Proposal for clarifying and improving review process for MSCs

[ara4n]

Documentation: https://docs.google.com/document/d/1miOde4v98lwxZNQqT40ksBcXGdliign8rfmobDfC5u0/edit

This was originally put together as notes of an IRL chat with the intention of turning it into a proper public MSC asap... except it got completely blocked on me wanting to first get alignment between whether to use discourse & github before releasing it (and then getting sucked into other stuff).

Releasing it now better late than never, as it overlaps significantly with #1421.

Best place for discussion is probably #matrix-spec-process:matrix.org or #discourse-matrix-spec:thebeckmeyers.xyz (the one for #1421)

[ara4n]

This is intended to obsolete #1308

[erikjohnston]

Personally, I think the main objectives should be to ensure that:

1. It is easy to follow and to contribute to proposals and their discussions asynchronously
2. Contributions are thought out and constructive.

The first is important as it allows the global matrix community to engage in discussions and proposals, ensuring that stakeholders in the community have a chance to be represented in proposals that effect their areas. The more effort required to follow new proposals the less likely people will engage.

The second is a more nebulous goal, but the tooling and processes we use will have an impact as to exactly how people engage. For example, using tooling that focuses on small inline comments will encourage nitpicking over exact wording, rather than discussing larger ideas introduced.

I feel like using GitHub as the main venue fulfils those two goals the best, as long as we have recommendations on how to contribute to discussions¹. The majority of community members that interact with matrix APIs will have GitHub accounts and be used to the tooling, reducing the friction for them to "watch" a proposal or to jump in to the discussion, without having to deal with a new and unfamiliar platform.

While proposals are likely to often spark lots of discussions, they're not the only way that the community interacts. Contributing to matrix projects─either via PRs or issues─will require using GitHub, and some of those will also spark large amounts of discussion. Having these different interactions use the same tools has obvious benefits, including helping ensure that contributions to issues/PRs are as civil and constructive as those for proposals.

(Its worth noting that Rust has tried this approach and it works reasonably well². They also have a bot that we can easily use that does everything needed in the proposal.)

Given the above, I think the alternatives must have material benefits to be considered being used instead of plain GitHub.

Some of the points in favour of using Discourse + Google Docs mentioned in the proposal are:

[Discourse:] All discussions can be threaded, letting insightful content be upvoted and exposed and less insightful stuff be downvoted and hidden; making it easier to organise, digest & archive the discussion

AFAIK Discourse does not support this. The UI is a flat structure that allows you to link to the message you are replying to, but does not expose a tree view. The only reactions exposed is a "like" button.

The links back to the quoted message is useful feature that GH doesn't have, but fundamentally they are both designed to be consumed as a linear timeline.

Discourse lets you split threads up and move them into their own topics as a way to manage off-topic discussion and generally curate what’s going on.

Potentially useful, but I think its going to be a lot rarer for a discussion to become offtopic than it is in e.g. a matrix channel. If it is rare then simply copying, editing and/or deleting comments may be good enough.

Annotations in Google Docs stick around as the document evolves until they are explicitly resolved.

Having useful contributions disappear simply because the owner thinks its resolved feels annoying. I would be in favour of ensuring that the vast amount of useful discussion happens outside of inline comments.

Ease of editing rich text content

This is important. I feel like rapid changes to the content shouldn't happen during the MSC process, not least because it becomes confusing to follow the comments if the doc has changed a lot. It may be good to suggest that all potential proposals should be posted in #matrix-spec to let people help suggest changes to increase the clarity of the proposed ideas involved (e.g. fixing up wording, adding examples, clarifications etc.).

If there are significant changes to the core ideas, it may be better to start a new MSC (with links back to the old one), rather than having a single, long, sprawling

Historical experience

It unclear how discourse would fundamentally change how the linked discussions happened.

---

At the end of the day it feels like Discourse gives a more glossy UI, while losing having inline comments and doc change notifications in one place. My personal inclination is to use github (and existing tooling) for now and if it turns out that there are features we need from Discourse we can revisit later.

---

^1. This could include recommendations about when to use inline comments, to be thoughtful about responses by responding to all points rather than nitpicking particular points, don't force push changes etc.

^2. See the blog series "Listening and trust" at http://aturon.github.io/

[uhoreg]

The links back to the quoted message is useful feature that GH doesn't have

If needed, links to specific comments can be added manually in GH. The UX probably won't be as smooth, of course, but may help in some cases of long threads.

[turt2live]

There's been some discussion over the last few days in #discourse-matrix-spec:thebeckmeyers.xyz about different approaches to take for this workflow. This is an attempt to summarize the discussion (if I've misrepresented anything, please tell me so I can correct it).

The discussion quickly turned to giving the GitHub approach a chance in a real-world scenario given Git has really good versioning support and Google Docs/Discourse are lacking in this area. There are concerns that we may need a third MSC to discuss the GitHub idea given it is so wildly different from the core ideas here and on #1426, however I'm documenting it here to have it somewhere.

Building off of what Erik said, there's concerns about using GitHub instead of GitLab/Gitea/something due alternatives being more open-source friendly. This is also mirrored by various unprompted conversations with people outside the room who aren't aware of this proposal in the first place.

Moving to gitlab/etc now would mean that we're particularly set on using gitlab when other options may be more suited. Given the time capacity of the team, and the ease of using something familiar, a decision was made to keep using github for now. This same ease of use is also incorporated into the argument of using git for proposals, at least for a trial run: if the process fails then it is relatively easy to pick up the pieces and move them to another platform, be it Discourse, GitLab, or something completely different. Going with another platform to trial the process means that failing does not have a very usable safety net (it's harder to import into Github than it is to export).

Losing the powerful capabilities of Google Docs (namely annotations and the wysiwyg) isn't great, however the alternatives provided by GitHub are worth a shot. Using github also gives the advantage of re-using existing tooling to manage the timings and state machine of proposals.

There was some discussion about how many repositories we should have for the proposal process and ultimately decided we'd re-use matrix-doc. Rust, the community much of this is based upon, has two repositories: one for RFCs, and one for the spec. If Matrix were to go this same direction, we'd eventually have conflicts in our numbering given proposals are already assigned numbers in matrix-doc. Instead of having an MSC number acquired by matrix-doc to open a PR on another repository, it seems best to keep everything in one place and consider breaking it out in the future if need be.

One possibility to deal with the numbering issue is to have the proposal PR be the MSC number. This has the benefit of github being really good at tracking links between things, however it does mean there's no canonical place for tracking the state of a given proposal. Concerns about people leaving comments on the issue instead of the PR were raised, and the solution to lock the conversation thread on the issue was suggested (and enacted in the case of State Resolution: Reloaded). This should hopefully drive conversation to the appropriate place rather than have feedback lost in the sea of potential threads.

Using a different repository would also mean being able to have a different contributing guidelines document. Although we can put it into matrix-doc's, it could quickly become unhelpful to have that much information in a single document. This doesn't feel significant enough at this time to warrant a whole new repository be made, however. The contributing document can also easily go in a subdirectory for now.

A potential solution for keeping issue counts low is to re-use an existing issue on matrix-doc to turn it into an MSC if it is appropriate. This has a number of downsides which ultimately lead to it not being implemented. The first is that it doesn't allow for multiple proposals to solve the same issue: by having a 1:1 relationship between issue and proposal, competition is suppressed. That 1:1 relationship also means that proposals which target multiple issues are harder to do. Given that it's fairly common for a proposal to target multiple issues/problems, it doesn't make a whole lot of sense to limit them to just one problem at a time. The final point was that re-using an issue can easily lead to confusion and possibly have the thread feel unrelated to the proposal while it was discussing potential solutions. This same confusion concern is what lead to today's proposal process being changed to not recommend re-using a given issue.

The last concern about not having a canonical issue for tracking the proposal state machine is the possibility that proposal PRs get closed due to massive formatting problems or major restructuring. If the proposal were to have 2 PRs, one closed, then there would be two MSCs out there even though the PRs are the same logical proposal. The likelihood of this happening is relatively low, however.

Using Gists was also suggested during the conversation, however it was met with immediate negative feedback (with no reasoning to back it up or support it).

Discussion around having a Matrix room per-MSC was brought up, however the decision was left out in the open: the proposer could opt to have a room if they like, however some thoughts on the subject are against having the room at all (due to concerns of room list size and having more places to have lost discussions, primarily).

A minor point about labelling proposal PRs to avoid accidental merging was brought up and solved by having a new label added: proposal-pr.

This discussion ultimately ended up including most of the timing and workflow points in #1426 and incorporating the highly valuable feedback of #1421 to weigh Discourse versus GitHub, ending with a decision to try GitHub for a while. GitHub's ease of portability gives it an advantage over other platform suggestions for a test bed of the workflow details, not to mention the conclusions referenced in the above paragraphs.

---

TLDR (at the end, because I'm mean):

We're trying out the GitHub workflow on State Resolution: Reloaded (#1441) to see how it goes. With this process, we're using the proposal PR's comment thread for major feedback and having inline comments be reserved for minor/formatting suggestions. We've also gone ahead and locked the conversation on #1441 to avoid accidentally having comments lost in a different thread. Provided the proposal passes review, it'll be merged pending a PR for the formal specification.

[ara4n]

The story on this in the end is that we experimented with a bunch of the options over a series of MSCs, and ended up with a new set of best practices, which are being written up as a plain spec PR at #1697.

[turt2live]

We ended up merging the spec PR (#1697), so labelling this as a success and closing. Future proposals to improve the process should be done on their own.