MSC4000: Forwards fill (/backfill forwards)

proposals/4000-forwards-fill.md

@@ -0,0 +1,150 @@
+# MSC4000: Forwards fill (`/backfill` forwards)
+
+Currently, `/messages` only backfills events when you paginate backwards with `?dir=b`.
+The limiting factor here is `/backfill` which only looks backwards to find preceding
+events.
+
+The spec just says this:
+
+> Since users in that room are able to request the history by the `/messages` client API
+> endpoint, it’s possible that they might step backwards far enough into history before
+> the homeserver itself was a member of that room.
+> 
+> To cover this case, the federation API provides a server-to-server analog of the
+> `/messages` client API, allowing one homeserver to fetch history from another. This is
+> the `/backfill` API.
+> 
+> *-- https://spec.matrix.org/v1.5/server-server-api/#backfilling-and-retrieving-missing-events*
+
+This proposal is after adding a way to `/backfill` forwards, also known as *forwards fill*.
+
+This also opens the door to adding backfill support to `/context` which returns events
+around a given event in both directions. Please note, that this MSC doesn't dictate when
+a server should `/backfill` or not. We can address this in a future MSC with spec or
+guidelines which is tracked by https://github.com/matrix-org/matrix-spec/issues/1281
+
+Navigating the DAG forwards is harder because by the nature of `prev_events`, which
+connect the DAG together, gives us no indication of what comes next. We could give a
+list of events that we have as forward extremeties and ask other homeservers for any
+successor events (events that reference the given events in their `prev_events`).
+
+## Proposal
+
+Add a new server API endpoint `GET /_matrix/federation/v1/forwards_fill/{roomId}` which
+is basically a mirror image of `/backfill`.
+
+Retrieves a sliding-window history of subsequent PDUs that occurred in the given room.
+Starting from the PDU ID(s) given in the `from_event_id` query parameter, the PDUs that
+succeed them are retrieved, up to the total number given by the `limit`.
+
+|   |   |
+--- | ---
+Rate-limited | No
+Requires authentication | Yes
+
+#### Request
+
+##### Request parameters
+
+Name| Type | Description
+--- | --- | ---
+`roomId` | `string` | **Required:** The room ID to backfill.
+
+##### Query parameters
+
+Name| Type | Description
+--- | --- | ---
+`limit` | `integer` | **Required:** The maximum number of PDUs to retrieve, including the given events.
+`from_event_id` | `[string]` | **Required:** The event IDs to find successors from. TODO: Should we spec a hard limit to the number of events you can specify?
+
+
+#### Responses
+
+Status | Description
+--- | ---
+`200` | A transaction containing the PDUs that succeeded the given event(s), up to the given limit.
+
+##### 200 response transaction
+
+Name| Type | Description
+--- | --- | ---
+`origin` | `string` | **Required:** The server_name of the homeserver sending this transaction.
+`origin_server_ts` | `integer` | **Required:** POSIX timestamp in milliseconds on originating homeserver when this transaction started.
+`pdus` | `[PDU]` | **Required:** List of persistent updates to rooms. Note that events have a different format depending on the room version - check the room version specification for precise event formats.
+
+
+## Potential issues
+
+Forwards fill has the same normal problems that `/backfill` has where the server will
+probably only exhaust a single chain of events up to the `limit` from the many given
+events to navigate from. This makes it harder to fill in all the gaps in one pass. But
+servers can keep hitting the endpoint with the new extremities until they've filled in
+everything. Is there a smarter heuristic guideline that servers should follow? Or
+something we should spec for how much depth/breadth a server should return given a list
+of many events to navigate from?
+
+For example with Synapse, when it does a `/messages?dir=b` and there are backward
+extremities, it just does a single `/backfill` with the closest 5 extremities and just
+moves on with the `/messages` response regardless of how much timeline is missing.
+Perhaps this is better solved by allowing the server to indicate where the gaps are so
+the client can choose if it's important to them to fill them in, see
+*[MSC3871](https://github.com/matrix-org/matrix-spec-proposals/pull/3871): Gappy
+timelines*.
+
+
+## Alternatives
+
+We could add a direction query parameter to `/backfill` but the semantics for are
+slightly different as backfill normally includes the events you requested with `v` as
+well preceding events versus with going forward, we just want to find the successors.
+
+
+## Security considerations
+
+We're only exposing messages according to the existing message history setting in the
+room (`m.room.history_visibility`). No extra data is exposed, just a new way to navigate
+and fetch it all.
+
+
+## Auxiliary references
+
+Other links:
+
+ - https://github.com/matrix-org/matrix-spec-proposals/pull/3030#discussion_r802851952
+ - https://github.com/matrix-org/matrix-spec/issues/601#issuecomment-1082198910
+ - https://github.com/matrix-org/matrix-public-archive/issues/72
+
+
+## Unstable prefix
+
+While this feature is in development, the endpoint can be used with the unstable prefix
+at `GET /_matrix/federation/org.matrix.msc4000/forwards_fill/{roomId}` instead of it's
+stable `/v1` location as described above.
+
+### While the MSC is unstable
+
+During this period, servers should optimistically try the unstable endpoint and look for
+the `404` `M_UNRECOGNIZED` error code to determine support (see [MSC3743: *Standardized
+error response for unknown
+endpoints*](https://github.com/matrix-org/matrix-spec-proposals/pull/3743)).
+
+### Once the MSC is merged but not in a spec version
+
+Once this MSC is merged, but is not yet part of the spec, servers should optimistically
+use the stable endpoint and fallback to the unstable endpoint to have maximum
+compatibility if desired.
+
+### Once the MSC is in a spec version
+
+Once this MSC becomes a part of a spec version, servers should optimistically use the
+stable endpoint. If a given server doesn't support the endpoint, another server in the
+room can be tried until all are exhausted.
+
+Servers can keep falling back to the unstable endpoint and are encouraged (not
+obligated) to serve the unstable fallback endpoint for a reasonable amount of time to
+help smooth over the transition for other servers. "Reasonable" is intentionally left as
+an implementation detail, however the MSC process currently recommends *at most* 2
+months from the date of spec release.
+
+
+