Support for discovering API endpoints via .well-known URIs (SPEC-121)

[matrixbot]

Documentation: https://docs.google.com/document/d/1OdEj06qA7diURofyonIMgTR3fB_pWf12Txye41qd-U4/edit, https://docs.google.com/document/d/1vF-uWlUYmf1Xo161m871H1upJbwiIPeikWGWzaE_lrU/edit#
Author: @maxidor, others
Shepherd: @uhoreg
PRs: #1359

We have several reasons why we might want to use .well-known URIs to discover API endpoints:

• Clients which can't query SRV records (e.g. webclients trying to discover the C-S API for a given user ID's domain)
• Admins who don't want to use the /_matrix convention, and adhere to RFC5785 instead
• Admins who can't create SRV records
• Dynamic .well-known URIs to help discover nomadic homeservers

See also SYWEB-224 and SYN-167

We should just get on and do it. Unsure whether SRV should trump .well-known URIs or not for server-server traffic.

(Imported from https://matrix.org/jira/browse/SPEC-121)

(Reported by @ara4n)

[matrixbot]

Jira watchers: @ara4n

[matrixbot]

Links exported from Jira:

relates to https://github.com/matrix-org/matrix-doc/issues/434

[matrixbot]

Some potential considerations regarding SSL SNI (server name indicators):

eternaleye (IRC)
M-Erik: ISTR there having been a problematic interaction in the past between SRV and SNI with how Matrix used them, which showed up when one person using CloudFlare tried to set it up
Jul 6 22:45
M-Erik: And there was a discussion re: using .well-known/ JSON to do what is currently being done with SRV
Jul 6 22:46
M-Erik: Did that wind up going anywhere?

...

[02:32] [01:42:58] Arathorn: IIRC, the issue with CloudFlare was that SNI + SRV sends SNI for the name the SRV record is on, while the cert was for the name it points to
[02:32] [01:44:30] Arathorn: The relevant RFC lays out that the SNI being for the domain the record is on is the correct one: https://tools.ietf.org/html/rfc6125#section-6
[02:32] [01:44:32] Title: RFC 6125 - Representation and Verification of Domain-Based Application Service Identity within Internet Public Key Infrastructure Using X.509 (PKIX) Certificates in the Context of Transport Layer Security (TLS) (at tools.ietf.org)
[02:32] [01:45:56] Arathorn: So _matrix._tcp.matrix.org would use an SNI value of "matrix.org", even if the SRV record points to "blargfoo.org"
[02:32] [01:47:52] Arathorn: #433 came out of that discussion, but I dunno if a ticket got filed for that actual issue
[02:32] M-NEBot/#matrix [01:48:06] https://matrix.org/jira/browse/SPEC-121 : Support for discovering API endpoints via .well-known URIs [Pending Triage,P2,reporter=Matthew Hodgson,assignee=]
[02:32] [01:48:38] Arathorn: If well-known is used, though, that "resolution" happens in the application level, and SNI will almost certainly be "blargfoo.org" there.
[02:32] [01:48:47] Arathorn: Not sure what's better, though
[02:32] [01:49:46] Arathorn: The current behavior prevents some common things on existing sites from coexisting with Matrix, while the latter behavior would make running multiple Matrix services on one port infeasible.
[02:32] [01:50:11] Arathorn: I'll also note that CNAME behaves similarly to SRV here - the "from" is in the SNI, not the "to"

-- @ara4n

[madduck]

I was asked in element-hq/riot-android#727 to chime in here. This is a high-priority feature we should add ASAP, as I think it greatly impacts user experience, and currently in negative ways:

When I try to log in to Riot-Android (F-Droid version here), I am asked for my "email address or username". There's an option to specify custom server options, but it's labelled advanced and I should not need to touch it.

However, when I try to log in as madduck@madduck.net with the correct password, I am told the password is wrong. This is probably because Riot tries to log me in to the default matrix.org homeserver with the username madduck@madduck.net, when in fact I expect it to do a SRV lookup on the domain to figure out the homeserver:

% dig +short SRV _matrix._tcp.madduck.net
0 0 8448 matrix.madduck.net.

I also tried @madduck:madduck.net, but that also fails.

If I enter https://matrix.madduck.net:8448/ manually into the advanced server field and change the username to just madduck, then login works. But this is not what users expect, especially not those who don't want to be confronted with "advanced" settings.

XMPP clients do use SRV records to determine which server to login to (_xmpp-client._tcp.example.org), but I am willing to accept that SRV records are for federation, and that well-known files are better suited for client-side apps.

It shouldn't matter anyway. The important thing IMHO is to make sure that users can log in with least effort and least surprises.

[dbkr]

This has just come up again on #matrix-dev: https://matrix.to/#/!XqBunHwQIXUiqCaoxq:matrix.org/$1492701491891267IJVIs:matrix.org

I've put what I think are the options into a spec proposal google doc: https://docs.google.com/document/d/1RHQ5DEA6_2IZ5m2UIysPf8CuYaup-NrincEbV4Ef3jE/edit#heading=h.ouq2tlo7ae5z

[ara4n]

Let's try to avoid this getting lost for another 2 years. A major reason for Mastodon's success seems to be not having to piss around with DNS, for instance.

Does anyone actually object to just adding .well-known URIs as the preferred discovery mechanism, keeping SRV as backup for weird people (e.g. Leo, Amdocs) who would find it easier to do DNS rather than edit their web root?