[RFC] Rationalise OSM API authentication: one authorization flow, one place to enter the parameters

[wangi]

Updated after the feedback in comment:2.
This is a request for comments before any code is written. It grows out of #24889, #24890 and #24925, which are all the same underlying problem seen from three directions: authenticating against an openstreetmap-website instance that is not openstreetmap.org or openhistoricalmap.org.

Prerequisites. Three bug fixes should land first. They are independent of each other and can go in any order, but all are assumed by what follows.

• #24907 — a trailing slash on the API URL breaks OAuth authentication. It normalises the API URL in one place, which everything below relies on: there is no point agreeing on a single key for the OAuth parameters and the token while https://example.org/api and https://example.org/api/ are still two different keys.
• #24925 — authentication on upload fails against a server with no built-in client id. The OAuth parameters are stored under the API URL but were looked up with the host of the request, so the lookup missed and JOSM opened the browser with an empty client_id. Fixed there; the reason it belongs here is that nothing stopped the two keys drifting apart in the first place, and the next caller holding only one of them will hit the same wall. Settling that is point (d) below.
• #24932 — "Remove token" leaves the OAuth access token in the preferences. It fixes the method that (d) reworks, and what removing a token deletes has to be right before the token and the parameters share one key.

1. What the code looks like today

Everything belonging to a server is keyed two different ways. OAuth20Parameters.rememberPreferences() stores the parameters under the API URL, while OAuthAccessTokenHolder and CredentialsManager key the token by host, since getAccessToken()/setAccessToken() reduce whatever they are given with URI.create(api).getHost(). A caller therefore has to know which of the two it is holding, and nothing says which. #24925 is what that costs in practice.

Three authorization procedures are offered by AuthorizationProcedure, but:

• SEMI_AUTOMATIC is dead. It is referenced only by its own getText() and getDescription(). OAuthAuthorizationWizard.refreshAuthorisationProcedurePanel() throws UnsupportedOperationException for it, and nothing constructs a wizard with it.
• FULLY_AUTOMATIC does not use the wizard window at all. showDialog() short-circuits straight to authorize() for OAuth 2, so the wizard, its header panel, its "Accept Access Token" button and FullyAutomaticAuthorizationUI only ever appear for MANUALLY.
• MANUALLY is expert-only (ExpertToggleAction.addVisibilitySwitcher) and duplicates the advanced parameters panel inside a second tab of its own window.

OsmConnection.OAuthAccessTokenFetcher, the static fetcher field and OAuthAuthorizationWizard.obtainAccessToken(URL) are also dead: the fetcher is assigned in MainApplication and in two test harnesses but never invoked, because the upload path calls obtainOAuth20Token() directly.

AdvancedOAuthPropertiesPanel has problems of its own:

• oauth.settings.use-default is a single global boolean, not per API URL, so the choice made for one server is applied to the next one.
• When it is ticked, rememberPreferences() calls OAuthParameters.createDefault() with no argument, i.e. the parameters of the default OSM API, and stores them under the default OSM API's key, whichever server is actually being configured.
• There is no field for the access token; that lives only in the separate manual window.
• setChildComponentsEnabled() only walks JosmTextField and JLabel children.
• The field labelled "Redirect URL" is still called tfRequestTokenURL, left over from OAuth 1.0a.

Finally, the endpoint defaults are a guess. OAuthParameters.getDefaultOAuth20Parameters() first tries RFC 8414 discovery at <scheme>://<host>/.well-known/oauth-authorization-server, which openstreetmap-website does answer — for openstreetmap.org it returns

"authorization_endpoint":"https://www.openstreetmap.org/oauth2/authorize"
"token_endpoint":"https://www.openstreetmap.org/oauth2/token"

— but if that request fails, or the resource is marked offline, the fallback is OAuth20Parameters(clientId, secret, baseUrl, …) with baseUrl = the API URL, which produces <api url>/authorize and <api url>/token. Those are wrong for every openstreetmap-website instance, so whether a third-party server works at all currently depends on whether one HTTP request to a well-known URL succeeded, and the user is given no hint that this is what happened.

2. Proposal

(a) Basic Authentication for experts only. Keep it as an OSM API authentication method, as a quick way to test one's own server, but show the option only in expert mode — or when it is the method currently selected, so that a profile still using it does not lose sight of its own setting. As today, AuthenticationPreferencesPanel.updateAcceptableAuthenticationMethods() keeps it disabled for the default OSM API, which does not accept it. (Originally proposed for removal; changed after comment:2.)

Publishing /.well-known/oauth-authorization-server is the supported way for a third-party instance to work with JOSM without needing a client id built into JOSM, and it is not documented anywhere a server operator would find it; see the wiki page in section 3.

(a2) Remove what is left of OAuth 1.0a. The mechanism is already gone — OAuthVersion has only OAuth20 and OAuth21 — but the vocabulary and some classes outlived it:

• OsmPrivileges and OsmPrivilegesPanel: the 1.0a privilege checkboxes. OAuth 2 uses OsmScopes instead. OsmPrivilegesPanel is used only by FullyAutomaticAuthorizationUI, and OsmPrivileges only by that panel and its own test, so both go with (b)/(c).
• FullyAutomaticPropertiesPanel: the username/password panel of the old screen-scraping flow. Already completely unused — referenced only by its own unit test.
• AccessTokenInfoPanel: used only by FullyAutomaticAuthorizationUI.
• The preference keys oauth.access-token.key and oauth.access-token.secret are 1.0a token storage. They are still registered as sensitive keys in AbstractPreferences and still handled in UserIdentityManager.preferenceChanged(), but nothing writes them any more.
• Stale wording: OAuthAuthenticationPreferencesPanel's class javadoc still says "The preferences panel for the OAuth 1.0a preferences"; AuthenticationPreferencesPanel throws "One of OAuth 2.0, OAuth 1.0a, or Basic authentication must be checked"; AdvancedOAuthPropertiesPanel names its fields tfConsumerKey / tfConsumerSecret although they are labelled Client ID and Client Secret, and tfRequestTokenURL although it is labelled Redirect URL.
• OAuthAuthenticationPreferencesPanel.AlreadyAuthorisedPanel adds a JLabel reading "Access Token Secret:" with no field after it. OAuth 2 tokens have no secret; the label is a leftover and is visible in the preferences today.

(b) and (c) One authorization flow, no wizard window. Delete AuthorizationProcedure, OAuthAuthorizationWizard, FullyAutomaticAuthorizationUI, FullyAutomaticPropertiesPanel, ManualAuthorizationUI, AbstractAuthorizationUI, AccessTokenInfoPanel, OsmPrivilegesPanel and the dead OAuthAccessTokenFetcher. What remains is one "Authorize now" button that runs the browser-based OAuth 2 dance with the parameters currently shown in the panel. The manual path stays: only its window goes, and a token obtained outside JOSM is pasted into the access token field of (d).

(d) Access token becomes a field of the advanced parameters. Add "Access Token" to AdvancedOAuthPropertiesPanel, next to Client ID, Client Secret, Redirect URI, Authorize URL and Access Token URL. Show the whole group whenever "Use default settings" is unticked, rather than hiding it behind a second checkbox. Pasting a token then stores it against the same server as the parameters it was pasted next to, rather than depending on which of the two keys the caller happened to be holding. As the fallback for when everything else breaks, a pasted token must work with nothing else: no client id, no endpoints, no RFC 8414 discovery, no remote control and no browser. Today it cannot, because OAuth20Token requires complete OAuth20Parameters.

While doing this, settle on one key for everything belonging to a server: the host. JosmPreferencesCredentialAgent already stores the token, and the parameters it was issued with, under oauth.access-token.parameters.OAuth20.<host> — the same prefix rememberPreferences() writes to under the API URL, so a configured server ends up with two parameter entries. With the host, rememberPreferences() writes the entry the token store already uses, every stored token stays valid, and only custom parameters need reading once from their old key. Removing a token must then remove the token only, not the client id the user entered. The global oauth.settings.use-default goes too: "use default settings" simply means that no custom parameters are stored for that host.

(e) Tooltips. Each field explains what it holds and where it comes from, e.g.:

• Client ID — the Client ID of an OAuth 2 application registered on this server. On an openstreetmap-website server, register it under My Settings > OAuth 2 applications.
• Client Secret — only needed if the application is registered as confidential. The applications JOSM has built in are public clients and have none.
• Redirect URI — must match exactly the redirect URI registered with the application. JOSM listens on http://127.0.0.1:8111/oauth_authorization, which needs remote control enabled.
• Authorize URL / Access Token URL — taken from the server's RFC 8414 metadata when it publishes any; on openstreetmap-website these are <site>/oauth2/authorize and <site>/oauth2/token. Note <site>, not the API URL. When discovery fails they are pre-filled with exactly those, rather than today's <api url>/authorize, and stay editable.
• Access Token — paste a token issued outside JOSM instead of pressing "Authorize now".

The permission scopes JOSM requests should also be listed somewhere, since the application has to be registered with at least read_prefs, write_api, write_notes, read_gpx and write_gpx.

(f) Finish off authorization on upload. #24925 makes it work and stops it opening a browser when the client id is empty, but it leaves the user with the generic "no Access Token configured, open the Preferences Dialog" message. With the access token and the parameters in one place (d), this can become a message that says which server has no client id and offers to open the authentication preferences there and then.

Two smaller things in the same method: obtainOAuth20Token() saves the token under OsmApi.getOsmApi().getServerUrl() while the caller reads it back with the request URL — harmless today only because the holder reduces both to the host, and another reason to settle (d) — and the while (done.getCount() >= 0 && counter < 5) loop condition is always true, so it should just be a bounded await.

3. Feedback

1. Is dropping basic authentication acceptable? openstreetmap.org, openhistoricalmap.org and OpenGeofiction no longer need it, and the one other instance we know of can publish the RFC 8414 metadata and use OAuth 2 instead. We cannot survey private deployments, though, and osm-server.username / osm-server.password may be read by plugins. If it goes, what migration do you want for a profile that still has osm-server.auth-method=basic — switch it silently to OAuth 2, or warn once on startup? — No: it stays, for experts; see (a).
2. Is deleting the wizard and the manual window acceptable? Two of the three procedures are already dead or expert-only, but these are public classes and a plugin may reference them. Is the usual deprecation cycle wanted here, or is removal fine given @since churn in this area? — A manual path must stay: it does, as the access token field; see (b)/(c) and (d). Deprecation cycle or removal: still open.
3. Host or API URL as the key for tokens and parameters — which would you prefer? — Host; see (d).
4. Should oauth.settings.use-default become per API URL? Today one global flag decides it for every server. — Yes, it falls out of (d).
5. What should happen when RFC 8414 discovery fails? This matters most for third-party servers, because discovery is the main way a third-party instance works without a client id built into JOSM. Keep guessing <api url>/authorize, which is wrong for every openstreetmap-website instance? Guess <site>/oauth2/authorize instead? Or leave the fields empty and tell the user the endpoints must be entered by hand? — <site>/oauth2/…, editable, which is the manual fallback of question 2; see (e).
6. Would you accept a short wiki page for server operators saying what an openstreetmap-website instance has to do to work with JOSM: publish /.well-known/oauth-authorization-server, register a public client with redirect URI http://127.0.0.1:8111/oauth_authorization, and grant the scopes JOSM asks for? We can write it up as part of this work. — Yes.

4. Out of scope

OAuth20Token.sign() compares hosts with this.oauthParameters.getApiUrl().contains(host), so a token for api.openstreetmap.org would also sign a request to the host openstreetmap.org. That is pre-existing and security-adjacent; it deserves its own ticket rather than being folded in here.

[wangi]

This would make sense for the next milestone after 26.09: 26.11. This would allow the 26.09 release to support both Basic Auth and OAuth 2 for the OpenGeofiction use-case, aiding the migration to OAuth there.

[stoecker]

1) I'm against Basic Auth dropping. While that clearly is no longer acceptable for production services it will always be a fast way to setup own services. It can be hidden from normal users, but should be available to experts.

2) I don't like dropping the manual process. That was always a fallback when processes changed. It can be streamlined, reduced, ... but there should be a method if everything breaks and current maintainers can't care to "use this crude workaround".

3) What better fits.

4) When it fits.

5) And that points back to answer 2 :-)

6) Wiki is free. You can add there anything relevant to JOSM.

[wangi]

Thanks for the quick feedback.

1. Believe me, I appreciate the simplicity of Basic Auth, especially given how OAuth has ended up as the main time sink for the recent OGF migration. However, what are you testing against - an old pull of openstreetmap-website?

2. Plan would be to delete the manual window, but to keep the ability to paste in an access token and thereby effect a manual authorisation.

[wangi]

Thanks, i've updated the description to match; the changes are marked in place, with answers in section 3.

1) Basic Auth stays, for experts. The option is shown only in expert mode, or when it is the method currently selected, so that a profile still using it doesn't lose sight of its own setting. It stays disabled for the default OSM API, as it is today. The code behind it is untouched, so the migration question goes away.

2) A manual path stays. The window goes, but pasting an access token stays, as a field next to the other OAuth parameters. To make it a real fallback for "if everything breaks", a pasted token must work with nothing else: no client id, no endpoints, no RFC 8414 discovery, no remote control, no browser. That isn't true today — OAuth20Token needs complete OAuth20Parameters — so it becomes an explicit requirement. One part of your question is still open: these are public classes, but I assume there is no need for a deprecation cycle for OAuthAuthorizationWizard, ManualAuthorizationUI and friends, and a straight removal is fine?

3) The host. JosmPreferencesCredentialAgent already stores the token, and the parameters it was issued with, under oauth.access-token.parameters.OAuth20.<host>. That is the same prefix rememberPreferences() writes to under the API URL, so today a configured server has two parameter entries. Keying by host means every stored token stays valid and nobody has to authorize again; only custom parameters are read once from their old key.

4) This falls out of 3: "use default settings" just means no custom parameters are stored for that host, and the global oauth.settings.use-default goes.

5) When discovery fails, the endpoints are pre-filled with <site>/oauth2/authorize and <site>/oauth2/token — correct for any openstreetmap-website instance, unlike today's <api url>/authorize — and stay editable. So the manual fallback covers this as well.

6) I'll write the server-operator page.

While checking 3) I found #24932: "Remove token" never deleted the token itself, only the parameters stored with it, so the bearer token stayed in the preferences in clear text. That patch touches the same method this ticket reworks, and what removing a token deletes has to be right before the token and the parameters share one key, so I've added it as a third prerequisite alongside #24907 and #24925.

[anonymous]

Replying to wangi:

1. Believe me, I appreciate the simplicity of Basic Auth, especially given how OAuth has ended up as the main time sink for the recent OGF migration. However, what are you testing against - an old pull of openstreetmap-website?

Well. A KI can probably implement an OSM-API for own data in a matter of minutes :-) A human in less than a day.