[PATCH] Avoid leaking `Authorization` header on redirects in HttpClient

[Kmpopppe]

What steps will reproduce the problem?

1. Install "Direct Download" plugin for downloading GPS tracks from openstreetmap.org
2. Select any GPS track and click download

What is the expected result?

Download and open of selected GPS track

What happens instead?

"Error fetching URL" error message with the "correct" URL (that oneself can use to correctly download the GPS data).

Please provide any additional information below. Attach a screenshot if possible.

URL:https://josm.openstreetmap.de/svn/trunk
Repository:UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
Last:Changed Date: 2022-03-06 10:38:55 +0100 (Sun, 06 Mar 2022)
Build-Date:2022-03-07 02:31:09
Revision:18387
Relative:URL: ^/trunk

Identification: JOSM/1.5 (18387 en_GB) Windows 11 64-Bit
OS Build number: Windows 10 Pro 2009 (22000)
Memory Usage: 865 MB / 7241 MB (620 MB allocated, but free)
Java version: 1.8.0_321-b07, Oracle Corporation, Java HotSpot(TM) 64-Bit Server VM
Look and Feel: com.sun.java.swing.plaf.windows.WindowsLookAndFeel
Screen: \Display0 1920×1080 (scaling 1.00×1.00)
Maximum Screen Size: 1920×1080
Best cursor sizes: 16×16→32×32, 32×32→32×32
System property file.encoding: Cp1252
System property sun.jnu.encoding: Cp1252
Locale info: en_GB
Numbers with default locale: 1234567890 -> 1234567890
VM arguments: [-Djava.security.manager, -Djava.security.policy=file:<java.home>\lib\security\javaws.policy, -DtrustProxy=true, -Djnlpx.home=<java.home>\bin, -Djnlpx.origFilenameArg=%UserProfile%\AppData\LocalLow\Sun\Java\Deployment\cache\6.0\56\1ee8cfb8-5fd7ddc6, -Djnlpx.remove=false, -Djava.util.Arrays.useLegacyMergeSort=true, -Djnlpx.splashport=54345, -Djnlp.application.href=https://josm.openstreetmap.de/download/josm.jnlp, -Djnlpx.jvm=<java.home>\bin\javaw.exe]
Dataset consistency test: No problems found

Plugins:
+ DirectDownload (35893)
+ notesolver (0.3.3)
+ utilsplugin2 (35893)

Last errors/warnings:
- 00007.641 W: Unable to request certificate of https://roottest-g3.pkioverheid.nl
- 00008.605 W: Unable to request certificate of https://roottest-g3.pkioverheid.nl
- 00013.715 W: java.nio.file.FileSystemException: %UserProfile%\AppData\Local\Temp\josm_exec_powershell_660736094689018128.txt: Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess verwendet wird.
- 00013.715 W: Unable to request certificate of https://grca.nat.gov.tw
- 00022.682 W: Region [TMS_BLOCK_v2] : Problem verifying disk.
- 01608.119 W: java.io.IOException: stream is closed
- 01745.488 W: java.io.IOException: stream is closed
- 01761.383 W: java.io.IOException: stream is closed

[taylor.smock]

It looks like OSM is now redirecting to Amazon S3 for downloads, and apparently they don't like having more than one authorization.

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidArgument</Code><Message>Only one auth mechanism allowed; only the X-Amz-Algorithm query parameter, Signature query string parameter or the Authorization header should be specified</Message><ArgumentName>Authorization</ArgumentName><ArgumentValue>OAuth oauth_consumer_key="..trimmed..", oauth_nonce="..trimmed..", oauth_signature="..trimmed..", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1647988292", oauth_token="..trimmed..", oauth_version="1.0"</ArgumentValue><RequestId>..trimmed..</RequestId><HostId>..trimmed..=</HostId></Error>

[taylor.smock]

In 35944/osm:

DirectDownload: fix #21935 and #21952

#21935: "Error fetching URL" when trying to download a GPS track

This was caused due to a change in where OSM stores GPX data files.
OSM now uses s3 buckets, and redirects using a signed URL. S3 does
not like multiple authentication methods.

#21952: Only the first tag in the Tags column in "Download Track" window is shown

This was caused due to only keeping the last encountered tag.

[taylor.smock]

In 35945/osm:

DirectDownload (dist): see #21935 and #21952

#21935: "Error fetching URL" when trying to download a GPS track

This was caused due to a change in where OSM stores GPX data files.
OSM now uses s3 buckets, and redirects using a signed URL. S3 does
not like multiple authentication methods.

#21952: Only the first tag in the Tags column in "Download Track" window is shown

This was caused due to only keeping the last encountered tag.

[taylor.smock]

This is also a core problem. Its just that most people don't download specific GPS traces. And I didn't think to check this until now.

Anyway, steps to reproduce:
1) Log in to OpenStreetMap
2) File -> Open Location
3) Enter https://www.openstreetmap.org/user/vorpalblade/traces/3958616 (trace is identifiable)
4) Get a bad request window, logging will show

org.openstreetmap.josm.io.OsmApiException: ResponseCode=400, Error Body=<<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidArgument</Code><Message>Only one auth mechanism allowed; only the X-Amz-Algorithm query parameter, Signature query string parameter or the Authorization header should be specified</Message><ArgumentName>Authorization</ArgumentName><ArgumentValue>OAuth oauth_consumer_key="..trimmed..", oauth_nonce="..trimmed..", oauth_signature="..trimmed..  0", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1648039451", oauth_token="..trimmed..", oauth_version="1.0"</ArgumentValue><RequestId>EYT7FEAG08DG2Z38</RequestId><HostId>/mfw6JoZ83JZGcAHrMh9KsPkoUmoRhFlY/CwZdnp+FXsNycldpN34jKJL1RVWfY9h+L6rI5EaD4=</HostId></Error>>
org.openstreetmap.josm.io.OsmApiException: ResponseCode=400, Error Body=<<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidArgument</Code><Message>Only one auth mechanism allowed; only the X-Amz-Algorithm query parameter, Signature query string parameter or the Authorization header should be specified</Message><ArgumentName>Authorization</ArgumentName><ArgumentValue>OAuth oauth_consumer_key="..trimmed..", oauth_nonce="..trimmed..", oauth_signature="..trimmed..", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1648039451", oauth_token="..trimmed..", oauth_version="1.0"</ArgumentValue><RequestId>EYT7FEAG08DG2Z38</RequestId><HostId>/mfw6JoZ83JZGcAHrMh9KsPkoUmoRhFlY/CwZdnp+FXsNycldpN34jKJL1RVWfY9h+L6rI5EaD4=</HostId></Error>>

We do the following

• Convert https://www.openstreetmap.org/user/vorpalblade/traces/3958616 to https://www.openstreetmap.org/trace/3958616/data in DownloadGpsTask#L71​
• Redirects are as follows:
  • https://www.openstreetmap.org/trace/3958616/data
  • -> https://www.openstreetmap.org/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBN3ROb2c9PSIsImV4cCI6bnVsbCwicHVyIjoiYmxvYl9pZCJ9fQ==--26fa060911e78b5e82e4730c2e0167d70a9a3c30/3958616.gpx.gz?disposition=attachment
  • -> https://openstreetmap-gps-traces.s3.dualstack.eu-west-1.amazonaws.com/drn9wcmfplehgrka3wybfz3oj602?response-content-disposition=attachment%3B%20filename%3D%223958616.gpx.gz%22%3B%20filename%2A%3DUTF-8%27%273958616.gpx.gz&response-content-type=application%2Fgzip&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIASQUXHPE7AMJQRFOS%2F20220323%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20220323T130611Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host&X-Amz-Signature=e1a782fd5c06cbd950efa0bb3e4324148f2c83d77b2254a65077d40a1dbc685e

For the first two, we should be able to send authorization headers. For the last one, we should not send the authorization header for security reasons (different domain name). The latter was a CVE in python-requests (​CVE-2014-1829).

[taylor.smock]

Initial (minimal) patch

[taylor.smock]

attachment:21935.patch​ is fairly minimal. I'm trying to figure out if we should be adding any response headers (i.e., server response has Authorization header that is expected by the next server).

[taylor.smock]

Add non-regression test

[taylor.smock]

Does anyone have any use cases where dropping authorization headers when redirecting to a different host would break a workflow? This is (apparently) standard practice in web browsers, so I would presume not. But there are also 4 days left in the current milestone.

[taylor.smock]

In 18437/josm:

Fix #21935: Avoid leaking Authorization headers on redirects in HttpClient

This was found due to a change in where OSM stores GPX data files.
OSM now uses s3 buckets, and redirects using a signed URL. S3 does
not like multiple authentication methods.

[stoecker]

Milestone renamed