Modify

Opened 3 months ago

Closed 3 months ago

Last modified 3 months ago

#18812 closed defect (fixed)

JOSM sends invalid HTTP Accept headers

Reported by: mmd Owned by: simon04
Priority: normal Milestone: 20.03
Component: Core Version:
Keywords: http httpclient headers accept Cc:

Description

When downloading OSM raw data via the OSM API 0.6 /map endpoint, JOSM currently sends a non-conforming HTTP Accept header (according to RFC 7231).

See https://github.com/openstreetmap/openstreetmap-website/pull/2485#issuecomment-592441007 for details.

Attachments (0)

Change History (9)

comment:1 Changed 3 months ago by mmd

Quoting from the Github issue:

What they are currently sending is the following:

text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2

There are a number of issues here:

  1. * is not a valid mime type, the spec requires <MIME_type>/<MIME_subtype>
  2. The extra space after the semicolon is not permitted
  3. The number format in q=.2 is invalid, that should be q=0.2

You can try this out for yourself on the Rails console:

bundle exec rails console
Mime::Type.parse("text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2")
Traceback (most recent call last):
        1: from (irb):8
Mime::Type::InvalidMimeType ("*; q=.2" is not a valid MIME type)

comment:2 Changed 3 months ago by simon04

In 15967/josm:

see #18812 - HttpClient: debug logging of request headers

comment:3 Changed 3 months ago by simon04

Keywords: http httpclient headers accept added

Are you sure the headers are coming from JOSM? With --debug, I get the following console output:

2020-02-29 21:55:57.127 FINE: REQUEST HEADERS: {Accept-Encoding=gzip, deflate}
2020-02-29 21:55:57.307 INFO: GET https://api.openstreetmap.org/api/0.6/map?bbox=11.3826118,47.2644319,11.3867297,47.2661308 -> HTTP/1.1 200 (181 ms)
2020-02-29 21:55:57.308 FINE: RESPONSE HEADERS: {Transfer-Encoding=[chunked], Keep-Alive=[timeout=5, max=100], null=[HTTP/1.1 200 OK], Strict-Transport-Security=[max-age=31536000; includeSubDomains; preload, max-age=31536000; includeSubDomains; preload], Cache-Control=[private, max-age=0, must-revalidate], Server=[Apache/2.4.29 (Ubuntu)], Content-Disposition=[attachment; filename="map.osm"], Connection=[Keep-Alive], Content-Encoding=[gzip], Date=[Sat, 29 Feb 2020 20:55:57 GMT], Content-Type=[text/xml; charset=utf-8], Expect-CT=[max-age=0, report-uri="https://openstreetmap.report-uri.com/r/d/ct/reportOnly", max-age=0, report-uri="https://openstreetmap.report-uri.com/r/d/ct/reportOnly"]}

comment:4 Changed 3 months ago by mmd

I started josm with -Djavax.net.debug=all and can see those headers in the log. I also tested against a local server as well, so no other user was triggering the issue. Maybe this is some default header added by Java?

comment:5 Changed 3 months ago by simon04

Milestone: 20.03
Owner: changed from team to simon04
Status: newassigned

Setting osm-server.url=http://api.openstreetmap.org/api reveals this HTTP request in Wireshark:

GET /api/0.6/map?bbox=11.3833058,47.2647132,11.3859022,47.2657835 HTTP/1.1
User-Agent: JOSM/1.5 (15950 SVN en_GB) Linux Arch Linux Java/1.8.0_242
Accept-Encoding: gzip, deflate
Host: api.openstreetmap.org
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive

HTTP/1.1 301 Moved Permanently
Date: Sat, 29 Feb 2020 23:21:21 GMT
Server: Apache/2.4.29 (Ubuntu)
Cache-Control: max-age=31536000
Expires: Tue, 19 Jan 2038 03:14:07 GMT
Location: https://www.openstreetmap.org/api/0.6/map?bbox=11.3833058,47.2647132,11.3859022,47.2657835
Content-Length: 385
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://www.openstreetmap.org/api/0.6/map?bbox=11.3833058,47.2647132,11.3859022,47.2657835">here</a>.</p>
<hr>
<address>Apache/2.4.29 (Ubuntu) Server at api.openstreetmap.org Port 80</address>
</body></html>

So, yes, it's coming from Java.


Sending Accept=application/xml, */*;q=0.8 should be fine for all https://www.openstreetmap.org/api/0.6 requests?

comment:6 Changed 3 months ago by simon04

Resolution: fixed
Status: assignedclosed

In 15968/josm:

fix #18812 - HttpClient: specify Accept=*/* to prevent Java from adding Accept=text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2

comment:7 Changed 3 months ago by simon04

In 15969/josm:

see #18812 - OsmApi, OsmServerReader, NameFinder: specify Accept=application/xml, */*;q=0.8

comment:8 Changed 3 months ago by mmd

Sending Accept=application/xml, */*;q=0.8 should be fine for all ​https://www.openstreetmap.org/api/0.6 requests

That should be fine for endpoints returning XML. Did you already test this change with JSON based endpoints, like for OSM Notes? I think this should also work as is, I'm just double checking here.

comment:9 Changed 3 months ago by simon04

I did; for notes, we're also using the XML format returned by queries such as https://api.openstreetmap.org/api/0.6/notes?limit=1000&closed=7&bbox=11.3373158,47.2402315,11.3993994,47.2657867

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain simon04.
as The resolution will be set.
The resolution will be deleted.

Add Comment


E-mail address and name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.