Fetch from Overpass API using HTTPS

[skyper]

I am testing josm with ssl and found own preferences for utilsplugin2

1. I would expect that the plugin is using the same settings as core to browse the OSM-history but it is not.
2. I would prefer to have a checkbox (similar to #9776 or even use this one) to change to ssl for all connections.
   • at least wikipedia is offering ssl.

Repository Root: http://josm.openstreetmap.de/svn
Build-Date: 2014-03-08 02:33:07
Last Changed Author: Don-vip
Revision: 6901
Repository UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
URL: http://josm.openstreetmap.de/svn/trunk
Last Changed Date: 2014-03-07 11:35:00 +0100 (Fri, 07 Mar 2014)
Last Changed Rev: 6901

Identification: JOSM/1.5 (6901 en) Linux Debian GNU/Linux 7.4 (wheezy)
Memory Usage: 173 MB / 672 MB (101 MB allocated, but free)
Java version: 1.7.0_25, Oracle Corporation, OpenJDK 64-Bit Server VM
Java package: openjdk-7-jre:amd64-7u25-2.3.10-1~deb7u1
VM arguments: [-Djosm.home=$HOME/.test]
Program arguments: [--language=en]

Plugin: utilsplugin2 (30277)
Plugin: wikipedia (30277)

[roland.olbricht]

The plugin mirrored_download is designed to communicate with overpass-api.de.

I'm running that service and am offering a self-signed certificate. That in turn won't work out-of-the-box with Java, so https have remained turned off. The reason not to have a certificate is that I haven't found a trustworthy certificate authority so far. Thus I'm currently not planning to spend time or money for a CA certificate.

In total, enabling https on the plugin won't make sense when its counterpart doesn't offer https.

[skyper]

Replying to roland.olbricht@…:

The plugin mirrored_download is designed to communicate with overpass-api.de.

I'm running that service and am offering a self-signed certificate.

Nice, it is available then. Did not know that overpass itself is offering ssl-encrypted connections.

That in turn won't work out-of-the-box with Java, so https have remained turned off. The reason not to have a certificate is that I haven't found a trustworthy certificate authority so far. Thus I'm currently not planning to spend time or money for a CA certificate.

I know the problem.

In total, enabling https on the plugin won't make sense when its counterpart doesn't offer https.

I was referring to the URLs listed in the preferences (wikipedia and OSM-History)

[Don-vip]

Replying to roland.olbricht@…:

The reason not to have a certificate is that I haven't found a trustworthy certificate authority so far. Thus I'm currently not planning to spend time or money for a CA certificate.

Same reason for us until a few weeks. You may have a look to ​https://www.globalsign.com/ssl/ssl-open-source/ :)

[Don-vip]

I have changed a minor point in [o30354] (https access to JOSM website). Plugin not rebuilt yet.

[simon04]

mirrored_download is now part of core (see #11428)

[simon04]

See ​https://github.com/drolbr/Overpass-API/issues/233

[simon04]

Due to #12264 the StartSSL certificate is no longer a problem. However, fetching via ​https://overpass-api.de/ break on "handshake alert: unrecognized_name":

Caused by: javax.net.ssl.SSLProtocolException: handshake alert:  unrecognized_name
	at sun.security.ssl.ClientHandshaker.handshakeAlert(ClientHandshaker.java:1441)
	at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2016)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1125)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
	at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153)
	at org.openstreetmap.josm.tools.HttpClient.connect(HttpClient.java:122)
	at org.openstreetmap.josm.io.OsmServerReader.getInputStreamRaw(OsmServerReader.java:145)
	... 15 more

See also the test result from ​https://www.ssllabs.com/ssltest/analyze.html?d=overpass-api.de&s=46.4.41.83

[bastiK]

Seems to be related to ​javabug:7177232 and ​javabug:7127374. Looking at the test results from ssllabs.com, every client but Java >= 7 ignores this issue and carries on with the connection.

One solution would be to fix the configuration of the overpass server.

Another solution would be to work around this Java quirk. Switching off SNI with

System.setProperty("jsse.enableSNIExtension", "false");

as suggested by some people is not a good idea. For example it no longer connects to ​https://helloworld.letsencrypt.org.

[Don-vip]

Replying to bastiK:

Switching off SNI as suggested by some people is not a good idea.

Yep, see 12152#comment:5 + r9218

[roland.olbricht]

I've fixed the issue on the server side. There was an old "ServerName" directive. And apparently nobody else ever noticed the problem.

[stoecker]

Maybe we should re-report that issue to Java? It should be easy to reproduce. Setup a server where the ServerName directive of the first VHost does not match the certificate name (i.e. a fallback vhost) and also no other vhost does. In my eyes that is no misconfiguration as they state in the mentioned tickets above, but a (maybe unusual) default scenario.

[Don-vip]

Go ahead :) You can mention this bug report comes from JOSM through the Quality Outreach (you can specify in the bug report to add "josm-found" JIRA label), it should improve the chances of positive answer.

[Stereo]

The download from overpass-api now fails again.

How to reproduce:

Find something interesting on overpass-turbo.eu, and click on Export > JOSM

Expected result:

The interesting stuff opens in JOSM

Actual result:

Error message:

Failed to upload data or download data from '​https://overpass-api.de/[...]' due to a problem with transferring data.
Details (untranslated): sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

URL:http://josm.openstreetmap.de/svn/trunk
Repository:UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
Last:Changed Date: 2016-03-13 01:17:21 +0100 (Sun, 13 Mar 2016)
Build-Date:2016-03-13 02:35:00
Revision:9979
Relative:URL: ^/trunk

Identification: JOSM/1.5 (9979 en_GB) Mac OS X 10.11.3
Memory Usage: 707 MB / 3641 MB (327 MB allocated, but free)
Java version: 1.8.0_31-b13, Oracle Corporation, Java HotSpot(TM) 64-Bit Server VM

Plugins:
- AddrInterpolation (31772)
- CommandLine (31927)
- DirectDownload (31934)
- DirectUpload (31912)
- FixAddresses (31772)
- HouseNumberTaggingTool (31772)
- ImportImagePlugin (31772)
- Mapillary (32040)
- OSMRecPlugin (31772)
- OpeningHoursEditor (31772)
- PicLayer (31895)
- RoadSigns (31895)
- ShapeTools (1000)
- alignways (31895)
- apache-commons (31895)
- apache-http (31895)
- areaselector (263)
- buildings_tools (31895)
- cadastre-fr (32060)
- conflation (0.1.9)
- contourmerge (1014)
- dataimport (31772)
- download_along (31772)
- ejml (31895)
- geotools (31895)
- gson (31895)
- imagery-xml-bounds (31772)
- imagery_offset_db (32046)
- importvec (31772)
- jts (31772)
- livegps (31772)
- log4j (31895)
- mapdust (31772)
- merge-overlap (31967)
- namemanager (31772)
- opendata (32071)
- pdfimport (32019)
- public_transport (31895)
- public_transport_layer (31932)
- rasterfilters (31895)
- reverter (32005)
- scripting (30722)
- splinex (31895)
- surveyor (31772)
- tageditor (31772)
- terracer (31895)
- turnlanes (31772)
- turnrestrictions (31895)
- undelete (31895)
- utilsplugin2 (32099)

Last errors/warnings:
- E: Failed to locate image ''
- W: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target. Cause: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target. Cause: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
- E: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target. Cause: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target. Cause: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
- E: org.openstreetmap.josm.io.OsmTransferException: Could not connect to the OSM server. Please check your internet connection.. Cause: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target. Cause: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target. Cause: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

[bastiK]

@Stereo: You need version r9995 or later for it to work with https.

[roland.olbricht]

For the sake of completeness: I've have renewed the overpass-api.de certificate, because the old one is going to expire. The CA is now "Let's Encyrpt" and I intend to stay with "Let's Encrypt" for a longer time.

[bastiK]

Overpass unit test fails again (since [10140]).

[stoecker]

The SSL Test for ​overpass-api.de says "Incorrect SNI alerts".

Reason probably: Missing ServerName or ServerAlias directive in the vhost.

Works for ​www.overpass-api.de, so I assume

ServerAlias overpass-api.de

is missing.

[stoecker]

Ah, I now also found the necessary commandline call

openssl s_client -connect overpass-api.de:443 -servername www.overpass-api.de -msg 2>&1 </dev/null |grep unrecognized

openssl s_client -connect overpass-api.de:443 -servername overpass-api.de -msg 2>&1 </dev/null |grep unrecognized
<<< TLS 1.2 Alert [length 0002], warning unrecognized_name

[roland.olbricht]

FWIW, I've added www.overpass-api.de to the ServerAlias in the Apache config. I don't know whether this has an influence.

[stoecker]

Replying to roland.olbricht@…:

FWIW, I've added www.overpass-api.de to the ServerAlias in the Apache config. I don't know whether this has an influence.

Hmm. "overpass-api.de" is missing, not "www.overpass-api.de".

It should look like

<VirtualHost www.overpass-api.de:443>
    ServerName www.overpass-api.de
    ServerAlias overpass-api.de
    ...
</VirtualHost>

ServerName should be the major name used. ServerAlias should be all the other names this entry is accessed with. Which is name and which alias depends on what you define as the main one. I don't think it makes a difference except in error messages.

i.e. it also could be:

<VirtualHost overpass-api.de:443>
    ServerName overpass-api.de
    ServerAlias www.overpass-api.de
    ...
</VirtualHost>

In case you still have trouble, please send me the config file by mail - I can have a closer look then.

BTW: I personally don't like the possibility to access one service with multiple names. Thus I have for all of mine except the main ones secondary entries, which redirect to the main ones.

[Don-vip]

In 10196/josm:

fix #9795 - Fetch from Overpass API using HTTPS