Dutch WMS http/https not working on Windows/macOS

[stoecker]

You changed the Dutch WMS from https to http. Referencing Qualys is no real reason, as JOSM is not pure Java, see ticket #14649.

Do you have a reason that it does no work beside the internet check?

[Stereo]

​https://forum.openstreetmap.org/viewtopic.php?id=61391

You seriously thought I'd come up with an ssl check on Qualys on a Dutch WMS randomly? :)

[stoecker]

@Vincent:

Any reason why adding the cert does not work? Did it change?

[stoecker]

You seriously thought I'd come up with an ssl check on Qualys on a Dutch WMS randomly? :)

Sure. People do things in this wiki for so many reasons, that I can't guess what they though. When you refer Qualys and not the thread, then I assume this was the reason.

[Don-vip]

We have a unit test and it works.

What does the forum say? I still don't speak German.

[stoecker]

Mainly that "​https://geodata.nationaalgeoregister.nl:443" is not trusted for at least two members. Works with JNLP. No Java or JOSM version information, but one Guy says it is for a while now.

Maybe the said "no" to some request to install the cert?

I still don't speak German.

It's probably at least as good as my French.

[stoecker]

@Stereo:

Please tell them to report their systeminformation here, so we can find the reason. The downgrade to HTTP is not really a wanted solution.

[Don-vip]

Replying to stoecker:

It's probably at least as good as my French.

I just know how to get enough bier and wurtz to survive in festivals. Not very useful to fix Java bugs :D

Does anyone here confirm the problem?

[stoecker]

No reaction, reverted the change to use https again until someone describes why it fails.

The attribution URL seems broken btw.

[stoecker]

P.S. Stereo - Please no xmltidy or similar automatic stuff on the XML pages:

• It is hard to track changes after a reformatting
• The new long lines are really ugly
• The output JOSM uses anyway is reformatted

[Stereo]

I think it would be good to have the http version for now, so that people can at least map until this is fixed in JOSM.

I was able to reproduce the error with:

URL:http://josm.openstreetmap.de/svn/trunk
Repository:UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
Last:Changed Date: 2018-01-28 23:08:56 +0100 (Sun, 28 Jan 2018)
Build-Date:2018-01-28 22:25:44
Revision:13367
Relative:URL: ^/trunk

Identification: JOSM/1.5 (13367 en_GB) Mac OS X 10.13.3
OS Build number: Mac OS X 10.13.3 (17D47)
Memory Usage: 1253 MB / 7282 MB (119 MB allocated, but free)
Java version: 1.8.0_161-b12, Oracle Corporation, Java HotSpot(TM) 64-Bit Server VM
Screen: Display 69733632 1680x1050, Display 458659266 3008x1692
Maximum Screen Size: 3008x1692
VM arguments: [-Djava.library.path=/Applications/JOSM.app/Contents/MacOS, -DLibraryDirectory=${HOME}/Library, -DDocumentsDirectory=${HOME}/Documents, -DApplicationSupportDirectory=${HOME}/Library/Application Support, -DCachesDirectory=${HOME}/Library/Caches, -DSandboxEnabled=false, -Dapple.laf.useScreenMenuBar=true, -Dcom.apple.macos.use-file-dialog-packages=true, -Dcom.apple.macos.useScreenMenuBar=true, -Dcom.apple.mrj.application.apple.menu.about.name=JOSM, -Dcom.apple.smallTabs=false, -Dawt.useSystemAAFontSettings=on, -Dswing.aatext=true, -Dsun.java2d.xrender=true

The error says:

2018-02-22 15:40:06.921 INFO: GET https://geodata.nationaalgeoregister.nl/luchtfoto/rgb/wms?REQUEST=GetMap&LAYERS=Actueel_ortho25&STYLES=&SRS=EPSG:3857&WIDTH=512&HEIGHT=512&BBOX=607826.3616108,6708725.6713892,607979.2356608,6708878.5454392 -> !!!
2018-02-22 15:40:07.020 WARNING: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target. Cause: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target. Cause: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
	at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1937)
	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302)
	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1478)
	at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:212)
	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:957)
	at sun.security.ssl.Handshaker.process_record(Handshaker.java:892)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1050)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1363)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1391)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1375)
	at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:563)
	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153)
	at org.openstreetmap.josm.tools.HttpClient.connect(HttpClient.java:141)
	at org.openstreetmap.josm.tools.HttpClient.connect(HttpClient.java:87)
	at org.openstreetmap.josm.data.cache.JCSCachedTileLoaderJob.loadObject(JCSCachedTileLoaderJob.java:326)
	at org.openstreetmap.josm.data.cache.JCSCachedTileLoaderJob.run(JCSCachedTileLoaderJob.java:235)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387)
	at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
	at sun.security.validator.Validator.validate(Validator.java:260)
	at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1460)
	... 17 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:145)
	at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:131)
	at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382)
	... 23 more

(Noted for the xml tidy. I had made a bit of a mess, and thought this'd clean up nicely.)

[Don-vip]

Are the forum guys using macOS too? On Linux and Windows I have no problem with two out of the three Dutch servers:

• PDOK aerial imagery Beeldmateriaal.nl 25cm latest mirror server => HTTPS OK
• PDOK aerial imagery Beeldmateriaal.nl 25cm (WMTS) => HTTPS OK
• PDOK aerial imagery Beeldmateriaal.nl 25cm latest => Error not linked to HTTPS

The "PDOK aerial imagery Beeldmateriaal.nl 25cm latest" entry seems to be invalid:

<?xml version="1.0"?>
<!DOCTYPE ServiceExceptionReport SYSTEM "http://schemas.opengis.net/wms/1.1.1/exception_1_1_1.dtd">
<ServiceExceptionReport version="1.1.1">
    <ServiceException>missing parameters ['version', 'format']</ServiceException>
</ServiceExceptionReport>

[Stereo]

Shouldn't that wmts be a mirror of the first anyway?

Both https://josm.openstreetmap.de/mapsview?entry=PDOK%20aerial%20imagery%20Beeldmateriaal.nl%2025cm%20latest and https://josm.openstreetmap.de/mapsview?entry=PDOK%20aerial%20imagery%20Beeldmateriaal.nl%2025cm%20latest&mirror=1 seem to load images, but very slightly different ones. What a strange server.

[Stereo]

aixbrick on the forum has the same problem on Windows 10 with the latest tested josm jar.

I also use the jar in the macOS app

[Don-vip]

In 13449/josm:

see #15992 - display and log ServiceException error messages

[Don-vip]

Replying to Stereo:

aixbrick on the forum has the same problem on Windows 10 with the latest tested josm jar.

It should work since ticket:14649#comment:24 / r11943 . It definitively works on my Windows 10.

I also use the jar in the macOS app

Does it fail with WebStart? Also, are you able to access the https links using Safari?

[stoecker]

Replying to Don-vip:

• PDOK aerial imagery Beeldmateriaal.nl 25cm latest => Error not linked to HTTPS

The "PDOK aerial imagery Beeldmateriaal.nl 25cm latest" entry seems to be invalid:

<?xml version="1.0"?>
<!DOCTYPE ServiceExceptionReport SYSTEM "http://schemas.opengis.net/wms/1.1.1/exception_1_1_1.dtd">
<ServiceExceptionReport version="1.1.1">
    <ServiceException>missing parameters ['version', 'format']</ServiceException>
</ServiceExceptionReport>

The preview works, but this reconstructs the URL. Added format and version parameters to make URL work properly

[Don-vip]

Issue reproduced on macOS, fix in progress

[Don-vip]

In 13450/josm:

fix #15992 - load native certificates from macOS system root trust store, see ​https://support.apple.com/en-us/HT208127

[Stereo]

So what do we tell the guy on Win 10 with 13367 in the forums, that he should switch to macOS? :)

[stoecker]

Replying to Stereo:

So what do we tell the guy on Win 10 with 13367 in the forums, that he should switch to macOS? :)

Tell him to try the most recent version and when it does not work report here with his exact system information so Vincent maybe can find out why it does not work.

[stoecker]

@Vincent: The report here seems interesting: ​https://forum.openstreetmap.org/viewtopic.php?pid=686380#p686380

[Don-vip]

Maybe linked to the Windows installer, I only checked WebStart and java -jar

[Don-vip]

OK indeed it doesn't work with Windows Installer, no idea why.

[Don-vip]

Investigated deeper. Not caused by windows installer. With Java 9 the Windows-ROOT KeyStore contains only 46 entries instead of ​the full list, don't know why.

[Don-vip]

Not caused by Java! The root trust store of my laptop contains only 45 entries, and does not contain the Dutch ones. This is a pure Windows problem.

[Don-vip]

OK problem understood. Here is ​how the root trust store works on Windows:

How Root Certificate Distribution Works

Starting with the release of Windows Vista, root certificates are updated on Windows automatically. When a user visits a secure Web site (by using HTTPS SSL), reads a secure email (S/MIME), or downloads an ActiveX control that is signed (code signing) and encounters a new root certificate, the Windows certificate chain verification software checks the appropriate Microsoft Update location for the root certificate. If it finds it, it downloads it to the system. To the user, the experience is seamless. The user does not see any security dialog boxes or warnings. The download happens automatically, behind the scenes.

Confirmed: If I visit ​https://geodata.nationaalgeoregister.nl/luchtfoto/rgb/wms?REQUEST=GetCapabilities with Edge, the root certificate "Staat der Nederlanden Root CA - G2" is added. Not with Firefox, not from Java. We need to notify Windows somehow that we need this certificate.

[stoecker]

Replying to Don-vip:

The download happens automatically, behind the scenes.

I've read about this concept which makes "trust" a strange concept taking control out of users hands to 100%.

We need to notify Windows somehow that we need this certificate.

Call Edge once to load the page, but only when that target is used and throws a security exception?

[Don-vip]

Concerning the control by end users: this feature is enabled by default but can easily be disabled with the "Turn off Automatic Root Certificates Update" group policy.

Replying to stoecker:

Call Edge once to load the page, but only when that target is used and throws a security exception?

I thought about that but it would be a crude workaround. I found out that there is no special API used by Edge or IE11. This happens inside the CryptoAPI itself. So in theory it should happen with every software calling MS CAPI. The JDK calls this API too, so maybe there's a pure Java way to achieve this.

I also found out that the certificate is not really "downloaded on demand". The complete list of trusted roots (CTL: certificate trust list) is downloaded on a periodical basis and saved into the Windows Registry, encoded in HKLM\SOFTWARE\Microsoft\SystemCertificates\AuthRoot\AutoUpdate\EncodedCtl. When needed, the certificate is simply copied from this entry to another entry of the Windows Registry, so this is really fast.

Right now I'm able to get this information but not to decode it yet:

Path tmp1 = Files.createTempFile("josmctl_", ".pkcs7");
// Extract trusted root certificates from Windows Registry. We cannot use WinRegistry class (binary data)
Utils.execOutput(Arrays.asList("powershell", "-Command",
    "[IO.File]::WriteAllBytes('" + tmp1.toString() +
    "',(Get-ItemProperty -Path 'HKLM:\\SOFTWARE\\Microsoft\\SystemCertificates\\AuthRoot\\AutoUpdate').EncodedCtl)"));

I tried to play with certutil; it can decode it but I only get human readable output, not standard PEM format.

[stoecker]

Hmm, I lost countenance in the referenced forum. You're trying to fix this rather complicated issue and that guy complains about the way I told him that he please may report bugs in the bugtracker.

Now he really has a post to complain about ;-)

[Don-vip]

In 13451/josm:

fix #15992 - force Windows to update its root CA trust store before we search for known CA in it

[Don-vip]

From ticket:16009#comment:6:

Version 13457 does not show up on Windows 7 64 bit. It gets stuck with the debug line:

FEIN: powershell -Command [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12;Invoke-WebRequest https://roottest-g2.pkioverheid.nl

@Hb---: What happens if you run the command directly in powershell?

[Don-vip]

In 13458/josm:

fix #15992 - add compatibility with older versions of Windows where Invoke-WebRequest is not available

[Don-vip]

In 13463/josm:

see #15992 - make sure .NET framework 4.5 or later is installed, as previous versions do not support TLS 1.2

[Hb---]

Even with 13463 java.exe freezes at the debug line mentioned above.

But the Powershell /NET-Framework installation here on Win7 64bit may be crap. The system control lists .NET Framework 4.6.1 (german) and .NET Framework 4.7 as installed. The command line shows:

C:\Users\name\Desktop>powershell
Windows PowerShell
Copyright (C) 2009 Microsoft Corporation. Alle Rechte vorbehalten.

PS C:\Users\name\Desktop> $PSVersionTable

Name                           Value
----                           -----
CLRVersion                     2.0.50727.8762
BuildVersion                   6.1.7601.17514
PSVersion                      2.0
WSManStackVersion              2.0
PSCompatibleVersions           {1.0, 2.0}
SerializationVersion           1.1.0.1
PSRemotingProtocolVersion      2.1

[Don-vip]

In 13465/josm:

fix #15992 - also make sure recent version of PowerShell is installed

[Hb---]

With NET 4.7.1 java.exe dies too:

C:\Users\name\Desktop>java  -jar josm-latest.jar --language=en --debug
47.679 INFORMATION: Protokollierungsgrad ist bei FEIN (FINE, 500)
48.140 FINE: System property 'http.agent' set to 'JOSM/1.5 (13465 en) Windows 7 64-Bit'. Old value was 'null'
48.151 FINE: System property 'user.language' set to ''. Old value was 'de'
48.280 FINE: System property 'sun.awt.fontconfig' set to 'C:\Users\hb\AppData\Local\JOSM\cache\fontconfig.properties'. Old value was 'null'
48.671 FINE: System property 'java.protocol.handler.pkgs' set to 'org.openstreetmap.josm.io.protocols'. Old value was 'null'
48.900 WARNING: Could not open/create prefs root node Software\JavaSoft\Prefs at root 0x80000002. Windows RegCreateKeyEx(...) returned error code 5.
48.910 FINE: powershell -Command $PSVersionTable.PSVersion.Major

[stoecker]

Would "format c:" help to fix this issue finally? ;-)

[Hb---]

Replying to stoecker:

Would "format c:" help to fix this issue finally? ;-)

To prevent this, I uninstalled .NET 4.7.1. See the astonishing result below.

Same crash when starting with --skip-plugins and with --offline=all.

Feels like JOSM depends on Microsoft .NET Framework and cannot start offline any more. That feeling doesn't make me smile.

URL:https://josm.openstreetmap.de/svn/trunk
Repository:UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
Last:Changed Date: 2018-02-26 01:10:01 +0100 (Mon, 26 Feb 2018)
Build-Date:2018-02-26 00:12:54
Revision:13465
Redirecting:to URL 'https://josm.openstreetmap.de/svn/trunk':
Relative:URL: ^/trunk

Identification: JOSM/1.5 (13465 en) Windows 7 64-Bit
OS Build number: Windows 7 Professional (7601)
Memory Usage: 243 MB / 3604 MB (189 MB allocated, but free)
Java version: 1.8.0_161-b12, Oracle Corporation, Java HotSpot(TM) 64-Bit Server VM
Screen: \Display0 1280x1024
Maximum Screen Size: 1280x1024
Program arguments: [--language=en, --debug]

Plugins:
+ apache-commons
+ photo_geotagging
+ photoadjust

Last errors/warnings:
- E: Handled by bug report queue: java.lang.NullPointerException


=== REPORTED CRASH DATA ===
BugReportExceptionHandler#handleException:
No data collected.

Warning issued by: BugReportExceptionHandler#handleException

=== STACK TRACE ===
Thread: main (1)
java.lang.NullPointerException
	at java.util.regex.Matcher.getTextLength(Unknown Source)
	at java.util.regex.Matcher.reset(Unknown Source)
	at java.util.regex.Matcher.<init>(Unknown Source)
	at java.util.regex.Pattern.matcher(Unknown Source)
	at org.openstreetmap.josm.tools.PlatformHookWindows.isDotNet45Installed(PlatformHookWindows.java:706)
	at org.openstreetmap.josm.tools.PlatformHookWindows.webRequest(PlatformHookWindows.java:744)
	at org.openstreetmap.josm.tools.PlatformHookWindows.getX509Certificate(PlatformHookWindows.java:451)
	at org.openstreetmap.josm.io.CertificateAmendment.addMissingCertificates(CertificateAmendment.java:218)
	at org.openstreetmap.josm.gui.MainApplication.mainJOSM(MainApplication.java:976)
	at org.openstreetmap.josm.gui.MainApplication$2.processArguments(MainApplication.java:279)
	at org.openstreetmap.josm.gui.MainApplication.main(MainApplication.java:851)

[Don-vip]

In 13466/josm:

fix #15992 - fix NPE

[Don-vip]

In 13467/josm:

fix #15992 - make sure PowerShell call is never blocking

[Don-vip]

In 13468/josm:

fix #15992 - fix codestyle issues

[freeExec]

getPowerShellVersion() endlessly waiting input.readLine(), because the second line will never be. As a result, josm does not start.

[Hb---]

r13468 starts fine here on Windows 7 without Powershell. Only minor WARNING in console, see #16015.

[Don-vip]

@freeExec: with r13468?

[Don-vip]

In 13472/josm:

see #15992 - redirect error stream of called processes to standard output

[Don-vip]

@freeExec can you please try with r13472? Also, can you please tell me what Windows version you're using, and if powershell is installed?

[Don-vip]

If r13472 is not enough, a ​more robust solution might be to redirect process output to a temp file, wait for process completion, then read the file.

[freeExec]

Test - Revision: 13472, josm does not start.
Windows - win7 x64
Powershell installed - PSVersion 2.0

[Don-vip]

OK I'll implement the file-based approach tonight.

[Don-vip]

In 13473/josm:

fix #15992 - more robust implementation of Utils.execOutput. Ensure calling a process never blocks the JVM

[Don-vip]

@freeExec, Hb--: can you please both check that JOSM r13473 starts correctly? The jar is already available.

[Hb---]

Screenshot of error message

[Hb---]

Network timeout error while creating the "Bildeinstellungen" on the first start. Second start was fine.

Why is https://api.openstreetmap.org/api/capabilities blamed in the error message? That server seems to work well.

[Don-vip]

You encountered a random network issue. It happens. The message could be improved, see #10733. At least JOSM starts fine now :)

[freeExec]

Despite the errors, josm was launched.

java -Dfile.encoding=Cp866 -Xmx1200M -jar josm-latest.jar
2018-02-27 23:44:32.917 INFO: Уровень журналирования: INFO (INFO, 800)
2018-02-27 23:44:33.663 WARNING: Could not open/create prefs root node Software\JavaSoft\Prefs at root 0x80000002. Windo
ws RegCreateKeyEx(...) returned error code 5.
2018-02-27 23:44:35.697 SEVERE: java.util.concurrent.ExecutionException: [powershell, -Command, $PSVersionTable.PSVersio
n.Major]
java.util.concurrent.ExecutionException: [powershell, -Command, $PSVersionTable.PSVersion.Major]
        at org.openstreetmap.josm.tools.Utils.execOutput(Utils.java:859)
        at org.openstreetmap.josm.tools.PlatformHookWindows.getPowerShellVersion(PlatformHookWindows.java:728)
        at org.openstreetmap.josm.tools.PlatformHookWindows.webRequest(PlatformHookWindows.java:748)
        at org.openstreetmap.josm.tools.PlatformHookWindows.getX509Certificate(PlatformHookWindows.java:452)
        at org.openstreetmap.josm.io.CertificateAmendment.addMissingCertificates(CertificateAmendment.java:218)
        at org.openstreetmap.josm.gui.MainApplication.mainJOSM(MainApplication.java:976)
        at org.openstreetmap.josm.gui.MainApplication$2.processArguments(MainApplication.java:279)
        at org.openstreetmap.josm.gui.MainApplication.main(MainApplication.java:851)

2018-02-27 23:44:37.835 SEVERE: java.util.concurrent.ExecutionException: [powershell, -Command, $PSVersionTable.PSVersio
n.Major]
java.util.concurrent.ExecutionException: [powershell, -Command, $PSVersionTable.PSVersion.Major]
        at org.openstreetmap.josm.tools.Utils.execOutput(Utils.java:859)
        at org.openstreetmap.josm.tools.PlatformHookWindows.getPowerShellVersion(PlatformHookWindows.java:728)
        at org.openstreetmap.josm.tools.PlatformHookWindows.webRequest(PlatformHookWindows.java:748)
        at org.openstreetmap.josm.tools.PlatformHookWindows.getX509Certificate(PlatformHookWindows.java:452)
        at org.openstreetmap.josm.io.CertificateAmendment.addMissingCertificates(CertificateAmendment.java:218)
        at org.openstreetmap.josm.gui.MainApplication.mainJOSM(MainApplication.java:976)
        at org.openstreetmap.josm.gui.MainApplication$2.processArguments(MainApplication.java:279)
        at org.openstreetmap.josm.gui.MainApplication.main(MainApplication.java:851)

[Klumbumbus]

I lost the thread here a bit and don't know if there is anything notable from this ticket which should be added to Changelog.

[Don-vip]

Not really, it's just a bugfix for my incomplete resolution of #14649. Basically it worked only on Linux, now it works too on macOS and Windows (8, 10).

[Don-vip]

In 13487/josm:

see #15992 - do not log PS 2.0 errors as such (debug level is enough)