Avoid initial capabilities request

[Fabi2]

Java does not find the system proxy on Linux x86_64 with latest OpenJDK, and so josm hangs on the new capabilities request.

$ java -Xmx3500M  -jar josm-latest.jar 
Repository Root: http://josm.openstreetmap.de/svn
Build-Date: 2011-02-28 02:31:27
Last Changed Author: bastiK
Revision: 3936
Repository UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
URL: http://josm.openstreetmap.de/svn/trunk
Last Changed Date: 2011-02-27 13:33:37 +0100 (Sun, 27 Feb 2011)
Last Changed Rev: 3936

Warnung: Eigenschaft "proxy.policy" nicht gefunden.
Proxy wird nicht verwendet.
GET http://api.openstreetmap.org/api/capabilities... GET http://api.openstreetmap.org/api/capabilities... GET http://api.openstreetmap.org/api/capabilities... GET http://api.openstreetmap.org/api/capabilities... GET http://api.openstreetmap.org/api/capabilities... GET http://api.openstreetmap.org/api/capabilities... Warnung: die Nachricht des Tages konnte nicht von 'http://josm.openstreetmap.de/wiki/De:StartupPage' gelesen werden. Fehlermeldung war: java.net.ConnectException: Connection timed out

System enviroment variables for the proxy are set:

$ set | grep proxy=
http_proxy=h**p://10.0.0.3:3128
https_proxy=h**p://10.0.0.3:3128
no_proxy=10.0.0.1/8

$ java -version
java version "1.6.0_20"
OpenJDK Runtime Environment (IcedTea6 1.9.7) (ArchLinux-6.b20_1.9.7-2-x86_64)
OpenJDK 64-Bit Server VM (build 17.0-b16, mixed mode)

[bastiK]

Actually I don't like that JOSM calls ​http://api.openstreetmap.org/api/capabilities at each start. E.g. it is a privacy issue for paranoid folks.

If the blacklist (including a date) was signed, we could cache it.

We should sort this out before the release.

[framm]

I'll accept your "privacy issue" argument if and only if any calls to the JOSM server are treated the same. If anything, the JOSM server offers *less* privacy than the OSM server, not more; and calls to the OSM server are only made if OSM is your configured API.

The initial call to OSM API is there because otherwise someone could open JOSM, activate a blacklisted imagery source, trace stuff, then upload. Upon the first call to the API, the blacklisted imagery source will be disabled but it is too late then.

A possible solution would be to record which imagery sources the user has been using during his session, and ban uploading with an error message if a blacklisted source has been used. The thing I don't like about this is that it catches the user out cold; they don't know that they're doing wrong until they try to upload...

[Zverikk]

Why does it access api.openstreetmap.org at the start? User can choose different server url, for example. And what happens if the server is down? When going to edit offline, for example. Also, can a user trace google imagery into his own private .osm file?

[framm]

It currently accesses the configured API server, whatever it is. When that doesn't work, JOSM starts up normally (but the reporter of this bug says there are timeout issues which need to be investigated).

Technically any displaying of Google tiles in JOSM, no matter what the user does with it, violates Google's terms and conditions. The line currently being taken by the JOSM team is that we don't care if you violate Google's terms and conditions as long as you don't upload your resulting data to OSM.

[bastiK]

Replying to framm:

I'll accept your "privacy issue" argument if and only if any calls to the JOSM server are treated the same. If anything, the JOSM server offers *less* privacy than the OSM server, not more; and calls to the OSM server are only made if OSM is your configured API.

Ok, you are right, the Message of the day is quite the same. Then let's separate the issues: First one is the long timeout and numerous retries that lead to long freeze at startup (original bug report) and second how we can generally avoid the call at startup.

The initial call to OSM API is there because otherwise someone could open JOSM, activate a blacklisted imagery source, trace stuff, then upload. Upon the first call to the API, the blacklisted imagery source will be disabled but it is too late then.

A possible solution would be to record which imagery sources the user has been using during his session, and ban uploading with an error message if a blacklisted source has been used. The thing I don't like about this is that it catches the user out cold; they don't know that they're doing wrong until they try to upload...

For each capabilities request (at download, upload, etc.) we could save the blacklist to the preferences. This cache file could be used to warn the user. However, for upload, JOSM would not use that cached version, but require a "fresh" request, so any manipulation of the preference file would be in vain.

The user can only be surprised if he/she was very eager tracing from some blacklisted source and tries to upload, but never ever downloaded anything from the server before.

[stoecker]

Regarding the privacy issue: All the other accesses are cached, so the maximum access is once a day to once a week. I see no reason why a blacklist should not be cached once a day as well. It will not change that frequently. It should simply use the mirrored input stream like everything else.

Why is OSM server better than JOSM server regarding privacy? We have no possibility to assign accesses to individual users, where OSM server can do that simply. So I would say it is the other way.

To solve your delayed-blacklist issue: When you open a WMS/TMS layer and no blacklist is downloaded yet, then you can access the API again and when this fails note the user about the fact. There must be a network connection when TMS/WMS is used, which is not the fact on every JOSM start. Also the refresh should be done here, as there is no reason to refresh/fetch blacklist at all for users which don't use WMS/TMS.

[bastiK]

In [3956/josm]:

Turn off the new features introduced in [3934] for now. There are still minor technical problems (see #6037) and a release is due.

[anonymous]

This kind auf blacklisting is a broken concept, which means, that I can still trace from Google, if save my work and restart JOSM before the upload (or edit the *.osm file). So don:t try to play the police man. But the user should be warned and pointed to more information, using a cached copy of the blacklist, this helps at least in some cases.

At least, the blacklist should be only downloaded, if the user intents to make a network connection with JOSM, as they must e.g. first configure the network settings to get connectivity.

[framm]

It was clear from the outset that we cannot make tracing impossible, and it was never intended to make it impossible. The intention of this kind of blacklisting is to make it very clear that tracing from Google is not tacitly condoned by OSM. If we have a situation where you have to employ tricks to trace from Google, then any judge in the world will see clearly that you are the one violating copyright. If, on the other hand, you just have to close a little warning window, then someone could say "the user didn't understand what he was doing". That is the whole point of this exercise.

[bastiK]

Is there any progress including the blacklist in the capabilities response as described in [3934]? I guess it would be easier to move on if this was working.

[framm]

When you set your HTTP proxy in the JOSM configuration, it will use that proxy even on the initial API contact. This is independent of any environment variables, and works perfectly with the initial API capabilities call.

If you want JOSM to use your system proxy settings, you have to run it with -Djava.net.useSystemProxies=true *and* have proxy.policy=use-system-settings in your config file. Neither of which the person reporting the error seems to have used!

I have turned down the connection timeout for the initial API connection in r3993 just to be on the safe side but I don't see how one can reproduce this issue and at the same time have a working JOSM instance - proxy not used initially would also mean that the proxy is not used later for other API requests.

I am downgrading but not closing this bug because the ideas suggested for avoiding the initial call altogether still have merit and should be looked into.

JOSM will use
-Djava.net.useSystemProxies=true

[framm]

Please ignore the last line in the above comment.

[Fabi2]

Replying to framm:

When you set your HTTP proxy in the JOSM configuration, it will use that proxy even on the initial API contact. This is independent of any environment variables, and works perfectly with the initial API capabilities call.

Yes, this was what I did, but to do this, I have at leat do one JOSM start to change the configuration, if I not want to use the text editor. I most time do not remember the name of the configuration variable to do this. And so you have to wait for tghis initally long connection time out. Yes, this bug was also more reported for the files.

If you want JOSM to use your system proxy settings, you have to run it with -Djava.net.useSystemProxies=true *and* have proxy.policy=use-system-settings in your config file. Neither of which the person reporting the error seems to have used!

I tried this a few months ago and at this time, this did not work with all of this settings in OpenJDK. But can make a new test, but think think this may be an upstream problem of OpenJDK.

I have turned down the connection timeout for the initial API connection in r3993 just to be on the safe side but I don't see how one can reproduce this issue and at the same time have a working JOSM instance - proxy not used initially would also mean that the proxy is not used later for other API requests.

You a right, but the proxy must be configured at the first start and JOSM can then be used after a restart.
But with this fix, I already waiting for the first (e.g. UMTS/GSM) dialup user (mobile mapper) which ask why JOSM want to make a connection at their start. But at least it solves the proxy issues.

[Don-vip]

​http://api.openstreetmap.org/api/capabilities now provides the imagery blacklist so the capabilities call must remain. However it can still be improved, see #7140