Default encoding problems

[akks]

People from forum (Xmypblu, LLlypuk82 etc.) noticed that notes and scripting plugins are showing incorrectly encoded messages
(on some Windows machines by default, can be fixed with -Dfile.encoding=utf8)

Debugging showed that the reason is in JOSM core:
BufferedReader in = new BufferedReader(new InputStreamReader(i));
in OsmApi.java uses default encoding (not always UTF8).

I have added Utils.UTF_8 to the call and hope there will be no regressions.

The same default InputStreamReader is used in other classes too:
AbstractOAuthProvider, OAuth, AutosaveTask, ServerSidePreferences, Projections, MapPaintDialog, MapPaintStyles, XmlStyleSource, OsmOAuthAuthorizationClient, ApiUrlTestTask, NmeaReader, OsmServerReader (???), PlatformHookUnixoid, Utils.execOutput.

Not sure if they can cause analogous problems, can someone help?

[akks]

Forgot to mention ticket in changeset:

In [6966/josm]:

Fix default encoding for non-UTF consoles (for Notes plugin etc.)

[akks]

To get sample of old incorrect encoding on old build in UTF8 console you need to run
java -Dfile.encoding=cp866 -jar josm-latest.jar
(only way to get Russian characters in Windows XP console)

[skyper]

Replying to akks:

Forgot to mention ticket in changeset:

In [6966/josm]:

Fix default encoding for non-UTF consoles (for Notes plugin etc.)

Does this fix #9858 ?

[akks]

Yes, I hope so. I just missed the tickets (searched for "encoding" instead of "charset")...

[akks]

I was asked to fix another encoding-related bug:
the following link

​http://127.0.0.1:8111/import?url=http://overpass-api.de/api/interpreter?data=relation[%22admin_level%22=%224%22][%22name%22=%22%D0%9A%D0%B0%D0%BB%D0%B8%D0%BD%D0%B8%D0%BD%D0%B3%D1%80%D0%B0%D0%B4%D1%81%D0%BA%D0%B0%D1%8F+%D0%BE%D0%B1%D0%BB%D0%B0%D1%81%D1%82%D1%8C%22];(way(r:%22outer%22);node(w););out%20meta;

works correctly only for utf console. Running it with java -Dfile.encoding=cp866 -jar josm-latest.jar or from Windows console do not download anything. I was not able to fix it. OSMReader keeps returning empty dataset.

[bastiK]

The remote control handler (correctly) decodes the url argument with URLDecoder.decode(param, "UTF-8");.

The result is the following string:
http://overpass-api.de/api/interpreter?data=relation["admin_level"="4"]["name"="Калининградская область"];(way(r:"outer");node(w););out meta;

This is not a valid URL, because it contains non ASCII characters. So this is in fact an error on the user site.

We can handle this anyway if we percent-encode non ASCII characters at this point.

[akks]

I agree that we need to send import-request as url-encoded (then it works correctly).
​http://overpass-api.de/api/interpreter?data=relation[%22admin_level%22=%224%22][%22name%22=%22%D0%9A%D0%B0%D0%BB%D0%B8%D0%BD%D0%B8%D0%BD%D0%B3%D1%80%D0%B0%D0%B4%D1%81%D0%BA%D0%B0%D1%8F+%D0%BE%D0%B1%D0%BB%D0%B0%D1%81%D1%82%D1%8C%22];(way(r:%22outer%22);node(w););out%20meta;
This type of requests are rather common for local validators and other web-based tools.

[akks]

By the way, the request to ​http://127.0.0.1:8111/... was not working behind proxy, so I have added 127.0.0.1 to the exceptions list:

[6966/josm] / Add proxy.exceptions advanced parameter (localhost and 127.0.0.1 by default)

(is it OK that I did not create separate ticket for it?)

[bastiK]

In 6972/josm:

see #9906 - remotecontrol import handler: add workaround when URLs with multibyte
characters are only one time percent encoded

[bastiK]

Replying to akks:

I was asked to fix another encoding-related bug:
the following link

​http://127.0.0.1:8111/import?url=http://overpass-api.de/api/interpreter?data=relation[%22admin_level%22=%224%22][%22name%22=%22%D0%9A%D0%B0%D0%BB%D0%B8%D0%BD%D0%B8%D0%BD%D0%B3%D1%80%D0%B0%D0%B4%D1%81%D0%BA%D0%B0%D1%8F+%D0%BE%D0%B1%D0%BB%D0%B0%D1%81%D1%82%D1%8C%22];(way(r:%22outer%22);node(w););out%20meta;

works correctly only for utf console. Running it with java -Dfile.encoding=cp866 -jar josm-latest.jar or from Windows console do not download anything. I was not able to fix it. OSMReader keeps returning empty dataset.

This should work now.

[akks]

Thank you very much! It really works.
I guess currently to open something with /import?, the authors of tools now just need to append this text to "​http://127.0.0.1:8111/import?url=".

[Don-vip]

In 6995/josm:

fix #9906 - fix reliance on default encoding