Modify

Opened 6 years ago

Closed 6 years ago

#9906 closed defect (fixed)

Default encoding problems

Reported by: akks Owned by: team
Priority: normal Milestone: 14.04
Component: Core Version:
Keywords: Cc:

Description

People from forum (Xmypblu, LLlypuk82 etc.) noticed that notes and scripting plugins are showing incorrectly encoded messages
(on some Windows machines by default, can be fixed with -Dfile.encoding=utf8)

Debugging showed that the reason is in JOSM core:
BufferedReader in = new BufferedReader(new InputStreamReader(i));
in OsmApi.java uses default encoding (not always UTF8).

I have added Utils.UTF_8 to the call and hope there will be no regressions.

The same default InputStreamReader is used in other classes too:
AbstractOAuthProvider, OAuth, AutosaveTask, ServerSidePreferences, Projections, MapPaintDialog, MapPaintStyles, XmlStyleSource, OsmOAuthAuthorizationClient, ApiUrlTestTask, NmeaReader, OsmServerReader (???), PlatformHookUnixoid, Utils.execOutput.

Not sure if they can cause analogous problems, can someone help?

Attachments (1)

A7aPK1E[1].png (11.1 KB) - added by akks 6 years ago.

Download all attachments as: .zip

Change History (14)

Changed 6 years ago by akks

Attachment: A7aPK1E[1].png added

comment:1 Changed 6 years ago by akks

Forgot to mention ticket in changeset:

In [6966/josm]:

Fix default encoding for non-UTF consoles (for Notes plugin etc.)

comment:2 Changed 6 years ago by akks

To get sample of old incorrect encoding on old build in UTF8 console you need to run
java -Dfile.encoding=cp866 -jar josm-latest.jar
(only way to get Russian characters in Windows XP console)

comment:3 in reply to:  1 Changed 6 years ago by skyper

Replying to akks:

Forgot to mention ticket in changeset:

In [6966/josm]:

Fix default encoding for non-UTF consoles (for Notes plugin etc.)

Does this fix #9858 ?

comment:4 Changed 6 years ago by akks

Yes, I hope so. I just missed the tickets (searched for "encoding" instead of "charset")...

Last edited 6 years ago by akks (previous) (diff)

comment:5 Changed 6 years ago by Don-vip

Milestone: 14.04

comment:6 Changed 6 years ago by akks

I was asked to fix another encoding-related bug:
the following link

http://127.0.0.1:8111/import?url=http://overpass-api.de/api/interpreter?data=relation[%22admin_level%22=%224%22][%22name%22=%22%D0%9A%D0%B0%D0%BB%D0%B8%D0%BD%D0%B8%D0%BD%D0%B3%D1%80%D0%B0%D0%B4%D1%81%D0%BA%D0%B0%D1%8F+%D0%BE%D0%B1%D0%BB%D0%B0%D1%81%D1%82%D1%8C%22];(way(r:%22outer%22);node(w););out%20meta;

works correctly only for utf console. Running it with java -Dfile.encoding=cp866 -jar josm-latest.jar or from Windows console do not download anything. I was not able to fix it. OSMReader keeps returning empty dataset.

comment:7 Changed 6 years ago by bastiK

The remote control handler (correctly) decodes the url argument with URLDecoder.decode(param, "UTF-8");.

The result is the following string:
http://overpass-api.de/api/interpreter?data=relation["admin_level"="4"]["name"="Калининградская область"];(way(r:"outer");node(w););out meta;

This is not a valid URL, because it contains non ASCII characters. So this is in fact an error on the user site.

We can handle this anyway if we percent-encode non ASCII characters at this point.

comment:9 Changed 6 years ago by akks

By the way, the request to http://127.0.0.1:8111/... was not working behind proxy, so I have added 127.0.0.1 to the exceptions list:

[6966/josm] / Add proxy.exceptions advanced parameter (localhost and 127.0.0.1 by default)

(is it OK that I did not create separate ticket for it?)

comment:10 Changed 6 years ago by bastiK

In 6972/josm:

see #9906 - remotecontrol import handler: add workaround when URLs with multibyte
characters are only one time percent encoded

comment:11 in reply to:  6 Changed 6 years ago by bastiK

Replying to akks:

I was asked to fix another encoding-related bug:
the following link

http://127.0.0.1:8111/import?url=http://overpass-api.de/api/interpreter?data=relation[%22admin_level%22=%224%22][%22name%22=%22%D0%9A%D0%B0%D0%BB%D0%B8%D0%BD%D0%B8%D0%BD%D0%B3%D1%80%D0%B0%D0%B4%D1%81%D0%BA%D0%B0%D1%8F+%D0%BE%D0%B1%D0%BB%D0%B0%D1%81%D1%82%D1%8C%22];(way(r:%22outer%22);node(w););out%20meta;

works correctly only for utf console. Running it with java -Dfile.encoding=cp866 -jar josm-latest.jar or from Windows console do not download anything. I was not able to fix it. OSMReader keeps returning empty dataset.

This should work now.

comment:12 Changed 6 years ago by akks

Thanl you very much! It really works.
I guess currently to open something with /import?, the authors of tools now just need to append this text to "http://127.0.0.1:8111/import?url=".

Version 0, edited 6 years ago by akks (next)

comment:13 Changed 6 years ago by Don-vip

Resolution: fixed
Status: newclosed

In 6995/josm:

fix #9906 - fix reliance on default encoding

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain team.
as The resolution will be set.
The resolution will be deleted.

Add Comment


E-mail address and name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.