Modify

Opened 7 years ago

Closed 7 years ago

#10492 closed defect (fixed)

Plugins list unreadable outside en_US locale: response sent as double-encoded-UTF8, with mangled multibyte characters

Reported by: Piskvor Owned by: stoecker
Priority: major Milestone:
Component: Trac Version:
Keywords: Cc:

Description (last modified by Don-vip)

Steps to reproduce:

  1. Launch JOSM in a locale which does not exclusively use characters inside ASCII (Czech, French, Japanese, whatever).
  2. Open Preferences (F12)
  3. Go to tab "Plugins"
  4. Click "Download list" to get the available list of plugins.
  5. See bogus characters in place of any multi-byte character (single-byte characters are not affected).

Simpler steps to reproduce:

  1. Go to https://josm.openstreetmap.de/plugin - the list of plugins is downloaded from this address
  2. See bogus characters (anything starting with "Ã" or "Ð").

This is not, strictly speaking, a bug in the JOSM program, but rather in the JOSM *website* - note that the HTTP response is giving out the correct charset, but all multibyte characters are encoded into double their size: for a two-byte character, four bytes are sent.

From other experiences with UTF-8, it seems to me that the plugin list is already UTF-8, yet before it's sent to the HTTP client, something is again taking that bytestream and reencoding to UTF-8 as if it were Latin-1, which gives out the results seen here. Is there now perhaps an automated UTF-8 conversion filter in place? I did not see this issue in the plugins list a week ago. See e.g. this: http://www.i18nqa.com/debug/bug-double-conversion.html

Status report:

Repository Root: http://josm.openstreetmap.de/svn
Build-Date: 2014-09-08 01:33:36
Last Changed Author: stoecker
Revision: 7512
Repository UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
Relative URL: ^/trunk
URL: http://josm.openstreetmap.de/svn/trunk
Last Changed Date: 2014-09-07 23:35:38 +0200 (Sun, 07 Sep 2014)
Last Changed Rev: 7512

Identification: JOSM/1.5 (7512 cs) Linux Ubuntu 14.04.1 LTS
Memory Usage: 350 MB / 1765 MB (113 MB allocated, but free)
Java version: 1.7.0_67, Oracle Corporation, Java HotSpot(TM) 64-Bit Server VM
VM arguments: [-Djosm.restart=true, -Djosm.home=/home/honza/.josm-latest, -Djava.net.useSystemProxies=true]

Plugins:
- ElevationProfile (30416)
- continuosDownload (28565)
- geochat (30513)
- notes (v0.9.4)
- turnrestrictions (30454)

Attachments (0)

Change History (10)

comment:1 Changed 7 years ago by Don-vip

Component: CoreTrac
Owner: changed from team to stoecker

Another unexpected side effect of the server migration

comment:2 Changed 7 years ago by Don-vip

Description: modified (diff)

comment:3 Changed 7 years ago by stoecker

Priority: normalmajor

comment:4 Changed 7 years ago by stoecker

Looks like trac.wsgi (or apache config) need to be changed to "don't touch I/O". I bet it is trac.wsgi.

comment:5 in reply to:  4 Changed 7 years ago by Don-vip

Replying to stoecker:

I bet it is trac.wsgi.

The problem is before that. The file /home/josm/auto/plugins.txt is badly encoded so it's transfered "as it".

comment:6 Changed 7 years ago by stoecker

Ah ok. Already wondered why the webpage works elsewhere. Is an issue with the new perl version. I temporary fixed it by reverting the encoding handling.

comment:7 Changed 7 years ago by stoecker

Which means that I again get a warning email for each cronjob run till this is fixed. :-(

comment:8 Changed 7 years ago by Don-vip

Do you have some more details on the perl issue? What's wrong?

comment:9 Changed 7 years ago by stoecker

The

use encoding 'utf8';
use open ':utf8';

is deprecated and should be replaced by this:

use open qw/:std :encoding(utf8)/;

But this has a slightly different behaviour, so some other parts must be fixed as well. It's not the first time I encounter that. Needs some cleanup all-over the code, but result is much less "magic".

comment:10 Changed 7 years ago by stoecker

Resolution: fixed
Status: newclosed

Fixed. Please report any other similar issues.

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain stoecker.
as The resolution will be set.
The resolution will be deleted.

Add Comment


E-mail address and name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.