Plugins list unreadable outside en_US locale: response sent as double-encoded-UTF8, with mangled multibyte characters

[Piskvor]

Steps to reproduce:

1. Launch JOSM in a locale which does not exclusively use characters inside ASCII (Czech, French, Japanese, whatever).
2. Open Preferences (F12)
3. Go to tab "Plugins"
4. Click "Download list" to get the available list of plugins.
5. See bogus characters in place of any multi-byte character (single-byte characters are not affected).

Simpler steps to reproduce:

1. Go to https://josm.openstreetmap.de/plugin - the list of plugins is downloaded from this address
2. See bogus characters (anything starting with "Ã" or "Ð").

This is not, strictly speaking, a bug in the JOSM program, but rather in the JOSM *website* - note that the HTTP response is giving out the correct charset, but all multibyte characters are encoded into double their size: for a two-byte character, four bytes are sent.

From other experiences with UTF-8, it seems to me that the plugin list is already UTF-8, yet before it's sent to the HTTP client, something is again taking that bytestream and reencoding to UTF-8 as if it were Latin-1, which gives out the results seen here. Is there now perhaps an automated UTF-8 conversion filter in place? I did not see this issue in the plugins list a week ago. See e.g. this: ​http://www.i18nqa.com/debug/bug-double-conversion.html

Status report:

Repository Root: http://josm.openstreetmap.de/svn
Build-Date: 2014-09-08 01:33:36
Last Changed Author: stoecker
Revision: 7512
Repository UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
Relative URL: ^/trunk
URL: http://josm.openstreetmap.de/svn/trunk
Last Changed Date: 2014-09-07 23:35:38 +0200 (Sun, 07 Sep 2014)
Last Changed Rev: 7512

Identification: JOSM/1.5 (7512 cs) Linux Ubuntu 14.04.1 LTS
Memory Usage: 350 MB / 1765 MB (113 MB allocated, but free)
Java version: 1.7.0_67, Oracle Corporation, Java HotSpot(TM) 64-Bit Server VM
VM arguments: [-Djosm.restart=true, -Djosm.home=/home/honza/.josm-latest, -Djava.net.useSystemProxies=true]

Plugins:
- ElevationProfile (30416)
- continuosDownload (28565)
- geochat (30513)
- notes (v0.9.4)
- turnrestrictions (30454)

[Don-vip]

Another unexpected side effect of the server migration

[stoecker]

Looks like trac.wsgi (or apache config) need to be changed to "don't touch I/O". I bet it is trac.wsgi.

[Don-vip]

Replying to stoecker:

I bet it is trac.wsgi.

The problem is before that. The file /home/josm/auto/plugins.txt is badly encoded so it's transfered "as it".

[stoecker]

Ah ok. Already wondered why the webpage works elsewhere. Is an issue with the new perl version. I temporary fixed it by reverting the encoding handling.

[stoecker]

Which means that I again get a warning email for each cronjob run till this is fixed. :-(

[Don-vip]

Do you have some more details on the perl issue? What's wrong?

[stoecker]

The

use encoding 'utf8';
use open ':utf8';

is deprecated and should be replaced by this:

use open qw/:std :encoding(utf8)/;

But this has a slightly different behaviour, so some other parts must be fixed as well. It's not the first time I encounter that. Needs some cleanup all-over the code, but result is much less "magic".

[stoecker]

Fixed. Please report any other similar issues.