Modify

Opened 4 weeks ago

Closed 3 weeks ago

Last modified 3 weeks ago

#18608 closed defect (fixed)

Command line output unreadable on Windows

Reported by: gaben Owned by: Don-vip
Priority: normal Milestone: 20.01
Component: Core Version: latest
Keywords: template_report windows console encoding Cc: Don-vip

Description (last modified by gaben)

What steps will reproduce the problem?

  1. Use Windows, set lang to Hungarian
  2. Set JOSM language to Hungarian
  3. Start JOSM from command line (cmd or powershell, doesn't matter):

java -jar "josm-snapshot-15752.jar"

What is the expected result?

Readable output with valid characters.

What happens instead?

JOSM printing invalid characters and it's not readable easily. See below:

2020-01-23 17:28:11.200 INFO: Naplˇzßsi szint: INFO (INFO, 800)
2020-01-23 17:28:11.370 INFO: Removed preference default remotecontrol.https.enabled since it is no longer used
2020-01-23 17:28:11.428 INFO: Saved updated default preferences.
2020-01-23 17:28:11.853 INFO: IPv6 hßlˇzat Úszlelve; IPv6 preferßlßsa IPv4 ellenÚben.
2020-01-23 17:28:12.777 INFO: 'jaxb' b§vÝtmÚny bet÷ltÚse (35092 verziˇ)
2020-01-23 17:28:13.171 INFO: Reassigning shortcut 'file:download_along' from null to 640 because of conflict with Alt+Shift+D
2020-01-23 17:28:13.172 INFO: Csendes gyorsbillentyű konfliktus: 'file:download_along' ßtmozgatßsra kerŘlt 'continuosdownload:activate'-r§l 'Ctrl+Alt+F1'-re.
2020-01-23 17:28:13.539 INFO: GET https://api.openstreetmap.org/api/0.6/user/details -> HTTP_1 200 (683 B)
2020-01-23 17:28:14.040 INFO: GET https://josm.openstreetmap.de/wiki/Hu:StartupPage -> HTTP_1 200
2020-01-23 17:28:14.120 INFO: Obtained 51 Tag2Link rules from https://query.wikidata.org/sparql
2020-01-23 17:28:14.175 INFO: Obtained 11 Tag2Link rules from https://sophox.org/sparql
2020-01-23 17:28:14.408 INFO: "Bing műholdkÚp" lÚgifelvÚtel frissÝtÚse
2020-01-23 17:28:14.408 INFO: "Esri World Imagery" lÚgifelvÚtel frissÝtÚse
2020-01-23 17:28:14.410 INFO: "Esri World Imagery (Clarity) Beta" lÚgifelvÚtel frissÝtÚse

Please provide any additional information below. Attach a screenshot if possible.

For example, the first entry is
2020-01-23 17:28:11.200 INFO: Naplˇzßsi szint: INFO (INFO, 800)
instead of
2020-01-23 17:28:11.200 INFO: Naplózási szint: INFO (INFO, 800)

Older JOSM versions work fine. I checked r10526 and it printed readable characters.

URL:https://josm.openstreetmap.de/svn/trunk
Repository:UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
Last:Changed Date: 2020-01-22 22:31:24 +0100 (Wed, 22 Jan 2020)
Build-Date:2020-01-23 02:30:55
Revision:15752
Relative:URL: ^/trunk

Identification: JOSM/1.5 (15752 hu) Windows 10 64-Bit
OS Build number: Windows 10 Pro 1909 (18363)
Memory Usage: 1066 MB / 1820 MB (666 MB allocated, but free)
Java version: 1.8.0_231-b11, Oracle Corporation, Java HotSpot(TM) 64-Bit Server VM
Screen: \Display0 1920x1200
Maximum Screen Size: 1920x1200
VM arguments: [-Djava.security.manager, -Djava.security.policy=file:<java.home>\lib\security\javaws.policy, -DtrustProxy=true, -Djnlpx.home=<java.home>\bin, -Djnlpx.origFilenameArg=%UserProfile%\AppData\LocalLow\Sun\Java\Deployment\cache\6.0\31\583aa85f-3ed5e881, -Djnlpx.remove=false, -Djava.util.Arrays.useLegacyMergeSort=true, -Djnlpx.heapsize=NULL,2048m, -Djnlpx.splashport=57967, -Djnlp.application.href=https://josm.openstreetmap.de/download/josm-latest.jnlp, -Djnlpx.jvm=<java.home>\bin\javaw.exe]

Attachments (1)

18608v1.patch (2.0 KB) - added by gaben 3 weeks ago.
my first patch ever on the internet :)

Download all attachments as: .zip

Change History (18)

comment:1 Changed 4 weeks ago by Don-vip

Keywords: windows console encoding added

comment:2 Changed 4 weeks ago by gaben

Just found JOSM archive so I could do some research :) The last working archive version is r10895 and the first wrong one is r10899.

The r10899 printing this new entry while older versions not:
FINE: System property 'java.locale.providers' set to 'JRE,CLDR'. Old value was 'null'

Changed 3 weeks ago by gaben

Attachment: 18608v1.patch added

my first patch ever on the internet :)

comment:3 Changed 3 weeks ago by gaben

Summary: Command line output unreadable on Windows[PATCH] Command line output unreadable on Windows

comment:4 Changed 3 weeks ago by gaben

I'm not sure if I broke anything, but it works fine for me. Please check your output.

comment:5 Changed 3 weeks ago by gaben

It fixed only the IDE terminal output, the system terminal got new charset which is still invalid. Maybe a terminal encoding check needed?

comment:6 Changed 3 weeks ago by simon04

Hello gaben and welcome to JOSM development! :)

Thank you for your patch. I did some testing and managed to get correct console output on Windows' cmd.exe by doing the following:

Microsoft Windows [Version 10.0.18363.592]
(c) 2019 Microsoft Corporation. Alle Rechte vorbehalten.

C:\Program Files (x86)\JOSM>chcp 65001
Active Codepage: 65001.

C:\Program Files (x86)\JOSM>java -Dfile.encoding=UTF-8 -jar josm-tested.jar --language=ru --debug
...
2020-01-26 23:16:34.435 FINE: Инициализация
2020-01-26 23:16:34.451 FINE: Определение модулей для скачивания...
2020-01-26 23:16:34.451 FINE: Plugins list initialized to []
2020-01-26 23:16:34.451 FINE: Удаление нежелательных модулей...
2020-01-26 23:16:34.451 FINE: Удаление неподдерживаемых модулей...
2020-01-26 23:16:34.451 FINE: Plugins list is finally set to []
2020-01-26 23:16:34.451 FINE: Чтение информации об установленных модулях...
2020-01-26 23:16:34.451 FINE: Модуль обработки файлов кэша сайта...
2020-01-26 23:16:34.467 FINE: Обработка файла 'site-josm.openstreetmap.de-_pluginicons.txt'
2020-01-26 23:16:34.857 FINE: Установка обновлённых модулей
2020-01-26 23:16:34.857 FINE: Загрузка ранних модулей
2020-01-26 23:16:34.857 FINE: Загрузка модулей с сервера...
2020-01-26 23:16:34.873 FINE: Проверка входных условий для модулей...
2020-01-26 23:16:34.873 FINE: Настройки по умолчанию
2020-01-26 23:16:35.201 FINE: Создание интерфейса
2020-01-26 23:16:35.217 FINE: Инициализация формата координат
2020-01-26 23:16:35.232 FINE: Инициализация формата координат завершено за  (15 мс)
2020-01-26 23:16:35.232 FINE: Запуск наблюдения за файлами
2020-01-26 23:16:35.232 FINE: Запуск наблюдения за файлами завершено за  (0 мс)
2020-01-26 23:16:35.232 FINE: File watcher thread started
2020-01-26 23:16:35.232 FINE: Исполнение стартовых процедур
2020-01-26 23:16:35.232 FINE: java.lang.ClassNotFoundException: com.sun.deploy.config.BuiltInProperties
2020-01-26 23:16:35.232 FINE: Исполнение стартовых процедур завершено за  (0 мс)
2020-01-26 23:16:35.232 FINE: Создание основного меню
2020-01-26 23:16:36.467 FINE: Создание основного меню завершено за  (1,2 с)
2020-01-26 23:16:36.467 FINE: Обновление интерфейса пользователя
2020-01-26 23:16:36.529 FINE: Обновление интерфейса пользователя завершено за  (62 мс)
2020-01-26 23:16:36.529 FINE: Инициализация внутренних данных о границах
2020-01-26 23:16:36.560 FINE: Подготовка данных...
2020-01-26 23:16:36.560 FINE: Разбор данных OSM...
...

So basically, one has to invoke chcp 65001 in the cmd.exe window before launching JOSM. The -Dfile.encoding=UTF-8 has a similar effect as your changes from attachment:18608v1.patch

I updated the documentation at Help/CommandLineOptions#Javaoptions.

comment:7 Changed 3 weeks ago by gaben

Thanks, Simon! :)

Checked your findings and could confirm the workaround. It's working in cmd but unfortunately not in PowerShell. With the patch, I wanted to replicate the pre r10899 behaviour which works fine without any extra arguments or console tweaking in both environments.

You can download it from here https://josm.openstreetmap.de/download/Archiv/josm-snapshot-10895.jar

(The patch indeed equivalent with the -Dfile.encoding=UTF-8 flag. Nevermind, at least I tried.)

comment:8 Changed 3 weeks ago by gaben

Summary: [PATCH] Command line output unreadable on WindowsCommand line output unreadable on Windows

comment:9 Changed 3 weeks ago by stoecker

Cc: Don-vip added

Is the behaviour wrong since r10899 or is no longer UTF-8 since then? As far as I know in Windows shells there is no proper way to determine the correct encoding, so it can only be guessed.

So I assume output was broken before r10899 for standard Windows.

Your patch is wrong because of this. UTF-8 is not the correct choice for all systems. Either there is a way to autodetect the correct choice in JOSM (BTW: did you try to set proper LANG environment variables like ru_RU.UTF-8 in the system) or you need to pass UTF-8 as argument.

comment:10 in reply to:  9 Changed 3 weeks ago by gaben

Description: modified (diff)

Replying to stoecker:

So I assume output was broken before r10899 for standard Windows.

I just tested the behaviour in a virtual machine with

  • an official Windows 10 1909 x64 (Hungarian) ISO
  • with JDK 1.8.241 (x64) and JDK 11.0.6 (x64)
  • in PowerShell and cmd

and I can say this is a regression. It's broken since r10899.

Updated the steps.

comment:11 Changed 3 weeks ago by stoecker

Milestone: 20.02

comment:12 Changed 3 weeks ago by Don-vip

Confirmed:

PS C:\SVN\josm\core> chcp
Page de codes active : 850

PS C:\SVN\josm\core> java -jar .\dist\josm-custom.jar
2020-01-27 23:35:57.158 INFOS: Le niveau de journalisation est Ó INFOS (INFO, 800)
2020-01-27 23:35:57.920 INFOS: RÚseau IPv6 utilisable dÚtectÚ, prÚfÚrer IPv6 Ó IPv4.

PS C:\SVN\josm\core> java -jar .\dist\josm-snapshot-10895.jar
INFO : Un réseau IPv6 a été détecté, il sera utilisé de préférence à IPv4.

PS C:\SVN\josm\core> java -jar .\dist\josm-snapshot-10899.jar
janv. 27, 2020 11:37:47 PM org.openstreetmap.josm.Main debug
PR╔CIS: System property 'java.locale.providers' set to 'JRE,CLDR'. Old value was 'null'
janv. 27, 2020 11:37:47 PM org.openstreetmap.josm.Main info
INFOS: Log level is at
janv. 27, 2020 11:37:47 PM org.openstreetmap.josm.Main info
INFOS: Un rÚseau IPv6 a ÚtÚ dÚtectÚ, il sera utilisÚ de prÚfÚrence Ó IPv4.
Last edited 3 weeks ago by Don-vip (previous) (diff)

comment:13 Changed 3 weeks ago by Don-vip

It works out of the box because System.out is initialized by native code retrieving the Windows console encoding ("Cp850" for me). We lose this feature when using ConsoleHandler, as it relies on Charset.defaultCharset() instead ("windows-1252" for me). We cannot get the information from Windows ourselves (native code) but we can simply use the "sun.stdout.encoding" property.

comment:14 Changed 3 weeks ago by Don-vip

Milestone: 20.0220.01
Owner: changed from team to Don-vip
Status: newassigned

comment:15 Changed 3 weeks ago by Don-vip

Resolution: fixed
Status: assignedclosed

In 15791/josm:

fix #18608 - use the correct console encoding on Windows (regression from r10899)

comment:16 Changed 3 weeks ago by gaben

Fix confirmed, thank you @Don-vip!

comment:17 Changed 3 weeks ago by Don-vip

It was a complex but interesting issue to understand why System.out.println() did not produce the same output as a ConsoleHandler configured with System.out :)

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain Don-vip.
as The resolution will be set.
The resolution will be deleted.

Add Comment


E-mail address and name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.