id summary reporter owner description type status priority milestone component version resolution keywords cc 24940 [PATCH] Fall back to a UTF-8 locale on Linux, and explain unmappable file names instead of asking for a bug report wangi team "This follows up #14596, which is closed as invalid but still collects duplicates: more than 30 so far, and it was updated again this week. All of them are `java.nio.file.InvalidPathException: Malformed input or input contains unmappable characters`, thrown by the file chooser when a folder holds a file with a non-ASCII name. ==== What steps will reproduce the problem? 1. On Linux, start JOSM with a UTF-8 locale which is not installed, e.g. `LANG=fr_FR.UTF-8` on a system that has only `en_US.UTF-8` — which is what happens inside a Flatpak sandbox — or with `LANG=C`. 2. File > Open, and go to a folder which holds a file called `Capture d'écran.png`. ==== What is the expected result? The folder is listed. ==== What happens instead? `InvalidPathException`, and JOSM asks for a bug report. ==== Please provide any additional information below. Attach a screenshot if possible. The reasoning in #14596 is that the user chose `LANG=C`. That doesn't match most of the duplicates: since the status report gained the encodings ([comment:41:ticket:14596 #14596 comment:41]), many show `LANG=en_GB.UTF-8` next to `sun.jnu.encoding=ANSI_X3.4-1968`. That is a UTF-8 locale which is '''not installed''': glibc falls back to C, and Java then decodes every file name as ASCII. Measured with OpenJDK 25: ||= Environment =||= `sun.jnu.encoding` =||= non-ASCII file name =|| || `LANG=en_GB.UTF-8` (installed) || UTF-8 || OK || || `LANG=C` || ANSI_X3.4-1968 || `InvalidPathException` || || `LANG=fr_FR.UTF-8` (not installed) || ANSI_X3.4-1968 || `InvalidPathException` || || `LANG=C` `-Dsun.jnu.encoding=UTF-8` || ANSI_X3.4-1968 || `InvalidPathException` || || `LANG=C` `-Dfile.encoding=UTF-8` || ANSI_X3.4-1968 || `InvalidPathException` || || `LANG=C.UTF-8` || UTF-8 || OK || || `LANG=fr_FR.UTF-8` `LC_CTYPE=C.UTF-8` || ANSI_X3.4-1968 || `InvalidPathException` || || `LANG=fr_FR.UTF-8` `LC_ALL=C.UTF-8` || UTF-8 || OK || So JOSM cannot fix this once Java is running: `sun.jnu.encoding` is fixed at startup, and setting it with `-D` is ignored (the suggestion in [comment:40:ticket:14596 #14596 comment:40] does not work); `file.encoding` is not used for file names at all. It has to be fixed in the environment before Java starts — which, as [comment:74:ticket:14596 #14596 comment:74] says, depends on the installer. Note also the second-to-last row: setting only `LC_CTYPE` is not enough, because `setlocale(LC_ALL, """")` fails as a whole when any category names a locale which is not installed. The attached patches are independent of each other. '''utf8-file-names-launcher.patch''' — the installer part. The Linux launchers `native/linux/{tested,latest}/usr/bin/josm*` set `LC_ALL=C.UTF-8` when the locale cannot be set or is ASCII, provided `C.UTF-8` is available (it is built into glibc 2.35 and later): {{{#!sh if { [ -n ""$(locale 2>&1 >/dev/null)"" ] || [ ""$(locale charmap 2>/dev/null)"" = ""ANSI_X3.4-1968"" ]; } \ && [ -z ""$(LC_ALL=C.UTF-8 locale 2>&1 >/dev/null)"" ]; then export LC_ALL=C.UTF-8 fi }}} * This reaches Flatpak users as well: the Flathub package runs this script (its only patch adds HiDPI scaling). * It does not change the language of JOSM. When the locale cannot be set, Java already falls back to English. * Locales which work are left alone, including non-UTF-8 ones such as ISO-8859-1, where `sun.jnu.encoding` is legitimately not UTF-8. If `locale` is missing, or `C.UTF-8` is not available, the check does nothing. * Tested by running both scripts, with a stand-in for `java` that lists such a folder, under `LANG=C`, a missing `LANG`, a missing `LANG` with a valid `LC_CTYPE`, and a working locale. All four list the folder; the unmodified script fails with the missing `LANG`. `shellcheck` 0.11.0 reports nothing new: its only finding, SC1091 about the sourced `/etc/default/josm`, is the same on the unmodified scripts. * One thing I could not check: whether `/usr/bin/locale` is present in the Freedesktop runtime. If it is not, the check does nothing there. '''utf8-file-names-dialog.patch''' — for the ways of starting JOSM whose launcher JOSM does not control: Snap (#22887), Java Web Start and plain `java -jar`. A JNLP file can set system properties but not environment variables, so Web Start cannot be fixed from `josm.jnlp`. New `ReportedException.isUnmappableFileName()` recognises such an `InvalidPathException` in the cause chain while `sun.jnu.encoding` is not UTF-8, and `BugReportDialog.showFor()` then explains the problem instead of asking for a bug report, as it already does for an `OutOfMemoryError`: {{{ JOSM cannot handle the name of a file, because it is running with the character set ANSI_X3.4-1968 for file names, which cannot represent it. This happens when JOSM is started without a UTF-8 locale, for example with LANG=C, or with a locale which is not installed. Please start JOSM with the environment variable LC_ALL=C.UTF-8. }}} It is shown once per session, since browsing another folder throws the same exception again. Requiring `sun.jnu.encoding` not to be UTF-8 keeps it from blaming the locale when it is not at fault, e.g. for a file name in ISO-8859-1 on a UTF-8 system. Unit tests: `ReportedExceptionTest.testIsUnmappableFileName()` covers the exception on its own and as a cause, another `InvalidPathException`, an unrelated exception, and a UTF-8 file name encoding. New `BugReportDialogTest` checks that the message is shown, exactly once, and that no bug report dialog is opened. Both fail without the patch. All tests in `org.openstreetmap.josm.tools.bugreport` and `org.openstreetmap.josm.gui.bugreport` pass, and checkstyle and PMD report nothing for the changed files. Both patches are against r19636. " enhancement new normal Core linux locale encoding InvalidPathException flatpak template_report