Confusion between digit group separators and decimal marks

[sommerluk]

In JOSM, settings, audio player, speed, I can see “1.3” and that seems to mean “one (and three tenth). In JOSM, settings, WMS/TMS, settings, cache size, I can see “20.000” and that means “twenty thousand”. So, sometimes the full stop “.” is used as digit group separator and sometimes it is used as decimal mark. That’s extremely confusing and definitively wrong.

Furthermore, this is also a localization issue. In some countries, “,” is the decimal mark, in others “.” is the decimal mark. The world seems to be divided fifty-fifty. As I’m using JOSM with the localization “German (Germany)”, so I would expect JOSM to use “,” as decimal mark, but JOSM doesn’t. Maybe it is intentionally that JOSM ignores the localization of the decimal mark, because Geo-URLs often use the decimal point as separator and JOSM doesn’t want to confuse the user furthermore?

Anyway, for digit group separators it would be a good idea to use always consistently (and independent of the localization) in all fields a space, just as ISO 31-0 tells. The character U+202F NARROW NO-BREAK SPACE might be a good space character to use.

[stoecker]

Hmm, the issue is:

• All input fields are English standard "." as decimal separator
• All translated displayed texts (i.e. the cache size) are using the localized variant, because of the MessageFormat function used to create the text.

JOSM should use the English standard always, as that is also used in the database.

[sommerluk]

JOSM should use the English standard always,
as that is also used in the database.

It sounds indeed reasonable to use everywhere in JOSM the same style as in the OSM database. So:

Decimal mark: Always “.” (independently from language and locale)

Digit group separator: Never used (independently from language and locale)

Not using the digit group separator would make long numbers harder to read, but it’s the only way to get a consistent feeling between JOSM and the database.

[Don-vip]

Agreed. Concerning ISO, it's now ​ISO 80000-1, which states in its chapter 7.3.1 (Numbers, General):

To facilitate the reading of numbers with many digits, these may be separated into groups of three, counting from the decimal sign towards the left and the right. No group shall contain more than three digits. Where such separation into groups of three is used, the groups shall be separated by a small space and not by a point or a comma or by any other means.

In the case where there is no decimal part (and thus no decimal sign), the counting shall be from the rightmost digit towards the left.

The separation into groups of three should not be used for ordinal numbers used as reference numbers, e.g. ISO 80000-1.

[bastiK]

In 12917/josm:

see #14602 - don't use thousands separator when displaying the JOSM version

[Don-vip]

@bastiK: I have edited your svn log message to reference this bug report :)

[bastiK]

Replying to Don-vip:

@bastiK: I have edited your svn log message to reference this bug report :)

Each time I'm amazed it doesn't crash everything. :)

[stoecker]

Replying to bastiK:

Replying to Don-vip:

@bastiK: I have edited your svn log message to reference this bug report :)

Each time I'm amazed it doesn't crash everything. :)

Well, it is not CVS, RCS or SCCS. It's a bit more modern ;-)

[bastiK]

This removes the thousands separator (plainIntegerFormat) or changes it to thin space (smallSpaceIntegerFormat) for integers:

src/org/openstreetmap/josm/tools/I18n.java

@@ -10 +10 @@
 import java.lang.annotation.RetentionPolicy;
 import java.net.URL;
 import java.nio.charset.StandardCharsets;
+import java.text.DecimalFormat;
+import java.text.DecimalFormatSymbols;
 import java.text.MessageFormat;
+import java.text.NumberFormat;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collection;
@@ -114 +117 @@
      */
     public static String tr(String text, Object... objects) {
         if (text == null) return null;
-        return MessageFormat.format(gettext(text, null), objects);
+        return customMessageFormat(gettext(text, null), objects);
     }
 
+    public static final NumberFormat plainIntegerFormat = NumberFormat.getIntegerInstance();
+    public static final NumberFormat smallSpaceIntegerFormat = NumberFormat.getIntegerInstance();
+    static {
+        plainIntegerFormat.setGroupingUsed(false);
+        if (smallSpaceIntegerFormat instanceof DecimalFormat) {
+            DecimalFormat df = (DecimalFormat) smallSpaceIntegerFormat;
+            DecimalFormatSymbols dfs = df.getDecimalFormatSymbols();
+            dfs.setGroupingSeparator('\u2009');
+            df.setDecimalFormatSymbols(dfs);
+        }
+    }
+
+    public static String customMessageFormat(String text, Object... objects) {
+        MessageFormat mf = new MessageFormat(text);
+        for (int i = 0; i < objects.length; i++) {
+            if (objects[i] instanceof Integer || objects[i] instanceof Long) {
+                mf.setFormatByArgumentIndex(i, plainIntegerFormat);
+            }
+        }
+        return mf.format(objects);
+    }
+
     /**
      * Translates some text in a context for the current locale.
      * There can be different translations for the same text within different contexts.

[Don-vip]

Don't commit it please, I'm working on a totally different approach but it takes some time to debug Java internals...

[Don-vip]

OK, here's my solution:

build.xml

@@ -63 +63 @@
     <target name="create-revision-eclipse">
         <property name="revision.dir" value="bin"/>
         <antcall target="create-revision"/>
+        <mkdir dir="bin/META-INF/services"/>
+        <echo encoding="UTF-8" file="bin/META-INF/services/java.text.spi.DecimalFormatSymbolsProvider">org.openstreetmap.josm.tools.JosmDecimalFormatSymbolsProvider</echo>
     </target>
     <!--
       ** Initializes the REVISION.XML file from SVN information
@@ -150 +152 @@
                 <attribute name="Add-Exports" value="java.base/sun.security.util java.base/sun.security.x509 java.desktop/com.apple.eawt java.desktop/com.sun.imageio.spi javafx.graphics/com.sun.javafx.application jdk.deploy/com.sun.deploy.config" />
                 <attribute name="Add-Opens" value="java.base/java.lang java.base/jdk.internal.loader java.desktop/javax.imageio.spi java.desktop/javax.swing.text.html java.prefs/java.util.prefs" />
             </manifest>
+            <service type="java.text.spi.DecimalFormatSymbolsProvider" provider="org.openstreetmap.josm.tools.JosmDecimalFormatSymbolsProvider" />
             <zipfileset dir="images" prefix="images"/>
             <zipfileset dir="data" prefix="data"/>
             <zipfileset dir="styles" prefix="styles"/>

src/org/openstreetmap/josm/tools/I18n.java

@@ -89 +89 @@
     private static volatile Map<String, String> strings;
     private static volatile Map<String, String[]> pstrings;
     private static Map<String, PluralMode> languages = new HashMap<>();
+    static {
+        //languages.put("ar", PluralMode.MODE_AR);
+        languages.put("ast", PluralMode.MODE_NOTONE);
+        languages.put("bg", PluralMode.MODE_NOTONE);
+        languages.put("be", PluralMode.MODE_RU);
+        languages.put("ca", PluralMode.MODE_NOTONE);
+        languages.put("ca@valencia", PluralMode.MODE_NOTONE);
+        languages.put("cs", PluralMode.MODE_CS);
+        languages.put("da", PluralMode.MODE_NOTONE);
+        languages.put("de", PluralMode.MODE_NOTONE);
+        languages.put("el", PluralMode.MODE_NOTONE);
+        languages.put("en_AU", PluralMode.MODE_NOTONE);
+        languages.put("en_GB", PluralMode.MODE_NOTONE);
+        languages.put("es", PluralMode.MODE_NOTONE);
+        languages.put("et", PluralMode.MODE_NOTONE);
+        //languages.put("eu", PluralMode.MODE_NOTONE);
+        languages.put("fi", PluralMode.MODE_NOTONE);
+        languages.put("fr", PluralMode.MODE_GREATERONE);
+        languages.put("gl", PluralMode.MODE_NOTONE);
+        //languages.put("he", PluralMode.MODE_NOTONE);
+        languages.put("hu", PluralMode.MODE_NOTONE);
+        languages.put("id", PluralMode.MODE_NONE);
+        //languages.put("is", PluralMode.MODE_NOTONE);
+        languages.put("it", PluralMode.MODE_NOTONE);
+        languages.put("ja", PluralMode.MODE_NONE);
+        // fully supported only with Java 8 and later (needs CLDR)
+        languages.put("km", PluralMode.MODE_NONE);
+        languages.put("lt", PluralMode.MODE_LT);
+        languages.put("nb", PluralMode.MODE_NOTONE);
+        languages.put("nl", PluralMode.MODE_NOTONE);
+        languages.put("pl", PluralMode.MODE_PL);
+        languages.put("pt", PluralMode.MODE_NOTONE);
+        languages.put("pt_BR", PluralMode.MODE_GREATERONE);
+        //languages.put("ro", PluralMode.MODE_RO);
+        languages.put("ru", PluralMode.MODE_RU);
+        languages.put("sk", PluralMode.MODE_SK);
+        //languages.put("sl", PluralMode.MODE_SL);
+        languages.put("sv", PluralMode.MODE_NOTONE);
+        //languages.put("tr", PluralMode.MODE_NONE);
+        languages.put("uk", PluralMode.MODE_RU);
+        languages.put("vi", PluralMode.MODE_NONE);
+        languages.put("zh_CN", PluralMode.MODE_NONE);
+        languages.put("zh_TW", PluralMode.MODE_NONE);
+    }
 
     /**
      * Translates some text for the current locale.
@@ -300 +344 @@
         return languages.containsKey(code);
     }
 
+    static void setupJavaLocaleProviders() {
+        // Look up SPI providers first (for JosmDecimalFormatSymbolsProvider).
+        // Enable CLDR locale provider on Java 8 to get additional languages, such as Khmer.
+        // http://docs.oracle.com/javase/8/docs/technotes/guides/intl/enhancements.8.html#cldr
+        // FIXME: This must be updated after we switch to Java 9.
+        // See https://docs.oracle.com/javase/9/docs/api/java/util/spi/LocaleServiceProvider.html
+        System.setProperty("java.locale.providers", "SPI,JRE,CLDR"); // Don't call Utils.updateSystemProperty to avoid spurious log at startup
+    }
+
     /**
      * I18n initialization.
      */
     public static void init() {
-        // Enable CLDR locale provider on Java 8 to get additional languages, such as Khmer.
-        // http://docs.oracle.com/javase/8/docs/technotes/guides/intl/enhancements.8.html#cldr
-        // FIXME: This can be removed after we switch to a minimal version of Java that enables CLDR by default
-        // or includes all languages we need in the JRE. See http://openjdk.java.net/jeps/252 for Java 9
-        System.setProperty("java.locale.providers", "JRE,CLDR"); // Don't call Utils.updateSystemProperty to avoid spurious log at startup
-
-        //languages.put("ar", PluralMode.MODE_AR);
-        languages.put("ast", PluralMode.MODE_NOTONE);
-        languages.put("bg", PluralMode.MODE_NOTONE);
-        languages.put("be", PluralMode.MODE_RU);
-        languages.put("ca", PluralMode.MODE_NOTONE);
-        languages.put("ca@valencia", PluralMode.MODE_NOTONE);
-        languages.put("cs", PluralMode.MODE_CS);
-        languages.put("da", PluralMode.MODE_NOTONE);
-        languages.put("de", PluralMode.MODE_NOTONE);
-        languages.put("el", PluralMode.MODE_NOTONE);
-        languages.put("en_AU", PluralMode.MODE_NOTONE);
-        languages.put("en_GB", PluralMode.MODE_NOTONE);
-        languages.put("es", PluralMode.MODE_NOTONE);
-        languages.put("et", PluralMode.MODE_NOTONE);
-        //languages.put("eu", PluralMode.MODE_NOTONE);
-        languages.put("fi", PluralMode.MODE_NOTONE);
-        languages.put("fr", PluralMode.MODE_GREATERONE);
-        languages.put("gl", PluralMode.MODE_NOTONE);
-        //languages.put("he", PluralMode.MODE_NOTONE);
-        languages.put("hu", PluralMode.MODE_NOTONE);
-        languages.put("id", PluralMode.MODE_NONE);
-        //languages.put("is", PluralMode.MODE_NOTONE);
-        languages.put("it", PluralMode.MODE_NOTONE);
-        languages.put("ja", PluralMode.MODE_NONE);
-        // fully supported only with Java 8 and later (needs CLDR)
-        languages.put("km", PluralMode.MODE_NONE);
-        languages.put("lt", PluralMode.MODE_LT);
-        languages.put("nb", PluralMode.MODE_NOTONE);
-        languages.put("nl", PluralMode.MODE_NOTONE);
-        languages.put("pl", PluralMode.MODE_PL);
-        languages.put("pt", PluralMode.MODE_NOTONE);
-        languages.put("pt_BR", PluralMode.MODE_GREATERONE);
-        //languages.put("ro", PluralMode.MODE_RO);
-        languages.put("ru", PluralMode.MODE_RU);
-        languages.put("sk", PluralMode.MODE_SK);
-        //languages.put("sl", PluralMode.MODE_SL);
-        languages.put("sv", PluralMode.MODE_NOTONE);
-        //languages.put("tr", PluralMode.MODE_NONE);
-        languages.put("uk", PluralMode.MODE_RU);
-        languages.put("vi", PluralMode.MODE_NONE);
-        languages.put("zh_CN", PluralMode.MODE_NONE);
-        languages.put("zh_TW", PluralMode.MODE_NONE);
+        setupJavaLocaleProviders();
 
         /* try initial language settings, may be changed later again */
         if (!load(LanguageInfo.getJOSMLocaleCode())) {

src/org/openstreetmap/josm/tools/JosmDecimalFormatSymbolsProvider.java

@@ +1 @@
+// License: GPL. For details, see LICENSE file.
+package org.openstreetmap.josm.tools;
+
+import java.text.DecimalFormatSymbols;
+import java.text.spi.DecimalFormatSymbolsProvider;
+import java.util.Locale;
+
+/**
+ * JOSM implementation of the {@link java.text.DecimalFormatSymbols DecimalFormatSymbols} class,
+ * consistent with OSM API and ISO 80000-1.
+ * This class will only be used with Java 9 and later runtimes, as Java 8 implementation relies
+ * on Java Extension Mechanism only, while Java 9 supports application classpath.
+ * See {@link java.util.spi.LocaleServiceProvider LocaleServiceProvider} javadoc for more details.
+ * @since xxx
+ */
+public class JosmDecimalFormatSymbolsProvider extends DecimalFormatSymbolsProvider {
+
+    @Override
+    public DecimalFormatSymbols getInstance(Locale locale) {
+        DecimalFormatSymbols symbols = new DecimalFormatSymbols(locale);
+        // Override decimal mark to be consistent with OSM API
+        symbols.setDecimalSeparator('.');
+        // Override digit group separator to be consistent across languages with ISO 80000-1, chapter 7.3.1
+        symbols.setGroupingSeparator('\u202F'); // U+202F: NARROW NO-BREAK SPACE
+        return symbols;
+    }
+
+    @Override
+    public Locale[] getAvailableLocales() {
+        return I18n.getAvailableTranslations();
+    }
+}

src/org/openstreetmap/josm/tools/Logging.java

@@ -51 +51 @@
     private static final RememberWarningHandler WARNINGS = new RememberWarningHandler();
 
     static {
+        // We need to be sure java.locale.providers system property is initialized by JOSM, not by JRE
+        // The call to ConsoleHandler constructor makes the JRE access this property by side effect
+        I18n.setupJavaLocaleProviders();
+
         LOGGER.setLevel(Level.ALL);
         LOGGER.setUseParentHandlers(false);

Looks simple but it took me quite a few hours to understand a serious design limitation of Java 6/7/8 removed in Java 9.

With this, we register a new ​LocaleServiceProvider (a ​DecimalFormatSymbolsProvider to be exact) which will override the decimal symbols for all locales supported by JOSM. Absolutely everywhere, so there is no need for a new public API or to track the formatting calls we make.

I also create the META-INF/services folder in Eclipse bin directory so that the mechanism works when running JOSM from Eclipse. Ant takes care of creating it in the jar by itself.

The fun point: this API introduced in Java 6 is clearly elegant, but it was absolutely worthless until Java 9. Why? Because up to Java 8 it works only with service providers registered through the ​Java Extension Mechanism (putting jars in the JRE directory...) which is non-practical for us, and a deprecated technology removed in Java 9. Java 9 removes this horrible restriction and allows now service providers registered in the application classpath to override JRE locale providers, which is exactly what we need.

So the good point: the code does not require Java 9 to compile, as the API is there since Java 6.
the bad point: the code require Java 9 at runtime. But as it is a very minor defect, and we will likely migrate to Java 9 next year anyway, I don't think that's a problem.

Thoughts?

[bastiK]

Your solution looks much more like a proper configuration than my crude hack. It is a good idea to get rid of localized group separator and use thin space instead.

I'm not sure it is okay to ditch the decimal mark localization for the sake of consistency, though. It looks strange to have for example "0.5" in the middle of a German sentence. This would seem as if it is incorrectly or only partially translated.

[stoecker]

+1 Please restore proper decimal point/comma. That would be a serious step back to introduce a point everywhere.

[stoecker]

Question remains how we can keep the point as standard for OSM data without destroying the comma/point as language standard for the UI.

An auto-conversion does probably more harm than good.

[bastiK]

So where do we have floating point input?

• Tools add node: Both point and comma is already allowed for input
• Audio preferences: I would switch to localized decimal mark, but still allow decimal point as input (auto-conversion when saving to the preferences)
• Imagery layer Offset - same here (coordinates are separated by semicolon, so no problem)
• Tag value: User must be educated, that decimal point is the standard for OSM tagging, no way around that. (Validator warnings should give a hint)
• Download dialog / Bounding Box: Not sure, coordinates are likely to be copy & pasted to and from the OSM website
• Advanced preferences: clue is in the name, user must know what they are doing

[Don-vip]

In 12931/josm:

see #14602 - Override digit group separator to be consistent across languages with ISO 80000-1 + checkstyle fixes

[Don-vip]

In 13050/josm:

fix #14602 - allow both dot and comma decimal separator everywhere possible for user-entered values

[Don-vip]

See #15552