DefaultNameFormatter: add missing U+202C POP DIRECTIONAL FORMATTING

[simon04]

Follow up of #10239.

To avoid nasty control characters in various Swing components (see https://josm.openstreetmap.de/attachment/ticket/19703/relation%20editor.PNG), we can make use of the Unicode characters FIRST-STRONG ISOLATE and POP DIRECTIONAL ISOLATE.

Further reading:

• ​https://www.w3.org/International/questions/qa-bidi-unicode-controls
• ​https://www.w3.org/International/articles/inline-bidi-markup/uba-basics
• ​https://www.iamcal.com/understanding-bidirectional-text/

[simon04]

In 16937/josm:

fix #19717 - DefaultNameFormatter: use FIRST-STRONG ISOLATE for bidirectional texts

[gaben]

Something is not right with today's build. r16932 is fine, but showing artefacts with r16942.

(screenshot from Windows)

[gaben]

Just checked, reverting the patch solves the issue.

[stoecker]

Ticket #19719 has been marked as a duplicate of this ticket.

[simon04]

@gaben, @skyper: a full status report (showing the Java version and operating system) would be highly appreciated! (That's why we ask for it via newticket…)

[Klumbumbus]

Similar problem for me.

URL:https://josm.openstreetmap.de/svn/trunk
Repository:UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
Last:Changed Date: 2020-08-26 23:18:51 +0200 (Wed, 26 Aug 2020)
Build-Date:2020-08-27 01:30:46
Revision:16942
Relative:URL: ^/trunk

Identification: JOSM/1.5 (16942 de) Windows 10 64-Bit
OS Build number: Windows 10 Pro 1909 (18363)
Memory Usage: 829 MB / 1820 MB (309 MB allocated, but free)
Java version: 1.8.0_261-b12, Oracle Corporation, Java HotSpot(TM) 64-Bit Server VM
Look and Feel: com.sun.java.swing.plaf.windows.WindowsLookAndFeel
Screen: \Display0 1680x1050 (scaling 1.0x1.0)
Maximum Screen Size: 1680x1050
Best cursor sizes: 16x16 -> 32x32, 32x32 -> 32x32
VM arguments: [-Djava.security.manager, -Djava.security.policy=file:<java.home>\lib\security\javaws.policy, -DtrustProxy=true, -Djnlpx.home=<java.home>\bin, -Djnlpx.origFilenameArg=C:\Program Files (x86)\josm-latest.jnlp, -Djnlpx.remove=false, -Djava.util.Arrays.useLegacyMergeSort=true, -Djnlpx.heapsize=NULL,2048m, -Djnlpx.splashport=53845, -Djnlpx.jvm=<java.home>\bin\javaw.exe]
Dataset consistency test: No problems found

Plugins:
+ DirectUpload (35499)
+ HouseNumberTaggingTool (35508)
+ OpeningHoursEditor (35414)
+ PicLayer (35405)
+ RoadSigns (35510)
+ SimplifyArea (35405)
+ apache-commons (35524)
+ buildings_tools (35500)
+ editgpx (35248)
+ imagery-xml-bounds (35313)
+ imagery_offset_db (35405)
+ javafx-windows (35458)
+ jaxb (35092)
+ jna (35092)
+ jogl (1.2.3)
+ measurement (35405)
+ photo_geotagging (35499)
+ photoadjust (35405)
+ reltoolbox (35529)
+ reverter (35499)
+ rex (53)
+ tageditor (35258)
+ tagging-preset-tester (35267)
+ terracer (35499)
+ turnlanes-tagging (283)
+ turnrestrictions (35515)
+ undelete (35521)
+ utilsplugin2 (35487)

Last errors/warnings:
- 00422,642 W: Unable to use English input method

[gaben]

Reproducible on both Windows 10 19041 with Java 8 and Ubuntu 20.04.1 with Java 11. Plugins not affecting this, because on my dev VM I don't have any modification, not even settings.

Windows

URL:https://josm.openstreetmap.de/svn/trunk
Repository:UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
Last:Changed Date: 2020-08-26 23:18:51 +0200 (Wed, 26 Aug 2020)
Build-Date:2020-08-27 01:30:46
Revision:16942
Relative:URL: ^/trunk

Identification: JOSM/1.5 (16942 hu) Windows 10 64-Bit
OS Build number: Windows 10 Pro for Workstations 2004 (19041)
Memory Usage: 990 MB / 1820 MB (468 MB allocated, but free)
Java version: 1.8.0_261-b12, Oracle Corporation, Java HotSpot(TM) 64-Bit Server VM
Look and Feel: com.sun.java.swing.plaf.windows.WindowsLookAndFeel
Screen: \Display0 1920x1200 (scaling 1.0x1.0)
Maximum Screen Size: 1920x1200
Best cursor sizes: 16x16 -> 32x32, 32x32 -> 32x32
VM arguments: [-Djava.security.manager, -Djava.security.policy=file:<java.home>\lib\security\javaws.policy, -DtrustProxy=true, -Djnlpx.home=<java.home>\bin, -Djnlpx.origFilenameArg=%UserProfile%\AppData\LocalLow\Sun\Java\Deployment\cache\6.0\31\583aa85f-29eff323, -Djnlpx.remove=false, -Djava.util.Arrays.useLegacyMergeSort=true, -Djnlpx.heapsize=NULL,2048m, -Djnlpx.splashport=63730, -Djnlp.application.href=https://josm.openstreetmap.de/download/josm-latest.jnlp, -Djnlpx.jvm=<java.home>\bin\javaw.exe]

Linux

Revision:16943
Is-Local-Build:true
Build-Date:2020-08-27 20:26:52

Identification: JOSM/1.5 (16943 SVN hu) Linux Ubuntu 20.04.1 LTS
Memory Usage: 274 MB / 2990 MB (157 MB allocated, but free)
Java version: 11.0.8+10-post-Ubuntu-0ubuntu120.04, Ubuntu, OpenJDK 64-Bit Server VM
Look and Feel: javax.swing.plaf.metal.MetalLookAndFeel
Screen: :0.0 1918x1020 (scaling 1.0x1.0)
Maximum Screen Size: 1918x1020
Best cursor sizes: 16x16 -> 16x16, 32x32 -> 32x32
Desktop environment: ubuntu:GNOME
Java package: openjdk-11-jre:amd64-11.0.8+10-0ubuntu1~20.04
Java ATK Wrapper package: libatk-wrapper-java:all-0.37.1-1
fonts-noto: fonts-noto:-
VM arguments: [-Dfile.encoding=UTF-8]
Dataset consistency test: No problems found

[simon04]

Thank you.

My tests:

r16812 suffers from https://josm.openstreetmap.de/attachment/ticket/19703/relation%20editor.PNG (Windows and Linux)

Relative:URL: ^/trunk
Repository:UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
Last:Changed Date: 2020-07-30 12:44:04 +0200 (Thu, 30 Jul 2020)
Revision:16812
Build-Date:2020-07-31 01:30:49
URL:https://josm.openstreetmap.de/svn/trunk

Identification: JOSM/1.5 (16812 en_GB) Windows 10 64-Bit
OS Build number: Windows 10 Pro 2004 (19041)
Memory Usage: 308 MB / 2048 MB (106 MB allocated, but free)
Java version: 14.0.2+12, AdoptOpenJDK, OpenJDK 64-Bit Server VM
Look and Feel: com.sun.java.swing.plaf.windows.WindowsLookAndFeel
Screen: \Display0 1920x1200 (scaling 1.0x1.0)
Maximum Screen Size: 1920x1200
Best cursor sizes: 16x16 -> 32x32, 32x32 -> 32x32
Dataset consistency test: No problems found

Relative:URL: ^/trunk
Repository:UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
Last:Changed Date: 2020-07-30 12:44:04 +0200 (Thu, 30 Jul 2020)
Revision:16812
Build-Date:2020-07-31 01:30:49
URL:https://josm.openstreetmap.de/svn/trunk

Identification: JOSM/1.5 (16812 en) Linux Arch Linux
Memory Usage: 634 MB / 7870 MB (261 MB allocated, but free)
Java version: 11.0.8+10, N/A, OpenJDK 64-Bit Server VM
Look and Feel: javax.swing.plaf.metal.MetalLookAndFeel
Screen: :0.0 3840x2160 (scaling 1.0x1.0)
Maximum Screen Size: 3840x2160
Best cursor sizes: 16x16 -> 16x16, 32x32 -> 32x32
VM arguments: [-Djosm.restart=true]
Dataset consistency test: No problems found

r16942 suffers from attachment:print_error.png​​ (Windows only?)

Relative:URL: ^/trunk
Repository:UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
Last:Changed Date: 2020-08-26 23:18:51 +0200 (Wed, 26 Aug 2020)
Revision:16942
Build-Date:2020-08-27 01:30:46
URL:https://josm.openstreetmap.de/svn/trunk

Identification: JOSM/1.5 (16942 en_GB) Windows 10 64-Bit
OS Build number: Windows 10 Pro 2004 (19041)
Memory Usage: 180 MB / 2046 MB (84 MB allocated, but free)
Java version: 14.0.2+12, AdoptOpenJDK, OpenJDK 64-Bit Server VM
Look and Feel: com.sun.java.swing.plaf.windows.WindowsLookAndFeel
Screen: \Display0 1920x1200 (scaling 1.0x1.0)
Maximum Screen Size: 1920x1200
Best cursor sizes: 16x16 -> 32x32, 32x32 -> 32x32
Dataset consistency test: No problems found

The used bidirectional isolate characters are recommended according to ​https://en.wikipedia.org/wiki/Bidirectional_text#Isolates

Let's see whether we can resolve this issue. As a fallback, we can easily revert the change for the 20.08 release…

[simon04]

In 16947/josm:

see #19717 - Fix unit tests

[stoecker]

The last checkin is ugly:

• Your commit hides a change to the source with a comment related only to test stuff
• You extract logic from the formater elsewhere. When this is needed then please do the replace inside the Formatter and e.g. call a fix function like DefaultNameFormatter.fixBIDI(text)

[simon04]

We've been replacing U+200E at various places before, too, see r16937. Sure, we can clean the code, but first we have to resolve the artefacts (on Windows). Any suggestion to tackle the displayed artefacts?

[stoecker]

Replying to simon04:

We've been replacing U+200E at various places before, too, see r16937.

That we did it before doesn't make it better :-)

but first we have to resolve the artefacts (on Windows). Any suggestion to tackle the displayed artefacts?

Not really. Anyway I'm happy that my Linux version looks good again. I've seen the issue with the small lines first time on Tuesday and now it is gone. In worst case we've to make a switch between the two methods via the Platform hooks.

[simon04]

In 16950/josm:

Revert "fix #19717 - DefaultNameFormatter: use FIRST-STRONG ISOLATE for bidirectional texts"

This reverts commit r16937

[simon04]

In 16951/josm:

fix #19717 - DefaultNameFormatter: add missing U+202C POP DIRECTIONAL FORMATTING

[simon04]

In 16952/josm:

see #19717 - Extract DefaultNameFormatter.removeBiDiCharacters