JOSM not seeing differences with special characters like LRM

[glenn@…]

Certain tags seem to contain special chars like ​LRM in the content.

What steps will reproduce the problem?

1. Use this overpass query to find problematic entries: ​http://overpass-turbo.eu/s/3c4
2. Fix the website tag: notice that you need to hit backspace twice to remove the last character in the URLs for each of the entries

Taglink has more of those entries: ​http://taginfo.openstreetmap.org/search?q=website%3D%E2%80%8E

3. Try uploading the results, JOSM will not do it, it's not seeing the differences. I tried: adding dummy note tag, but josm does not include those keys in the diff. Tried manual fixing in osm file, open in new layer, merge them (works). But josm also refuses to upload that result.

Take a look at attached osm files with a decent editor that sees them like vim. There is a extraneous LRM char after the url. This was noticed on the OSM-Talk list. The person that spotted this in the data (John Packer) also states: "It seems this also affect other keys (like phone), but the problem is not as bad as in the keys 'website' and 'contact:website'."

What is the expected result?

Josm should account for differences outside the visible character set, special characters should be accounted for.

A nice to have is to prevent those type of special characters from being introduced, they have no merit.

What happens instead?

Refuses to upload

Please provide any additional information below. Attach a screenshot if
possible.

Revision: 6942
Repository Root: http://josm.openstreetmap.de/svn
Last Changed Author: Don-vip
Last Changed Date: 2014-03-29 00:45:51 +0100 (Sat, 29 Mar 2014)
Build-Date: 2014-03-29 02:35:49
URL: http://josm.openstreetmap.de/svn/trunk
Repository UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
Last Changed Rev: 6942

Identification: JOSM/1.5 (6942 en) Linux Ubuntu 13.10
Memory Usage: 132 MB / 1749 MB (41 MB allocated, but free)
Java version: 1.8.0, Oracle Corporation, Java HotSpot(TM) 64-Bit Server VM
VM arguments: [-Djosm.restart=true, -Djosm.home=/home/glenn/.josm-latest, -Djava.net.useSystemProxies=true]
Dataset consistency test: No problems found

Plugin: FixAddresses (30277)
Plugin: PicLayer (30277)
Plugin: RoadSigns (30320)
Plugin: buildings_tools (30277)
Plugin: mapdust (30277)
Plugin: mirrored_download (30197)
Plugin: notes (v0.9.2)
Plugin: openstreetbugs (30277)
Plugin: terracer (30277)
Plugin: turnlanes (30277)
Plugin: turnrestrictions (30307)
Plugin: utilsplugin2 (30370)

[glenn@…]

File with bad chars

[glenn@…]

merged results set with bad chars removed

[glenn@…]

the response

[Don-vip]

Replying to glenn@…:

Josm should account for differences outside the visible character set, special characters should be accounted for.

...

Refuses to upload

I cannot reproduce this with both Java 7u55 and 8u5 on Windows. JOSM correctly detects the changes and allows to upload corrected objects. Can you update to 8u5 to check if that part is not a problem specific to Java 1.8.0?

I'll try with Ubuntu to see if it's Linux-specific.

[Don-vip]

No problem neither with

Identification: JOSM/1.5 (7018 fr) Linux Ubuntu 14.04 LTS
Java version: 1.7.0_51, Oracle Corporation, OpenJDK Client VM
Java package: openjdk-7-jre:i386-7u51-2.4.6-1ubuntu4

[glenn@…]

Sorry, took me a while to pick this back up, I tested the following:

Revision: 7050
Repository Root: http://josm.openstreetmap.de/svn
Last Changed Author: Don-vip
Last Changed Date: 2014-05-02 03:01:08 +0200 (Fri, 02 May 2014)
Build-Date: 2014-05-02 01:32:39
URL: http://josm.openstreetmap.de/svn/trunk
Repository UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
Last Changed Rev: 7050

Identification: JOSM/1.5 (7050 en) Linux Ubuntu 13.10
Memory Usage: 127 MB / 1749 MB (63 MB allocated, but free)
Java version: 1.8.0, Oracle Corporation, Java HotSpot(TM) 64-Bit Server VM
VM arguments: [-Djosm.restart=true, -Djosm.home=/home/glenn/.josm-latest, -Djava.net.useSystemProxies=true, -Dawt.useSystemAAFontSettings=lcd]
Dataset consistency test: No problems found

It's not working on this one either. Chances it's java 8 are upped now.

[glenn@…]

And this one does't work either:

Repository Root: http://josm.openstreetmap.de/svn
Build-Date: 2014-05-02 01:32:39
Last Changed Author: Don-vip
Revision: 7050
Repository UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
URL: http://josm.openstreetmap.de/svn/trunk
Last Changed Date: 2014-05-02 03:01:08 +0200 (Fri, 02 May 2014)
Last Changed Rev: 7050

Identification: JOSM/1.5 (7050 en) Linux Ubuntu 13.10
Memory Usage: 255 MB / 1749 MB (183 MB allocated, but free)
Java version: 1.7.0_51, Oracle Corporation, Java HotSpot(TM) 64-Bit Server VM
VM arguments: [-Djosm.restart=true, -Djosm.home=/home/glenn/.josm-latest, -Djava.net.useSystemProxies=true, -Dawt.useSystemAAFontSettings=lcd]
Dataset consistency test: No problems found

[glenn@…]

Now this is an interesting one, notice how it reports both oracle java but the package is openjdk

Repository Root: http://josm.openstreetmap.de/svn
Build-Date: 2014-05-02 01:32:39
Last Changed Author: Don-vip
Revision: 7050
Repository UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
URL: http://josm.openstreetmap.de/svn/trunk
Last Changed Date: 2014-05-02 03:01:08 +0200 (Fri, 02 May 2014)
Last Changed Rev: 7050

Identification: JOSM/1.5 (7050 en) Linux Ubuntu 13.10
Memory Usage: 261 MB / 1749 MB (173 MB allocated, but free)
Java version: 1.7.0_51, Oracle Corporation, OpenJDK 64-Bit Server VM
Java package: openjdk-7-jre:amd64-7u51-2.4.4-0ubuntu0.13.10.1
VM arguments: [-Djosm.restart=true, -Djosm.home=/home/glenn/.josm-latest, -Djava.net.useSystemProxies=true, -Dawt.useSystemAAFontSettings=lcd]
Dataset consistency test: No problems found

Command line was:

glenn@slicky:~$ /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Djosm.restart=true 
-Djosm.home=/home/glenn/.josm-latest -Djava.net.useSystemProxies=true 
-jar /usr/share/josm-latest/josm-latest-test.jar
Picked up _JAVA_OPTIONS: -Dawt.useSystemAAFontSettings=lcd

Keep getting this so it seems

[Don-vip]

Is there a change of behaviour if you unset _JAVA_OPTIONS ? I don't know the impact of this awt property.

[glenn@…]

IF I unset this the drop down menu buttons work but I can't keep the selections open, so I cannot choose a submenu. It's a strange problem, this is been like that for a while now. And the fonts get really ugly too.

I don't think it has anything to do with this, for some reason java oracle number 8 works best with all the stuff I have running that needs it (solr / tradestation / jetty ). They all seem to work fine so far.

I'll give open-jdk-7 a try in a fresh VM soon.