Modify

Opened 10 years ago

Closed 10 years ago

#9960 closed defect (irreproducible)

JOSM not seeing differences with special characters like LRM

Reported by: glenn@… Owned by: glenn@…
Priority: normal Milestone:
Component: Core Version:
Keywords: template_report unicode lrm character Cc:

Description

Certain tags seem to contain special chars like LRM in the content.

What steps will reproduce the problem?

  1. Use this overpass query to find problematic entries: http://overpass-turbo.eu/s/3c4
  2. Fix the website tag: notice that you need to hit backspace twice to remove the last character in the URLs for each of the entries

Taglink has more of those entries: http://taginfo.openstreetmap.org/search?q=website%3D%E2%80%8E

  1. Try uploading the results, JOSM will not do it, it's not seeing the differences. I tried: adding dummy note tag, but josm does not include those keys in the diff. Tried manual fixing in osm file, open in new layer, merge them (works). But josm also refuses to upload that result.

Take a look at attached osm files with a decent editor that sees them like vim. There is a extraneous LRM char after the url. This was noticed on the OSM-Talk list. The person that spotted this in the data (John Packer) also states: "It seems this also affect other keys (like phone), but the problem is not as bad as in the keys 'website' and 'contact:website'."

What is the expected result?

Josm should account for differences outside the visible character set, special characters should be accounted for.

A nice to have is to prevent those type of special characters from being introduced, they have no merit.

What happens instead?

Refuses to upload

Please provide any additional information below. Attach a screenshot if
possible.

Revision: 6942
Repository Root: http://josm.openstreetmap.de/svn
Last Changed Author: Don-vip
Last Changed Date: 2014-03-29 00:45:51 +0100 (Sat, 29 Mar 2014)
Build-Date: 2014-03-29 02:35:49
URL: http://josm.openstreetmap.de/svn/trunk
Repository UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
Last Changed Rev: 6942

Identification: JOSM/1.5 (6942 en) Linux Ubuntu 13.10
Memory Usage: 132 MB / 1749 MB (41 MB allocated, but free)
Java version: 1.8.0, Oracle Corporation, Java HotSpot(TM) 64-Bit Server VM
VM arguments: [-Djosm.restart=true, -Djosm.home=/home/glenn/.josm-latest, -Djava.net.useSystemProxies=true]
Dataset consistency test: No problems found

Plugin: FixAddresses (30277)
Plugin: PicLayer (30277)
Plugin: RoadSigns (30320)
Plugin: buildings_tools (30277)
Plugin: mapdust (30277)
Plugin: mirrored_download (30197)
Plugin: notes (v0.9.2)
Plugin: openstreetbugs (30277)
Plugin: terracer (30277)
Plugin: turnlanes (30277)
Plugin: turnrestrictions (30307)
Plugin: utilsplugin2 (30370)

Attachments (3)

badchar_osm.osm (2.3 KB ) - added by glenn@… 10 years ago.
File with bad chars
fix_osm_merged.osm (2.1 KB ) - added by glenn@… 10 years ago.
merged results set with bad chars removed
Screenshot from 2014-04-29 12:08:11.png (6.9 KB ) - added by glenn@… 10 years ago.
the response

Download all attachments as: .zip

Change History (13)

by glenn@…, 10 years ago

Attachment: badchar_osm.osm added

File with bad chars

by glenn@…, 10 years ago

Attachment: fix_osm_merged.osm added

merged results set with bad chars removed

by glenn@…, 10 years ago

the response

in reply to:  description comment:1 by Don-vip, 10 years ago

Replying to glenn@…:

Josm should account for differences outside the visible character set, special characters should be accounted for.

...

Refuses to upload

I cannot reproduce this with both Java 7u55 and 8u5 on Windows. JOSM correctly detects the changes and allows to upload corrected objects. Can you update to 8u5 to check if that part is not a problem specific to Java 1.8.0?

I'll try with Ubuntu to see if it's Linux-specific.

comment:2 by Don-vip, 10 years ago

No problem neither with

Identification: JOSM/1.5 (7018 fr) Linux Ubuntu 14.04 LTS
Java version: 1.7.0_51, Oracle Corporation, OpenJDK Client VM
Java package: openjdk-7-jre:i386-7u51-2.4.6-1ubuntu4

comment:3 by Don-vip, 10 years ago

Owner: changed from team to glenn@…
Status: newneedinfo

comment:4 by glenn@…, 10 years ago

Sorry, took me a while to pick this back up, I tested the following:

Revision: 7050
Repository Root: http://josm.openstreetmap.de/svn
Last Changed Author: Don-vip
Last Changed Date: 2014-05-02 03:01:08 +0200 (Fri, 02 May 2014)
Build-Date: 2014-05-02 01:32:39
URL: http://josm.openstreetmap.de/svn/trunk
Repository UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
Last Changed Rev: 7050

Identification: JOSM/1.5 (7050 en) Linux Ubuntu 13.10
Memory Usage: 127 MB / 1749 MB (63 MB allocated, but free)
Java version: 1.8.0, Oracle Corporation, Java HotSpot(TM) 64-Bit Server VM
VM arguments: [-Djosm.restart=true, -Djosm.home=/home/glenn/.josm-latest, -Djava.net.useSystemProxies=true, -Dawt.useSystemAAFontSettings=lcd]
Dataset consistency test: No problems found

It's not working on this one either. Chances it's java 8 are upped now.

comment:5 by glenn@…, 10 years ago

And this one does't work either:

Repository Root: http://josm.openstreetmap.de/svn
Build-Date: 2014-05-02 01:32:39
Last Changed Author: Don-vip
Revision: 7050
Repository UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
URL: http://josm.openstreetmap.de/svn/trunk
Last Changed Date: 2014-05-02 03:01:08 +0200 (Fri, 02 May 2014)
Last Changed Rev: 7050

Identification: JOSM/1.5 (7050 en) Linux Ubuntu 13.10
Memory Usage: 255 MB / 1749 MB (183 MB allocated, but free)
Java version: 1.7.0_51, Oracle Corporation, Java HotSpot(TM) 64-Bit Server VM
VM arguments: [-Djosm.restart=true, -Djosm.home=/home/glenn/.josm-latest, -Djava.net.useSystemProxies=true, -Dawt.useSystemAAFontSettings=lcd]
Dataset consistency test: No problems found

comment:6 by glenn@…, 10 years ago

Now this is an interesting one, notice how it reports both oracle java but the package is openjdk

Repository Root: http://josm.openstreetmap.de/svn
Build-Date: 2014-05-02 01:32:39
Last Changed Author: Don-vip
Revision: 7050
Repository UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
URL: http://josm.openstreetmap.de/svn/trunk
Last Changed Date: 2014-05-02 03:01:08 +0200 (Fri, 02 May 2014)
Last Changed Rev: 7050

Identification: JOSM/1.5 (7050 en) Linux Ubuntu 13.10
Memory Usage: 261 MB / 1749 MB (173 MB allocated, but free)
Java version: 1.7.0_51, Oracle Corporation, OpenJDK 64-Bit Server VM
Java package: openjdk-7-jre:amd64-7u51-2.4.4-0ubuntu0.13.10.1
VM arguments: [-Djosm.restart=true, -Djosm.home=/home/glenn/.josm-latest, -Djava.net.useSystemProxies=true, -Dawt.useSystemAAFontSettings=lcd]
Dataset consistency test: No problems found

Command line was:

glenn@slicky:~$ /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Djosm.restart=true 
-Djosm.home=/home/glenn/.josm-latest -Djava.net.useSystemProxies=true 
-jar /usr/share/josm-latest/josm-latest-test.jar
Picked up _JAVA_OPTIONS: -Dawt.useSystemAAFontSettings=lcd

Keep getting this so it seems

comment:7 by Don-vip, 10 years ago

Is there a change of behaviour if you unset _JAVA_OPTIONS ? I don't know the impact of this awt property.

comment:8 by glenn@…, 10 years ago

IF I unset this the drop down menu buttons work but I can't keep the selections open, so I cannot choose a submenu. It's a strange problem, this is been like that for a while now. And the fonts get really ugly too.

I don't think it has anything to do with this, for some reason java oracle number 8 works best with all the stuff I have running that needs it (solr / tradestation / jetty ). They all seem to work fine so far.

I'll give open-jdk-7 a try in a fresh VM soon.

comment:9 by Don-vip, 10 years ago

Keywords: unicode lrm character added

comment:10 by Don-vip, 10 years ago

Resolution: irreproducible
Status: needinfoclosed

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain glenn@….
as The resolution will be set.
The resolution will be deleted. Next status will be 'reopened'.

Add Comment


E-mail address and name can be saved in the Preferences .
 
Note: See TracTickets for help on using tickets.