Serious autocomplete performance degradation

[SafwatHalaby]

Intro

I deal with a huge dataset (about 30k bus stops nodes) almost daily.
This was never a problem before, but since recently, JOSM often hangs when I edit the bus stop tags.
Waiting for a few seconds (can be up to ~30) always resolves this.

It happens seems to happen when editing massively used keys. For instance, all stops have a
"ref" key, and when I click "edit" on a single stop's "ref", JOSM
stalls. A similar issue occurs when adding "description" to changeset
uploads. Description is also used by all stops. So it appears this is triggered whenever auto-complete is in use.

Profiling

"I can reproduce this locally on JOSM 13106. According to jvisualvm cpu
profiling (on Oracle JDK), most time is spent in method
org.openstreetmap.josm.data.tagging.ac.AutoCompletionSet.add"
-- mmd

It's a CPU starvation issue and not a memory overuse issue.
(Core at 100% for 10-30 seconds).

Test case

Run the Overpass query below to obtain a huge bus stop dataset, then try editing the ref or name of a stop.
12712 works flawlessly.
13053 hangs for 10-30 seconds.

Query

[out:xml][timeout:90][bbox:29.4013195,33.8818359,33.4131022,36.0791016]
;
(
area(3601473946); // Israel
area(3601803010); // Judea and Samaria district
)->.a;
(
  node["highway"="bus_stop"](area.a);
  way["highway"="bus_stop"](area.a);
)->.b;
(rel(bw.b);rel(bn.b))->.routes;
(.b;way(bn.b);)->.b;
(.b;node(w.b);)->.b;
(.b;.routes;);
out meta;

Status report

URL:http://josm.openstreetmap.de/svn/trunk
Repository:UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
Last:Changed Date: 2017-09-02 14:47:18 +0200 (Sat, 02 Sep 2017)
Build-Date:2017-09-02 20:53:16
Revision:12712
Relative:URL: ^/trunk

Identification: JOSM/1.5 (12712 en) Linux Debian GNU/Linux 9.2 (stretch)
Memory Usage: 468 MB / 811 MB (227 MB allocated, but free)
Java version: 1.8.0_141-8u141-b15-1~deb9u1-b15, Oracle Corporation, OpenJDK 64-Bit Server VM
Screen: :0.0 1366x768
Maximum Screen Size: 1366x768
Java package: openjdk-8-jre:amd64-8u141-b15-1~deb9u1
Java ATK Wrapper package: libatk-wrapper-java:all-0.33.3-13
Dataset consistency test: No problems found

Nothing fancy installed, everything is default. No extra plugins, validator rules, map paint styles, presets, etc.

[Don-vip]

In 13121/josm:

fix #15547 - fix major performance drawback of AutoCompletionSet (regression from r12859)

[Don-vip]

I think it's OK now but I have not tested it much against potential side effects. Can you please tell me if you find anything unusual in autocompletion tomorrow?

[mmd]

I did a very quick check on r13121, performance issue seems to be solved.

[SafwatHalaby]

Thank you for the quick response. I will test this when I have the time.

For the record, the issue was originally posted and discussed here: ​https://lists.openstreetmap.org/pipermail/talk/2017-November/079535.html

There is also a separate discussion which was started up at around the same time, but it also talks about memory errors. I don't know whether or not the issues are related:

​https://forum.openstreetmap.org/viewtopic.php?id=60403

[anonymous]

13124 didn't solve the issue.

URL:http://josm.openstreetmap.de/svn/trunk
Repository:UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
Last:Changed Date: 2017-11-15 20:09:06 +0100 (Wed, 15 Nov 2017)
Build-Date:2017-11-16 02:31:57
Revision:13124
Relative:URL: ^/trunk

Identification: JOSM/1.5 (13124 es) Linux openSUSE Tumbleweed
Memory Usage: 435 MB / 3545 MB (73 MB allocated, but free)
Java version: 1.8.0_144-b01, Oracle Corporation, OpenJDK 64-Bit Server VM
Screen: :0.0 1920x1080
Maximum Screen Size: 1920x1080
Java package: java-1_8_0-openjdk:x86_64-1.8.0.144
Dataset consistency test: No problems found

Plugins:
+ Create_grid_of_ways (32699)
+ ElevationProfile (33815)
+ FixAddresses (33718)
+ HouseNumberTaggingTool (33714)
+ ImproveOsm (122)
+ ImproveWay (12)
+ PicLayer (33822)
+ apache-commons (33668)
+ conflation (0.5.5)
+ contourmerge (1032)
+ dataimport (33581)
+ editgpx (33782)
+ ejml (32680)
+ geotools (33380)
+ graphview (33004)
+ imagery_offset_db (33774)
+ jts (32699)
+ junctionchecking (33778)
+ lakewalker (33712)
+ log4j (32699)
+ measurement (33760)
+ merge-overlap (33436)
+ michigan_left (33004)
+ opendata (33791)
+ pbf (33568)
+ pdfimport (33579)
+ public_transport (33817)
+ reverter (33572)
+ routing (33794)
+ turnlanes (33294)
+ turnrestrictions (33780)
+ undelete (33835)
+ utilsplugin2 (33742)

Tagging presets:
+ http://somewhere.dk/email/josm/source.xml
+ https://josm.openstreetmap.de/josmfile?page=Presets/Quick-highways&zip=1

Map paint styles:
- https://josm.openstreetmap.de/josmfile?page=Styles/Maxspeed&style&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/Noname&style&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/Surface&style&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/Surface-DataEntry&style&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Styles/Lane_and_Road_Attributes&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/BesideTheRoad_Speed&zip=1

Last errors/warnings:
- W: Actualizar complementos - org.openstreetmap.josm.plugins.PluginHandler$UpdatePluginsMessagePanel[,0,0,0x0,invalid,layout=java.awt.GridBagLayout,alignmentX=0.0,alignmentY=0.0,border=,flags=9,maximumSize=,minimumSize=,preferredSize=]
- W: Invalid jar file ''<josm.pref>/plugins/PicLayer.jar.new'' (exists: false, canRead: false)
- W: Invalid jar file ''<josm.pref>/plugins/ElevationProfile.jar.new'' (exists: false, canRead: false)
- W: Invalid jar file ''<josm.pref>/plugins/undelete.jar.new'' (exists: false, canRead: false)
- W: Invalid jar file ''<josm.pref>/plugins/public_transport.jar.new'' (exists: false, canRead: false)
- W: No configuration settings found.  Using hardcoded default values for all pools.

...Deleted...

[Don-vip]

Who's speaking?

[SafwatHalaby]

I don't know. It was not me. I have not tested this yet.

[Don-vip]

OK. Sometimes we have random anonymous comments on totally different subjects. Unless we get concrete information, I will only consider your feedback ;) When will you be able to test it?

[SafwatHalaby]

Tested with 13150.
I could not reproduce the original "Test case" described in my first post. There were no noticeable hangs.