[Patch]Download object: Too slow when downloading a relation with members with OSM api

[GerdP]

What steps will reproduce the problem?

1. Download a complex relation like ​r9488835 with members to a new layer (not using overpass)
2. Watch how it takes some seconds to actually download all the data and much more time to process it before a the rendered data is shown.
3. Save to *.osm file
4. Load from *.osm file

What is the expected result?

It should take more or less the same time to process the downloaded data as it takes to open the file.

What happens instead?

The downloaded data takes much longer.

Please provide any additional information below. Attach a screenshot if possible.

I used VisualVM to monitor the program and found that a lot of time is spent in DatasetMerger.fixReferences(). It turned out that the data that is retrieved with the 1st call

2021-03-21 08:44:43.115 INFO: GET https://api.openstreetmap.org/api/0.6/relations?relations=9488835 -> HTTP/1.1 200 (322 ms)
2021-03-21 08:44:53.634 INFO: GET https://api.openstreetmap.org/api/0.6/relation/9488835/full -> HTTP/1.1 200 (10.0 s)

is causing the trouble. The result of the call is the incomplete relation. This incomplete data is stored in field MultiFetchServerObjectReader.outputDataSet and later, all the detailed data is merged with this incomplete relation. This forces lots of rather slow QuadBucketPrimitiveStore.reindexRelation()
calls. This merging isn't really needed, the only reason for the initial download is to find out if the relation exists.

URL:https://josm.openstreetmap.de/svn/trunk
Repository:UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
Last:Changed Date: 2021-03-17 18:58:40 +0100 (Wed, 17 Mar 2021)
Build-Date:2021-03-17 17:59:56
Revision:17580
Relative:URL: ^/trunk

Identification: JOSM/1.5 (17580 en) Windows 10 64-Bit
OS Build number: Windows 10 Home 2004 (19041)
Memory Usage: 412 MB / 3641 MB (221 MB allocated, but free)
Java version: 1.8.0_221-b11, Oracle Corporation, Java HotSpot(TM) 64-Bit Server VM
Look and Feel: com.sun.java.swing.plaf.windows.WindowsLookAndFeel
Screen: \Display0 1920×1080 (scaling 1.00×1.00)
Maximum Screen Size: 1920×1080
Best cursor sizes: 16×16→32×32, 32×32→32×32
VM arguments: [-XX:StartFlightRecording=name=MyRecording2,settings=d:\dbg\gerd.jfc, -XX:FlightRecorderOptions=defaultrecording=true,dumponexit=true,dumponexitpath=e:\ld\perf_20210321_082504.jfr]

[GerdP]

screenshot VisualVM

[GerdP]

I assume we see the same delay in other situations. The patch only fixes the reported problem.

[GerdP]

I wonder if it is possible to detect the special case and defer the refreshing of the spatial index QuadBucketsStore while merging two DataSet instances?

[GerdP]

Tried a few alternatives like

• only add relations to the visited hashset in Relation.addToBBox()
• DataSet without spatial index

They did not show comparable improvements, so I'll commit my simple patch...

[GerdP]

In 17635/josm:

fix #20633: Download object: Too slow when downloading a relation with members with OSM api

• reduce work for DatasetMerger: avoid to add incomplete relation to downloaded dataset when full relation will be downloaded later. Saves lots of complex reindexRelation calls, esp. with complex relations.