Opened 3 years ago

Closed 3 years ago

Last modified 18 months ago

#16102 closed enhancement (fixed)

kml import is very slow compared to gpx import

Reported by: GerdP Owned by: Don-vip
Priority: normal Milestone:
Component: Plugin opendata Version:
Keywords: performance kml Cc:


I've got a kml file and a gpx file, both describe the same track.
I've noted that it takes much longer to load the kml file into JOSM (many seconds).
I think the reason is that uses
for each single node instead of adding all nodes to a List and calling
once the data is complete.
Each addNode() triggers the complex

Attached is a patch which might work. I have problems compiling the plugin, so I cannot test the patch.

Attachments (4)

kml.patch (3.0 KB) - added by GerdP 3 years ago.
kml-v2.patch (3.2 KB) - added by GerdP 3 years ago.
GpsiesTrack.kmz (310.7 KB) - added by GerdP 3 years ago.
simplifyWay-performance.patch (1.7 KB) - added by GerdP 3 years ago.

Download all attachments as: .zip

Change History (14)

Changed 3 years ago by GerdP

Attachment: kml.patch added

Changed 3 years ago by GerdP

Attachment: kml-v2.patch added

comment:1 Changed 3 years ago by GerdP

I was finally able to compile the plugin and found some errors. 2nd version works, is much faster compared to the original code,
but still much slower compared to the gpx file import.
My files contain ~41000 points in a single way.
Loading the gpx takes ~ 1 sec, loading the kml with the patch takes 15 secs, without the patch ~60 secs.

I'll have a closer look at the code that imports gpx tomorrow.

comment:2 Changed 3 years ago by GerdP

Was too curious...
The rest of the time is used for SimplifyWayAction. When I comment the lines in
the kml file is also imported in ~ 1 sec.
Why this is this method called? I think it should not be done or at least it should be optional.

comment:3 in reply to:  2 Changed 3 years ago by Don-vip

Replying to GerdP:

Why this is this method called?

This is necessary for some files having way too much points... Can you please attach your file?

Changed 3 years ago by GerdP

Attachment: GpsiesTrack.kmz added

comment:4 Changed 3 years ago by GerdP

Done. I also wonder why SimplifyWayAction needs so much time for this. It doesn't seem to be this specific data set that is so slow,
I see it with all kml / kmz files.

Changed 3 years ago by GerdP

comment:5 Changed 3 years ago by GerdP

I've attached simplifyWay-performance.patch which addresses two problems in SimplifyWayAction:
1) the method isRequiredNode always does the most complex action first (count the occurence of a node)
while the simple check for tags is done last. The patch reverses that order.
This has nearly no effect for nodes without tags but is much faster when the node has tags.
2) The case that SimplifyWay did not find a node to remove can be checked by comparing the number of nodes in the ways.
This is much faster than the addAll() + removeAll() combination that is done now.

With this additional patch the kml import is nearly as fast as gpx, and I see no negative impact when I use
Shift+Y on a large collection of ways.

comment:6 Changed 3 years ago by Don-vip

In 13540/josm:

see #16102 - SimplifyWayAction performance improvements (patch by GerdP)

comment:7 Changed 3 years ago by Don-vip

Resolution: fixed
Status: newclosed

Fixed in [o34089:34090]. Thanks for the patches! :)

comment:8 Changed 18 months ago by GerdP

In 14970/josm:

see #16102: Drastically improve performance for SimplifyWayAction when used with very complex ways.
For a kml file produced by brouter with ~50000 points the import took > 50 seconds, now it is done in 1.5 seconds.

comment:9 Changed 18 months ago by Don-vip


comment:10 Changed 18 months ago by GerdP

I've tried to run the unit test but it doesn't seem to work on my machine :(
Three of four tests fail. A part of the log shows

Testcase: testMoreThanTenWays took 0,406 sec
	Caused an ERROR
ReportedException [thread=Thread[Timeout runner,5,], exception=java.lang.reflect.InvocationTargetException, methodWarningFrom=null]
	at org.openstreetmap.josm.gui.util.GuiHelper.runInEDTAndWaitWithException(
	at org.openstreetmap.josm.gui.layer.LayerManager.removeLayer(
	at org.openstreetmap.josm.actions.SimplifyWayActionTest.testMoreThanTenWays(
	at org.openstreetmap.josm.testutils.JOSMTestRules$
Caused by: java.lang.reflect.InvocationTargetException
	at java.awt.EventQueue.invokeAndWait(
	at java.awt.EventQueue.invokeAndWait(
	at javax.swing.SwingUtilities.invokeAndWait(
	at org.openstreetmap.josm.gui.util.GuiHelper.runInEDTAndWaitWithException(
Caused by: java.lang.IllegalArgumentException: OsmDataLayer [name=, associatedFile=null] is not managed by us.
	at org.openstreetmap.josm.gui.layer.LayerManager.checkContainsLayer(
	at org.openstreetmap.josm.gui.layer.LayerManager.realRemoveLayer(
	at org.openstreetmap.josm.gui.layer.LayerManager.lambda$removeLayer$1(
	at java.awt.event.InvocationEvent.dispatch(
	at java.awt.EventQueue.dispatchEventImpl(
	at java.awt.EventQueue.access$500(
	at java.awt.EventQueue$
	at java.awt.EventQueue$
	at Method)
	at java.awt.EventQueue.dispatchEvent(
	at java.awt.EventDispatchThread.pumpOneEventForFilters(
	at java.awt.EventDispatchThread.pumpEventsForFilter(
	at java.awt.EventDispatchThread.pumpEventsForHierarchy(
	at java.awt.EventDispatchThread.pumpEvents(
	at java.awt.EventDispatchThread.pumpEvents(

Modify Ticket

Change Properties
Set your email in Preferences
as closed The owner will remain Don-vip.
as The resolution will be set.
The resolution will be deleted.

Add Comment

E-mail address and name can be saved in the Preferences.

Note: See TracTickets for help on using tickets.