CrossingWays causes OutOfMemory (was: josm runs out of memory after clicking "upload", upload is impossible)

[Cobra]

I can't reproduce this reliably yet, but it happened several times now.

Immediately after clicking the upload button, josm consumes as much cpu and ram as possible until a "out of memory" message pops up. The upload dialog doesn't open but josm stays usable. I have to save the osm layer and restart josm to upload it.

I'll try to investigate this further but this may take some time.

Repository Root: http://josm.openstreetmap.de/svn
Build-Date: 2011-07-04 01:31:38
Last Changed Author: stoecker
Revision: 4201
Repository UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
URL: http://josm.openstreetmap.de/svn/trunk
Last Changed Date: 2011-07-04 00:07:05 +0200 (Mon, 04 Jul 2011)
Last Changed Rev: 4201

Identification: JOSM/1.5 (4201 en)
Memory Usage: 647 MB / 910 MB (564 MB allocated, but free)
Java version: 1.6.0_24, Sun Microsystems Inc., Java HotSpot(TM) 64-Bit Server VM
Operating system: Linux
Dataset consistency test: No problems found

Plugin: ImproveWayAccuracy (26174)
Plugin: PicLayer (26174)
Plugin: alignways (26174)
Plugin: contourmerge (25626)
Plugin: measurement (26174)
Plugin: openstreetbugs (26174)
Plugin: openvisible (26174)
Plugin: proj4j (26174)
Plugin: reltoolbox (26174)
Plugin: reverter (26174)
Plugin: terracer (26174)
Plugin: undelete (26242)
Plugin: utilsplugin2 (26174)

[jttt]

Please run java with -XX:+HeapDumpOnOutOfMemoryError parameter. Next time OutOfMemory happens, java will generate heap dump that we can use to analyze what happened. You should use OAuth instead of plain text password, otherwise it's possible to get your password from the dump.

[Cobra]

I finally managed to reproduce it again.

Now I've got a 1.3GB large heap dump file. I also have the file I loaded before trying to upload.

Context: I was using josm for some time, saved my changes to a file, deleted all layers and loaded an older file. When I tried to upload changes in this file, josm went unresponsive with 100% CPU load, used lots of RAM and finally displayed a dialog labeled "error" but blank like josm's main window.

[jttt]

Can you upload the file somewhere? It shouldn't be too big zipped.

[Cobra]

​http://dl.dropbox.com/u/31535737/josm_6536.hprof.7z

[jttt]

OutOfMemory is caused by ineffective algorithm used for CrossingWays validator test. I'll replace it either with some proper line segment intersection algorithm like Bentley Ottman or the naive algorithm speeded up by QuadBuckets.

[simon04]

@jttt: Did you start with the implementation? Just asking to avoid potential duplicate work on the same issue.

[jttt]

Nope, sorry, I've completely forgot about this. If you want, please go ahead and fix it. As far as I remember, problem was that it parcels checked area into cells with predefined size, so if area is big then it takes huge amount of memory. Also used structures were unnecessarily memory hungry, like caching thinks that can be easily calculated on demand.

If you want to use Bentley Ottmann, ​here is sample library. But it probably shouldn't be used as is because it's really academical (for example it's using arbitrary precision numbers which will probably really slow it down and also eat lots of memory)

[simon04]

In 6580/josm:

see #6536 - Refactor crossing way test in order to decrease memory footprint

Drop data structure ExtendedSegment and obtain needed values on demand
from the way tags.

[GerdP]

I think this ticket should be closed, the memory footprint for the test is (now) rather small.