Large GPX files combined with Bing Imagery causes memory exhaustion and crash

[mcpicoli]

What steps will reproduce the problem?

1. Open JOSM, drag one or more LARGE (> 20MB each) GPX files into it, or open them using the "File" menu. Wait for it to finish loading.
2. Add Bing Imagery to a new layer, using the "Imagery" menu.
3. Zoom in, and start dragging around, WITHOUT adding data layers, or anything else. Just keep dragging around, preferably when zoomed in and showing parts of the GPX tracks.

What is the expected result?

It is expected that JOSM will keep working normally - background consisting of Bing Imagery, overlayed by parts of the tracks defined in one or more of the GPX files loaded.
It is expected that JOSM's performance will be mostly constant along time, if no other operations besides zooming in/out and dragging the screen around are done.

What happens instead?

Almost immediately, performance (measured by "speed" of dragging the map) degrades VERY sharply, and gets much worse (to the point of being unusable) after only a few minutes. If one insists some more, JOSM will crash with an "out of memory" error, and not respond anymore to user input (most of the times).

Please provide any additional information below. Attach a screenshot if possible.

• All plugins were disabled
• Reverting to an VERY old and obsolete version of JOSM (5608) - with the ancient imagery cache directory - solves completely the problem
• Reverting to a recent version like 8159 that uses the newer (but not newest, DB based) cache mechanism - does NOT solve the problem.
• Installing a plugin like "ImageryCache" does NOT solve the problem.
• Increasing the JVM's memory allocation postpones the problem, but performance gets really unusable, very quickly.
• Also tried: lowering the maximum number of concurrent downloads to 1, lowering "jcs.cache.max_objects_in_memory" from 10k to 100. None changed anything related to the problem.
• My GPX files consist of many (a few hundreds in each file) tracks.
• While it is expected that with large GPX files the performance won't be as good as with small files, it is expected that it wouldn't degrade with time by simply dragging the map around.
• Probably not related, but may be useful info: Intel i7-3770 CPU, 32GB RAM, 2x Apple Cinema Displays (via TB).

Repository Root: http://josm.openstreetmap.de/svn
Build-Date: 2015-05-08 01:31:23
Last Changed Author: stoecker
Revision: 8339
Repository UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
Relative URL: ^/trunk
URL: http://josm.openstreetmap.de/svn/trunk
Last Changed Date: 2015-05-07 17:30:43 +0200 (Thu, 07 May 2015)
Last Changed Rev: 8339

Identification: JOSM/1.5 (8339 en) Windows 7 64-Bit
Memory Usage: 166 MB / 455 MB (62 MB allocated, but free)
Java version: 1.7.0_21, Oracle Corporation, Java HotSpot(TM) 64-Bit Server VM

[wiktorn]

I've tested on 26MB gpx file, which contains around 20 tracks, but many points in each track. After FullGC I see memory usage around 125MB, without Bing imagery loaded.

With Bing Imagery loaded, I see memory use around 170MB.

I do not notice any performance problems when browsing.

Tweaking jcs.cache.max_objects_in_memory doesn't affect the number of objects cached in memory for Bing layer, as this limit is fixed at 200. I plan to make it autosize based on screen resolution.

What I recommend, is that you first update your Java version - you've got 1.7.0_21 which is quite old. Then, if the problem continues, it would help to debug the problem if you could share the gpx file.

In mean time, you can also try to use jvisualvm, which is installed in your Java installation folder (if you install JDK) - and see, how much memory is used when you just load gpx file.

[Don-vip]

can you please try with josm latest + java 8u45? If it's still this bad, sharing these big gpx files will help us to investigate.

[mcpicoli]

Sorry for taking so long.

• First, upgraded to the latest JVM (1.8.0_45) according to JOSM's About dialog. Uninstalled the old version using the tool provided by Oracle. What happened was somewhat strange. Before, when using 1.7.0_21, when dragging the map (with imagery), CPU usage was steadily increasing on all 4 "real" cores and all of the 4 "virtual" ones (hyperthreads), of my i7 3770. Just before crashing with an out of memoy error, CPU usage was 100% across all eight for many seconds on end. After the upgrade, however, CPU usage was still high, but never on more than a single "real" core, when using the same JOSM version (8339) as before. That meant it took a little longer to crash, because a single core took a little longer to run to the point of memory exhaustion.

• Second, upgraded to the latest (at the time) unstable version of JOSM (8456). It seems to be somewhat faster than before (sorry for not taking objective notes, measuring times, etc.), but degradation still occurs, and the crash is simply an "exception", and not an out-of-memory error like before. Still using only a single core of the CPU. It is generally possible to continue using JOSM after this exception.

• Discovered that if I load all of the problem GPX files (7) from the largest first to the smallest last, performance seems to be better, and if I "unload" (deleting layers) all of them, performance does not return to the previous levels.

About sharing the problem files, I believe that it is a matter of privacy, since they contain sensitive information (at least for me) - the place I live, work, common routes, etc. Is there any way to anonymize them? Sugestions? If they are really key to the problem, we'll find a way.

[wiktorn]

Replying to marcelo@…:

• Second, upgraded to the latest (at the time) unstable version of JOSM (8456). It seems to be somewhat faster than before (sorry for not taking objective notes, measuring times, etc.), but degradation still occurs, and the crash is simply an "exception", and not an out-of-memory error like before. Still using only a single core of the CPU. It is generally possible to continue using JOSM after this exception.

Can you describe more about what kind of the "exception" you're receving? Does "report bug" dialog shows? Can you provide information, what kind of exception it is?

About sharing the problem files, I believe that it is a matter of privacy, since they contain sensitive information (at least for me) - the place I live, work, common routes, etc. Is there any way to anonymize them? Sugestions? If they are really key to the problem, we'll find a way.

It would be great if we would have a way to reproduce the problem. Maybe you can try to combine some publicly available GPX-es (such as available here: ​http://www.openstreetmap.org/traces/ ) and create gpx file, for which you observe the same problem.

[mcpicoli]

This is the text of the Exception given. I'll try to compose a GPX file capable of triggering the problems from public sources later today:

Revision: 8456
Repository Root: http://josm.openstreetmap.de/svn
Relative URL: ^/trunk
Last Changed Author: Don-vip
Last Changed Date: 2015-06-03 04:36:57 +0200 (Wed, 03 Jun 2015)
Build-Date: 2015-06-03 02:39:48
URL: http://josm.openstreetmap.de/svn/trunk
Repository UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
Last Changed Rev: 8456

Identification: JOSM/1.5 (8456 en) Windows 7 64-Bit
Memory Usage: 494 MB / 494 MB (144 MB allocated, but free)
Java version: 1.8.0_45, Oracle Corporation, Java HotSpot(TM) Client VM


Last errors/warnings:
- W: JCS TMS Cache - error creating URL for tile 17/48560/74388@Bing aerial imagery: Attribution is not loaded yet
- W: No url returned for: 17/48560/74388@Bing aerial imagery, skipping
- E: java.lang.NullPointerException

java.lang.NullPointerException
	at org.openstreetmap.josm.data.cache.JCSCachedTileLoaderJob.loadObject(JCSCachedTileLoaderJob.java:385)
	at org.openstreetmap.josm.data.cache.JCSCachedTileLoaderJob.run(JCSCachedTileLoaderJob.java:248)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

[Don-vip]

Replying to marcelo@…:

This is the text of the Exception given.

Please update to latest, this has been fixed in r8462.

[mcpicoli]

After using version 8491 for more than one week and now 8561, I am not able to reproduce the problem anymore.

While the performance still degrades sharply after a few minutes when using the large GPX files combined with Bing's aerial imagery - specially when compared to the performance of very old versions like 5608 - never again the memory was exhausted, or any exception thrown.

[wiktorn]

If memory exception doesn't occur - I'm closing it for now. Did you tried raising the memory limit for JOSM and did it affect the performance?