I can reproduce the problem. Adding data.clear() to OsmDataLayer.java#destroy helps: Java then reports that more of the heap is free ("allocated, but free" in JOSM’s status report).
Analyzing the heap dumps using Eclipse’s MAT gives about the same results: If data.clear is called the heap is around 60 MB with top consumers being ImageCaches in swing.….GTKEngine and bufferedImages in MapView (so MapView seems to leak as well; at least I believe it is destroyed when there are no more layers.).
Without data clear the heap uses 320 MB of which 280 belong to DataSet. The follow-ups are again GTKEngine and bufImgs in MapView.
Unfortunately, Java rarely shrinks the heap once allocated. So even if you system is running OOM, Java will not free the "free heap". Apparently there are some JVM options (see (1)), but they did not work for me (somewhere on stackoverflow it was mentioned that the Min/MaxFreeHeapRatio won’t work with the parallel(?) GC and that the JVM auto-detects this because of multiple CPUs and enough RAM. This applies to my system, thus my heap will probably never shrink).
In any case I’ll attach the patch since having a free heap is still better than nothing. If I find no more issues, I’ll commit it.
(1) https://stopcoding.wordpress.com/2010/04/12/more-on-the-incredible-shrinking-jvm-heap/
