#4057 closed defect (fixed)
JOSM loses data on save: Approx 20% data loss of ways but none of nodes/relations
| Reported by: | Owned by: | team | |
|---|---|---|---|
| Priority: | critical | Milestone: | |
| Component: | Core | Version: | latest |
| Keywords: | Cc: |
Description
After reverting the failed lakewalker trace described in this comment to bug #3784 I tried to save the .osm file I was working on. The resulting saved file has around 20% of its ways missing. I didn't delete these because there are no modified objects in the dataset nor are there any pending deletions.
This indicates the approximate dataloss, note that this isn't entirely accurate since Iceland.osm doesn't take into account changes in this changeset, but that's not more than <100 ways and a few thousand nodes:
avar@aoeu:~/src/josm$ grep -c "<way" Iceland.osm Iceland-fail.osm Iceland.osm:17159 Iceland-fail.osm:13899 avar@aoeu:~/src/josm$ grep -c "<relation" Iceland.osm Iceland-fail.osm Iceland.osm:374 Iceland-fail.osm:374 avar@aoeu:~/src/josm$ grep -c "<node" Iceland.osm Iceland-fail.osm Iceland.osm:327739 Iceland-fail.osm:328779
When I search for type:way in the JOSM instance I saved from I get 13899 even though by looking at mappaint it's actually drawing closer to 17159 ways. I.e. I can select some ways in JOSM but I can't find them using the search engine, nor are those same ways saved if I save to .osm.
I'm keeping the JOSM instance which failed like this open for a while in case there's anything more I can do to debug it while it's running. I'm also attaching .osm files & screenshots.
Here's a status report:
Build-Date: 2009-11-30 13:15:54 Revision: 2554 Is-Local-Build: true Memory Usage: 729 MB / 986 MB (238 MB allocated, but free) Java version: 1.6.0_16 Dataset consistency test: No problems found Plugins: lakewalker,multipoly,openvisible,remotecontrol,utilsplugin,validator,walkingpapers,wmsplugin Plugin lakewalker Version: 18828 Plugin multipoly Version: 18832 Plugin openvisible Version: 17536 Plugin remotecontrol Version: 18678 Plugin utilsplugin Version: 18621 Plugin validator Version: 18848 Plugin walkingpapers Version: 18597 Plugin wmsplugin Version: 18762 agpifoj.visible=false cache.motd.html=1259597144 cache.motd.html.lang=Is: cache.motd.html.version=2555 cache.wmsplugin.expire=-1 cache.wmsplugin.maxsize=-1 color.node=#FF0000 conflictresolutiondialog.height=342 conflictresolutiondialog.width=1001 conflictresolutiondialog.x=159 conflictresolutiondialog.y=209 coordinates=DECIMAL_DEGREES download.places.history=Perth brisbane Santiago, Chile Dortmund draw.fullscreen=true draw.rawgps.colorTracksTune=10 draw.rawgps.colors=1 draw.rawgps.direction=true draw.rawgps.large=true draw.splashscreen=false filter.visible=false gui.geometry=1280x751+0+25 help.window.bounds=72,133,800,600 josm.version=2555 language=is lastDirectory=/home/avar/src/josm marker.audiofromstart=false marker.buttonlabels=false markers.lastaudiodirectory=/home/avar measure.visible=true message.convert_to_data=false message.movedManyElements=false message.pluginmanger.version=false mirror.http://josm.openstreetmap.de/presets=1257355888008;/home/avar/.josm/mirror_http___josm.openstreetmap.de_presets mirror.http://josm.openstreetmap.de/styles=1257355887701;/home/avar/.josm/mirror_http___josm.openstreetmap.de_styles mirror.http://svn.openstreetmap.org/applications/editors/josm/plugins/validator/ignoretags.cfg=1259079054131;/home/avar/.josm/plugins/validator/mirror_http___svn.openstreetmap.org_applications_editors_josm_plugins_validator_ignoretags.cfg mirror.http://svn.openstreetmap.org/applications/editors/josm/plugins/validator/tagchecker.cfg=1259079054235;/home/avar/.josm/plugins/validator/mirror_http___svn.openstreetmap.org_applications_editors_josm_plugins_validator_tagchecker.cfg mirror.http://svn.openstreetmap.org/applications/editors/josm/plugins/wmsplugin/sources.cfg=1259079055582;/home/avar/.josm/plugins/wmsplugin/mirror_http___svn.openstreetmap.org_applications_editors_josm_plugins_wmsplugin_sources.cfg mirror.http://svn.openstreetmap.org/applications/utils/planet.osm/java/speller/words.cfg=1259079053988;/home/avar/.josm/plugins/validator/mirror_http___svn.openstreetmap.org_applications_utils_planet.osm_java_speller_words.cfg org.openstreetmap.josm.actions.UploadAction$UploadDialogPanel=x=374,y=59,width=536,height=400 org.openstreetmap.josm.gui.actionsupport.DeleteFromRelationConfirmationDialog.geometry=x=659,y=435,width=400,height=200 org.openstreetmap.josm.gui.conflict.tags.CombinePrimitiveResolverDialog.geometry=x=165,y=294,width=788,height=445 org.openstreetmap.josm.gui.download.DownloadDialog.geometry=x=187,y=176,width=1000,height=600 org.openstreetmap.josm.gui.io.CloseChangesetDialog.geometry=x=396,y=272,width=562,height=305 org.openstreetmap.josm.gui.io.UploadDialog.geometry=x=255,y=333,width=752,height=423 org.openstreetmap.josm.gui.io.UploadSelectionDialog.geometry=x=241,y=237,width=345,height=400 osb.api.disabled=false osb.auto_download=true osb.include_date=true osb.uri.close=http://openstreetbugs.schokokeks.org/api/0.1/closePOIexec osb.uri.download=http://openstreetbugs.schokokeks.org/api/0.1/getBugs osb.uri.edit=http://openstreetbugs.schokokeks.org/api/0.1/editPOIexec osb.uri.new=http://openstreetbugs.schokokeks.org/api/0.1/addPOIexec osm-download.bounds=65.81065554839948;-16.6387939453125;65.93639447843205;-16.27349853515625 osm-primitives.showid=true pluginmanager.lastupdate=1259587006343 pluginmanager.version=2554 plugins=lakewalker multipoly openvisible remotecontrol utilsplugin validator walkingpapers wmsplugin proxy.anonymous=false proxy.enable=false save.keepbackup=false selectionlist.bounds=470,310,340,179 shortcut.groups.1=128 shortcut.groups.10=64 shortcut.groups.11=576 shortcut.groups.12=64 shortcut.groups.13=512 shortcut.groups.15=640 shortcut.groups.16=640 shortcut.groups.17=576 shortcut.groups.18=512 shortcut.groups.19=128 shortcut.groups.2=128 shortcut.groups.20=512 shortcut.groups.3=0 shortcut.groups.4=512 shortcut.groups.5=0 shortcut.groups.6=512 shortcut.groups.8=192 shortcut.groups.9=192 shortcut.groups.configured=true slippymap.autoload_tiles=true slippymap.autozoom=true slippymap.draw_debug=false slippymap.fade_background=0.0 slippymap.last_zoom_lvl=7 slippymap.max_zoom_lvl=21 slippymap.tile_source=NearMap Australia tagimages.lastdirectory=/home/avar/src/cam/2009-07-29-Mapping-Bankholzen toggleDialogs.width=357 upload.comment.history=Tracing lakes, rivers and riverbanks from Landsat imagery Tracing water north of Vatnajökull from Landsat imagery Foo Tracing Jökulsá á Fjöllum from Landsat imagery Connect two parts of Öxafjarðarheiði Just tag the ice skating ring with an area, not a node Tag a sport / fitness center as such Correctly tag an ice skating stadium Tracing tracks from NearMap Add a footway leading up to Hlíðarvegur Tracing lakes, rivers and coastlines from Landsat imagery Connect Strandvegur & Djúpivegur Tracing residential features from nearmap mapping some features from nearmap imagery Tracing lakes from Landsat imagery validator.TagChecker.checkComplexBeforeUpload=false validator.TagChecker.checkFixmesBeforeUpload=false validator.TagChecker.checkKeysBeforeUpload=false validator.TagChecker.checkValuesBeforeUpload=false validator.TagChecker.paintBeforeUpload=false validator.tests=Coastlines=true,SimilarNamedWays=true,UntaggedNode=true,CrossingWays=true,NodesWithSameName=true,TagChecker=true,OverlappingWays=true,UnclosedWays=true,WronglyOrderedWays=true,UnconnectedWays=true,SelfIntersectingWay=true,DuplicatedWayNodes=true,NameMismatch=true,DuplicateWay=true,UntaggedWay=true,DuplicateNode=true validator.testsBeforeUpload=Coastlines=false,SimilarNamedWays=false,UntaggedNode=false,CrossingWays=true,NodesWithSameName=false,TagChecker=false,OverlappingWays=true,UnclosedWays=false,WronglyOrderedWays=false,UnconnectedWays=false,SelfIntersectingWay=true,DuplicatedWayNodes=false,NameMismatch=true,DuplicateWay=true,UntaggedWay=false,DuplicateNode=false walkingpapers.last-used-id=bnp7xl4b wmsplugin.default.html:http://josm.openstreetmap.de/wmsplugin/YahooDirect.html?=true wmsplugin.default.http://onearth.jpl.nasa.gov/wms.cgi?request_GetMap&layers_global_mosaic&styles_&format_image/jpeg&=true wmsplugin.default.http://openaerialmap.org/wms/?VERSION_1.0&request_GetMap&layers_world&styles_&format_image/jpeg&=true wmsplugin.url.0.name=Czech CUZK:KM wmsplugin.url.0.url=http://wms.cuzk.cz/wms.asp?service=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG:4326&LAYERS=kn,def_budovy,prehledky&FORMAT=image/png&TRANSPARENT=TRUE& wmsplugin.url.1.name=Czech UHUL:ORTOFOTO wmsplugin.url.1.url=http://geoportal2.uhul.cz/cgi-bin/oprl.asp?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG:4326&LAYERS=Ortofoto_cb&STYLES=default&FORMAT=image/jpeg&TRANSPARENT=TRUE& wmsplugin.url.10.name=Streets NRW Geofabrik.de wmsplugin.url.10.url=http://tools.geofabrik.de/osmi/view/strassennrw/josmwms? wmsplugin.url.11.name=Terraserver Topo wmsplugin.url.11.url=http://terraservice.net/ogcmap.ashx?version=1.1.1&request=GetMap&Layers=drg&styles=&format=image/jpeg& wmsplugin.url.12.name=Terraserver Urban wmsplugin.url.12.url=http://terraservice.net/ogcmap.ashx?version=1.1.1&request=GetMap&Layers=urbanarea&styles=&format=image/jpeg& wmsplugin.url.13.name=TilesAtHome wmsplugin.url.13.url=html:http://josm.openstreetmap.de/wmsplugin/TilesAtHome.html? wmsplugin.url.14.name=Yahoo Sat wmsplugin.url.14.url=html:http://josm.openstreetmap.de/wmsplugin/YahooDirect.html? wmsplugin.url.3.name=Landsat wmsplugin.url.3.url=http://onearth.jpl.nasa.gov/wms.cgi?request=GetMap&layers=global_mosaic&styles=&format=image/jpeg& wmsplugin.url.4.name=NPE Maps wmsplugin.url.4.url=http://nick.dev.openstreetmap.org/openpaths/freemap.php?layers=npe& wmsplugin.url.5.name=NPE Maps (Tim) wmsplugin.url.5.url=http://dev.openstreetmap.org/~timsc/wms2/map.php? wmsplugin.url.6.name=NearMap wmsplugin.url.6.url=html:http://map-data.bigtincan.com/wms/nearmap.php? wmsplugin.url.7.name=Open Aerial Map wmsplugin.url.7.url=http://openaerialmap.org/wms/?VERSION=1.0&request=GetMap&layers=world&styles=&format=image/jpeg& wmsplugin.url.8.name=OpenCycleMap wmsplugin.url.8.url=html:http://josm.openstreetmap.de/wmsplugin/OpenCycleMap.html? wmsplugin.url.9.name=OpenStreetMap wmsplugin.url.9.url=html:http://josm.openstreetmap.de/wmsplugin/OpenStreetMap.html? wmsplugin.url.overlap=false wmsplugin.url.overlapEast=14 wmsplugin.url.overlapNorth=4
Attachments (11)
Change History (19)
by , 16 years ago
| Attachment: | Iceland.osm.bz2aa added |
|---|
by , 16 years ago
| Attachment: | Iceland.osm.bz2ab added |
|---|
The dump of Iceland with approx. all the changes, osm.bz2 split with split(1)
by , 16 years ago
| Attachment: | Iceland.osm.bz2ac added |
|---|
The dump of Iceland with approx. all the changes, osm.bz2 split with split(1)
by , 16 years ago
| Attachment: | Iceland-fail.osm.bz2aa added |
|---|
The broken save of Iceland with ~20% ways missing, osm.bz2 split with split(1)
by , 16 years ago
| Attachment: | Iceland-fail.osm.bz2ab added |
|---|
The broken save of Iceland with ~20% ways missing, osm.bz2 split with split(1)
by , 16 years ago
| Attachment: | Iceland-fail.osm.bz2ac added |
|---|
The broken save of Iceland with ~20% ways missing, osm.bz2 split with split(1)
by , 16 years ago
| Attachment: | akureyri-incomplete.png added |
|---|
Example area on the map viewed in JOSM->save->JOSM
by , 16 years ago
| Attachment: | reykjavik-also-complete.png added |
|---|
Example area on the map viewed in JOSM->save->JOSM: No missing ways in this area like the other one
by , 16 years ago
| Attachment: | akureyri-lundahverfi-there-but-not-there.png added |
|---|
In JOSM->save->JOSM: I can select some ways but JOSM doesn't notice that they're in the data when searching through it
comment:1 by , 16 years ago
Also doing the data consistency test in JOSM or JOSM->save->JOSM yields "no problems found".
follow-up: 3 comment:2 by , 16 years ago
Do you still have the broken JOSM running? It might be useful if you do heap dump. You can use jvisualvm (gui tool) or following command:
jmap -dump:live,file=heap.bin <josm pid>
But the generated file will be huge (700M) and chances that we find something useful there are pretty low...
Otherwise only idea I had so far is that QuadBuckets got somehow broken. So iteration over all primitives no longer work (SearchAction, saving to file) but QuadBuckets.search() still works (mappaint, selecting primitive)
comment:3 by , 16 years ago
Replying to jttt:
Do you still have the broken JOSM running? It might be useful if you do heap dump. You can use jvisualvm (gui tool) or following command:
jmap -dump:live,file=heap.bin <josm pid>
But the generated file will be huge (700M) and chances that we find something useful there are pretty low...
Yes I've made a heap dump. It's 374M or 82M bzip2 compressed. I can give you a download link to it but I'd rather not make it publicly available. You can give me your email or send a request to avarab@… and I'll give you a HTTP link to it.
Otherwise only idea I had so far is that QuadBuckets got somehow broken. So iteration over all primitives no longer work (SearchAction, saving to file) but QuadBuckets.search() still works (mappaint, selecting primitive)
comment:4 by , 16 years ago
I've been tracing lakes in the same dataset as the Iceland dump and doing lots of node creation and deletion, as well as redo/undo. I haven't been able to reproduce this, yet. Still working on it, though.
comment:5 by , 16 years ago
avar sent me the heap dump and I've found one QBLevel that have all children set to null. That breaks QBLevel.firstChild method and leads to problems described above (search works, iterator doesn't work)
comment:6 by , 16 years ago
| Resolution: | → fixed |
|---|---|
| Status: | new → closed |
comment:7 by , 16 years ago
Note that I can only hope that the bug is fixed, because I've never managed to reproduce it and I've only fixed one place that hypotetically could create QBLevel with all children set to null.
That behaviour described in this ticket was caused by QBLevel with all children set to null was confirmed in the heapdump from avar.
The dump of Iceland with approx. all the changes, osm.bz2 split with split(1)