Modify

Opened 11 years ago

Closed 11 years ago

Last modified 11 years ago

#4057 closed defect (fixed)

JOSM loses data on save: Approx 20% data loss of ways but none of nodes/relations

Reported by: avarab@… Owned by: team
Priority: critical Milestone:
Component: Core Version: latest
Keywords: Cc:

Description

After reverting the failed lakewalker trace described in this comment to bug #3784 I tried to save the .osm file I was working on. The resulting saved file has around 20% of its ways missing. I didn't delete these because there are no modified objects in the dataset nor are there any pending deletions.

This indicates the approximate dataloss, note that this isn't entirely accurate since Iceland.osm doesn't take into account changes in this changeset, but that's not more than <100 ways and a few thousand nodes:

avar@aoeu:~/src/josm$ grep -c "<way" Iceland.osm Iceland-fail.osm 
Iceland.osm:17159
Iceland-fail.osm:13899
avar@aoeu:~/src/josm$ grep -c "<relation" Iceland.osm Iceland-fail.osm 
Iceland.osm:374
Iceland-fail.osm:374
avar@aoeu:~/src/josm$ grep -c "<node" Iceland.osm Iceland-fail.osm 
Iceland.osm:327739
Iceland-fail.osm:328779

When I search for type:way in the JOSM instance I saved from I get 13899 even though by looking at mappaint it's actually drawing closer to 17159 ways. I.e. I can select some ways in JOSM but I can't find them using the search engine, nor are those same ways saved if I save to .osm.

I'm keeping the JOSM instance which failed like this open for a while in case there's anything more I can do to debug it while it's running. I'm also attaching .osm files & screenshots.

Here's a status report:

Build-Date: 2009-11-30 13:15:54		
Revision: 2554
Is-Local-Build: true

Memory Usage: 729 MB / 986 MB (238 MB allocated, but free)
Java version: 1.6.0_16

Dataset consistency test:
No problems found


Plugins: lakewalker,multipoly,openvisible,remotecontrol,utilsplugin,validator,walkingpapers,wmsplugin
Plugin lakewalker Version: 18828
Plugin multipoly Version: 18832
Plugin openvisible Version: 17536
Plugin remotecontrol Version: 18678
Plugin utilsplugin Version: 18621
Plugin validator Version: 18848
Plugin walkingpapers Version: 18597
Plugin wmsplugin Version: 18762

agpifoj.visible=false
cache.motd.html=1259597144
cache.motd.html.lang=Is:
cache.motd.html.version=2555
cache.wmsplugin.expire=-1
cache.wmsplugin.maxsize=-1
color.node=#FF0000
conflictresolutiondialog.height=342
conflictresolutiondialog.width=1001
conflictresolutiondialog.x=159
conflictresolutiondialog.y=209
coordinates=DECIMAL_DEGREES
download.places.history=Perth
brisbane
Santiago, Chile
Dortmund
draw.fullscreen=true
draw.rawgps.colorTracksTune=10
draw.rawgps.colors=1
draw.rawgps.direction=true
draw.rawgps.large=true
draw.splashscreen=false
filter.visible=false
gui.geometry=1280x751+0+25
help.window.bounds=72,133,800,600
josm.version=2555
language=is
lastDirectory=/home/avar/src/josm
marker.audiofromstart=false
marker.buttonlabels=false
markers.lastaudiodirectory=/home/avar
measure.visible=true
message.convert_to_data=false
message.movedManyElements=false
message.pluginmanger.version=false
mirror.http://josm.openstreetmap.de/presets=1257355888008;/home/avar/.josm/mirror_http___josm.openstreetmap.de_presets
mirror.http://josm.openstreetmap.de/styles=1257355887701;/home/avar/.josm/mirror_http___josm.openstreetmap.de_styles
mirror.http://svn.openstreetmap.org/applications/editors/josm/plugins/validator/ignoretags.cfg=1259079054131;/home/avar/.josm/plugins/validator/mirror_http___svn.openstreetmap.org_applications_editors_josm_plugins_validator_ignoretags.cfg
mirror.http://svn.openstreetmap.org/applications/editors/josm/plugins/validator/tagchecker.cfg=1259079054235;/home/avar/.josm/plugins/validator/mirror_http___svn.openstreetmap.org_applications_editors_josm_plugins_validator_tagchecker.cfg
mirror.http://svn.openstreetmap.org/applications/editors/josm/plugins/wmsplugin/sources.cfg=1259079055582;/home/avar/.josm/plugins/wmsplugin/mirror_http___svn.openstreetmap.org_applications_editors_josm_plugins_wmsplugin_sources.cfg
mirror.http://svn.openstreetmap.org/applications/utils/planet.osm/java/speller/words.cfg=1259079053988;/home/avar/.josm/plugins/validator/mirror_http___svn.openstreetmap.org_applications_utils_planet.osm_java_speller_words.cfg
org.openstreetmap.josm.actions.UploadAction$UploadDialogPanel=x=374,y=59,width=536,height=400
org.openstreetmap.josm.gui.actionsupport.DeleteFromRelationConfirmationDialog.geometry=x=659,y=435,width=400,height=200
org.openstreetmap.josm.gui.conflict.tags.CombinePrimitiveResolverDialog.geometry=x=165,y=294,width=788,height=445
org.openstreetmap.josm.gui.download.DownloadDialog.geometry=x=187,y=176,width=1000,height=600
org.openstreetmap.josm.gui.io.CloseChangesetDialog.geometry=x=396,y=272,width=562,height=305
org.openstreetmap.josm.gui.io.UploadDialog.geometry=x=255,y=333,width=752,height=423
org.openstreetmap.josm.gui.io.UploadSelectionDialog.geometry=x=241,y=237,width=345,height=400
osb.api.disabled=false
osb.auto_download=true
osb.include_date=true
osb.uri.close=http://openstreetbugs.schokokeks.org/api/0.1/closePOIexec
osb.uri.download=http://openstreetbugs.schokokeks.org/api/0.1/getBugs
osb.uri.edit=http://openstreetbugs.schokokeks.org/api/0.1/editPOIexec
osb.uri.new=http://openstreetbugs.schokokeks.org/api/0.1/addPOIexec
osm-download.bounds=65.81065554839948;-16.6387939453125;65.93639447843205;-16.27349853515625
osm-primitives.showid=true
pluginmanager.lastupdate=1259587006343
pluginmanager.version=2554
plugins=lakewalker
multipoly
openvisible
remotecontrol
utilsplugin
validator
walkingpapers
wmsplugin
proxy.anonymous=false
proxy.enable=false
save.keepbackup=false
selectionlist.bounds=470,310,340,179
shortcut.groups.1=128
shortcut.groups.10=64
shortcut.groups.11=576
shortcut.groups.12=64
shortcut.groups.13=512
shortcut.groups.15=640
shortcut.groups.16=640
shortcut.groups.17=576
shortcut.groups.18=512
shortcut.groups.19=128
shortcut.groups.2=128
shortcut.groups.20=512
shortcut.groups.3=0
shortcut.groups.4=512
shortcut.groups.5=0
shortcut.groups.6=512
shortcut.groups.8=192
shortcut.groups.9=192
shortcut.groups.configured=true
slippymap.autoload_tiles=true
slippymap.autozoom=true
slippymap.draw_debug=false
slippymap.fade_background=0.0
slippymap.last_zoom_lvl=7
slippymap.max_zoom_lvl=21
slippymap.tile_source=NearMap Australia
tagimages.lastdirectory=/home/avar/src/cam/2009-07-29-Mapping-Bankholzen
toggleDialogs.width=357
upload.comment.history=Tracing lakes, rivers and riverbanks from Landsat imagery
Tracing water north of Vatnajökull from Landsat imagery
Foo
Tracing Jökulsá á Fjöllum from Landsat imagery
Connect two parts of Öxafjarðarheiði
Just tag the ice skating ring with an area, not a node
Tag a sport / fitness center as such
Correctly tag an ice skating stadium
Tracing tracks from NearMap
Add a footway leading up to Hlíðarvegur
Tracing lakes, rivers and coastlines from Landsat imagery
Connect Strandvegur & Djúpivegur
Tracing residential features from nearmap
mapping some features from nearmap imagery
Tracing lakes from Landsat imagery
validator.TagChecker.checkComplexBeforeUpload=false
validator.TagChecker.checkFixmesBeforeUpload=false
validator.TagChecker.checkKeysBeforeUpload=false
validator.TagChecker.checkValuesBeforeUpload=false
validator.TagChecker.paintBeforeUpload=false
validator.tests=Coastlines=true,SimilarNamedWays=true,UntaggedNode=true,CrossingWays=true,NodesWithSameName=true,TagChecker=true,OverlappingWays=true,UnclosedWays=true,WronglyOrderedWays=true,UnconnectedWays=true,SelfIntersectingWay=true,DuplicatedWayNodes=true,NameMismatch=true,DuplicateWay=true,UntaggedWay=true,DuplicateNode=true
validator.testsBeforeUpload=Coastlines=false,SimilarNamedWays=false,UntaggedNode=false,CrossingWays=true,NodesWithSameName=false,TagChecker=false,OverlappingWays=true,UnclosedWays=false,WronglyOrderedWays=false,UnconnectedWays=false,SelfIntersectingWay=true,DuplicatedWayNodes=false,NameMismatch=true,DuplicateWay=true,UntaggedWay=false,DuplicateNode=false
walkingpapers.last-used-id=bnp7xl4b
wmsplugin.default.html:http://josm.openstreetmap.de/wmsplugin/YahooDirect.html?=true
wmsplugin.default.http://onearth.jpl.nasa.gov/wms.cgi?request_GetMap&layers_global_mosaic&styles_&format_image/jpeg&=true
wmsplugin.default.http://openaerialmap.org/wms/?VERSION_1.0&request_GetMap&layers_world&styles_&format_image/jpeg&=true
wmsplugin.url.0.name=Czech CUZK:KM
wmsplugin.url.0.url=http://wms.cuzk.cz/wms.asp?service=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG:4326&LAYERS=kn,def_budovy,prehledky&FORMAT=image/png&TRANSPARENT=TRUE&
wmsplugin.url.1.name=Czech UHUL:ORTOFOTO
wmsplugin.url.1.url=http://geoportal2.uhul.cz/cgi-bin/oprl.asp?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG:4326&LAYERS=Ortofoto_cb&STYLES=default&FORMAT=image/jpeg&TRANSPARENT=TRUE&
wmsplugin.url.10.name=Streets NRW Geofabrik.de
wmsplugin.url.10.url=http://tools.geofabrik.de/osmi/view/strassennrw/josmwms?
wmsplugin.url.11.name=Terraserver Topo
wmsplugin.url.11.url=http://terraservice.net/ogcmap.ashx?version=1.1.1&request=GetMap&Layers=drg&styles=&format=image/jpeg&
wmsplugin.url.12.name=Terraserver Urban
wmsplugin.url.12.url=http://terraservice.net/ogcmap.ashx?version=1.1.1&request=GetMap&Layers=urbanarea&styles=&format=image/jpeg&
wmsplugin.url.13.name=TilesAtHome
wmsplugin.url.13.url=html:http://josm.openstreetmap.de/wmsplugin/TilesAtHome.html?
wmsplugin.url.14.name=Yahoo Sat
wmsplugin.url.14.url=html:http://josm.openstreetmap.de/wmsplugin/YahooDirect.html?
wmsplugin.url.3.name=Landsat
wmsplugin.url.3.url=http://onearth.jpl.nasa.gov/wms.cgi?request=GetMap&layers=global_mosaic&styles=&format=image/jpeg&
wmsplugin.url.4.name=NPE Maps
wmsplugin.url.4.url=http://nick.dev.openstreetmap.org/openpaths/freemap.php?layers=npe&
wmsplugin.url.5.name=NPE Maps (Tim)
wmsplugin.url.5.url=http://dev.openstreetmap.org/~timsc/wms2/map.php?
wmsplugin.url.6.name=NearMap
wmsplugin.url.6.url=html:http://map-data.bigtincan.com/wms/nearmap.php?
wmsplugin.url.7.name=Open Aerial Map
wmsplugin.url.7.url=http://openaerialmap.org/wms/?VERSION=1.0&request=GetMap&layers=world&styles=&format=image/jpeg&
wmsplugin.url.8.name=OpenCycleMap
wmsplugin.url.8.url=html:http://josm.openstreetmap.de/wmsplugin/OpenCycleMap.html?
wmsplugin.url.9.name=OpenStreetMap
wmsplugin.url.9.url=html:http://josm.openstreetmap.de/wmsplugin/OpenStreetMap.html?
wmsplugin.url.overlap=false
wmsplugin.url.overlapEast=14
wmsplugin.url.overlapNorth=4

Attachments (11)

Iceland.osm.bz2aa (1.7 MB) - added by avarab@… 11 years ago.
The dump of Iceland with approx. all the changes, osm.bz2 split with split(1)
Iceland.osm.bz2ab (1.7 MB) - added by avarab@… 11 years ago.
The dump of Iceland with approx. all the changes, osm.bz2 split with split(1)
Iceland.osm.bz2ac (1.2 MB) - added by avarab@… 11 years ago.
The dump of Iceland with approx. all the changes, osm.bz2 split with split(1)
Iceland-fail.osm.bz2aa (1.7 MB) - added by avarab@… 11 years ago.
The broken save of Iceland with ~20% ways missing, osm.bz2 split with split(1)
Iceland-fail.osm.bz2ab (1.7 MB) - added by avarab@… 11 years ago.
The broken save of Iceland with ~20% ways missing, osm.bz2 split with split(1)
Iceland-fail.osm.bz2ac (966.2 KB) - added by avarab@… 11 years ago.
The broken save of Iceland with ~20% ways missing, osm.bz2 split with split(1)
akureyri-complete.png (173.9 KB) - added by avarab@… 11 years ago.
Example area on the map viewed in JOSM
akureyri-incomplete.png (142.3 KB) - added by avarab@… 11 years ago.
Example area on the map viewed in JOSM->save->JOSM
reykjavík-complete.png (168.1 KB) - added by avarab@… 11 years ago.
Example area on the map viewed in JOSM
reykjavik-also-complete.png (167.8 KB) - added by avarab@… 11 years ago.
Example area on the map viewed in JOSM->save->JOSM: No missing ways in this area like the other one
akureyri-lundahverfi-there-but-not-there.png (160.9 KB) - added by avarab@… 11 years ago.
In JOSM->save->JOSM: I can select some ways but JOSM doesn't notice that they're in the data when searching through it

Change History (19)

Changed 11 years ago by avarab@…

Attachment: Iceland.osm.bz2aa added

The dump of Iceland with approx. all the changes, osm.bz2 split with split(1)

Changed 11 years ago by avarab@…

Attachment: Iceland.osm.bz2ab added

The dump of Iceland with approx. all the changes, osm.bz2 split with split(1)

Changed 11 years ago by avarab@…

Attachment: Iceland.osm.bz2ac added

The dump of Iceland with approx. all the changes, osm.bz2 split with split(1)

Changed 11 years ago by avarab@…

Attachment: Iceland-fail.osm.bz2aa added

The broken save of Iceland with ~20% ways missing, osm.bz2 split with split(1)

Changed 11 years ago by avarab@…

Attachment: Iceland-fail.osm.bz2ab added

The broken save of Iceland with ~20% ways missing, osm.bz2 split with split(1)

Changed 11 years ago by avarab@…

Attachment: Iceland-fail.osm.bz2ac added

The broken save of Iceland with ~20% ways missing, osm.bz2 split with split(1)

Changed 11 years ago by avarab@…

Attachment: akureyri-complete.png added

Example area on the map viewed in JOSM

Changed 11 years ago by avarab@…

Attachment: akureyri-incomplete.png added

Example area on the map viewed in JOSM->save->JOSM

Changed 11 years ago by avarab@…

Attachment: reykjavík-complete.png added

Example area on the map viewed in JOSM

Changed 11 years ago by avarab@…

Attachment: reykjavik-also-complete.png added

Example area on the map viewed in JOSM->save->JOSM: No missing ways in this area like the other one

Changed 11 years ago by avarab@…

In JOSM->save->JOSM: I can select some ways but JOSM doesn't notice that they're in the data when searching through it

comment:1 Changed 11 years ago by avarab@…

Also doing the data consistency test in JOSM or JOSM->save->JOSM yields "no problems found".

comment:2 Changed 11 years ago by jttt

Do you still have the broken JOSM running? It might be useful if you do heap dump. You can use jvisualvm (gui tool) or following command:
jmap -dump:live,file=heap.bin <josm pid>
But the generated file will be huge (700M) and chances that we find something useful there are pretty low...

Otherwise only idea I had so far is that QuadBuckets got somehow broken. So iteration over all primitives no longer work (SearchAction, saving to file) but QuadBuckets.search() still works (mappaint, selecting primitive)

comment:3 in reply to:  2 Changed 11 years ago by avarab@…

Replying to jttt:

Do you still have the broken JOSM running? It might be useful if you do heap dump. You can use jvisualvm (gui tool) or following command:
jmap -dump:live,file=heap.bin <josm pid>
But the generated file will be huge (700M) and chances that we find something useful there are pretty low...

Yes I've made a heap dump. It's 374M or 82M bzip2 compressed. I can give you a download link to it but I'd rather not make it publicly available. You can give me your email or send a request to avarab@… and I'll give you a HTTP link to it.

Otherwise only idea I had so far is that QuadBuckets got somehow broken. So iteration over all primitives no longer work (SearchAction, saving to file) but QuadBuckets.search() still works (mappaint, selecting primitive)

comment:4 Changed 11 years ago by hansendc

I've been tracing lakes in the same dataset as the Iceland dump and doing lots of node creation and deletion, as well as redo/undo. I haven't been able to reproduce this, yet. Still working on it, though.

comment:5 Changed 11 years ago by jttt

avar sent me the heap dump and I've found one QBLevel that have all children set to null. That breaks QBLevel.firstChild method and leads to problems described above (search works, iterator doesn't work)

comment:6 Changed 11 years ago by jttt

Resolution: fixed
Status: newclosed

(In [2896]) Fixed #4057 JOSM loses data on save: Approx 20% data loss of ways but none of nodes/relations

comment:7 Changed 11 years ago by jttt

Note that I can only hope that the bug is fixed, because I've never managed to reproduce it and I've only fixed one place that hypotetically could create QBLevel with all children set to null.

That behaviour described in this ticket was caused by QBLevel with all children set to null was confirmed in the heapdump from avar.

comment:8 Changed 11 years ago by anonymous

Ticket #4324 has been marked as a duplicate of this ticket.

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain team.
as The resolution will be set.
The resolution will be deleted.

Add Comment


E-mail address and name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.