[Patch] DataIntegrityProblemException when reverting CS 47770943

[GerdP]

What steps will reproduce the problem?

1. Start JOSM with reverter plugin
2. Try to revert 47770943

What is the expected result?

reverted data, probably with conflicts

What happens instead?

popup with this error message

Please provide any additional information below. Attach a screenshot if possible.

Build-Date:2017-09-13 15:14:52
Revision:12839
Is-Local-Build:true

Identification: JOSM/1.5 (12839 SVN de) Windows 10 64-Bit
OS Build number: Windows 10 Home 1703 (15063)
Memory Usage: 1801 MB / 5461 MB (1365 MB allocated, but free)
Java version: 1.8.0_121-b13, Oracle Corporation, Java HotSpot(TM) 64-Bit Server VM
Screen: \Display0 1920x1080
Maximum Screen Size: 1920x1080

Dataset consistency test:
[DELETED REFERENCED] {Relation id=1653406 version=3 MT [Linie 120591203, Linie 120591205]} refers to deleted primitive {Way id=120591203 version=5 D nodes=[]}


Plugins:
+ OpeningHoursEditor (33185)
+ apache-commons (33517)
+ buildings_tools (33004)
+ ejml (32680)
+ geotools (33380)
+ jts (32699)
+ o5m (33566)
+ opendata (33613)
+ pbf (33568)
+ poly (33570)
+ reverter (33572)
+ utilsplugin2 (33579)

Last errors/warnings:
- W: New conflict: Conflict [my={Node id=919155558 version=6 D }, their={Node id=919155558 version=0 IV lat=52.537147,lon=17.5932961}]
- W: New conflict: Conflict [my={Node id=919155565 version=3 D }, their={Node id=919155565 version=0 IV lat=52.5368374,lon=17.5932426}]
- W: New conflict: Conflict [my={Node id=1880045050 version=3 D }, their={Node id=1880045050 version=0 IV lat=52.5367725,lon=17.5933115}]
- W: New conflict: Conflict [my={Node id=898267401 version=5 D }, their={Node id=898267401 version=0 IV lat=52.5367707,lon=17.5932898}]
- W: New conflict: Conflict [my={Node id=1880045048 version=2 D }, their={Node id=1880045048 version=0 IV lat=52.5367705,lon=17.5932756}]
- E: Unable to revert {Way id=107997507 version=5 VT nodes=[{Node id=1239920255 version=0 IV lat=52.536723,lon=17.5967509}, {Node id=1239920258 version=0 IV lat=52.5367594,lon=17.5967488}, {Node id=1239920268 version=0 IV lat=52.5367605,lon=17.5967052}, {Node id=1239920270 version=0 IV lat=52.5369863,lon=17.5967198}, {Node id=1239920272 version=0 IV lat=52.5369678,lon=17.5973649}, {Node id=1239920275 version=0 IV lat=52.5368867,lon=17.5973573}, {Node id=1239920277 version=0 IV lat=52.5368898,lon=17.5972744}, {Node id=1239920280 version=0 IV lat=52.5366627,lon=17.5972448}, {Node id=1239920282 version=0 IV lat=52.5366664,lon=17.5970526}, {Node id=1239920283 version=0 IV lat=52.5367193,lon=17.5970546}, {Node id=1239920255 version=0 IV lat=52.536723,lon=17.5967509}]} as it produces 0 nodes way {Way id=107997507 version=7 MT nodes=[]}
- W: New conflict: Conflict [my={Node id=4318089454 version=2 D }, their={Node id=4318089454 version=0 IV lat=52.5367962,lon=17.5933824}]
- W: New conflict: Conflict [my={Node id=4318089455 version=2 D }, their={Node id=4318089455 version=0 IV lat=52.5367948,lon=17.5933416}]
- W: New conflict: Conflict [my={Node id=4318078376 version=3 D }, their={Node id=4318078376 version=0 IV lat=52.5367233,lon=17.5933456}]
- E: Handled by bug report queue: org.openstreetmap.josm.data.osm.DataIntegrityProblemException: Deleted member referenced: {Relation id=1653406 version=3 MT [Linie 120591203, Linie 120591205]}


=== REPORTED CRASH DATA ===
BugReportExceptionHandler#handleException:
No data collected.

Warning issued by: BugReportExceptionHandler#handleException

=== STACK TRACE ===
Thread: AWT-EventQueue-0 (19) of main
org.openstreetmap.josm.data.osm.DataIntegrityProblemException: Deleted member referenced: {Relation id=1653406 version=3 MT [Linie 120591203, Linie 120591205]}
	at org.openstreetmap.josm.data.osm.Relation.checkMembers(Relation.java:512)
	at org.openstreetmap.josm.data.osm.Relation.fireMembersChanged(Relation.java:524)
	at org.openstreetmap.josm.data.osm.Relation.setMembers(Relation.java:65)
	at org.openstreetmap.josm.data.osm.Relation.cloneFrom(Relation.java:258)
	at org.openstreetmap.josm.command.ChangeCommand.executeCommand(ChangeCommand.java:81)
	at org.openstreetmap.josm.command.SequenceCommand.executeCommand(SequenceCommand.java:80)
	at org.openstreetmap.josm.data.UndoRedoHandler.addNoRedraw(UndoRedoHandler.java:71)
	at org.openstreetmap.josm.data.UndoRedoHandler.add(UndoRedoHandler.java:99)
	at reverter.RevertChangesetTask$3.run(RevertChangesetTask.java:126)
	at java.awt.event.InvocationEvent.dispatch(Unknown Source)
	at java.awt.EventQueue.dispatchEventImpl(Unknown Source)
	at java.awt.EventQueue.access$500(Unknown Source)
	at java.awt.EventQueue$3.run(Unknown Source)
	at java.awt.EventQueue$3.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
	at java.awt.EventQueue.dispatchEvent(Unknown Source)
	at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
	at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
	at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
	at java.awt.WaitDispatchSupport$2.run(Unknown Source)
	at java.awt.WaitDispatchSupport$4.run(Unknown Source)
	at java.awt.WaitDispatchSupport$4.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.awt.WaitDispatchSupport.enter(Unknown Source)
	at java.awt.Dialog.show(Unknown Source)
	at java.awt.Component.show(Unknown Source)
	at java.awt.Component.setVisible(Unknown Source)
	at java.awt.Window.setVisible(Unknown Source)
	at java.awt.Dialog.setVisible(Unknown Source)
	at org.openstreetmap.josm.gui.progress.swing.PleaseWaitProgressMonitor.lambda$doBeginTask$3(PleaseWaitProgressMonitor.java:256)
	at org.openstreetmap.josm.gui.progress.swing.PleaseWaitProgressMonitor.lambda$doInEDT$0(PleaseWaitProgressMonitor.java:114)
	at java.awt.event.InvocationEvent.dispatch(Unknown Source)
	at java.awt.EventQueue.dispatchEventImpl(Unknown Source)
	at java.awt.EventQueue.access$500(Unknown Source)
	at java.awt.EventQueue$3.run(Unknown Source)
	at java.awt.EventQueue$3.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
	at java.awt.EventQueue.dispatchEvent(Unknown Source)
	at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
	at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
	at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
	at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
	at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
	at java.awt.EventDispatchThread.run(Unknown Source)

[Don-vip]

Ticket #15373 has been marked as a duplicate of this ticket.

[GerdP]

I tried to find an error in the reverter plugin but I think there might be an error in the OSM DB.
The history for way 107997507 says that all nodes of the way were deleted, but not the way itself.
​https://www.openstreetmap.org/way/107997507/history
Is that really possible? I thought a way without nodes is not allowed?

[GerdP]

Or maybe that is a misinterpretation of the web server. According to the api the nodes were not removed, only all tags:
​https://www.openstreetmap.org/api/0.6/way/107997507/history

[GerdP]

Forget the comments above, I got confused by the fact that the history browser shows the current status of the objects. I always assumed that it shows the changeset actions, means, I thought a stroked id means the object was removed in that cs.

[GerdP]

I cannot reproduce exactly the same error but I think the problem is this:
In CS 47770943 many relations are changed, at the moment I see

Deleted member referenced: {Relation id=1535571 version=3 MT [way 107997507, way 107997508]}

The current status of this relation and both ways is "deleted". The plugin seems to have problems with this case.
Before CS 47770943 all three objects existed.
In CS 47770943 the tags of way 107997507 and rel 1535571 are modified, but way 107997508 is not changed.
When reverter downloads data to revert the change the way 107997508 is never downloaded. I found no code which would calculate the
version of the way that needs to be downloaded.

[GerdP]

@upliner: One more info: When I change preference debug.checkDeleteReferenced to false the revert works (with 226 conflicts).
The dataset consistency check (right click on data layer) shows

[DELETED REFERENCED] {Relation id=1653406 version=3 MT [way 120591203, way 120591205]} refers to deleted primitive {Way id=120591203 version=5 D nodes=[]}
[DELETED REFERENCED] {Relation id=1535571 version=3 MT [way 107997507, way 107997508]} refers to deleted primitive {Way id=107997507 version=7 D nodes=[]}

Now I can use the undetele plugin to undelete w120591203 w107997507 to a new layer.
Finally I can merge the two layers and I see no more consistency problems.
Not sure if the reverter should do exactly the same, but something like that.

[GerdP]

I still found no fix for this. I think the revert crashes whenever we get to this line in reverter.DataSetCommandMerger and the source way has parent relations.

  Logging.error("Unable to revert "+source+" as it produces 0 nodes way "+newTarget);

[GerdP]

I think I finally found a reason for the problem. The method fixNodesWithoutCoordinates() is not prepared for the case that a node
was deleted or changed after the changeset which should be reverted was closed.
This results in a lot of wrong conflicts or ChangeCommands.
My understanding is that this routine should not only set the coordinates of the nodes without coordinates. Instead it should find the version that was active when the changeset was closed.
The attached patch fixes this but doesn't solve other problems like the one in #17312
Probably the same problem exists for nodes and ways.

[GerdP]

improved version

[GerdP]

Please review, if I hear no complains I'll commit this patch on friday (after fixing checkstyle issues)

[GerdP]

See [o34930:34931]