DataIntegrityProblemException: Relation member must be part of the same dataset as relation

[skyper]

Heureka, I found one reproducible way:

What steps will reproduce the problem?

1. Open relation editor with a relation
2. Create a new way. I did split an existing one.
3. Select way and add it to the relation in the relation manager but do not save or close it.
4. Undo last change (creation of new way)
5. Save relation in relation manager or close relation manager
6. (later creates a conflict)
7. (try to solve conflict by opening it through conflict list panel)

What is the expected result?

No exception

What happens instead?

DataIntegrityProblemException: Relation member must be part of the same dataset as relation

• either by saving in relation manager
• or bigger problem, by opening the conflict dialog

Please provide any additional information below. Attach a screenshot if possible.

I had to manually delete the non-existing way from the relation to get rid of the data inconsistency.

Relative:URL: ^/trunk
Repository:UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
Last:Changed Date: 2020-06-03 21:59:58 +0200 (Wed, 03 Jun 2020)
Revision:16540
Build-Date:2020-06-04 01:30:48
URL:https://josm.openstreetmap.de/svn/trunk

Dataset consistency test:
[NO DATASET] {Way id=-116460 version=0 MVT> nodes=[{Node id=7595952827 version=1 VT lat=47.62064273589,lon=8.21870135149}, {Node id=7595952836 version=1 V lat=47.62076724308,lon=8.21864507466}]} is referenced by {Relation id=10751781 version=3 VT [way 813199195, way 156641560, way 813199198, way 91989673, way 813199197, way 37642856, way 111655397, way 708195987, way 32898067, way 319382676, way 319382672, way 319382674, way 319382675, way 218499396, way 319382677, way 319193447, way 319339813, way 813199206, way 813199207, way 689747754, way 405108448, way 405108453, way 668347833, way 405108455, way 405292499, way 319339818, way 688139903, way 405210448, way 319339816, way 573396449, way 405108441, way 405108468, way 405108445, way 446474138, way 446474137, way 813199208, way 689753641, way 668347836, way 668347839, way 668347840, way 493408784, way 319339822, way 668347821, way 319339809, way 813199215, way 813199211, way 813199212, way 813199209, way 813199210, way 668347842, way 319339820, way 319339815, way 813199216, way 319341061, way 319341073, way 319341055, way 319341056, way 813199214, way 813199213, way 81069707, way 467290797, way 319341053, way 230699852, way 103191373, way 319389360, way 25962870, way 319385270, way 37609618, way 37609619, way 319385269, way 38319435, way 104556361, way 465470098, way 38319381, way 38319369, way 284902139, way 465470099, way 364419876, way 30771261, way 25711490, way 25711720, way 581455538, way 581455532, way 416771371, way 371934806, way 32416794, way 284902144, way 320856209, way 38319233, way 465468464, way 464296093, way 284902147, way 379623073, way 32416819, way 374167583, way 464131663, way 508017645, way 374167789, way 464132575, way 508017646, way 217429289, way 41127555, way 321307082, way 41184530, way 464338716, way 41184531, way 284902172, way 284902176, way 435059395, way 561487667, way 284909301, way 561478952, way 284909312, way 457972497, way 561478955, way 561478956, way 284909313, way 561478957, way 561478958, way 561478959, way 284909319, way 32404662, way 284909321, way 561478951, way 776328787, node 3762662985, node 3762666733, way -116460]} but not found in dataset

Last errors/warnings:
- E: Handled by bug report queue: org.openstreetmap.josm.data.osm.DataIntegrityProblemException: Relation member must be part of the same dataset as relation(relation 10751781, way -116460)


=== REPORTED CRASH DATA ===
BugReportExceptionHandler#handleException:
No data collected.

Warning issued by: BugReportExceptionHandler#handleException

=== STACK TRACE ===
Thread: AWT-EventQueue-0 (18) of main
org.openstreetmap.josm.data.osm.DataIntegrityProblemException: Relation member must be part of the same dataset as relation(relation 10751781, way -116460)
	at org.openstreetmap.josm.data.osm.Relation.checkMembers(Relation.java:485)
	at org.openstreetmap.josm.data.osm.Relation.fireMembersChanged(Relation.java:503)
	at org.openstreetmap.josm.data.osm.Relation.setMembers(Relation.java:61)
	at org.openstreetmap.josm.data.osm.Relation.cloneFrom(Relation.java:258)
	at org.openstreetmap.josm.data.osm.OsmPrimitive.cloneFrom(OsmPrimitive.java:864)
	at org.openstreetmap.josm.command.ChangeCommand.executeCommand(ChangeCommand.java:67)
	at org.openstreetmap.josm.data.UndoRedoHandler.addNoRedraw(UndoRedoHandler.java:299)
	at org.openstreetmap.josm.data.UndoRedoHandler.add(UndoRedoHandler.java:353)
	at org.openstreetmap.josm.gui.dialogs.relation.actions.SavingAction.applyExistingNonConflictingRelation(SavingAction.java:101)
	at org.openstreetmap.josm.gui.dialogs.relation.actions.SavingAction.applyChanges(SavingAction.java:174)
	at org.openstreetmap.josm.gui.dialogs.relation.actions.ApplyAction.actionPerformed(ApplyAction.java:31)
	at java.desktop/javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:1967)
	at java.desktop/javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2308)
	at java.desktop/javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:405)
	at java.desktop/javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:262)
	at java.desktop/javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:279)
	at java.desktop/java.awt.AWTEventMulticaster.mouseReleased(AWTEventMulticaster.java:297)
	at java.desktop/java.awt.Component.processMouseEvent(Component.java:6631)
	at java.desktop/javax.swing.JComponent.processMouseEvent(JComponent.java:3342)
	at java.desktop/java.awt.Component.processEvent(Component.java:6396)
	at java.desktop/java.awt.Container.processEvent(Container.java:2263)
	at java.desktop/java.awt.Component.dispatchEventImpl(Component.java:5007)
	at java.desktop/java.awt.Container.dispatchEventImpl(Container.java:2321)
	at java.desktop/java.awt.Component.dispatchEvent(Component.java:4839)
	at java.desktop/java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4918)
	at java.desktop/java.awt.LightweightDispatcher.processMouseEvent(Container.java:4547)
	at java.desktop/java.awt.LightweightDispatcher.dispatchEvent(Container.java:4488)
	at java.desktop/java.awt.Container.dispatchEventImpl(Container.java:2307)
	at java.desktop/java.awt.Window.dispatchEventImpl(Window.java:2772)
	at java.desktop/java.awt.Component.dispatchEvent(Component.java:4839)
	at java.desktop/java.awt.EventQueue.dispatchEventImpl(EventQueue.java:772)
	at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:721)
	at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:715)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:85)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:95)
	at java.desktop/java.awt.EventQueue$5.run(EventQueue.java:745)
	at java.desktop/java.awt.EventQueue$5.run(EventQueue.java:743)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:85)
	at java.desktop/java.awt.EventQueue.dispatchEvent(EventQueue.java:742)
	at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:203)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:124)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:113)
	at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:109)
	at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
	at java.desktop/java.awt.EventDispatchThread.run(EventDispatchThread.java:90)

[simon04]

Interestingly, a conflict is being reported after clicking away the BugReportDialog.

---

With #19343, the exception will be

DataIntegrityProblemException: Relation member must be part of the same dataset as relation(relation 2260670, way -106787) (changed by the following commands: [Sequence: Add a new node to an existing way (undone)])

[GerdP]

I think problem is that the code in RelationEditor only checks rm.getMember().isDeleted(), but in this case the member is "unusable" because it has rm.getMember().getDataSet() == null.

[GerdP]

There is dead code which should produce a warning popup

            String msg = tr("One or more members of this new relation have been deleted while the relation editor\n" +
            "was open. They have been removed from the relation members list.");
            JOptionPane.showMessageDialog(MainApplication.getMainFrame(), msg, tr("Warning"), JOptionPane.WARNING_MESSAGE);

With the current code this popup never appears. Should I remove the dead code or should I fix it so that the popup is shown?
Why would we only show it for new relations?

[GerdP]

patch to fix the error and remove the dead code reg. warning

[GerdP]

alternative patch to fix the error and make the warning work (only for new relations)

[skyper]

Replying to GerdP:

There is dead code which should produce a warning popup

            String msg = tr("One or more members of this new relation have been deleted while the relation editor\n" +
            "was open. They have been removed from the relation members list.");
            JOptionPane.showMessageDialog(MainApplication.getMainFrame(), msg, tr("Warning"), JOptionPane.WARNING_MESSAGE);

With the current code this popup never appears.

Which change did change this?

Should I remove the dead code or should I fix it so that the popup is shown?

Good question. If you delete the just added way instead of undo the problem is a bit different but exists, too, this time even with objects ids > 0:

• Ok button and closing the relation manager silently remove the non-existing members from the members list.
• Save raises the exception.

I'd prefer a message in all cases instead of adjusting the members list silently.

Why would we only show it for new relations?

Does not make sense as my examples are valid for existing relations and existing ways.

[GerdP]

Replying to skyper:

Replying to GerdP:

There is dead code which should produce a warning popup

            String msg = tr("One or more members of this new relation have been deleted while the relation editor\n" +
            "was open. They have been removed from the relation members list.");
            JOptionPane.showMessageDialog(MainApplication.getMainFrame(), msg, tr("Warning"), JOptionPane.WARNING_MESSAGE);

With the current code this popup never appears.

Which change did change this?

It is dead code since r11684.

Should I remove the dead code or should I fix it so that the popup is shown?

Good question. If you delete the just added way instead of undo the problem is a bit different but exists, too, this time even with objects ids > 0:

• Ok button and closing the relation manager silently remove the non-existing members from the members list.
• Save raises the exception.

I'd prefer a message in all cases instead of adjusting the members list silently.

Why would we only show it for new relations?

Does not make sense as my examples are valid for existing relations and existing ways.

What kind of message? A popup that has to be confirmed or a notification?
The current code (and also the patched code) doesn't update the member list, the removed way is shown as long as the relation editor is not closed.

[skyper]

Replying to GerdP:

Replying to skyper:

Why would we only show it for new relations?

Does not make sense as my examples are valid for existing relations and existing ways.

What kind of message? A popup that has to be confirmed or a notification?
The current code (and also the patched code) doesn't update the member list, the removed way is shown as long as the relation editor is not closed.

Think a notification should be enough, as the changes were made outside of the relation editor.
In cases of only using save without closing the relation editor, the member list needs to be updated.

[GerdP]

Thinking again about it we have to take care about the case that a user first uses undo and than redo. Would be a bad idea to update the member list while the editor is still open.

[GerdP]

Not sure if this is what you want. With 19353-show-notification.patch the notification appears when you copy or save the relation. You probably want it in the moment where you delete the object that is shown in the relation editor.
This requires a different approach. Also, the notification might not be visible when the relation editor window is in the lower left corner.
The concept of the relation editor is that it ignores deleted objetcs, I don't dare to change this.
There is this comment in org.openstreetmap.josm.gui.dialogs.relation.MemberTableModel:

    public void primitivesRemoved(PrimitivesRemovedEvent event) {
        // ignore - the relation in the editor might become out of sync with the relation
        // in the dataset. We will deal with it when the relation editor is closed or
        // when the changes in the editor are applied.
    }

My conclusion: 19353.patch is the best choice so far.
Maybe "invalid" members could be marked somehow, e.g. with a different colour, but I think that would be a different ticket.

[skyper]

A notification at the moment the object is deleted contradicts current independence of the relation editor.

Looking at it again I think it is more tricky. We have two cases:

1. Closing the relation dialog.
2. Saving or copying the relation.

You are right about the possible hidden notification which is a general problem as I often have several relation editors open at the same time and undocked panels, history viewer and changeset manager could hide it, too. Is it possible to get the notifications on top for all cases?

Saving and copying need to update the member list to reflect corresponding layer data.

Then we have the problem with undo/redo:

1. Closing the relation dialog should not be a problem
2. Saving might be tricky.

Did not try with some edits in between and undo/redo several edits at once.

My major concern is the correct member list when the relation editor is not closed but saved several times and then some undo/redo action takes place. At least the reload button needs to be active in cases where the member list gets out of sync with the layer data.

[GerdP]

In 17063/josm:

fix #19353: DataIntegrityProblemException: Relation member must be part of the same dataset as relation

• recognize members which are not part of dataset (happens when undoing an add action)
• remove dead code which showed popup
• change logic which enables the refresh button (logic taken from CancelAction)

[GerdP]

@skyper: Please create a new ticket if you think the refresh still doesn't work as wanted.

[Klumbumbus]

This possibly fixed some of the other "Relation member must be part of the same dataset as relation" bugs too?
https://josm.openstreetmap.de/query?status=assigned&status=needinfo&status=new&status=reopened&description=~Relation+member+must+be+part+of+the+same+dataset+as+relation&col=id&col=summary&col=status&col=type&col=priority&col=milestone&col=component&order=priority

[GerdP]

Possibly, even likely. #13773 looks different, more likely a problem in the terracer plugin?
I see no way to make sure that they were all caused by the same problem.

[GerdP]

I've now closed those which contain relation.actions.SavingAction in the traceback. Many (maybe all) of the others contain a text like This relation has been changed outside of the editor. (or the i18n translation) which probably means the user faced the same problem but somehow managed to continue.
I see terracer plugin and type=associatedStreet relations in some of the remaining. I'll take a closer look at this combination...

[Klumbumbus]

Ticket #14405 has been marked as a duplicate of this ticket.

[Klumbumbus]

Ticket #18689 has been marked as a duplicate of this ticket.

[Klumbumbus]

Ticket #19424 has been marked as a duplicate of this ticket.

[Klumbumbus]

Ticket #19707 has been marked as a duplicate of this ticket.

[Klumbumbus]

Ticket #19737 has been marked as a duplicate of this ticket.

[skyper]

Replying to GerdP:

@skyper: Please create a new ticket if you think the refresh still doesn't work as wanted.

Sorry, I am a bit late, but did not work with relations that much. Sadly, this does not work at all, see #19913.

[GerdP]

Ticket #19946 has been marked as a duplicate of this ticket.