Modify

Opened 6 years ago

Closed 8 days ago

#10585 closed defect (duplicate)

DataIntegrityProblemException: Deleted node referenced

Reported by: anonymous Owned by: team
Priority: normal Milestone:
Component: Plugin tracer Version:
Keywords: Cc:

Description

zaseknutí JOSM při trasování polí

Repository Root: http://josm.openstreetmap.de/svn
Build-Date: 2014-10-01 01:34:13
Last Changed Author: Don-vip
Revision: 7591
Repository UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
Relative URL: ^/trunk
URL: http://josm.openstreetmap.de/svn/trunk
Last Changed Date: 2014-09-30 23:56:05 +0200 (Tue, 30 Sep 2014)
Last Changed Rev: 7591

Identification: JOSM/1.5 (7591 cs) Windows 7 64-Bit
Memory Usage: 957 MB / 989 MB (298 MB allocated, but free)
Java version: 1.7.0_67, Oracle Corporation, Java HotSpot(TM) Client VM

Dataset consistency test:
[DELETED REFERENCED] {Way id=-38634 version=0 VT nodes=[{Node id=-38635 version=0 MV lat=50.1382636,lon=15.2213157}, {Node id=-38637 version=0 MVD lat=50.13684480000001,lon=15.2210666}, {Node id=3107044983 version=1 MV lat=50.1368452,lon=15.2210675}, {Node id=3107044975 version=1 MV lat=50.1366159,lon=15.2252755}, {Node id=3107044971 version=1 MV lat=50.13643879999999,lon=15.228719500000002}, {Node id=3107044968 version=1 MV lat=50.13643589999999,lon=15.2297197}, {Node id=-38642 version=0 MV lat=50.1369169,lon=15.2296828}, {Node id=-38643 version=0 MV lat=50.1373955,lon=15.2296731}, {Node id=-38644 version=0 MV lat=50.1376694,lon=15.2296868}, {Node id=-38645 version=0 MV lat=50.1378077,lon=15.2297047}, {Node id=-38646 version=0 MV lat=50.13973,lon=15.2298279}, {Node id=-38647 version=0 MV lat=50.1405903,lon=15.2298868}, {Node id=-38648 version=0 MV lat=50.1406606,lon=15.2293597}, {Node id=-38649 version=0 MV lat=50.1409683,lon=15.2241084}, {Node id=-38650 version=0 MV lat=50.1411181,lon=15.2217773}, {Node id=-38635 version=0 MV lat=50.1382636,lon=15.2213157}]} refers to deleted primitive {Node id=-38637 version=0 MVD lat=50.13684480000001,lon=15.2210666}


Plugins:
- Tracer-testing (1411840389)
- geotools (30569)
- jts (30416)
- pointInfo (30470)
- reltoolbox (30587)
- reverter (30521)

Last errors/warnings:
- W: javax.imageio.IIOException: Inconsistent metadata read from stream
- W: javax.imageio.IIOException: Inconsistent metadata read from stream
- W: javax.imageio.IIOException: Inconsistent metadata read from stream
- W: javax.imageio.IIOException: Inconsistent metadata read from stream
- E: java.lang.reflect.InvocationTargetException. Přičina: org.openstreetmap.josm.data.osm.DataIntegrityProblemException: Deleted node referenced: {Way id=-38634 version=0 VT nodes=[{Node id=-38635 version=0 MV lat=50.1382636,lon=15.2213157}, {Node id=-38637 version=0 MVD lat=50.13684480000001,lon=15.2210666}, {Node id=3107044983 version=1 MV lat=50.1368452,lon=15.2210675}, {Node id=3107044975 version=1 MV lat=50.1366159,lon=15.2252755}, {Node id=3107044971 version=1 MV lat=50.13643879999999,lon=15.228719500000002}, {Node id=3107044968 version=1 MV lat=50.13643589999999,lon=15.2297197}, {Node id=-38642 version=0 MV lat=50.1369169,lon=15.2296828}, {Node id=-38643 version=0 MV lat=50.1373955,lon=15.2296731}, {Node id=-38644 version=0 MV lat=50.1376694,lon=15.2296868}, {Node id=-38645 version=0 MV lat=50.1378077,lon=15.2297047}, {Node id=-38646 version=0 MV lat=50.13973,lon=15.2298279}, {Node id=-38647 version=0 MV lat=50.1405903,lon=15.2298868}, {Node id=-38648 version=0 MV lat=50.1406606,lon=15.2293597}, {Node id=-38649 version=0 MV lat=50.1409683,lon=15.2241084}, {Node id=-38650 version=0 MV lat=50.1411181,lon=15.2217773}, {Node id=-38635 version=0 MV lat=50.1382636,lon=15.2213157}]}

java.lang.reflect.InvocationTargetException
	at java.awt.EventQueue.invokeAndWait(Unknown Source)
	at java.awt.EventQueue.invokeAndWait(Unknown Source)
	at org.openstreetmap.josm.gui.PleaseWaitRunnable.doRealRun(PleaseWaitRunnable.java:87)
	at org.openstreetmap.josm.gui.PleaseWaitRunnable.run(PleaseWaitRunnable.java:150)
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
Caused by: org.openstreetmap.josm.data.osm.DataIntegrityProblemException: Deleted node referenced: {Way id=-38634 version=0 VT nodes=[{Node id=-38635 version=0 MV lat=50.1382636,lon=15.2213157}, {Node id=-38637 version=0 MVD lat=50.13684480000001,lon=15.2210666}, {Node id=3107044983 version=1 MV lat=50.1368452,lon=15.2210675}, {Node id=3107044975 version=1 MV lat=50.1366159,lon=15.2252755}, {Node id=3107044971 version=1 MV lat=50.13643879999999,lon=15.228719500000002}, {Node id=3107044968 version=1 MV lat=50.13643589999999,lon=15.2297197}, {Node id=-38642 version=0 MV lat=50.1369169,lon=15.2296828}, {Node id=-38643 version=0 MV lat=50.1373955,lon=15.2296731}, {Node id=-38644 version=0 MV lat=50.1376694,lon=15.2296868}, {Node id=-38645 version=0 MV lat=50.1378077,lon=15.2297047}, {Node id=-38646 version=0 MV lat=50.13973,lon=15.2298279}, {Node id=-38647 version=0 MV lat=50.1405903,lon=15.2298868}, {Node id=-38648 version=0 MV lat=50.1406606,lon=15.2293597}, {Node id=-38649 version=0 MV lat=50.1409683,lon=15.2241084}, {Node id=-38650 version=0 MV lat=50.1411181,lon=15.2217773}, {Node id=-38635 version=0 MV lat=50.1382636,lon=15.2213157}]}
	at org.openstreetmap.josm.data.osm.Way.checkNodes(Way.java:593)
	at org.openstreetmap.josm.data.osm.Way.fireNodesChanged(Way.java:609)
	at org.openstreetmap.josm.data.osm.Way.setNodes(Way.java:74)
	at org.openstreetmap.josm.data.osm.Way.cloneFrom(Way.java:317)
	at org.openstreetmap.josm.command.ChangeCommand.executeCommand(ChangeCommand.java:65)
	at org.openstreetmap.josm.command.SequenceCommand.executeCommand(SequenceCommand.java:53)
	at org.openstreetmap.josm.command.SequenceCommand.executeCommand(SequenceCommand.java:53)
	at org.openstreetmap.josm.command.SequenceCommand.executeCommand(SequenceCommand.java:53)
	at org.openstreetmap.josm.data.UndoRedoHandler.addNoRedraw(UndoRedoHandler.java:43)
	at org.openstreetmap.josm.data.UndoRedoHandler.add(UndoRedoHandler.java:69)
	at org.openstreetmap.josm.plugins.tracer.modules.lpis.LpisModule$LpisTracerTask.createTracedPolygon(LpisModule.java:283)
	at org.openstreetmap.josm.plugins.tracer.modules.lpis.LpisModule$LpisTracerTask.finish(LpisModule.java:158)
	at org.openstreetmap.josm.gui.PleaseWaitRunnable$1.run(PleaseWaitRunnable.java:90)
	at java.awt.event.InvocationEvent.dispatch(Unknown Source)
	at java.awt.EventQueue.dispatchEventImpl(Unknown Source)
	at java.awt.EventQueue.access$200(Unknown Source)
	at java.awt.EventQueue$3.run(Unknown Source)
	at java.awt.EventQueue$3.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.security.ProtectionDomain$1.doIntersectionPrivilege(Unknown Source)
	at java.awt.EventQueue.dispatchEvent(Unknown Source)
	at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
	at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
	at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
	at java.awt.WaitDispatchSupport$2.run(Unknown Source)
	at java.awt.WaitDispatchSupport$4.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.awt.WaitDispatchSupport.enter(Unknown Source)
	at java.awt.Dialog.show(Unknown Source)
	at java.awt.Component.show(Unknown Source)
	at java.awt.Component.setVisible(Unknown Source)
	at java.awt.Window.setVisible(Unknown Source)
	at java.awt.Dialog.setVisible(Unknown Source)
	at org.openstreetmap.josm.gui.progress.PleaseWaitProgressMonitor$4.run(PleaseWaitProgressMonitor.java:172)
	at java.awt.event.InvocationEvent.dispatch(Unknown Source)
	at java.awt.EventQueue.dispatchEventImpl(Unknown Source)
	at java.awt.EventQueue.access$200(Unknown Source)
	at java.awt.EventQueue$3.run(Unknown Source)
	at java.awt.EventQueue$3.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.security.ProtectionDomain$1.doIntersectionPrivilege(Unknown Source)
	at java.awt.EventQueue.dispatchEvent(Unknown Source)
	at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
	at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
	at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
	at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
	at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
	at java.awt.EventDispatchThread.run(Unknown Source)

Attachments (1)

del_node_ref.png (33.2 KB) - added by Mkyral 6 years ago.
Way without nodes

Download all attachments as: .zip

Change History (21)

comment:1 Changed 6 years ago by stoecker

Component: CorePlugin tracer

A first idea who may be responsible for similar errors :-)

comment:2 Changed 6 years ago by stoecker

Summary: pád JOSMDeleted node referenced.

See #6529.

comment:3 Changed 6 years ago by Mkyral

Hi,
I'm working on update of the Tracer plugin. It is still in testing state. It uses another data sources and I'm trying to improve the connectWays class which connects the traced way to other ways (and solve overlaping of ways of the same type - building, landuse)

I know about this issue, but I have problem to reproduce it.

Repository is on https://github.com/mkyral/josm-tracer
Plugin is in list as Tracer-testing.

comment:4 Changed 6 years ago by Don-vip

What's the difference with "tracer2"? I'd like to have a single tracer plugin, we currently have 3: tracer, tracer-testing and tracer2.

comment:5 Changed 6 years ago by Mkyral

I think, this is not a right place for discussion, Josm-dev will be better place.

In short. Tracer-testing is an testing version of improved original Tracer plugin and I hope to replace the original Tracer plugin with the new one some day - after known issues will be fixed and TODOs finished.

Main changes:

*) Based on original Tracer
*) The updated connectWays class from Tracer2 plugin was adopted (and later several times rewrited ;-) )
*) Configuration dialog adopted from Tracer2 plugin
*) Added support for modules. Currently Classic (original), RUIAN (digitalized buildings from Czech cadastral department), RUIAN-Lands (digitalized lands - experimental) and LPIS (digitalized lands from Czech Ministry of Agriculture). Add Tracer2 module should be not problem.
*) In configuration you can choose, which modules will be active (and in which order)
*) You can have more modules active and switch between them by simple hint "T" key. First "T" - enter mode and enable last active module. Next press of "T" - switch to next module. If on last active module, the first one is used.

I need to update readme on Github (and finally make a wiki page on JOSM wiki). Development, testing and discussion is mainly on Talk-cz@.

If you are interested, discussion and help is welcome.

comment:6 Changed 6 years ago by Don-vip

OK thanks. Well when you manage to finish this plugin, I'd be happy to replacer tracer plugin by this one, and deprecate tracer2 as well, to keep things simple. That's all I wanted to know :)

comment:7 Changed 6 years ago by Mkyral

OK ;-)

Back to the issue. I've contacted the author. He was able to reproduce the issue several times on day of report (in new instance of JOSM). But when I tried, I was not able to reproduce. Also when author tried next day, issue just gone :-(

I have no idea what is wrong and how to fix it. Is the issue in Tracer or in JOSM? Sometimes JOSM did not recognise that data are inconsistent and upload fails with "missing placeholder" issue. The only way, how to solve it, is save the work to the file, manually edit them, load back to JOSM, fix corrupted areas and upload to OSM.

But this is not a way for the normal user (who even does not know XML file structure). In ideal case, this should never happen, but:

  1. Could be the Data consistency check run automatically before upload to OSM?
  2. Could be the Data consistency check extended to allow solving of the data inconsistency? Eg. simply delete incorrect reference(s) and show user on which ways the issue happen so he can correct them.
  3. When inconsistent file is loaded into the JSOM, it always finish on first error. Will be much better to upload the whole file and show all data inconsistency, not just the first one. And then allow user to fix them.

comment:8 Changed 6 years ago by stoecker

This issue happens, when you don't handle the arguments passed to Command actions properly. Problem is, that the exception happens not at the same place, but can be delayed, or only caused when some actions (e.g. undo) are used. Not so easy to find.

Getting it reproducible is a major step.

comment:9 Changed 6 years ago by stoecker

I'm not sure if we want to "fix" such issues automatically. That can cause much more trouble when the data then is uploaded.

comment:10 in reply to:  9 ; Changed 6 years ago by Mkyral

Replying to stoecker:

I'm not sure if we want to "fix" such issues automatically. That can cause much more trouble when the data then is uploaded.


Well, at least consistency check before upload could be done. When the "missing placeholder issue" happen, the result is a lot of isolated nodes uploaded to OSM.

comment:11 in reply to:  10 ; Changed 6 years ago by stoecker

Replying to Mkyral:

Replying to stoecker:

I'm not sure if we want to "fix" such issues automatically. That can cause much more trouble when the data then is uploaded.


Well, at least consistency check before upload could be done. When the "missing placeholder issue" happen, the result is a lot of isolated nodes uploaded to OSM.

I was convinced we have an integrity check before upload. Either I'm totally off the track or this exception should be the result of this check.

comment:12 in reply to:  8 ; Changed 6 years ago by Mkyral

Replying to stoecker:

This issue happens, when you don't handle the arguments passed to Command actions properly. Problem is, that the exception happens not at the same place, but can be delayed, or only caused when some actions (e.g. undo) are used. Not so easy to find.

Getting it reproducible is a major step.


I think, the little delay could be the issue. Especially when two ways sharing nodes and tries to connect to the same third way. Is there a way, how to ensure that there are no Commands to apply?

comment:13 in reply to:  11 ; Changed 6 years ago by Mkyral

Replying to stoecker:

Replying to Mkyral:

Replying to stoecker:

I'm not sure if we want to "fix" such issues automatically. That can cause much more trouble when the data then is uploaded.


Well, at least consistency check before upload could be done. When the "missing placeholder issue" happen, the result is a lot of isolated nodes uploaded to OSM.

I was convinced we have an integrity check before upload. Either I'm totally off the track or this exception should be the result of this check.


Then there is something wrong, when it is possible to start upload that will fail on the "missing placeholder" issue. It even happen to me, that consistency check says, all is consistent, but save and reload failed on inconsistency. Unfortunately I can't reproduce it.

comment:14 in reply to:  13 Changed 6 years ago by stoecker

Then there is something wrong, when it is possible to start upload that will fail on the "missing placeholder" issue. It even happen to me, that consistency check says, all is consistent, but save and reload failed on inconsistency. Unfortunately I can't reproduce it.

That should happen only when something was changed on the server inbetween. This is a different issue. This here is an inconsistency in the JOSM dataset which is caused by a malfunction somewhere in the editing process.

comment:15 in reply to:  12 Changed 6 years ago by Mkyral

Replying to Mkyral:

Replying to stoecker:

This issue happens, when you don't handle the arguments passed to Command actions properly. Problem is, that the exception happens not at the same place, but can be delayed, or only caused when some actions (e.g. undo) are used. Not so easy to find.

Getting it reproducible is a major step.


I think, the little delay could be the issue. Especially when two ways sharing nodes and tries to connect to the same third way. Is there a way, how to ensure that there are no Commands to apply?


Ou, I've read one more time. You mean delay between when it happen and when the exception happen.

In Tracer, it happen when I'm apply the changes.

How Tracer works:

  • User click inside of the object to trace (land in this case)
  • Tracer sent request to the server and receive geometry and additional attributes
  • New nodes and way according to traced geometry is created (all commands are stored in the List, are not applied to the dataset yet)
  • Then Tracer looks for an existing way with landuse key. If it is found, the geometry and attributes of the existing way is updated
  • Next Tracer checks close ways with landuse keys that shares the same border and tries to connect them (merge common nodes)
  • Finally, if some way with landuse key overlaps the traced way, overlaping is solved

All this is stored in the List of commands (create, delete, update node and create, update, delete way) and at the end, all is applied.

The problem could be threads - one thread Traces one way and the next thread is started before previous thread finished completely. In such way it is possible, that second thread works with nodes that are removed in first thread.

I'm not a java programmer and know almost nothing about threads. Martin Svec helped me with this a lot, but there still could be some bug.

comment:16 Changed 6 years ago by stoecker

That sounds like a possible reason. Java has different methods to synchronize, but I'm no expect here either. I know Perl a pthreads for C better. Also here you need to be in sync between main data set and the temporary modified data set. Not trivial.

A possible suggestion would be to add consistency checks when joining the threaded results and restart a process when conflicts occur (passing the changed state as argument).

comment:17 in reply to:  16 Changed 6 years ago by Mkyral

Replying to stoecker:

That sounds like a possible reason. Java has different methods to synchronize, but I'm no expect here either. I know Perl a pthreads for C better. Also here you need to be in sync between main data set and the temporary modified data set. Not trivial.

A possible suggestion would be to add consistency checks when joining the threaded results and restart a process when conflicts occur (passing the changed state as argument).


It happen again.

Today I've traced a lot of lands, no exception happen. Then I run validation and started upload to OSM (forgot about the consistency check :-( ). All went good, but near to end, OSM API returned http 400 code (missing placeholder issue).

So I have inserted a new layer, download data from server and tested. This time, I got the "Deleted node referenced" exception. And even in newly started JOSM, still the same exception.

So good think is, I have a test case, so I should be able to find the reason and fix it. I hope will not disappear again.

But maybe this is not the last issue and I would like to handle the exception correctly. But when exception happen, the result is a way with original shape, but without nodes. See attachement. I can't do undo, there are no command in stack. Is there some Savepoint/Rollback mechanism? Or how I can handle this correctly?


Changed 6 years ago by Mkyral

Attachment: del_node_ref.png added

Way without nodes

comment:18 in reply to:  2 Changed 4 years ago by skyper

Replying to stoecker:

See #6529.

Yes, probably this core bug.

comment:19 Changed 13 months ago by Don-vip

Summary: Deleted node referenced.DataIntegrityProblemException: Deleted node referenced

comment:20 Changed 8 days ago by GerdP

Resolution: duplicate
Status: newclosed

Closed as duplicate of #19438.

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain team.
as The resolution will be set.
The resolution will be deleted.

Add Comment


E-mail address and name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.