DataIntegrityProblemException in Test DuplicateWay

[anonymous]

What steps will reproduce the problem?

1. I hit upload button.

What is the expected result?

What happens instead?

Please provide any additional information below. Attach a screenshot if possible.

Revision:18969
Build-Date:2024-02-06 12:12:39

Identification: JOSM/1.5 (18969 en) Windows 10 64-Bit
OS Build number: Windows 10 Pro 2009 (19045)
Memory Usage: 5376 MB / 8176 MB (627 MB allocated, but free)
Java version: 17.0.10+7-LTS, Azul Systems, Inc., OpenJDK 64-Bit Server VM
Look and Feel: com.sun.java.swing.plaf.windows.WindowsLookAndFeel
Screen: \Display0 1920×1080 (scaling 1.00×1.00) \Display1 1600×900 (scaling 1.00×1.00) \Display2 1920×1080 (scaling 1.00×1.00)
Maximum Screen Size: 1920×1080
Best cursor sizes: 16×16→32×32, 32×32→32×32
System property file.encoding: Cp1252
System property sun.jnu.encoding: Cp1252
Locale info: en_US
Numbers with default locale: 1234567890 -> 1234567890
VM arguments: [-Djpackage.app-version=1.5.18969, --add-modules=java.scripting,java.sql,javafx.controls,javafx.media,javafx.swing,javafx.web, --add-exports=java.base/sun.security.action=ALL-UNNAMED, --add-exports=java.desktop/com.sun.imageio.plugins.jpeg=ALL-UNNAMED, --add-exports=java.desktop/com.sun.imageio.spi=ALL-UNNAMED, --add-opens=java.base/java.lang=ALL-UNNAMED, --add-opens=java.base/java.nio=ALL-UNNAMED, --add-opens=java.base/jdk.internal.loader=ALL-UNNAMED, --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED, --add-opens=java.desktop/javax.imageio.spi=ALL-UNNAMED, --add-opens=java.desktop/javax.swing.text.html=ALL-UNNAMED, --add-opens=java.prefs/java.util.prefs=ALL-UNNAMED, -Djpackage.app-path=%UserProfile%\AppData\Local\JOSM\JOSM.exe]

Dataset consistency test:
[NO DATASET] {Node id=-1495245 version=0 MV lat=40.87988059373646,lon=-74.73155535824597} is referenced by {Way id=-271212 version=0 V nodes=[{Node id=-1495245 version=0 MV lat=40.87988059373646,lon=-74.73155535824597}, {Node id=-1495246 version=0 MV lat=40.879934842006335,lon=-74.73142795331778}]} but not found in dataset
[NO DATASET] {Node id=-1495246 version=0 MV lat=40.879934842006335,lon=-74.73142795331778} is referenced by {Way id=-271212 version=0 V nodes=[{Node id=-1495245 version=0 MV lat=40.87988059373646,lon=-74.73155535824597}, {Node id=-1495246 version=0 MV lat=40.879934842006335,lon=-74.73142795331778}]} but not found in dataset
[SEARCH WAYS] {Way id=-271212 version=0 V nodes=[{Node id=-1495245 version=0 MV lat=40.87988059373646,lon=-74.73155535824597}, {Node id=-1495246 version=0 MV lat=40.879934842006335,lon=-74.73142795331778}]} not found using Dataset.containsWay()


Plugins:
+ FixAddresses (36126)
+ HouseNumberTaggingTool (36196)
+ OpeningHoursEditor (36196)
+ PolygonCutOut (v0.7.3)
+ SimplifyArea (36200)
+ continuosDownload (103)
+ fastdriveway (v1.0.1)
+ gridify (1606242219)
+ imagery_offset_db (36126)
+ intersection (0.0.7)
+ splinex (36126)
+ turnrestrictions (36200)

Tagging presets:
+ https://raw.githubusercontent.com/yopaseopor/traffic_signs_preset_JOSM/master/US.zip

Map paint styles:
+ https://raw.githubusercontent.com/yopaseopor/traffic_signs_style_JOSM/master/US.zip
+ https://josm.openstreetmap.de/josmfile?page=Styles/Direction&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Styles/iD&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Styles/Lane_and_Road_Attributes&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/Surface&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/TreeCrownDiameter&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Styles/MappingAccessibility&zip=1

Last errors/warnings:
- 471752.291 W: Already here java.net.SocketTimeoutException: Read timed out
- 471752.404 E: Failed to locate image 'https://www.fs.usda.gov/sites/all/themes/fs_hq/images/usfslogo.png'
- 471782.489 W: java.net.SocketTimeoutException: Read timed out. Cause: java.net.SocketTimeoutException: Read timed out
- 471782.489 W: Already here java.net.SocketTimeoutException: Read timed out
- 471782.598 E: Failed to locate image 'https://www.fs.usda.gov/sites/all/themes/fs_hq/images/usfslogo.png'
- 471825.346 E: Handled by bug report queue: org.openstreetmap.josm.data.osm.DataIntegrityProblemException: Primitive must be part of the dataset: {Node id=-1495246 version=0 MV lat=40.879934842006335,lon=-74.73142795331778}
- 471908.856 W: java.net.SocketTimeoutException: Read timed out. Cause: java.net.SocketTimeoutException: Read timed out
- 471908.856 W: Already here java.net.SocketTimeoutException: Read timed out
- 471908.967 E: Failed to locate image 'https://www.fs.usda.gov/sites/all/themes/fs_hq/images/usfslogo.png'
- 472528.389 E: Handled by bug report queue: org.openstreetmap.josm.data.osm.DataIntegrityProblemException: Primitive must be part of the dataset: {Node id=-1495246 version=0 MV lat=40.879934842006335,lon=-74.73142795331778}



=== REPORTED CRASH DATA ===
BugReportExceptionHandler#handleException:
No data collected.

Warning issued by: BugReportExceptionHandler#handleException

=== STACK TRACE ===
Thread: AWT-EventQueue-0 (24) of main
org.openstreetmap.josm.data.osm.DataIntegrityProblemException: Primitive must be part of the dataset: {Node id=-1495246 version=0 MV lat=40.879934842006335,lon=-74.73142795331778}
	at org.openstreetmap.josm.data.osm.OsmPrimitive.checkDataset(OsmPrimitive.java:198)
	at org.openstreetmap.josm.data.osm.OsmPrimitive.referrers(OsmPrimitive.java:702)
	at org.openstreetmap.josm.data.osm.OsmPrimitive.referrers(OsmPrimitive.java:723)
	at org.openstreetmap.josm.data.osm.Node.getParentWays(Node.java:366)
	at org.openstreetmap.josm.data.validation.tests.DuplicateWay.endTest(DuplicateWay.java:135)
	at org.openstreetmap.josm.data.validation.ValidationTask.realRun(ValidationTask.java:192)
	at org.openstreetmap.josm.gui.PleaseWaitRunnable.doRealRun(PleaseWaitRunnable.java:94)
	at org.openstreetmap.josm.gui.PleaseWaitRunnable.run(PleaseWaitRunnable.java:142)
	at org.openstreetmap.josm.actions.upload.ValidateUploadHook.checkUpload(ValidateUploadHook.java:61)
	at org.openstreetmap.josm.actions.UploadAction.lambda$checkPreUploadConditionsAsync$0(UploadAction.java:252)
	at java.base/java.util.stream.MatchOps$1MatchSink.accept(Unknown Source)
	at java.base/java.util.LinkedList$LLSpliterator.tryAdvance(Unknown Source)
	at java.base/java.util.stream.ReferencePipeline.forEachWithCancel(Unknown Source)
	at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(Unknown Source)
	at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
	at java.base/java.util.stream.MatchOps$MatchOp.evaluateSequential(Unknown Source)
	at java.base/java.util.stream.MatchOps$MatchOp.evaluateSequential(Unknown Source)
	at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)
	at java.base/java.util.stream.ReferencePipeline.allMatch(Unknown Source)
	at org.openstreetmap.josm.actions.UploadAction.lambda$checkPreUploadConditionsAsync$1(UploadAction.java:252)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)

[GerdP]

I thought this could be a regression of r18960 but I am not sure. How can a node be referenced by a usable way when it has dataSet=null?

[taylor.smock]

It could be one of the plugins. I've poked the person I think is the reporter on OSM JOSM (OSM World Discord).

I'm guessing that one of the plugins added a node to a way but did not add the node to a dataset (Way#isUsable just checks to see if the way or nodes are incomplete).

[GerdP]

OK, I see. I think this should be fixed in the calling code so that visit() is not called for such invalid objects.

[taylor.smock]

It sounds like they used the driveway plugin, gridify, and a spline tool (probably splinex).

[GerdP]

Of course we should try to find the cause of the problem but I wonder what to do in cases like this.
The report shows

Dataset consistency test:
[NO DATASET] {Node id=-1495245 version=0 MV lat=40.87988059373646,lon=-74.73155535824597} is referenced by {Way id=-271212 version=0 V nodes=[{Node id=-1495245 version=0 MV lat=40.87988059373646,lon=-74.73155535824597}, {Node id=-1495246 version=0 MV lat=40.879934842006335,lon=-74.73142795331778}]} but not found in dataset
[NO DATASET] {Node id=-1495246 version=0 MV lat=40.879934842006335,lon=-74.73142795331778} is referenced by {Way id=-271212 version=0 V nodes=[{Node id=-1495245 version=0 MV lat=40.87988059373646,lon=-74.73155535824597}, {Node id=-1495246 version=0 MV lat=40.879934842006335,lon=-74.73142795331778}]} but not found in dataset
[SEARCH WAYS] {Way id=-271212 version=0 V nodes=[{Node id=-1495245 version=0 MV lat=40.87988059373646,lon=-74.73155535824597}, {Node id=-1495246 version=0 MV lat=40.879934842006335,lon=-74.73142795331778}]} not found using Dataset.containsWay()

So a way that is bogus (possibly invisible?) should be uploaded and is therefore passed to the validation test(s) which probably are not able to handle that data.
I think we should always perform DatasetConsistencyTest prior to the validation tests, but I am not sure if we would be able to use the result in a sane way.
Many users would probably prefer to upload a few "bad" objects when the changeset is large and hours of work would be lost otherwise. An expert might prefer to stop the upload and fix the bad data, possibly by editing the *.osm file with a text editor.

Or is there a safe way to cleanup the bad data first?

[GerdP]

I am able to reproduce a similar problem with a sequence like this

            Way w = new Way();
            w.addNode(new Node(new LatLon(1,2)));
            w.addNode(new Node(new LatLon(2,2)));
            try {
                ds.addPrimitive(w);
            } catch (DataIntegrityProblemException e) {
                Logging.error(e);
            }

added to the UploadAction. I also see that ExtrudeAction also catches DataIntegrityProblemException because of #12870 but I've not yet found a case where this leads to invalid data. Still it looks very wrong to catch this.

[GerdP]

I've tried to reproduce the problem with the mentioned plugins (fastdriveway, gridify, and splinex). I was able to produce some crashes or invalid data problems but not exactly the above case. The ​fastdriveway may generate a SplitWayCommand and that's always a potential source for problems when the split way is member of a relation, but so far I only saw NullPointerExecptions.
Would it be a good idea to perform the DatasetConsistencyTest?

[taylor.smock]

Would it be a good idea to perform the DatasetConsistencyTest?

We could probably do it as part of the autosave functionality (default 5 minutes, IIRC, look at OsmDataLayer#autosave). And if it fails, add a log of the commands to the reported exception.