Allow autofix for duplicate nodes where possible

[Cartographer10]

I would like to see a tool in JOSM that can merge duplicate nodes into one.

Usecase: I had the situation where I had a way in JOSM. From a shape file, I copied another way with (95%+) matching nodes with the already present way. I had to manually merge them by selecting duplicated nodes and hit M, and move on to the following pair. I would like an automatic tool for that.

[GerdP]

Try Replace Geometry from utilsplugin2 : ​https://wiki.openstreetmap.org/wiki/JOSM/Plugins/utilsplugin2

[Cartographer10]

Part of the geometry is new so there is nothing to replace. See the image below as example. The red selected way is copied from the shape file layer and I need to merge the nodes with the nodes of the already present, grey ways/nodes.

​https://i.imgur.com/f0qcZCQ.png

[GerdP]

Did you try it?

[skyper]

Doesn't validator have an autofix for that? Select the two ways, run validator, select all duplicate nodes warnings and click on fix.

Contour merge plugin should work if you make closed ways in advance.

[Cartographer10]

Replying to GerdP:

Did you try it?

Yes, Just tried. I copied the new way to the active layer but I need to select another way to get replaced but there is nothing to replace because the geometry is new.

Replying to skyper:

Doesn't validator have an autofix for that? Select the two ways, run validator, select all duplicate nodes warnings and click on fix.

Contour merge plugin should work if you make closed ways in advance.

There is no fix from the validator. When I select all nodes from the warning, the fix button stays inactive.
I can't get the contour plugin to work. But on the image above, I need to keep all geometry but with merged ways.

[GerdP]

Seems we don't understand what data you have. Is any of the ways already in OSM?

[Cartographer10]

Replying to GerdP:

Seems we don't understand what data you have. Is any of the ways already in OSM?

A sorry for the misunderstand then. In this image: ​https://i.imgur.com/f0qcZCQ.png , the grey lines are closed ways with the tag place=neighbourhood (already present in OSM). I have a shape file with all other neighbourhoods, even the one not in OSM yet. I load the shape file in JOSM ( it loads into a seperate data layer) and I copy one of the closed ways from the shape file layer to the OSM data layer. The nodes match perfectly, hence, the duplicates. I just need to merge all the duplicate nodes so that they ways are attached to eachother. That proces of merging the duplicates is what I want to do automatically.

Does that make the situation clear?

[GerdP]

Select the OSM way and your copied way and use Replace Geometry

[Cartographer10]

Replying to GerdP:

Select the OSM way and your copied way and use Replace Geometry

Both ways need to stay and my new way is matching with multiple ways (see image). I just need to merge the common nodes of the new way with the already existing nodes on the same location.

[GerdP]

Hm, if validator doesn't find duplicate nodes they are probably not exactly at the same place. If you use Replace Geometry and and then copy the way again there should be exact duplicates and validator will find them.

[Cartographer10]

Replying to GerdP:

Hm, if validator doesn't find duplicate nodes they are probably not exactly at the same place. If you use Replace Geometry and and then copy the way again there should be exact duplicates and validator will find them.

Ow, the validator does find them, ​https://i.imgur.com/LX8xYVN.png . It is just that the "fix (herstellen)" button does not get activated so it can't fix it.

[GerdP]

OK, try to add the tags of the old way to the new way.

[Cartographer10]

Replying to GerdP:

OK, try to add the tags of the old way to the new way.

I added the name and place tag, just like the already present ways have but the fix button still stays inactive.

[GerdP]

Do you have the OSM way inside a download area?

[Cartographer10]

Replying to GerdP:

Do you have the OSM way inside a download area?

I download the needed data via overpass, not by selecting a rectangle to download all data from. In the view menu, you have the button for showing a hatch around your downloaded area, with that enabled, no hatch shows up so I suppose it kind of is inside the downloaded area.

[GerdP]

If you download an area with overpass you should clear the text box with the overpass statements. If not, JOSM assumes that the data is not complete and thus shows no hatched area and thus no fix for duplicate nodes.

[Cartographer10]

Replying to GerdP:

If you download an area with overpass you should clear the text box with the overpass statements. If not, JOSM assumes that the data is not complete and thus shows no hatched area and thus no fix for duplicate nodes.

I see. Well, I use overpass because I am working in a large area and I don't want all data because it clutters the view. When I clear the overpass statement, it asks me to download all data in the area, which is way to much.

[GerdP]

You can try the "download along way" action. If you want to work without a download area you have to merge manually and - of course - make sure that all parents of the merged nodes are downloaded before.

[GerdP]

Wait. I just wonder if there is any risk to corrupt other - not downloaded - data when you merge an old node with a new one and both have the same position. Maybe this is a special case where the autofix could be enabled? The action should always delete the new node and maybe modify parents which are eiher also new or already modified.

[Cartographer10]

Replying to GerdP:

Wait. I just wonder if there is any risk to corrupt other - not downloaded - data when you merge an old node with a new one and both have the same position. Maybe this is a special case where the autofix could be enabled? The action should always delete the new node and maybe modify parents which are eiher also new or already modified.

Yeah, I was thinking the same. I am not sure how the validator works but in this case, you have a new node and an already existing node. If you just merge with the already existing node, nothing should get corrupt indeed.

Ofcouse, because I download via overpass, I have to make sure all duplicate nodes are downloaded but that is outside the scope of this validation script because that one is for merging the downloaded duplicate nodes.

[GerdP]

So, maybe change the subject to "Allow autofix for duplicate nodes with new nodes"?

[Cartographer10]

Replying to GerdP:

So, maybe change the subject to "Allow autofix for duplicate nodes with new nodes"?

What about "Allow autofix for duplicate nodes for overpass downloaded area" since the key here is that the autofix works, only not when specific data is downloaded via overpass. Not specifically related to merging old with new nodes.

[GerdP]

Not really. It is related to the fact that autofix is always refused if a node is not inside a download area. You can have a download area with overpass download if you don't use a filter, and you can have no download area if you use e.g. "Download object".

[Cartographer10]

Replying to GerdP:

Not really. It is related to the fact that autofix is always refused if a node is not inside a download area. You can have a download area with overpass download if you don't use a filter, and you can have no download area if you use e.g. "Download object".

Isn't "Allow autofix for duplicate nodes outside downloaded areas" then more accurate?

[GerdP]

I don't want to change that. I just want to allow the autofix if it possible to remove only new nodes. The phrase "outside downloaded area" is problematic when there is no download area at all.
We can say "Allow autofix for duplicate nodes where possible"

[Cartographer10]

Replying to GerdP:

I don't want to change that. I just want to allow the autofix if it possible to remove only new nodes. The phrase "outside downloaded area" is problematic when there is no download area at all.
We can say "Allow autofix for duplicate nodes where possible"

Now I understand the situation. I changed the title to the above.

[skyper]

I do not see any problem in merging untagged nodes with id:0 onto existing ones (id > 0). With tags, it is ok, as long as the old node has the same tags among its tags. Only if the new node has a tag which the old node does not have, user's action is needed.
Same evaluation should be made for memberships but be careful to take differences in roles into account.

@Cartographer10:
For now, you could take a look at conflation plugin, where it should be possible to define the nodes and merging is only one action, but maybe the act of defining the nodes is too much manual work and therefore no gain in your case.

[GerdP]

@skyper: OK, it's probably difficult to define "possible". I would limit it to untagged new nodes (id:0) which are not members in any old relation (I assume that's what we have here).
If this requires more than two or three lines of code I would stop and prefer to add a new action to download minimal (1x1 mm) areas around selected nodes, presuming that this would also solve the problem.

[skyper]

Oh, I was too fast. We need to look at parent ways, too. In this case it does not make a difference but we do not want to create invalid ways or relations or do we?
E.g. if both nodes are part of the same way or both have one parent way member of the same relation it is already a problem and that can even happen with parent ways with id:0

I did not understand if you plan to still separate cases between inside and outside download area but the only general same rule is that a new node with only new parents (ways + relations) and no parent way with membership nor having the other node as child (in nodelist), too.

Hope I got this right now, as I did rewrite it three times, now.

With complete data it gets more flexible concerning memberships and id (new <> old) of the parent way.

[GerdP]

Hm, not sure where this is going. Cartographer10 says he can do what he wants when he selects the nodes manually and presses M. I don't want to change the code in class MergeNodesAction. I would just want to enable the autofix if that calls the same method as M does. If M silently merges nodes which should not be merged we have a different problem.

[skyper]

The huge difference is that I need to select the node manually and individually, e.g. I might take a look at the parent ways and the relations while with the autofix it gets easy to select and merge hundreds of node pairs at once without taking any glance at the spots.

Think that was the initial thought behind denying autofix outside download area , in general. Let the user create invalid ways and broken relations but do not blame validator for doing that.

[GerdP]

OK, if there is a chance that data is corrupted we should not do it. How does the conflation plugin work around this problem?

[skyper]

Conflation works in two steps:

1. defining pairs by all kinds of filters
2. conflating with either Replace Geometry or self-defined method.

In all cases, the user is responsible for the changes.

[GerdP]

What do you think about the "download 1x1 mm area around nodes" idea? See also #17776 and #11446

[skyper]

I need to take a deeper look what is allowed at the moment but I fear even inside download area, many cases could be a problem. At least all parent ways and relations are downloaded that way and validator has a chance to report problems like self-crossing, self-intersecting or overlapping ways and broken relations (if they are complete ?!) like MP with two overlapping outer.

[skyper]

Replying to GerdP:

What do you think about the "download 1x1 mm area around nodes" idea? See also #17776 and #11446

+1 for the idea in general. Though, it won`t help me as long as it is not possible to tie the drawing of download boundaries to eaach layer and have a default general setting.

I our case it would simplify the situation as we do not have to take care about the unknown. Still it would not help much cause in my opinion the problem is lying in merging without taking care about the parent objects but that is probably out of the scope of this ticket.