Create conflation plugin

[joshdoe]

I started a conflation plugin last year, but it languished before producing anything useful. After my recent additions to Replace Geometry from utilsplugin2, I'm taking a more simplified approach to this by taking advantage of this conflating capability. For some background read ​the OSM wiki here. Last year I checked in what I had as a plugin named conflation, but it was unusable. I've begun reworking it, so it's sort of in a usable state, at least where it can generate matches based on distance, draw arrows between them, and display a list of matches that can be clicked to zoom/center and select. For now the Replace Geometry command has to be run manually. There are numerous other problems, like using a Netbeans form, not listening to dataset changes, etc., but it's a start.

[joshdoe]

In [o27751] I've made changes so this plugin is somewhat usable. No JAR provided yet since it's highly experimental. To try it out compile, install, and start plugin with a data layer open. Select some objects you want to replace and click freeze under Source, then select some that will be the replacements and click freeze under Target. Click Next. Notice the cost measure is currently just distance. Click Conflate. Click one of the matches to zoom and center to the pair, and click Replace Geometry. Note that there is still a lot of work to do, but I've tried it for upgrading GNIS park nodes to county polygons and it's worked pretty well.

[joshdoe]

Just a note that I've refactored this quite a bit, where a ToggleDialog is now used. The plugin list isn't updating right now (because we're in stabilization phase?), but you can get it from SVN. I sent a ​message soliciting feedback on josm-dev@, and you can read more about the plugin and see a screenshot ​on the OSM wiki.

[akks]

The algorithm and the whole plugin is really impressing!

Some smaller bugs are shortcut confilcts we tried to eliminate :) . With latest published JAR there are warnings

Silent shortcut conflict: 'Conflation' moved by 'MapDust' to 'Ctrl+Alt+F1'.
Silent shortcut conflict: 'conflation:configure' moved by 'reltoolbox:find' to 'Ctrl+Alt+F2'.
Silent shortcut conflict: 'conflation:replace' moved by 'reltoolbox:find' to 'Ctrl+Alt+F3'.

I also could not recompile it with ant dist - it can not find ReplaceGeometryAction.

[joshdoe]

Replying to akks:

The algorithm and the whole plugin is really impressing!

Thanks, but it has a way to go before being Really Useful.

Some smaller bugs are shortcut confilcts we tried to eliminate :) . With latest published JAR there are warnings

Silent shortcut conflict: 'Conflation' moved by 'MapDust' to 'Ctrl+Alt+F1'.
Silent shortcut conflict: 'conflation:configure' moved by 'reltoolbox:find' to 'Ctrl+Alt+F2'.
Silent shortcut conflict: 'conflation:replace' moved by 'reltoolbox:find' to 'Ctrl+Alt+F3'.

I honestly wasn't even thinking about proper shortcuts at this point, I just copied and pasted the code from somewhere else. The code will be in flux, so I didn't want to settle on specific actions and shortcuts.

I also could not recompile it with ant dist - it can not find ReplaceGeometryAction.

It depends on utilsplugin2, and some recent changes from there as well. So make sure that's updated first.

[akks]

Replying to joshdoe:

I also could not recompile it with ant dist - it can not find ReplaceGeometryAction.

It depends on utilsplugin2, and some recent changes from there as well. So make sure that's updated first.

I did full svn update (core+plugins), ant dist , ant install in utilsplugin2 folder. But Conflation still can not compile... Maybe some problem with environment variables or build.xml?

16. S. It seems I found the problem : move line

<property name="utilsplugin2" location="${plugin.dist.dir}/utilsplugin2.jar"/>

after

<import file="../build-common.xml"/> :)

[joshdoe]

I've now refactored things to use the Java Conflation Suite (JCS), and no longer need the Hungarian or assignment methods. This also significantly expands the possibilities for generating matches, such as based on Hausdorff distance, overlap, symmetric difference, and more (see implementing classes of ​FeatureMatcher). Now the hard part will be creating a user friendly GUI for setting the myriad of matching possibilities.

After a little more cleanup, I'll commit the code which will simply use centroid distance as it does now.

[Cobra]

Replacing address nodes with existing buildings works fine. But doing so needs definitely too many clicks: first select the nodes, freeze, select the buildings, freeze, select an entry, click replace, click apply in the tag/relation conflicts window, select the next entry, click replace ...

I'd like to have a select all or replace all button/function.

Also, is there a possibility to suppress the warning about any conflicts regarding tags/relations?

I'm planning to use this plugin for at least 268 addr:housenumber nodes, so I don't want to do too many steps for this.

[joshdoe]

Replying to Cobra:

Replacing address nodes with existing buildings works fine. But doing so needs definitely too many clicks: first select the nodes, freeze, select the buildings, freeze, select an entry, click replace, click apply in the tag/relation conflicts window, select the next entry, click replace ...

I'd like to have a select all or replace all button/function.

Also, is there a possibility to suppress the warning about any conflicts regarding tags/relations?

I'm planning to use this plugin for at least 268 addr:housenumber nodes, so I don't want to do too many steps for this.

Agreed that this should be implemented. I already have this on my TODO list, but knowing someone will find it immediately useful I'll implement this next (by this weekend likely).

Any suggestions as to how to implement? Obviously I'll use the standard multiple-selection behavior for lists. Just use the same button? How to determine whether to always ask about conflicts are to always use the reference tags? Maybe a right-click menu with "Use subject tags, subject geometry" (i.e. ignore/remove match), "Use subject tags, reference geometry", "Use reference tags, subject geometry", or "Use reference tags, reference geometry"?

It could be a little more complicated, because you could want to ignore/remove all subject tags and only use reference tags, or you could want to merge tags with one or the other taking priority for each key, and then you might want to concatenate values together. Which would you be using?

[Cobra]

The reason for ignoring/suppressing the conflict window is simple: the nodes are members of relations, the buildings aren't yet. Is it possible for a plugin to access the conflict dialog and add a "do this for any future conflicts" checkbox?

[joshdoe]

I've tweaked the behavior of ReplaceGeometry in utilsplugin2 so it will only show the dialog if multiple values exist for the same key, or if both objects have relation memberships. This should significantly reduce the number of times this dialog will be shown.

I've also made several other enhancements to the conflation plugin which you can see ​here, most notably for you being that you can select multiple matches and merge them at once. It seems to work well for the test case you provided (#7616, 7616.osm). Select and freeze ways as reference, select and freeze nodes as subject, OK, click in conflation dialog list, select all (Ctrl+A), then simply click Conflate. Still lots of clicks, but I'm not sure how to reduce this further.

I've just released v0.1.2 of the conflation plugin and [o28350] of utilsplugin2, which should be available in JOSM in the next 10 minutes. Please let me know how it goes and continue to provide feedback.

I'm closing this ticket since I believe 0.1.2 is actually a somewhat useful version of this plugin. Please open additional tickets for bugs and enhancement requests.

[Cobra]

Now that's what I wanted. perfect!

If you want to implement more "magic" you could try to find the corresponding building without the need to select address node and building first. So one could select all address nodes and then click "automagically find subjects" and all selected address nodes would get associated to the biggest nearest building, ready for conflation. (no, this isn't a feature request)

[skyper]

Thanks for this wonderful plugin