Sync remaining maps from EII and JOSM

[stoecker]

The image list at ImageryCompare contains still a number of red entries. They should either be added to JOSM's wiki or marked to be ignored.

If done we can automate add results into the automated build checking process.

See also these tickets:

• #10760 - original ticket for sync, contains probably some valid information for ignores
• #11022 - similar request
• #12313 - wontfix for "best" implementation

[Don-vip]

Milestone renamed

[Klumbumbus]

What do the = 1 and = 3 behind the ignore entries mean? https://josm.openstreetmap.de/browser/josm/trunk/scripts/sync_editor_imagery_index.groovy?rev=9653#L97

[Klumbumbus]

And what is about the entries at the end of ImageryCompare?

+++ Obsolete skip entry:

I don't see a common schema there.

[Klumbumbus]

EII uses: Hike & Bike - ​http://toolserver.org/tiles/hikebike/{zoom}/{x}/{y}.png
JOSM uses: Hike & Bike - ​http://{switch:a,b,c}.tiles.wmflabs.org/hikebike/{zoom}/{x}/{y}.png

Both work. What's the difference? Which is "better"?

[stoecker]

What do the = 1 and = 3 behind the ignore entries mean? https://josm.openstreetmap.de/browser/josm/trunk/scripts/sync_editor_imagery_index.groovy?rev=9653#L97

The number of lines to skip.

And what is about the entries at the end of ImageryCompare?

+++ Obsolete skip entry:

I don't see a common schema there.

There are entries to ignore differences in the script, but nothing was ignored. That's an indication that the entry got changed in between. E.g. an Country code was added or something alike or in best case the difference has been resolved.

Either correct the text of the ignore (when a similar unignored entry exists which is not better than JOSM's entry) or simply remove it (when it has been resolved).

EII uses: Hike & Bike - ​​http://toolserver.org/tiles/hikebike/{zoom}/{x}/{y}.png
JOSM uses: Hike & Bike - ​​http://{switch:a,b,c}.tiles.wmflabs.org/hikebike/{zoom}/{x}/{y}.png

Both work. What's the difference? Which is "better"?

Toolserver is an very old address (go to Hike&Bike map and see what it uses), maybe also using old data. That's why it's better to fix it on both sides instead of using the ignores, so the question needs to be asked only once: But I personally did't want to make EII submit requests for data - in this point I'm petulant - I'll work with them, but not for them.

[Klumbumbus]

Replying to stoecker:

What do the = 1 and = 3 behind the ignore entries mean? https://josm.openstreetmap.de/browser/josm/trunk/scripts/sync_editor_imagery_index.groovy?rev=9653#L97

The number of lines to skip.

Which lines?

[Klumbumbus]

Could we please have one more category "updates which need to be solved in EII lists" so we can better distinguish all the entries?

Does the current ignore list only include entries with minor differences (like different name or z instead of zoom) or does it also contain entries which are outdated in EII and needs to be fixed there?

[stoecker]

Which lines?

Of the displayed text. It's a simple string compare. E.g. the "name differs" entry consist of a line with URL and two lines with changed data. That's why the count is 3.

Could we please have one more category "updates which need to be solved in EII lists" so we can better distinguish all the entries?

Does the current ignore list only include entries with minor differences (like different name or z instead of zoom) or does it also contain entries which are outdated in EII and needs to be fixed there?

The ignores are ATM all relative to JOSM. Usually that means fixing the entry in EII (or sometimes in JOSM?) would make them obsolete. Where EII uses a TMS instead of the original WMS we can add these as mirror in our config.

[Klumbumbus]

Replying to stoecker:

Which lines?

Of the displayed text. It's a simple string compare. E.g. the "name differs" entry consist of a line with URL and two lines with changed data. That's why the count is 3.

OK, now I understand.

Could we please have one more category "updates which need to be solved in EII lists" so we can better distinguish all the entries?

Does the current ignore list only include entries with minor differences (like different name or z instead of zoom) or does it also contain entries which are outdated in EII and needs to be fixed there?

The ignores are ATM all relative to JOSM. Usually that means fixing the entry in EII (or sometimes in JOSM?) would make them obsolete. Where EII uses a TMS instead of the original WMS we can add these as mirror in our config.

OK, it would be usefull to have 2 ignore lists, one for the entries which are actually correct on both sides (slightly different name or z vs. zoom) and one for the entries which should be fixed at the EII side. This way we can sort the entries without the need to fix them at EII.

Also it would be useful if the ignore lists could be maintained in the wiki. This way only a wiki edit instead of a svn commit is required to update the ignore lists.

[Klumbumbus]

In 11234/josm:

see #12706 - update EII ignore list

[stoecker]

In 11238/josm:

see #12706 - move skiplist to wiki, add a second color code for ignores

[stoecker]

In 11239/josm:

see #12706 - move skiplist to wiki, add a second color code for ignores - better color

[stoecker]

Replying to Klumbumbus:

OK, it would be usefull to have 2 ignore lists, one for the entries which are actually correct on both sides (slightly different name or z vs. zoom) and one for the entries which should be fixed at the EII side. This way we can sort the entries without the need to fix them at EII.

Also it would be useful if the ignore lists could be maintained in the wiki. This way only a wiki edit instead of a svn commit is required to update the ignore lists.

Done.

[Klumbumbus]

Replying to stoecker:

Done.

Great!

[stoecker]

In 11242/josm:

see #12706 - change regexp, so that «blancos for copy» don't produce broken skip entry :-)

[Klumbumbus]

In 11245/josm:

see #12706 - use a dark yellow color (two greens are too similar)

[Klumbumbus]

In 11246/josm:

see #12706 - typos

[Klumbumbus]

I don't understand this change. https://josm.openstreetmap.de/wiki/Maps/Estonia?action=diff&version=5 Isn't Mercator the "normal" projection? The previous URLs with -geo support EPSG4326 and work in Merkator. (I checked for Estonia Basemap (Maaamet))

[Klumbumbus]

If https is available should that be preferred over http? (e.g. + OpenStreetMap Carto (Standard layer) - https://{switch:a,b,c}.tile.openstreetmap.org/{zoom}/{x}/{y}.png)

[stoecker]

@bastiK:

Did you see comment 22?

[stoecker]

I'd say https should be preferred nowadays.

[bastiK]

Replying to Klumbumbus:

I don't understand this change. https://josm.openstreetmap.de/wiki/Maps/Estonia?action=diff&version=5 Isn't Mercator the "normal" projection? The previous URLs with -geo support EPSG4326 and work in Merkator. (I checked for Estonia Basemap (Maaamet))

I changed it because displaying stretched WGS 84 tiles on a Mercator map is a bit of a hack.

Nevertheless, it works fine for this server, so feel free to switch back.

[Klumbumbus]

OK, I switched back and moved the EPSG:3301 tiles into <mirror>

[stoecker]

Thanks a lot Klumbumbus for the cleanups!

JOSM's side of fixes is now (mainly?) done for the moment, so #12313 could be reconsidered. I'm still not really convinced of the "best" flag, but I can see some benefits. Opinions?

And also maybe we could improve the check scripts to detect differences in the polygons between EII and JOSM (and glitches in our own database like unclosed rings) or even other values?

Also now we could add a build check for red entries in the list. Should we?

[Klumbumbus]

Replying to stoecker:

Thanks a lot Klumbumbus for the cleanups!

JOSM's side of fixes is now (mainly?) done for the moment,

I checked all blue entries if they still deliver tiles and found some, which don't. I will post the list in an own ticket for review.

so #12313 could be reconsidered. I'm still not really convinced of the "best" flag, but I can see some benefits. Opinions?

I also think it's not the best concept. However it would be good to have this flag in our database too, the reasons given in #12313 are valid . Maybe in JOSM we could display a little ​star in front of the imagery name when it has the best flag.

And also maybe we could improve the check scripts to detect differences in the polygons between EII and JOSM (and glitches in our own database like unclosed rings) or even other values?

Would be good.

Also now we could add a build check for red entries in the list. Should we?

Would be good too. However only for the current checks. Possible polygon check should not yet be included. I guess fixing the polygons will be a longer task.

[stoecker]

In 11410/josm:

see #12706 - show mismatching shape information

[Klumbumbus]

Uh, thats a lot. Is there a way to automatically fix these 236 unclosed JOSM shapes by a script? Doing this by hand looks like a lot of boring work.

There seems to be an error in the shape comparison. E.g. imagico.de OSM images for mapping: Eastern Devon Island coast (Canada) is reported at ImageryCompare, but it has the same shape in maps and in the ELI xml output (in the ELI source file there are some more decimal places, though).

[stoecker]

Replying to Klumbumbus:

Uh, thats a lot. Is there a way to automatically fix these 236 unclosed JOSM shapes by a script? Doing this by hand looks like a lot of boring work.

That would be probably more work that manual. :-) I started yesterday but overlooked that the list is unsorted. I was so happy that I fixed 3 whole countries... The fix is usually simple, copy the first point to the end as well.

There seems to be an error in the shape comparison. E.g. imagico.de OSM images for mapping: Eastern Devon Island coast (Canada) is reported at ImageryCompare, but it has the same shape in maps and in the ELI xml output (in the ELI source file there are some more decimal places, though).

There is no approximate check. It checks for equality, so more decimal places means "not equal".

[stoecker]

In 11411/josm:

see #12706 - improve shape difference display, ignore shapes equivalent to bounds

[Klumbumbus]

Replying to stoecker:

There is no approximate check. It checks for equality, so more decimal places means "not equal".

I thought you use the ELI XML output for the comparison, but you use the json file. The ELI XML file (where I took some shapes from, e.g. all the imagico entries) cuts the last decimal places. So thats the reason for the difference.

BTW geojson is now the default format on ELI and json only legacy. I don't know if it makes sense to change our script.

[stoecker]

In 11412/josm:

see #12706 - switch to geojson

[stoecker]

No sense using an old format when the switch is relatively easy.

[stoecker]

For the coordinates we probably need a better equals() function taking a certain epsilon into account :-)

[Klumbumbus]

When does it report No EII shape:? When there are only 4 different points?

[stoecker]

In 11413/josm:

see #12706 - better equals function and better difference description

[stoecker]

Now it's much better - less reports and better details.

Also removed most of the unclosed polygons.

Replying to Klumbumbus:

When does it report No EII shape:? When there are only 4 different points?

I added a special detection for this bounding box similar shapes. Maybe that's wrong. Which entry?

[stoecker]

In 11414/josm:

see #12706 - small cleanup

[Klumbumbus]

All the imagico entries

[stoecker]

In 11415/josm:

see #12706 - fix 5 point shape handling

[stoecker]

Hmm, is it useful to add a check for the remaining keyword elements? Polygons have been less work than I expected it to be. I thought it would be a nightmare.

[Klumbumbus]

Replying to stoecker:

Hmm, is it useful to add a check for the remaining keyword elements?

• icon and projectons would be useful
• id seems not useful
• maybe a check for the josm database if bounds and shape fit together
• copyright/attribution information seems important but I think it is not really possible atm to compare them because ELI has only one property (attribution) while in JOSM we have two (attribution-url and terms-of-use-url) (even three if you count also eula). These two are necessary for some sources like the opencyclemap ​http://thunderforest.com/terms/

Attribution must be given to both “Thunderforest” and “OpenStreetMap contributors”.
Users of your site/application must have a working links to www.thunderforest.com and www.openstreetmap.org/copyright

[Klumbumbus]

Also a check if <id> is present would be useful. Some weeks ago I fixed an entry without id.

[stoecker]

In 11420/josm:

see #12706 - add some more checks

[stoecker]

In 11422/josm:

see #12706 - add bounds check - no epsilon check, as we can fix these in josm wiki easily

[stoecker]

In 11423/josm:

see #12706 - add bounds check - use single quote

[stoecker]

@Vincent: Can you add this to automatic checks testing for "red color" in the output?

[Don-vip]

Sure, should be easy.

[Don-vip]

In 11426/josm:

see #12706 - add automatic test for red entries

[stoecker]

The "best" handling is a separate ticket, so I consider this done.

[Don-vip]

The automatic test works!

See jenkins/job/JOSM/lastCompletedBuild/jdk=JDK8/testReport/org.openstreetmap.josm.data.imagery/ImageryCompareTestIT/testImageryEntries/

[Klumbumbus]

In 11599/josm:

see #12706, see #12313 - enable eli-best comparison script

[Klumbumbus]

In 11665/josm:

see #12706 - enable ELI date comparison script

[Klumbumbus]

see #14655 for following ELI/JOSM sync enhancements