Modify

Opened 13 months ago

Closed 4 months ago

Last modified 7 weeks ago

#12706 closed defect (fixed)

Sync remaining maps from EII and JOSM

Reported by: stoecker Owned by: team
Priority: normal Milestone:
Component: External imagery source Version:
Keywords: imagery maps EII Cc: bastiK

Description

The image list at ImageryCompare contains still a number of red entries. They should either be added to JOSM's wiki or marked to be ignored.

If done we can automate add results into the automated build checking process.

See also these tickets:

  • #10760 - original ticket for sync, contains probably some valid information for ignores
  • #11022 - similar request
  • #12313 - wontfix for "best" implementation

Attachments (0)

Change History (56)

comment:1 Changed 13 months ago by Don-vip

Milestone: 16.0416.03

comment:2 Changed 13 months ago by Don-vip

Milestone: 16.0316.04

Milestone renamed

comment:3 Changed 12 months ago by Don-vip

Milestone: 16.0416.05

comment:4 Changed 11 months ago by Don-vip

Milestone: 16.0516.06

comment:5 Changed 10 months ago by Don-vip

Milestone: 16.06

comment:6 Changed 6 months ago by Klumbumbus

Last edited 6 months ago by Klumbumbus (previous) (diff)

comment:7 Changed 6 months ago by Klumbumbus

And what is about the entries at the end of ImageryCompare?

+++ Obsolete skip entry:

I don't see a common schema there.

Last edited 6 months ago by Klumbumbus (previous) (diff)

comment:8 Changed 6 months ago by Klumbumbus

EII uses: Hike & Bike - http://toolserver.org/tiles/hikebike/{zoom}/{x}/{y}.png
JOSM uses: Hike & Bike - http://{switch:a,b,c}.tiles.wmflabs.org/hikebike/{zoom}/{x}/{y}.png

Both work. What's the difference? Which is "better"?

comment:9 Changed 6 months ago by stoecker

What do the = 1 and = 3 behind the ignore entries mean? https://josm.openstreetmap.de/browser/josm/trunk/scripts/sync_editor_imagery_index.groovy?rev=9653#L97

The number of lines to skip.

And what is about the entries at the end of ImageryCompare?

+++ Obsolete skip entry:

I don't see a common schema there.

There are entries to ignore differences in the script, but nothing was ignored. That's an indication that the entry got changed in between. E.g. an Country code was added or something alike or in best case the difference has been resolved.

Either correct the text of the ignore (when a similar unignored entry exists which is not better than JOSM's entry) or simply remove it (when it has been resolved).

EII uses: Hike & Bike - ​http://toolserver.org/tiles/hikebike/{zoom}/{x}/{y}.png
JOSM uses: Hike & Bike - ​http://{switch:a,b,c}.tiles.wmflabs.org/hikebike/{zoom}/{x}/{y}.png

Both work. What's the difference? Which is "better"?

Toolserver is an very old address (go to Hike&Bike map and see what it uses), maybe also using old data. That's why it's better to fix it on both sides instead of using the ignores, so the question needs to be asked only once: But I personally did't want to make EII submit requests for data - in this point I'm petulant - I'll work with them, but not for them.

comment:10 in reply to:  9 Changed 6 months ago by Klumbumbus

Replying to stoecker:

What do the = 1 and = 3 behind the ignore entries mean? https://josm.openstreetmap.de/browser/josm/trunk/scripts/sync_editor_imagery_index.groovy?rev=9653#L97

The number of lines to skip.

Which lines?

comment:11 Changed 6 months ago by Klumbumbus

Could we please have one more category "updates which need to be solved in EII lists" so we can better distinguish all the entries?

Does the current ignore list only include entries with minor differences (like different name or z instead of zoom) or does it also contain entries which are outdated in EII and needs to be fixed there?

comment:12 Changed 5 months ago by stoecker

Which lines?

Of the displayed text. It's a simple string compare. E.g. the "name differs" entry consist of a line with URL and two lines with changed data. That's why the count is 3.

Could we please have one more category "updates which need to be solved in EII lists" so we can better distinguish all the entries?

Does the current ignore list only include entries with minor differences (like different name or z instead of zoom) or does it also contain entries which are outdated in EII and needs to be fixed there?

The ignores are ATM all relative to JOSM. Usually that means fixing the entry in EII (or sometimes in JOSM?) would make them obsolete. Where EII uses a TMS instead of the original WMS we can add these as mirror in our config.

comment:13 in reply to:  12 ; Changed 5 months ago by Klumbumbus

Replying to stoecker:

Which lines?

Of the displayed text. It's a simple string compare. E.g. the "name differs" entry consist of a line with URL and two lines with changed data. That's why the count is 3.

OK, now I understand.

Could we please have one more category "updates which need to be solved in EII lists" so we can better distinguish all the entries?

Does the current ignore list only include entries with minor differences (like different name or z instead of zoom) or does it also contain entries which are outdated in EII and needs to be fixed there?

The ignores are ATM all relative to JOSM. Usually that means fixing the entry in EII (or sometimes in JOSM?) would make them obsolete. Where EII uses a TMS instead of the original WMS we can add these as mirror in our config.

OK, it would be usefull to have 2 ignore lists, one for the entries which are actually correct on both sides (slightly different name or z vs. zoom) and one for the entries which should be fixed at the EII side. This way we can sort the entries without the need to fix them at EII.

Also it would be useful if the ignore lists could be maintained in the wiki. This way only a wiki edit instead of a svn commit is required to update the ignore lists.

comment:14 Changed 5 months ago by Klumbumbus

In 11234/josm:

see #12706 - update EII ignore list

comment:15 Changed 5 months ago by stoecker

In 11238/josm:

see #12706 - move skiplist to wiki, add a second color code for ignores

comment:16 Changed 5 months ago by stoecker

In 11239/josm:

see #12706 - move skiplist to wiki, add a second color code for ignores - better color

comment:17 in reply to:  13 ; Changed 5 months ago by stoecker

Replying to Klumbumbus:

OK, it would be usefull to have 2 ignore lists, one for the entries which are actually correct on both sides (slightly different name or z vs. zoom) and one for the entries which should be fixed at the EII side. This way we can sort the entries without the need to fix them at EII.

Also it would be useful if the ignore lists could be maintained in the wiki. This way only a wiki edit instead of a svn commit is required to update the ignore lists.

Done.

comment:18 in reply to:  17 Changed 5 months ago by Klumbumbus

Replying to stoecker:

Done.

Great!

comment:19 Changed 5 months ago by stoecker

In 11242/josm:

see #12706 - change regexp, so that «blancos for copy» don't produce broken skip entry :-)

comment:20 Changed 5 months ago by Klumbumbus

In 11245/josm:

see #12706 - use a dark yellow color (two greens are too similar)

comment:21 Changed 5 months ago by Klumbumbus

In 11246/josm:

see #12706 - typos

comment:22 Changed 5 months ago by Klumbumbus

Cc: bastiK added

I don't understand this change. https://josm.openstreetmap.de/wiki/Maps/Estonia?action=diff&version=5 Isn't Mercator the "normal" projection? The previous URLs with -geo support EPSG4326 and work in Merkator. (I checked for Estonia Basemap (Maaamet))

comment:23 Changed 5 months ago by Klumbumbus

If https is available should that be preferred over http? (e.g. + OpenStreetMap Carto (Standard layer) - https://{switch:a,b,c}.tile.openstreetmap.org/{zoom}/{x}/{y}.png)

comment:24 Changed 5 months ago by stoecker

@bastiK:

Did you see comment 22?

comment:25 Changed 5 months ago by stoecker

I'd say https should be preferred nowadays.

comment:26 in reply to:  22 Changed 5 months ago by bastiK

Replying to Klumbumbus:

I don't understand this change. https://josm.openstreetmap.de/wiki/Maps/Estonia?action=diff&version=5 Isn't Mercator the "normal" projection? The previous URLs with -geo support EPSG4326 and work in Merkator. (I checked for Estonia Basemap (Maaamet))

I changed it because displaying stretched WGS 84 tiles on a Mercator map is a bit of a hack.

Nevertheless, it works fine for this server, so feel free to switch back.

comment:27 Changed 5 months ago by Klumbumbus

OK, I switched back and moved the EPSG:3301 tiles into <mirror>

Last edited 5 months ago by Klumbumbus (previous) (diff)

comment:28 Changed 5 months ago by stoecker

Thanks a lot Klumbumbus for the cleanups!

JOSM's side of fixes is now (mainly?) done for the moment, so #12313 could be reconsidered. I'm still not really convinced of the "best" flag, but I can see some benefits. Opinions?

And also maybe we could improve the check scripts to detect differences in the polygons between EII and JOSM (and glitches in our own database like unclosed rings) or even other values?

Also now we could add a build check for red entries in the list. Should we?

comment:29 in reply to:  28 Changed 5 months ago by Klumbumbus

Replying to stoecker:

Thanks a lot Klumbumbus for the cleanups!

JOSM's side of fixes is now (mainly?) done for the moment,

I checked all blue entries if they still deliver tiles and found some, which don't. I will post the list in an own ticket for review.

so #12313 could be reconsidered. I'm still not really convinced of the "best" flag, but I can see some benefits. Opinions?

I also think it's not the best concept. However it would be good to have this flag in our database too, the reasons given in #12313 are valid . Maybe in JOSM we could display a little star in front of the imagery name when it has the best flag.

And also maybe we could improve the check scripts to detect differences in the polygons between EII and JOSM (and glitches in our own database like unclosed rings) or even other values?

Would be good.

Also now we could add a build check for red entries in the list. Should we?

Would be good too. However only for the current checks. Possible polygon check should not yet be included. I guess fixing the polygons will be a longer task.

comment:30 Changed 4 months ago by stoecker

In 11410/josm:

see #12706 - show mismatching shape information

comment:31 Changed 4 months ago by Klumbumbus

Uh, thats a lot. Is there a way to automatically fix these 236 unclosed JOSM shapes by a script? Doing this by hand looks like a lot of boring work.

There seems to be an error in the shape comparison. E.g. imagico.de OSM images for mapping: Eastern Devon Island coast (Canada) is reported at ImageryCompare, but it has the same shape in maps and in the ELI xml output (in the ELI source file there are some more decimal places, though).

comment:32 in reply to:  31 ; Changed 4 months ago by stoecker

Replying to Klumbumbus:

Uh, thats a lot. Is there a way to automatically fix these 236 unclosed JOSM shapes by a script? Doing this by hand looks like a lot of boring work.

That would be probably more work that manual. :-) I started yesterday but overlooked that the list is unsorted. I was so happy that I fixed 3 whole countries... The fix is usually simple, copy the first point to the end as well.

There seems to be an error in the shape comparison. E.g. imagico.de OSM images for mapping: Eastern Devon Island coast (Canada) is reported at ImageryCompare, but it has the same shape in maps and in the ELI xml output (in the ELI source file there are some more decimal places, though).

There is no approximate check. It checks for equality, so more decimal places means "not equal".

comment:33 Changed 4 months ago by stoecker

In 11411/josm:

see #12706 - improve shape difference display, ignore shapes equivalent to bounds

comment:34 in reply to:  32 Changed 4 months ago by Klumbumbus

Replying to stoecker:

There is no approximate check. It checks for equality, so more decimal places means "not equal".

I thought you use the ELI XML output for the comparison, but you use the json file. The ELI XML file (where I took some shapes from, e.g. all the imagico entries) cuts the last decimal places. So thats the reason for the difference.

BTW geojson is now the default format on ELI and json only legacy. I don't know if it makes sense to change our script.

comment:35 Changed 4 months ago by stoecker

In 11412/josm:

see #12706 - switch to geojson

comment:36 Changed 4 months ago by stoecker

No sense using an old format when the switch is relatively easy.

comment:37 Changed 4 months ago by stoecker

For the coordinates we probably need a better equals() function taking a certain epsilon into account :-)

comment:38 Changed 4 months ago by Klumbumbus

When does it report No EII shape:? When there are only 4 different points?

comment:39 Changed 4 months ago by stoecker

In 11413/josm:

see #12706 - better equals function and better difference description

comment:40 in reply to:  38 Changed 4 months ago by stoecker

Now it's much better - less reports and better details.

Also removed most of the unclosed polygons.

Replying to Klumbumbus:

When does it report No EII shape:? When there are only 4 different points?

I added a special detection for this bounding box similar shapes. Maybe that's wrong. Which entry?

comment:41 Changed 4 months ago by stoecker

In 11414/josm:

see #12706 - small cleanup

comment:42 Changed 4 months ago by Klumbumbus

All the imagico entries

comment:43 Changed 4 months ago by stoecker

In 11415/josm:

see #12706 - fix 5 point shape handling

comment:44 Changed 4 months ago by stoecker

Hmm, is it useful to add a check for the remaining keyword elements? Polygons have been less work than I expected it to be. I thought it would be a nightmare.

comment:45 in reply to:  44 Changed 4 months ago by Klumbumbus

Replying to stoecker:

Hmm, is it useful to add a check for the remaining keyword elements?

  • icon and projectons would be useful
  • id seems not useful
  • maybe a check for the josm database if bounds and shape fit together
  • copyright/attribution information seems important but I think it is not really possible atm to compare them because ELI has only one property (attribution) while in JOSM we have two (attribution-url and terms-of-use-url) (even three if you count also eula). These two are necessary for some sources like the opencyclemap http://thunderforest.com/terms/

Attribution must be given to both “Thunderforest” and “OpenStreetMap contributors”.
Users of your site/application must have a working links to www.thunderforest.com and www.openstreetmap.org/copyright

comment:46 Changed 4 months ago by Klumbumbus

Also a check if <id> is present would be useful. Some weeks ago I fixed an entry without id.

comment:47 Changed 4 months ago by stoecker

In 11420/josm:

see #12706 - add some more checks

comment:48 Changed 4 months ago by stoecker

In 11422/josm:

see #12706 - add bounds check - no epsilon check, as we can fix these in josm wiki easily

comment:49 Changed 4 months ago by stoecker

In 11423/josm:

see #12706 - add bounds check - use single quote

comment:50 Changed 4 months ago by stoecker

@Vincent: Can you add this to automatic checks testing for "red color" in the output?

comment:51 Changed 4 months ago by Don-vip

Sure, should be easy.

comment:52 Changed 4 months ago by Don-vip

In 11426/josm:

see #12706 - add automatic test for red entries

comment:53 Changed 4 months ago by stoecker

Resolution: fixed
Status: newclosed

The "best" handling is a separate ticket, so I consider this done.

comment:55 Changed 2 months ago by Klumbumbus

In 11599/josm:

see #12706, see #12313 - enable eli-best comparison script

comment:56 Changed 7 weeks ago by Klumbumbus

In 11665/josm:

see #12706 - enable ELI date comparison script

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain team.
as The resolution will be set. Next status will be 'closed'.
The resolution will be deleted. Next status will be 'reopened'.

Add Comment


E-mail address and name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.