#16073 closed enhancement (fixed)
ImageryCompare for dead layers
| Reported by: | marc_marc | Owned by: | team |
|---|---|---|---|
| Priority: | normal | Milestone: | 18.12 |
| Component: | Unit tests | Version: | |
| Keywords: | Cc: |
Description
eli check and remove some not working layers.
those layers are show in "URLs found in JOSM but not in ELI" (that's right) but in blue (= should be checked).
To avoid the need to make the same check "by hand" for josm or to make a lookup in eli history for every layers, it would be nice if ImageryCompare (or a dedicated page) could make the same checks and/or used those checks. for exemple missing layer in josm that return a 404 could be tagged in red to easy see that they need to be renove also from josm.
of course it would be nice it the same tools could be shared between both "index" (I also did the same a request at eli)
NB: here is a POC that show removed eli layers not already removed from josm
NB: I start removing those 12 layers by hand, but adding this info in ImageryCompare 'll be usefull for everyone that use it.
Attachments (2)
Change History (106)
comment:1 by , 7 years ago
comment:2 by , 7 years ago
I'm thinking about a way to detect dead entries. But it may take some time to think and implement a proper solution.
follow-up: 4 comment:3 by , 7 years ago
Martin over at ELI is working on the problem too. He's requesting tiles within the layer's bbox to make sure it's returning something.
comment:4 by , 7 years ago
Replying to Stereo:
Martin over at ELI is working on the problem too. He's requesting tiles within the layer's bbox to make sure it's returning something.
Jupp. That's one of the points ;-)
follow-up: 9 comment:7 by , 6 years ago
Regarding https://josm.openstreetmap.de/wiki/Maps/France?action=diff&version=146 I think it was supposed to display the EULA in the language which is set in JOSM if available. Currently available are english and french: https://wiki.openstreetmap.org/wiki/WikiProject_France/CRAIG/EULA
follow-up: 14 comment:8 by , 6 years ago
Seems there are some errors:
- https://öpnvkarte.de/ works for me
https://download.kortforsyningen.dk/content/vilkaar-og-betingelser does not issue an 404it does- Messages like "https://öpnvkarte.de/ -> öpnvkarte.de" aren't helpful. Does the exception have a better output?
I think you should output >= 300 HTTP errors, so we can fix redirects as well. They should be warnings, not errors.
We have some certificate issues, probably due to missing full chain.
- https://www.ssllabs.com/ssltest/analyze.html?d=idecor.cba.gov.ar&s=201.234.102.22&latest → send e-mail to site operator
- https://www.ssllabs.com/ssltest/analyze.html?d=www.linz.govt.nz&latest → send e-mail to site operator
comment:9 by , 6 years ago
Replying to Klumbumbus:
Regarding https://josm.openstreetmap.de/wiki/Maps/France?action=diff&version=146 I think it was supposed to display the EULA in the language which is set in JOSM if available. Currently available are english and french: https://wiki.openstreetmap.org/wiki/WikiProject_France/CRAIG/EULA
Yes. There is code in ImageryPreference.java for this. The Map change should be reverted and the check simply replace {lang} with the default "" for this test here.
comment:14 by , 6 years ago
Replying to stoecker:
- https://www.ssllabs.com/ssltest/analyze.html?d=idecor.cba.gov.ar&s=201.234.102.22&latest → send e-mail to site operator
- https://www.ssllabs.com/ssltest/analyze.html?d=www.linz.govt.nz&latest → send e-mail to site operator
Do you mean you have already sent an e-mail, or do you want someone to do it?
comment:17 by , 6 years ago
| Component: | External imagery source → Unit tests |
|---|---|
| Milestone: | → 18.12 |
comment:19 by , 6 years ago
For wiki:/Maps/Taiwan#NLSCOpenDataWMTS JOSM says "No layers defined by getCapabilities document: https://maps.nlsc.gov.tw/OpenData/wmts". Wrong WMTS file or a bug in JOSM?
follow-up: 21 comment:20 by , 6 years ago
https://maps.nlsc.gov.tw seems dead, I have sent an e-mail to them
by , 6 years ago
comment:21 by , 6 years ago
by , 6 years ago
| Attachment: | notforme.png added |
|---|
comment:22 by , 6 years ago
Because of this. But I don't know why, this is the only site I cannot access:
follow-up: 25 comment:23 by , 6 years ago
Is it IPv6 only? My DNS resolves to an IPv4 address while JOSM server resolves to an IPv6 one.
comment:25 by , 6 years ago
Replying to Don-vip:
Is it IPv6 only? My DNS resolves to an IPv4 address while JOSM server resolves to an IPv6 one.
Yes. IPv6 is broken for their site: https://[2001:e10:6040:143::28]/
comment:27 by , 6 years ago
To test that max zoom is correct I need to know where to check. For local providers I simply choose the center of the bbox, but for global providers it is more tricky, as the max zoom isn't available globally. For example with ESRI, they provide the available resolutions per area. It appears the more precise location is Santa Rosa, California, USA:
Question is: how do we store this information in the wiki? I suggest a new attribute, something like <test-max-zoom-location>38.438631,-122.719048<test-max-zoom-location>
We could also add a <test-no-tile-location> to test the validity of <no-tile-header>.
Comments?
comment:29 by , 6 years ago
Don't get too sophisticated. A general availability check should be enough.
comment:35 by , 6 years ago
According to AbstractTileSourceLayer the default Max-Zoom we use is 20. You could use that for tests, so that we can fix these which have 18 only.
comment:36 by , 6 years ago
Or you use the function of the tile layer to get the MaxZoom, which takes the default into account :-)
comment:43 by , 6 years ago
How do we want to handle known invalid entries like the TLS problems? Add an Ignore list to ImageryCompareIgnores in some form? Maybe a second table?
follow-up: 46 comment:45 by , 6 years ago
Replying to stoecker:
Drop SPOT in sources as well: [o34776].
Fixed in [o34777]. You should use an IDE to spot compile errors :p
Replying to stoecker:
How do we want to handle known invalid entries like the TLS problems? Add an Ignore list to ImageryCompareIgnores in some form?
A separate page with the same idea sounds good.
comment:46 by , 6 years ago
Replying to Don-vip:
Replying to stoecker:
Drop SPOT in sources as well: [o34776].
Fixed in [o34777]. You should use an IDE to spot compile errors :p
Compiling also for simple fixes would be enough as well. ;-)
How do we want to handle known invalid entries like the TLS problems? Add an Ignore list to ImageryCompareIgnores in some form?
A separate page with the same idea sounds good.
Why separate? ImageryCompare already handles a lot of other non-remote validity checks. Keeping that in one place is not strange I'd say.
follow-up: 48 comment:47 by , 6 years ago
The point of this test is not to compare with ELI but check the validity of our own sources. I have other tests like this (validity of our plugins, presets, validator rules, etc.) that also fail quite often and could benefit from a common page like IntegrationTestIgnores. Right now I exclude some presets directly in the Java source code, it's not ideal.
Also the TLS stuff from example has nothing to do with ELI: from a web browser, it works. It only concerns Java.
follow-up: 49 comment:48 by , 6 years ago
Replying to Don-vip:
The point of this test is not to compare with ELI but check the validity of our own sources.
I know. Nevertheless ImageryCompare actually is an ImageryCompareAndCheckValidity test already. A lot of that stuff has nothing to do with ELI as well. It was only a place where these checks could be added without duplicating a lot of effort.
I have other tests like this (validity of our plugins, presets, validator rules, etc.) that also fail quite often and could benefit from a common page like IntegrationTestIgnores. Right now I exclude some presets directly in the Java source code, it's not ideal.
A generic page for all integration tests is not a bad idea. Format could be similar to the ignores page, but adding a integration test name as first column.
comment:49 by , 6 years ago
Replying to stoecker:
Format could be similar to the ignores page, but adding a integration test name as first column.
That's what I had in mind :)
comment:71 by , 6 years ago
For some reason the test is performed in the area where only zoom 14 is available.
comment:73 by , 6 years ago
Why min-zoom 13 for the big shape? --> https://gravitystorm.dev.openstreetmap.org/imagery/philippines/12/3424/1874.png
follow-up: 76 comment:75 by , 6 years ago
I didn't know about this check. It should not depend on the order of shapes. My test always takes the first shape to check for tile validity:
* Different number of points for shape 1 (10 ! = 5)): [PH] Pangasinán/Bulacan (Philippines HiRes) - https://gravitystorm.dev.openstreetmap.org/imagery/philippines/{zoom}/{x}/{y}.png
* Different number of points for shape 2 (5 ! = 10)): [PH] Pangasinán/Bulacan (Philippines HiRes) - https://gravitystorm.dev.openstreetmap.org/imagery/philippines/{zoom}/{x}/{y}.png
comment:76 by , 6 years ago
Replying to Don-vip:
I didn't know about this check. It should not depend on the order of shapes. My test always takes the first shape to check for tile validity:
* Different number of points for shape 1 (10 ! = 5)): [PH] Pangasinán/Bulacan (Philippines HiRes) - https://gravitystorm.dev.openstreetmap.org/imagery/philippines/{zoom}/{x}/{y}.png * Different number of points for shape 2 (5 ! = 10)): [PH] Pangasinán/Bulacan (Philippines HiRes) - https://gravitystorm.dev.openstreetmap.org/imagery/philippines/{zoom}/{x}/{y}.png
Hmm, then the sync script needs to handle that situation...
follow-up: 79 comment:77 by , 6 years ago
I guess the script can sort the shapes itself before comparing them?
comment:78 by , 6 years ago
Other possibility would be to allow min-zoom/max-zoom elements in shapes, not only in entries.
comment:79 by , 6 years ago
Replying to Don-vip:
I guess the script can sort the shapes itself before comparing them?
It could, but we need to keep original position for the output or users are confused.
Other possibility would be to allow min-zoom/max-zoom elements in shapes, not only in entries.
No. That's useless complexity. For the tests we could add an "test-position" field to choose a defined position for tests in these cases where automatics will fail. Or we simply catch that with the ignores page (probably the easiest and best solution).
comment:80 by , 6 years ago
I don't like ignoring the issue in this case, this means we ignore completely to check a server working correctly.
After more thoughts I can simply check each shape. The test will last longer but after all that's the only way to be sure the geometry is (almost) correct. Thus I won't need to change the XML (which is good)
follow-up: 83 comment:81 by , 6 years ago
Whoah 261 Ignores on the page already for fixes you did. This test really was necessary...
follow-up: 84 comment:83 by , 6 years ago
Replying to stoecker:
Whoah 261 Ignores on the page already for fixes you did. This test really was necessary...
Indeed. Almost finished :) We will really have a nice database now!
follow-up: 86 comment:84 by , 6 years ago
Indeed. Almost finished :) We will really have a nice database now!
Dream on 😁 What do you think I thought when I added many sanity checks in the sync script and fixed the related errors in our database...
comment:85 by , 6 years ago
The test should at the end output unused ignores, so that we have a chance from time to time to get rid of old ones.
follow-up: 87 comment:86 by , 6 years ago
| Resolution: | → fixed |
|---|---|
| Status: | new → closed |
Replying to stoecker:
Dream on 😁 What do you think I thought when I added many sanity checks in the sync script and fixed the related errors in our database...
I'm sure it was far worse back then :)
Replying to stoecker:
The test should at the end output unused ignores, so that we have a chance from time to time to get rid of old ones.
Problem is that many ignores won't trigger because I duplicated error messages for switches. During the next few weeks I'll check links directly from IntegrationTestIgnores.
comment:87 by , 6 years ago
The test should at the end output unused ignores, so that we have a chance from time to time to get rid of old ones.
Problem is that many ignores won't trigger because I duplicated error messages for switches. During the next few weeks I'll check links directly from IntegrationTestIgnores.
Don't make them an error like for sync script, but output nevertheless.
For duplicated error message you may find a way to de-duplicate the message, e.g. additionally allow RegExp syntax ignores?
follow-up: 95 comment:94 by , 6 years ago
2018-12-30 13:19:31.731 WARNING: Ignore line unused: http://wms.ign.gob.ar:80/geoserver/ows?SERVICE=WMS&FORMAT=image/png&TRANSPARENT=TRUE&VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&LAYERS=topografico:area_de_limites&STYLES=&SRS=EPSG:3857&WIDTH=512&HEIGHT=512&BBOX=-20037508.3427892,-20037506.6204108,20037506.6204108,20037508.3427892 -> zoom 1 -> Could not find layer topografico:area_de_limites 2018-12-30 13:19:31.731 WARNING: Ignore line unused: http://wms.ign.gob.ar:80/geoserver/ows?SERVICE=WMS&FORMAT=image/png&TRANSPARENT=TRUE&VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&LAYERS=topografico:area_de_limites&STYLES=&SRS=EPSG:3857&WIDTH=512&HEIGHT=512&BBOX=-7083572.8419892,-4696289.9548108,-7064004.9635892,-4676722.0764108 -> zoom 12 -> Could not find layer topografico:area_de_limites
Where do these two come from? That ":80" looks strange to me.
comment:95 by , 6 years ago
Replying to stoecker:
Where do these two come from? That ":80" looks strange to me.
Maps/Argentina's ign-wms entry is a WMS endpoint. The test checks the first layer defined in capabilities, which is:
<Layer queryable="1" opaque="0"> <Name>topografico:area_de_limites</Name> <Title>Area de Límites</Title> <Abstract/> <KeywordList><Keyword>features</Keyword><Keyword>area_de_limites</Keyword></KeywordList> <CRS>EPSG:4326</CRS> <CRS>CRS:84</CRS> <EX_GeographicBoundingBox> <westBoundLongitude>-181.800003051758</westBoundLongitude> <eastBoundLongitude>181.800018310547</eastBoundLongitude> <southBoundLatitude>-90.8538589477539</southBoundLatitude> <northBoundLatitude>81.6239395141602</northBoundLatitude> </EX_GeographicBoundingBox> <BoundingBox CRS="CRS:84" minx="-181.800003051758" miny="-90.8538589477539" maxx="181.800018310547" maxy="81.6239395141602"/> <BoundingBox CRS="EPSG:4326" minx="-90.8538589477539" miny="-181.800003051758" maxx="81.6239395141602" maxy="181.800018310547"/> <Style> <Name>generic</Name> <Title>Generic</Title> <Abstract>Generic style</Abstract> <LegendURL width="20" height="20"> <Format>image/png</Format> <OnlineResource xlink:type="simple" xlink:href="http://wms.ign.gob.ar:80/geoserver/ows?service=WMS&request=GetLegendGraphic&format=image%2Fpng&width=20&height=20&layer=topografico%3Aarea_de_limites"/> </LegendURL> </Style> </Layer>
follow-up: 97 comment:96 by , 6 years ago
Are you sure your code works? The test has been ignored as expected:
Skip Message
{AR={ImageryInfo{name='National Geographic Institute (WMS)', countryCode='AR', url='https://wms.ign.gob.ar/geoserver/ows?service=wms&version=1.3.0&request=GetCapabilities', imageryType=WMS_ENDPOINT}=
[
http://wms.ign.gob.ar:80/geoserver/ows?SERVICE=WMS&FORMAT=image/png&TRANSPARENT=TRUE&VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&LAYERS=topografico:area_de_limites&STYLES=&SRS=EPSG:3857&WIDTH=512&HEIGHT=512&BBOX=-20037508.3427892,-20037506.6204108,20037506.6204108,20037508.3427892 -> zoom 1 -> Could not find layer topografico:area_de_limites
,
http://wms.ign.gob.ar:80/geoserver/ows?SERVICE=WMS&FORMAT=image/png&TRANSPARENT=TRUE&VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&LAYERS=topografico:area_de_limites&STYLES=&SRS=EPSG:3857&WIDTH=512&HEIGHT=512&BBOX=-7083572.8419892,-4696289.9548108,-7064004.9635892,-4676722.0764108 -> zoom 12 -> Could not find layer topografico:area_de_limites
]}
comment:97 by , 6 years ago
comment:98 by , 6 years ago
IGN websites have been unreachable all day yesterday, maybe that's why you get this message.
follow-up: 100 comment:99 by , 6 years ago
@stoecker I don't understand why this new entry is unreachable from JOSM server. It works fine from my machine but we get connect timeout errors from Jenkins.
https://ogrip.oit.ohio.gov/ProjectsInitiatives/StatewideImagery.aspx -> java.net.SocketTimeoutException: connect timed out https://ogrip.oit.ohio.gov/Portals/0/PDFs/OSIP%20Program%20Description.pdf#page=2 -> java.net.SocketTimeoutException: connect timed out https://geo.oit.ohio.gov/arcgis/services/OSIP/osip_best_avail_1ft/ImageServer/WMSServer?FORMAT=image/png&TRANSPARENT=TRUE&VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&LAYERS=0&STYLES=&SRS=EPSG:4326&WIDTH=512&HEIGHT=512&BBOX=-180.0000000,-269.9999845,179.9999845,90.0000000 -> java.net.SocketTimeoutException: connect timed out https://geo.oit.ohio.gov/arcgis/services/OSIP/osip_best_avail_1ft/ImageServer/WMSServer?FORMAT=image/png&TRANSPARENT=TRUE&VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&LAYERS=0&STYLES=&SRS=EPSG:4326&WIDTH=512&HEIGHT=512&BBOX=-82.7929729,40.0781271,-82.6171917,40.2539084 -> java.net.SocketTimeoutException: connect timed out
<entry> <name>Ohio Statewide Imagery Program</name> <id>Ohio_OSIP_1ft</id> <category>photo</category> <date>2011;2014</date> <country-code>US</country-code> <type>wms</type> <url><![CDATA[https://geo.oit.ohio.gov/arcgis/services/OSIP/osip_best_avail_1ft/ImageServer/WMSServer?FORMAT=image/png&TRANSPARENT=TRUE&VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&LAYERS=0&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}]]></url> <min-zoom>1</min-zoom> <max-zoom>20</max-zoom> <projections> <code>EPSG:4326</code> </projections> <attribution-text>Ohio Statewide Imagery Program</attribution-text> <attribution-url>https://ogrip.oit.ohio.gov/ProjectsInitiatives/StatewideImagery.aspx</attribution-url> <permission-ref><![CDATA[https://ogrip.oit.ohio.gov/Portals/0/PDFs/OSIP%20Program%20Description.pdf#page=2]]></permission-ref> <bounds min-lat='38.3974528' min-lon='-84.8447141' max-lat='41.9993921' max-lon='-80.5016116'/> </entry>
follow-up: 101 comment:100 by , 6 years ago
Replying to Don-vip:
@stoecker I don't understand why this new entry is unreachable from JOSM server. It works fine from my machine but we get connect timeout errors from Jenkins.
I'd simply say they block the IP. I tried serveral other servers from Hetzner and had no success. So they probably block the whole Hetzner IP range or even more. Overblocking is a common behaviour for US services.
comment:101 by , 6 years ago
Replying to stoecker:
I'd simply say they block the IP. I tried serveral other servers from Hetzner and had no success. So they probably block the whole Hetzner IP range or even more. Overblocking is a common behaviour for US services.
OK I've sent them an e-mail.
My POC works but is unreadable :). Basically it looks at the URLs that have disappeared in imagery.geojson since an arbitrary commit at the beginning of the year, keeps only those with } to filter out attribution URLs etc., and compares the URLs to the ones ImageryCompare says are missing.