Modify

Opened 6 months ago

Closed 6 months ago

Last modified 3 months ago

#14655 closed defect (fixed)

JOSM / ELI synchronization

Reported by: Don-vip Owned by: Klumbumbus
Priority: normal Milestone: 17.04
Component: Core imagery Version:
Keywords: Cc: stoecker, Klumbumbus, Zverikk

Description

The test is failing for several days now:

Error: at least a red line has been found, see https://josm.openstreetmap.de/wiki/ImageryCompare for details
Displaying only red entries. The test fails if at least one is found
*** URLs found in ELI but not in JOSM (36): ***
-  [BR] Alagoas Litoral 2006 - http://geo.seplande.al.gov.br/teogc/terraogcwms.cgi?LAYERS=Ortofotos%202006&SERVICE=WMS&FORMAT=image/png&TRANSPARENT=TRUE&VERSION=1.1.0&SERVICE=WMS&REQUEST=GetMap&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
-  [BR] Arapiraca AL - http://geo.seplande.al.gov.br/teogc/terraogcwms.cgi?LAYERS=Arapiraca&SERVICE=WMS&FORMAT=image/png&TRANSPARENT=TRUE&VERSION=1.1.0&SERVICE=WMS&REQUEST=GetMap&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
-  [BR] Branquinha AL - http://geoserver.dados.al.gov.br:8080/geoserver/Alagoas/ows?service=WMS&version=1.1.0&request=GetMap&layers=Branquinha&SERVICE=WMS&FORMAT=image/png&TRANSPARENT=TRUE&VERSION=1.1.0&REQUEST=GetMap&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
-  [BR] Cajueiro AL - http://geo.seplande.al.gov.br/teogc/terraogcwms.cgi?LAYERS=Cajueiro&SERVICE=WMS&FORMAT=image/png&TRANSPARENT=TRUE&VERSION=1.1.0&SERVICE=WMS&REQUEST=GetMap&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
-  [BR] Campo Alegre AL - http://geo.seplande.al.gov.br/teogc/terraogcwms.cgi?LAYERS=Campo%20Alegre&SERVICE=WMS&FORMAT=image/png&TRANSPARENT=TRUE&VERSION=1.1.0&SERVICE=WMS&REQUEST=GetMap&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
-  [BR] Capela AL - http://geo.seplande.al.gov.br/teogc/terraogcwms.cgi?LAYERS=Capela&SERVICE=WMS&FORMAT=image/png&TRANSPARENT=TRUE&VERSION=1.1.0&SERVICE=WMS&REQUEST=GetMap&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
-  [BR] DER-ES - http://portal.der.es.gov.br/geoserver/wms?SERVICE=WMS&FORMAT=image/png&TRANSPARENT=TRUE&VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&LAYERS=deres-ext:TRECHOINFO&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
-  [BR] Delmiro Gouveia AL - http://geo.seplande.al.gov.br/teogc/terraogcwms.cgi?LAYERS=Delmiro%20Gouveia&SERVICE=WMS&FORMAT=image/png&TRANSPARENT=TRUE&VERSION=1.1.0&SERVICE=WMS&REQUEST=GetMap&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
-  [BR] Estrela de Alagoas - http://geo.seplande.al.gov.br/teogc/terraogcwms.cgi?LAYERS=Estrela%20de%20Alagoas&SERVICE=WMS&FORMAT=image/png&TRANSPARENT=TRUE&VERSION=1.1.0&SERVICE=WMS&REQUEST=GetMap&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
-  [BR] Girau do Ponciano - http://geo.seplande.al.gov.br/teogc/terraogcwms.cgi?LAYERS=Girau%20do%20Ponciano&SERVICE=WMS&FORMAT=image/png&TRANSPARENT=TRUE&VERSION=1.1.0&SERVICE=WMS&REQUEST=GetMap&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
-  [BR] Major Isidoro AL - http://geo.seplande.al.gov.br/teogc/terraogcwms.cgi?LAYERS=Major%20Isidoro&SERVICE=WMS&FORMAT=image/png&TRANSPARENT=TRUE&VERSION=1.1.0&SERVICE=WMS&REQUEST=GetMap&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
-  [BR] Mar Vermelho AL - http://geo.seplande.al.gov.br/teogc/terraogcwms.cgi?LAYERS=Mar%20Vermelho&SERVICE=WMS&FORMAT=image/png&TRANSPARENT=TRUE&VERSION=1.1.0&SERVICE=WMS&REQUEST=GetMap&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
-  [BR] Minador do Negrão - http://geo.seplande.al.gov.br/teogc/terraogcwms.cgi?LAYERS=Minador%20do%20Negrao&SERVICE=WMS&FORMAT=image/png&TRANSPARENT=TRUE&VERSION=1.1.0&SERVICE=WMS&REQUEST=GetMap&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
-  [BR] Olho d'Agua do Casado AL - http://geo.seplande.al.gov.br/teogc/terraogcwms.cgi?LAYERS=Olho%20Dagua%20do%20Casado&SERVICE=WMS&FORMAT=image/png&TRANSPARENT=TRUE&VERSION=1.1.0&SERVICE=WMS&REQUEST=GetMap&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
-  [BR] Palestina AL - http://geo.seplande.al.gov.br/teogc/terraogcwms.cgi?LAYERS=Palestina&SERVICE=WMS&FORMAT=image/png&TRANSPARENT=TRUE&VERSION=1.1.0&SERVICE=WMS&REQUEST=GetMap&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
-  [BR] Palmeira dos Indios AL - http://geo.seplande.al.gov.br/teogc/terraogcwms.cgi?LAYERS=Palmeira%20dos%20Indios&SERVICE=WMS&FORMAT=image/png&TRANSPARENT=TRUE&VERSION=1.1.0&SERVICE=WMS&REQUEST=GetMap&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
-  [BR] Piranhas AL - wms:http://geo.seplande.al.gov.br/teogc/terraogcwms.cgi?LAYERS=Piranhas&SERVICE=WMS&FORMAT=image/png&TRANSPARENT=TRUE&VERSION=1.1.0&SERVICE=WMS&REQUEST=GetMap&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
-  [BR] Poço das Trincheiras AL - http://geo.seplande.al.gov.br/teogc/terraogcwms.cgi?LAYERS=Poco%20das%20Trincheiras&SERVICE=WMS&FORMAT=image/png&TRANSPARENT=TRUE&VERSION=1.1.0&SERVICE=WMS&REQUEST=GetMap&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
-  [BR] Pão de Açucar AL - http://geo.seplande.al.gov.br/teogc/terraogcwms.cgi?LAYERS=Pao%20de%20Acucar&SERVICE=WMS&FORMAT=image/png&TRANSPARENT=TRUE&VERSION=1.1.0&SERVICE=WMS&REQUEST=GetMap&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
-  [BR] Santana do Ipanema AL - http://geo.seplande.al.gov.br/teogc/terraogcwms.cgi?LAYERS=Santana%20do%20Ipanema&SERVICE=WMS&FORMAT=image/png&TRANSPARENT=TRUE&VERSION=1.1.0&SERVICE=WMS&REQUEST=GetMap&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
-  [BR] São Miguel dos Campos AL - http://geo.seplande.al.gov.br/teogc/terraogcwms.cgi?LAYERS=Sao_miguel_dos_campos&SERVICE=WMS&FORMAT=image/png&TRANSPARENT=TRUE&VERSION=1.1.0&SERVICE=WMS&REQUEST=GetMap&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
-  [BR] Teotonio Vilela AL - http://geo.seplande.al.gov.br/teogc/terraogcwms.cgi?LAYERS=Teotonio%20Vilela&SERVICE=WMS&FORMAT=image/png&TRANSPARENT=TRUE&VERSION=1.1.0&SERVICE=WMS&REQUEST=GetMap&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
-  [BR] União dos Palmares AL - http://geo.seplande.al.gov.br/teogc/terraogcwms.cgi?LAYERS=Uniao%20dos%20Palmares&SERVICE=WMS&FORMAT=image/png&TRANSPARENT=TRUE&VERSION=1.1.0&SERVICE=WMS&REQUEST=GetMap&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
-  [BR] Viçosa AL - http://geo.seplande.al.gov.br/teogc/terraogcwms.cgi?LAYERS=Vicosa&SERVICE=WMS&FORMAT=image/png&TRANSPARENT=TRUE&VERSION=1.1.0&SERVICE=WMS&REQUEST=GetMap&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}

Attachments (2)

wow.PNG (3.1 KB) - added by Don-vip 6 months ago.
josm_perfo.PNG (13.6 KB) - added by Don-vip 3 months ago.

Download all attachments as: .zip

Change History (91)

comment:1 Changed 6 months ago by Klumbumbus

Owner: changed from team to Klumbumbus
Status: newassigned

comment:2 Changed 6 months ago by Klumbumbus

Resolution: fixed
Status: assignedclosed

comment:3 Changed 6 months ago by Don-vip

new errors:

Displaying only red entries. The test fails if at least one is found
*** Mismatching shapes: ***
+++ JOSM shape 1 unclosed: [DE] Deutsche Bahn VzG lines Nov 2013 - https://wms.michreichert.de/vzg-strecken?FORMAT=image/png&TRANSPARENT=TRUE&VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&LAYERS=vzg_strecken,station_codes,level_crossings&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
*** Miscellaneous checks: ***
* Bounds do not match shape (is 47.2363,5.53806,55.2817,15.3711, calculated <bounds min-lat='47.391238' min-lon='6.003688' max-lat='54.990006' max-lon='15.039257'>): [DE] Deutsche Bahn VzG lines Nov 2013 - https://wms.michreichert.de/vzg-strecken?FORMAT=image/png&TRANSPARENT=TRUE&VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&LAYERS=vzg_strecken,station_codes,level_crossings&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}

comment:4 in reply to:  3 Changed 6 months ago by stoecker

Replying to Don-vip:

new errors:

Well, these messages already contained the way to fix them in the message itself.

  • Replace the slightly wrong bounds with the calculated ones.
  • Copy the last point to close the polygon

comment:5 Changed 6 months ago by Don-vip

Probably, but I'm not interested in ELI synchronization. I added the test because you asked to :)

comment:6 in reply to:  5 Changed 6 months ago by stoecker

Replying to Don-vip:

Probably, but I'm not interested in ELI synchronization. I added the test because you asked to :)

Hah, I got you. That message doesn't come from the syncing, but from the additional sanity checks implemented in the same script. And you are very well into cleanup and warning fixing. You did this mostly at least the last 2 years...

;-)

comment:7 Changed 6 months ago by Don-vip

:)

comment:8 Changed 6 months ago by bastiK

Commits to ELI come at least weekly, almost daily, many of which require JOSM wiki update. Who feels responsible for ELI synchronization? If no one does, we should disable this test.

Tests should not fail for a long period of time. The person responsible should take note and make an effort to fix it, so the broken tests do not pile up like last month.
https://josm.openstreetmap.de/jenkins/job/JOSM/jdk=JDK8/test/trend

comment:9 Changed 6 months ago by stoecker

While that comment in general is right, here is the wrong target. The ELI issues have been synced mostly by Klumbumbus and sometimes me usually very fast. Only the bunch of 24 Brazilian imagery entries took some days longer.

comment:10 Changed 6 months ago by bastiK

Okay, fair enough.

comment:11 Changed 6 months ago by stoecker

BTW: It would be much easier if the quality of the entries would be good enough to simply copy new entries...

comment:12 Changed 6 months ago by Klumbumbus

I usually try to fix the sync issues quickly. I was a bit demotivated to add the 24 brazilian layers as adding the two czech entries before took me very long (one or two hours), so I let the brazilian layers wait some days :)

Is there a RSS feed or email notification for single Jenkins tests? Using the whole Jenkins RSS feed (https://josm.openstreetmap.de/jenkins/job/JOSM/rssFailed) doesn't really help me as most of the time Jenkins fails anyway as shown in the graphic above.

comment:13 in reply to:  12 Changed 6 months ago by bastiK

Replying to Klumbumbus:

Is there a RSS feed or email notification for single Jenkins tests? Using the whole Jenkins RSS feed (https://josm.openstreetmap.de/jenkins/job/JOSM/rssFailed) doesn't really help me as most of the time Jenkins fails anyway as shown in the graphic above.

Possibly by making it a separate project in Jenkins?

comment:14 Changed 6 months ago by Don-vip

I was thinking about it. It would make sens to extract the integration tests from our main job. The tests that fail very often because of external factor are:

  • plugins integration test
  • IANA TLD integration test
  • ELI synchronization test
  • presets integration test
  • map paint styles integration test
  • taginfo popular tags test

Changed 6 months ago by Don-vip

Attachment: wow.PNG added

comment:15 Changed 6 months ago by Don-vip

@Klumbumbus: Don't despair, you can fix all of them!


comment:16 Changed 6 months ago by stoecker

Hey. Don't spread debug info ... BTW the first number were blue entries (no reds ATM) and the second the data age of the last test :-)

@Klumbumbus: Well, now you should get such an message when there are red entries. Data is updated once a day or when the compare page is called (which is done by the test I think).

@Don-vip, bastiK: As TRAC_ADMINs get all permissions you also get that note if I find no way to check individual permission settings beside the TRAC_ADMIN.

That information could be expanded for other status stuff as well. Ideas?

comment:17 Changed 6 months ago by Klumbumbus

Nice :)

As I'm also interested when some of the other jenkins tests fail, I think I found a solution to monitor individual jenkins test results via https://addons.mozilla.org/en-us/firefox/addon/alertbox/

So if this works as expected (it will take some time to test) the trac message could be removed again if it bothers other team members.

comment:18 Changed 6 months ago by Don-vip

OK done:

  • JOSM job now only runs real unit tests. By real I mean tests that should not fail unless we have changed some Java code.
  • JOSM-Integration job runs integration tests that fail frequently depending on the external changes in OSM ecosystem. It will run every 6 hours and should be fast.

comment:19 in reply to:  18 ; Changed 6 months ago by bastiK

Replying to Don-vip:

OK done:

  • JOSM job now only runs real unit tests. By real I mean tests that should not fail unless we have changed some Java code.
  • JOSM-Integration job runs integration tests that fail frequently depending on the external changes in OSM ecosystem. It will run every 6 hours and should be fast.

Thanks, this is a great improvement!

comment:20 Changed 6 months ago by stoecker

Aaargh. Again 11 differences and again many errors in ELI - Actually the argument that ELI has peer review is void. There is NO peer review. Broken PRs are simply accepted.

comment:21 Changed 6 months ago by Klumbumbus

@stoecker: regarding

Unjustified large zoom
ELI 	* Maxzoom differs (20 != 19): [CH] Kanton Aargau 25cm (AGIS 2014) - http://mapproxy.osm.ch:8080/tiles/AGIS2014/EPSG900913/{zoom}/{x}/{y}.png?origin=nw
ELI 	* Maxzoom differs (20 != 19): [CH] Kanton Aargau 25cm (AGIS 2016) - http://mapproxy.osm.ch:8080/tiles/AGIS2016/EPSG900913/{zoom}/{x}/{y}.png?origin=nw

according to https://github.com/osmlab/editor-layer-index/pull/308#issuecomment-295737054 the base zoom level is z20

comment:22 in reply to:  21 Changed 6 months ago by stoecker

according to https://github.com/osmlab/editor-layer-index/pull/308#issuecomment-295737054 the base zoom level is z20

Then maybe he made an error in server setup? All places I tried zoom 20 is a simple upscale of 19.

comment:23 Changed 6 months ago by stoecker

@Klumbumbus:

I've seen that the double point I added as ignore was in our data and you fixed it. I though I got rid of them already, that's why I was sure it is an ignore :-)

In r11964 I added another sanity check for the double points and also a first step to get proper formatted XML directly from the GeoJSON, so that an "ant imageryindexdownload" in SVN produces a file directly usable for Copy&Paste. Currently contains only name, id and the shape/bounds, which usually need most handwork. Will add the other parts at the weekend.

comment:24 Changed 6 months ago by Klumbumbus

👍

comment:25 Changed 6 months ago by stoecker

In 11975/josm:

see #14655 - implement nearly all features in ELI sync XML output

comment:26 Changed 6 months ago by stoecker

As I added code to read most properties now anyway - Should I add any additional ELI<->JOSM checks for not yet checked properties? URLs, projections, ...?

comment:27 Changed 6 months ago by Don-vip

In 11977/josm:

see #14655 - fix javadoc warnings

comment:28 Changed 6 months ago by Klumbumbus

projections and description would be nice. All the license/terms_of_use/attribution URLs would be important but I think thats not really possible atm as josm and ELI has different properties.

comment:29 Changed 6 months ago by stoecker

In 11981/josm:

see #14655 - add more ELI - JOSM checks, new results comment by SKIP text (does not trigger error notes)

comment:30 Changed 6 months ago by stoecker

Well, a lot of new output, all marked as SKIP to silence the checks.

Permission-ref and terms-of-use both map to license_url. I'm not sure how the links to generic CC-BY-SA and alike should be handled. Plain text in license_url sounds like "Ignore" for me.

comment:31 in reply to:  23 Changed 6 months ago by Klumbumbus

Replying to stoecker:

proper formatted XML directly from the GeoJSON, so that an "ant imageryindexdownload" in SVN produces a file directly usable for Copy&Paste.

Is this file already somewhere available or do I need to create it myself?

comment:32 Changed 6 months ago by stoecker

You need to call "ant imageryindexdownload" or "int imageryindex" (no new download) in SVN. File is not automatically created somewhere. I don't plan to release it: Neither the ELI not the JOSM output is fully complete.

P.S. When copying the descriptions from ELI I think they should be cleaned. E.g. the imagico entries have ugly information in there, which is not really for human consumption.

comment:33 in reply to:  8 Changed 6 months ago by Klumbumbus

Replying to bastiK:

https://josm.openstreetmap.de/jenkins/job/JOSM/jdk=JDK8/test/trend

This graph is gone at Jenkins :(

Last edited 6 months ago by Klumbumbus (previous) (diff)

comment:34 Changed 6 months ago by stoecker

@Klumbumbus:

I did some long planned changes this week:

  • The preview now handles also WMS servers not supporting EPSG:4326 and EPSG:3857.
  • There is also a new hidden feature: In the preview you can add "&epsg=" and either 1...x (for the first, second, ... entry) or the epsg number to choose a specific ESPG code to the URL. This allows testing if all EPSGs work on the server.

comment:35 in reply to:  23 Changed 6 months ago by Klumbumbus

Replying to stoecker:

"ant imageryindexdownload" in SVN produces a file directly usable for Copy&Paste.

I need a little help here. What I already did:

  • install ant
  • install jdk
  • install groovy
  • added the environment variables for these three
  • install wget

when calling ant imageryindexdownload in my josm folder I get the following error. Also doing a new checkout via svn co https://josm.openstreetmap.de/svn/trunk josm (as suggested here) didn't help. I was not asked to accept a certificate.

C:\Users\stefa\Documents\OSM\josm>ant imageryindexdownload
Buildfile: C:\Users\stefa\Documents\OSM\josm\build.xml

imageryindexdownload:
     [exec] --2017-05-02 22:14:48--  https://josm.openstreetmap.de/maps
     [exec] Resolving josm.openstreetmap.de (josm.openstreetmap.de)... 78.47.65.34
     [exec] Connecting to josm.openstreetmap.de (josm.openstreetmap.de)|78.47.65.34|:443... connected.
     [exec] ERROR: cannot verify josm.openstreetmap.de's certificate, issued by 'CN=Let\'s Encrypt Authority X3,O=Let\'s Encrypt,C=US':
     [exec]   Self-signed certificate encountered.
     [exec] To connect to josm.openstreetmap.de insecurely, use `--no-check-certificate'.
     [exec] Result: 5
     [exec] --2017-05-02 22:14:48--  https://josm.openstreetmap.de/wiki/ImageryCompareIgnores?format=txt
     [exec] Resolving josm.openstreetmap.de (josm.openstreetmap.de)... 78.47.65.34
     [exec] Connecting to josm.openstreetmap.de (josm.openstreetmap.de)|78.47.65.34|:443... connected.
     [exec] ERROR: cannot verify josm.openstreetmap.de's certificate, issued by 'CN=Let\'s Encrypt Authority X3,O=Let\'s Encrypt,C=US':
     [exec]   Self-signed certificate encountered.
     [exec] To connect to josm.openstreetmap.de insecurely, use `--no-check-certificate'.
     [exec] Result: 5
     [exec] --2017-05-02 22:14:49--  https://raw.githubusercontent.com/osmlab/editor-layer-index/gh-pages/imagery.geojson
     [exec] Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.12.133
     [exec] Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.12.133|:443... connected.
     [exec] ERROR: cannot verify raw.githubusercontent.com's certificate, issued by 'CN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US':
     [exec]   Unable to locally verify the issuer's authority.
     [exec] To connect to raw.githubusercontent.com insecurely, use `--no-check-certificate'.
     [exec] Result: 5

init-properties:

imageryindex:
     [echo] Checking editor imagery difference

BUILD FAILED
C:\Users\stefa\Documents\OSM\josm\build.xml:730: The following error occurred while executing this line:
C:\Users\stefa\Documents\OSM\josm\build.xml:702: Script Failed: org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
C:\Users\stefa\Documents\OSM\josm\scripts\SyncEditorLayerIndex.groovy: 21: unable to resolve class javax.json.JsonArray
 @ line 21, column 1.
   import javax.json.JsonArray
   ^

C:\Users\stefa\Documents\OSM\josm\scripts\SyncEditorLayerIndex.groovy: 26: unable to resolve class org.openstreetmap.josm.data.imagery.Shape
 @ line 26, column 1.
   import org.openstreetmap.josm.data.imagery.Shape
   ^

C:\Users\stefa\Documents\OSM\josm\scripts\SyncEditorLayerIndex.groovy: 27: unable to resolve class org.openstreetmap.josm.io.imagery.ImageryReader
 @ line 27, column 1.
   import org.openstreetmap.josm.io.imagery.ImageryReader
   ^

C:\Users\stefa\Documents\OSM\josm\scripts\SyncEditorLayerIndex.groovy: 22: unable to resolve class javax.json.JsonObject
 @ line 22, column 1.
   import javax.json.JsonObject
   ^

C:\Users\stefa\Documents\OSM\josm\scripts\SyncEditorLayerIndex.groovy: 23: unable to resolve class javax.json.JsonReader
 @ line 23, column 1.
   import javax.json.JsonReader
   ^

C:\Users\stefa\Documents\OSM\josm\scripts\SyncEditorLayerIndex.groovy: 20: unable to resolve class javax.json.Json
 @ line 20, column 1.
   import javax.json.Json
   ^

C:\Users\stefa\Documents\OSM\josm\scripts\SyncEditorLayerIndex.groovy: 25: unable to resolve class org.openstreetmap.josm.data.imagery.ImageryInfo
 @ line 25, column 1.
   import org.openstreetmap.josm.data.imagery.ImageryInfo
   ^

7 errors



Total time: 2 seconds
URL:http://josm.openstreetmap.de/svn/trunk
Repository:UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
Last:Changed Date: 2017-05-02 12:26:59 +0200 (Tue, 02 May 2017)
Build-Date:2017-05-02 18:19:23
Revision:12038
Relative:URL: ^/trunk

Identification: JOSM/1.5 (12038 de) Windows 10 64-Bit
Memory Usage: 1808 MB / 3641 MB (1289 MB allocated, but free)
Java version: 1.8.0_131-b11, Oracle Corporation, Java HotSpot(TM) 64-Bit Server VM
Screen: \Display0 1680x1050
Maximum Screen Size: 1680x1050
VM arguments: [-Djava.security.manager, -Djava.security.policy=file:%JAVA_HOME%\jre\lib\security\javaws.policy, -DtrustProxy=true, -Djnlpx.home=%JAVA_HOME%\jre\bin, -Djnlpx.origFilenameArg=C:\Program Files (x86)\josm-latest-mehr-RAM.jnlp, -Djnlpx.remove=true, -Djava.util.Arrays.useLegacyMergeSort=true, -Djnlpx.heapsize=1024m,4096m, -Djnlpx.splashport=58499, -Djnlpx.jvm=%JAVA_HOME%\jre\bin\javaw.exe]

comment:36 Changed 6 months ago by Don-vip

in your shell can you please post the output for the following commands:

ant -version
java -version
javac -version
echo %JAVA_HOME%

You should have something like:

>ant -version
Apache Ant(TM) version 1.9.7 compiled on April 9 2016

>java -version
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

>javac -version
javac 1.8.0_131

>echo %JAVA_HOME%
C:\Program Files\Java\jdk1.8.0_131
Last edited 6 months ago by Don-vip (previous) (diff)

comment:37 Changed 6 months ago by Don-vip

Once everything is setup correctly, running ant dist imageryindexdownload should give you this:

...
dist:
     [echo] Revision 12039
     [copy] Copying 1 file to C:\SVN\josm\core\build
     [copy] Copying 1 file to C:\SVN\josm\core\build
     [copy] Copying 1 file to C:\SVN\josm\core\build
      [jar] Building jar: C:\SVN\josm\core\dist\josm-custom.jar
imageryindexdownload:
     [exec] --2017-05-02 23:56:15--  https://josm.openstreetmap.de/maps
     [exec] Resolving josm.openstreetmap.de (josm.openstreetmap.de)... 2a01:4f8:c17:2906::2, 78.47.65.34
     [exec] Connecting to josm.openstreetmap.de (josm.openstreetmap.de)|2a01:4f8:c17:2906::2|:443... connected.
     [exec] HTTP request sent, awaiting response... 200 OK
     [exec] Length: 2825028 (2.7M) [text/xml]
     [exec] Saving to: 'imagery_josm.imagery.xml'
     [exec]      0K .......... .......... .......... .......... ..........  1%  692K 4s
...
     [exec]   2750K ........                                              100% 79.8M=0.5s
     [exec] 2017-05-02 23:56:15 (5.86 MB/s) - 'imagery_josm.imagery.xml' saved [2825028/2825028]
     [exec] --2017-05-02 23:56:16--  https://josm.openstreetmap.de/wiki/ImageryCompareIgnores?format=txt
     [exec] Resolving josm.openstreetmap.de (josm.openstreetmap.de)... 2a01:4f8:c17:2906::2, 78.47.65.34
     [exec] Connecting to josm.openstreetmap.de (josm.openstreetmap.de)|2a01:4f8:c17:2906::2|:443... connected.
     [exec] HTTP request sent, awaiting response... 200 Ok
     [exec] Length: 27208 (27K) [text/plain]
     [exec] Saving to: 'imagery_josm.ignores.txt'
     [exec]      0K .......... .......... ......                          100%  745K=0.04s
     [exec] 2017-05-02 23:56:16 (745 KB/s) - 'imagery_josm.ignores.txt' saved [27208/27208]
     [exec] --2017-05-02 23:56:16--  https://raw.githubusercontent.com/osmlab/editor-layer-index/gh-pages/imagery.geojson
     [exec] Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.128.133, 151.101.0.133, 151.101.192.133, ...
     [exec] Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.128.133|:443... connected.
     [exec] HTTP request sent, awaiting response... 200 OK
     [exec] Length: 4530932 (4.3M) [text/plain]
     [exec] Saving to: 'imagery_eli.geojson'
     [exec]      0K .......... .......... .......... .......... ..........  1% 1.22M 3s
...
     [exec]   4400K .......... .......... ....                            100% 26.2M=0.6s
     [exec] 2017-05-02 23:56:17 (6.65 MB/s) - 'imagery_eli.geojson' saved [4530932/4530932]
init-properties:
imageryindex:
     [echo] Checking editor imagery difference
   [groovy] 2017-05-02 23:56:18.390 FINE: Starting SAX parsing of org.xml.sax.InputSource@4c3ac34 using org.openstreetmap.josm.io.imagery.ImageryReader$Parser@64e2cd66
   [groovy] 2017-05-02 23:56:18.602 FINE: SAX parsing done in 274 ms
   [groovy] *** Loaded 522 entries (JOSM). ***
   [groovy] *** Loaded 378 entries (ELI). ***
   [groovy] *** URLs found in ELI but not in JOSM (12): ***
   [groovy] *** URLs found in JOSM but not in ELI (187): ***
   [groovy] *** Same URL, but different name: ***
   [groovy] *** Same URL, but different type: ***
   [groovy] *** Same URL, but different zoom bounds: ***
   [groovy] *** Same URL, but different country code: ***
   [groovy] *** Same URL, but different quality: ***
   [groovy] *** Same URL, but different dates: ***
   [groovy] *** Same URL, but different information: ***
   [groovy] *** Mismatching shapes: ***
   [groovy] *** Mismatching icons: ***
   [groovy] *** Miscellaneous checks: ***
BUILD SUCCESSFUL
Total time: 1 minute 2 seconds

comment:38 Changed 6 months ago by Klumbumbus

JRE setup was not right, I fixed it and now I get the following version output:

C:\Users\stefa>ant -version
Apache Ant(TM) version 1.10.1 compiled on February 2 2017

C:\Users\stefa>java -version
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

C:\Users\stefa>javac -version
javac 1.8.0_131

C:\Users\stefa>echo %JAVA_HOME%
C:\Program Files\Java\jdk1.8.0_131

However ant imageryindexdownload still gives the same error.

ant dist imageryindexdownload produces:

C:\Users\stefa\Documents\OSM\josm>ant dist imageryindexdownload
Buildfile: C:\Users\stefa\Documents\OSM\josm\build.xml

init-properties:

init:
    [mkdir] Created dir: C:\Users\stefa\Documents\OSM\josm\build
    [mkdir] Created dir: C:\Users\stefa\Documents\OSM\josm\dist

javacc:
    [mkdir] Created dir: C:\Users\stefa\Documents\OSM\josm\src\org\openstreetmap\josm\gui\mappaint\mapcss\parsergen
     [java] Warning: Line 4, Column 3: Command line setting of "OUTPUT_DIRECTORY" modifies option value in file.
     [java] Java Compiler Compiler Version 7.0.2 (Parser Generator)
     [java] (type "javacc" with no arguments for help)
     [java] Reading from file C:\Users\stefa\Documents\OSM\josm\src\org\openstreetmap\josm\gui\mappaint\mapcss/MapCSSParser.jj . . .
     [java] Warning: Line 169, Column 5: Non-ASCII characters used in regular expression.
     [java] Please make sure you use the correct Reader when you create the parser, one that can handle your character set.
     [java] File "TokenMgrError.java" does not exist.  Will create one.
     [java] File "ParseException.java" does not exist.  Will create one.
     [java] File "Token.java" does not exist.  Will create one.
     [java] File "SimpleCharStream.java" does not exist.  Will create one.
     [java] Parser generated with 0 errors and 2 warnings.

compile:
    [javac] Compiling 378 source files to C:\Users\stefa\Documents\OSM\josm\build
    [javac] C:\Users\stefa\Documents\OSM\josm\src\org\apache\commons\compress\compressors\brotli\BrotliCompressorInputStream.java:34: error: package org.brotli.dec does not exist
    [javac]     private final org.brotli.dec.BrotliInputStream decIS;
    [javac]                                 ^
    [javac] C:\Users\stefa\Documents\OSM\josm\src\org\apache\commons\compress\compressors\brotli\BrotliCompressorInputStream.java:37: error: package org.brotli.dec does not exist
    [javac]         this.decIS = new org.brotli.dec.BrotliInputStream(in);
    [javac]                                        ^
    [javac] Note: C:\Users\stefa\Documents\OSM\josm\src\oauth\signpost\AbstractOAuthProvider.java uses or overrides a deprecated API.
    [javac] Note: Recompile with -Xlint:deprecation for details.
    [javac] Note: Some input files use unchecked or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked for details.
    [javac] 2 errors

BUILD FAILED
C:\Users\stefa\Documents\OSM\josm\build.xml:228: Compile failed; see the compiler error output for details.

Total time: 4 seconds

comment:39 Changed 6 months ago by Klumbumbus

OK, after updating my svn working copy to r12039 I'm one step further. ant dist imageryindexdownload produces:

C:\Users\stefa\Documents\OSM\josm>ant dist imageryindexdownload
Buildfile: C:\Users\stefa\Documents\OSM\josm\build.xml

init-properties:

init:

javacc:

compile:
    [javac] Compiling 376 source files to C:\Users\stefa\Documents\OSM\josm\build
    [javac] Note: C:\Users\stefa\Documents\OSM\josm\src\oauth\signpost\AbstractOAuthProvider.java uses or overrides a deprecated API.
    [javac] Note: Recompile with -Xlint:deprecation for details.
    [javac] Note: Some input files use unchecked or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked for details.
    [javac] Compiling 54 source files to C:\Users\stefa\Documents\OSM\josm\build
    [javac] Compiling 1487 source files to C:\Users\stefa\Documents\OSM\josm\build
    [javac] C:\Users\stefa\Documents\OSM\josm\src\org\openstreetmap\josm\gui\MapView.java:497: warning: [deprecation] isChanged() in Layer has been deprecated
    [javac]             if (l.isChanged() || invalidated.contains(l)) {
    [javac]                  ^
    [javac] C:\Users\stefa\Documents\OSM\josm\src\org\openstreetmap\josm\gui\layer\GpxLayer.java:259: warning: [deprecation] isChanged() in Layer has been deprecated
    [javac]     public boolean isChanged() {
    [javac]                    ^
    [javac] C:\Users\stefa\Documents\OSM\josm\src\org\openstreetmap\josm\gui\layer\AbstractTileSourceLayer.java:1682: warning: [deprecation] isChanged() in Layer has been deprecated
    [javac]     public boolean isChanged() {
    [javac]                    ^
    [javac] 3 warnings
     [copy] Copying 1 file to C:\Users\stefa\Documents\OSM\josm\build

init-svn-revision-xml:
     [exec] Execute failed: java.io.IOException: Cannot run program "svn" (in directory "C:\Users\stefa\Documents\OSM\josm"): CreateProcess error=2, Das System kann die angegebene Datei nicht finden

init-git-revision-xml:
     [exec] Execute failed: java.io.IOException: Cannot run program "git" (in directory "C:\Users\stefa\Documents\OSM\josm"): CreateProcess error=2, Das System kann die angegebene Datei nicht finden
[replaceregexp] The following file is missing: 'C:\Users\stefa\Documents\OSM\josm\REVISION.XML'

create-revision:

check-schemas:

epsg-compile:
    [mkdir] Created dir: C:\Users\stefa\Documents\OSM\josm\build2
    [javac] Compiling 1 source file to C:\Users\stefa\Documents\OSM\josm\build2

epsg:
    [touch] Creating C:\Users\stefa\Documents\OSM\josm\data\projection\custom-epsg

dist:
     [echo] Revision UNKNOWN
     [copy] Copying 1 file to C:\Users\stefa\Documents\OSM\josm\build
     [copy] Copying 1 file to C:\Users\stefa\Documents\OSM\josm\build
     [copy] Copying 1 file to C:\Users\stefa\Documents\OSM\josm\build
      [jar] Building jar: C:\Users\stefa\Documents\OSM\josm\dist\josm-custom.jar

imageryindexdownload:
     [exec] --2017-05-03 00:12:07--  https://josm.openstreetmap.de/maps
     [exec] Resolving josm.openstreetmap.de (josm.openstreetmap.de)... 78.47.65.34
     [exec] Connecting to josm.openstreetmap.de (josm.openstreetmap.de)|78.47.65.34|:443... connected.
     [exec] ERROR: cannot verify josm.openstreetmap.de's certificate, issued by 'CN=Let\'s Encrypt Authority X3,O=Let\'s Encrypt,C=US':
     [exec]   Self-signed certificate encountered.
     [exec] To connect to josm.openstreetmap.de insecurely, use `--no-check-certificate'.
     [exec] Result: 5
     [exec] --2017-05-03 00:12:07--  https://josm.openstreetmap.de/wiki/ImageryCompareIgnores?format=txt
     [exec] Resolving josm.openstreetmap.de (josm.openstreetmap.de)... 78.47.65.34
     [exec] Connecting to josm.openstreetmap.de (josm.openstreetmap.de)|78.47.65.34|:443... connected.
     [exec] ERROR: cannot verify josm.openstreetmap.de's certificate, issued by 'CN=Let\'s Encrypt Authority X3,O=Let\'s Encrypt,C=US':
     [exec]   Self-signed certificate encountered.
     [exec] To connect to josm.openstreetmap.de insecurely, use `--no-check-certificate'.
     [exec] Result: 5
     [exec] --2017-05-03 00:12:08--  https://raw.githubusercontent.com/osmlab/editor-layer-index/gh-pages/imagery.geojson
     [exec] Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.12.133
     [exec] Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.12.133|:443... connected.
     [exec] ERROR: cannot verify raw.githubusercontent.com's certificate, issued by 'CN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US':
     [exec]   Unable to locally verify the issuer's authority.
     [exec] To connect to raw.githubusercontent.com insecurely, use `--no-check-certificate'.
     [exec] Result: 5

init-properties:

imageryindex:
     [echo] Checking editor imagery difference
   [groovy] 2017-05-03 00:12:09.992 FINE: Starting SAX parsing of org.xml.sax.InputSource@72b8b063 using org.openstreetmap.josm.io.imagery.ImageryReader$Parser@439e19a0

BUILD FAILED
C:\Users\stefa\Documents\OSM\josm\build.xml:731: The following error occurred while executing this line:
C:\Users\stefa\Documents\OSM\josm\build.xml:703: org.codehaus.groovy.runtime.InvokerInvocationException: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Premature end of file.
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:100)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:810)
        at SyncEditorLayerIndex.invokeMethod(SyncEditorLayerIndex.groovy)
        at org.codehaus.groovy.runtime.InvokerHelper$1.run(InvokerHelper.java:452)
        at org.codehaus.groovy.ant.Groovy.parseAndRunScript(Groovy.java:491)
        at org.codehaus.groovy.ant.Groovy.execGroovy(Groovy.java:438)
        at org.codehaus.groovy.ant.Groovy.execute(Groovy.java:303)
        at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:293)
        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
        at org.apache.tools.ant.Task.perform(Task.java:348)
        at org.apache.tools.ant.Target.execute(Target.java:435)
        at org.apache.tools.ant.Target.performTasks(Target.java:456)
        at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1405)
        at org.apache.tools.ant.helper.SingleCheckExecutor.executeTargets(SingleCheckExecutor.java:38)
        at org.apache.tools.ant.Project.executeTargets(Project.java:1260)
        at org.apache.tools.ant.taskdefs.Ant.execute(Ant.java:441)
        at org.apache.tools.ant.taskdefs.CallTarget.execute(CallTarget.java:105)
        at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:293)
        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
        at org.apache.tools.ant.Task.perform(Task.java:348)
        at org.apache.tools.ant.Target.execute(Target.java:435)
        at org.apache.tools.ant.Target.performTasks(Target.java:456)
        at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1405)
        at org.apache.tools.ant.Project.executeTarget(Project.java:1376)
        at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
        at org.apache.tools.ant.Project.executeTargets(Project.java:1260)
        at org.apache.tools.ant.Main.runBuild(Main.java:857)
        at org.apache.tools.ant.Main.startAnt(Main.java:236)
        at org.apache.tools.ant.launch.Launcher.run(Launcher.java:287)
        at org.apache.tools.ant.launch.Launcher.main(Launcher.java:113)
Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Premature end of file.
        at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:203)
        at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
        at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:400)
        at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327)
        at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1472)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:1014)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:602)
        at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:112)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:505)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:841)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:770)
        at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
        at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
        at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643)
        at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:327)
        at org.openstreetmap.josm.tools.Utils.parseSafeSAX(Utils.java:1355)
        at org.openstreetmap.josm.io.imagery.ImageryReader.parse(ImageryReader.java:88)
        at org.openstreetmap.josm.io.imagery.ImageryReader$parse.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:117)
        at SyncEditorLayerIndex.loadJosmEntries(SyncEditorLayerIndex.groovy:299)
        at SyncEditorLayerIndex$loadJosmEntries$2.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:117)
        at SyncEditorLayerIndex.main(SyncEditorLayerIndex.groovy:56)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
        ... 38 more

Total time: 1 minute 9 seconds

comment:40 Changed 6 months ago by Don-vip

You should make sure your SVN client (TortoiseSVN I assume) is in the PATH. Mine contains C:\Program Files\TortoiseSVN\bin so svn can be called from command line.

comment:41 Changed 6 months ago by Don-vip

Also it seems there's a problem with your wget. Where did you install it from? I use this one.

comment:42 Changed 6 months ago by Klumbumbus

I fiddled a bit around with subversion and wget and now it works. :) Thanks for your help!

comment:43 Changed 6 months ago by Klumbumbus

There is one little problem left. special characters are not handles correctly. see script output extract below. Also the generated xml files have strange chars included.

init-properties:

imageryindex:
     [echo] Checking editor imagery difference
   [groovy] 2017-05-03 21:34:51.340 FINE: Starting SAX parsing of org.xml.sax.InputSource@16fc5622 using org.openstreetmap.josm.io.imagery.ImageryReader$Parser@516c20de
   [groovy] 2017-05-03 21:34:51.605 FINE: SAX parsing done in 350 ms
   [groovy] *** Loaded 531 entries (JOSM). ***
   [groovy] *** Loaded 387 entries (ELI). ***
   [groovy] *** URLs found in ELI but not in JOSM (11): ***
   [groovy] *** URLs found in JOSM but not in ELI (186): ***
   [groovy] *** Same URL, but different name: ***
   [groovy] * Name differs ('Kanton Z³rich 2015  10cm' != 'Kanton Z³rich Orthophotos 2014/2015 10cm'): http://mapproxy.osm.ch:8080/tiles/KTZUERICH2015/EPSG900913/{zoom}/{x}/{y}.png?origin=nw
   [groovy] * Name differs ('Lithuania - NÄT ORT10LT' != 'ORT10LT (Lithuania)'): http://mapproxy.openmap.lt/ort10lt/g/{zoom}/{x}/{y}.jpeg
   [groovy] * Name differs ('Olho d'Agua do Casado AL' != 'Olho dÆAgua do Casado AL'): http://geo.seplande.al.gov.br/teogc/terraogcwms.cgi?LAYERS=Olho%20Dagua%20do%20Casado&SERVICE=WMS&FORMAT=image/png&TRANSPARENT=TRUE&VERSION=1.1.0&SERVICE=WMS&REQUEST=GetMap&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
   [groovy] * Name differs ('S¾sk·t, Pusztazßmor, Tßrnok, Di¾sd ortophoto 2017' != 'S¾sk·t, Pusztazßmor, Tßrnok, Di¾sd orthophoto 2017'): http://adam.openstreetmap.hu/mapproxy/tiles/1.0.0/Soskut-Tarnok-Pusztazamor-Diosd/mercator/{zoom}/{x}/{y}.png
   [groovy] * Name differs ('▄bersichtsplan Z³rich' != 'Stadt Z³rich ▄bersichtsplan (Strassen, Gebõude, Hausnummer)'): http://mapproxy.sosm.ch:8080/tiles/zh_uebersichtsplan/EPSG900913/{zoom}/{x}/{y}.png?origin=nw
   [groovy] *** Same URL, but different type: ***
   [groovy] *** Same URL, but different zoom bounds: ***
   [groovy] * Minzoom differs (null != 11): [HU] S¾sk·t, Pusztazßmor, Tßrnok, Di¾sd orthophoto 2017 - http://adam.openstreetmap.hu/mapproxy/tiles/1.0.0/Soskut-Tarnok-Pusztazamor-Diosd/mercator/{zoom}/{x}/{y}.png
   [groovy] * Maxzoom differs (null != 20): [HU] S¾sk·t, Pusztazßmor, Tßrnok, Di¾sd orthophoto 2017 - http://adam.openstreetmap.hu/mapproxy/tiles/1.0.0/Soskut-Tarnok-Pusztazamor-Diosd/mercator/{zoom}/{x}/{y}.png
   [groovy] *** Same URL, but different country code: ***
   [groovy] *** Same URL, but different quality: ***
   [groovy] *** Same URL, but different dates: ***
   [groovy] *** Same URL, but different information: ***
   [groovy] *** Mismatching shapes: ***
   [groovy] * Different coordinate for point 1 of shape 1: [CH] Stadt Z³rich Luftbild 2011 - http://mapproxy.sosm.ch:8080/tiles/zh_luftbild2011/EPSG900913/{zoom}/{x}/{y}.png?origin=nw
   [groovy] * Different number of points for shape 1 (108 ! = 495)): [CH] Kanton Z³rich Orthophotos 2014/2015 10cm - http://mapproxy.osm.ch:8080/tiles/KTZUERICH2015/EPSG900913/{zoom}/{x}/{y}.png?origin=nw
   [groovy] * Different number of shapes (1 != 3): [CH] Canton de GÚnÞve 5cm (SITG 2011) - http://etat.geneve.ch/ags2/services/Orthophotos_2011/MapServer/WMSServer?FORMAT=image/jpeg&VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&Layers=0&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
   [groovy] * Different number of points for shape 1 (5 ! = 474)): [CH] Stadtplan Z³rich - http://mapproxy.sosm.ch:8080/tiles/zh_stadtplan/EPSG900913/{zoom}/{x}/{y}.png?origin=nw
   [groovy] * Different number of points for shape 1 (5 ! = 167)): [CH] Stadt Z³rich ▄bersichtsplan (Strassen, Gebõude, Hausnummer) - http://mapproxy.sosm.ch:8080/tiles/zh_uebersichtsplan/EPSG900913/{zoom}/{x}/{y}.png?origin=nw
   [groovy] *** Mismatching icons: ***
   [groovy] *** Miscellaneous checks: ***
   [groovy] +++ Obsolete skip entry: * Name differs ('Lithuania - NśT ORT10LT' != 'ORT10LT (Lithuania)'): http://mapproxy.openmap.lt/ort10lt/g/{zoom}/{x}/{y}.jpeg
   [groovy] +++ Obsolete skip entry: * Name differs ('Übersichtsplan Zürich' != 'Stadt Zürich Übersichtsplan (Strassen, Gebäude, Hausnummer)'): http://mapproxy.sosm.ch:8080/tiles/zh_uebersichtsplan/EPSG900913/{zoom}/{x}/{y}.png?origin=nw
   [groovy] +++ Obsolete skip entry: * Name differs ('Kanton Z├╝rich 2015  10cm' != 'Kanton Z├╝rich Orthophotos 2014/2015 10cm'): http://mapproxy.osm.ch:8080/tiles/KTZUERICH2015/EPSG900913/{zoom}/{x}/{y}.png?origin=nw
   [groovy] +++ Obsolete skip entry: * Name differs ('Sóskút, Pusztazámor, Tárnok, Diósd ortophoto 2017' != 'Sóskút, Pusztazámor, Tárnok, Diósd orthophoto 2017'): http://adam.openstreetmap.hu/mapproxy/tiles/1.0.0/Soskut-Tarnok-Pusztazamor-Diosd/mercator/{zoom}/{x}/{y}.png
   [groovy] +++ Obsolete skip entry: +  [AT] VoGIS: H├Âhen und Gel├ñnde - http://vogis.cnv.at/mapserver/mapserv?map=i_hoehen_und_gelaende_r_wms.map
   [groovy] +++ Obsolete skip entry: * Different number of points for shape 1 (5 ! = 474)): [CH] Stadtplan Z├╝rich - http://mapproxy.sosm.ch:8080/tiles/zh_stadtplan/EPSG900913/{zoom}/{x}/{y}.png?origin=nw
   [groovy] +++ Obsolete skip entry: * Different number of points for shape 1 (5 ! = 167)): [CH] Stadt Zürich Übersichtsplan (Strassen, Gebäude, Hausnummer) - http://mapproxy.sosm.ch:8080/tiles/zh_uebersichtsplan/EPSG900913/{zoom}/{x}/{y}.png?origin=nw
   [groovy] +++ Obsolete skip entry: * Different number of shapes (1 != 3): [CH] Canton de G├®n├¿ve 5cm (SITG 2011) - http://etat.geneve.ch/ags2/services/Orthophotos_2011/MapServer/WMSServer?FORMAT=image/jpeg&VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&Layers=0&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
   [groovy] +++ Obsolete skip entry: * Different coordinate for point 1 of shape 1: [CH] Stadt Z├╝rich Luftbild 2011 - http://mapproxy.sosm.ch:8080/tiles/zh_luftbild2011/EPSG900913/{zoom}/{x}/{y}.png?origin=nw
   [groovy] +++ Obsolete skip entry: * Different number of points for shape 1 (108 ! = 495)): [CH] Kanton Z├╝rich Orthophotos 2014/2015 10cm - http://mapproxy.osm.ch:8080/tiles/KTZUERICH2015/EPSG900913/{zoom}/{x}/{y}.png?origin=nw
   [groovy] +++ Obsolete skip entry: * Minzoom differs (null != 11): [HU] Sóskút, Pusztazámor, Tárnok, Diósd orthophoto 2017 - http://adam.openstreetmap.hu/mapproxy/tiles/1.0.0/Soskut-Tarnok-Pusztazamor-Diosd/mercator/{zoom}/{x}/{y}.png
   [groovy] +++ Obsolete skip entry: * Maxzoom differs (null != 20): [HU] Sóskút, Pusztazámor, Tárnok, Diósd orthophoto 2017 - http://adam.openstreetmap.hu/mapproxy/tiles/1.0.0/Soskut-Tarnok-Pusztazamor-Diosd/mercator/{zoom}/{x}/{y}.png

BUILD SUCCESSFUL
Total time: 52 seconds

comment:45 Changed 6 months ago by Klumbumbus

It doesn't work completely. I changed font to lucida console and entered chcp 65001
In the first part the characters are still wrong, which produces the "obsolete Skip" entries, while there the characters are right

imageryindex:
     [echo] Checking editor imagery difference
   [groovy] 2017-05-04 22:14:55.335 FINE: Starting SAX parsing of org.xml.sax.InputSource@16fc5622 using org.openstreetmap.josm.io.imagery.ImageryReader$Parser@516c20de
   [groovy] 2017-05-04 22:14:55.816 FINE: SAX parsing done in 593 ms
   [groovy] *** Loaded 531 entries (JOSM). ***
   [groovy] *** Loaded 387 entries (ELI). ***
   [groovy] *** URLs found in ELI but not in JOSM (11): ***
   [groovy] *** URLs found in JOSM but not in ELI (186): ***
   [groovy] *** Same URL, but different name: ***
   [groovy] * Name differs ('Kanton Z�rich 2015  10cm' != 'Kanton Z�rich Orthophotos 2014/2015 10cm'): http://mapproxy.osm.ch:8080/tiles/KTZUERICH2015/EPSG900913/{zoom}/{x}/{y}.png?origin=nw
   [groovy] * Name differs ('Lithuania - N�T ORT10LT' != 'ORT10LT (Lithuania)'): http://mapproxy.openmap.lt/ort10lt/g/{zoom}/{x}/{y}.jpeg
   [groovy] * Name differs ('Olho d'Agua do Casado AL' != 'Olho d�Agua do Casado AL'): http://geo.seplande.al.gov.br/teogc/terraogcwms.cgi?LAYERS=Olho%20Dagua%20do%20Casado&SERVICE=WMS&FORMAT=image/png&TRANSPARENT=TRUE&VERSION=1.1.0&SERVICE=WMS&REQUEST=GetMap&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
   [groovy] * Name differs ('S�sk�t, Pusztaz�mor, T�rnok, Di�sd ortophoto 2017' != 'S�sk�t, Pusztaz�mor, T�rnok, Di�sd orthophoto 2017'): http://adam.openstreetmap.hu/mapproxy/tiles/1.0.0/Soskut-Tarnok-Pusztazamor-Diosd/mercator/{zoom}/{x}/{y}.png
   [groovy] * Name differs ('�bersichtsplan Z�rich' != 'Stadt Z�rich �bersichtsplan (Strassen, Geb�ude, Hausnummer)'): http://mapproxy.sosm.ch:8080/tiles/zh_uebersichtsplan/EPSG900913/{zoom}/{x}/{y}.png?origin=nw
   [groovy] *** Same URL, but different type: ***
   [groovy] *** Same URL, but different zoom bounds: ***
   [groovy] * Minzoom differs (null != 11): [HU] S�sk�t, Pusztaz�mor, T�rnok, Di�sd orthophoto 2017 - http://adam.openstreetmap.hu/mapproxy/tiles/1.0.0/Soskut-Tarnok-Pusztazamor-Diosd/mercator/{zoom}/{x}/{y}.png
   [groovy] * Maxzoom differs (null != 20): [HU] S�sk�t, Pusztaz�mor, T�rnok, Di�sd orthophoto 2017 - http://adam.openstreetmap.hu/mapproxy/tiles/1.0.0/Soskut-Tarnok-Pusztazamor-Diosd/mercator/{zoom}/{x}/{y}.png
   [groovy] *** Same URL, but different country code: ***
   [groovy] *** Same URL, but different quality: ***
   [groovy] *** Same URL, but different dates: ***
   [groovy] *** Same URL, but different information: ***
   [groovy] *** Mismatching shapes: ***
   [groovy] * Different coordinate for point 1 of shape 1: [CH] Stadt Z�rich Luftbild 2011 - http://mapproxy.sosm.ch:8080/tiles/zh_luftbild2011/EPSG900913/{zoom}/{x}/{y}.png?origin=nw
   [groovy] * Different number of points for shape 1 (108 ! = 495)): [CH] Kanton Z�rich Orthophotos 2014/2015 10cm - http://mapproxy.osm.ch:8080/tiles/KTZUERICH2015/EPSG900913/{zoom}/{x}/{y}.png?origin=nw
   [groovy] * Different number of shapes (1 != 3): [CH] Canton de G�n�ve 5cm (SITG 2011) - http://etat.geneve.ch/ags2/services/Orthophotos_2011/MapServer/WMSServer?FORMAT=image/jpeg&VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&Layers=0&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
   [groovy] * Different number of points for shape 1 (5 ! = 474)): [CH] Stadtplan Z�rich - http://mapproxy.sosm.ch:8080/tiles/zh_stadtplan/EPSG900913/{zoom}/{x}/{y}.png?origin=nw
   [groovy] * Different number of points for shape 1 (5 ! = 167)): [CH] Stadt Z�rich �bersichtsplan (Strassen, Geb�ude, Hausnummer) - http://mapproxy.sosm.ch:8080/tiles/zh_uebersichtsplan/EPSG900913/{zoom}/{x}/{y}.png?origin=nw
   [groovy] *** Mismatching icons: ***
   [groovy] *** Miscellaneous checks: ***
   [groovy] +++ Obsolete skip entry: * Name differs ('Lithuania - NŽT ORT10LT' != 'ORT10LT (Lithuania)'): http://mapproxy.openmap.lt/ort10lt/g/{zoom}/{x}/{y}.jpeg
   [groovy] +++ Obsolete skip entry: * Name differs ('Übersichtsplan Zürich' != 'Stadt Zürich Übersichtsplan (Strassen, Gebäude, Hausnummer)'): http://mapproxy.sosm.ch:8080/tiles/zh_uebersichtsplan/EPSG900913/{zoom}/{x}/{y}.png?origin=nw
   [groovy] +++ Obsolete skip entry: * Name differs ('Kanton Zürich 2015  10cm' != 'Kanton Zürich Orthophotos 2014/2015 10cm'): http://mapproxy.osm.ch:8080/tiles/KTZUERICH2015/EPSG900913/{zoom}/{x}/{y}.png?origin=nw
   [groovy] +++ Obsolete skip entry: * Name differs ('Sóskút, Pusztazámor, Tárnok, Diósd ortophoto 2017' != 'Sóskút, Pusztazámor, Tárnok, Diósd orthophoto 2017'): http://adam.openstreetmap.hu/mapproxy/tiles/1.0.0/Soskut-Tarnok-Pusztazamor-Diosd/mercator/{zoom}/{x}/{y}.png
   [groovy] +++ Obsolete skip entry: +  [AT] VoGIS: Höhen und Gelände - http://vogis.cnv.at/mapserver/mapserv?map=i_hoehen_und_gelaende_r_wms.map
   [groovy] +++ Obsolete skip entry: * Different number of points for shape 1 (5 ! = 474)): [CH] Stadtplan Zürich - http://mapproxy.sosm.ch:8080/tiles/zh_stadtplan/EPSG900913/{zoom}/{x}/{y}.png?origin=nw
   [groovy] +++ Obsolete skip entry: * Different number of points for shape 1 (5 ! = 167)): [CH] Stadt Zürich Übersichtsplan (Strassen, Gebäude, Hausnummer) - http://mapproxy.sosm.ch:8080/tiles/zh_uebersichtsplan/EPSG900913/{zoom}/{x}/{y}.png?origin=nw
   [groovy] +++ Obsolete skip entry: * Different number of shapes (1 != 3): [CH] Canton de Génève 5cm (SITG 2011) - http://etat.geneve.ch/ags2/services/Orthophotos_2011/MapServer/WMSServer?FORMAT=image/jpeg&VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&Layers=0&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
   [groovy] +++ Obsolete skip entry: * Different coordinate for point 1 of shape 1: [CH] Stadt Zürich Luftbild 2011 - http://mapproxy.sosm.ch:8080/tiles/zh_luftbild2011/EPSG900913/{zoom}/{x}/{y}.png?origin=nw
   [groovy] +++ Obsolete skip entry: * Different number of points for shape 1 (108 ! = 495)): [CH] Kanton Zürich Orthophotos 2014/2015 10cm - http://mapproxy.osm.ch:8080/tiles/KTZUERICH2015/EPSG900913/{zoom}/{x}/{y}.png?origin=nw
   [groovy] +++ Obsolete skip entry: * Minzoom differs (null != 11): [HU] Sóskút, Pusztazámor, Tárnok, Diósd orthophoto 2017 - http://adam.openstreetmap.hu/mapproxy/tiles/1.0.0/Soskut-Tarnok-Pusztazamor-Diosd/mercator/{zoom}/{x}/{y}.png
   [groovy] +++ Obsolete skip entry: * Maxzoom differs (null != 20): [HU] Sóskút, Pusztazámor, Tárnok, Diósd orthophoto 2017 - http://adam.openstreetmap.hu/mapproxy/tiles/1.0.0/Soskut-Tarnok-Pusztazamor-Diosd/mercator/{zoom}/{x}/{y}.png

BUILD SUCCESSFUL

comment:46 Changed 6 months ago by stoecker

In 12061/josm:

see #14655 - UTF-8 fixes on Windows

comment:47 Changed 6 months ago by stoecker

Better? If not Vincent needs to have a look as I have no Windows systems.

comment:48 Changed 6 months ago by Klumbumbus

The console is clean now. However when I open the output files imagery_eliout.imagery.xml and imagery_josmout.imagery.xml in notepad++ there are still "wrong" characters <name>Kanton Z��h 2015 10cm</name>.

The coding is set to UTF-8 without BOM in notepad++. When I change it to ANSI the characters are displayed correctly: Zürich. I usually never change the coding in notepad++. It is not a big deal for me to change the coding in notepad++, but I don't know if this is actually still a bug of the groovy script.

comment:49 Changed 6 months ago by stoecker

In 12066/josm:

see #14655 - enforce UTF-8 for output in files

comment:50 Changed 6 months ago by stoecker

I wonder if Windows ever will follow all the others and use UTF-8 as default...

comment:51 Changed 6 months ago by stoecker

P.S. I have a special detection for "channels 123" texts. It can be replaced by proper English "123 channels" :-)

comment:52 in reply to:  51 Changed 6 months ago by Klumbumbus

Replying to stoecker:

P.S. I have a special detection for "channels 123" texts. It can be replaced by proper English "123 channels" :-)

If I understand correct, "channels 123" means as much as "the channels number 1, 2 and 3", see https://github.com/osmlab/editor-layer-index/pull/269#issuecomment-299567528, so it would be correct this way in my eyes.

comment:53 Changed 6 months ago by stoecker

ELI should be fixed then as well to either use "1+2+3" or "1,2,3". Writing this as one number is not understandable.

comment:54 Changed 6 months ago by stoecker

In 12068/josm:

see #14655 - remove channels workaround

comment:55 Changed 6 months ago by Klumbumbus

Imagico descriptions were changed in ELI, see https://github.com/osmlab/editor-layer-index/pull/322

comment:56 Changed 6 months ago by stoecker

Should I also drop the startsWith() workaround, so that only exact descriptions are taken as equal?

comment:57 Changed 6 months ago by Klumbumbus

Yes, makes sense I think.

comment:58 Changed 6 months ago by Klumbumbus

I already updated the imagico description, which were already in JOSM, so there should no additional warning pop up.

comment:59 Changed 6 months ago by stoecker

In 12086/josm:

see #14655 - drop another workaround

comment:60 Changed 5 months ago by bastiK

Cc: Zverikk added

Is it already syncing the id? That would useful for imagery_offset_db. The URL based imagery identifier algorithm is okay, but it is unstable when the URL changes. For this reason, a separate id database has been created. It would be ideal if ELI and JOSM ids were the same and we could use those in imagery_offset_db to identify an imagery layer.

comment:61 Changed 5 months ago by stoecker

No. id is not synced.

That makes only sense, when there would really be a two-way-syncing of the two datasets with active participation of ELI and iD.

What we do at the moment is a one-way syncing from ELI to JOSM and Klumbumbus fixes the most obvious issues in ELI as this is easier than to maintain an extremely large ignore list.

Syncing id now would mean we have to copy id's from ELI with resulting trouble for our users. As the result still would not mean we have stability I don't consider such a break worth the effort. Currently there are 30 differing id's and additionally 17 differences due to mirrors.

Klumbumbus: Correct me if I'm wrong.

In all the time I did not yet see any data flow from JOSM to ELI not initiated from JOSM contributors (except the very first dataset).

comment:62 Changed 5 months ago by stoecker

In 12126/josm:

see #14655 - add (commented) ID check

comment:63 Changed 5 months ago by bastiK

Output of the check:

 *** Same URL, but different Id: ***
 * Id differs ('Solothurn-sogis2014-wms' != 'Solothurn-RGB-SOGIS'): http://geoweb.so.ch/wms/sogis_orthofoto.wms?FORMAT=image/png&TRANSPARENT=TRUE&VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&LAYERS=Orthofoto_SO&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
 * Id differs ('Czech_CUZK-KM-tms' != 'Czech_CUZK-KM_mirror1'): http://osm-{switch:a,b,c}.zby.cz/tiles_cuzk.php/{zoom}/{x}/{y}.png
 * Id differs ('geodata.state.nj.us-Infrared2015' != 'geodata.state.nj.us-Infrared2015_mirror1'): http://whoots.mapwarper.net/tms/{zoom}/{x}/{y}/Infrared2015/http://geodata.state.nj.us/imagerywms/Infrared2015
 * Id differs ('Zuerich-zh_luftbild2011-tms' != 'Zuerich-aerial_2011_mirror1'): http://mapproxy.sosm.ch:8080/tiles/zh_luftbild2011/EPSG900913/{zoom}/{x}/{y}.png?origin=nw
 * Id differs ('MAPNIK' != 'standard'): https://{switch:a,b,c}.tile.openstreetmap.org/{zoom}/{x}/{y}.png
 * Id differs ('FR-BAN' != 'bano'): http://{switch:a,b,c}.layers.openstreetmap.fr/bano/{zoom}/{x}/{y}.png
 * Id differs ('South_Africa-CapeTown-Aerial' != 'South-Africa-City-of-Cape-Town-2015-Aerial'): http://{switch:a,b,c}.coct.aerial.openstreetmap.org.za/layer/za_coct_aerial_2015/{zoom}/{x}/{y}.jpg
 * Id differs ('tirol.gv.at-contourlines' != 'tirol.gv.at-tigris-contourlines'): https://gis.tirol.gv.at/arcgis/services/Service_Public/terrain/MapServer/WmsServer?FORMAT=image/png&TRANSPARENT=TRUE&VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&LAYERS=Hoehenschichtlinien 20m&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
 * Id differs ('KTZUERICH2015' != 'Kanton-Zürich-2015_mirror1'): http://mapproxy.osm.ch:8080/tiles/KTZUERICH2015/EPSG900913/{zoom}/{x}/{y}.png?origin=nw
 * Id differs ('tf-cycle' != 'opencylemap'): https://{switch:a,b,c}.tile.thunderforest.com/cycle/{zoom}/{x}/{y}.png
 * Id differs ('UrbisAdmFR' != 'URBISfr'): https://geoservices-urbis.irisnet.be/geoserver/ows/?SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&FORMAT=image/png&TRANSPARENT=true&LAYERS=UrbisAdm:UrbisFR&WIDTH={width}&HEIGHT={height}&CRS={proj}&STYLES=&BBOX={bbox}
 * Id differs ('Torokbalint-orthophoto-2015' != 'Törökbálint_2015'): http://terkep.torokbalint.hu/mapproxy/service?FORMAT=image/png&VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&LAYERS=TBORTO_2015_20160218&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}&TRANSPARENT=true
 * Id differs ('Torokbalint-orthophoto-2013' != 'Törökbálint_2013'): http://terkep.torokbalint.hu/mapproxy/service?FORMAT=image/png&VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&LAYERS=ORTO_2013_5CM_2013SZEPT_TAKARASSAL_512_512&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}&TRANSPARENT=true
 * Id differs ('Bern-bern2016-wms' != 'Bern-bern2016_mirror1'): http://map.bern.ch/wms/OpenData/proxy.php?FORMAT=image/jpeg&VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&LAYERS=0&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
 * Id differs ('geodata.md.gov-MD_ColorBasemap' != 'geodata.md.gov-MD_ColorBasemap_mirror1'): http://whoots.mapwarper.net/tms/{zoom}/{x}/{y}/MD_ColorBasemap/http://geodata.md.gov/imap/services/Transportation/MD_ColorBasemap/MapServer/WmsServer
 * Id differs ('AGIV' != 'AGIV_mirror1'): http://tile.informatievlaanderen.be/ws/raadpleegdiensten/wmts?SERVICE=WMTS&REQUEST=GetTile&VERSION=1.0.0&LAYER=omwrgbmrvl&STYLE=&FORMAT=image/png&tileMatrixSet=GoogleMapsVL&tileMatrix={zoom}&tileRow={y}&tileCol={x}
 * Id differs ('South-Tyrol-Topomap' != 'sdi.provinz.bz.it-WMTS_TOPOMAP_APB-PAB'): http://geoservices.buergernetz.bz.it/geoserver/gwc/service/wmts/?SERVICE=WMTS&REQUEST=GetTile&VERSION=1.0.0&LAYER=P_BZ_BASEMAP_TOPO&STYLE=default&TILEMATRIXSET=GoogleMapsCompatible&TILEMATRIX=GoogleMapsCompatible%3A{zoom}&TILEROW={y}&TILECOL={x}&FORMAT=image%2Fjpeg
 * Id differs ('maaamet.ee-pohi_vr2' != 'Maaamet-Estonia_Basemap'): http://kaart.maaamet.ee/wms/alus-geo?FORMAT=image/jpeg&VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&Layers=pohi_vr2&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
 * Id differs ('GRB' != 'AGIVFlandersGRB'): http://tile.informatievlaanderen.be/ws/raadpleegdiensten/wmts?SERVICE=WMTS&REQUEST=GetTile&VERSION=1.0.0&LAYER=grb_bsk&STYLE=&FORMAT=image/png&tileMatrixSet=GoogleMapsVL&tileMatrix={zoom}&tileRow={y}&tileCol={x}
 * Id differs ('FR-Cadastre' != 'Cadastre'): http://tms.cadastre.openstreetmap.fr/*/tout/{zoom}/{x}/{y}.png
 * Id differs ('GeolittoralV2-Orthophotos' != 'Geolittoral_V2-Orthophotos-2011_2014'): http://geolittoral.application.developpement-durable.gouv.fr/wms2/metropole?FORMAT=image/jpeg&VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&LAYERS=ortholittorale_v2_rvb&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
 * Id differs ('geodata.md.gov-MD_SixInchImagery' != 'geodata.md.gov-MD_SixInchImagery_mirror1'): http://whoots.mapwarper.net/tms/{zoom}/{x}/{y}/MD_SixInchImagery/http://geodata.md.gov/imap/services/Imagery/MD_SixInchImagery/MapServer/WmsServer
 * Id differs ('kelowna_roads' != 'Kelowna-RO'): http://{switch:a,b,c,d}.tile.paulnorman.ca/kelowna_overlay/{zoom}/{x}/{y}.png
 * Id differs ('NLSC-EMAP5' != 'TW_NLSC_WMS_EMAP5'): http://maps.nlsc.gov.tw/S_Maps/wmts?SERVICE=WMTS&REQUEST=GetTile&VERSION=1.0.0&LAYER=EMAP5_OPENDATA&STYLE=_null&TILEMATRIXSET=EPSG:3857&TILEMATRIX=EPSG:3857:{zoom}&TILEROW={y}&TILECOL={x}&FORMAT=image/png
 * Id differs ('Solothurn-sogis2014-tms' != 'Solothurn-RGB-SOGIS_mirror1'): http://mapproxy.osm.ch:8080/tiles/sogis2014/EPSG900913/{zoom}/{x}/{y}.png?origin=nw
 * Id differs ('South-Tyrol-Orthofoto2011' != 'sdi.provinz.bz.it-WMTS_OF2011_APB-PAB'): http://geoservices.buergernetz.bz.it/geoserver/gwc/service/wmts/?SERVICE=WMTS&REQUEST=GetTile&VERSION=1.0.0&LAYER=P_BZ_OF_2011_EPSG3857&STYLE=default&TILEMATRIXSET=GoogleMapsCompatible&TILEMATRIX=GoogleMapsCompatible%3A{zoom}&TILEROW={y}&TILECOL={x}&FORMAT=image%2Fjpeg
 * Id differs ('bavaria-DOP80' != 'bavaria-80cm_mirror1'): http://mapproxy.osm.ch:8080/tiles/BAYERNDOP80/EPSG900913/{zoom}/{x}/{y}.png?origin=nw
 * Id differs ('PNOA-Spain-TMS' != 'PNOA-Spain_mirror1'): http://www.ign.es/wmts/pnoa-ma?request=GetTile&service=WMTS&VERSION=1.0.0&Layer=OI.OrthoimageCoverage&Style=default&Format=image/png&TileMatrixSet=GoogleMapsCompatible&TileMatrix={zoom}&TileRow={y}&TileCol={x}
 * Id differs ('kelowna_2012' != 'Kelowna-OF-2012'): http://{switch:a,b,c,d}.tile.paulnorman.ca/kelowna2012/{zoom}/{x}/{y}.png
 * Id differs ('IPR-orotofoto-last-tms' != 'PrahaIPRlatestorthophoto_mirror1'): http://osm-{switch:a,b,c}.zby.cz/tiles_ipr_last.php/{zoom}/{x}/{y}.jpg
 * Id differs ('maaamet.ee-orto' != 'Maaamet-Estonia_Ortho_mirror2'): http://kaart.maakaart.ee/orto/{zoom}/{x}/{y}.jpeg
 * Id differs ('tirol.gv.at-DGM' != 'tirol.gv.at-tigris-DGM'): https://gis.tirol.gv.at/arcgis/services/Service_Public/terrain/MapServer/WmsServer?FORMAT=image/jpeg&VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&LAYERS=Image Schummerung_Gelaendemodell&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
 * Id differs ('UrbISOrtho2016' != 'URBIS2016'): https://geoservices-urbis.irisnet.be/geoserver/ows/?SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&FORMAT=image/png&TRANSPARENT=true&LAYERS=Urbis:Ortho2016&WIDTH={width}&HEIGHT={height}&CRS={proj}&STYLES=&BBOX={bbox}
 * Id differs ('Delaware2012Orthophotography' != 'Delaware2012Orthophotography_mirror1'): http://whoots.mapwarper.net/tms/{zoom}/{x}/{y}/0/https://firstmap.delaware.gov/arcgis/services/DE_Imagery/DE_Imagery_2012/ImageServer/WMSServer
 * Id differs ('UrbisAdmNL' != 'URBISnl'): https://geoservices-urbis.irisnet.be/geoserver/ows/?SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&FORMAT=image/png&TRANSPARENT=true&LAYERS=UrbisAdm:UrbisNL&WIDTH={width}&HEIGHT={height}&CRS={proj}&STYLES=&BBOX={bbox}
 * Id differs ('tirol.gv.at-DOM' != 'tirol.gv.at-tigris-DOM'): https://gis.tirol.gv.at/arcgis/services/Service_Public/terrain/MapServer/WmsServer?FORMAT=image/jpeg&VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&LAYERS=Image Schummerung_Oberflaechenmodell&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
 * Id differs ('geodata.state.nj.us-Natural2015' != 'geodata.state.nj.us-Natural2015_mirror1'): http://whoots.mapwarper.net/tms/{zoom}/{x}/{y}/Natural2015/http://geodata.state.nj.us/imagerywms/Natural2015
 * Id differs ('NSW_LPI_Imagery' != 'LPI_NSW_Imagery'): https://maps.six.nsw.gov.au/arcgis/rest/services/public/NSW_Imagery/MapServer/tile/{zoom}/{y}/{x}
 * Id differs ('NSW_LPI_BaseMap' != 'LPI_NSW_Base_Map'): https://maps.six.nsw.gov.au/arcgis/rest/services/public/NSW_Base_Map/MapServer/tile/{zoom}/{y}/{x}
 * Id differs ('Geolittoral-Orthophotos2000' != 'Geolittoral-Orthophotos-2000'): http://geolittoral.application.developpement-durable.gouv.fr/wms2/metropole?FORMAT=image/jpeg&VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&LAYERS=ortholittorale&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
 * Id differs ('NSW_LPI_TopographicMap' != 'LPI_NSW_Topographic_Map'): https://maps.six.nsw.gov.au/arcgis/rest/services/public/NSW_Topo_Map/MapServer/tile/{zoom}/{y}/{x}
 * Id differs ('maaamet.ee-reljeef' != 'Maaamet-Estonia_Hillshade'): http://kaart.maaamet.ee/wms/alus-geo?FORMAT=image/jpeg&VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&Layers=reljeef&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
 * Id differs ('IPR-orotofoto-vege-tms' != 'PrahaIPRlow-vegetationorthophoto_mirror1'): http://osm-{switch:a,b,c}.zby.cz/tiles_ipr_vege.php/{zoom}/{x}/{y}.jpg
 * Id differs ('maaamet.ee-of10000' != 'Maaamet-Estonia_Ortho'): http://kaart.maaamet.ee/wms/alus-geo?VERSION=1.1.1&REQUEST=GetMap&LAYERS=of10000&SRS={proj}&FORMAT=image/jpeg&WIDTH={width}&HEIGHT={height}&BBOX={bbox}
 * Id differs ('Zuerich-zh_uebersichtsplan-tms' != 'Zuerich-overview_mirror1'): http://mapproxy.sosm.ch:8080/tiles/zh_uebersichtsplan/EPSG900913/{zoom}/{x}/{y}.png?origin=nw
 * Id differs ('Bern-bern2016-tms' != 'Bern-bern2016'): http://mapproxy.osm.ch:8080/tiles/bern2016/EPSG900913/{zoom}/{x}/{y}.png?origin=nw
 * Id differs ('maaamet.ee-cir_ngr' != 'Maaamet-Estonia_Forestry'): http://kaart.maaamet.ee/wms/alus-geo?FORMAT=image/jpeg&VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&Layers=cir_ngr&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}

comment:64 in reply to:  61 ; Changed 5 months ago by bastiK

Replying to stoecker:

Syncing id now would mean we have to copy id's from ELI (...)

Correct. As ELI sets out to be a multi-editor project (currently iD, Potlatch 2 and Vespucci) and iD is widely exposing the imagery id with the imagery_used tag in changesets, ELI clearly has a higher legitimacy for stable id.

(...) with resulting trouble for our users.

On the other hand, we have a good handle on the damage it does and even the option to add migration code if needed.

I think it is worth the effort and there is no problem to keep it synced. If an inconsistency is detected early, it can be changed in ELI, otherwise we change it in the JOSM wiki.

comment:65 in reply to:  64 Changed 5 months ago by stoecker

Replying to bastiK:

Replying to stoecker:

Syncing id now would mean we have to copy id's from ELI (...)

Correct. As ELI sets out to be a multi-editor project (currently iD, Potlatch 2 and Vespucci) and iD is widely exposing the imagery id with the imagery_used tag in changesets, ELI clearly has a higher legitimacy for stable id.

Well, actually only iD counts. The others aren't important enough to be considered. http://wiki.openstreetmap.org/wiki/Editor_usage_stats

I think it is worth the effort

Could be.

We could add an <id old="...">...</id> now in JOSM and auto-update the ids and in some months start to replace the ids in wiki.

and there is no problem to keep it synced. If an inconsistency is detected early, it can be changed in ELI, otherwise we change it in the JOSM wiki.

As said, that would require an awareness for issues caused for JOSM on iD/ELI side. Currently this does not exist at all. The is no communication or feedback not initiated from us. Otherwise it will break again and again in the future as it does now (which we fix with the Ignores system, which is less than sub-optimal).

comment:66 Changed 5 months ago by stoecker

In 12143/josm:

see #14655 - drop some SKIP texts not needed anymore

comment:67 Changed 5 months ago by stoecker

In 12144/josm:

see #14655 - don't report missing projections for ELI TMS

comment:68 Changed 5 months ago by stoecker

In 12145/josm:

see #14655 - add missing apostrophe

comment:69 in reply to:  61 ; Changed 5 months ago by Klumbumbus

Replying to stoecker:

Klumbumbus: Correct me if I'm wrong.

Yeah, the situation is pretty much like you describe in this post from my point of view. (I think the main reason is the missing xml -> geojson converter).

Once we are done with the remaining "SKIP" entries at ImageryCompare, I'll announce again at https://github.com/osmlab/editor-layer-index/issues/88 and invite to update the ELI database.

comment:70 in reply to:  69 Changed 5 months ago by stoecker

Replying to Klumbumbus:

I think the main reason is the missing xml -> geojson converter

I can do this. I didn't do it till now based on an internal discussion at least a year ago to not heat the flames between our two projects, as an GeoJSON export of our database is the last step to say "ELI is unnecessary". Our maps page simple could get an parameter "?format=geojson".

comment:71 Changed 5 months ago by bastiK

A converter so we can contribute something back... Now that's a much more constructive motivation than was expressed at the time. ;)

comment:72 Changed 5 months ago by stoecker

Ok. When I find the time I'll implement GeoJSON output.

comment:73 Changed 5 months ago by stoecker

In 12179/josm:

see #14655 - drop two more skip types

comment:74 Changed 5 months ago by stoecker

Please test if GeoJSON works: Pretty: maps, Default: maps

comment:75 Changed 5 months ago by Klumbumbus

The bounds are missing.

comment:76 Changed 5 months ago by Klumbumbus

see r12226 and r12227

So basically now everything which can be compared by the script is compared now?

comment:77 in reply to:  75 Changed 5 months ago by stoecker

Replying to Klumbumbus:

The bounds are missing.

Ooops. Commented them, as they made the output hard to compare for other features. Forgot the enable it again. Fixed now.

So basically now everything which can be compared by the script is compared now?

Think so. As far as I see JOSM checks against any possible feature from ELI, but not the other way round.

comment:79 Changed 5 months ago by stoecker

In 12242/josm:

see #14655 - handle api keys in URL compare

comment:80 Changed 5 months ago by stoecker

+ SKIP * Attribution URL differs ('http://fototerkep.hu/' != 'http://www.torokbalint.hu/'): [HU] Törökbálint orthophoto 2015...
+ SKIP * Attribution text differs ('Fototerkep.hu' != 'Törökbálint'): [HU] Törökbálint orthophoto 2015 - ...
+ SKIP * Attribution URL differs ('http://terinfo.ujbuda.hu/' != 'https://www.ujbuda.hu/'): [HU] Budapest district XI orthophoto 2015 ...
+ SKIP * Attribution text differs ('SZTE TFGT - University of Szeged' != 'Szegedi Tudományegyetem Természeti Földrajzi és Geoinformatikai Tanszék'): [HU] Szeged orthophoto 2011 ...
+ SKIP - Missing JOSM license URL (http://wiki.openstreetmap.org/wiki/WikiProject_Hungary/Ortofot%C3%B3k): [HU] Sóskút, Pusztazámor, Tárnok, Diósd orthophoto 2017 ...

There are 5 SKIP entries left. Do you have any idea yet about them?

  • The first two look like an error in ELI to me.
  • For the third the ELI link seems mor helpful, but has no real relevance
  • Forth is probably a matter of taste and
  • Fifth seems wrong - at least I found nothing related to this on the page

comment:81 Changed 5 months ago by Klumbumbus

I didn't yet have a look at them. However it is on my todo list for this week ;)

Meanwhile something went wrong... 186 red entries in ImageryCompare

Last edited 5 months ago by Klumbumbus (previous) (diff)

comment:82 in reply to:  81 Changed 5 months ago by stoecker

Replying to Klumbumbus:

I didn't yet have a look at them. However it is on my todo list for this week ;)

Meanwhile something went wrong... 186 red entries in ImageryCompare

Jup, was already hard at work fixing it. Was a bug introduced with GeoJSON output and your last change triggered it.

comment:83 Changed 5 months ago by stoecker

In 12245/josm:

see #14655, new color for the unhandled ID entries

comment:84 Changed 5 months ago by Klumbumbus

Great. All SKIP entries done now.

comment:85 Changed 5 months ago by stoecker

In 12266/josm:

see #14655 - drop remaining SKIP types

comment:86 in reply to:  84 Changed 5 months ago by stoecker

Replying to Klumbumbus:

Great. All SKIP entries done now.

I'm not totally happy with the three Hungarian entries, see mail :-)

Changed 3 months ago by Don-vip

Attachment: josm_perfo.PNG added

comment:87 in reply to:  19 Changed 3 months ago by Don-vip

Replying to bastiK:

Thanks, this is a great improvement!

I've just extracted the performance tests out of the main job, on the same model than integration tests:

  • they're slow by definition
  • they're almost always ok

=> jenkins/job/JOSM-Performance

I have enabled JDK8 and JDK9. We can see JDK9 is faster, this is great :)

This speeds up the standard unit tests by 15 minutes, this is also a nice improvement:


Last edited 3 months ago by Don-vip (previous) (diff)

comment:88 Changed 3 months ago by bastiK

Okay.

Concerning [12537], I would only use uppercase variable name for constants (static final field, instance of unmodifiable class or primitive value), i.e. what would be considered a constant in C++.

comment:89 Changed 3 months ago by Don-vip

okay, done in r12542

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain Klumbumbus.
as The resolution will be set.
The resolution will be deleted.

Add Comment


E-mail address and name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.