Modify

Opened 8 years ago

Closed 17 months ago

Last modified 16 months ago

#14361 closed defect (fixed)

Support for image/webp for WMTS satellite imagery?

Reported by: SanderH Owned by: team
Priority: normal Milestone: 17.02
Component: Core imagery Version:
Keywords: wmts webp Cc:

Description

I'm trying to get a satellite imagery working that's available in the netherlands since a few days but don't get the WMTS version working.
Configured the WMTS imagery for this url:
https://geodata.nationaalgeoregister.nl/luchtfoto/wmts?&request=GetCapabilities&service=WMS
The provider prefers consumers to use WMTS over WMS (which is working fine) to reduce load on their servers.

When selecting the WMTS from the Imagery menu, I get a nice box to choose from the 3 projections for the 2 layers.
Whatever you choose there it won't work. JOSM keeps complaining "Could not load image file from server".

The tile properties show for example this url with http status code 200:
http://geodata.nationaalgeoregister.nl/luchtfoto/wmts?SERVICE=WMTS&REQUEST=GetTile&VERSION=1.0.0&LAYER=Actueel_ortho25&STYLE=default&FORMAT=image/webp&tileMatrixSet=EPSG:28992&tileMatrix=8&tileRow=134&tileCol=131

I noticed the default format is image/webp, which I'm unable to change to one of the known working image formats (png, png8, jpg) which is also supported for this WMTS.

Since everything else seems working OK (url does work in the browser, status code 200) the only thing I can think of is that the image format itself could be the problem.
Do you know if JOSM supports image/webp?

Can you have a look and see what's going wrong and maybe fix it accordingly if it's something JOSM can do about it?

Thanks in advance.

Attachments (0)

Change History (28)

comment:1 by Don-vip, 8 years ago

Keywords: wmts webp added

comment:2 by Don-vip, 8 years ago

WebP is not supported by the JDK, see javabug:8116096. We must make sure to use only supported formats.

comment:3 by Don-vip, 8 years ago

Resolution: fixed
Status: newclosed

In 11559/josm:

fix #14361 - WMTS: download only supported image formats

comment:4 by Stereo, 4 years ago

Resolution: fixed
Status: closedreopened

comment:5 by Don-vip, 4 years ago

Milestone: 17.02
Resolution: fixed
Status: reopenedclosed

In source program, coders need to put native lib files like .so/.dll/.dylib into the folder of java.library.path.

No go, sorry.

comment:6 by anonymous, 2 years ago

comment:7 by Sreeram K, 2 years ago

Resolution: fixed
Status: closedreopened

Reopening to start a discussion..

comment:8 by taylor.smock, 2 years ago

I'll be honest, this kind of sounds like it could be done in a plugin (and probably should be done in a plugin). I don't know if JOSM really wants to bundle a bunch of jar files it doesn't directly depend upon. The few jars I checked from TwelveMonkeys seemed to be reasonably sized (<100kb).

comment:9 by Sreeram K <kandimalla.sreeram@…>, 2 years ago

I made an attempt at https://github.com/ramSeraph/josm-webp-plugin , did not work. I am not very familiar with java. it almost feels like the imageio available to the imagery code was already initialised by the time the plugins load.

in reply to:  9 comment:10 by taylor.smock, 2 years ago

Replying to Sreeram K <kandimalla.sreeram@…>:

I made an attempt at https://github.com/ramSeraph/josm-webp-plugin , did not work. I am not very familiar with java. it almost feels like the imageio available to the imagery code was already initialised by the time the plugins load.

Notes:

  • I probably would have called it josm-imageio-extensions or something like that, and then downloaded common extensions from the maven repository (example: avif/heif in the future).
    • Use the plugin manifest Class-Path instead of packing the imageio extensions into the jar file if you do this.
    • This is more complicated. You probably don't want to do that right now.
  • To get the current plugin to build, I would do this: git mv lib/build.gradle.kts .; git mv lib/src . in the root directory of the repository.
  • Apply the following patch to get the WebP ImageIO extension registered on plugin load
    • src/main/java/org/openstreetmap/josm/plugins/webp/WebpPlugin.java

      diff --git a/src/main/java/org/openstreetmap/josm/plugins/webp/WebpPlugin.java b/src/main/java/org/openstreetmap/josm/plugins/webp/WebpPlugin.java
      index 8cc90aa..b890418 100644
      a b  
      11package org.openstreetmap.josm.plugins.webp;
      22
       3import javax.imageio.ImageIO;
      34import org.openstreetmap.josm.plugins.Plugin;
      45import org.openstreetmap.josm.plugins.PluginInformation;
      56
      public class WebpPlugin extends Plugin {  
      1112   */
      1213  public WebpPlugin(PluginInformation info) {
      1314     super(info);
       15     ImageIO.scanForPlugins();
      1416  }
      1517}

comment:11 by Sreeram K <kandimalla.sreeram@…>, 2 years ago

The plugin in the current form builds.. though I would definitely want to get rid of that extra sub directory structure.

And apparently the problem was not with the plugin, but with the webp layer I was testing.. I checked with another webp layer I found on the internet.. it seems to be working fine. even without the changes you suggested. I would like to debug further as to what is going wrong with the layer I am trying.

About the suggestion regarding the Class-Path and the name, would this be a prerequisite for adding it into the main plugin repository?

in reply to:  11 comment:12 by taylor.smock, 2 years ago

Replying to Sreeram K <kandimalla.sreeram@…>:

The plugin in the current form builds.. though I would definitely want to get rid of that extra sub directory structure.

You probably have to run gradle inside the lib directory instead of ./gradlew for your current directory structure. I generally avoid gradle just due to versioning issues.

And apparently the problem was not with the plugin, but with the webp layer I was testing.. I checked with another webp layer I found on the internet.. it seems to be working fine. even without the changes you suggested. I would like to debug further as to what is going wrong with the layer I am trying.

I don't know then -- I wasn't seeing WebP when I was stepping through the code without the scanForPlugins bit.

About the suggestion regarding the Class-Path and the name, would this be a prerequisite for adding it into the main plugin repository?

No. Using ant, however, is (just due to how the JOSM CI works). If you want to do dual ant/gradle build scripts, see the mapillary or MapWithAI plugins for examples.

If you want a very basic release CI for GitHub actions, I created one for Lanes that you can use. I should look into making it a discrete repository so that changes can be more easily propagated. Right now, the only change you should have to make is to change the default branch name from master to main in the workflow (at the time I wrote the script, I was unable to use a variable for that).

Just keep in mind that someday you may want to add additional formats (specifically AVIF, and maybe JPEG XL), and then you would either have to refactor the plugin, add another plugin, or start fiddling with different build specifications.

comment:13 by Sreeram K <kandimalla.sreeram@…>, 2 years ago

I will look into the options you provided and get ping back here once I am done. For now I have another problem/query.. the imagery layer I had problems with, which also happens to be the reason i attempted this plugin is hosted on Google Cloud Storage by me and looks like JOSM blacklisted all "*.googleapis.*".. this seems a bit extreme. Do you know anyway for me to override this?

comment:14 by taylor.smock, 2 years ago

The *.googleapis.* is actually blacklisted by the OpenStreetMap API.

See https://api.openstreetmap.org/api/0.6/capabilities (<blacklist regex=".*\.google(apis)?\..*/.*"/>).

The only suggestion I have is to set up a domain that is not google related, and use that instead. We are not going to work around the OpenStreetMap blacklist.

With that being said, feel free to open a ticket on the openstreetmap-website repository, if you feel that the blacklist is overly broad. But I don't think it will go anywhere (just due to not wanting to chance copyright issues with Google Maps).

comment:15 by Sreeram K <kandimalla.sreeram@…>, 2 years ago

Thanks for the pointers.. I will consider my options.

comment:16 by Sreeram K <kandimalla.sreeram@…>, 2 years ago

Published the plugin to the plugin wiki page. Shows up as an external plugin in the plugin list. Has been tested on windows, linux and mac. Works nicely. In the end I had to put in the scanForPlugins line. Doesn't work consistently otherwise. Thanks for all the help @taylor.smock

comment:17 by taylor.smock, 2 years ago

Resolution: fixed
Status: reopenedclosed

I'm going to go ahead and close this again since Sreeram released a plugin for webp.

comment:18 by anonymous, 19 months ago

The Sreeram's webp plugin is does not work on Apple M1 or non-x86 architectures (the docs are wrong).

https://github.com/haraldk/TwelveMonkeys as suggested earlier is a brilliant drop-in replacement for ImageIO with better compatibility and WebP support. It is also faster.

comment:19 by kandimalla.sreeram@…, 19 months ago

When I last looked at it, TwelveMonkeys didn't have support full read support for WEBP, That's why I dropped the idea( I was the one who suggested it BTW :) ). Also, about the existing plugin not working on non-x86 Apple M1, I did a spot check on my laptop.. it worked. I am guessing my JVM might have been running in x86 compatibility mode. Looks like proper WEBP support is now available in TwelveMonkeys. Maybe you can submit a pull request to my repo or start a new plugin pulling in all of the image formats supported by TwelveMonkeys?

comment:20 by Stereo, 19 months ago

Yeah, JOSM runs in x86 compatibility mode on M1 Macs, because automatic builds using github actions are complicated (https://github.com/github/roadmap/issues/528) in the way we're currently building things.

On my homemade M1 build, the webp plugin installs, but adding https://stereo.dev.openstreetmap.org/testglobe.webp as a test tile layer produces an exception:

STACK TRACE

Thread: TMS-downloader-9 (116)
java.lang.NoClassDefFoundError: Could not initialize class com.luciad.imageio.webp.WebPDecoderOptions
	at com.luciad.imageio.webp.WebPReadParam.<init>(WebPReadParam.java:24)
	at com.luciad.imageio.webp.WebPReader.getDefaultReadParam(WebPReader.java:147)
	at java.desktop/javax.imageio.ImageIO.read(Unknown Source)
	at java.desktop/javax.imageio.ImageIO.read(Unknown Source)
	at org.openstreetmap.gui.jmapviewer.Tile.loadImage(Tile.java:247)
	at org.openstreetmap.josm.data.imagery.TMSCachedTileLoaderJob.tryLoadTileImage(TMSCachedTileLoaderJob.java:328)
	at org.openstreetmap.josm.data.imagery.TMSCachedTileLoaderJob.loadingFinished(TMSCachedTileLoaderJob.java:210)
	at org.openstreetmap.josm.data.cache.JCSCachedTileLoaderJob.finishLoading(JCSCachedTileLoaderJob.java:265)
	at org.openstreetmap.josm.data.cache.JCSCachedTileLoaderJob.run(JCSCachedTileLoaderJob.java:234)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.UnsatisfiedLinkError: Can't load library: /var/folders/1y/hwgwq23x3jbb7cyj0j998yrh0000gp/T/6605156055280924213libwebp-imageio.dylib [in thread "JCS Downloading: https://stereo.dev.openstreetmap.org/testglobe.webp"]
	at java.base/java.lang.ClassLoader.loadLibrary(Unknown Source)
	at java.base/java.lang.Runtime.load0(Unknown Source)
	at java.base/java.lang.System.load(Unknown Source)
	at com.luciad.imageio.webp.NativeLibraryUtils.loadFromJar(WebP.java:131)
	at com.luciad.imageio.webp.WebP.loadNativeLibrary(WebP.java:31)
	at com.luciad.imageio.webp.WebP.<clinit>(WebP.java:38)
	at com.luciad.imageio.webp.WebPDecoderOptions.<clinit>(WebPDecoderOptions.java:20)
	... 12 more

comment:21 by anonymous, 17 months ago

Resolution: fixed
Status: closedreopened

comment:22 by Stereo, 17 months ago

Reopening since there's only a partial solution.

comment:23 by stoecker, 17 months ago

Resolution: fixed
Status: reopenedclosed

Instead of opening an old ticket here, open a ticket for the plugin at github where it can be fixed.

comment:24 by taylor.smock, 16 months ago

In 36093/osm:

See #14361: Support for image/webp

comment:25 by taylor.smock, 16 months ago

@stoecker: I'm not going to make a binary for the plugin I just pushed to svn yet -- I want to give you a chance to make certain that the download code isn't going to kill our servers (it downloads the ImageIO dependencies off of our nexus).

I don't think it will, but I'd rather not kill our servers accidentally.

comment:26 by stoecker, 16 months ago

Static downloads aren't an issue for the server. ;-)

in reply to:  26 comment:27 by taylor.smock, 16 months ago

Replying to stoecker:

Static downloads aren't an issue for the server. ;-)

Good to know. I didn't know if the downloads were static or generated on demand (I've seen some interesting implementations for stuff before).

comment:28 by taylor.smock, 16 months ago

In 36094/osm:

See #14361 (dist): Support for image/webp

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain team.
as The resolution will be set.
The resolution will be deleted. Next status will be 'reopened'.

Add Comment


E-mail address and name can be saved in the Preferences .
 
Note: See TracTickets for help on using tickets.