Support Sentinel 2 data directly

[stoecker]

We already have some reused sentinel data in our Maps overview, but that is not really a good solution. Sentinel provides a large amount of imagery and a dedicated interface would be good.

Especially there needs to be a time span selection to easily choose proper data (i.e. no clouds, ...).

Conditions: ​https://sentinel.esa.int/documents/247904/690755/Sentinel_Data_Legal_Notice
Data directly: ​http://sentinel-pds.s3-website.eu-central-1.amazonaws.com/
SciHub: ​https://scihub.copernicus.eu/
A WMS/TMS/WMTS service: ​http://www.sentinel-hub.com/apps/wms

It looks like asking www.sentinel-hub.com for JOSM/OSM conditions would be the fastest solution, but probably direct access to Sentinel data is worth the effort?

[bastiK]

Sentinel-2 with 10m resolution is a better than Landsat (30m), but not that much better: You can barely see individual buildings. The value is really in the frequent updates (complete earth coverage about every 10 days).

One use case I can see already is to observe bodies of water over the seasons, so you can distinguish proper lakes from temporarily flooded areas ("Überflutungswiesen").

Replying to stoecker:

It looks like asking www.sentinel-hub.com for JOSM/OSM conditions would be the fastest solution,

Yes, it looks like they provide all that we could ask for.

but probably direct access to Sentinel data is worth the effort?

The data on the AWS will be hard to use directly: It is in proprietary jpeg 2000 format and the tiles are rather big (~300 MB each).

[wiktorn]

The other use case is to check how the road network looks after major (re-)construction.

I'm not sure about the conditions as from different parts I see different conditions, as for example ​here where I come from ​AWS S3 page

(note - only non-commercial use).

Sentinel Hub does cutting and reformatting to jpeg/png/tiff, so probably - we would need to implement some kind of their WMS-Configurator to allow our users to choose images easily.

Accessing S3 directly would require to setup something similar to Sentinel Hub on our infrastructure.

[stoecker]

Sentinel Hub does cutting and reformatting to jpeg/png/tiff, so probably - we would need to implement some kind of their WMS-Configurator to allow our users to choose images easily.

As far as my tests showed they provide normal WMS/TMS/WMTS service. With an additional TIME parameter for image selection.

[bastiK]

Replying to wiktorn:

I'm not sure about the conditions as from different parts I see different conditions, as for example ​here where I come from ​AWS S3 page

(note - only non-commercial use).

I don't see any ambiguity here, quote:

Any Sentinel data available through the Sentinel Data Hub will be governed by the ​Legal Notice on the use of Copernicus Sentinel Data and Service Information.

Any other contents of ESA's Sentinel Data Hub websites are intended for non-commercial use.

We don't want to use any contents other than the Sentinel data, so it is fine.

Sentinel Hub does cutting and reformatting to jpeg/png/tiff, so probably - we would need to implement some kind of their WMS-Configurator to allow our users to choose images easily.

Accessing S3 directly would require to setup something similar to Sentinel Hub on our infrastructure.

Maybe possible for a single time slice, but all the data? This would require a major investment in hardware and a person working full time on it ...

[Don-vip]

Another way to access data from Sentinel products, from the CNES (French Space Research Agency): ​https://peps.cnes.fr
There are web services / OpenSearch API described (in French) here: ​https://peps.cnes.fr/rocket/plus/plus.htm

[bastiK]

It doesn't look like they have options to download smaller tiles, just the areas of several square km like ESA. They have a "suggestions" box, maybe we can ask them to add WMS/TMS/WMTS service? :)

[Don-vip]

I can ask :)

[Don-vip]

Done. I note also in the project description the following evolution: "By 2018, a European platform involving the Big Data and Cloud industries will make Copernicus data available to all citizens, scientists and industry worldwide." Maybe this one will offer easier access if PEPS does not.

[Don-vip]

Good news! I have a positive answer from CNES:

the functionality to offer TMS/WMS/WMTS services is planned, under development and will be available soon.

\o/

[Don-vip]

Hello,
On Tuesday morning I will likely meet the IGN (French Geographic Agency) in the scope of this project:
​https://landsense.eu/Themes/Urban-Landscape-Dynamics
I don't know anything about it yet, but they're talking about "OpenStreetMap tools", and:

In the mid-Pyrenees, including the city of Toulouse in France, IGN France will engage citizens and public authorities in updating a number of their mapping products. Using the LandSense change detection service, citizens will become detectives and investigate urban change in real-time supported by the European Sentinel-2 satellite image data infrastructure.

I'll let you know of the implications, this is exciting :)

[bastiK]

Great, sounds very promising!

[Don-vip]

​The project is, among other things, dealing with improving landuse/landcover thanks to automatic change detection based on Sentinel-2 imagery. But the changes are provided as a service, not the imagery itself. I asked about our interest for a direct imagery access, they told me they will see with LandSense (it's not the IGN who run this service but ​GeoVille). Stay tuned!

@Simon: there's ​something fun in Austria. We will also use this application in Toulouse, but without the money :D Are you by chance in contact with GeoVille, IIASA, UBA, G2K? This project is definitively led by Austria!

[bastiK]

If you like to check out how the Sentinel 2 data looks in JOSM: The people from ​https://s2maps.eu/ have compiled a cloud-free snapshot and provide ​WMTS access. The attribution requirement is not compatible with OSM, though.

They may allow OSM use if we ask nicely, but I still see the value of Sentinel 2 in the frequent updates, not a single snapshot.

[Don-vip]

Replying to bastiK:

I still see the value of Sentinel 2 in the frequent updates, not a single snapshot.

I agree. I'd love to implement a GUI in JOSM allowing to update imagery background based on date & cloud settings.

[Don-vip]

See #15749 where support for jpeg 2000 is asked, but I closed it as wontfix as I don't see how we could support this format in JOSM.

[Don-vip]

To reschedule when we have concrete news

[gmilcinski]

Dear OSM community,
Bence has reached to us as we were not aware of this need/ticket.

We at Sentinel Hub are happy to support your cause and provide you free access to our services. As there are some costs attached to operation of the services we have to introduce some limits, e.g. 1 MIO tiles per month. Based on our discussion with Bence, this should be sufficient due to the nature (and limitations) of S-2 data.

I am not familiar with the OSM tooling (but most happy user of the data, thank you for this effort) but I believe it would be best if the service would be integrated in a way so that user can choose date and cloud parameters, to be able to get the best image possible. Similarly as it is done in Sentinel Playground (​http://apps.sentinel-hub.com/sentinel-playground/).

There is QGIS plugin in place for Sentinel Hub as well, if it makes it any easier:
​https://medium.com/sentinel-hub/control-sentinel-hub-from-within-qgis-2a83eb7f13db
​https://plugins.qgis.org/plugins/SentinelHub/

If there is someone on your side willing to put an effort into integration, he/she should reach to us at info@… and we will communicate details.

Best,
Grega

[Don-vip]

Hi Grega,
Thanks a lot! I think a JOSM Sentinel Hub plugin would make a great GSoC project for next summer, I would be happy to mentor it.
The QGIS plugin will definitively be helpful for this project.

[simon@…]

eox.at already has a global, cloudless, sentinel layer available. I'm just in the process of nailing down the few licence issues and then I suspect we will be able to use it directly (a plugin would have the advantage of being able to use current imagery, but ease of use would point rather to a layer).

[katpatuka]

Latest news (thanks simonpoole):

• ​https://github.com/osmlab/editor-layer-index/pull/644#event-2284284588
• ​https://wiki.openstreetmap.org/wiki/Eox.at

For use in josm add WMS address ​https://tiles.maps.eox.at/wms?service=wms&request=getcapabilities to get a list of layers from eox.at (select Store WMS endpoint only) - Note: selecting WMS layer entries including 4326 do not work but show a HTTP 400 error.

[Don-vip]

EOX added: https://josm.openstreetmap.de/wiki/Maps/Worldwide?action=diff&version=51&old_version=50

[OSM-user:smaprs]

Great achievement. Just for info, I tried it right now, and noticed that the resolution of [tiles.maps.eox.at...] is far below that of sentinel-hub; can't trace the axis of a road, or the borders of farmland/wood. I've been testing [services.sentinel-hub.com/ogc/wms/...] from personal account, its TCI/True Color Image is about 10m/px. Image comparing EOX vs Sentinel-Hub at ​https://i.imgur.com/k3l04vL.png. I guess that EOX image won't be much useful.

[simon04]

code-de.org provides a WMS service which allows specifying the creation interval using the WMS parameter time= (such as time=2019-08-31T00:00:00Z/2019-09-01T00:00:00Z): ​https://code-de.org/en/view-record/39b6073d-7ef4-4aa9-a3e4-07f079709b52

This could be integrated into JOSM in the following way: The user adds this WMS layer; JOSM asks the user to input the interval (maybe by showing simple text box); the selected interval is pre-filled for the next layer usage.

[imagico]

Example for a custom image layer entry in JOSM would be:
​https://geoservice.code-de.org/Sentinel2/wms?SERVICE=WMS&VERSION=1.1.0&REQUEST=GetMap&FORMAT=image%2Fpng&TRANSPARENT=true&LAYERS=S2_MSI_L1C&time=2019-09-09T00%3A00%3A00.000Z%2F2019-09-24T00%3A00%3A00.000Z&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}

At the low zoom levels you get image footprints, at the higher zoom levels the images themselves. Images are recorded in most land areas every five days and are available usually within a day or two after recording.

Slightly off-topic: The same could also be used for GIBS (​https://earthdata.nasa.gov/gibs) which offers low resolution layers with daily near real time updates - see ​https://wiki.earthdata.nasa.gov/display/GIBS/GIBS+Available+Imagery+Products

[simon04]

With attachment:14921.patch​​ the user is asked to input a time filter for the following WMS definition:

• URL: wms:https://geoservice.code-de.org/Sentinel2/ows?FORMAT=image/jpeg&VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&LAYERS=S2_MSI_L1C&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}&time={time}

The input string in inserted into the {time} template parameter.

[Don-vip]

It's fine for a first step. In the long term we should offer a proper UI. We should enhance the JavaFX plugin to propose ​DatePicker instead of a text field.

[simon04]

In 15739/josm:

see #14921 - WMS: support time={time} for Sentinel-2 imagery

[Don-vip]

In 15744/josm:

see #14921 - fix NPE

[Don-vip]

{time} documented in https://josm.openstreetmap.de/wiki/Maps?action=diff&version=569&old_version=568

[Don-vip]

Layers added to default list: https://josm.openstreetmap.de/wiki/Maps/Worldwide?action=diff&version=76

[Don-vip]

In 15810/josm:

see #14921 - hardcode valid time range for Sentinel-2 sources in imagery integration test

[pch14@…]

Der hostname geoservice.code-de.org kann nicht aufgelöst werden. Wurde das wms verlegt?

[simon04]

Maybe it's related to the CODE-DE Relaunch announced on ​https://code-de.org/news

[anonymous]

browser.code-de.org image of the Airport Innsbruck - ​https://services.sentinel-hub.com/ogc/wms/ed64bf38-72da-4723-9c06-568b76b8add0?service=WMS&request=GetMap&layers=1_TRUE_COLOR&styles=&format=image%2Fpng&transparent=true&version=1.1.1&showlogo=false&name=Sentinel-2%20L2A&height=512&width=512&pane=activeLayer&maxcc=100&time=2020-04-21%2F2020-04-21&srs=EPSG%3A3857&bbox=1262128.2110448303,5982879.077937317,1264574.1959499558,5985325.0628424445 - Nearly same location from apps.sentinel-hub.com - ​https://services.sentinel-hub.com/ogc/wms/26fc2e12-219a-42eb-9dde-f9f0287a7823?version=1.1.1&service=WMS&request=GetMap&format=image%2Fjpeg&crs=EPSG%3A3857&layers=1_TRUE_COLOR%2CDATE&bbox=1262128.2110448303%2C5982879.077937317%2C1264574.195949956%2C5985325.0628424445&time=2019-10-01T00%3A00%3A00.000Z%2F2020-04-22T23%3A59%3A59.999Z&width=512&height=512&showlogo=false&transparent=true&preview=2&bgcolor=000000 - The address looks suspiciously like like the geoservice.code-de.org WMS above; What is in the uuid? Swapping uuid does not change picture

[Hungerburg]

Sorry for the noise, but the feature is too intriguing; Below address pasted into the code-de setting gives a useable background in JOSM; max zoom capped at 14, no reason to burden the network with interpolating images; max cloudiness set to 50, just a guess; uuid is some kind of customer ID, it must not be left out nor replaced with a random one

wms[14]:​https://services.sentinel-hub.com/ogc/wms/ed64bf38-72da-4723-9c06-568b76b8add0?FORMAT=image/jpeg&VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&LAYERS=1_TRUE_COLOR%2CDATE&maxcc=50&name=Sentinel-2%20L2A&showlogo=false&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}

time parameter deliberately left out, so JOSM will not ask for a date. Then the latest image, that meets maxcc, will show instead - perhaps max cloudiness percentage might be an easier to answer query for the user, mightn't it? Yet, unfortunately, "{maxcc} is not a valid WMS argument"…

[simon04]

@Hungerburg, thank you for your research on imagery providers (and greetings from Tyrol (as well?))!

The current Sentinel imagery offered in JOSM are defined in Maps/Worldwide. Everyone is welcome to extend this list by useful additional Sentinel imagery providers.

Replying to Hungerburg:

time parameter deliberately left out, so JOSM will not ask for a date.

The interactive date selection was discussed/drafted in comment:33 and comment:34.

[imagico]

Some background info that might be helpful - Code-DE is the German platform for distributing Copernicus Sentinel data. Its operation has recently been contracted to a new operator and continuity for existing users is apparently not a main priority (and neither is service reliability - the whole site is down at the moment).

sentinel-hub.com is a commerical service that also - among other things - provides WMS services. They make money by selling paid services so what they offer for free is for attracting potential paying customers and can be changed or turned off at any time. That the new Code-DE operator makes use of sentinel-hub.com services is possible (or even likely since they cooperate in other projects). But that does not mean that Code-DE (and the German taxpayer) pay sentinel-hub.com for allowing external users to use sentinel-hub.com WMS services. I was going to look if there is any info on the new Code-DE regarding terms of use for this service but since it is down right now i can't do that. So far the TOS that apply would be ​https://sentinel-hub.com/tos. It also needs to be expected that the API keys will become invalid relatively frequently.

Regarding the maxcc parameter - that is probably a threshold to filter images based on automatic cloud cover assessment during the specified time frame. Automatic cloud cover assessment is not very reliable so this is to be used with care because it might lead to unexpected results for the user.

[Hungerburg]

Hello imagico, I made a an enhancement issue for the cloud coverage parameter - paired with date-stamping images I'd still consider it worthwhile.

Thank you for the background - I'd have thought sentinel-hub an ESA project. Reading sentinel-hub.com TOS I get the sense, that they try to adhere to the DSGVO (what is it in English) and are not any less user-privacy-friendly than other imagery sources; Still it looks evident, that I may use the WMS privately with the public-relations api-key nor the CODE-DE key, however often they may change, but must not put it into the JOSM Maps wiki?

[simon04]

Someone opened a separate ticket for the cloud coverage: #19141

[Don-vip]

Christoph, Simon, what should we do regarding our imagery entry referencing geoservice.code-de.org? The hostname is down, is there a new one, or should we delete the entry?

[imagico]

It seems rather unlikely this is going to come back so it probably should be removed.

There seems to be another WMS for Sentinel-2 images:

​https://view.onda-dias.eu/instance00/ows?language=eng&&SERVICE=WMS&VERSION=1.3.0&REQUEST=GetCapabilities
​https://view.onda-dias.eu/instance00/ows?language=eng&&SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&BBOX=1340370.084915278479,5691188.690827084705,1405450.65930740349,5737823.96873374097&CRS=EPSG:3857&WIDTH=1497&HEIGHT=1072&LAYERS=S2L1C_TRUE_COLOR&STYLES=&FORMAT=image/png&DPI=100&MAP_RESOLUTION=100&FORMAT_OPTIONS=dpi:100&TRANSPARENT=TRUE

Documentation is here:

​https://www.onda-dias.eu/cms/services/catalogues/view/

and indicates this is supposed to offer a three month time period (code-de was not limited in that regard). But according to

​https://www.gael-systems.com/dias-inspire-services/#S-VIEW

it is clearly indicated for public use without constraints and requires no API key. Unfortunately it seems to be really slow.

[pch14@…]

Indeed, it is slow, but under fees GetCapabilities states no conditions apply

wms[14]:https://view.onda-dias.eu/instance00/ows?service=WMS&version=1.1.1&request=GetMap&format=image%2Fjpeg&layers=S2L2A_TRUE_COLOR&srs={proj}&width={width}&height={height}&bbox={bbox}

JPEG chosen over PNG to minimize traffic, no use for DPI neither…

Resolution is less than sentinel hub, but also best at zoom level 14, for comparison:

• ​https://view.onda-dias.eu/instance00/ows?service=WMS&version=1.1.1&request=GetMap&format=image%2Fjpeg&layers=S2L2A_TRUE_COLOR&srs=EPSG:3857&width=512&height=512&bbox=1262127.2956108,5982879.6819892,1267019.2652108,5987771.6515892
• ​https://services.sentinel-hub.com/ogc/wms/ed64bf38-72da-4723-9c06-568b76b8add0?FORMAT=image/jpeg&VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&LAYERS=1_TRUE_COLOR%2CDATE&maxcc=50&name=Sentinel-2%20L2A&showlogo=false&STYLES=&SRS=EPSG:3857&WIDTH=512&HEIGHT=512&BBOX=1262127.2956108,5982879.6819892,1267019.2652108,5987771.6515892

[Don-vip]

Replying to imagico:

It seems rather unlikely this is going to come back so it probably should be removed.

Done in https://josm.openstreetmap.de/wiki/Maps/Worldwide?action=diff&version=100

[anonymous]

Hello Don, this might be a replacement section for Sentinel Data in the WIKI; regarding three months - imagico might have read that wrong, it is not a limit on access but a limit on data availability when using some undocumented date parameter, perhaps. As below, it always gets the latest image, however cloudy…

    <entry>
        <name>Sentinel-2 L2A (ONDE-DIAS)</name>
        <id>Sentinel2-L2A-CodeDE</id>
        <category>photo</category>
        <type>wms</type>
        <url><![CDATA[https://view.onda-dias.eu/instance00/ows?service=WMS&version=1.3.0&request=GetMap&format=image%2Fjpeg&layers=S2L2A_TRUE_COLOR&crs={proj}&width={width}&height={height}&bbox={bbox}]]></url>
        <min-zoom>1</min-zoom>
        <max-zoom>14</max-zoom>
        <projections>/
            <code>EPSG:3857</code>
            <code>EPSG:4326</code>
        </projections>
        <attribution-text mandatory="true">Sentinel-2 L2A - ONDE-DIAS (Copernicus Sentinel-2)</attribution-text>
        <attribution-url>https://www.onda-dias.eu/cms/services/catalogues/view/ - https://www.copernicus.eu/en/access-data/dias</attribution-url>
        <permission-ref><![CDATA[https://view.onda-dias.eu/instance00/ows?service=wms&request=getcapabilities]]></permission-ref>
        <icon>data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQBAMAAADt3eJSAAAABGdBTUEAALGPC/xhBQAAABtQTFRFMQAAbNHvbNHwbdHvbdHwbNLvbNLwbdLvbdLw51NFqwAAAAF0Uk5TAEDm2GYAAAABYktHRACIBR1IAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAB3RJTUUH5AUIEzIl9EHJtAAAAB1pVFh0Q29tbWVudAAAAAAAQ3JlYXRlZCB3aXRoIEdJTVBkLmUHAAAAXElEQVQI101OwQ2AMBCixhifklj77QpuYDdwhDqCGzi60D6UB8dBwh3wYeCWPUcKVeK2iEAg00o6iXYfOQswiQJPtK05XdQuZiaAPExFVereXShXGVuioxfz74cXR+oIUa54uGUAAAAASUVORK5CYII=</icon>
        <description lang="en">This collection contains Sentinel-2 Level-2A products which consist of bottom-of-atmosphere reflectances in cartographic geometry.</description>
    </entry>

[simon04]

Can someone check and confirm whether the following Sentinel-2 services may be included in JOSM (taking into account its license and access restrictions)? Thanks!

• onda-dias.eu
• sentinel-hub.com

[pch14@…]

Hello Simon,

onda-dias should be fine, the getcapabilities request literally states, closest to the subject as can be:

<Fees>no conditions apply</Fees>
<AccessConstraints>None</AccessConstraints>

I found nothing to the contrary on their site, maybe I haven't look thoroughly enough? Of course, their processing makes less brilliant pictures than sentinel-hub and the web service is awfully slow.

Sentinel hub though clearly does not want anybody to use their service for free. It would not be nice of JOSMs to work around their changing of api-key for their promotion product to work around their wish.

The personal opinion of mine,
Peter

[anonymous]

"Sentinel hub though clearly does not want anybody to use their service for free".
I can't see that clearly they don't want that.
They offer free use (through registering, but OSM editing requires this too).
They have some categories of users and services. I can use all features in EO-Browser for free.
If this changes in the future, that seems not the main issue.
Maxar and Bing do the same offering images to OSM. It's propaganda for their paid services.
Sérgio - OSMuser:smaprs

[ff5722]

Replying to anonymous:

"Sentinel hub though clearly does not want anybody to use their service for free".
I can't see that clearly they don't want that.
They offer free use (through registering, but OSM editing requires this too).
They have some categories of users and services. I can use all features in EO-Browser for free.
If this changes in the future, that seems not the main issue.
Maxar and Bing do the same offering images to OSM. It's propaganda for their paid services.
Sérgio - OSMuser:smaprs

Non-commercial use and media: The use of EO Browser application is free of charge for non-commercial use and any use in media.
CC BY 4.0 license: Data exported from EO Browser can be used under CC BY 4.0 license. Please mention Sentinel Hub EO Browser along with the relevant data set in the credits.
​https://apps.sentinel-hub.com/eo-browser/
​https://wiki.openstreetmap.org/wiki/Creative_Commons#Importing_creative_commons_licenced_data_into_OpenStreetMap

Sentinel Playground has a © Sentinel-hub in the bottom-right corner.

As far as I can tell, this makes their generated imagery incompatible with OSM until they give explicit permission.

[anonymous]

Hi, Copernicus have added a limited WMTS service of their own where I can find no mention of a CC BY license other than the one for which OSM has been granted access by the Copernicus programme. See ​https://documentation.dataspace.copernicus.eu/APIs/SentinelHub/OGC/WMTS.html for the service.
Yes, it obiously seems to be based on technology SentinelHub, but I haven't found any additional ToS from them limiting the use of the imagery.
It is unfortunately still a limited and registration-based service

[pch14@…]

The new location of the online sentinel data renderings browser is ​https://dataspace.copernicus.eu/browser/ - the firefox network console does not show, where the sat images come from, obviously there is some browser enforced standardized policy hiding them that I forgot the name of. As "the most recent image" will not always be wanted, there is much GUI E needed to make this useful. E.g. when I looked at the massive rock fall on Fluchthorn, it is always in clouded, except on Sep. 8, where the cloudiness suggests 30%, and the Fluchthorn is cloudless.

[anonymous]

"there is much GUI E needed to make this useful"
Regarding the problem of clouding, you can select a maximum threshold in EO Browser as well as the WMS/WMTS service