Modify

Opened 3 years ago

Last modified 5 months ago

#14921 reopened enhancement

Support Sentinel 2 data directly

Reported by: stoecker Owned by: team
Priority: major Milestone:
Component: Core imagery Version:
Keywords: sentinel gsoc-candidate sotm19 Cc: wiktorn, Klumbumbus, simon04, naoliv, michael2402, lists@…, imagico, SimonPoole

Description (last modified by stoecker)

We already have some reused sentinel data in our Maps overview, but that is not really a good solution. Sentinel provides a large amount of imagery and a dedicated interface would be good.

Especially there needs to be a time span selection to easily choose proper data (i.e. no clouds, ...).

Conditions: https://sentinel.esa.int/documents/247904/690755/Sentinel_Data_Legal_Notice
Data directly: http://sentinel-pds.s3-website.eu-central-1.amazonaws.com/
SciHub: https://scihub.copernicus.eu/
A WMS/TMS/WMTS service: http://www.sentinel-hub.com/apps/wms

It looks like asking www.sentinel-hub.com for JOSM/OSM conditions would be the fastest solution, but probably direct access to Sentinel data is worth the effort?

Attachments (2)

14921.patch (5.5 KB) - added by simon04 9 months ago.
2020-01-19-225834_607x106_scrot.png (4.7 KB) - added by simon04 9 months ago.

Download all attachments as: .zip

Change History (62)

comment:1 Changed 3 years ago by stoecker

Description: modified (diff)

comment:2 Changed 3 years ago by stoecker

Cc: Klumbumbus added

comment:3 in reply to:  description Changed 3 years ago by bastiK

Sentinel-2 with 10m resolution is a better than Landsat (30m), but not that much better: You can barely see individual buildings. The value is really in the frequent updates (complete earth coverage about every 10 days).

One use case I can see already is to observe bodies of water over the seasons, so you can distinguish proper lakes from temporarily flooded areas ("Überflutungswiesen").

Replying to stoecker:

It looks like asking www.sentinel-hub.com for JOSM/OSM conditions would be the fastest solution,

Yes, it looks like they provide all that we could ask for.

but probably direct access to Sentinel data is worth the effort?

The data on the AWS will be hard to use directly: It is in proprietary jpeg 2000 format and the tiles are rather big (~300 MB each).

comment:4 Changed 3 years ago by wiktorn

The other use case is to check how the road network looks after major (re-)construction.

I'm not sure about the conditions as from different parts I see different conditions, as for example here where I come from AWS S3 page

(note - only non-commercial use).

Sentinel Hub does cutting and reformatting to jpeg/png/tiff, so probably - we would need to implement some kind of their WMS-Configurator to allow our users to choose images easily.

Accessing S3 directly would require to setup something similar to Sentinel Hub on our infrastructure.

comment:5 in reply to:  4 Changed 3 years ago by stoecker

Sentinel Hub does cutting and reformatting to jpeg/png/tiff, so probably - we would need to implement some kind of their WMS-Configurator to allow our users to choose images easily.

As far as my tests showed they provide normal WMS/TMS/WMTS service. With an additional TIME parameter for image selection.

comment:6 in reply to:  4 Changed 3 years ago by bastiK

Replying to wiktorn:

I'm not sure about the conditions as from different parts I see different conditions, as for example here where I come from AWS S3 page

(note - only non-commercial use).

I don't see any ambiguity here, quote:

Any Sentinel data available through the Sentinel Data Hub will be governed by the Legal Notice on the use of Copernicus Sentinel Data and Service Information.

Any other contents of ESA's Sentinel Data Hub websites are intended for non-commercial use.

We don't want to use any contents other than the Sentinel data, so it is fine.

Sentinel Hub does cutting and reformatting to jpeg/png/tiff, so probably - we would need to implement some kind of their WMS-Configurator to allow our users to choose images easily.

Accessing S3 directly would require to setup something similar to Sentinel Hub on our infrastructure.

Maybe possible for a single time slice, but all the data? This would require a major investment in hardware and a person working full time on it ...

comment:7 Changed 3 years ago by Don-vip

Milestone: 17.0717.08

comment:8 Changed 3 years ago by Don-vip

Milestone: 17.0817.09

comment:9 Changed 3 years ago by Don-vip

Another way to access data from Sentinel products, from the CNES (French Space Agency): https://peps.cnes.fr
There are web services / OpenSearch API described (in French) here: https://peps.cnes.fr/rocket/plus/plus.htm

Last edited 3 years ago by Don-vip (previous) (diff)

comment:10 Changed 3 years ago by bastiK

It doesn't look like they have options to download smaller tiles, just the areas of several square km like ESA. They have a "suggestions" box, maybe we can ask them to add WMS/TMS/WMTS service? :)

comment:11 Changed 3 years ago by Don-vip

I can ask :)

comment:12 Changed 3 years ago by Don-vip

Done. I note also in the project description the following evolution: "By 2018, a European platform involving the Big Data and Cloud industries will make Copernicus data available to all citizens, scientists and industry worldwide." Maybe this one will offer easier access if PEPS does not.

comment:13 Changed 3 years ago by Don-vip

Good news! I have a positive answer from CNES:

the functionality to offer TMS/WMS/WMTS services is planned, under development and will be available soon.

\o/

Last edited 3 years ago by Don-vip (previous) (diff)

comment:14 Changed 3 years ago by Don-vip

Hello,
On Tuesday morning I will likely meet the IGN (French Geographic Agency) in the scope of this project:
https://landsense.eu/Themes/Urban-Landscape-Dynamics
I don't know anything about it yet, but they're talking about "OpenStreetMap tools", and:

In the mid-Pyrenees, including the city of Toulouse in France, IGN France will engage citizens and public authorities in updating a number of their mapping products. Using the LandSense change detection service, citizens will become detectives and investigate urban change in real-time supported by the European Sentinel-2 satellite image data infrastructure.

I'll let you know of the implications, this is exciting :)

comment:15 Changed 3 years ago by bastiK

Great, sounds very promising!

comment:16 Changed 3 years ago by Don-vip

Cc: simon04 added

The project is, among other things, dealing with improving landuse/landcover thanks to automatic change detection based on Sentinel-2 imagery. But the changes are provided as a service, not the imagery itself. I asked about our interest for a direct imagery access, they told me they will see with LandSense (it's not the IGN who run this service but GeoVille). Stay tuned!

@Simon: there's something fun in Austria. We will also use this application in Toulouse, but without the money :D Are you by chance in contact with GeoVille, IIASA, UBA, G2K? This project is definitively led by Austria!

comment:17 Changed 3 years ago by Don-vip

Milestone: 17.0917.10

comment:18 Changed 3 years ago by bastiK

If you like to check out how the Sentinel 2 data looks in JOSM: The people from https://s2maps.eu/ have compiled a cloud-free snapshot and provide WMTS access. The attribution requirement is not compatible with OSM, though.

They may allow OSM use if we ask nicely, but I still see the value of Sentinel 2 in the frequent updates, not a single snapshot.

comment:19 in reply to:  18 Changed 3 years ago by Don-vip

Replying to bastiK:

I still see the value of Sentinel 2 in the frequent updates, not a single snapshot.

I agree. I'd love to implement a GUI in JOSM allowing to update imagery background based on date & cloud settings.

comment:20 Changed 3 years ago by Don-vip

Milestone: 17.1017.11

comment:21 Changed 3 years ago by Don-vip

Milestone: 17.1117.12

comment:22 Changed 3 years ago by Don-vip

Keywords: sentinel added
Milestone: 17.1218.01

comment:23 Changed 3 years ago by Don-vip

See #15749 where support for jpeg 2000 is asked, but I closed it as wontfix as I don't see how we could support this format in JOSM.

comment:24 Changed 3 years ago by Don-vip

Cc: naoliv added

comment:25 Changed 3 years ago by Don-vip

Milestone: 18.01

To reschedule when we have concrete news

comment:26 Changed 23 months ago by gmilcinski

Dear OSM community,
Bence has reached to us as we were not aware of this need/ticket.

We at Sentinel Hub are happy to support your cause and provide you free access to our services. As there are some costs attached to operation of the services we have to introduce some limits, e.g. 1 MIO tiles per month. Based on our discussion with Bence, this should be sufficient due to the nature (and limitations) of S-2 data.

I am not familiar with the OSM tooling (but most happy user of the data, thank you for this effort) but I believe it would be best if the service would be integrated in a way so that user can choose date and cloud parameters, to be able to get the best image possible. Similarly as it is done in Sentinel Playground (http://apps.sentinel-hub.com/sentinel-playground/).

There is QGIS plugin in place for Sentinel Hub as well, if it makes it any easier:
https://medium.com/sentinel-hub/control-sentinel-hub-from-within-qgis-2a83eb7f13db
https://plugins.qgis.org/plugins/SentinelHub/

If there is someone on your side willing to put an effort into integration, he/she should reach to us at info@… and we will communicate details.

Best,
Grega

comment:27 Changed 23 months ago by Don-vip

Cc: michael2402 added
Keywords: gsoc-candidate added

Hi Grega,
Thanks a lot! I think a JOSM Sentinel Hub plugin would make a great GSoC project for next summer, I would be happy to mentor it.
The QGIS plugin will definitively be helpful for this project.

comment:28 Changed 22 months ago by Aun Johnsen <lists@…>

Cc: lists@… added

comment:29 Changed 21 months ago by simon@…

eox.at already has a global, cloudless, sentinel layer available. I'm just in the process of nailing down the few licence issues and then I suspect we will be able to use it directly (a plugin would have the advantage of being able to use current imagery, but ease of use would point rather to a layer).

comment:30 Changed 19 months ago by katpatuka

Latest news (thanks simonpoole):

For use in josm add WMS address https://tiles.maps.eox.at/wms?service=wms&request=getcapabilities to get a list of layers from eox.at (select Store WMS endpoint only) - Note: selecting WMS layer entries including 4326 do not work but show a HTTP 400 error.

Last edited 19 months ago by katpatuka (previous) (diff)

comment:32 Changed 18 months ago by OSM-user:smaprs

Great achievement. Just for info, I tried it right now, and noticed that the resolution of [tiles.maps.eox.at...] is far below that of sentinel-hub; can't trace the axis of a road, or the borders of farmland/wood. I've been testing [services.sentinel-hub.com/ogc/wms/...] from personal account, its TCI/True Color Image is about 10m/px. Image comparing EOX vs Sentinel-Hub at https://i.imgur.com/k3l04vL.png. I guess that EOX image won't be much useful.

comment:33 Changed 13 months ago by simon04

Cc: imagico added
Keywords: sotm19 added

code-de.org provides a WMS service which allows specifying the creation interval using the WMS parameter time= (such as time=2019-08-31T00:00:00Z/2019-09-01T00:00:00Z): https://code-de.org/en/view-record/39b6073d-7ef4-4aa9-a3e4-07f079709b52

This could be integrated into JOSM in the following way: The user adds this WMS layer; JOSM asks the user to input the interval (maybe by showing simple text box); the selected interval is pre-filled for the next layer usage.

comment:34 Changed 13 months ago by imagico

Example for a custom image layer entry in JOSM would be:
https://geoservice.code-de.org/Sentinel2/wms?SERVICE=WMS&VERSION=1.1.0&REQUEST=GetMap&FORMAT=image%2Fpng&TRANSPARENT=true&LAYERS=S2_MSI_L1C&time=2019-09-09T00%3A00%3A00.000Z%2F2019-09-24T00%3A00%3A00.000Z&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}

At the low zoom levels you get image footprints, at the higher zoom levels the images themselves. Images are recorded in most land areas every five days and are available usually within a day or two after recording.

Slightly off-topic: The same could also be used for GIBS (https://earthdata.nasa.gov/gibs) which offers low resolution layers with daily near real time updates - see https://wiki.earthdata.nasa.gov/display/GIBS/GIBS+Available+Imagery+Products

Changed 9 months ago by simon04

Attachment: 14921.patch added

Changed 9 months ago by simon04

comment:35 Changed 9 months ago by simon04

Milestone: 20.01
Owner: changed from team to simon04
Status: newassigned
Summary: Support Sentinel 2 data directly[Patch] Support Sentinel 2 data directly

With attachment:14921.patch​ the user is asked to input a time filter for the following WMS definition:

  • URL: wms:https://geoservice.code-de.org/Sentinel2/ows?FORMAT=image/jpeg&VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&LAYERS=S2_MSI_L1C&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}&time={time}


The input string in inserted into the {time} template parameter.

comment:36 Changed 9 months ago by Don-vip

It's fine for a first step. In the long term we should offer a proper UI. We should enhance the JavaFX plugin to propose DatePicker instead of a text field.

comment:37 Changed 9 months ago by simon04

In 15739/josm:

see #14921 - WMS: support time={time} for Sentinel-2 imagery

comment:38 Changed 9 months ago by simon04

Summary: [Patch] Support Sentinel 2 data directlySupport Sentinel 2 data directly

comment:39 Changed 9 months ago by Don-vip

In 15744/josm:

see #14921 - fix NPE

comment:40 Changed 9 months ago by simon04

Owner: changed from simon04 to team
Status: assignednew

comment:42 Changed 9 months ago by Don-vip

Resolution: fixed
Status: newclosed

comment:43 Changed 9 months ago by Don-vip

In 15810/josm:

see #14921 - hardcode valid time range for Sentinel-2 sources in imagery integration test

comment:44 Changed 6 months ago by pch14@…

Der hostname geoservice.code-de.org kann nicht aufgelöst werden. Wurde das wms verlegt?

comment:45 Changed 6 months ago by simon04

Maybe it's related to the CODE-DE Relaunch announced on https://code-de.org/news

comment:47 Changed 6 months ago by Hungerburg

Sorry for the noise, but the feature is too intriguing; Below address pasted into the code-de setting gives a useable background in JOSM; max zoom capped at 14, no reason to burden the network with interpolating images; max cloudiness set to 50, just a guess; uuid is some kind of customer ID, it must not be left out nor replaced with a random one

wms[14]:https://services.sentinel-hub.com/ogc/wms/ed64bf38-72da-4723-9c06-568b76b8add0?FORMAT=image/jpeg&VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&LAYERS=1_TRUE_COLOR%2CDATE&maxcc=50&name=Sentinel-2%20L2A&showlogo=false&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}

time parameter deliberately left out, so JOSM will not ask for a date. Then the latest image, that meets maxcc, will show instead - perhaps max cloudiness percentage might be an easier to answer query for the user, mightn't it? Yet, unfortunately, "{maxcc} is not a valid WMS argument"…

comment:48 in reply to:  47 Changed 6 months ago by simon04

@Hungerburg, thank you for your research on imagery providers (and greetings from Tyrol (as well?))!

The current Sentinel imagery offered in JOSM are defined in Maps/Worldwide. Everyone is welcome to extend this list by useful additional Sentinel imagery providers.

Replying to Hungerburg:

time parameter deliberately left out, so JOSM will not ask for a date.

The interactive date selection was discussed/drafted in comment:33 and comment:34.

comment:49 Changed 6 months ago by imagico

Some background info that might be helpful - Code-DE is the German platform for distributing Copernicus Sentinel data. Its operation has recently been contracted to a new operator and continuity for existing users is apparently not a main priority (and neither is service reliability - the whole site is down at the moment).

sentinel-hub.com is a commerical service that also - among other things - provides WMS services. They make money by selling paid services so what they offer for free is for attracting potential paying customers and can be changed or turned off at any time. That the new Code-DE operator makes use of sentinel-hub.com services is possible (or even likely since they cooperate in other projects). But that does not mean that Code-DE (and the German taxpayer) pay sentinel-hub.com for allowing external users to use sentinel-hub.com WMS services. I was going to look if there is any info on the new Code-DE regarding terms of use for this service but since it is down right now i can't do that. So far the TOS that apply would be https://sentinel-hub.com/tos. It also needs to be expected that the API keys will become invalid relatively frequently.

Regarding the maxcc parameter - that is probably a threshold to filter images based on automatic cloud cover assessment during the specified time frame. Automatic cloud cover assessment is not very reliable so this is to be used with care because it might lead to unexpected results for the user.

comment:50 in reply to:  49 Changed 6 months ago by Hungerburg

Hello imagico, I made a an enhancement issue for the cloud coverage parameter - paired with date-stamping images I'd still consider it worthwhile.

Thank you for the background - I'd have thought sentinel-hub an ESA project. Reading sentinel-hub.com TOS I get the sense, that they try to adhere to the DSGVO (what is it in English) and are not any less user-privacy-friendly than other imagery sources; Still it looks evident, that I may use the WMS privately with the public-relations api-key nor the CODE-DE key, however often they may change, but must not put it into the JOSM Maps wiki?

comment:51 Changed 6 months ago by simon04

Someone opened a separate ticket for the cloud coverage: #19141

comment:52 Changed 6 months ago by Don-vip

Christoph, Simon, what should we do regarding our imagery entry referencing geoservice.code-de.org? The hostname is down, is there a new one, or should we delete the entry?

comment:53 Changed 6 months ago by imagico

It seems rather unlikely this is going to come back so it probably should be removed.

There seems to be another WMS for Sentinel-2 images:

https://view.onda-dias.eu/instance00/ows?language=eng&&SERVICE=WMS&VERSION=1.3.0&REQUEST=GetCapabilities
https://view.onda-dias.eu/instance00/ows?language=eng&&SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&BBOX=1340370.084915278479,5691188.690827084705,1405450.65930740349,5737823.96873374097&CRS=EPSG:3857&WIDTH=1497&HEIGHT=1072&LAYERS=S2L1C_TRUE_COLOR&STYLES=&FORMAT=image/png&DPI=100&MAP_RESOLUTION=100&FORMAT_OPTIONS=dpi:100&TRANSPARENT=TRUE

Documentation is here:

https://www.onda-dias.eu/cms/services/catalogues/view/

and indicates this is supposed to offer a three month time period (code-de was not limited in that regard). But according to

https://www.gael-systems.com/dias-inspire-services/#S-VIEW

it is clearly indicated for public use without constraints and requires no API key. Unfortunately it seems to be really slow.

comment:54 Changed 6 months ago by pch14@…

Indeed, it is slow, but under fees GetCapabilities states no conditions apply

wms[14]:https://view.onda-dias.eu/instance00/ows?service=WMS&version=1.1.1&request=GetMap&format=image%2Fjpeg&layers=S2L2A_TRUE_COLOR&srs={proj}&width={width}&height={height}&bbox={bbox}

JPEG chosen over PNG to minimize traffic, no use for DPI neither…

Resolution is less than sentinel hub, but also best at zoom level 14, for comparison:

comment:55 in reply to:  53 Changed 6 months ago by Don-vip

Replying to imagico:

It seems rather unlikely this is going to come back so it probably should be removed.

Done in https://josm.openstreetmap.de/wiki/Maps/Worldwide?action=diff&version=100

comment:56 Changed 6 months ago by Don-vip

Milestone: 20.01
Resolution: fixed
Status: closedreopened

comment:57 Changed 6 months ago by anonymous

Hello Don, this might be a replacement section for Sentinel Data in the WIKI; regarding three months - imagico might have read that wrong, it is not a limit on access but a limit on data availability when using some undocumented date parameter, perhaps. As below, it always gets the latest image, however cloudy…

    <entry>
        <name>Sentinel-2 L2A (ONDE-DIAS)</name>
        <id>Sentinel2-L2A-CodeDE</id>
        <category>photo</category>
        <type>wms</type>
        <url><![CDATA[https://view.onda-dias.eu/instance00/ows?service=WMS&version=1.3.0&request=GetMap&format=image%2Fjpeg&layers=S2L2A_TRUE_COLOR&crs={proj}&width={width}&height={height}&bbox={bbox}]]></url>
        <min-zoom>1</min-zoom>
        <max-zoom>14</max-zoom>
        <projections>/
            <code>EPSG:3857</code>
            <code>EPSG:4326</code>
        </projections>
        <attribution-text mandatory="true">Sentinel-2 L2A - ONDE-DIAS (Copernicus Sentinel-2)</attribution-text>
        <attribution-url>https://www.onda-dias.eu/cms/services/catalogues/view/ - https://www.copernicus.eu/en/access-data/dias</attribution-url>
        <permission-ref><![CDATA[https://view.onda-dias.eu/instance00/ows?service=wms&request=getcapabilities]]></permission-ref>
        <icon>data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQBAMAAADt3eJSAAAABGdBTUEAALGPC/xhBQAAABtQTFRFMQAAbNHvbNHwbdHvbdHwbNLvbNLwbdLvbdLw51NFqwAAAAF0Uk5TAEDm2GYAAAABYktHRACIBR1IAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAB3RJTUUH5AUIEzIl9EHJtAAAAB1pVFh0Q29tbWVudAAAAAAAQ3JlYXRlZCB3aXRoIEdJTVBkLmUHAAAAXElEQVQI101OwQ2AMBCixhifklj77QpuYDdwhDqCGzi60D6UB8dBwh3wYeCWPUcKVeK2iEAg00o6iXYfOQswiQJPtK05XdQuZiaAPExFVereXShXGVuioxfz74cXR+oIUa54uGUAAAAASUVORK5CYII=</icon>
        <description lang="en">This collection contains Sentinel-2 Level-2A products which consist of bottom-of-atmosphere reflectances in cartographic geometry.</description>
    </entry>
Last edited 5 months ago by simon04 (previous) (diff)

comment:58 Changed 5 months ago by simon04

Cc: SimonPoole added

Can someone check and confirm whether the following Sentinel-2 services may be included in JOSM (taking into account its license and access restrictions)? Thanks!

  • onda-dias.eu
  • sentinel-hub.com

comment:59 Changed 5 months ago by pch14@…

Hello Simon,

onda-dias should be fine, the getcapabilities request literally states, closest to the subject as can be:

<Fees>no conditions apply</Fees>
<AccessConstraints>None</AccessConstraints>

I found nothing to the contrary on their site, maybe I haven't look thoroughly enough? Of course, their processing makes less brilliant pictures than sentinel-hub and the web service is awfully slow.

Sentinel hub though clearly does not want anybody to use their service for free. It would not be nice of JOSMs to work around their changing of api-key for their promotion product to work around their wish.

The personal opinion of mine,
Peter

comment:60 Changed 5 months ago by anonymous

"Sentinel hub though clearly does not want anybody to use their service for free".
I can't see that clearly they don't want that.
They offer free use (through registering, but OSM editing requires this too).
They have some categories of users and services. I can use all features in EO-Browser for free.
If this changes in the future, that seems not the main issue.
Maxar and Bing do the same offering images to OSM. It's propaganda for their paid services.
Sérgio - OSMuser:smaprs

Modify Ticket

Change Properties
Set your email in Preferences
Action
as reopened The owner will remain team.
as The resolution will be set.
to The owner will be changed from team to the specified user.
The owner will change to stoecker
as duplicate The resolution will be set to duplicate.The specified ticket will be cross-referenced with this ticket

Add Comment


E-mail address and name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.