Modify

Opened 3 years ago

Closed 2 months ago

Last modified 2 months ago

#14921 closed enhancement (fixed)

Support Sentinel 2 data directly

Reported by: stoecker Owned by: team
Priority: major Milestone: 20.01
Component: Core imagery Version:
Keywords: sentinel gsoc-candidate sotm19 Cc: wiktorn, Klumbumbus, simon04, naoliv, michael2402, lists@…, imagico

Description (last modified by stoecker)

We already have some reused sentinel data in our Maps overview, but that is not really a good solution. Sentinel provides a large amount of imagery and a dedicated interface would be good.

Especially there needs to be a time span selection to easily choose proper data (i.e. no clouds, ...).

Conditions: https://sentinel.esa.int/documents/247904/690755/Sentinel_Data_Legal_Notice
Data directly: http://sentinel-pds.s3-website.eu-central-1.amazonaws.com/
SciHub: https://scihub.copernicus.eu/
A WMS/TMS/WMTS service: http://www.sentinel-hub.com/apps/wms

It looks like asking www.sentinel-hub.com for JOSM/OSM conditions would be the fastest solution, but probably direct access to Sentinel data is worth the effort?

Attachments (2)

14921.patch (5.5 KB) - added by simon04 2 months ago.
2020-01-19-225834_607x106_scrot.png (4.7 KB) - added by simon04 2 months ago.

Download all attachments as: .zip

Change History (45)

comment:1 Changed 3 years ago by stoecker

Description: modified (diff)

comment:2 Changed 3 years ago by stoecker

Cc: Klumbumbus added

comment:3 in reply to:  description Changed 3 years ago by bastiK

Sentinel-2 with 10m resolution is a better than Landsat (30m), but not that much better: You can barely see individual buildings. The value is really in the frequent updates (complete earth coverage about every 10 days).

One use case I can see already is to observe bodies of water over the seasons, so you can distinguish proper lakes from temporarily flooded areas ("Überflutungswiesen").

Replying to stoecker:

It looks like asking www.sentinel-hub.com for JOSM/OSM conditions would be the fastest solution,

Yes, it looks like they provide all that we could ask for.

but probably direct access to Sentinel data is worth the effort?

The data on the AWS will be hard to use directly: It is in proprietary jpeg 2000 format and the tiles are rather big (~300 MB each).

comment:4 Changed 3 years ago by wiktorn

The other use case is to check how the road network looks after major (re-)construction.

I'm not sure about the conditions as from different parts I see different conditions, as for example here where I come from AWS S3 page

(note - only non-commercial use).

Sentinel Hub does cutting and reformatting to jpeg/png/tiff, so probably - we would need to implement some kind of their WMS-Configurator to allow our users to choose images easily.

Accessing S3 directly would require to setup something similar to Sentinel Hub on our infrastructure.

comment:5 in reply to:  4 Changed 3 years ago by stoecker

Sentinel Hub does cutting and reformatting to jpeg/png/tiff, so probably - we would need to implement some kind of their WMS-Configurator to allow our users to choose images easily.

As far as my tests showed they provide normal WMS/TMS/WMTS service. With an additional TIME parameter for image selection.

comment:6 in reply to:  4 Changed 3 years ago by bastiK

Replying to wiktorn:

I'm not sure about the conditions as from different parts I see different conditions, as for example here where I come from AWS S3 page

(note - only non-commercial use).

I don't see any ambiguity here, quote:

Any Sentinel data available through the Sentinel Data Hub will be governed by the Legal Notice on the use of Copernicus Sentinel Data and Service Information.

Any other contents of ESA's Sentinel Data Hub websites are intended for non-commercial use.

We don't want to use any contents other than the Sentinel data, so it is fine.

Sentinel Hub does cutting and reformatting to jpeg/png/tiff, so probably - we would need to implement some kind of their WMS-Configurator to allow our users to choose images easily.

Accessing S3 directly would require to setup something similar to Sentinel Hub on our infrastructure.

Maybe possible for a single time slice, but all the data? This would require a major investment in hardware and a person working full time on it ...

comment:7 Changed 3 years ago by Don-vip

Milestone: 17.0717.08

comment:8 Changed 3 years ago by Don-vip

Milestone: 17.0817.09

comment:9 Changed 3 years ago by Don-vip

Another way to access data from Sentinel products, from the CNES (French Space Agency): https://peps.cnes.fr
There are web services / OpenSearch API described (in French) here: https://peps.cnes.fr/rocket/plus/plus.htm

Last edited 3 years ago by Don-vip (previous) (diff)

comment:10 Changed 3 years ago by bastiK

It doesn't look like they have options to download smaller tiles, just the areas of several square km like ESA. They have a "suggestions" box, maybe we can ask them to add WMS/TMS/WMTS service? :)

comment:11 Changed 3 years ago by Don-vip

I can ask :)

comment:12 Changed 3 years ago by Don-vip

Done. I note also in the project description the following evolution: "By 2018, a European platform involving the Big Data and Cloud industries will make Copernicus data available to all citizens, scientists and industry worldwide." Maybe this one will offer easier access if PEPS does not.

comment:13 Changed 3 years ago by Don-vip

Good news! I have a positive answer from CNES:

the functionality to offer TMS/WMS/WMMS/WMTS services is planned, under development and will be available soon.

\o/

Version 0, edited 3 years ago by Don-vip (next)

comment:14 Changed 3 years ago by Don-vip

Hello,
On Tuesday morning I will likely meet the IGN (French Geographic Agency) in the scope of this project:
https://landsense.eu/Themes/Urban-Landscape-Dynamics
I don't know anything about it yet, but they're talking about "OpenStreetMap tools", and:

In the mid-Pyrenees, including the city of Toulouse in France, IGN France will engage citizens and public authorities in updating a number of their mapping products. Using the LandSense change detection service, citizens will become detectives and investigate urban change in real-time supported by the European Sentinel-2 satellite image data infrastructure.

I'll let you know of the implications, this is exciting :)

comment:15 Changed 3 years ago by bastiK

Great, sounds very promising!

comment:16 Changed 3 years ago by Don-vip

Cc: simon04 added

The project is, among other things, dealing with improving landuse/landcover thanks to automatic change detection based on Sentinel-2 imagery. But the changes are provided as a service, not the imagery itself. I asked about our interest for a direct imagery access, they told me they will see with LandSense (it's not the IGN who run this service but GeoVille). Stay tuned!

@Simon: there's something fun in Austria. We will also use this application in Toulouse, but without the money :D Are you by chance in contact with GeoVille, IIASA, UBA, G2K? This project is definitively led by Austria!

comment:17 Changed 3 years ago by Don-vip

Milestone: 17.0917.10

comment:18 Changed 2 years ago by bastiK

If you like to check out how the Sentinel 2 data looks in JOSM: The people from https://s2maps.eu/ have compiled a cloud-free snapshot and provide WMTS access. The attribution requirement is not compatible with OSM, though.

They may allow OSM use if we ask nicely, but I still see the value of Sentinel 2 in the frequent updates, not a single snapshot.

comment:19 in reply to:  18 Changed 2 years ago by Don-vip

Replying to bastiK:

I still see the value of Sentinel 2 in the frequent updates, not a single snapshot.

I agree. I'd love to implement a GUI in JOSM allowing to update imagery background based on date & cloud settings.

comment:20 Changed 2 years ago by Don-vip

Milestone: 17.1017.11

comment:21 Changed 2 years ago by Don-vip

Milestone: 17.1117.12

comment:22 Changed 2 years ago by Don-vip

Keywords: sentinel added
Milestone: 17.1218.01

comment:23 Changed 2 years ago by Don-vip

See #15749 where support for jpeg 2000 is asked, but I closed it as wontfix as I don't see how we could support this format in JOSM.

comment:24 Changed 2 years ago by Don-vip

Cc: naoliv added

comment:25 Changed 2 years ago by Don-vip

Milestone: 18.01

To reschedule when we have concrete news

comment:26 Changed 16 months ago by gmilcinski

Dear OSM community,
Bence has reached to us as we were not aware of this need/ticket.

We at Sentinel Hub are happy to support your cause and provide you free access to our services. As there are some costs attached to operation of the services we have to introduce some limits, e.g. 1 MIO tiles per month. Based on our discussion with Bence, this should be sufficient due to the nature (and limitations) of S-2 data.

I am not familiar with the OSM tooling (but most happy user of the data, thank you for this effort) but I believe it would be best if the service would be integrated in a way so that user can choose date and cloud parameters, to be able to get the best image possible. Similarly as it is done in Sentinel Playground (http://apps.sentinel-hub.com/sentinel-playground/).

There is QGIS plugin in place for Sentinel Hub as well, if it makes it any easier:
https://medium.com/sentinel-hub/control-sentinel-hub-from-within-qgis-2a83eb7f13db
https://plugins.qgis.org/plugins/SentinelHub/

If there is someone on your side willing to put an effort into integration, he/she should reach to us at info@… and we will communicate details.

Best,
Grega

comment:27 Changed 16 months ago by Don-vip

Cc: michael2402 added
Keywords: gsoc-candidate added

Hi Grega,
Thanks a lot! I think a JOSM Sentinel Hub plugin would make a great GSoC project for next summer, I would be happy to mentor it.
The QGIS plugin will definitively be helpful for this project.

comment:28 Changed 15 months ago by Aun Johnsen <lists@…>

Cc: lists@… added

comment:29 Changed 14 months ago by simon@…

eox.at already has a global, cloudless, sentinel layer available. I'm just in the process of nailing down the few licence issues and then I suspect we will be able to use it directly (a plugin would have the advantage of being able to use current imagery, but ease of use would point rather to a layer).

comment:30 Changed 12 months ago by katpatuka

Latest news (thanks simonpoole):

For use in josm add WMS address https://tiles.maps.eox.at/wms?service=wms&request=getcapabilities to get a list of layers from eox.at (select Store WMS endpoint only) - Note: selecting WMS layer entries including 4326 do not work but show a HTTP 400 error.

Last edited 12 months ago by katpatuka (previous) (diff)

comment:32 Changed 12 months ago by OSM-user:smaprs

Great achievement. Just for info, I tried it right now, and noticed that the resolution of [tiles.maps.eox.at...] is far below that of sentinel-hub; can't trace the axis of a road, or the borders of farmland/wood. I've been testing [services.sentinel-hub.com/ogc/wms/...] from personal account, its TCI/True Color Image is about 10m/px. Image comparing EOX vs Sentinel-Hub at https://i.imgur.com/k3l04vL.png. I guess that EOX image won't be much useful.

comment:33 Changed 6 months ago by simon04

Cc: imagico added
Keywords: sotm19 added

code-de.org provides a WMS service which allows specifying the creation interval using the WMS parameter time= (such as time=2019-08-31T00:00:00Z/2019-09-01T00:00:00Z): https://code-de.org/en/view-record/39b6073d-7ef4-4aa9-a3e4-07f079709b52

This could be integrated into JOSM in the following way: The user adds this WMS layer; JOSM asks the user to input the interval (maybe by showing simple text box); the selected interval is pre-filled for the next layer usage.

comment:34 Changed 6 months ago by imagico

Example for a custom image layer entry in JOSM would be:
https://geoservice.code-de.org/Sentinel2/wms?SERVICE=WMS&VERSION=1.1.0&REQUEST=GetMap&FORMAT=image%2Fpng&TRANSPARENT=true&LAYERS=S2_MSI_L1C&time=2019-09-09T00%3A00%3A00.000Z%2F2019-09-24T00%3A00%3A00.000Z&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}

At the low zoom levels you get image footprints, at the higher zoom levels the images themselves. Images are recorded in most land areas every five days and are available usually within a day or two after recording.

Slightly off-topic: The same could also be used for GIBS (https://earthdata.nasa.gov/gibs) which offers low resolution layers with daily near real time updates - see https://wiki.earthdata.nasa.gov/display/GIBS/GIBS+Available+Imagery+Products

Changed 2 months ago by simon04

Attachment: 14921.patch added

Changed 2 months ago by simon04

comment:35 Changed 2 months ago by simon04

Milestone: 20.01
Owner: changed from team to simon04
Status: newassigned
Summary: Support Sentinel 2 data directly[Patch] Support Sentinel 2 data directly

With attachment:14921.patch​ the user is asked to input a time filter for the following WMS definition:

  • URL: wms:https://geoservice.code-de.org/Sentinel2/ows?FORMAT=image/jpeg&VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&LAYERS=S2_MSI_L1C&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}&time={time}


The input string in inserted into the {time} template parameter.

comment:36 Changed 2 months ago by Don-vip

It's fine for a first step. In the long term we should offer a proper UI. We should enhance the JavaFX plugin to propose DatePicker instead of a text field.

comment:37 Changed 2 months ago by simon04

In 15739/josm:

see #14921 - WMS: support time={time} for Sentinel-2 imagery

comment:38 Changed 2 months ago by simon04

Summary: [Patch] Support Sentinel 2 data directlySupport Sentinel 2 data directly

comment:39 Changed 2 months ago by Don-vip

In 15744/josm:

see #14921 - fix NPE

comment:40 Changed 2 months ago by simon04

Owner: changed from simon04 to team
Status: assignednew

comment:42 Changed 2 months ago by Don-vip

Resolution: fixed
Status: newclosed

comment:43 Changed 2 months ago by Don-vip

In 15810/josm:

see #14921 - hardcode valid time range for Sentinel-2 sources in imagery integration test

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain team.
as The resolution will be set.
The resolution will be deleted.

Add Comment


E-mail address and name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.