Modify

Opened 16 months ago

Last modified 3 months ago

#22596 new enhancement

[Patch] Elevation at current map location / Automatic download of SRTM HGT files

Reported by: hhtznr Owned by: team
Priority: normal Milestone:
Component: Plugin elevationprofile Version:
Keywords: elevation, SRTM HGT, current map location, automatic download Cc: OliverW

Description (last modified by hhtznr)

What?
I wrote this patch, which will display the elevation at the current location on the map next to the coordinates in the MapStatus line. Optionally, the necessary SRTM HGT files providing the elevation can automatically be downloaded, if they are missing on disk.

Why?
This is typical functionality in most map applications, but it is missing in JOSM. Elevation at the current map location (mouse pointer) can be useful in determining the proper direction of waterways, the location of summits etc.

How does it work?
It builds on HgtReader from ElevationProfile plugin written by Oliver Wieland. HgtReader checks whether the HGT file covering the current map location is available on disk. If so, it reads it into memory and determines the elevation at the current location.

I extended HgtReader with HgtDownloader, which will download any missing HGT file in a background thread from NASA's Land Processes Distributed Active Archive Center (LP DAAC) at https://e4ftl01.cr.usgs.gov/MEASURES/SRTMGL3.003/2000.02.11/.

I created a preferences tab, where the elevation feature can be enabled and where optional auto-downloading can turned on.

Why not as plugin?
I did not see a way, how to feed the current map location (mouse pointer) into a plugin.

What's the content of the patch?
New files:

  • src/org/openstreetmap/josm/data/elevation/ElevationHelper.java (from Oliver Wieland)
  • src/org/openstreetmap/josm/data/elevation/HgtDownloader.java
  • src/org/openstreetmap/josm/data/elevation/HgtDownloadListener.java
  • src/org/openstreetmap/josm/data/elevation/HgtPreferences.java
  • src/org/openstreetmap/josm/data/elevation/HgtReader.java (builds on Oliver Wieland)
  • src/org/openstreetmap/josm/data/elevation/gpx/GeoidCorrectionKind.java (from Oliver Wieland)
  • src/org/openstreetmap/josm/gui/preferences/elevation/HgtPreference.java
  • src/org/openstreetmap/josm/gui/preferences/elevation/HgtPreferencesPanel.java
  • resources/images/preferences/elevation.svg
  • resources/images/statusline/ele.svg

Updated files:

  • src/org/openstreetmap/josm/gui/MapStatus.java
  • src/org/openstreetmap/josm/gui/preferences/PreferenceTabbedPane.java

What next?
I would appreciate it, if the demonstrated functionality could be integrated into mainline JOSM.

Screenshots
Screenshot of elevation information in MapStatus line Screenshot of elevation data preferences tab

Attachments (7)

JOSM_18616_Elevation_git_diff.txt (63.8 KB ) - added by hhtznr 16 months ago.
Git diff of elevation patch
JOSM_18616_Elevation_new_updated_src_files.tar.gz (32.5 KB ) - added by hhtznr 16 months ago.
Archive with new and updated source files
JOSM_with_elevation_in_MapSatus.png (3.6 MB ) - added by hhtznr 16 months ago.
Screenshot of elevation information in MapStatus line
JOSM_Elevation_Data_Preferences.png (145.7 KB ) - added by hhtznr 16 months ago.
Screenshot of elevation data preferences tab
JOSM_ElevationProfile_18494.diff (58.5 KB ) - added by hhtznr 16 months ago.
Git diff of reworked elevation patch against ElevationProfile
HttpClient.diff (2.8 KB ) - added by hhtznr 5 months ago.
JosmNasaEarthdataAuthTest.java (3.8 KB ) - added by hhtznr 5 months ago.

Change History (37)

by hhtznr, 16 months ago

Git diff of elevation patch

by hhtznr, 16 months ago

Archive with new and updated source files

by hhtznr, 16 months ago

Screenshot of elevation information in MapStatus line

by hhtznr, 16 months ago

Screenshot of elevation data preferences tab

comment:1 by hhtznr, 16 months ago

Description: modified (diff)

comment:2 by hhtznr, 16 months ago

Description: modified (diff)

comment:3 by hhtznr, 16 months ago

Description: modified (diff)

comment:4 by taylor.smock, 16 months ago

Why not as plugin?
I did not see a way, how to feed the current map location (mouse pointer) into a plugin.

It is possible to do so:

        Point mousePosition = MainApplication.getMap().mapView.getMousePosition();
        LatLon latLon = MainApplication.getMap().mapView.getLatLon(mousePosition.getX(), mousePosition.getY());

Also, you should have renamed attachment:JOSM_18616_Elevation_git_diff.txt such that it ended with .diff or .patch.

Anyway, skimming through your patch:

  • Why unzip the HGT file? It takes more space on disk, and there are several methods in Java to get a decompressed input stream. If you profiled it, and it took fewer resources, that would be helpful to know.
  • Please don't set the HttpClient factory. (HttpClient.setFactory(Http1Client::new);). This should only be set by JOSM itself or a plugin which provides its own factory (like the http2 plugin).
  • I'm not a fan of the TODO comments sprinkled throughout the code
  • Is there any documentation for the login process for the government server? Does it use OAuth 2.0? If so, does it allow redirects to localhost/127.0.0.1? I've got a WIP patch for OAuth 2.0 support in JOSM, and it takes advantage of the remote control server to receive initial server redirect.

in reply to:  4 ; comment:5 by hhtznr, 16 months ago

Replying to taylor.smock:

Why not as plugin?
I did not see a way, how to feed the current map location (mouse pointer) into a plugin.

It is possible to do so:

        Point mousePosition = MainApplication.getMap().mapView.getMousePosition();
        LatLon latLon = MainApplication.getMap().mapView.getLatLon(mousePosition.getX(), mousePosition.getY());

Thank you for pointing this out. It helped me to figure out how to do it outside the scope of MapStatus. I think, your proposal would not work to get the position as a result of mouse motion. But being pointed to mapView, I came across that it could be done like this, copying the listener code from MapStatus into an own plugin class and putting it in Plugin.mapFrameInitialized:

@Override
public void mapFrameInitialized(MapFrame oldFrame, MapFrame newFrame) {
    super.mapFrameInitialized(oldFrame, newFrame);

    if (newFrame != null) {
        newFrame.mapView.addMouseMotionListener(new MouseMotionListener() {
            @Override
            public void mouseDragged(MouseEvent e) {
                mouseMoved(e);
            }

            @Override
            public void mouseMoved(MouseEvent e) {
                if (newFrame.mapView.getCenter() == null)
                    return;
                // Do not update the view if ctrl or right button is pressed.
                if ((e.getModifiersEx() & (MouseEvent.CTRL_DOWN_MASK | MouseEvent.BUTTON3_DOWN_MASK)) == 0)
                    updateEleText(newFrame.mapView.getLatLon(e.getX(), e.getY()));
            }
        });

        }
    }
}

Also, you should have renamed attachment:JOSM_18616_Elevation_git_diff.txt such that it ended with .diff or .patch.

Anyway, skimming through your patch:

  • Why unzip the HGT file? It takes more space on disk, and there are several methods in Java to get a decompressed input stream. If you profiled it, and it took fewer resources, that would be helpful to know.

Modern file systems use compression so it does not take more space when the original file is decompressed. Should I create a config option so the user can decide whether to decompress the downloaded files or to leave them as-is?

  • Please don't set the HttpClient factory. (HttpClient.setFactory(Http1Client::new);). This should only be set by JOSM itself or a plugin which provides its own factory (like the http2 plugin).

Is this better?

Http1Client httpClient = new Http1Client(url, "GET");
  • I'm not a fan of the TODO comments sprinkled throughout the code

NP, can be deleted.

  • Is there any documentation for the login process for the government server? Does it use OAuth 2.0? If so, does it allow redirects to localhost/127.0.0.1? I've got a WIP patch for OAuth 2.0 support in JOSM, and it takes advantage of the remote control server to receive initial server redirect.

They have documentation here: https://urs.earthdata.nasa.gov/documentation

It's not really explained in depth, but it seems they use OAuth2: https://urs.earthdata.nasa.gov/documentation/for_users/oauth2

They also have a Java code example, which uses java.net.HttpURLConnection: https://urs.earthdata.nasa.gov/documentation/for_users/data_access/java

However, it uses basic authorization with username + password instead of the authorization bearer. Would this be preferred? I thought otherwise.

Sorry, I'm absolutely no expert on this web stuff. So some support on this would be appreciated.

in reply to:  5 comment:6 by taylor.smock, 16 months ago

Replying to hhtznr:

Modern file systems use compression so it does not take more space when the original file is decompressed. Should I create a config option so the user can decide whether to decompress the downloaded files or to leave them as-is?

Many of the common linux filesystems do not enable transparent compression by default. BTRFS requires the user to pick a compression type, ext4 doesn't support transparent compression, and so on.

  • Please don't set the HttpClient factory. (HttpClient.setFactory(Http1Client::new);). This should only be set by JOSM itself or a plugin which provides its own factory (like the http2 plugin).

Is this better?

Http1Client httpClient = new Http1Client(url, "GET");

Not really. You should be using the factory (HttpClient.create), and the variable type should just be the base HttpClient type instead of a specific subclass. All of the methods you called on Http1Client and Http1Response are available in the super class/interface (HttpClient and HttpResponse respectively).

  • Is there any documentation for the login process for the government server? Does it use OAuth 2.0? If so, does it allow redirects to localhost/127.0.0.1? I've got a WIP patch for OAuth 2.0 support in JOSM, and it takes advantage of the remote control server to receive initial server redirect.

They have documentation here: https://urs.earthdata.nasa.gov/documentation

It's not really explained in depth, but it seems they use OAuth2: https://urs.earthdata.nasa.gov/documentation/for_users/oauth2

They also have a Java code example, which uses java.net.HttpURLConnection: https://urs.earthdata.nasa.gov/documentation/for_users/data_access/java

However, it uses basic authorization with username + password instead of the authorization bearer. Would this be preferred? I thought otherwise.

Sorry, I'm absolutely no expert on this web stuff. So some support on this would be appreciated.

It does look like they do use OAuth 2, and that would be greatly preferred over username + password. See #20768 for the WIP patch for JOSM OAuth 2 support.

comment:7 by hhtznr, 16 months ago

Is a "display local elevation" feature of interest for integration into the main application or should I rather convert my proposal into a plugin?

in reply to:  7 ; comment:8 by taylor.smock, 16 months ago

Replying to hhtznr:

Is a "display local elevation" feature of interest for integration into the main application or should I rather convert my proposal into a plugin?

I'd probably just apply the patch to the ElevationProfile plugin.

in reply to:  8 comment:9 by hhtznr, 16 months ago

Replying to taylor.smock:

Replying to hhtznr:

Is a "display local elevation" feature of interest for integration into the main application or should I rather convert my proposal into a plugin?

I'd probably just apply the patch to the ElevationProfile plugin.

OK, I'll consider your valued feedback on HttpClient and decompression, incorporate my code into ElevationProfile and send a new diff/patch.

by hhtznr, 16 months ago

Git diff of reworked elevation patch against ElevationProfile

comment:10 by hhtznr, 16 months ago

I integrated my code into ElevationProfile plugin and created this diff: https://josm.openstreetmap.de/attachment/ticket/22596/JOSM_ElevationProfile_18494.diff

An incompatible change is the directory, where the HGT files are saved.

The original code used:

for (String location : Preferences.getAllPossiblePreferenceDirs()) {
    String fullPath = new File(location + File.separator + "elevation", file).getPath();
    [...]
}

I use ...

public static final File DEFAULT_HGT_DIRECTORY =
    Paths.get(Preferences.main().getDirs().getCacheDirectory(true).toString(), "elevation", "SRTM3").toFile();

... to make sure, I have a directory that is writable when downloading and to ensure that SRTM3 HGT files can be distinguished from other elevation data in case that this should be necessary in the future.

What is your opinion on this patch?

comment:11 by skyper, 16 months ago

Component: CorePlugin elevationprofile
Owner: changed from team to OliverW
Version: latest

comment:12 by skyper, 16 months ago

Cc: OliverW added
Owner: changed from OliverW to team

comment:13 by hhtznr, 6 months ago

Did you ever consider this proposal/patch?

comment:14 by hhtznr, 6 months ago

I published my Elevation plugin on GitHub: https://github.com/hhtznr/JOSM-Elevation-Plugin

It does not include the other functionality from the ElevationProfile plugin. So it will not interfere.

comment:15 by gaben, 6 months ago

@hhtznr: Is it possible to download elevation data at specific node locations?

If yes, I may fork it :)

comment:16 by hhtznr, 6 months ago

Provided a suitable SRTM file is available, the SRTM file reader returns the elevation value where the location in the SRTM tile raster is closest to a given org.openstreetmap.josm.data.coor.ILatLon (no interpolation). Therefore, it could also be used to get the approximate elevation of a node on the map.

However, as the current functionality is focused on providing the elevation at the mouse pointer location on the map, file reading and downloading is done in a non-blocking manner (separate threads). Due to this, the SRTM file reader will report a data void instead of a usable elevation value as long as the relevant SRTM tile has not been cached.

From my perspective, obtaining the elevation of a node should encompass interpolation and waiting for usable values to be available.

in reply to:  13 ; comment:17 by taylor.smock, 6 months ago

Replying to hhtznr:

Did you ever consider this proposal/patch?

I forget stuff. And sometimes more important things come up. Like fixing CI issues.

If a patch doesn't have a response after two weeks, ping it (and cc me in the ticket if I'm not already involved). And keep pinging until someone (probably me) responds.

Anyway, as I noted in comment:6, the authentication information can be obtained via oauth 2. JOSM has support for oauth 2. I would strongly prefer to use that support.

Beyond that, I'm not a fan of the singleton instance for HgtReader. In most cases, if you "need" a singleton, you can get away with static methods. There are a few exceptions. In this case, I think the exception is for having multiple instances with different configuration. In this case, there is only one instance of a new HgtReader; in the singleton method. I probably would have made a private static class for the HgtDownloadListener (the other possible exception).

I'll try to find time to take a deeper look this week.

in reply to:  17 comment:18 by hhtznr, 6 months ago

Replying to taylor.smock:

Replying to hhtznr:

Did you ever consider this proposal/patch?

I forget stuff. And sometimes more important things come up. Like fixing CI issues.

If a patch doesn't have a response after two weeks, ping it (and cc me in the ticket if I'm not already involved). And keep pinging until someone (probably me) responds.

Anyway, as I noted in comment:6, the authentication information can be obtained via oauth 2. JOSM has support for oauth 2. I would strongly prefer to use that support.

Beyond that, I'm not a fan of the singleton instance for HgtReader. In most cases, if you "need" a singleton, you can get away with static methods. There are a few exceptions. In this case, I think the exception is for having multiple instances with different configuration. In this case, there is only one instance of a new HgtReader; in the singleton method. I probably would have made a private static class for the HgtDownloadListener (the other possible exception).

I'll try to find time to take a deeper look this week.

Thank you for the reply. I'm planning to work on the plugin during the cold period and would propose pinging you for a review afterwards.

My from-the-scratch version at https://github.com/hhtznr/JOSM-Elevation-Plugin already contains a few features that were not included in last year's patch:

  • Handling of SRTM1 datasets in addition to SRTM3
  • Size limit of in-memory cache
  • More work delegation to threads
  • Disabling of elevation functionality if zoomed out too much
  • Timer triggered caching of SRTM tiles to cover complete map view

I have more ideas in mind and will also consider your recommendations.

in reply to:  17 comment:19 by hhtznr, 5 months ago

Replying to taylor.smock:

Replying to hhtznr:
Anyway, as I noted in comment:6, the authentication information can be obtained via oauth 2. JOSM has support for oauth 2. I would strongly prefer to use that support.

I considered your proposal to use OAuth2 for authentication at NASA Earthdata servers instead of using an authorization bearer token.

It is straightforward, to do it the way proposed by NASA ...

... following their redirects, doing basic authentication when arriving at their URS server and then being granted session authorization.

However, I do not see how to get it to work based on the JOSM API. There are two major obstacles:

  1. The OAuth2 code in JOSM looks very OSM centric to me, for example the last two assignments in the OAuthParameters constructor:
    public OAuthParameters(String consumerKey, String consumerSecret,
                           String requestTokenUrl, String accessTokenUrl,
                           String authoriseUrl, String osmLoginUrl, String osmLogoutUrl) {
        this.consumerKey = consumerKey;
        this.consumerSecret = consumerSecret;
        this.requestTokenUrl = requestTokenUrl;
        this.accessTokenUrl = accessTokenUrl;
        this.authoriseUrl = authoriseUrl;
        this.osmLoginUrl = osmLoginUrl;
        this.osmLogoutUrl = osmLogoutUrl;
    }
    
    So, at first glance, it looks as if it might be necessary to subclass/override some of the code to get it to work with non-OSM servers.
  1. HttpClient handles redirects immediately in its connect() method, where it removes the authorization header when being redirected. This is a good idea when being redirected somewhere to prevent leaking the auth data. But this prevents doing OAuth the NASA way following their redirects and putting the header in place when arriving at their auth host. Unluckily, I cannot find a documented approach how to directly connect their URS server in order to get auth done without the need to follow redirects.

The two solutions that I see at the moment are:

  1. Stick to HttpClient and at the same time stick to the authorization bearer token. This is not a perfect solution as the user not only has to generate the token once on the NASA pages, but each time that a generated token has expired.
  1. Use the NASA proposed OAuth2 approach. This would mean dropping HttpClient in favor of HttpURLConnection, not using JOSM OAuth2 code at all, but trying to make use of CredentialsManager to get username and password stored the JOSM way.

Can you give me a few hints how to resolve this? Is there a more elegant third way that I just don't see?

Last edited 5 months ago by hhtznr (previous) (diff)

comment:20 by taylor.smock, 5 months ago

However, I do not see how to get it to work based on the JOSM API. There are two major obstacles:

  1. You want OAuth20Parameters, not OAuthParameters. It doesn't (currently) refresh the token, but with additional users, it will be worthwhile to add that ability.
  2. We had an issue where we were sending authentication information to servers that were not the original server. Example: osm -> aws for GPX traces. Feel free to file a patch so that we can send authentication information to servers that match the following:
    • Same protocol (http != https) -- http -> https is probably fine, but I'd rather not define the security of the protocol.
    • Same host (data.usgs.gov != other.data.usgs.gov)
Last edited 5 months ago by taylor.smock (previous) (diff)

in reply to:  20 comment:21 by hhtznr, 5 months ago

Replying to taylor.smock:

However, I do not see how to get it to work based on the JOSM API. There are two major obstacles:

  1. You want OAuth20Parameters, not OAuthParameters. It doesn't (currently) refresh the token, but with additional users, it will be worthwhile to add that ability.

I will have a look. I missed this out as I am currently building the plugin for an older version of JOSM.

  1. We had an issue where we were sending authentication information to servers that were not the original server. Example: osm -> aws for GPX traces. Feel free to file a patch so that we can send authentication information to servers that match the following:
    • Same protocol (http != https) -- http -> https is probably fine, but I'd rather not define the security of the protocol.
    • Same host (data.usgs.gov != other.data.usgs.gov)

With NASA, we would always want to send auth information to a different host. However, we know the hostname in advance as it is specified. We want to get the desired resource from e4ftl01.cr.usgs.gov, but we will be redirected to urs.earthdata.nasa.gov for auth and redirected back to the resource on the initial host afterwards.

Last edited 5 months ago by hhtznr (previous) (diff)

by hhtznr, 5 months ago

Attachment: HttpClient.diff added

by hhtznr, 5 months ago

in reply to:  20 ; comment:22 by hhtznr, 5 months ago

Replying to taylor.smock:

However, I do not see how to get it to work based on the JOSM API. There are two major obstacles:

  1. You want OAuth20Parameters, not OAuthParameters. It doesn't (currently) refresh the token, but with additional users, it will be worthwhile to add that ability.
  2. We had an issue where we were sending authentication information to servers that were not the original server. Example: osm -> aws for GPX traces. Feel free to file a patch so that we can send authentication information to servers that match the following:
    • Same protocol (http != https) -- http -> https is probably fine, but I'd rather not define the security of the protocol.
    • Same host (data.usgs.gov != other.data.usgs.gov)

With the OAuth20[...] API from JOSM it also does not seem to be feasible to authenticate at the NASA servers. But the good thing is that it is not necessary at all and the approach proposed by NASA for contacting their servers is much simpler.

I attached a small patch for HttpClient and a small demo how to use it in this scope.

The patch is not elegant, it is just intended to show how HttpClient could be adapted to cooperate.

When running the test ...

Accessing resource https://e4ftl01.cr.usgs.gov/MEASURES/SRTMGL3.003/2000.02.11/N44E010.SRTMGL3.hgt.zip.xml
2023-11-17 19:17:56.280 INFO: GET https://e4ftl01.cr.usgs.gov/MEASURES/SRTMGL3.003/2000.02.11/N44E010.SRTMGL3.hgt.zip.xml -> HTTP/1.1 302 (862 ms; 503 B)
2023-11-17 19:17:56.283 INFO: Download redirected to https://urs.earthdata.nasa.gov/oauth/authorize?scope=uid&app_type=401&client_id=ijpRZvb9qeKCK5ctsn75Tg&response_type=code&redirect_uri=https%3A%2F%2Fe4ftl01.cr.usgs.gov%2Foauth&state=aHR0cHM6Ly9lNGZ0bDAxLmNyLnVzZ3MuZ292L01FQVNVUkVTL1NSVE1HTDMuMDAzLzIwMDAuMDIuMTEvTjQ0RTAxMC5TUlRNR0wzLmhndC56aXAueG1s
2023-11-17 19:17:56.923 INFO: GET https://urs.earthdata.nasa.gov/oauth/authorize?scope=uid&app_type=401&client_id=ijpRZvb9qeKCK5ctsn75Tg&response_type=code&redirect_uri=https%3A%2F%2Fe4ftl01.cr.usgs.gov%2Foauth&state=aHR0cHM6Ly9lNGZ0bDAxLmNyLnVzZ3MuZ292L01FQVNVUkVTL1NSVE1HTDMuMDAzLzIwMDAuMDIuMTEvTjQ0RTAxMC5TUlRNR0wzLmhndC56aXAueG1s -> HTTP/1.1 302 (639 ms)
2023-11-17 19:17:56.923 INFO: Download redirected to https://e4ftl01.cr.usgs.gov/oauth?code=ep1Pmzw6iRYMHSaAVAZButda8oaUBJQnm4ZtRUX-4zYDzfmn3ILpiJhawmgV22u7ZA&state=aHR0cHM6Ly9lNGZ0bDAxLmNyLnVzZ3MuZ292L01FQVNVUkVTL1NSVE1HTDMuMDAzLzIwMDAuMDIuMTEvTjQ0RTAxMC5TUlRNR0wzLmhndC56aXAueG1s
2023-11-17 19:17:56.924 INFO: Download redirected to different host (urs.earthdata.nasa.gov -> e4ftl01.cr.usgs.gov), removing authorization headers
2023-11-17 19:17:57.776 INFO: GET https://e4ftl01.cr.usgs.gov/oauth?code=ep1Pmzw6iRYMHSaAVAZButda8oaUBJQnm4ZtRUX-4zYDzfmn3ILpiJhawmgV22u7ZA&state=aHR0cHM6Ly9lNGZ0bDAxLmNyLnVzZ3MuZ292L01FQVNVUkVTL1NSVE1HTDMuMDAzLzIwMDAuMDIuMTEvTjQ0RTAxMC5TUlRNR0wzLmhndC56aXAueG1s -> HTTP/1.1 302 (851 ms; 271 B)
2023-11-17 19:17:57.777 INFO: Download redirected to https://e4ftl01.cr.usgs.gov/MEASURES/SRTMGL3.003/2000.02.11/N44E010.SRTMGL3.hgt.zip.xml
2023-11-17 19:17:58.352 INFO: GET https://e4ftl01.cr.usgs.gov/MEASURES/SRTMGL3.003/2000.02.11/N44E010.SRTMGL3.hgt.zip.xml -> HTTP/1.1 200 (575 ms; 2.86 kB)
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE GranuleMetaDataFile SYSTEM "http://ecsinfo.gsfc.nasa.gov/ECSInfo/ecsmetadata/dtds/DPL/ECS/ScienceGranuleMetadata.dtd">
[....]

Accessing resource https://e4ftl01.cr.usgs.gov/MEASURES/SRTMGL3.003/2000.02.11/N44E011.SRTMGL3.hgt.zip.xml
2023-11-17 19:17:58.915 INFO: GET https://e4ftl01.cr.usgs.gov/MEASURES/SRTMGL3.003/2000.02.11/N44E011.SRTMGL3.hgt.zip.xml -> HTTP/1.1 200 (548 ms; 2.87 kB)
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE GranuleMetaDataFile SYSTEM "http://ecsinfo.gsfc.nasa.gov/ECSInfo/ecsmetadata/dtds/DPL/ECS/ScienceGranuleMetadata.dtd">
[...]

... we are redirected for auth to the URS server the first time that we request a resource from the download server. There, we do basic auth with our Earthdata credentials. The next time, the download server immediately provides us the requested resource. We are recognized based on a session cookie.

What is your opinion on this solution?

in reply to:  22 ; comment:23 by stoecker, 5 months ago

The patch is not elegant, it is just intended to show how HttpClient could be adapted to cooperate.

It's not bad I'd say.

in reply to:  23 comment:24 by hhtznr, 5 months ago

Replying to stoecker:

It's not bad I'd say.

Any further considerations on this patch?

comment:25 by taylor.smock, 5 months ago

I don't think so. Poke this ticket in a few days so I don't forget about this. I just did a release, and I have a policy of not doing non-urgent fixes for a few days prior and after the release.

in reply to:  25 comment:26 by hhtznr, 5 months ago

Replying to taylor.smock:

I don't think so. Poke this ticket in a few days so I don't forget about this. I just did a release, and I have a policy of not doing non-urgent fixes for a few days prior and after the release.

This is the requested friendly reminder on the HttpClient patch enabling auth upon redirection to a predefined host:

https://josm.openstreetmap.de/attachment/ticket/22596/HttpClient.diff

comment:27 by taylor.smock, 4 months ago

In 18913/josm:

See #22596: Some hosts redirect to another host for authentication (patch by hhtznr, modified)

This lets plugins do authentication with hosts that redirect to another host
for authentication.

in reply to:  27 comment:28 by hhtznr, 4 months ago

Replying to taylor.smock:

In 18913/josm:

See #22596: Some hosts redirect to another host for authentication (patch by hhtznr, modified)

This lets plugins do authentication with hosts that redirect to another host
for authentication.

Great, thanks a lot!

in reply to:  17 comment:29 by hhtznr, 4 months ago

Replying to taylor.smock:

Anyway, as I noted in comment:6, the authentication information can be obtained via oauth 2. JOSM has support for oauth 2. I would strongly prefer to use that support.


With commit 3e08a78, I have updated the Elevation plugin to use the JOSM OAuth 2.0 implementation for authentication based on the authorization bearer token that an Earthdata user can create on the Generate Token tab at Earthdata Login. (I will implement an alternative based on username and password using the patched HttpClient in a later commit.)

To provide feedback on the JOSM OAuth 2.0 implementation:

  • It is quite some research and implementation effort to get the bearer token into the HTTP header this way.
  • In the previous version, I used a dedicated pref parameter to store the token. And I simply added an authorization header containing Bearer _BEARER_TOKEN_ to the HttpClient.
  • Now, the bearer token is still stored in the same preferences file, but there is a lot of additional information with most of it just beeing added because the parameters are not allowed to be null.
  • This piece of code shows what must be added in addition to token_type and access_token to satisfy the constraints of the implementation:
private static OAuth20Token createEarthdataOAuthToken(String bearerToken) {
    // Either client ID or bearer token must be non-null/non-blank
    if (bearerToken == null || Utils.isBlank(bearerToken))
        return null;
    JsonObjectBuilder jsonObjectBuilder = Json.createObjectBuilder();
    jsonObjectBuilder.add("token_type", "bearer");
    jsonObjectBuilder.add("access_token", bearerToken);
    String json = jsonObjectBuilder.build().toString();

    String earthDataURS = "https://" + EARTHDATA_SSO_HOST + "/";
    String earthDataDL = "https://" + EARTHDATA_DOWNLOAD_HOST + "/";
    // Earthdata neither provides us a client ID nor a client secret
    String clientId = "";
    String clientSecret = null;
    // Following two URLs are set because the parameters may not be null, but are
    // never used
    String authorizeUrl = earthDataURS;
    String redirectUri = earthDataURS;
    // These two parameters need to be set to the URL of the SRTM download server
    // Otherwise, OAuth20Parameters will refuse to be initialized and
    // OAuthToken.sign(HttpClient) will refuse to do its job
    String tokenUrl = earthDataDL;
    String apiUrl = earthDataDL;

    OAuth20Parameters parameters = new OAuth20Parameters(clientId, clientSecret, tokenUrl, authorizeUrl, apiUrl,
            redirectUri);
    try {
        return new OAuth20Token(parameters, json);
    } catch (OAuth20Exception e) {
        Logging.error("Elevation: " + e);
        return null;
    }
}
  • It is to note that we do not know a client ID in this specific case. As it may not be null, I set it to a blank string in order to pass the null check in the OAuth20Parameters constructor.
  • It would be more convenient if fields like OAuth20Token.ACCESS_TOKEN and OAuth20Token.TOKEN_TYPE were public.
  • When I got it right from OAuth.com, OAuth 2.0 is quite flexible with respect to the parameters that are in place for a specific use case. Therefore, if it is desired that plugins make use of the OAuth 2.0 implementation, it would be of advantage that more flexible constructors for OAuth20Parameters and OAuth20Token are provided which are not based on assumptions that are specific to OSM.
Last edited 4 months ago by hhtznr (previous) (diff)

in reply to:  17 comment:30 by hhtznr, 3 months ago

Replying to taylor.smock:

I'll try to find time to take a deeper look this week.

Now is the time that I would like to come back to the offered review. I have implemented all the features I had in mind. And I have considered your recommendations. The plugin is working stable for me. So far, no bugs were reported.

The Elevation plugin source is found here: https://github.com/hhtznr/JOSM-Elevation-Plugin

Documentation on configuration and usage is found here: https://wiki.openstreetmap.org/wiki/JOSM/Plugins/Elevation

Looking forward to your feedback!

Modify Ticket

Change Properties
Set your email in Preferences
Action
as new The owner will remain team.
as The resolution will be set. Next status will be 'closed'.
to The owner will be changed from team to the specified user.
Next status will be 'needinfo'. The owner will be changed from team to hhtznr.
as duplicate The resolution will be set to duplicate. Next status will be 'closed'. The specified ticket will be cross-referenced with this ticket.
The owner will be changed from team to anonymous. Next status will be 'assigned'.

Add Comment


E-mail address and name can be saved in the Preferences .
 
Note: See TracTickets for help on using tickets.