Modify

Opened 5 months ago

Closed 3 months ago

Last modified 3 months ago

#18440 closed enhancement (fixed)

Heads up: Maxar implementing new anti-scraping techniques

Reported by: Stereo Owned by: Don-vip
Priority: normal Milestone: 20.02
Component: Core imagery Version:
Keywords: maxar api key Cc: sommerluk

Attachments (0)

Change History (33)

comment:1 Changed 5 months ago by Boggedy

This may also require an auto expire feature for the JOSM imagery cache where Maxar and ESRI tiles are all autodeleted on exit.

comment:2 Changed 5 months ago by sommerluk

Cc: sommerluk added

comment:3 Changed 5 months ago by Don-vip

Ticket #18473 has been marked as a duplicate of this ticket.

comment:4 Changed 5 months ago by Don-vip

Keywords: maxar added

comment:5 Changed 5 months ago by Don-vip

Ticket #18516 has been marked as a duplicate of this ticket.

comment:6 Changed 5 months ago by Don-vip

See https://josm.openstreetmap.de/ticket/18473#comment:3 comments about the removal from default entries

comment:7 Changed 3 months ago by Don-vip

Milestone: 20.02
Owner: changed from team to Don-vip
Status: newassigned

comment:8 Changed 3 months ago by Don-vip

I've received an e-mail from Maxar providing us (JOSM) a new API key to restore access to Maxar imagery. Their only condition is to not make the API key public on the wiki or the source code.

I already began related work in #17744 / [o35017] when the Bing API key was unavailable. I'll work on this tomorrow during GeoFabrik hack week-end.

comment:9 Changed 3 months ago by stoecker

If the key should not be in the wiki or the source, where should it be?

comment:10 Changed 3 months ago by Don-vip

I was thinking about providing it at build time and storing it somewhat in JOSM jar. Not completely hidden, but at least not easily crawlable. They seem ok with this.

comment:11 Changed 3 months ago by stoecker

That's not possible. It would be a license violation of JOSM GPL.

comment:12 Changed 3 months ago by Don-vip

Ok. Does it work if JOSM retrieves the key from JOSM website using a new endpoint? I'm thinking of a POST request with a user-agent check.

comment:13 Changed 3 months ago by stoecker

That would work.

comment:14 Changed 3 months ago by stoecker

If so, please make it expandable to other JOSM specific keys.

comment:15 Changed 3 months ago by Don-vip

In 15853/josm:

see #18440 - non-blocking warning in case of invalid TMS entries

comment:16 Changed 3 months ago by Don-vip

JMapViewer updated in [o35320].

comment:17 in reply to:  14 ; Changed 3 months ago by Don-vip

Replying to stoecker:

If so, please make it expandable to other JOSM specific keys.

What other keys do you have in mind?

comment:18 in reply to:  17 ; Changed 3 months ago by stoecker

Replying to Don-vip:

Replying to stoecker:

If so, please make it expandable to other JOSM specific keys.

What other keys do you have in mind?

We have a few other JOSM specific map keys. If we setup such a feature it should be possible to support other services as well. It should not be Maxar only.

comment:19 Changed 3 months ago by stoecker

E.g.

https://josm.openstreetmap.org/mapkey?id=maxar

returns the maxar key and ...

comment:20 Changed 3 months ago by SimonPoole

@stoecker the GPL in general does not apply to configuration.

comment:21 in reply to:  20 ; Changed 3 months ago by stoecker

Replying to SimonPoole:

@stoecker the GPL in general does not apply to configuration.

That's not configuration. GPL applies to sources and the tools/toolchain to build an executable from this. So there cannot be any secret in the build process. The code signing we do for JOSM is already a border case.

comment:22 in reply to:  18 Changed 3 months ago by Don-vip

Replying to stoecker:

If we setup such a feature it should be possible to support other services as well. It should not be Maxar only.

No problem, this is what I had in mind.
EDIT: These are the entries I found with an API key:

  • Bing
  • City_of_Melbourne_Feb_2019
  • City_of_Melbourne_May_2018
  • geoimage.at
  • geovekst-nib
  • IBGE_DF_Addresses
  • IBGE_Nomes_Ruas
  • IBGE_Salvador_Streets
  • lantmateriet-orto1960
  • lantmateriet-orto1975
  • lantmateriet-topowebb
  • LINZ_NZ_Aerial_Imagery
  • LINZ_NZ_Topo50_Gridless_Maps
  • Mapbox
  • mapbox_locator_overlay
  • Maxar-Premium
  • Maxar-Standard
  • New_and_Misaligned_TIGER_Roads-2013
  • opencylemap
  • tf-landscape
  • US_Forest_Service_roads_overlay
  • US-TIGER-Roads-2017
  • US-TIGER-Roads-2018
  • US-TIGER-Roads-2019
  • WSDOT-Roads-2017

Any missing?

Last edited 3 months ago by Don-vip (previous) (diff)

comment:23 Changed 3 months ago by Don-vip

In 15855/josm:

see #18440 - Fetch imagery API keys from JOSM website. Restore access to Maxar imagery. Requires JMapViewer 2.13

comment:24 in reply to:  21 Changed 3 months ago by anonymous

Replying to stoecker:

Replying to SimonPoole:

@stoecker the GPL in general does not apply to configuration.

That's not configuration. GPL applies to sources and the tools/toolchain to build an executable from this. So there cannot be any secret in the build process. The code signing we do for JOSM is already a border case.

This not even the interpretation of the FSF.

comment:25 Changed 3 months ago by stoecker

The source code for a work means the preferred form of the work for making modifications to it. For an executable work, complete source code means all the source code for all modules it contains, plus any associated interface definition files, plus the scripts used to control compilation and installation of the executable. However, as a special exception, the source code distributed need not include anything that is normally distributed (in either source or binary form) with the major components (compiler, kernel, and so on) of the operating system on which the executable runs, unless that component itself accompanies the executable.

I don't care about any "interpretation", especially given by an anonymous commenter. It follow the GPL texts and these are very clear in this fact.

Version 0, edited 3 months ago by stoecker (next)

comment:26 Changed 3 months ago by Don-vip

In 15868/josm:

see #18440 - cache API keys in memory

comment:27 Changed 3 months ago by Don-vip

Keywords: api key added
Resolution: fixed
Status: assignedclosed

Done for Maxar. Follow-up in #18726

comment:28 Changed 3 months ago by Don-vip

In 15871/josm:

see #18440 - register API key provider in integration test

comment:29 Changed 3 months ago by stoecker

Could you change the pattern to lowercase {apikey}? that's not a Java variable name.

comment:30 in reply to:  29 Changed 3 months ago by Don-vip

Replying to stoecker:

Could you change the pattern to lowercase {apikey}? that's not a Java variable name.

Done in [o35329].

comment:31 Changed 3 months ago by Klumbumbus

There are at least two ways to display the connectId in JOSM in cleartext. Is that a problem?

comment:32 in reply to:  31 Changed 3 months ago by stoecker

Replying to Klumbumbus:

There are at least two ways to display the connectId in JOSM in cleartext. Is that a problem?

Hopefully not. Current solution is the best we can do in an OpenSource software.

comment:33 in reply to:  31 Changed 3 months ago by Don-vip

Replying to Klumbumbus:

There are at least two ways to display the connectId in JOSM in cleartext. Is that a problem?

No, I warned them about this and they're ok. They just don't want the key to be too easily scraped (using a search engine for example).

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain Don-vip.
as The resolution will be set.
The resolution will be deleted.

Add Comment


E-mail address and name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.