Modify

Opened 5 years ago

Closed 4 years ago

Last modified 21 months ago

#9210 closed enhancement (fixed)

Add missing google domains to blacklist

Reported by: pnorman Owned by: pnorman
Priority: normal Milestone:
Component: Core imagery Version: latest
Keywords: blacklist Cc: framm, stoecker, bastiK

Description (last modified by pnorman)

For reference, http://josm.openstreetmap.de/browser/josm/trunk/src/org/openstreetmap/josm/io/OsmApi.java#L242 is where the current list is.

We are not currently blocking google.ru, google.pl, and googleapis.com, all of which have seen use.

".*\\.google(apis)?\\.[^/]*/.*" might be a better regex than adding them all individually, but it'd need checking as I'm not 100% sure everything is escaped right.

I also get 173.194.33.0-14, 173.194.41.128-142, 74.125.228.0-14 as IP ranges

Attachments (0)

Change History (25)

comment:1 Changed 5 years ago by pnorman

Description: modified (diff)

comment:2 Changed 5 years ago by skyper

Keywords: blacklist added
Priority: normalmajor
Version: latest

comment:3 Changed 5 years ago by Don-vip

Owner: changed from team to pnorman
Status: newneedinfo

The proper solution should be to define imagery blacklist in OSM API capabilities (as described in r3934, see http://josm.openstreetmap.de/ticket/6037#comment:11).

Can you ask to API developers if this is planned ?

comment:4 Changed 5 years ago by pnorman

I don't believe anyone has planned adding a blacklist to /capabilities, nor do I think this was ever planned.

comment:5 Changed 5 years ago by Don-vip

Cc: framm stoecker bastiK added

@team: do you remember if it was really planned on server side ? If not we should ask for it :)

comment:6 Changed 5 years ago by Don-vip

I have created this ticket: https://trac.openstreetmap.org/ticket/5024

I hope it will be accepted...

comment:7 Changed 5 years ago by Don-vip

Good thing: the ticket has not be closed as wontfix in the 5 minutes of its creation like I often see on trac.osm.org when we request something.

Bad thing: no response at all, I don't know if it's the good way to ask for it. If it's something that you really want at DWG, I'm sure you know how to make it happen, the ball is in your court :)

comment:8 Changed 5 years ago by Don-vip

Priority: majornormal
Type: defectenhancement

Well, this doesn't look like so major.

comment:9 Changed 5 years ago by pnorman

I've raised the issue of if clients should be responsible for blocking imagery or if it should come from the server with the LWG, since there may be legal implications, but we've not met over Christmas.

comment:10 Changed 5 years ago by Don-vip

Ok thanks for update. I assume it will be discussed in next minute, here ? http://www.osmfoundation.org/wiki/Working_Group_Minutes#Licensing_Working_Group

comment:11 Changed 5 years ago by Don-vip

Is the LWG still alive?

comment:12 Changed 5 years ago by Don-vip

Hello, I finally see some activity from Feb 25th stating:

Getting JOSM’s blacklist update is important for the DWG, be it by modifying the API to return a blacklist or not. The question is, should this be delivered by the OpenStreetMap API? The positive is that it actively helps prevent well meaning map data and editor software contributors from contaminating the OSM database with data from non-allowed sources. The potential negative to this is that it may imply that other source not on the blacklist are OK. After discussion, it was felt that the right balance could be struck by additionally adding a whitelist, which may also be of use in its own right. Therefore:

LWG recommends and supports an API-delivered blacklist AND whitelist. We urge that it be made clear in supporting documentation that if a source is not on the blacklist, that does not imply that it OK and research should be done before adding it.

Potential whitelist https://github.com/osmlab/editor-imagery-index

The next meeting was scheduled on March 25th but I can't find the minutes?

comment:13 Changed 5 years ago by stoecker

A note: I won't support a whitelist. Especially not the mentioned one. But actually this is also nothing they should care about at all.

If they API supplies a reasonable blacklist we'll use it, but editor development is still our task and not that of a licensing working group.

Version 0, edited 5 years ago by stoecker (next)

comment:14 in reply to:  13 ; Changed 5 years ago by bastiK

Replying to stoecker:

A note: I won't support a whitelist. Especially not the mentioned one. But actually this is also nothing they should care about at all.

If the API supplies a reasonable blacklist we'll use it, but editor development is still our task and not that of a licensing working group.


It's a good thing if the licensing working group checks the usage terms of imagery providers and compiles a list of "officially" approved servers. It would give confidence to the users that the work put into tracing stuff is not wasted. Everything not on the whitelist or the blacklist is simply not yet evaluated by the LWG. So as I understand this suggestion, it would be an additional service and by no means a restriction.

comment:15 in reply to:  14 Changed 5 years ago by stoecker

So as I understand this suggestion, it would be an additional service and by no means a restriction.

I already told Frederik when he added the blacklist, that this in principle cannot work. It is a compromise to prevent the most obvious misuse. Otherwise we still need to trust our users. A white-list is too much. I'll never agree to it.

Think of something like HOT. Should they wait weeks or months until LWG approves their imagery? Would make the project totally useless.

I anyway want to introduce a verfied="date" tag to our maps list, so outdated entries get easier to spot. If LWG wants, they can also provide a list of verified URL's and we simply add an lwg_approved="date" to the entries and display that to the user. But not more. If they do so - will LWG be liable for wrong entries: Now we tell the user always "you are responsible"? A simple API list will not be enough in any case. It must be a more official document of the type "The LWG states:".

Last edited 5 years ago by stoecker (previous) (diff)

comment:16 Changed 4 years ago by anonymous

I'm not quite sure if the problem is just one of wording (aka using the term "whitelist"), as was already said, the idea is that it should be a list of imagery sources that have already had a number of eyeballs on it and where there is a certain level of confidence that the sources are ok for use in OSM. It was never intended to block custom imagery, and in fact it has been stated multiple times that naturally custom imagery should still be possible, just that you need to check the suitability for OSM use on your "own". Or to put it differently, it tries to answer the question for developers of a new editor, which imagery sources can I use without needing to check each one myself?

As to not implementing a positive list: de facto JOSM already does so from an user perspective, just as most of the other editors, the pre-configured imagery sources already have an implied suitability for use in OSM.

At this point in time both the black list and a potential positive list are simply an offer of support and have not been declared mandatory (has anything ever been declared mandatory wrt editors?). It would just seem that supporting a black list through the API would enable all to react faster to ongoing misuse as it happens.

comment:17 Changed 4 years ago by Don-vip

Any news on this subject from working groups side?

comment:18 Changed 4 years ago by pnorman

See https://trac.openstreetmap.org/ticket/5024.

I don't see any reason to hold off updating the in-JOSM list while waiting for new API functionality.

comment:19 Changed 4 years ago by bastiK

Blacklist is up:

$ wget -O- http://api.openstreetmap.org/api/capabilities
<?xml version="1.0" encoding="UTF-8"?>
<osm version="0.6" generator="OpenStreetMap server" copyright="OpenStreetMap and contributors" attribution="http://www.openstreetmap.org/copyright" license="http://opendatacommons.org/licenses/odbl/1-0/">
  <api>
    <version minimum="0.6" maximum="0.6"/>
    <area maximum="0.25"/>
    <tracepoints per_page="5000"/>
    <waynodes maximum="2000"/>
    <changesets maximum_elements="50000"/>
    <timeout seconds="300"/>
    <status database="online" api="online" gpx="online"/>
  </api>
  <policy>
    <imagery>
      <blacklist regex=".*\.googleapis\.com/.*"/>
      <blacklist regex=".*\.google\.com/.*"/>
      <blacklist regex=".*\.google\.ru/.*"/>
    </imagery>
  </policy>
</osm>

From our code (OsmApi.java) I get that the JOSM internal blacklist is now ignored:

            /* This is an interim solution for openstreetmap.org not currently
             * transmitting their imagery blacklist in the capabilities call.
             * remove this as soon as openstreetmap.org adds blacklists.
             * If you want to update this list, please ask for update of
             * http://trac.openstreetmap.org/ticket/5024
             * This list should not be maintained by each OSM editor (see #9210) */
            if (this.serverUrl.matches(".*openstreetmap.org/api.*") && capabilities.getImageryBlacklist().isEmpty()) {
                capabilities.put("blacklist", "regex", ".*\\.google\\.com/.*");
                capabilities.put("blacklist", "regex", ".*209\\.85\\.2\\d\\d.*");
                capabilities.put("blacklist", "regex", ".*209\\.85\\.1[3-9]\\d.*");
                capabilities.put("blacklist", "regex", ".*209\\.85\\.12[89].*");
            }

Who is responsible for maintaining the blacklist, so we can request to add the IP patterns?

comment:20 Changed 4 years ago by stoecker

Resolution: fixed
Status: needinfoclosed

comment:21 Changed 21 months ago by stoecker

In 11532/josm:

see #9210 - remove interim imagery blacklist solution, the api now has a proper system

comment:22 Changed 21 months ago by Don-vip

Milestone: 17.02

comment:23 Changed 21 months ago by stoecker

Hmm. Really? Claiming a ticket closed 2 years ago?

comment:24 Changed 21 months ago by Don-vip

well you cleaned up the code :)

comment:25 Changed 21 months ago by Don-vip

Milestone: 17.02

My bad didn't see it was already closed.

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain pnorman.
as The resolution will be set.
The resolution will be deleted.

Add Comment


E-mail address and name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.