Modify

Opened 4 years ago

Last modified 4 years ago

#11045 new enhancement

New optional parameters to "import" remote control handler for tag replacement and OSM API download

Reported by: StefanB Owned by: team
Priority: normal Milestone:
Component: Core remotecontrol Version:
Keywords: import tag replacement download Cc:

Description

shp/dbf file format is rather old and lacks support for colons in field names, which makes it problematic for massive community imports where source:date and other source data (eg source_prefix:id must be specified in OSM tags. Having to search and replace manually for each tag during manual community import might be too demanding and error prone for comunity import.

According to wiki, the /import remote control command only has two parameters:
Parameter Required/Optional Meaning
url R
new_layer O If true, downloads to a new layer

It would help us greatly to support these optional parameters:
tagSearchAndReplace O Searches and replaces the tags when importing .shp files. Multiple parameters
loadAlsoFromApi O If true it also downloads the data from API on the bbox of the imported file (which is zoomed already by default). It can also add 10% (fixed or specified) margin of loaded data for better context

Example:
tagSearchAndReplace=ID=.*|source_prefix:id=.*&tagSearchAndReplace=DATE=.*|source:date=.*
or more correctly with url encoding:
tagSearchAndReplace=ID%3D.%2A%7Csource_prefix%3Aid%3D.%2A&tagSearchAndReplace=DATE%3D.%2A%7Csource%3Adate%3D.%2A

Note, we did strongly consider using ogr2osm, but there are still open problems, namely duplicated ways (https://github.com/pnorman/ogr2osm/issues/28 , solved in JOSM as #10761 ) is a blocker and long ways (https://github.com/pnorman/ogr2osm/issues/30 ) is an inconvenience. Importing shp into JOSM seems more feasable with this being the only obstacles.

We're open for other/modified solution proposals.

Thanks,
Stefan

Attachments (0)

Change History (5)

comment:1 Changed 4 years ago by StefanB

Summary: Additional import parametersSupport colons in tags during SHP file import
Type: defectenhancement

comment:2 Changed 4 years ago by Don-vip

Component: Plugin opendataCore remotecontrol
Keywords: import tag replacement download added
Owner: changed from Don-vip to team
Summary: Support colons in tags during SHP file importNew optional parameters to "import" remote control handler for tag replacement and OSM API download

comment:3 Changed 4 years ago by StefanB

For the immediate problem of colons the workaround was to do replacements in the shapefile's dbf file itself using bbe (binary block editor) utility, see http://manpages.ubuntu.com/manpages/utopic/en/man1/bbe.1.html

bbe -e "s/RABA_ID/raba:id/" x-orig.dbf -o x.dbf

To allow a bit longer tag names (shapefile and gis tools limit this to 10 characters, but dbf allows up to 11) you can use this hack:

bbe -e "s/SOURCEDATE\x00/source:date/" x-orig.dbf -o x.dbf

For good measure both can be done at the same time:

bbe -e "s/RABA_ID/raba:id/" -e "s/SOURCEDATE\x00/source:date/" x-orig.dbf -o x.dbf

Longer tags and other special characters in tag names would still need some change in JOSM.

Last edited 4 years ago by StefanB (previous) (diff)

comment:4 Changed 4 years ago by StefanB

Interestingly, there are 100k+ instances of wrong source_date tag (http://taginfo.openstreetmap.org/keys/source_date ) in the database.

comment:5 Changed 4 years ago by StefanB

It would be also very useful to specify source tag for the next changeset (or more specifically, the one that would be uploading the imported data).
eg ...&source=City+of+X+GIS+office

Until community imports rely on importers to manually add the source tag to changeset we have to prepare the data in such way, that it will include the source tag within the imported data.

Modify Ticket

Change Properties
Set your email in Preferences
Action
as new The owner will remain team.
as The resolution will be set.
to The owner will be changed from team to the specified user.
The owner will change to StefanB
as duplicate The resolution will be set to duplicate.The specified ticket will be cross-referenced with this ticket
The owner will be changed from team to anonymous.

Add Comment


E-mail address and name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.