New optional parameters to "import" remote control handler for tag replacement and OSM API download

[StefanB]

shp/dbf file format is rather old and lacks support for colons in field names, which makes it problematic for massive community imports where source:date and other source data (eg source_prefix:id must be specified in OSM tags. Having to search and replace manually for each tag during manual community import might be too demanding and error prone for comunity import.

According to wiki, the /import remote control command only has two parameters:
Parameter Required/Optional Meaning
url R
new_layer O If true, downloads to a new layer

It would help us greatly to support these optional parameters:
tagSearchAndReplace O Searches and replaces the tags when importing .shp files. Multiple parameters
loadAlsoFromApi O If true it also downloads the data from API on the bbox of the imported file (which is zoomed already by default). It can also add 10% (fixed or specified) margin of loaded data for better context

Example:
tagSearchAndReplace=ID=.*|source_prefix:id=.*&tagSearchAndReplace=DATE=.*|source:date=.*
or more correctly with url encoding:
tagSearchAndReplace=ID%3D.%2A%7Csource_prefix%3Aid%3D.%2A&tagSearchAndReplace=DATE%3D.%2A%7Csource%3Adate%3D.%2A

Note, we did strongly consider using ogr2osm, but there are still open problems, namely duplicated ways (​https://github.com/pnorman/ogr2osm/issues/28 , solved in JOSM as #10761 ) is a blocker and long ways (​https://github.com/pnorman/ogr2osm/issues/30 ) is an inconvenience. Importing shp into JOSM seems more feasable with this being the only obstacles.

We're open for other/modified solution proposals.

Thanks,
Stefan

[StefanB]

For the immediate problem of colons the workaround was to do replacements in the shapefile's dbf file itself using bbe (binary block editor) utility, see ​http://manpages.ubuntu.com/manpages/utopic/en/man1/bbe.1.html

bbe -e "s/RABA_ID/raba:id/" x-orig.dbf -o x.dbf

To allow a bit longer tag names (shapefile and gis tools limit this to 10 characters, but dbf allows up to 11) you can use this hack:

bbe -e "s/SOURCEDATE\x00/source:date/" x-orig.dbf -o x.dbf

For good measure both can be done at the same time:

bbe -e "s/RABA_ID/raba:id/" -e "s/SOURCEDATE\x00/source:date/" x-orig.dbf -o x.dbf

Longer tags and other special characters in tag names would still need some change in JOSM.

[StefanB]

Interestingly, there are 100k+ instances of wrong source_date tag (​http://taginfo.openstreetmap.org/keys/source_date ) in the database.

[StefanB]

It would be also very useful to specify source tag for the next changeset (or more specifically, the one that would be uploading the imported data).
eg ...&source=City+of+X+GIS+office

Until community imports rely on importers to manually add the source tag to changeset we have to prepare the data in such way, that it will include the source tag within the imported data.