Improve the usability of importing a CSV file

[flaimo]

I tried importing data from a simple CSV file and I think that, at the moment, this use case is very unintuitive and not flexible enough.

a) First of all I think that – regardless of other file formats – the CSV import functionality should be implemented in the core, not just as a plugin. Nevertheless, if you go to the plugins tab in the settings and search for CSV you won't find the plugin (opendata) where it is currently implemented. That is bad. So either extract the CSV functionality and put it in the core or in a separate import plugin which has nothing to do with opendata portals (why are those two things mangled together anyway) and give it a fitting description.

b) There is basically no interaction with the user during the import procedure. If the data doesn't fit, you just get an error message and have to tingle with the CSV file yourself. The usability of this use case could be vastly improved if it was implemented like the import wizard of Microsoft Excel, which could look something like this:

1) User clicks File→Open and selects a CSV file.
2) The first dialog pops up. The user selects the file encoding and the column separator character and if the data is in quotation marks or not. The user can also define if the first line are the column headers or not. The wizard tries to preparse the file and suggest the right values in this dialog.
3) The User clicks "Next" and the next dialog shows up. Here the user can define, if there is one column that holds the list of coordinates that describes a closed way, or the two columns that represent latitude and longitude. Also the projection can be selected. Again the wizard tries to preparse the file and suggest the right values in this dialog.
4) The User clicks "Next" and the next dialog shows up. Here the user can map the remaining individual columns to OSM keys using text input fields. Again the wizard tries to prefill the text fields with the most fitting OSM key based on the column header (if available).
5) The User clicks "Next". If there is no data layer yet, a new one is created an the data is imported there. If there is a data layer open already, the wizard asks if the data should be added to the current layer or imported into a new layer.

Other thinks to consider:
c) How large can the CSV file be. Can JOSM handle the import of 100000 lines and above or does a maximum number of lines limit need to be implemented to ensure the stability of the running program?
d) In case the CSV file contains a column with an external ID there could be an addon functionality which tries to synchronise the objects of an open data layer with the ones of the CSV file based on that ID. The user could mark the ID column during the import using a simple checkbox and afterwards JOSM updates all existing objects with those IDs it can find in the data layer. Of course there would be the need to handle conflicts on a case per case basis but also in batches (aka “Skip all existing objects”, “Only add new tags”, “Replace data of all existing objects”, and so on.). The same sould be implemented if the CSV file contains a column with the OSM database ID of an object as long as the object type (node or way) matches the existing one.

Real life example: My home town provides a list of all managed trees as a CSV file (german "Baumkataster") which gets updated regularly. This way updating the data without having to have database knowledge would be a breeze.

Please provide any additional information below. Attach a screenshot if possible.

Revision: 8491
Repository Root: http://josm.openstreetmap.de/svn
Relative URL: ^/trunk
Last Changed Author: Don-vip
Last Changed Date: 2015-06-16 23:27:08 +0200 (Tue, 16 Jun 2015)
Build-Date: 2015-06-16 21:45:58
URL: http://josm.openstreetmap.de/svn/trunk
Repository UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
Last Changed Rev: 8491

Identification: JOSM/1.5 (8491 en) Mac OS X 10.10.4
Memory Usage: 409 MB / 910 MB (125 MB allocated, but free)
Java version: 1.8.0_45, Oracle Corporation, Java HotSpot(TM) 64-Bit Server VM
VM arguments: [-Djava.library.path=/Applications/JOSM.app/Contents/MacOS, -DLibraryDirectory=/Users/flaimo/Library, -DDocumentsDirectory=/Users/flaimo/Documents, -DApplicationSupportDirectory=/Users/flaimo/Library/Application Support, -DCachesDirectory=/Users/flaimo/Library/Caches, -DSandboxEnabled=false, -Dapple.laf.useScreenMenuBar=true, -Dcom.apple.macos.use-file-dialog-packages=true, -Dcom.apple.macos.useScreenMenuBar=true, -Dcom.apple.mrj.application.apple.menu.about.name=JOSM, -Dcom.apple.smallTabs=true]

Plugins:
- ColorPlugin (1414145445)
- HouseNumberTaggingTool (31241)
- OpeningHoursEditor (31241)
- PicLayer (31241)
- RoadSigns (31241)
- ShapeTools (1000)
- buildings_tools (31241)
- contourmerge (1013)
- geotools (31126)
- jts (31126)
- kendzi3d-jogl (37)
- kendzi3d_Improved_by_Andrei (1.0.179-SNAPSHOT)
- log4j (31231)
- measurement (31241)
- opendata (31241)
- reverter (31241)
- terracer (31241)
- turnlanes (31241)
- turnrestrictions (31241)
- undelete (31241)
- utilsplugin2 (31241)
- wikipedia (31241)

Last errors/warnings:
- E: Failed to locate image ''
- E: Failed to locate image ''
- E: Failed to locate image ''
- W: No valid coordinates have been found.
- E: java.lang.IllegalArgumentException: No valid coordinates have been found.

[simon04]

I think it's related to the opendata plugin.

[Don-vip]

Replying to flaimo:

a) First of all I think that – regardless of other file formats – the CSV import functionality should be implemented in the core, not just as a plugin. Nevertheless, if you go to the plugins tab in the settings and search for CSV you won't find the plugin (opendata) where it is currently implemented. That is bad. So either extract the CSV functionality and put it in the core or in a separate import plugin which has nothing to do with opendata portals (why are those two things mangled together anyway) and give it a fitting description.

I disagree. The main use case of opening CSV files is to open data published as open data by governements or municipalities. The opendata plugin handles a large amount of formats, csv is only one of them.

b) There is basically no interaction with the user during the import procedure. If the data doesn't fit, you just get an error message and have to tingle with the CSV file yourself.

It's supposed to work out of the box. You can simply create a ticket so that I can improve the column detection.

The usability of this use case could be vastly improved if it was implemented like the import wizard of Microsoft Excel, which could look something like this:

1) User clicks File→Open and selects a CSV file.
2) The first dialog pops up. The user selects the file encoding and the column separator character and if the data is in quotation marks or not. The user can also define if the first line are the column headers or not. The wizard tries to preparse the file and suggest the right values in this dialog.
3) The User clicks "Next" and the next dialog shows up. Here the user can define, if there is one column that holds the list of coordinates that describes a closed way, or the two columns that represent latitude and longitude. Also the projection can be selected. Again the wizard tries to preparse the file and suggest the right values in this dialog.
4) The User clicks "Next" and the next dialog shows up. Here the user can map the remaining individual columns to OSM keys using text input fields. Again the wizard tries to prefill the text fields with the most fitting OSM key based on the column header (if available).
5) The User clicks "Next". If there is no data layer yet, a new one is created an the data is imported there. If there is a data layer open already, the wizard asks if the data should be added to the current layer or imported into a new layer.

I am only one guy, I don't have the same manpower as Microsoft. What you are asking is nice, but requires a huge effort I'm not willing to spend. Most people are happy with the current csv support or just create small tickets to request a small adjustment of the column auto-detection.

Real life example: My home town provides a list of all managed trees as a CSV file (german "Baumkataster") which gets updated regularly.

So it's open data. You see?

This way updating the data without having to have database knowledge would be a breeze.

You don't have to. Just create a ticket with the file that can't be opened and I'll find a way to make the plugin smarter.