Paste tags from text

[overqu@…]

Need a way to paste to all selected objects tags from system clipboard, where tags are in plain text.
Current "Paste Tags" paste only tags copied from another object in JOSM and ignores system clipboard.

Applications:
1) Manual of half-automatic adjustment of object's tags
2) Selective copying tags from different sources, like web-based validators

In case of collisions by tags, previous tags of selected objects should be replaced (like "Paste Tags" do).
Accepted format(s) is not important - it could be tag=value (result of Copy all Keys/Values”) or "tag":"value" or OSM-XML or JSOM or else.
Preferred is to catch Unicode/UTF8 if it is present in system clipboard and to convert ASCII by local codepage if only ASCII is present.

[verdy_p]

For internal copy-pasting of multiple tags between objects within JOSM itself, JOSM support an internal prefered clipboard format that sill preserve the internal parsing of tag pairs as a Java PropertiesList (possibly a specialized subclass stating that this is effectively containing a lsit of tags).

This would cimplify a lot things when we *paste* things into another object : if the clipboard contains such a properties list or specialized Tags subclass it should use this format by default.

Otherwise JOSM would parse a plain-text format to see if it uses the "key:value" format (line-separated), or the "key=value" format (line-separated), or a CSV format (line-separated), or some HTML two-column table format or an XML format (both separated as document elements, using some HTML or XML parser). In case of doubt, this would behave a bit like what Excel proposes when pasting plain-text : it allows several parsers, if it does not autodetect the encoded format.

Supporting only an unparsed plain-text format when pasting into JOSM tags is really not convenient and unnecessarily slow : it requires complex manual editing of the pasted string, and manually splitting it into multiple fields. This causes various unexpected errors or ommisions in the resulting OSM document.

[akks]

I think this pasting should be additional action for now (not Ctrl-Shift-V or Ctrl-V), because you can not clean JOSM buffer after Ctrl-C (so no tags from any other source can be inserted).

I propose to add it into Utilplugin2 and insert it into core after adding all functionality and complete debugging.

[akks]

It is simple enough to implement but the format must chosen wisely - we need multiplpe tags with possible special-symbols in them.

How to separate one key-value from another?

key1=value1; key2=value2; ... => problem with ';' (consider first = is always separator)
key1=value1 key2=value2 ... => problem with spaces (consider first = is always separator for cases like name=x=y)
"key1"="value1","key2"="value2" ... (JSON-like) => problem with quotes
"key1"="hotel \"California\"","description"="......" ... (JSON-like) => is the format OK?

[anonymous]

All the plain-text formats share the same problem as found in CSV files or formats used for exporting from (or importing to) any database or spreadsheet document.
You may use varius separators you'll still need some escaping syntax. Even if you use quotes to surround keys or values, there will remain the need to also escape the quotes.
The same is also treu when you use some HTML or XML syntax.

But this is not true if the native (and prefered) clipboard format (used internally) is a properties list *object*. The internal Java clipboard implements a server that can keep a single internal format as an object, which will then enumerate the export formats it supports to the target clients trying to get data from the clipboard in order for them to perform the paste operation. Internally the tags editors in JOSM can also select the internal JOSM format as the prefered format as it requires absolutely no parsing: it knows that the clpboard format is in a known type other than just the basic "String" type, and then they can perform the best conversions without any loss.
Pasring from JOSM to another applicatin will be able to use the String type and will perceive it as plain text, only because the internal type will be convertible by JOSM as a String (this will be what JOSM will return to clients requesting this type. But JOSM could also reply to other clietns by performing conversions of its internal format according to the selection of format requested by the clipboard client.

My opinion is that the best text format exported by JOSM is a tabulated text (e.g. "key1\tvalue1\nkey2\tvalue2\n")
simply because we (hopefully) don't accept TABS and NEWLINEs in the OSM database for tag keys or tag values; it requires no special escaping or quoting. The newlines could be encodes as CR or LF or CR+LF

When pasting to JOSM (i.e. when JOSM is the clipboard client), and if the internal native format or the alternate prefered text format is not supported by the clipboard server, but only a plain-text format not obeying to the 2-column tabular format (using TAB+Newlines), then only JOSM could open a dialog to allow specifying how the file is encoded (most often it will be in CSV form; but the user may also need to map the columns and select which ne to use as the key, and which one to use as the value; and if keys or values are quoted, which quotation marks are used, and if they are escaped, which characters is used for escaping these quotes). There may be fields to ignore, or may be to parse lines one could provide a regexp (associated to a registrable text format) allowing basic parsing even of HTML or XML encoded data. This should not require a complex text parser, but if needed we could also have in the dialog the option to specify an XML or HTML parser, and a schema for XML, from which we will select keys and values.

Such import/export mechanism could be generic enough to become a full component of JOSM, performing conversions with known geographic formats (OSM XML, KML, shape files, OpenGIS XML...) not just for the clipboard but also from files or from webservices, or from a JDBC SQL database connector.

[Don-vip]

Replying to akks:

I think this pasting should be additional action for now (not Ctrl-Shift-V or Ctrl-V), because you can not clean JOSM buffer after Ctrl-C (so no tags from any other source can be inserted).

I think we can use the same action. If we remember the time at which a primitive has been copied in JOSM and when the system clipboard has been changed (could be done with ​FlavorListener), we can determine what source should be considered (the most recently modified).

[overqu@…]

Replying to anonymous:

My opinion is that the best text format exported by JOSM is a tabulated text (e.g. "key1\tvalue1\nkey2\tvalue2\n")
simply because we (hopefully) don't accept TABS and NEWLINEs in the OSM database for tag keys or tag values;

Good idea.
As far as I see in wiki, Tab and Newline are not forbidden in keys and values. But JOSM diplays a warning before uploading objects with such chars. So, I think we can just ignore smallest usecase for pasting keys/values with Tabs/Newlines from plain text.

[akks]

Conversion from different formats is currently done by FileExporter/FileImporter, JOSM also supports file drag and drop. I do not see the reason to duplicate this mechanism (anonymous). But maybe we could invoke that readers when bunch of files are copied and Ctrl-V is pressed (I can not do it myself, however).

Pasting tags from different sources is an easier task. I suggest it remains on Ctrl-Shift-W, pasting tags from JOSM or from external clipboard (the most recently modified), as Don-vip suggests.

Tab and Newline - separated text seems like the best format and should be implemented, but I think we should support some others formats too without asking user each time. I propose to support also "free format"
key1=very long key key2=short key ; key3=value3
(first word before = is the key, all separators between it and previous word are ignored) and "strict JSON-version" for web-based tools
{"key1"="value number 1", "key2"="v2", "key3"="road \"M5\""}

I think we can start from plugin action (considering stabilization phase) and then integrate it with Ctrl-Shift-V in core before next (march) tested is released.

[Don-vip]

Replying to akks:

Pasting tags from different sources is an easier task. I suggest it remains on Ctrl-Shift-W, pasting tags from JOSM or from external clipboard (the most recently modified), as Don-vip suggests.

Not so easy in fact. My idea of FlavorListener does not work: events are not triggered when new text is copied from the same application. Looks like there are two ways of getting notified of clipboard changes: a dedicated thread that reads clipboard periodically, or a systematic read when JOSM gains focus (as we can assume JOSM will always have lost the focus in that case).

[akks]

Replying to Don-vip:

Replying to akks:
Not so easy in fact. My idea of FlavorListener does not work: events are not triggered when new text is copied from the same application. Looks like there are two ways of getting notified of clipboard changes: a dedicated thread that reads clipboard periodically, or a systematic read when JOSM gains focus (as we can assume JOSM will always have lost the focus in that case).

While experimenting I accidentally noticed that Utils.getClipboardContent() not only returns the last copied string but also becomes null when some JOSM objects is copied after text (try inserting System.out.println(Utils.getClipboardContent()); to check).

It seems that is what we needed.

[akks]

I have added the tag-parsing functionality to Utilsplugin2 for testing (Ctrl-T), see [o29226]

It supports both "key1\tvalue1\nkey2\tvalue2\n" format and "free format".

In "free format" I had some problems with detecting "," delimiters, so not JSON-like text is consumed but strings like

• a=1 b=2 c=3
• "name" ="road M5" highway = "trunk"
• "name" = "the quotes are \"here\""
• a=b=3 c = " hello world" (means "a"="b=3" "c"="hello world")

or any of their combinations.

Please check the behavior and say if something needs to be changed.

[overqu@…]

Replying to akks:

I have added the tag-parsing functionality to Utilsplugin2 for testing (Ctrl-T), see [o29226]
Please check the behavior and say if something needs to be changed.

Looks very good! I will use it.

Found 1 point - if last or only line is

• a = b c

then nothing happens at all.

Absolutely not critical, but is it possible to 1) parse remaining tags 2) consider lines with one = as key = value ?

[akks]

Replying to overqu@…:

Looks very good! I will use it.

Found 1 point - if last or only line is

• a = b c

then nothing happens at all.

Absolutely not critical, but is it possible to 1) parse remaining tags 2) consider lines with one = as key = value ?

I forgot to mention,
a 2 b 3 c
4 g hello
will result to a=2,b=3,c=4,g=hello, spaces can be replaced with any number of \t \n (and not more than one = can be inserted between 2-b,3-c,4-g)
this is a very big tag, this is! gives this=is! a=very big=tag,

For a = b c it founds tag a, sign = and tag b without value, so it considers input as incorrect.
or a = b c 17 or a = b\n\n\t c\n\n\n\t 17 will give a=b,c=17.

If we need to convert a = b c to "a"="b c" at the end of each line, line breaks will mean too much, I think.
(if a = b c is the last fragment, I can make it work as "a"="b c", of course, but the format will become too free...)

Maybe the best solution would be to support extra "strict" formats:
tag number 1 = value number 1 \n
tag number 2 = value number 2 \n
without processing " and \
and JSON
{ "a" : "1", "b" : "2 the value with \"quotes" " }

Free format will work only if no strict format succeed.

[overqu@…]

Replying to akks:

Maybe the best solution would be to support extra "strict" formats:
tag number 1 = value number 1 \n
tag number 2 = value number 2 \n
without processing " and \
and JSON
{ "a" : "1", "b" : "2 the value with \"quotes" " }

Free format will work only if no strict format succeed.

I would prefer this way.

[rickmastfan67]

In [o29226], there was an update to utilsplugin2 plugin for this. Unfortunately, it has introduced two brand new Registered toolbar action problems on startup of JOSM.

Registered toolbar action pastetags overwritten: org.openstreetmap.josm.actions.PasteTagsAction gets org.openstreetmap.josm.plugins.utilsplugin2.actions.PasteTagsExtendedAction
Toolbar action pastetags overwritten: org.openstreetmap.josm.actions.PasteTagsAction gets org.openstreetmap.josm.plugins.utilsplugin2.actions.PasteTagsExtendedAction

Might want to fix this.

Repository Root: http://josm.openstreetmap.de/svn
Build-Date: 2013-02-04 02:31:36
Last Changed Author: simon04
Revision: 5693
Repository UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
URL: http://josm.openstreetmap.de/svn/trunk
Last Changed Date: 2013-02-03 11:48:30 +0100 (Sun, 03 Feb 2013)
Last Changed Rev: 5693

Identification: JOSM/1.5 (5693 en)
Memory Usage: 154 MB / 2730 MB (82 MB allocated, but free)
Java version: 1.7.0_13, Oracle Corporation, Java HotSpot(TM) 64-Bit Server VM
Operating system: Windows 7

Plugin: OpeningHoursEditor (29210)
Plugin: buildings_tools (29210)
Plugin: mapdust (29210)
Plugin: measurement (29210)
Plugin: mirrored_download (29210)
Plugin: openstreetbugs (29210)
Plugin: reverter (29216)
Plugin: turnrestrictions (29210)
Plugin: undelete (29210)
Plugin: utilsplugin2 (29226)

[akks]

OK. I'm preparing next version...

[akks]

There is a problem:
"power" = "line" should be still treated as power=line
but
name = "Strange" memorial should be treated as name="Strange" memorial
more regexping in needed ))
and I wonder what to do with name = "Strange". Free format -> name=strange, A=B format - the same?

[overqu@…]

Replying to akks:

name = "Strange" memorial should be treated as name="Strange" memorial
and I wonder what to do with name = "Strange". Free format -> name=strange, A=B format - the same?

IMHO, best case:
1) If line contains only one = then it divides key and value
2) If line contains more than one = then line should be checked for quoted key/value pair with escaping and (if fails) for free format
3) If key or value is completely in ", then it is formating quotes and raw value should be without them
So
1) name="Strange" memorial -> name\t"Strange" memorial
2) name = "Strange" -> name\tStrange

Good case:
1) If line contains " then this line should be either one "key"="value" pair with proper escaping (allowed only to omit quotes around key) or free format
So
1) name="Strange" memorial ->(discard memorial as error)-> name\tStrange
2) name = "Strange" -> name\tStrange

[akks]

@ overqu@.. : Thank you very much for idea and detailed descriptions!

The next version is ready, [o29234]
Added tags pasting from JOSM, now it can be used as replacement of Ctrl-Shift-V - please check it
Improved "a=b" format support, added JSON.

Examples are in attached text file:

[akks]

example of text to copy/paste

[overqu@…]

Replying to akks:

The next version is ready, [o29234]
Added tags pasting from JOSM, now it can be used as replacement of Ctrl-Shift-V - please check it
Improved "a=b" format support, added JSON.

Works very good for now. Ctrl-Shift-V also works.

One thing - in JOSMs 5697 and 5608 menu items "Paste tags [testing]" locked, but keyboard shortcut still works.

[akks]

Replying to overqu@…:

One thing - in JOSMs 5697 and 5608 menu items "Paste tags [testing]" locked, but keyboard shortcut still works.

Thank you, fixed in [o29239]:)

@team: please let me know when it will be ready for core integration
JOSM core copies id1,id2,id3 (sorry for mistake, not null) to text buffer when copying primitives, current code detects it and invoke old tag pasting. If other text is in buffer, it will be inserted by the parser.

Combinations of tags for arbitrary text in buffer may be 1) huge 2) absolutely meaningless. Do we need some warnings? How to detect this "bad" combinations? (number of tags, length of tags, length of values?)

[overqu@…]

@team: please let me know when it will be ready for core integration

Well, I have not yet tested a lot in real-life tasks, so can't vote for core integration. I plan to do extensive testing in a week or so.

Combinations of tags for arbitrary text in buffer may be 1) huge 2) absolutely meaningless. Do we need some warnings?
How to detect this "bad" combinations? (number of tags, length of tags, length of values?)

I have just tested huge ini-like text, about 100k unicode chars, resulting 700 tags on object (a lot of duplicates collapsed), it was rather fast (<1 second for 1 object) and does not affect performance of JOSM with resulting object. For many objects it gets more time. Meaningless data, IMHO, is not a problem, if user accidentally paste garbage he will undo it quickly.
Huge - could be an issue if a lot of objects are selected. So, I would suggest to ask user "You are about to paste N tags to K objects, do you really want to proceed?" if
1) more than 200 tags found in clipboard (do we have objects with 200+ tags? name:*= in every language?)
2) if N*K > 100000
Maybe we could also ask user if
1) Total length of data in clipboard more than 100k chars (this could be checked before parsing)
2) Length of detected keys/values more than 255 chars (API 0.6 limit)

[akks]

I have added basic validation.
For now, rules are rather strict:

• there are <=30 tags in the buffer
• key contain only characters a-zA-Z:_ , not more than 50 of them
• values has <255 characters

However, user can ignore the warnings and paste tags. Tags with empty symbols at the begging or end are trimmed automatically.

[akks]

What if I commit "tags pasting from text" into core, extending Ctrl-Shift-V and disable this functionality by default for now (add checkbox to preferences)?

[Don-vip]

If there is no visible regression (i.e pasting tags from OSM object still works as expected), I think it's ok to put it enabled without any preference.

[akks]

Pasting tags from OSM object still works as before (I hope at least), but the action should become always enabled (when there are selected objects).

[Don-vip]

Even if the clipboard does not contain anything ?

[akks]

Maybe. Or is there a good and fast way to detect all clipboard changes? (You have tried FlavorListener, you said)
We can just give the message "Sorry, the buffer is empty. Please copy some JOSM objects or text containing tags (see ...)", it can be faster (lower CPU usage) and more obvious for user.

[Don-vip]

This kind of popup sounds good to me. Maybe we can however use ConditionalOptionPaneUtil to allow user to disable it when he has acknowledged it once :)

[akks]

Replying to Don-vip:

This kind of popup sounds good to me. Maybe we can however use ConditionalOptionPaneUtil to allow user to disable it when he has acknowledged it once :)

Oh, yes, it is a good idea! I am adding it. Warnings are already skippable.
Do you want to check the patch before committing?

[Don-vip]

with pleasure :)

[akks]

So here it is. Fell free to modify or commit it :)

[Don-vip]

A minor typo error "ffrom".
I will review the rest of the patch tonight :)

[akks]

OK. There is also some debugging output :-)

[Don-vip]

Sorry, #8444 took me longer than expected, no more time tonight. Tomorrow, promised ! :)

[akks]

Ok, no problem.

[Don-vip]

new patch

[Don-vip]

Ok nice :) No problem seen, I have just updated some typo/javadoc; cleanup some code and removed debug info. You can commit when you want :) Nice job !

[akks]

Thank you for checking and help!
I have added advanced parameters , now committing. Later we'll need to fix the help page.

[akks]

In 5738/josm:

see #8384: Ctrl-Shift-V now can paste tags from text buffer like "a=2,c=3", "a 2 c 3", etc.

[akks]

Tag regexp-pattern is customizable now, digits are now allowed (4wd is ok).

[simon04]

Comparing those two lines

• ​http://josm.openstreetmap.de/browser/josm/trunk/src/org/openstreetmap/josm/tools/TextTagParser.java?rev=5738#L192
• ​http://josm.openstreetmap.de/browser/josm/trunk/src/org/openstreetmap/josm/tools/TextTagParser.java?rev=5738#L201

in one case \\n and in the other case \n is used in the regexp. Is that intended?

[akks]

Replying to simon04:

Comparing those two lines

• ​http://josm.openstreetmap.de/browser/josm/trunk/src/org/openstreetmap/josm/tools/TextTagParser.java?rev=5738#L192
• ​http://josm.openstreetmap.de/browser/josm/trunk/src/org/openstreetmap/josm/tools/TextTagParser.java?rev=5738#L201

in one case \\n and in the other case \n is used in the regexp. Is that intended?

Thank you for noticing. First expression is wrong of course (extra ], no \), first method is not working. I'm patching it..

[akks]

In 5755/josm:

Fix incorrect tag pasting from tab-delimited text, see #8384

[akks]

Maybe some unit testing is needed (after tested realease) :)

[Don-vip]

You can add unit tests before we relase, there is no problem with that :D

[akks]

In 5756/josm:

paste tags from text: fixed one more parser error, added JUnit tests, see #8384
NetBeans project: experimental testing support

[akks]

I have added the tests. One more serious bug in parsing ({ "a":"1", "b":"2 3 4" } was a=1, b=2) was fixed by the way, so the tests proved to be useful :)

I noticed that

• NetBeans project had no testing enabled - tried to add, but could not get rid of IDE false error highlighting.
• ant test gives an error on Windows (test path is bad, ${test.dir}/${build.dir} is duplicated absolute path)
• Many tests involving projections fail.

[akks]

In 5809/josm:

fix #8500, see #8384: added "Clear buffer" button to the messages shown for incorrect buffer (clipboard)
(next Ctrl-Shift-V will paste tags from JOSM-copied objects, if available)

[stoecker]

Please no stub javadocs - fill out the texts with something useful.

[Dakon]

Currently JOSM isn't able to paste it's own copied tags. Marking tags and using Ctrl-C gives me clipboard content like this:

amenity {school=1}

When I insert this using Ctrl-Shift-V the tag created is indeed amenity={school=1}, not amenity=school.

[akks]

Replying to Dakon:

Currently JOSM isn't able to paste it's own copied tags. Marking tags and using Ctrl-C gives me clipboard content like this:
amenity {school=1}
When I insert this using Ctrl-Shift-V the tag created is indeed amenity={school=1}, not amenity=school.

This is also described in #8508.
Please, let me know exactly how to reproduce this bug, because I can not do it (did once, but can not repeat)
(download smth. ofr open OSM, select smth., select tags, ctrl-c)

Please attach .osm to the ticket #8508 and specify your OS/Java version.

/ Edited: outdated

[akks]

Oh, I managed to reproduce it. Your mentioned pressing Ctrl-C - this is the key. Maybe some default Swing behaviour is invoked. I am fixing it.

[Dakon]

The trick seems to be that you need to have keys from a node being selected, i.e. after selecting node(s) you need to select them in the properties view.

I'm using JOSM 5836 on openSUSE 12.2.

java version "1.7.0_17"
OpenJDK Runtime Environment (IcedTea7 2.3.8) (suse-3.33.1-x86_64)
OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)

[akks]

Thank you again. It seems that it is standard JTable behaviour, which must be redefined.

[akks]

Fixed in [5843/josm].

[akks]

Can this be closed?

[overqu@…]

Yes, requested function is implemented.

[Kolesár András <kolesar@…>]

Paste tags fails when a value contains equal sign and another value contains a space, for example:

name=Main street
url=http://foo.bar/?id=1

Result after pasting text above (stable version #8969):

name=Main
street=url=http://foo.bar/?id=1

Workaround until fix: quoting the values. It's enough to quote the value with equal sign:

name=Main street
url="http://foo.bar/?id=1"

[simon04]

With the current implementation, this behaviour is intended: from org.openstreetmap.josm.tools.TextTagParser#unescape: '=' not in quotes will be treated as an error! …

[anonymous]

In this case copying tags to clipboard should escape values containing '=' sign.

[simon04]

In 15750/josm:

see #8384 - TextTagParser: add test case

[simon04]

Behaviour from comment:58 can no longer be reproduced.