OSM objects parameter via stdin are not XML since 2 XML documents

[jjaf.de]

Problem

While looking for a automatisation for WIWOSM-project i came to CommandLine.
When selecting an OSM object to be passed as parameter via stdin to a script you get two XML documents instead of one, e.g.:

<?xml version='1.0' encoding='UTF-8'?>
<osm version='0.6' generator='JOSM'>
</osm>
<?xml version='1.0' encoding='UTF-8'?>
<osm version='0.6' generator='JOSM'>
  <relation id='253676' timestamp='2009-09-18T01:18:49Z' uid='120146' user='TIGERcnl' visible='true' version='1' changeset='2518319'>
[...]
</osm>

This of course does not parse.

Cause

​http://svn.openstreetmap.org/applications/editors/josm/plugins/CommandLine/src/CommandLine/CommandLine.java Line 520 has unconditional

osmWriter.footer();

for ending referenced objects before conditional

osmWriter.header();
[...]
osmWriter.footer();

for parameters.

Resolution

Leave out the conditional statements and put unconditional footer at the end of parameters processing.

[Don-vip]

This should do the trick (needs testing however before committing)

[Don-vip]

Mmm patch is quite unreadable because of automatic indentation. Here's the new piece of code:

                osmWriter.header();
                DataSet ds = new DataSet();
                for (OsmPrimitive primitive : refObjects) {
                    ds.addPrimitive(primitive);
                    if (bbox == null)
                        bbox = new BBox(primitive.getBBox());
                    else
                        bbox.addPrimitive(primitive, 0.0);
                }
                for (Parameter parameter : parameters) {
                    if (!parameter.isOsm())
                        continue;
                    pObjects = parameter.getParameterObjects();
                    for (OsmPrimitive primitive : pObjects) {
                        ds.addPrimitive(primitive);
                        if (bbox == null)
                            bbox = new BBox(primitive.getBBox());
                        else
                            bbox.addPrimitive(primitive, 0.0);
                    }
                }
                osmWriter.writeContent(ds);
                osmWriter.footer();
                osmWriter.flush();

[Don-vip]

Can you see if it works ? (by replacing the plugin jar). Thanks :)

[jjaf.de]

Sorry when selecting one or more OSM objects to be passed as parameter it throws:

Repository Root: http://josm.openstreetmap.de/svn
Build-Date: 2013-02-06 02:31:34
Last Changed Author: Don-vip
Revision: 5697
Repository UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
URL: http://josm.openstreetmap.de/svn/trunk
Last Changed Date: 2013-02-06 01:35:48 +0100 (Wed, 06 Feb 2013)
Last Changed Rev: 5697

Identification: JOSM/1.5 (5697 de)
Memory Usage: 82 MB / 247 MB (30 MB allocated, but free)
Java version: 1.7.0_15, Oracle Corporation, Java HotSpot(TM) Client VM
Operating system: Windows XP
Dataset consistency test: No problems found

Plugin: CommandLine (29210)
Plugin: FixAddresses (29210)
Plugin: PicLayer (29210)
Plugin: ext_tools (29210)
Plugin: jts (28945)
Plugin: mirrored_download (29210)
Plugin: namemanager (29210)
Plugin: openstreetbugs (29210)
Plugin: pdfimport (29210)
Plugin: restart (29210)
Plugin: reverter (29216)
Plugin: tag2link (29210)
Plugin: terracer (29210)
Plugin: turnrestrictions (29210)
Plugin: waydownloader (29210)
Plugin: wayselector (29210)
Plugin: wikipedia (29218)

org.openstreetmap.josm.data.osm.DataIntegrityProblemException: Primitive cannot be included in more than one Dataset
	at org.openstreetmap.josm.data.osm.OsmPrimitive.setDataset(OsmPrimitive.java:309)
	at org.openstreetmap.josm.data.osm.Node.setDataset(Node.java:189)
	at org.openstreetmap.josm.data.osm.DataSet.addPrimitive(DataSet.java:359)
	at CommandLine.CommandLine$3.run(CommandLine.java:518)
	at java.lang.Thread.run(Unknown Source)

[Don-vip]

In 5737/josm:

see #8442 - Add write methods to OsmWriter to not rely only on DataSet

[Don-vip]

new patch

[Don-vip]

new piece of code:

                Collection<OsmPrimitive> contents = new ArrayList<OsmPrimitive>();
                for (OsmPrimitive primitive : refObjects) {
                    contents.add(primitive);
                    if (bbox == null)
                        bbox = new BBox(primitive.getBBox());
                    else
                        bbox.addPrimitive(primitive, 0.0);
                }
                for (Parameter parameter : parameters) {
                    if (!parameter.isOsm())
                        continue;
                    pObjects = parameter.getParameterObjects();
                    for (OsmPrimitive primitive : pObjects) {
                        contents.add(primitive);
                        if (bbox == null)
                            bbox = new BBox(primitive.getBBox());
                        else
                            bbox.addPrimitive(primitive, 0.0);
                    }
                }
                osmWriter.writeNodes(new SubclassFilteredCollection<OsmPrimitive, Node>(contents, OsmPrimitive.nodePredicate));
                osmWriter.writeWays(new SubclassFilteredCollection<OsmPrimitive, Way>(contents, OsmPrimitive.wayPredicate));
                osmWriter.writeRelations(new SubclassFilteredCollection<OsmPrimitive, Relation>(contents, OsmPrimitive.relationPredicate));
                osmWriter.footer();
                osmWriter.flush();

[Don-vip]

And new jar. Can you test it with tomorrow's latest ? (r5737+). Thanks.

[jjaf.de]

Test on 5737/josm with above patch positive!
Tested one and several relations without and and the same with referenced objects.

[Don-vip]

Fixed in [o29273]

[jjaf.de]

I believe some later changes made this symptom reappear.
For this case i did not delete the workaround JOSMticket8442() in ​http://osm.jjaf.de/wiwosm/bot/?source

[Don-vip]

Indeed my change has been reverted in r29817/osm.

@Hind, can you please see with with jjaf if he's using the plugin as you would expect? I though my fix was the way to handle this issue, it looks like not :)

[Hind]

Hi all :3
​First XML document contains dependences for parameters which type is OSM object.

In total, CommandLine passes 1 + N XMLs to external tool, where N is the number of such parameters.

[taylor.smock]

Stupid question: is this still a problem? Or can we close this issue?

[jjaf.de]

Thank you taylor.smock for trying to close this ticket. Sadly there seems to be an issue here still.

CommandLine test command

I created a test command for CommandLine that dumps the input to a file for debugging. Put these two files in the CommandLine-plugin directory (e.g. ~/.local/share/JOSM/plugins/CommandLine/) and modify output filename /tmp/stdin2file.out if required:

stdin2file.xml

<?xml version="1.0" encoding="UTF-8"?>
<command version="1" name="stdin2file" run="python stdin2file.py">
        <parameter required="true" type="any" maxinstances="0">
                <name>Objects</name>
                <description>Objects to write to file</description>
        </parameter>
</command>

stdin2file.py

#!/usr/bin/env python

import sys
import fileinput

file = open("/tmp/stdin2file.out", "w") 

for line in fileinput.input():
        file.write(line)

file.close()

Test

Tests where conducted with JOSM 18427 and CommandLine 35951.

node

Creating a new node and sending it to the command results in defect xml having to xml documents concatenated the first one empty:

<?xml version='1.0' encoding='UTF-8'?>
<osm version='0.6' generator='JOSM'>
</osm>
<?xml version='1.0' encoding='UTF-8'?>
<osm version='0.6' generator='JOSM'>
  <node id='-101762' visible='true' lat='52.36710389541' lon='9.73775417476' />
</osm>

way

Creating a new way shows that the first xml document is not always empty having node separated from way.

<?xml version='1.0' encoding='UTF-8'?>
<osm version='0.6' generator='JOSM'>
  <node id='-101761' visible='true' lat='52.3671080428' lon='9.7377493544' />
  <node id='-101762' visible='true' lat='52.36710389541' lon='9.73775417476' />
</osm>
<?xml version='1.0' encoding='UTF-8'?>
<osm version='0.6' generator='JOSM'>
  <way id='-101783' visible='true'>
    <nd ref='-101761' />
    <nd ref='-101762' />
  </way>
</osm>

relation

Also with relations the first document is not empty but unlike a way it contains the whole way (node and way) but separated from relation:

<?xml version='1.0' encoding='UTF-8'?>
<osm version='0.6' generator='JOSM'>
  <node id='-101761' visible='true' lat='52.3671080428' lon='9.7377493544' />
  <node id='-101762' visible='true' lat='52.36710389541' lon='9.73775417476' />
  <way id='-101783' visible='true'>
    <nd ref='-101761' />
    <nd ref='-101762' />
  </way>
</osm>
<?xml version='1.0' encoding='UTF-8'?>
<osm version='0.6' generator='JOSM'>
  <relation id='-99762' visible='true'>
    <member type='way' ref='-101783' role='' />
    <tag k='type' v='test' />
  </relation>
</osm>

[jjaf.de]

Replying to Hind:

​First XML document contains dependences for parameters which type is OSM object.

In total, CommandLine passes 1 + N XMLs to external tool, where N is the number of such parameters.

Seems to me CommandLine does behave like given in this description but it is not described that explicitly in the linked wiki article. If this is really the correct description then one could close this bug and have to handle two XML documents coming on the same stream in the receiving application.