Modify

Opened 7 years ago

Last modified 6 years ago

#8591 new defect

consecutive identical nodes in ways

Reported by: Oli-Wan Owned by: team
Priority: normal Milestone:
Component: Core Version:
Keywords: Cc:

Description

It appears that JOSM can create ways containing consecutive identical nodes, perhaps only under special circumstances. A similar behaviour (too old to be called a bug) has been known from Potlatch for quite a while, cf. https://trac.openstreetmap.org/ticket/2501 .

This way is the only example I am currently aware of: http://www.openstreetmap.org/browse/way/216494527 . There may be more, but the issue is certainly a rare one.

Attachments (2)

8591_ways.ods (7.2 KB) - added by Don-vip 7 years ago.
List of buggy ways created with JOSM
dup_nodes.zip (2.0 KB) - added by xeen 7 years ago.

Download all attachments as: .zip

Change History (54)

comment:1 Changed 7 years ago by Don-vip

Could a German-speaking user contact this one to ask him how he was able to do this ?

We can add a new validator test to detect this (EDIT: there's already a test for it: "Duplicated way nodes"), but the root cause should be identified as well.

Last edited 7 years ago by Don-vip (previous) (diff)

comment:2 in reply to:  1 ; Changed 7 years ago by skyper

Replying to Don-vip:

Could a German-speaking user contact this one to ask him how he was able to do this ?

Done

comment:3 in reply to:  2 Changed 7 years ago by skyper

Replying to skyper:

Replying to Don-vip:

Could a German-speaking user contact this one to ask him how he was able to do this ?

Done

Got an answer but the user does not know anymore how he was able to create this way.

comment:4 Changed 7 years ago by skyper

Considering plugins, he is/was using:

  • BuildingsTools
  • FixAddresses
  • mirrored download

comment:5 Changed 7 years ago by Oli-Wan

Scanning through Europe (as by Geofabrik's definition), I found a few more examples, all of them quite recent. Maybe the timestamps help in narrowing down the bug; otherwise I see no pattern in the list. The oldest offending way in the list is from October, 2012.

The number after the link is the version in which the repeated nodes first occurred; the editor info is taken from the created_by tag of the corresponding changeset.

I don't like saying this, but the erroneous edits created by JOSM outnumber those of all other editors together, including Potlatch 1/2 and iD. The issuse is starting to look more serious than I first thought.
http://www.openstreetmap.org/browse/way/216620170 v1 2013-04-09 JOSM/1.5 (5836 ru) Windows 7
http://www.openstreetmap.org/browse/way/216619279 v1 2013-04-09 JOSM/1.5 (5836 ru) Windows 7
http://www.openstreetmap.org/browse/way/216619266 v1 2013-04-09 JOSM/1.5 (5836 ru) Windows 7
http://www.openstreetmap.org/browse/way/216616639 v1 2013-04-09 JOSM/1.5 (5836 ru) Windows 7
http://www.openstreetmap.org/browse/way/216601461 v1 2013-04-09 JOSM/1.5 (5836 ru) Windows 7
http://www.openstreetmap.org/browse/way/216601437 v1 2013-04-09 JOSM/1.5 (5836 ru) Windows 7
http://www.openstreetmap.org/browse/way/216591726 v1 2013-04-09 JOSM/1.5 (5836 ru) Windows 7
http://www.openstreetmap.org/browse/way/216591725 v1 2013-04-09 JOSM/1.5 (5836 ru) Windows 7
http://www.openstreetmap.org/browse/way/216494527 v1 2013-04-08 JOSM/1.5 (5759 de)
http://www.openstreetmap.org/browse/way/216422814 v1 2013-04-08 JOSM/1.5 (5759 ru)
http://www.openstreetmap.org/browse/way/216419340 v1 2013-04-08 JOSM/1.5 (5759 ru)
http://www.openstreetmap.org/browse/way/216406948 v1 2013-04-08 JOSM/1.5 (5759 ru)
http://www.openstreetmap.org/browse/way/215188946 v1 2013-04-03 JOSM/1.5 (5759 en)
http://www.openstreetmap.org/browse/way/213887621 v1 2013-03-29 JOSM/1.5 (5759 nl)
http://www.openstreetmap.org/browse/way/213504767 v1 2013-03-28 JOSM/1.5 (5759 nl)
http://www.openstreetmap.org/browse/way/213504766 v1 2013-03-28 JOSM/1.5 (5759 nl)
http://www.openstreetmap.org/browse/way/213410410 v1 2013-03-28 JOSM/1.5 (5759 ru)
http://www.openstreetmap.org/browse/way/213410404 v1 2013-03-28 JOSM/1.5 (5759 ru)
http://www.openstreetmap.org/browse/way/213410403 v1 2013-03-28 JOSM/1.5 (5759 ru)
http://www.openstreetmap.org/browse/way/212360710 v1 2013-03-25 JOSM/1.5 (5759 nl)
http://www.openstreetmap.org/browse/way/212219016 v1 2013-03-25 JOSM/1.5 (5759 fr)
http://www.openstreetmap.org/browse/way/212016622 v1 2013-03-24 JOSM/1.5 (5759 fr)
http://www.openstreetmap.org/browse/way/212016621 v1 2013-03-24 JOSM/1.5 (5759 fr)
http://www.openstreetmap.org/browse/way/210932176 v1 2013-03-19 JOSM/1.5 (5697 ru)
http://www.openstreetmap.org/browse/way/209616128 v1 2013-03-12 JOSM/1.5 (5759 nl)
http://www.openstreetmap.org/browse/way/209514301 v1 2013-03-12 JOSM/1.5 (5759 cs)
http://www.openstreetmap.org/browse/way/209392863 v1 2013-03-11 JOSM/1.5 (5759 nl)
http://www.openstreetmap.org/browse/way/209073783 v1 2013-03-09 JOSM/1.5 (5576 ru)
http://www.openstreetmap.org/browse/way/208991576 v1 2013-03-09 JOSM/1.5 (5759 de)
http://www.openstreetmap.org/browse/way/208458782 v1 2013-03-05 JOSM/1.5 (5608 en)
http://www.openstreetmap.org/browse/way/208455226 v1 2013-03-05 JOSM/1.5 (5697 pl)
http://www.openstreetmap.org/browse/way/207515000 v1 2013-02-27 JOSM/1.5 (5697 en)
http://www.openstreetmap.org/browse/way/205827499 v1 2013-02-16 JOSM/1.5 (5576 ru)
http://www.openstreetmap.org/browse/way/205433774 v1 2013-02-14 JOSM/1.5 (5608 en)
http://www.openstreetmap.org/browse/way/205206821 v1 2013-02-12 JOSM/1.5 (5697 fr)
http://www.openstreetmap.org/browse/way/205206815 v1 2013-02-12 JOSM/1.5 (5697 fr)
http://www.openstreetmap.org/browse/way/205206813 v1 2013-02-12 JOSM/1.5 (5697 fr)
http://www.openstreetmap.org/browse/way/205206810 v1 2013-02-12 JOSM/1.5 (5697 fr)
http://www.openstreetmap.org/browse/way/204178585 v1 2013-02-04 JOSM/1.5 (5608 pl)
http://www.openstreetmap.org/browse/way/204178582 v1 2013-02-04 JOSM/1.5 (5608 pl)
http://www.openstreetmap.org/browse/way/204062730 v1 2013-02-03 JOSM/1.5 (5608 en)
http://www.openstreetmap.org/browse/way/204058151 v1 2013-02-03 JOSM/1.5 (5608 en)
http://www.openstreetmap.org/browse/way/204058148 v1 2013-02-03 JOSM/1.5 (5608 en)
http://www.openstreetmap.org/browse/way/204058144 v1 2013-02-03 JOSM/1.5 (5608 en)
http://www.openstreetmap.org/browse/way/204058141 v1 2013-02-03 JOSM/1.5 (5608 en)
http://www.openstreetmap.org/browse/way/203669975 v1 2013-01-31 JOSM/1.5 (5608 en)
http://www.openstreetmap.org/browse/way/203514851 v1 2013-01-30 JOSM/1.5 (5608 en_GB)
http://www.openstreetmap.org/browse/way/202847387 v1 2013-01-25 JOSM/1.5 (5608 es)
http://www.openstreetmap.org/browse/way/202847374 v1 2013-01-25 JOSM/1.5 (5608 es)
http://www.openstreetmap.org/browse/way/202448058 v1 2013-01-22 JOSM/1.5 (5608 pl)
http://www.openstreetmap.org/browse/way/202448055 v1 2013-01-22 JOSM/1.5 (5608 pl)
http://www.openstreetmap.org/browse/way/202045154 v1 2013-01-20 JOSM/1.5 (5608 pl)
http://www.openstreetmap.org/browse/way/201469862 v1 2013-01-16 JOSM/1.5 (5615 fr)
http://www.openstreetmap.org/browse/way/201126910 v1 2013-01-14 JOSM/1.5 (5615 fr)
http://www.openstreetmap.org/browse/way/200288143 v1 2013-01-09 JOSM/1.5 (5608 ru)
http://www.openstreetmap.org/browse/way/200288084 v1 2013-01-09 JOSM/1.5 (5608 ru)
http://www.openstreetmap.org/browse/way/199355531 v1 2013-01-03 JOSM/1.5 (5608 ru)
http://www.openstreetmap.org/browse/way/197048396 v1 2012-12-18 JOSM/1.5 (5576 ru)
http://www.openstreetmap.org/browse/way/196844716 v1 2012-12-17 JOSM/1.5 (5485 fr)
http://www.openstreetmap.org/browse/way/196841349 v1 2012-12-17 JOSM/1.5 (5539 ru)
http://www.openstreetmap.org/browse/way/196535654 v1 2012-12-15 JOSM/1.5 (5576 ru)
http://www.openstreetmap.org/browse/way/196519424 v1 2012-12-15 JOSM/1.5 (5608 ru)

Last edited 7 years ago by Don-vip (previous) (diff)

comment:6 Changed 7 years ago by Oli-Wan

here's the rest of the list. trac wouldn't allow me to post them all in one comment, claiming I was spamming...

http://www.openstreetmap.org/browse/way/196328780 v1 2012-12-14 JOSM/1.5 (5576 ru)
http://www.openstreetmap.org/browse/way/196328777 v1 2012-12-14 JOSM/1.5 (5576 ru)
http://www.openstreetmap.org/browse/way/196287440 v1 2012-12-14 JOSM/1.5 (5576 ru)
http://www.openstreetmap.org/browse/way/194252568 v1 2012-12-03 JOSM/1.5 (5539 ru)
http://www.openstreetmap.org/browse/way/194252369 v1 2012-12-03 JOSM/1.5 (5539 ru)
http://www.openstreetmap.org/browse/way/194174241 v1 2012-12-02 JOSM/1.5 (5576 ru)
http://www.openstreetmap.org/browse/way/194118698 v1 2012-12-02 JOSM/1.5 (5576 ru)
http://www.openstreetmap.org/browse/way/193768486 v1 2012-11-30 JOSM/1.5 (5576 pl)
http://www.openstreetmap.org/browse/way/193135634 v1 2012-11-27 JOSM/1.5 (5576 ru)
http://www.openstreetmap.org/browse/way/192423324 v1 2012-11-24 JOSM/1.5 (5576 pl)
http://www.openstreetmap.org/browse/way/192423321 v1 2012-11-24 JOSM/1.5 (5576 pl)
http://www.openstreetmap.org/browse/way/192423296 v1 2012-11-24 JOSM/1.5 (5576 pl)
http://www.openstreetmap.org/browse/way/192423293 v1 2012-11-24 JOSM/1.5 (5576 pl)
http://www.openstreetmap.org/browse/way/192420705 v1 2012-11-24 JOSM/1.5 (5576 pl)
http://www.openstreetmap.org/browse/way/192420695 v1 2012-11-24 JOSM/1.5 (5576 pl)
http://www.openstreetmap.org/browse/way/192420690 v1 2012-11-24 JOSM/1.5 (5576 pl)
http://www.openstreetmap.org/browse/way/192420669 v1 2012-11-24 JOSM/1.5 (5576 pl)
http://www.openstreetmap.org/browse/way/191975211 v1 2012-11-21 JOSM/1.5 (5576 ru)
http://www.openstreetmap.org/browse/way/191975182 v1 2012-11-21 JOSM/1.5 (5576 ru)
http://www.openstreetmap.org/browse/way/191907475 v2 2012-11-21 JOSM/1.5 (5576 en)
http://www.openstreetmap.org/browse/way/191897211 v1 2012-11-21 JOSM/1.5 (5531 fr)
http://www.openstreetmap.org/browse/way/191897109 v1 2012-11-21 JOSM/1.5 (5531 fr)
http://www.openstreetmap.org/browse/way/191882964 v1 2012-11-21 JOSM/1.5 (5576 fr)
http://www.openstreetmap.org/browse/way/190874247 v1 2012-11-15 JOSM/1.5 (5485 pl)
http://www.openstreetmap.org/browse/way/190643131 v1 2012-11-13 JOSM/1.5 (5531 fr)
http://www.openstreetmap.org/browse/way/190605955 v1 2012-11-13 JOSM/1.5 (5576 ru)
http://www.openstreetmap.org/browse/way/190529862 v1 2012-11-13 JOSM/1.5 (5485 pl)
http://www.openstreetmap.org/browse/way/189878752 v1 2012-11-09 JOSM/1.5 (5485 pl)
http://www.openstreetmap.org/browse/way/189560440 v1 2012-11-07 JOSM/1.5 (5387 ru)
http://www.openstreetmap.org/browse/way/189271556 v1 2012-11-05 JOSM/1.5 (5485 pl)
http://www.openstreetmap.org/browse/way/189246566 v1 2012-11-05 JOSM/1.5 (5531 ru)
http://www.openstreetmap.org/browse/way/184991848 v4 2013-03-20 JOSM/1.5 (5759 en)
http://www.openstreetmap.org/browse/way/184455063 v1 2012-10-06 JOSM/1.5 (5485 ru)
http://www.openstreetmap.org/browse/way/184454159 v1 2012-10-06 JOSM/1.5 (5485 ru)
http://www.openstreetmap.org/browse/way/177704837 v3 2013-03-06 JOSM/1.5 (5697 en)
http://www.openstreetmap.org/browse/way/122787902 v3 2012-10-05 JOSM/1.5 (5608 en)
http://www.openstreetmap.org/browse/way/120310023 v4 2013-03-12 JOSM/1.5 (5759 en)
http://www.openstreetmap.org/browse/way/112508498 v3 2012-12-03 JOSM/1.5 (5531 ru)
http://www.openstreetmap.org/browse/way/27316411 v15 2013-03-12 JOSM/1.5 (5759 en)
http://www.openstreetmap.org/browse/way/25475233 v12 2013-02-24 JOSM/1.5 (5697 nl)
http://www.openstreetmap.org/browse/way/24207158 v3 2013-04-05 JOSM/1.5 (5759 de)
http://www.openstreetmap.org/browse/way/22744248 v5 2013-01-30 JOSM/1.5 (5608 en_GB)
http://www.openstreetmap.org/browse/way/19077433 v19 2013-01-10 JOSM/1.5 (5267 en)

comment:7 Changed 7 years ago by stoecker

If we don't know the action which created such ways there is little chance to find the source. We had such issues in the past in core and they have been fixed. But we have so many actions, we can't do an complete code review to find such an issue without any additional info.

comment:8 Changed 7 years ago by Oli-Wan

True, but as I said: the timestamps/revision numbers might help pin down the issue. Afaik xybot stopped repairing such ways by the end of March, 2012 - so most offending ways created after that date should be unaffected. Since defective ways appeared to have shown up only from October, 2012 on, edits to code affecting the node structure of ways in the weeks before revision 5485 might be worth looking at.

Also, there is a relatively simple solution to avoid creating such ways in the database, even if the actual source is not found. Before uploading, iterate through each way's node list and kill all references which are identical to their predecessor. (And make the user file a bug report in such a case, or try to track down the source automatically - which plugins were involved, and so on. It would also be interesting if the issue was noticed by the validator, or why it slipped through.)

comment:9 in reply to:  8 Changed 7 years ago by Don-vip

Replying to Oli-Wan:

Before uploading, iterate through each way's node list and kill all references which are identical to their predecessor. (And make the user file a bug report in such a case, or try to track down the source automatically - which plugins were involved, and so on.

Good idea. A good way to have the missing info to understand this bug.

comment:10 Changed 7 years ago by akks

My FastDraw plugin was creating such ways in rare cases for about a year (fixed not far ago).

Validator shows an error in this case and can fix it automatically (if I understand correctly). Though I do not know how to repriduce it by core and Utilsplugin2.

Changed 7 years ago by Don-vip

Attachment: 8591_ways.ods added

List of buggy ways created with JOSM

comment:11 Changed 7 years ago by Don-vip

In the past month we have about a dozen of mappers who were able to create these ways (see attached file). We have their JOSM language so we can contact English/German/Russian/French-speaking ones :)

comment:12 Changed 7 years ago by Oli-Wan

I have set up a little blame-o-mat for the problematic ways. If everything works correctly, the data should be updated daily from the OSM daydiffs. (If not, I will fix that.) The columns list the object id, version, timestamp, changeset id and the editor to blame. Perhaps if asked soon afterwards, some of the mappers may be able to give helpful information.

http://toolserver.org/~mapjedi/blame.txt

comment:13 Changed 7 years ago by akks

Most of the lines from http://toolserver.org/~mapjedi/blame.txt and the list before seems to be created by my FastDraw before update :-(
I committed it 11 days ago, needed JOSM version was not increased for the fix, but it was already 5738. The bug was very old, so many ways was created by people ignoring validator :-)

But in some (very rare) cases there may be other reasons:
http://www.openstreetmap.org/browse/way/216494527

Last edited 7 years ago by akks (previous) (diff)

comment:14 Changed 7 years ago by Oli-Wan

@akks: If FastDraw really was responsible for the major part of the trouble, I'd call that good news since you already fixed that plugin. But since, unfortunately, revisions >5738 still create erroneous ways, there must be other sources as well - unless, of course, users have updated their JOSM, but kept an old version of that plugin.

A few numbers: There are 515 broken ways in the latest planet file, JOSM having created 247 of them. So the issue is not that big numerically, but there's still something fundamentally wrong somewhere in the code.

Forget my remarks above on revision 5485. Turns out that older versions are also affected. Also, my statement about xybot ceasing its work in March of 2012 appears not to be quite correct; the cutoff happened in October of 2012 (apart from three slightly older ways, which probably just slipped through).

Stats for JOSM revisions based on the latest planet file (two weeks old):

      1 4863
     19 5047
      1 5241
     11 5267
     38 5315
      6 5356
      1 5387
      8 5485
      1 5515
     10 5531
     11 5539
     29 5576
     43 5608
      2 5615
      1 5669
      2 5674
      1 5692
     31 5697
      2 5698
      3 5749
     25 5759
      1 5792

Leading digits only:

      1 48xx
     19 50xx
     12 52xx
     45 53xx
      8 54xx
     51 55xx
     82 56xx
     29 57xx

For recently created ways, see the current "blame file".

comment:15 Changed 7 years ago by akks

I'll try to lower main.version in FastDraw and insert version checking code (to enable fix for old JOSM users).

comment:16 in reply to:  15 Changed 7 years ago by stoecker

Replying to akks:

I'll try to lower main.version in FastDraw and insert version checking code (to enable fix for old JOSM users).

I don't see any sense in that.

When they don't update josm, why do you think they update the plugin?

Also I'm pretty sure lowering main version will result in a big confusion in the plugin loading mechanisms and the related server tasks :-)

comment:17 Changed 7 years ago by akks

I communicated to Trolleway , the only author of edits with JOSM 5836 in http://toolserver.org/~mapjedi/blame.txt
He was using FastDraw but did not update it :)

I have also sent messages to Alex Rollin and Ikks .

comment:18 Changed 7 years ago by Oli-Wan

@akks: Seems like you were right then, which is good news as there is no indication to any flaws in JOSM's core. Given that the average mapper uses a JOSM version which is three months old (and most hopefully update their plugins along with the core), we should see some improvement over the next few months, i.e. fewer JOSM-related entries in the blame file -- especially if you keep nagging the most notorious creators of buggy ways to update their JOSM+plugins ;) I just added another column to the blame file (new entries) to make that a tiny bit easier.

Btw. if http://toolserver.org/~mapjedi/blame.txt returns 404 once in a while, try again after a few minutes. There happen to be some issues with the web servers in the Wikemedia Toolserver cluster at the moment. The file is there, believe me ;)

comment:19 Changed 7 years ago by Don-vip

way	217762635	v1	2013-04-17T13:05:18Z	15762570	ArcGIS Editor for OpenStreetMap (2.1)	handerson
way	217762633	v1	2013-04-17T13:05:17Z	15762570	ArcGIS Editor for OpenStreetMap (2.1)	handerson
way	217762632	v1	2013-04-17T13:05:15Z	15762570	ArcGIS Editor for OpenStreetMap (2.1)	handerson
--------------------------------------------------------------------------------------------------------------------------
way	217857571	v1	2013-04-18T13:04:50Z	15773250	ArcGIS Editor for OpenStreetMap (2.1)	handerson
way	217857570	v1	2013-04-18T13:04:49Z	15773250	ArcGIS Editor for OpenStreetMap (2.1)	handerson
way	217857569	v1	2013-04-18T13:04:47Z	15773250	ArcGIS Editor for OpenStreetMap (2.1)	handerson

We should file a bug to ArcGIS mates :)

EDIT: bug filed: https://esriosmeditor.codeplex.com/workitem/11724

Last edited 7 years ago by Don-vip (previous) (diff)

comment:20 Changed 7 years ago by skyper

Still wonder why all the josm user are not fixing errors before uploading.

How about an additional warning for errors and/or always running error check (e.g. only disable warnings and infos).

comment:21 in reply to:  19 ; Changed 7 years ago by Oli-Wan

Replying to Don-vip:

We should file a bug to ArcGIS mates :)

Or ask its user (singular) to switch to JOSM.
Anyway, thanks for filing the bug report. I did not feel like signing up for yet another issue tracker after writing bug reports for Vespucci (fixed - within six hours - in the upcoming release) and iD (partially fixed). Guess I should write another one for "Go Map!!".

Is it possible to require a minimum version of certain plugins, i.e. automatically disable FastDraw if its version number is less than x? That would provide a smoking gun in case of another bug in the core: if JOSM revision y (the one in which the minimum version requirement was introduced) or higher appears in the blame file, there must be something broken elsewhere than in FastDraw. Right now one has to query the respective user about his plugin usage.

Replying to skyper:

Still wonder why all the josm user are not fixing errors before uploading.

Maybe they don't understand, or don't care? JOSM generates warnings for various other, not so fundamental errors - but people go ahead with their uploads anyway.

comment:22 in reply to:  21 ; Changed 7 years ago by stoecker

Replying to Oli-Wan:

Is it possible to require a minimum version of certain plugins, i.e. automatically disable FastDraw if its version number is less than x? That would provide a smoking gun in case of another bug in the core: if JOSM revision y (the one in which the minimum version requirement was introduced) or higher appears in the blame file, there must be something broken elsewhere than in FastDraw. Right now one has to query the respective user about his plugin usage.

It would be possible, but instead of that we implemented the auto-update system to keep plugins recent. But when user disable that they will have old plugins.

Replying to skyper:

Still wonder why all the josm user are not fixing errors before uploading.

Maybe they don't understand, or don't care? JOSM generates warnings for various other, not so fundamental errors - but people go ahead with their uploads anyway.

Actually the upload texts also says so: "When in doubt ignore them." (the errors). We decided that data with little errors is better than no data at all.

comment:23 Changed 7 years ago by akks

It seems that the main problem is very old JOSM version (older than 3 months). The patched plugins usually need newer JOSM and there is no possibility to patch "older" plugin version. I can "downgrade" FastDraw or compile special version for older JOSM (like 4xxx) in minutes, but there is no correct way to publish it.

However, in recent section http://toolserver.org/~mapjedi/blame.txt there seems to be more incorrect unchecked building imports, than FastDraw buggy ways.

@stoecker : or there is a possibility to replace old version of JAR file?

comment:24 in reply to:  19 ; Changed 7 years ago by Thomas Emge

I would like to investigate those duplicate nodes but the ways (id: 15762570, 15773250)were created before the ArcGIS Editor for OpenStreetMap existed. Certainly way before Version 2.1.

Thomas

Replying to Don-vip:

way	217762635	v1	2013-04-17T13:05:18Z	15762570	ArcGIS Editor for OpenStreetMap (2.1)	handerson
way	217762633	v1	2013-04-17T13:05:17Z	15762570	ArcGIS Editor for OpenStreetMap (2.1)	handerson
way	217762632	v1	2013-04-17T13:05:15Z	15762570	ArcGIS Editor for OpenStreetMap (2.1)	handerson
--------------------------------------------------------------------------------------------------------------------------
way	217857571	v1	2013-04-18T13:04:50Z	15773250	ArcGIS Editor for OpenStreetMap (2.1)	handerson
way	217857570	v1	2013-04-18T13:04:49Z	15773250	ArcGIS Editor for OpenStreetMap (2.1)	handerson
way	217857569	v1	2013-04-18T13:04:47Z	15773250	ArcGIS Editor for OpenStreetMap (2.1)	handerson

We should file a bug to ArcGIS mates :)

EDIT: bug filed: https://esriosmeditor.codeplex.com/workitem/11724

comment:25 Changed 7 years ago by stoecker

It seems that the main problem is very old JOSM version (older than 3 months). The patched plugins usually need newer JOSM and there is no possibility to patch "older" plugin version. I can "downgrade" FastDraw or compile special version for older JOSM (like 4xxx) in minutes, but there is no correct way to publish it.

As said. I don't believe that there will be a real benefit in updating old plugins, when core is not updated. That is also the reason why there is no mechanism for that. Managing multiple version takes a lot of effort for little benefit. JOSMs release model neither encourages nor requires maintaining of old components.

comment:26 in reply to:  24 Changed 7 years ago by Thomas Emge

Ha, I mixed up user id and way id columns.

We'll take a look at the underlying issue.

Thomas

Replying to Thomas Emge:

I would like to investigate those duplicate nodes but the ways (id: 15762570, 15773250)were created before the ArcGIS Editor for OpenStreetMap existed. Certainly way before Version 2.1.

Thomas

Replying to Don-vip:

way	217762635	v1	2013-04-17T13:05:18Z	15762570	ArcGIS Editor for OpenStreetMap (2.1)	handerson
way	217762633	v1	2013-04-17T13:05:17Z	15762570	ArcGIS Editor for OpenStreetMap (2.1)	handerson
way	217762632	v1	2013-04-17T13:05:15Z	15762570	ArcGIS Editor for OpenStreetMap (2.1)	handerson
--------------------------------------------------------------------------------------------------------------------------
way	217857571	v1	2013-04-18T13:04:50Z	15773250	ArcGIS Editor for OpenStreetMap (2.1)	handerson
way	217857570	v1	2013-04-18T13:04:49Z	15773250	ArcGIS Editor for OpenStreetMap (2.1)	handerson
way	217857569	v1	2013-04-18T13:04:47Z	15773250	ArcGIS Editor for OpenStreetMap (2.1)	handerson

We should file a bug to ArcGIS mates :)

EDIT: bug filed: https://esriosmeditor.codeplex.com/workitem/11724

comment:27 in reply to:  22 ; Changed 7 years ago by Oli-Wan

Replying to stoecker:

Replying to Oli-Wan:

Is it possible to require a minimum version of certain plugins, i.e. automatically disable FastDraw if its version number is less than x? [...]

It would be possible, but instead of that we implemented the auto-update system to keep plugins recent. But when user disable that they will have old plugins.

Yes, and I do not doubt that this was a wise decision. However, in a case such as the one at hand, where older versions of a given plugin are known to be seriously flawed a more imperative approach towards "encouraging" users to upgrade also their plugins would make sense in my view.

comment:28 in reply to:  27 Changed 7 years ago by stoecker

Replying to Oli-Wan:

However, in a case such as the one at hand, where older versions of a given plugin are known to be seriously flawed a more imperative approach towards "encouraging" users to upgrade also their plugins would make sense in my view.

Actually this is no serious flaw. If it would be, the API would simply reject such ways. We had serious issues in the past and got an API blocks for some JOSM versions. This is only a little problem.

And too much force for the user is not recommended. We already have a very good update rate. Others would be happy to have.

comment:29 Changed 7 years ago by xeen

I regularly create tags with duplicated nodes, but fix them before uploading. I’m using a combination of building tools, rectify, join nodes to ways, the extrude feature, the turn function, merge nodes, split/unglue nodes, “certain angle only drawing” (i.e. tab key), undo and probably a few others from time to time. I haven’t yet been able to narrow this down because I only ever validate at the end of a “session”. The main reason for this is that the validator runs on all data when checking manually, but only on my changes when uploading. Is there a way to make it “live”, i.e. check every 30s or so? This would make it much easier to find the reason. After adding buildings for a whole city block, I usually don’t remember what I did exactly for those two buildings – at least, it’s usually two adjacent buildings where I manage to create dup nodes.

comment:30 Changed 7 years ago by Oli-Wan

I would assume that this user, who only signed up and made his first edits on April 19, was not using an ancient version of FastDraw:

way	219444918	v1	2013-04-29T19:46:13Z	15913064	JOSM/1.5 (5836 fr) Linux	similia joseph

Similar for these:

way	219431102	v1	2013-04-29T17:35:12Z	15911300	JOSM/1.5 (5836 fr) Linux	Rodney Simeon
way	219413381	v1	2013-04-29T15:09:41Z	15909516	JOSM/1.5 (5836 fr) Linux	garcon frantz
way	219170236	v1	2013-04-28T02:16:12Z	15890220	JOSM/1.5 (5836 en) Windows XP	lamonganscout
way	219113302	v1	2013-04-27T13:58:42Z	15884086	JOSM/1.5 (5836 en) Windows XP	gusti putu

I am inclined to conclude that, unfortunately, there must be sources of repeated nodes other than the FastDraw plugin.

@xeen: I think it would be very much appreciated if you did make the effort of narrowing down which function or plugin those broken ways (probably) originate from. I myself have never received any warning from the Validator about repeated nodes in ways of my own.

comment:31 in reply to:  30 Changed 7 years ago by skyper

Replying to Oli-Wan:

I am inclined to conclude that, unfortunately, there must be sources of repeated nodes other than the FastDraw plugin.

@xeen: I think it would be very much appreciated if you did make the effort of narrowing down which function or plugin those broken ways (probably) originate from. I myself have never received any warning from the Validator about repeated nodes in ways of my own.

+1

I do not use building tool though.

comment:32 in reply to:  30 Changed 7 years ago by akks

Replying to Oli-Wan:

I am inclined to conclude that, unfortunately, there must be sources of repeated nodes other than the FastDraw plugin.

Then, I think, it is Angle Snapping. Because of the same author of code :)

comment:33 Changed 7 years ago by xeen

Yes, it appears to be angle snapping. After akks comment I tried using an empty file using angle snapping only and was able to create a way with two duplicated nodes – but only once. I have not managed to find a way to reproduce this.

comment:34 Changed 7 years ago by xeen

(to be more precise, it was two ways that shared some of their nodes, i.e. two adjacent polygons)

comment:35 Changed 7 years ago by xeen

okay, here is my theory: the angle snapping tool creates duplicates nodes iff it would like to draw at an angle, but the “merge into nearby nodes” code overrules AST’s decision. I’ll attach three .osm files shortly where I managed to reproduce this behaviour. Unfortunately, I can’t provide exact steps to reproduce.

Changed 7 years ago by xeen

Attachment: dup_nodes.zip added

comment:36 Changed 7 years ago by rickmastfan67

I just wanted to make a comment here that a few weeks ago, I was able to make duplicated nodes in the same place once. I never did upload it as I caught it before upload. But I have never had the plugin "FastDraw" installed on my system.

comment:37 in reply to:  35 Changed 7 years ago by akks

Replying to xeen:

okay, here is my theory: the angle snapping tool creates duplicates nodes iff it would like to draw at an angle, but the “merge into nearby nodes” code overrules AST’s decision. I’ll attach three .osm files shortly where I managed to reproduce this behaviour. Unfortunately, I can’t provide exact steps to reproduce.

Thank you, I managed to reproduce it one time...
Angle snapping when drawing near existing line tries to make angles strictly 90 (i.e.) that leads to very near points even if they not coincide. My theory is that they are inserted in nearby lines and make duplicates in some cases.
There are 2 solutions (may be combined)

  • Give nearest point under cursor priority to angle snapping (=> no more near points, but the angles vary a little)
  • When inserting into nearby segments, skip inserting into segment with near points.

comment:38 Changed 7 years ago by akks

Forgot to add "see #8591":
In [5922/josm]:

Avoid duplicate and very near points in Angle snapping draw mode
(will reuse existing nodes in mappaint.node.snap-distance pixel radius if possible)

Tolerance was 0.1 mm, now it is mappaint.node.snap-distance converted form pixels to meters. This should solve most of the cases of creating extremely near or duplicate nodes (and make harder drawing exact rectangles near existing lines, but who needs it?).

Last edited 7 years ago by akks (previous) (diff)

comment:39 in reply to:  29 Changed 7 years ago by stoecker

Replying to xeen:

The main reason for this is that the validator runs on all data when checking manually, but only on my changes when uploading. Is there a way to make it “live”, i.e. check every 30s or so?

There is no auto-check, but you simply can simulate upload behavior. Use "Select Modified" from first menu and then run validator. Validator runs on current selection or all data if nothing is selected.

comment:40 Changed 7 years ago by skyper

Still not fixed ? So iD is taking first place !

way	220772954	v1	2013-05-10T17:44:22Z	16068383	JOSM/1.5 (5939 de) Windows XP 32-Bit	ruph
way	220772948	v1	2013-05-10T17:44:22Z	16068383	JOSM/1.5 (5939 de) Windows XP 32-Bit	ruph
way	220772947	v1	2013-05-10T17:44:22Z	16068383	JOSM/1.5 (5939 de) Windows XP 32-Bit	ruph
way	148738309	v4	2013-05-10T18:34:10Z	16069141	JOSM/1.5 (5939 en) Linux Ubuntu 12.10	Nachtspazz
way	221250418	v1	2013-05-13T20:50:41Z	16116972	JOSM/1.5 (5939 de) Windows XP 32-Bit	ruph
way	221250417	v1	2013-05-13T20:50:41Z	16116972	JOSM/1.5 (5939 de) Windows XP 32-Bit	ruph
way	221181752	v3	2013-05-13T19:15:05Z	16115157	JOSM/1.5 (5939 pl) Windows XP 32-Bit	benek

comment:41 Changed 7 years ago by skyper

You are still able to upload these kind of ways if you import files in JOSM. This is what this user did. With the help of a spanish speaking user I got into contact with him.

Maybe we need some consistency checks and warnings when opening files.

comment:42 in reply to:  41 Changed 7 years ago by Oli-Wan

Replying to skyper:

You are still able to upload these kind of ways if you import files in JOSM. This is what this user did. With the help of a spanish speaking user I got into contact with him.

Maybe we need some consistency checks and warnings when opening files.

I on't think that yet another warning message will help much. I just saved a broken way in an .osm file, inverted the way's object id (text editor), then opened the file in JOSM to upload the way. On upload, the validator gives a warning, as it should. If people ignore this one (even when it appears on 1304 ways as in the case of jsuarezcarbajal's import), they will also ignore a warning that is shown earlier. Auto-correcting this special type of error on opening files and/or before upload might be a better choice. Or simply refuse to open such a file, as JOSM does when other errors are found.

The validator warning pops up only if the nodes are loaded as well; otherwise (i.e. if the file contains only the way itself) it will not. (Changing this behaviour won't solve the current problem either.)

comment:43 Changed 6 years ago by rickmastfan67

I've figured out just about a full proof way to create identical nodes in a single way!! Just happened to me tonight again and I noticed it right away and was able to pretty much continuously duplicate it after that.

Here's how to do it. I've been able to duplicate it almost always on command as long as I don't shift the location of my mouse while following the steps below.

  1. Start drawing a way (A) while holding down the Ctrl button (do not let go of this button till I say so otherwise, you will not get this to happen!!) to prevent the way from connecting to another.
  2. Now after you left click to create a new node, hit the right click button on your mouse. (DO NOT MOVE THE MOUSE AFTER YOU LEFT CLICK)
  3. Hit your "ESC" button (For some reason, Window's start menu pops up the first time you hit ESC while holding down the Ctrl button but just ignore this for now).
  4. Now, left click once again in the JOSM screen.
  5. Now, move the mouse and left click again to create another node and release the CTRL button.
  6. Now, hit the "s" button and select the area you were when you hit he "ESC" button.

What happens:
There are now two identical nodes in the same place in the same way.

Repository Root: http://josm.openstreetmap.de/svn
Build-Date: 2013-06-22 01:35:22
Last Changed Author: Don-vip
Revision: 6016
Repository UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
URL: http://josm.openstreetmap.de/svn/trunk
Last Changed Date: 2013-06-21 09:43:29 +0200 (Fri, 21 Jun 2013)
Last Changed Rev: 6016

Identification: JOSM/1.5 (6016 en) Windows 7 64-Bit
Memory Usage: 196 MB / 2730 MB (24 MB allocated, but free)
Java version: 1.7.0_25, Oracle Corporation, Java HotSpot(TM) 64-Bit Server VM
VM arguments: [-Xmx3072M]
Dataset consistency test: No problems found

Plugin: ImageryCache (29690)
Plugin: OpeningHoursEditor (29435)
Plugin: buildings_tools (29596)
Plugin: mapdust (29525)
Plugin: measurement (29625)
Plugin: mirrored_download (29643)
Plugin: openstreetbugs (29435)
Plugin: osmarender (29639)
Plugin: reverter (29663)
Plugin: turnrestrictions (29435)
Plugin: undelete (29555)
Plugin: utilsplugin2 (29672)
Last edited 6 years ago by rickmastfan67 (previous) (diff)

comment:44 Changed 6 years ago by akks

In 6027/josm:

see #8591: do not allow creating ways with duplicate nodes in Draw mode, even with Ctrl pressed

comment:45 in reply to:  43 Changed 6 years ago by akks

Replying to rickmastfan67:

I've figured out just about a full proof way to create identical nodes in a single way!! Just happened to me tonight again and I noticed it right away and was able to pretty much continuously duplicate it after that.

Thank you!
I tried to fix the problem "globally" by reusing nodes of selected way even when ctrl is pressed (only nodes from other nodes will be ignored).

comment:46 Changed 6 years ago by Oli-Wan

Just a few words of reassurance here. Looking at http://toolserver.org/~mapjedi/blame.txt you see that although several loopholes have been closed, even recent JOSM versions still show up, proving that there are still problems. So you might think that little actual progress has been made.

That discouraging impression is wrong. Obviously, there are still bugs present; but the efforts made so far have already lead to a significant reduction in the creation of broken ways. Let me back up this claim by a few numbers.

Consider the number of users involved in creating broken ways each day. Under a few reasonable assumptions, this number should (and in fact does) nicely follow a Poisson distribution, and therefore lends itself better to statistical analysis than the total number of broken ways created within one day (whose distribution is unknown and easily distorted by someone like jsuarezcarbajal creating some 4200 broken ways in a couple of days). The average numbers are:

JOSM, 4th quarter 2012: 1.80 ± 0.14
JOSM, 1st quarter 2013: 1.83 ± 0.15
JOSM, 2nd quarter 2013: 1.43 ± 0.14
JOSM, July 01-22, 2013: 0.81 ± 0.15

The decrease is obvious, and it is even statistically significant. So thanks for your efforts so far.
To find the remaining bugs, I would suggest to continue contacting users who create broken ways on a regular basis, like Vítor Dias. Someone here speaking Portuguese?

comment:47 in reply to:  46 ; Changed 6 years ago by skyper

Replying to Oli-Wan:

[...]

The decrease is obvious, and it is even statistically significant. So thanks for your efforts so far.
To find the remaining bugs, I would suggest to continue contacting users who create broken ways on a regular basis, like Vítor Dias. Someone here speaking Portuguese?

Thanks for the numbers. Maybe, we should once more point developers and other active mapper to this list, as speaking about JOSM more and more often the errors are made by the user and not the software.

I know that some people speaking Portuguese are listening at talk@osm.

@akks:
FMS seems to be speaking Russian.

comment:48 in reply to:  47 Changed 6 years ago by akks


I know that some people speakin
@akks:
FMS seems to be speaking Russian.

FMS was a well-known sociopate, I doubt someone can get any non-automatical trolling answer from him :) His sources are random but mostly legal. I guess it is some sort of dirty import again.

comment:49 Changed 6 years ago by Oli-Wan

I have contacted one of the JOSM users recently involved in creating a broken way (using an up-to-date version of JOSM) and received a very precise description that allows to reproduce the specific incident, as the user fortunately remembered the procedure in full detail. This is him:

way	231868995	v1	2013-07-30T21:54:50Z	17159087	JOSM/1.5 (6060 de)	XM-Franz

And here is his description: http://www.ruether24.de/JOSM/Beschreibung.pdf - in de, so for those who don't speak German I'll summarize it below.

Let me first introduce some terminology. Divide the creation of broken ways into two classes: primary and secondary creation, where primary means creating a broken way from scratch, while secondary means that there existed a broken way beforehand, which was then edited and split up, in the course of which a new broken way was created.

It turns out that the incident in question is of the latter type, resulting from a broken way previously created by anotheruser (and a different editor):

way	230951423	v1	2013-07-23T10:26:38Z	17060113	iD 1.0.1	spaghettieis

Way 231868995 was then created by splitting up way 230951423; this should have been caught by JOSM. While we are concerned only with secondary creation here, my hope is that primary and secondary creation can be prevented by one single safety measure in the code.

To reproduce the erroneus behaviour, you will want to work with OSM data from before the edits by XM-Franz. Here you are: https://toolserver.org/~mapjedi/duisburg.osm . This is the corresponding region taken from a Geofabrik extract of the night before.

Here's the summary of XM-Franz' description. Please have a look at the images in his file.

Starting point was an error message in keep right about a way containing more than one node several times. So zoom to way 230951423. It contains a loop between node 2393749427 and 2393749429 (i.e. between the stairs and the bridge), and also node 2393749427 twice in a row. Then using the crosshair on the longer segment, pull apart the double-line to obtain a triangular loop. (This step is not necessary to reproduce the broken way, but part of XM-Franz' description - and it helps to see things a bit more clearly.)

Now select nodes 2393749429 and 2393749427, split ("p") the way there. You will obtain a total of four ways (which is already a bit strange given that we're splitting at only two nodes), one of which contains only node 2393749427, but twice. (You can find it most easily by opening a rectangle selection box around that particular node.)

If you hit the upload button, the validator will complain; i.e. the error is detected.

Since no plugins are involved (XM-Franz even explicitly uninstalled all plugins on his machine to be sure), this time the issue is somewhere in the core.

comment:50 Changed 6 years ago by akks

Wow, thank you very much for you help! Give our thanks to XM-Franz too when it will be fixed :)
However, it was already double node... We should make splitting the line more smart, I guess.

Last edited 6 years ago by akks (previous) (diff)

comment:51 Changed 6 years ago by akks

I cannot fix it myself reliably enough (testing is near).

comment:52 Changed 6 years ago by Oli-Wan

While I still hope that the issue sketched in comment:49 can be addressed, let me come back to another problem. There seem to be tools around that generate OSM files that contain broken ways. People then open these files in JOSM and upload their data. JOSM will complain about the broken ways, but if people don't cancel the upload and actively hit the repair button, the broken ways will make their way into the database.

One example of those buggy third-party tools appears to be "Global Mapper 13.0", using which user Jestr88 recently created 112 broken ways; the 4208 broken ways by user jsuarezcarbajal have also been said to originate from some external tool.

I would like to repeat the suggestion that JOSM be more rigid towards such low-level data errors when working with files. Those errors should not end up in the database, no matter what. A warning message does not seem to suffice in this case: people won't understand it, or they won't know how to fix the problem (i.e. tell JOSM to do this automagically). So JOSM should either auto-fix any broken ways with a negative ID already in the process of opening the file, or apply an auto-fix to anything that is going to be uploaded.

(I am not suggesting to enforce *all* validator issues - the validator is wrong every once in a while. But the specific case discussed here is *always* an error.)

Modify Ticket

Change Properties
Set your email in Preferences
Action
as new The owner will remain team.
as The resolution will be set.
to The owner will be changed from team to the specified user.
The owner will change to Oli-Wan
as duplicate The resolution will be set to duplicate.The specified ticket will be cross-referenced with this ticket
The owner will be changed from team to anonymous.

Add Comment


E-mail address and name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.