#8591 (consecutive identical nodes in ways) – JOSM

follow-up: 2 comment:1 by Don-vip, 12 years ago

Could a German-speaking user contact this one to ask him how he was able to do this ?

~~We can add a new validator test to detect this~~ (EDIT: there's already a test for it: "Duplicated way nodes"), but the root cause should be identified as well.

Last edited 12 years ago by Don-vip (previous) (diff)

in reply to: 1 ; follow-up: 3 comment:2 by skyper, 12 years ago

Replying to Don-vip:

Could a German-speaking user contact this one to ask him how he was able to do this ?

Done

in reply to: 2 comment:3 by skyper, 12 years ago

Replying to skyper:

Replying to Don-vip:

Could a German-speaking user contact this one to ask him how he was able to do this ?

Done

Got an answer but the user does not know anymore how he was able to create this way.

comment:4 by skyper, 12 years ago

Considering plugins, he is/was using:

BuildingsTools
FixAddresses
mirrored download

comment:5 by Oli-Wan, 12 years ago

Scanning through Europe (as by Geofabrik's definition), I found a few more examples, all of them quite recent. Maybe the timestamps help in narrowing down the bug; otherwise I see no pattern in the list. The oldest offending way in the list is from October, 2012.

The number after the link is the version in which the repeated nodes first occurred; the editor info is taken from the created_by tag of the corresponding changeset.

Last edited 12 years ago by Don-vip (previous) (diff)

comment:6 by Oli-Wan, 12 years ago

here's the rest of the list. trac wouldn't allow me to post them all in one comment, claiming I was spamming...

comment:7 by stoecker, 12 years ago

If we don't know the action which created such ways there is little chance to find the source. We had such issues in the past in core and they have been fixed. But we have so many actions, we can't do an complete code review to find such an issue without any additional info.

follow-up: 9 comment:8 by Oli-Wan, 12 years ago

True, but as I said: the timestamps/revision numbers might help pin down the issue. Afaik xybot stopped repairing such ways by the end of March, 2012 - so most offending ways created after that date should be unaffected. Since defective ways appeared to have shown up only from October, 2012 on, edits to code affecting the node structure of ways in the weeks before revision 5485 might be worth looking at.

Also, there is a relatively simple solution to avoid creating such ways in the database, even if the actual source is not found. Before uploading, iterate through each way's node list and kill all references which are identical to their predecessor. (And make the user file a bug report in such a case, or try to track down the source automatically - which plugins were involved, and so on. It would also be interesting if the issue was noticed by the validator, or why it slipped through.)

in reply to: 8 comment:9 by Don-vip, 12 years ago

Replying to Oli-Wan:

Before uploading, iterate through each way's node list and kill all references which are identical to their predecessor. (And make the user file a bug report in such a case, or try to track down the source automatically - which plugins were involved, and so on.

Good idea. A good way to have the missing info to understand this bug.

comment:10 by akks, 12 years ago

My FastDraw plugin was creating such ways in rare cases for about a year (fixed not far ago).

Validator shows an error in this case and can fix it automatically (if I understand correctly). Though I do not know how to repriduce it by core and Utilsplugin2.

by Don-vip, 12 years ago

Attachment:	8591_ways.ods added

List of buggy ways created with JOSM

comment:11 by Don-vip, 12 years ago

In the past month we have about a dozen of mappers who were able to create these ways (see attached file). We have their JOSM language so we can contact English/German/Russian/French-speaking ones :)

comment:12 by Oli-Wan, 12 years ago

I have set up a little blame-o-mat for the problematic ways. If everything works correctly, the data should be updated daily from the OSM daydiffs. (If not, I will fix that.) The columns list the object id, version, timestamp, changeset id and the editor to blame. Perhaps if asked soon afterwards, some of the mappers may be able to give helpful information.

http://toolserver.org/~mapjedi/blame.txt

comment:13 by akks, 12 years ago

Most of the lines from http://toolserver.org/~mapjedi/blame.txt and the list before seems to be created by my FastDraw before update :-(
I committed it 11 days ago, needed JOSM version was not increased for the fix, but it was already 5738. The bug was very old, so many ways was created by people ignoring validator :-)

But in some (very rare) cases there may be other reasons:
http://www.openstreetmap.org/browse/way/216494527

Last edited 12 years ago by akks (previous) (diff)

comment:14 by Oli-Wan, 12 years ago

@akks: If FastDraw really was responsible for the major part of the trouble, I'd call that good news since you already fixed that plugin. But since, unfortunately, revisions >5738 still create erroneous ways, there must be other sources as well - unless, of course, users have updated their JOSM, but kept an old version of that plugin.

A few numbers: There are 515 broken ways in the latest planet file, JOSM having created 247 of them. So the issue is not that big numerically, but there's still something fundamentally wrong somewhere in the code.

Forget my remarks above on revision 5485. Turns out that older versions are also affected. Also, my statement about xybot ceasing its work in March of 2012 appears not to be quite correct; the cutoff happened in October of 2012 (apart from three slightly older ways, which probably just slipped through).

Stats for JOSM revisions based on the latest planet file (two weeks old):

Leading digits only:

For recently created ways, see the current "blame file".

follow-up: 16 comment:15 by akks, 12 years ago

I'll try to lower main.version in FastDraw and insert version checking code (to enable fix for old JOSM users).

in reply to: 15 comment:16 by stoecker, 12 years ago

Replying to akks:

I'll try to lower main.version in FastDraw and insert version checking code (to enable fix for old JOSM users).

I don't see any sense in that.

When they don't update josm, why do you think they update the plugin?

Also I'm pretty sure lowering main version will result in a big confusion in the plugin loading mechanisms and the related server tasks :-)

comment:17 by akks, 12 years ago

I communicated to Trolleway , the only author of edits with JOSM 5836 in http://toolserver.org/~mapjedi/blame.txt
He was using FastDraw but did not update it :)

I have also sent messages to Alex Rollin and Ikks .

comment:18 by Oli-Wan, 12 years ago

@akks: Seems like you were right then, which is good news as there is no indication to any flaws in JOSM's core. Given that the average mapper uses a JOSM version which is three months old (and most hopefully update their plugins along with the core), we should see some improvement over the next few months, i.e. fewer JOSM-related entries in the blame file -- especially if you keep nagging the most notorious creators of buggy ways to update their JOSM+plugins ;) I just added another column to the blame file (new entries) to make that a tiny bit easier.

Btw. if http://toolserver.org/~mapjedi/blame.txt returns 404 once in a while, try again after a few minutes. There happen to be some issues with the web servers in the Wikemedia Toolserver cluster at the moment. The file is there, believe me ;)

follow-ups: 21 24 comment:19 by Don-vip, 12 years ago

way	217762635	v1	2013-04-17T13:05:18Z	15762570	ArcGIS Editor for OpenStreetMap (2.1)	handerson
way	217762633	v1	2013-04-17T13:05:17Z	15762570	ArcGIS Editor for OpenStreetMap (2.1)	handerson
way	217762632	v1	2013-04-17T13:05:15Z	15762570	ArcGIS Editor for OpenStreetMap (2.1)	handerson
--------------------------------------------------------------------------------------------------------------------------
way	217857571	v1	2013-04-18T13:04:50Z	15773250	ArcGIS Editor for OpenStreetMap (2.1)	handerson
way	217857570	v1	2013-04-18T13:04:49Z	15773250	ArcGIS Editor for OpenStreetMap (2.1)	handerson
way	217857569	v1	2013-04-18T13:04:47Z	15773250	ArcGIS Editor for OpenStreetMap (2.1)	handerson

We should file a bug to ArcGIS mates :)

EDIT: bug filed: https://esriosmeditor.codeplex.com/workitem/11724

Last edited 12 years ago by Don-vip (previous) (diff)

comment:20 by skyper, 12 years ago

Still wonder why all the josm user are not fixing errors before uploading.

How about an additional warning for errors and/or always running error check (e.g. only disable warnings and infos).

in reply to: 19 ; follow-up: 22 comment:21 by Oli-Wan, 12 years ago

Replying to Don-vip:

We should file a bug to ArcGIS mates :)

Or ask its user (singular) to switch to JOSM.
Anyway, thanks for filing the bug report. I did not feel like signing up for yet another issue tracker after writing bug reports for Vespucci (fixed - within six hours - in the upcoming release) and iD (partially fixed). Guess I should write another one for "Go Map!!".

Is it possible to require a minimum version of certain plugins, i.e. automatically disable FastDraw if its version number is less than x? That would provide a smoking gun in case of another bug in the core: if JOSM revision y (the one in which the minimum version requirement was introduced) or higher appears in the blame file, there must be something broken elsewhere than in FastDraw. Right now one has to query the respective user about his plugin usage.

Replying to skyper:

Still wonder why all the josm user are not fixing errors before uploading.

Maybe they don't understand, or don't care? JOSM generates warnings for various other, not so fundamental errors - but people go ahead with their uploads anyway.

in reply to: 21 ; follow-up: 27 comment:22 by stoecker, 12 years ago

Replying to Oli-Wan:

Is it possible to require a minimum version of certain plugins, i.e. automatically disable FastDraw if its version number is less than x? That would provide a smoking gun in case of another bug in the core: if JOSM revision y (the one in which the minimum version requirement was introduced) or higher appears in the blame file, there must be something broken elsewhere than in FastDraw. Right now one has to query the respective user about his plugin usage.

It would be possible, but instead of that we implemented the auto-update system to keep plugins recent. But when user disable that they will have old plugins.

Replying to skyper:

Still wonder why all the josm user are not fixing errors before uploading.

Maybe they don't understand, or don't care? JOSM generates warnings for various other, not so fundamental errors - but people go ahead with their uploads anyway.

Actually the upload texts also says so: "When in doubt ignore them." (the errors). We decided that data with little errors is better than no data at all.

comment:23 by akks, 12 years ago

It seems that the main problem is very old JOSM version (older than 3 months). The patched plugins usually need newer JOSM and there is no possibility to patch "older" plugin version. I can "downgrade" FastDraw or compile special version for older JOSM (like 4xxx) in minutes, but there is no correct way to publish it.

However, in recent section http://toolserver.org/~mapjedi/blame.txt there seems to be more incorrect unchecked building imports, than FastDraw buggy ways.

@stoecker : or there is a possibility to replace old version of JAR file?

in reply to: 19 ; follow-up: 26 comment:24 by Thomas Emge, 12 years ago

I would like to investigate those duplicate nodes but the ways (id: 15762570, 15773250)were created before the ArcGIS Editor for OpenStreetMap existed. Certainly way before Version 2.1.

Thomas

Replying to Don-vip:

way	217762635	v1	2013-04-17T13:05:18Z	15762570	ArcGIS Editor for OpenStreetMap (2.1)	handerson
way	217762633	v1	2013-04-17T13:05:17Z	15762570	ArcGIS Editor for OpenStreetMap (2.1)	handerson
way	217762632	v1	2013-04-17T13:05:15Z	15762570	ArcGIS Editor for OpenStreetMap (2.1)	handerson
--------------------------------------------------------------------------------------------------------------------------
way	217857571	v1	2013-04-18T13:04:50Z	15773250	ArcGIS Editor for OpenStreetMap (2.1)	handerson
way	217857570	v1	2013-04-18T13:04:49Z	15773250	ArcGIS Editor for OpenStreetMap (2.1)	handerson
way	217857569	v1	2013-04-18T13:04:47Z	15773250	ArcGIS Editor for OpenStreetMap (2.1)	handerson

We should file a bug to ArcGIS mates :)

EDIT: bug filed: https://esriosmeditor.codeplex.com/workitem/11724

comment:25 by stoecker, 12 years ago

It seems that the main problem is very old JOSM version (older than 3 months). The patched plugins usually need newer JOSM and there is no possibility to patch "older" plugin version. I can "downgrade" FastDraw or compile special version for older JOSM (like 4xxx) in minutes, but there is no correct way to publish it.

As said. I don't believe that there will be a real benefit in updating old plugins, when core is not updated. That is also the reason why there is no mechanism for that. Managing multiple version takes a lot of effort for little benefit. JOSMs release model neither encourages nor requires maintaining of old components.

in reply to: 24 comment:26 by Thomas Emge, 12 years ago

Ha, I mixed up user id and way id columns.

We'll take a look at the underlying issue.

Thomas

Replying to Thomas Emge:

I would like to investigate those duplicate nodes but the ways (id: 15762570, 15773250)were created before the ArcGIS Editor for OpenStreetMap existed. Certainly way before Version 2.1.

Thomas

Replying to Don-vip:
way	217762635	v1	2013-04-17T13:05:18Z	15762570	ArcGIS Editor for OpenStreetMap (2.1)	handerson
way	217762633	v1	2013-04-17T13:05:17Z	15762570	ArcGIS Editor for OpenStreetMap (2.1)	handerson
way	217762632	v1	2013-04-17T13:05:15Z	15762570	ArcGIS Editor for OpenStreetMap (2.1)	handerson
--------------------------------------------------------------------------------------------------------------------------
way	217857571	v1	2013-04-18T13:04:50Z	15773250	ArcGIS Editor for OpenStreetMap (2.1)	handerson
way	217857570	v1	2013-04-18T13:04:49Z	15773250	ArcGIS Editor for OpenStreetMap (2.1)	handerson
way	217857569	v1	2013-04-18T13:04:47Z	15773250	ArcGIS Editor for OpenStreetMap (2.1)	handerson
We should file a bug to ArcGIS mates :)

EDIT: bug filed: https://esriosmeditor.codeplex.com/workitem/11724

in reply to: 22 ; follow-up: 28 comment:27 by Oli-Wan, 12 years ago

Replying to stoecker:

Replying to Oli-Wan:

Is it possible to require a minimum version of certain plugins, i.e. automatically disable FastDraw if its version number is less than x? [...]

It would be possible, but instead of that we implemented the auto-update system to keep plugins recent. But when user disable that they will have old plugins.

Yes, and I do not doubt that this was a wise decision. However, in a case such as the one at hand, where older versions of a given plugin are known to be seriously flawed a more imperative approach towards "encouraging" users to upgrade also their plugins would make sense in my view.

in reply to: 27 comment:28 by stoecker, 12 years ago

Replying to Oli-Wan:

However, in a case such as the one at hand, where older versions of a given plugin are known to be seriously flawed a more imperative approach towards "encouraging" users to upgrade also their plugins would make sense in my view.

Actually this is no serious flaw. If it would be, the API would simply reject such ways. We had serious issues in the past and got an API blocks for some JOSM versions. This is only a little problem.

And too much force for the user is not recommended. We already have a very good update rate. Others would be happy to have.

follow-up: 39 comment:29 by xeen, 12 years ago

I regularly create tags with duplicated nodes, but fix them before uploading. I’m using a combination of building tools, rectify, join nodes to ways, the extrude feature, the turn function, merge nodes, split/unglue nodes, “certain angle only drawing” (i.e. tab key), undo and probably a few others from time to time. I haven’t yet been able to narrow this down because I only ever validate at the end of a “session”. The main reason for this is that the validator runs on all data when checking manually, but only on my changes when uploading. Is there a way to make it “live”, i.e. check every 30s or so? This would make it much easier to find the reason. After adding buildings for a whole city block, I usually don’t remember what I did exactly for those two buildings – at least, it’s usually two adjacent buildings where I manage to create dup nodes.

follow-ups: 31 32 comment:30 by Oli-Wan, 12 years ago

I would assume that this user, who only signed up and made his first edits on April 19, was not using an ancient version of FastDraw:

way	219444918	v1	2013-04-29T19:46:13Z	15913064	JOSM/1.5 (5836 fr) Linux	similia joseph

Similar for these:

way	219431102	v1	2013-04-29T17:35:12Z	15911300	JOSM/1.5 (5836 fr) Linux	Rodney Simeon
way	219413381	v1	2013-04-29T15:09:41Z	15909516	JOSM/1.5 (5836 fr) Linux	garcon frantz
way	219170236	v1	2013-04-28T02:16:12Z	15890220	JOSM/1.5 (5836 en) Windows XP	lamonganscout
way	219113302	v1	2013-04-27T13:58:42Z	15884086	JOSM/1.5 (5836 en) Windows XP	gusti putu

I am inclined to conclude that, unfortunately, there must be sources of repeated nodes other than the FastDraw plugin.

@xeen: I think it would be very much appreciated if you did make the effort of narrowing down which function or plugin those broken ways (probably) originate from. I myself have never received any warning from the Validator about repeated nodes in ways of my own.

in reply to: 30 comment:31 by skyper, 12 years ago

Replying to Oli-Wan:

I am inclined to conclude that, unfortunately, there must be sources of repeated nodes other than the FastDraw plugin.

@xeen: I think it would be very much appreciated if you did make the effort of narrowing down which function or plugin those broken ways (probably) originate from. I myself have never received any warning from the Validator about repeated nodes in ways of my own.

+1

I do not use building tool though.

in reply to: 30 comment:32 by akks, 12 years ago

Replying to Oli-Wan:

I am inclined to conclude that, unfortunately, there must be sources of repeated nodes other than the FastDraw plugin.

Then, I think, it is Angle Snapping. Because of the same author of code :)

comment:33 by xeen, 12 years ago

Yes, it appears to be angle snapping. After akks comment I tried using an empty file using angle snapping only and was able to create a way with two duplicated nodes – but only once. I have not managed to find a way to reproduce this.

comment:34 by xeen, 12 years ago

(to be more precise, it was two ways that shared some of their nodes, i.e. two adjacent polygons)

follow-up: 37 comment:35 by xeen, 12 years ago

okay, here is my theory: the angle snapping tool creates duplicates nodes iff it would like to draw at an angle, but the “merge into nearby nodes” code overrules AST’s decision. I’ll attach three .osm files shortly where I managed to reproduce this behaviour. Unfortunately, I can’t provide exact steps to reproduce.

by xeen, 12 years ago

Attachment:	dup_nodes.zip added

comment:36 by rickmastfan67, 12 years ago

I just wanted to make a comment here that a few weeks ago, I was able to make duplicated nodes in the same place once. I never did upload it as I caught it before upload. But I have never had the plugin "FastDraw" installed on my system.

in reply to: 35 comment:37 by akks, 12 years ago

Replying to xeen:

okay, here is my theory: the angle snapping tool creates duplicates nodes iff it would like to draw at an angle, but the “merge into nearby nodes” code overrules AST’s decision. I’ll attach three .osm files shortly where I managed to reproduce this behaviour. Unfortunately, I can’t provide exact steps to reproduce.

Thank you, I managed to reproduce it one time...
Angle snapping when drawing near existing line tries to make angles strictly 90 (i.e.) that leads to very near points even if they not coincide. My theory is that they are inserted in nearby lines and make duplicates in some cases.
There are 2 solutions (may be combined)

Give nearest point under cursor priority to angle snapping (=> no more near points, but the angles vary a little)
When inserting into nearby segments, skip inserting into segment with near points.

comment:38 by akks, 12 years ago

Forgot to add "see #8591":
In [5922/josm]:

Avoid duplicate and very near points in Angle snapping draw mode
(will reuse existing nodes in mappaint.node.snap-distance pixel radius if possible)

Tolerance was 0.1 mm, now it is mappaint.node.snap-distance converted form pixels to meters. This should solve most of the cases of creating extremely near or duplicate nodes (and make harder drawing exact rectangles near existing lines, but who needs it?).

Last edited 12 years ago by akks (previous) (diff)

in reply to: 29 comment:39 by stoecker, 12 years ago

Replying to xeen:

The main reason for this is that the validator runs on all data when checking manually, but only on my changes when uploading. Is there a way to make it “live”, i.e. check every 30s or so?

There is no auto-check, but you simply can simulate upload behavior. Use "Select Modified" from first menu and then run validator. Validator runs on current selection or all data if nothing is selected.

comment:40 by skyper, 12 years ago

Still not fixed ? So iD is taking first place !

way	220772954	v1	2013-05-10T17:44:22Z	16068383	JOSM/1.5 (5939 de) Windows XP 32-Bit	ruph
way	220772948	v1	2013-05-10T17:44:22Z	16068383	JOSM/1.5 (5939 de) Windows XP 32-Bit	ruph
way	220772947	v1	2013-05-10T17:44:22Z	16068383	JOSM/1.5 (5939 de) Windows XP 32-Bit	ruph
way	148738309	v4	2013-05-10T18:34:10Z	16069141	JOSM/1.5 (5939 en) Linux Ubuntu 12.10	Nachtspazz
way	221250418	v1	2013-05-13T20:50:41Z	16116972	JOSM/1.5 (5939 de) Windows XP 32-Bit	ruph
way	221250417	v1	2013-05-13T20:50:41Z	16116972	JOSM/1.5 (5939 de) Windows XP 32-Bit	ruph
way	221181752	v3	2013-05-13T19:15:05Z	16115157	JOSM/1.5 (5939 pl) Windows XP 32-Bit	benek

follow-up: 42 comment:41 by skyper, 12 years ago

You are still able to upload these kind of ways if you import files in JOSM. This is what this user did. With the help of a spanish speaking user I got into contact with him.

Maybe we need some consistency checks and warnings when opening files.

in reply to: 41 comment:42 by Oli-Wan, 12 years ago

Replying to skyper:

You are still able to upload these kind of ways if you import files in JOSM. This is what this user did. With the help of a spanish speaking user I got into contact with him.

Maybe we need some consistency checks and warnings when opening files.

I on't think that yet another warning message will help much. I just saved a broken way in an .osm file, inverted the way's object id (text editor), then opened the file in JOSM to upload the way. On upload, the validator gives a warning, as it should. If people ignore this one (even when it appears on 1304 ways as in the case of jsuarezcarbajal's import), they will also ignore a warning that is shown earlier. Auto-correcting this special type of error on opening files and/or before upload might be a better choice. Or simply refuse to open such a file, as JOSM does when other errors are found.

The validator warning pops up only if the nodes are loaded as well; otherwise (i.e. if the file contains only the way itself) it will not. (Changing this behaviour won't solve the current problem either.)

follow-up: 45 comment:43 by rickmastfan67, 12 years ago

I've figured out just about a full proof way to create identical nodes in a single way!! Just happened to me tonight again and I noticed it right away and was able to pretty much continuously duplicate it after that.

Here's how to do it. I've been able to duplicate it almost always on command as long as I don't shift the location of my mouse while following the steps below.

Start drawing a way (A) while holding down the Ctrl button (do not let go of this button till I say so otherwise, you will not get this to happen!!) to prevent the way from connecting to another.
Now after you left click to create a new node, hit the right click button on your mouse. (DO NOT MOVE THE MOUSE AFTER YOU LEFT CLICK)
Hit your "ESC" button (For some reason, Window's start menu pops up the first time you hit ESC while holding down the Ctrl button but just ignore this for now).
Now, left click once again in the JOSM screen.
Now, move the mouse and left click again to create another node and release the CTRL button.
Now, hit the "s" button and select the area you were when you hit he "ESC" button.

What happens:
There are now two identical nodes in the same place in the same way.

Repository Root: http://josm.openstreetmap.de/svn
Build-Date: 2013-06-22 01:35:22
Last Changed Author: Don-vip
Revision: 6016
Repository UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
URL: http://josm.openstreetmap.de/svn/trunk
Last Changed Date: 2013-06-21 09:43:29 +0200 (Fri, 21 Jun 2013)
Last Changed Rev: 6016

Identification: JOSM/1.5 (6016 en) Windows 7 64-Bit
Memory Usage: 196 MB / 2730 MB (24 MB allocated, but free)
Java version: 1.7.0_25, Oracle Corporation, Java HotSpot(TM) 64-Bit Server VM
VM arguments: [-Xmx3072M]
Dataset consistency test: No problems found

Plugin: ImageryCache (29690)
Plugin: OpeningHoursEditor (29435)
Plugin: buildings_tools (29596)
Plugin: mapdust (29525)
Plugin: measurement (29625)
Plugin: mirrored_download (29643)
Plugin: openstreetbugs (29435)
Plugin: osmarender (29639)
Plugin: reverter (29663)
Plugin: turnrestrictions (29435)
Plugin: undelete (29555)
Plugin: utilsplugin2 (29672)

Last edited 12 years ago by rickmastfan67 (previous) (diff)

comment:44 by akks, 12 years ago

In 6027/josm:

see #8591: do not allow creating ways with duplicate nodes in Draw mode, even with Ctrl pressed

in reply to: 43 comment:45 by akks, 12 years ago

Replying to rickmastfan67:

I've figured out just about a full proof way to create identical nodes in a single way!! Just happened to me tonight again and I noticed it right away and was able to pretty much continuously duplicate it after that.

Thank you!
I tried to fix the problem "globally" by reusing nodes of selected way even when ctrl is pressed (only nodes from other nodes will be ignored).

follow-up: 47 comment:46 by Oli-Wan, 11 years ago

Just a few words of reassurance here. Looking at http://toolserver.org/~mapjedi/blame.txt you see that although several loopholes have been closed, even recent JOSM versions still show up, proving that there are still problems. So you might think that little actual progress has been made.

That discouraging impression is wrong. Obviously, there are still bugs present; but the efforts made so far have already lead to a significant reduction in the creation of broken ways. Let me back up this claim by a few numbers.

Consider the number of users involved in creating broken ways each day. Under a few reasonable assumptions, this number should (and in fact does) nicely follow a Poisson distribution, and therefore lends itself better to statistical analysis than the total number of broken ways created within one day (whose distribution is unknown and easily distorted by someone like jsuarezcarbajal creating some 4200 broken ways in a couple of days). The average numbers are:

JOSM, 4th quarter 2012: 1.80 ± 0.14
JOSM, 1st quarter 2013: 1.83 ± 0.15
JOSM, 2nd quarter 2013: 1.43 ± 0.14
JOSM, July 01-22, 2013: 0.81 ± 0.15

The decrease is obvious, and it is even statistically significant. So thanks for your efforts so far.
To find the remaining bugs, I would suggest to continue contacting users who create broken ways on a regular basis, like Vítor Dias. Someone here speaking Portuguese?

in reply to: 46 ; follow-up: 48 comment:47 by skyper, 11 years ago

Replying to Oli-Wan:

[...]

The decrease is obvious, and it is even statistically significant. So thanks for your efforts so far.
To find the remaining bugs, I would suggest to continue contacting users who create broken ways on a regular basis, like Vítor Dias. Someone here speaking Portuguese?

Thanks for the numbers. Maybe, we should once more point developers and other active mapper to this list, as speaking about JOSM more and more often the errors are made by the user and not the software.

I know that some people speaking Portuguese are listening at talk@osm.

@akks:
FMS seems to be speaking Russian.

in reply to: 47 comment:48 by akks, 11 years ago

I know that some people speakin
@akks:
FMS seems to be speaking Russian.

FMS was a well-known sociopate, I doubt someone can get any non-automatical trolling answer from him :) His sources are random but mostly legal. I guess it is some sort of dirty import again.

comment:49 by Oli-Wan, 11 years ago

I have contacted one of the JOSM users recently involved in creating a broken way (using an up-to-date version of JOSM) and received a very precise description that allows to reproduce the specific incident, as the user fortunately remembered the procedure in full detail. This is him:

way	231868995	v1	2013-07-30T21:54:50Z	17159087	JOSM/1.5 (6060 de)	XM-Franz

And here is his description: http://www.ruether24.de/JOSM/Beschreibung.pdf - in de, so for those who don't speak German I'll summarize it below.

Let me first introduce some terminology. Divide the creation of broken ways into two classes: primary and secondary creation, where primary means creating a broken way from scratch, while secondary means that there existed a broken way beforehand, which was then edited and split up, in the course of which a new broken way was created.

It turns out that the incident in question is of the latter type, resulting from a broken way previously created by anotheruser (and a different editor):

way	230951423	v1	2013-07-23T10:26:38Z	17060113	iD 1.0.1	spaghettieis

Way 231868995 was then created by splitting up way 230951423; this should have been caught by JOSM. While we are concerned only with secondary creation here, my hope is that primary and secondary creation can be prevented by one single safety measure in the code.

To reproduce the erroneus behaviour, you will want to work with OSM data from before the edits by XM-Franz. Here you are: https://toolserver.org/~mapjedi/duisburg.osm . This is the corresponding region taken from a Geofabrik extract of the night before.

Here's the summary of XM-Franz' description. Please have a look at the images in his file.

Starting point was an error message in keep right about a way containing more than one node several times. So zoom to way 230951423. It contains a loop between node 2393749427 and 2393749429 (i.e. between the stairs and the bridge), and also node 2393749427 twice in a row. Then using the crosshair on the longer segment, pull apart the double-line to obtain a triangular loop. (This step is not necessary to reproduce the broken way, but part of XM-Franz' description - and it helps to see things a bit more clearly.)

Now select nodes 2393749429 and 2393749427, split ("p") the way there. You will obtain a total of four ways (which is already a bit strange given that we're splitting at only two nodes), one of which contains only node 2393749427, but twice. (You can find it most easily by opening a rectangle selection box around that particular node.)

If you hit the upload button, the validator will complain; i.e. the error is detected.

Since no plugins are involved (XM-Franz even explicitly uninstalled all plugins on his machine to be sure), this time the issue is somewhere in the core.

comment:50 by akks, 11 years ago

Wow, thank you very much for you help! Give our thanks to XM-Franz too when it will be fixed :)
However, it was already double node... We should make splitting the line more smart, I guess.

Last edited 11 years ago by akks (previous) (diff)

comment:51 by akks, 11 years ago

I cannot fix it myself reliably enough (testing is near).

comment:52 by Oli-Wan, 11 years ago

While I still hope that the issue sketched in comment:49 can be addressed, let me come back to another problem. There seem to be tools around that generate OSM files that contain broken ways. People then open these files in JOSM and upload their data. JOSM will complain about the broken ways, but if people don't cancel the upload and actively hit the repair button, the broken ways will make their way into the database.

One example of those buggy third-party tools appears to be "Global Mapper 13.0", using which user Jestr88 recently created 112 broken ways; the 4208 broken ways by user jsuarezcarbajal have also been said to originate from some external tool.

I would like to repeat the suggestion that JOSM be more rigid towards such low-level data errors when working with files. Those errors should not end up in the database, no matter what. A warning message does not seem to suffice in this case: people won't understand it, or they won't know how to fix the problem (i.e. tell JOSM to do this automagically). So JOSM should either auto-fix any broken ways with a negative ID already in the process of opening the file, or apply an auto-fix to anything that is going to be uploaded.

(I am not suggesting to enforce *all* validator issues - the validator is wrong every once in a while. But the specific case discussed here is *always* an error.)

Reported by:	Oli-Wan	Owned by:	team
Priority:	normal	Milestone:
Component:	Core	Version:
Keywords:		Cc:

Summary:
Type:		Priority:
Milestone:		Component:
Version:		Keywords:
Cc:	Set your email in Preferences

Context Navigation

#8591 new defect

consecutive identical nodes in ways

Description

Attachments (2)

Change History (54)

follow-up: 2 comment:1 by Don-vip, 12 years ago

in reply to: 1 ; follow-up: 3 comment:2 by skyper, 12 years ago

in reply to: 2 comment:3 by skyper, 12 years ago

comment:4 by skyper, 12 years ago

comment:5 by Oli-Wan, 12 years ago

comment:6 by Oli-Wan, 12 years ago

comment:7 by stoecker, 12 years ago

follow-up: 9 comment:8 by Oli-Wan, 12 years ago

in reply to: 8 comment:9 by Don-vip, 12 years ago

comment:10 by akks, 12 years ago

by Don-vip, 12 years ago

comment:11 by Don-vip, 12 years ago

comment:12 by Oli-Wan, 12 years ago

comment:13 by akks, 12 years ago

comment:14 by Oli-Wan, 12 years ago

follow-up: 16 comment:15 by akks, 12 years ago

in reply to: 15 comment:16 by stoecker, 12 years ago

comment:17 by akks, 12 years ago

comment:18 by Oli-Wan, 12 years ago

follow-ups: 21 24 comment:19 by Don-vip, 12 years ago

comment:20 by skyper, 12 years ago

in reply to: 19 ; follow-up: 22 comment:21 by Oli-Wan, 12 years ago

in reply to: 21 ; follow-up: 27 comment:22 by stoecker, 12 years ago

comment:23 by akks, 12 years ago

in reply to: 19 ; follow-up: 26 comment:24 by Thomas Emge, 12 years ago

comment:25 by stoecker, 12 years ago

in reply to: 24 comment:26 by Thomas Emge, 12 years ago

in reply to: 22 ; follow-up: 28 comment:27 by Oli-Wan, 12 years ago

in reply to: 27 comment:28 by stoecker, 12 years ago

follow-up: 39 comment:29 by xeen, 12 years ago

follow-ups: 31 32 comment:30 by Oli-Wan, 12 years ago

in reply to: 30 comment:31 by skyper, 12 years ago

in reply to: 30 comment:32 by akks, 12 years ago

comment:33 by xeen, 12 years ago

comment:34 by xeen, 12 years ago

follow-up: 37 comment:35 by xeen, 12 years ago

by xeen, 12 years ago

comment:36 by rickmastfan67, 12 years ago

in reply to: 35 comment:37 by akks, 12 years ago

comment:38 by akks, 12 years ago

in reply to: 29 comment:39 by stoecker, 12 years ago

comment:40 by skyper, 12 years ago

follow-up: 42 comment:41 by skyper, 12 years ago

in reply to: 41 comment:42 by Oli-Wan, 12 years ago

follow-up: 45 comment:43 by rickmastfan67, 12 years ago

comment:44 by akks, 12 years ago

in reply to: 43 comment:45 by akks, 12 years ago

follow-up: 47 comment:46 by Oli-Wan, 11 years ago

in reply to: 46 ; follow-up: 48 comment:47 by skyper, 11 years ago

in reply to: 47 comment:48 by akks, 11 years ago

comment:49 by Oli-Wan, 11 years ago

comment:50 by akks, 11 years ago

comment:51 by akks, 11 years ago

comment:52 by Oli-Wan, 11 years ago

Modify Ticket

Add Comment

by anonymous

Download in other formats: