Modify

Opened 14 years ago

Last modified 5 years ago

#5215 new defect

Automatically detect and fix situations, where server accepted upload and JOSM assumes upload failure

Reported by: dieterdreist Owned by: team
Priority: major Milestone:
Component: Core Version: latest
Keywords: double, changeset, duplicate, node, and, way, conflict, management, upload-api-response Cc: AM909

Description (last modified by stoecker)

Just discovered it is all double. It is around 6200 changes...
i don't know exactly which one it was: changesets
*5140491
*5140246
*5140090
*5139835
I set this to blocker because I expect other users having the same issues due to the downtime recently.

Attachments (2)

JOSM_error.png (10.2 KB ) - added by dieterdreist 14 years ago.
JOSM_error2.png (10.9 KB ) - added by dieterdreist 14 years ago.

Download all attachments as: .zip

Change History (24)

comment:1 by stoecker, 14 years ago

Owner: changed from team to dieterdreist
Status: newneedinfo

Probably your uploaded twice as you thought the first upload failed?

in reply to:  1 ; comment:2 by dieterdreist, 14 years ago

Owner: changed from dieterdreist to stoecker
Priority: blockercritical
Status: needinfonew

Replying to stoecker:

Probably your uploaded twice as you thought the first upload failed?

: yes, that's exactly what I did, but I thought that JOSM would prevent to upload twice. It also seemed to be a slightly different number of changes.

in reply to:  2 ; comment:3 by skyper, 14 years ago

Owner: changed from stoecker to dieterdreist
Status: newneedinfo

Replying to dieterdreist:

Replying to stoecker:

Probably your uploaded twice as you thought the first upload failed?

: yes, that's exactly what I did, but I thought that JOSM would prevent to upload twice. It also seemed to be a slightly different number of changes.

Did you update your data before the second upload ?

in reply to:  3 ; comment:4 by dieterdreist, 14 years ago

Replying to skyper:

Replying to dieterdreist:

Replying to stoecker:

Probably your uploaded twice as you thought the first upload failed?

: yes, that's exactly what I did, but I thought that JOSM would prevent to upload twice. It also seemed to be a slightly different number of changes.

Did you update your data before the second upload ?

Actually I think I failed and didn't because I wasn't aware that I should do (the area is almost empty and I didn't expect other mappers)

in reply to:  4 ; comment:5 by dieterdreist, 14 years ago

Owner: changed from dieterdreist to stoecker
Priority: criticalblocker
Status: needinfonew

Replying to dieterdreist:

Replying to skyper:

Replying to dieterdreist:

Replying to stoecker:

Probably your uploaded twice as you thought the first upload failed?

: yes, that's exactly what I did, but I thought that JOSM would prevent to upload twice. It also seemed to be a slightly different number of changes.

Did you update your data before the second upload ?

Actually I think I failed and didn't because I wasn't aware that I should do (the area is almost empty and I didn't expect other mappers)

It happened again! http://www.openstreetmap.org/browse/changeset/5227820
This time there was a conflict reported for one node on upload (this also happened above), but while above I clicked on "synchronize whole ..." now I clicked on "cancel" and did "update data" first. On Update I was reported that approx. 2200 elements seem to be "deleted" on the server (the ones I wanted to upload I guess, before the changeset didn't upload "check your internet connection"). After this update there was 1 conflict (multipolygon-relation, I took "my version" not "their version"). After the conflict was resolved, there were 48 duplicate nodes reported. I selected the error and then on fix. Afterwards tried to upload but: 306 duplicate nodes. Again fixed. Afterwards fixed the multipolygon (1 way was double, 648 vs 650 nodes). Now I am uploading again...

in reply to:  5 comment:6 by dieterdreist, 14 years ago

Keywords: duplicate node and way conflict management added

fixed. Afterwards fixed the multipolygon (1 way was double, 648 vs 650 nodes). Now I am uploading again...

There is a mean bug somewhere there. After upload I got 966 duplicated nodes. Resolved them and fixed an area with duplicated nodes. After this (did nothing else) validator reported 36 duplicated ways that weren't reported 10 seconds before (the only thing I did in the mean time was selecting the duplicated nodes of one element (which was reported as area feature not closed) and merge them).

comment:7 by AM909, 14 years ago

Possibly related, though I'm afraid I don't have as much detail as I'd like.

I downloaded some neighboring bounding boxes, then did some work on one of them and uploaded it. Then, I did some work on the next one, attempted to upload it (579 ways to be modified), and got a complaint about the server having a newer version of one of my ways. I chose to sync that way only and tried again. It complained about another way, so I repeated the process. I did this several more times before biting the bullet and choosing to "sync entire dataset", which took a long time (as expected).

At some point, I looked at the open changeset with http://www.openstreetmap.org/browse/changeset/5257539 and it showed that it contained 579 ways.

When I then tried to upload again, I noticed that it said there were only 567 ways to upload. It's entirely possible that the difference (12) is the number of times that I sync'd individual ways, though I can't confirm this.

After the upload succeeded, I saved the dataset, and looked at the changeset again, and it now shows 1146 ways (579+567). If I download the changeset XML, I see that 567 of the ways are listed twice, with two successive version numbers (e.g. http://www.openstreetmap.org/browse/way/11234037/history). The version number in the file I saved after the successful upload is the latest one (3 in the example).

Note that, in my case, all the edits are to tags on existing ways - no geometry changes and no object adds/deletes.

HTH

comment:8 by AM909, 14 years ago

Forgot to mention: this happened running r3329.

comment:9 by AM909, 14 years ago

Cc: AM909 added

in reply to:  9 ; comment:10 by dieterdreist, 14 years ago

Replying to Am909:
It happened again with 3376: Uploaded 1808 changes. Got a message that one way is conflicted. I canceled synchronisation: 11 duplicate nodes. I tried update data. 2 errors happened (see attachments).

by dieterdreist, 14 years ago

Attachment: JOSM_error.png added

by dieterdreist, 14 years ago

Attachment: JOSM_error2.png added

in reply to:  10 comment:11 by dieterdreist, 14 years ago

Replying to dieterdreist:

Replying to Am909:
It happened again with 3376: Uploaded 1808 changes. Got a message that one way is conflicted. I canceled synchronisation: 11 duplicate nodes. I tried update data. 2 errors happened (see attachments).

Please also note that the conflicting data seems to be more (ca. 1900) than the actual number of edits I tried to upload (1808).

comment:12 by stoecker, 14 years ago

Summary: conflict-management: JOSM doubled a complete huge changeset without advertingAutomatically detect and fix situations, where server accepted upload and JOSM assumes upload failure
Type: defectenhancement

NOTE: When you have a situation like above: JOSM thinks upload failed, but server accepted upload, then you should drop your whole local dataset and reload it from server (you may load it into a different layer to verify if it really is complete).

There is no need to do conflict resolution, as there aren't any conflicts. Conflict handling for new elements is rather nonexisting, so whatever you do it will produce incorrect results.

JOSM may detect this situation and properly handle it in future, but don't expect this soon.

comment:13 by stanton, 13 years ago

Yes, this is just the problem: I've had at least two of these cases in which something did get uploaded but I don't know how much. Edits to existing objects aren't a problem as the server will reject the new upload due to the version number not matching, but new objects will get created a second time. Verifying how many of these actually made it to the server is a tedious chore when the number of new objects is large - so I believe this case should get handled somewhere, even if only to prevent garbage (duplicate objects) from being uploaded to the server. Frankly, I believe this should go not in the editor but in the API as the "choke point" - I had opened ticket 3521 on OSM but it was closed as "wontfix".

We may need to discuss this among a wider audience - other editors may at times encounter similar issues (I remember about the TIGER import creating massive amounts of dupes), hence this potentially affects all editors and the API.

comment:14 by stoecker, 11 years ago

Ticket #7321 has been marked as a duplicate of this ticket.

comment:15 by stoecker, 11 years ago

Description: modified (diff)
Owner: changed from stoecker to team

comment:16 by skyper, 10 years ago

I am not sure if it was #5783 without exception or this bug but I just stumbled over an upload which started to repeat its conflicts where new objects on the server had already an id but JOSM had them still listed as new objects (id=0). All I know is that I had three real conflicts in the beginning (2 releations + one node still in use) and I was uploading ~1000 objects but with more than 100 BBOXs and always tried to upload to one changeset which is still open (changeset: 18559678).

I did save the data after I got more and more conflicts and did restart. After careful inspecting all modified object and downloading/merging I let validator auto fix all the duplicates I ended up without any modified object left and only some deletions to upload which had been uploaded to server already.

comment:17 by planemad, 8 years ago

A recent example of this issue causing a major data disaster in Uzbekistan: http://www.openstreetmap.org/user/PlaneMad/diary/35762

comment:18 by simon04, 8 years ago

Keywords: upload-api-response added

comment:19 by bastiK, 8 years ago

Ticket #12601 has been marked as a duplicate of this ticket.

comment:20 by bastiK, 8 years ago

Link to osm ticket menitioned above: #osm3521.

I think there are pragmatic ways to detect this situation. Most simple would be to download a tiny bounding box around one of the nodes to upload. If the server returns a node with the same coordinates and tags, issue a warning or perform more thorough checks.

When the changeset contains modifications or deletions, the server will reject duplicate upload, so this is only necessary for changesets with additions only.

comment:21 by bastiK, 8 years ago

Priority: blockermajor
Type: enhancementdefect

comment:22 by mmd, 5 years ago

We're currently discussing potential changes to the OSM API to make the changeset upload idempotent. It means that you could re-upload the same changeset after some network failure without risking duplicates on the database, and at the same time still receive the original diffResult message.

Please feel free to join the discussion on https://github.com/openstreetmap/openstreetmap-website/issues/2201

There's even a small prototype in that issue to demonstrate the use of an Idempotency-Key for JOSM changeset upload.

Modify Ticket

Change Properties
Set your email in Preferences
Action
as new The owner will remain team.
as The resolution will be set. Next status will be 'closed'.
to The owner will be changed from team to the specified user.
Next status will be 'needinfo'. The owner will be changed from team to dieterdreist.
as duplicate The resolution will be set to duplicate. Next status will be 'closed'. The specified ticket will be cross-referenced with this ticket.
The owner will be changed from team to anonymous. Next status will be 'assigned'.

Add Comment


E-mail address and name can be saved in the Preferences .
 
Note: See TracTickets for help on using tickets.