[PATCH] JOSM does not remember what has been uploaded

[Nakor]

Upload changes in several requests in the same changeset
During one of the requests the upload fails because of a deleted node still used in a way
Fix the issue and try to upload again

JOSM will try to re-upload the whole thing including what has already been uploaded, potentially leading for duplicates for new objects.

[Nakor]

Also upload takes forever after if you have some deleted elements in the set.

[Ldp]

I can confirm this behaviour. Had this happen to me tonight, with r2957. I had deleted a node still in a way, got the conflict, resolved it (by keeping the node around), then went to upload again.

My chunk size was set to 2000, and every time it encountered an already deleted element, wrote that fact in the upload window text box, and tried to reupload the rest (1999 elements) of the chunk. Until it hit the next deleted element, and it retried with 1998 elements, 1997, 1996. Ad infinitum, until you hit 1 element still to do, hours later.

[Gubaer]

Here's a "test script" to reproduce:

• Create a new node named new.1. Upload in a JOSM instance JOSM.1

• Launch another JOSM instance JOSM.2. Download the data uploaded above. Create a way including the node new.1. Upload.

• Switch to JOSM.1. Delete new.1. Create 3 additional new nodes.
  • Open the upload dialog
  • Go to "Advanced settings" and select "chunked upload mode" with a chunk size of 2
• Upload
  • JOSM uploads the first chunk with 2 new nodes - OK
  • JOSM upload the next chunk with 1 new nodes and 1 delete node - Conflict on the delete node
  • Click on Prepare Conflict Resolution. This creates a conflict. Resolve it by accepting "my version" (which is the delete new.1)

• Click on "Upload".

In the dialog you can see that it tries to upload 1 new nodes. That's fine. Because the second chunk in the former upload failed 1 new nodes still needs to be upload. Note that the first chunk succeeded and 2 of the 3 new nodes are already uploaded.

Result: According to this test case everything looks fine.

How does one have to tweak the test case to reproduce the problem?

[bastiK]

I experienced this problem at the end of January 2010. Now I cannot reproduce it anymore. There was a reimplementation of the API at the be beginning of February. Apparently there has been a change in behaviour: They dismiss a partial upload if there is an error in the progress. This solves the problem for JOSM!

[Gubaer]

Hmm, they should always have done because the promise was that an individual upload be transactional: it either succeeds or doesn't have any effect at all.

[bastiK]

Replying to Ldp:

I can confirm this behaviour. Had this happen to me tonight, with r2957. I had deleted a node still in a way, got the conflict, resolved it (by keeping the node around), then went to upload again.

@Ldp: I missed your post. Can you try to reproduce? For me it was chunk size like 2000, as well. Maybe it happens for large number of nodes, only?

@Gubaer: Somehow I managed to upload 3 and 4 versions of the very same nodes and the upload was canceled a view times due to conflicts.

[bastiK]

Replying to Ldp:

I can confirm this behaviour. Had this happen to me tonight, with r2957.

Can you please check if the server has incremented the version number of the nodes for each upload? (Although the properties of the nodes haven't changed.)

[Ldp]

I don't really remember the changeset, but most of the times this has happened to me, JOSM went on to upload new nodes or new ways, not increase the version. After I notice this happened, I open a fresh JOSM, download that area, and run the validator to get rid of the duplicate ways and/or nodes that I created. If it did increase the version number, there would not be such a problem, only an extra entry in the OSM history tables.

Also, if you manually cancel a chunk*, JOSM doesn't seem to know whether the server has applied that chunk correctly. It doesn't mark the objects in the chunk as uploaded, and goes to reupload them the next time. This also leads to duplicate objects in OSM. Is there a way to check on the server to see if the chunk was already processed?

• Which is sometimes very easy, if you forget your focus is on JOSM and you press space, since the Cancel button has focus. A confirmation dialog could come in handy.

[Gubaer]

Also, if you manually cancel a chunk*, JOSM doesn't seem to know whether the server has applied that chunk correctly. It doesn't mark the objects in the chunk as uploaded, and goes to reupload them the next time. This also leads to duplicate objects in OSM. Is there a way to check on the server to see if the chunk was already processed?

This could indeed be a problem, I'd say it's a very basic problem in the way the RESTful API is designed today. An indivual upload (also the upload of an individual chunk) today is a long running task with transactional behaviour. If the server replies an error then the client can be sure nothing in the upload made into the OSM database. I don't know, though, what happens if the client "cancels" the upload. There is no "canceling" of uploads in the spec anyway, JOSM uses a brute force approach and simply disconnects the open connection. In contrast to changsets, individual uploads have no identity. There are no API calls to check the state of an upload (ok, failed, interrupted, rolled back, etc.). Just running an "update" after a canelled upload doesn't help either because it wouldn't match new local primitives with already uploaded primitives.

I think that JOSM should display a warning message when an upload is canceled. The user should be warned that the outcome of the upload is unknown and that he should try what Ldp did: (1) update, (2) then run validator.

We should try to fix this in API 0.7. I'm afraid there isn't much we can do as long as we are working with API 0.6

[Ldp]

The way I understand it, if you drop the connection on a cancel, but the osmChange was already uploaded in full and JOSM was only waiting for the confirmation from the server, chances are high that the upload will still be processed in full on the server. These days there is a non-trivial amount of time spent in waiting for the results of an upload.

There are a few ways to mitigate the issue:

• Use a confirmation dialog when the user presses Cancel on an upload. This circumvents the problem where the Cancel was pressed in error.
• Offer to cancel immediately (the current behaviour) or after the current chunk finishes.
• After the upload was canceled in such a way, check (in the OSM db) one or a few of the objects in the upload, to see if they were indeed updated on the server.

[Gubaer]

The way I understand it, if you drop the connection on a cancel, but the osmChange was already
uploaded in full and JOSM was only waiting for the confirmation from the server, chances
are high that the upload will still be processed in full on the server.

Exactly.

After the upload was canceled in such a way, check (in the OSM db) one or a few of
the objects in the upload, to see if they were indeed updated on the server.

Note that this isn't reliably possible for new objects but we can apply heuristics:

• download a small bounding box around the new objects and check whether there is a semantically equivalent object at the same position which was last updated in the cancelled changeset (check the changeset attribute of the object)

• download the changeset content. This is possible for open and for closed changesets. Match the objects in the changeset with the objects in the local dataset using the same heuristics.

The second approach can be a waste of resources if you cancel the 49th chunk of 1000 objects in a changeset with 50'000 objects. Downloading and merging 50'000 object would take some time.

[Ldp]

Replying to Gubaer:

The way I understand it, if you drop the connection on a cancel, but the osmChange was already
uploaded in full and JOSM was only waiting for the confirmation from the server, chances
are high that the upload will still be processed in full on the server.

Exactly.

​http://wiki.openstreetmap.org/wiki/API_v0.6#Diff_upload:_POST_.2Fapi.2F0.6.2Fchangeset.2F.23id.2Fupload

"This is guaranteed to be running in a transaction. So either all the changes are applied or none."

"If a diff is successfully applied a XML (content type text/xml) is returned"

What is *not* documented is the desired behaviour when that response cannot be sent to the client, due to a closing of the connection. Either from a user cancelling the upload, or due to network issues. The whole issue presented in this ticket could be circumvented if the succesful sending of this XML response is a part of the criteria for the transactional behaviour of OsmChange processing on the OSM server.

The second approach can be a waste of resources if you cancel the 49th chunk of 1000 objects in a changeset with 50'000 objects. Downloading and merging 50'000 object would take some time.

Agreed about the resources, but it's an equal or larger waste of time and effort to fix such an upload by hand.

[Gubaer]

What is *not* documented is the desired behaviour when that response cannot be sent
to the client, due to a closing of the connection. Either from a user cancelling the upload,
or due to network issues. The whole issue presented in this ticket could be circumvented if
the succesful sending of this XML response is a part of the criteria for the transactional
behaviour of OsmChange processing on the OSM server.

IMHO, that's an unrealistic criteria which is unlikely to make it in API 0.7 (or any other "next generation" API). "Successful sending" is difficult to asses withouth another handshake between server and client.

I rather think that an upcoming API should (1) give uploads an identity and (2) should process uploads asynchronously.

• Client allocates an upload ID from the API.
• Client submits an upload for this upload ID. Connection is closed by the server after the data is read from the client but before processing starts.
• Client can poll for the state of an upload: (processing, canceled, OK, failed (inkl. verbose error message, also for multiple objects)).
• Client can try to cancel an upload with a given upload ID. Server returns immediatelly and asynchronously tries to abort and roll-back the open db transaction, if possible. Client polls the state of the upload again.

Anyway, there's no chance to get something changed on the server at short notice. If JOSM wants to offer "cancelling" of uploads it has to rely on workarounds and heuristics.

[bastiK]

Replying to Ldp:

"This is guaranteed to be running in a transaction. So either all the changes are applied or none."

I can confirm, that there are no partial uploads: Just tried to upload a large number of objects and canceled before it was done. (I have the wireshark log.)
The new changeset was opened successfully (as expected), and afterwards the changeset was empty.

So the remaining problem is cancellation while the diff from the server is received.

Replying to Gubaer:

• download a small bounding box around the new objects and check whether there is a semantically equivalent object at the same position which was last updated in the cancelled changeset (check the changeset attribute of the object)

• download the changeset content. This is possible for open and for closed changesets. Match the objects in the changeset with the objects in the local dataset using the same heuristics.

The second approach can be a waste of resources if you cancel the 49th chunk of 1000 objects in a changeset with 50'000 objects. Downloading and merging 50'000 object would take some time.

The first approach is not that easy, because some objects are not directly returned by a bbox request (see #o2699). I think it is ok to download the entire changeset in this exceptional situation.

[Gubaer]

I think it is ok to download the entire changeset in this exceptional situation.

To summarize, a possible workaround would be:

• Whenever an upload is canceled (that's the exceptional situation)
  • download the content of the changeset the cancelled upload was submitted to
  • foreach object o in the changeset, process it as it was an element in a diff document replied by the server (purge deleted objects; update version and technical attributes of updated objects, remove modified flag; set ID, version and technical attributes on new objects)

[fsteggink]

I just experienced a similar problem. I'm not sure if it exactly the same. I uploaded waterbodies / coastlines and wooded areas in Canada, using build 2971. The import covers this area: ​http://www.openstreetmap.org/?minlon=-71&minlat=47&maxlon=-70.5&maxlat=47.25&box=yes

Changeset 1: ​http://www.openstreetmap.org/browse/changeset/3916569
This contains waterbodies. By convention, I converted the waterbodies which are in the St. Lawrence river to coastlines, and tied them up to the existing coastline. I removed the original coastline (PGS). This upload went successfully.

Changeset 2: ​http://www.openstreetmap.org/browse/changeset/3916856
This contains wooded areas. It needed 7 chunks, with a chunk size of 15000. I merged the new features in the _existing_ OSM file (which I downloaded before I uploaded changeset 1). However, when the last chunk started uploading, I got the conflict dialog which Nakor and Ldp described. Because this takes a very long time (one minute or more?) per conflicting feature to upload, and I recognized the first conflicting node ID as one in the PGS daaset, and knew there were 791 nodes, I aborted the upload. Since Cancel didn't work, I killed JOSM. So, that is why this changeset contains 15k nodes only.

Then I checked the area on the server, ran the validator, and saw that there were exactly 15k unconnected nodes. So, at least those nodes are uploaded well. I tried to upload the rest from the existing OSM file, but there were those 791 coastlines nodes waiting to be deleted.

Changeset 3: ​http://www.openstreetmap.org/history?bbox=-71.11%2C47.01%2C-70.321%2C47.242
Then I made a copy of that file, removed the 791 PGS nodes, and uploaded again. This worked well, and I had no conflicts with this upload.

After the upload I downloaded the area again, and checked its contents. The wooded areas look to be OK in a visual check.

Hope this helps.

[fsteggink]

Oops, changeset 3 above is: ​http://www.openstreetmap.org/browse/changeset/3917054

[fsteggink]

In order to avoid confusion: during the first import, the removal of the coastline went OK. It is only _after_ I reused the same OSM file for the second import that I got these problems. Only the PGS nodes were marked as deleted, not the ways. (There weren't any other deleted objects.)

So, it seems to me that JOSM doesn't register well which nodes have been deleted.

Generally I think it is better to download the area again, in order to start with a clean slate.

[bastiK]

Is not a release blocker.

[pinkduck]

Re: DIFF Upload of OsmChange - "This is guaranteed to be running in a transaction. So either all the changes are applied or none." (API v0.6)

This is not the case currently. I have just tried to update >50,000 nodes with corrected positional data. Nearly all of them were at version 1 on the server before I began. Now after uploading in chunks of 500 and encountering separate synchronisation conflicts due to 4 single nodes being deleted on server (duplicate node fixes) I check the version numbers at the server and find that they range between 2 and 5, when I expect nearly all to be version 2.

The server history confirms that repeated uploads of each node took place and were assigned new version numbers even when the nodes were unchanged, for example:

​http://www.openstreetmap.org/browse/node/799751822/history

Is the API not transacting the OsmChange upload unit? Is JOSM incorrectly increasing the version numbers of elements in failed changeset chunks?

It looks as though I will have to use the 'Upload each object individually' advanced option to avoid such duplication. Thankfully in this case I was updating existing objects rather than creating new. The latter case leads to the less desirable situation of duplicate, identical nodes, ways and relations.

[bastiK]

Replying to pinkduck:

Re: DIFF Upload of OsmChange - "This is guaranteed to be running in a transaction. So either all the changes are applied or none." (API v0.6)

This is not the case currently.

That's wrong, it still is the case!

I have just tried to update >50,000 nodes with corrected positional data. Nearly all of them were at version 1 on the server before I began. Now after uploading in chunks of 500

If you choose chunked upload, this means JOSM picks 500 modified objects and uploads them in a single upload request (but leaves the changeset open). Then, in a completely different transaction, it uploads the next 500 objects and so on.

and encountering separate synchronisation conflicts due to 4 single nodes being deleted on server (duplicate node fixes) I check the version numbers at the server and find that they range between 2 and 5, when I expect nearly all to be version 2.

The server history confirms that repeated uploads of each node took place and were assigned new version numbers even when the nodes were unchanged, for example:

​http://www.openstreetmap.org/browse/node/799751822/history

Normally, this only happens if you

• edit
• save data layer
• upload (e.g. only the first chunkes of a chunked upload, or the entire change)
• close JOSM without saving the data layer again
• open that file again and upload

Another scenario:

• begin upload
• abort the upload process while it is almost finished (i.e. all data is send to the server, but the server response has not been received or processed).
• upload again

(This also happens if there are network problems. In these cases, just check in the browser if the changeset has been accepted or not.)

If you are sure, that nothing like this happened, it would be absolutely amazing and beneficial for the project if you could reconstruct and repeat the steps that led to this increased version numbers.

Is the API not transacting the OsmChange upload unit? Is JOSM incorrectly increasing the version numbers of elements in failed changeset chunks?

JOSM can not assign version numbers, this is done by the server exclusively. (JOSM sends a version to the server; if this version does not match the server version, the entire upload request is rejected.)

It looks as though I will have to use the 'Upload each object individually' advanced option to avoid such duplication.

This won't help as it is similar to chunked mode with number of objects per chunk is 1.

Thankfully in this case I was updating existing objects rather than creating new. The latter case leads to the less desirable situation of duplicate, identical nodes, ways and relations.

true

[pinkduck]

What I stated happened with a single open JOSM session, no saves between start and completion but 4 upload attempt retries where each involved fixing a conflicting node. I did not abort any transaction. The JOSM console log showed that an error conflict was the cause of each chunk upload failure. I chose the 'Synchronise this node only' option on the conflict dialogue displayed, then opted to use the server-side visibility (deleted) then attempted chunked upload again to the same changeset. The stats showing the remaining chunks to upload were correct throughout.

JOSM must update the version number locally as how else is the local state updated having made an upload so that the next OsmChange transaction will succeed?

Uploading with single objects per chunk does help because then either a single node succeeds or fails and there are no other upload objects to be duplicated when the single node upload is resumed after conflict resolution. If the server fails to respond to the single chunk upload request then yes, a conflict would occur on the retry (but it didn't this time).

To recreate I suggest creating eight new nodes in OSM. Then download them with JOSM and save as an OSM file. Then use JOSM to delete two of the nodes on the server. Then open up the saved file and alter the tags on each of the nodes. Then try to upload with a chunk size of two objects. That will cause the conflicts and the chunk size of two should reveal the same issue already reported.

[pinkduck]

I just followed my recreation instructions and the same problem occurred. Below is the console trace from JOSM:

Repository Root: ​http://josm.openstreetmap.de/svn
Build-Date: 2010-07-17 01:31:31
Last Changed Author: bastiK
Revision: 3376
Repository UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
URL: ​http://josm.openstreetmap.de/svn/trunk
Last Changed Date: 2010-07-16 23:44:00 +0200 (Fri, 16 Jul 2010)
Last Changed Rev: 3376

loading plugin 'wmsplugin' (version 22286)
loading plugin 'remotecontrol' (version 21706)
loading plugin 'buildings_tools' (version 22164)
RemoteControl::Accepting connections on port 8111
loading plugin 'validator' (version 22204)
loading plugin 'terracer' (version 22169)
GET ​http://api.openstreetmap.org/api/capabilities... OK
Communications with ​http://api.openstreetmap.org/api established using protocol
version 0.6
GET ​http://api.openstreetmap.org/api/0.6/map?bbox=1.2270033,52.6118172,1.2305331
,52.6136738
PUT ​http://api.openstreetmap.org/api/0.6/changeset/create... OK
POST ​http://api.openstreetmap.org/api/0.6/changeset/5374888/upload... OK
PUT ​http://api.openstreetmap.org/api/0.6/changeset/5374888/close... OK
GET ​http://api.openstreetmap.org/api/0.6/map?bbox=1.2270033,52.6118172,1.2305331
,52.6136738
Keystroke pressed D is already assigned to org.openstreetmap.josm.actions.mapmod
e.DeleteAction@17a82f1, will be overridden by org.openstreetmap.josm.actions.map
mode.DeleteAction@85b8d
Keystroke pressed D is already assigned to org.openstreetmap.josm.actions.mapmod
e.DeleteAction@85b8d, will be overridden by org.openstreetmap.josm.actions.mapmo
de.DeleteAction@ca677f
PUT ​http://api.openstreetmap.org/api/0.6/changeset/create... OK
POST ​http://api.openstreetmap.org/api/0.6/changeset/5374899/upload... OK
PUT ​http://api.openstreetmap.org/api/0.6/changeset/5374899/close... OK
Open file: C:\Documents and Settings\Pink Duck\My Documents\Downloads\JOS
MTest.osm (1308 bytes)
PUT ​http://api.openstreetmap.org/api/0.6/changeset/create... OK
POST ​http://api.openstreetmap.org/api/0.6/changeset/5374921/upload... OK
POST ​http://api.openstreetmap.org/api/0.6/changeset/5374921/upload... Conflict
Error header: Version mismatch: Provided 1, server had: 2 of Node 842467895
GET ​http://api.openstreetmap.org/api/0.6/nodes?nodes=842467895
1 conflict has been resolved.
1 conflict has been resolved.
PUT ​http://api.openstreetmap.org/api/0.6/changeset/5374921... OK
POST ​http://api.openstreetmap.org/api/0.6/changeset/5374921/upload... OK
POST ​http://api.openstreetmap.org/api/0.6/changeset/5374921/upload... Conflict
Error header: Version mismatch: Provided 1, server had: 2 of Node 842467893
GET ​http://api.openstreetmap.org/api/0.6/nodes?nodes=842467893
1 conflict has been resolved.
1 conflict has been resolved.
PUT ​http://api.openstreetmap.org/api/0.6/changeset/5374921... OK
POST ​http://api.openstreetmap.org/api/0.6/changeset/5374921/upload... OK
POST ​http://api.openstreetmap.org/api/0.6/changeset/5374921/upload... OK
POST ​http://api.openstreetmap.org/api/0.6/changeset/5374921/upload... OK
PUT ​http://api.openstreetmap.org/api/0.6/changeset/5374921/close... OK
GET ​http://api.openstreetmap.org/api/0.6/map?bbox=1.2270033,52.6118172,1.2305331
,52.6136738
GET ​http://api.openstreetmap.org/api/0.6/node/842467897/history

I first created 8 nodes, uploaded them in a single changeset (#5374888) and saved that as JOSMTest.osm. Then I delete two nodes and uploaded that change in another changeset (#5374899). Then I deleted the layer and reopened JOSMTest.osm and assigned a note=n property to each node then attempted the upload with chunk size of 2 objects in changeset #5374921. Two upload retries were needed after each node conflict sequence of opting to resolve by taking the server-side visibility state (deleted). On completion I deleted the layer, redownloaded and checked the history of the first node (which should have gone from version 1 to version 2) and saw history showing version 1, 2, 3 and 4.

I did this at lat/lng box:
52.6117994 1.2265402
52.6136916 1.2309962

[bastiK]

Yes, I can reproduce, thanks.

@Upliner: This directly affects the things that are touched by [3362] and #5221, can you have a look?

[Upliner]

The bug appears because processedPrimitives.addAll() isn't called if writer.uploadOsm() throws an exception. I've fixed it, but I think that current architecture is quite error-prone and there is some risk to leave the dataset in inconsistent state. I think it's better to clear the "modified" flag directly in DiffResultProcessor or OsmApi.

[stoecker]

(In [3422]) fix #4401 - upload error - patch by Upliner