unexpected full recursive download of children relations makes some edits impossible

[verdyp@…]

something has changed in JOSM recently, causing it to be unable to resolve edit conflicts for local edits where a parent relation is concerned: the parent relation is now used to perform a FULL recursive download of all its children and subchildrens.
This is a major concern because this happens when trying to resolve an edit conflict: the download will take a considerable time when the parent category is a large and detailed country, causing JOSM trying to download millions of nodes, thousands subrelations, only to complete the dowland status of all of them. Additionallyt, in the 32-bit version you'll rapidly exhaust the memory capacity in the VM and JOSM will hang, error dialogs will not even open all unsaved edits will be lost, the edit conflict will not be solved; all we can do is to kill JOSM with a task manager. When relaunching JOSM, the unsaved edits will be reloaded, but the last submitted data will not be part of the restored data, and some objects that were already saved will be in the wrong state, causing more edit conflicts to resolve.
Now trying to complete only the relations deepest level and checking their update status and resolving these conflicts and submitting them again will not solve the problem for the parent relations.
Even outside the edit conflict editor, now if we have downloaded a relation that has incomplete members for their relation members, if we select that relation and click "download members" or "download incomplete members", both now perform a recursive download of subrelations and this never ends for large countries.

This is now a severely blocking behavior: we can no longer correctly, esily and speedily fix any edit conflicts, even with the 64-bit version of Java and a large VM (launch parameter: -mx8G). And this takes now several hours when we see the "downloading" dialog perform many, many download requests (and finally we get an error from the OSM API because we downloaded too much data in the current session).

Please restore the previous behavior: DO NOT recurse subrelations when downloading members, only recurse nodes of incomplete ways members and at most 1 level of recursion, do not download members referenced of member relations (only download their references: type, and id, possibly their tags like names).

[verdyp@…]

note: full recursive download of *parent* relations does not cause this issue
This occurs now when resolving local edit conflicts and transforms what was a local-only edit with a local-only edit conflict into a edit over a giant area (because it attempts to recurse on *all children* of parent relations; the recursion occurs in both directions)

[anonymous]

I don't know if this may be related to a recent change trying to fix bug #6529 (but I note that this occur because there were deleted items in the edit conflict, over a relation (that I did not edit directly) that was recently redacted and reverted but was part of the parent references and that had used some recently deleted objects (a way that was split and then unsplit by the revert).
For soem strange reasons, these objects that are alreadfy marked as deleted in the database can cause this behavior.

[verdyp@…]

See also old bug #4142 (containing its own, workaround, which no longer works as well) but can also trigger inconsistant data state. I think that edit conflicts now can cause such inconsistancies that forces JOSM to download much more than expected and force too many child-recursions (which in my opinion should still NEVER be needed; iD itself does not recurse all children that must be downloaded only manually on request, where iD still recurses successfully on all parent relations, without needing a manual CTRL+ALT+D like in JOSM to load referers).

[GerdP]

Might be a regression of #18566.
Please attach a status report and describe how to reproduce the problem, e.g. which relation you cannot download with members.

[anonymous]

Look at Spain (I did not want to edit it directly but some relations in France that I edited on a way fully in France, was part of a larger relation that has *other* ways on the international border with Spain; there was an edit conflict and now to solve it I need to change this international border, that has now holes, and it is impossible to resolve the conflict without downloading Spain completely and all its children; this requires running JOSM in 64-bit and use a giant VM size: now more than 12 Gigabytes; and downloading millions nodes in multiple requests over several days because the OSM API limits the data volume per user).This was caused by a revert caused by an administrative edit when someone changed the borders of Spanish Sovereign bases in the north of Morocco (this was reverted by the DWG, but then caused edit conflicts with every other non-Spanish relations touching Spain)

[GerdP]

Why do you download all members when that causes problems?

[verdyp@…]

You're right, it is definitely related to #18566: now the full recursive download is performed in all cases.
This is a very bad behavior. This full recursion on children should NEVER be applied: only one level recursion should occur.

The edit conflict resolver now uses the full recursion with this new behavior, and the manual "download members" action should only never recurse down more than 1 level.

If I use JOSM and not iD, it's specifically to avoid such recursions that make some edits impossible to do correctly in iD. iD never recurses down more than 1 level at a time, it requires a manual click on the download button of listed relation members of relations. This is the only safe behavior.

[skyper]

Thanks for your report, however your ticket is incomplete and therefore not helpful in its current form.

Please add all needed information according to this list:

• The required parts of the Status Report from your JOSM.
  • Please, use Report Bug from Help menu and copy & paste.
• Describe what behaviour you expected.
• Describe what did happen instead.
• Describe if and how the issue is reproducible.
• Add any relevant information like error messages or screenshots.

To ensure that all technical relevant information is contained, create new tickets by clicking in JOSMs Main Menu on Help → Report Bug.

Remember: This is a generic notice so we don't need to write the same stuff again and again. It may only apply in parts to the specific case!

How about a step by step description and attaching a small OSM data file to reproduce ?

You can also download older version to see download.

[anonymous]

@GerdP: I don't want to download ALL members. I just want the nodes of ways (1 level max of recursion) or the list of relation members without their own members. But the edit conflict resolver wants to download everything, and now has loaded unexpected relations that have their own conflicts even if I did not touch them.

[GerdP]

I do not yet understand where this full download is started. What do you do before "the edit conflict resolver wants to download everything"?

[GerdP]

Regarding Spain: The ways of relation 1311341 don't show a hole, I see several complete rings.

[GerdP]

I can reproduce the problem that the OSM api stops with "You have downloaded too much data" when you try to download all members of relation 1311341. I have no problems when I download only the way members of that relation. This is done in less than a minute and results in ~30.000 nodes.
I can also download the complete relation using overpass, this gives much more data (> 1.635.000 nodes) but still fits into a JOSM session started with -Xmx2G
I didn't find a button in the conflict dialog which would download a relation either.

[anonymous]

Also when I use "update selected objects", the update of just the relation attempts now to do a full recursive update. This is clearly a defect, it should jsut update the tags, the list of members, and the nodes of member ways, not the members of child relations.

[anonymous]

There was a hole in Spain, I could find it but it was fixed by someone else while I attempted to solve it (and could not do it at all for several days with JOSM, making now thousands API requests on all administrative areas at all admin_levels, even in areas not concerned by the changes).
This is a major defect with the conflict resolver, because this can no longer be done in many cases where there are some ways shared by may parent relations having many recursive children (recursing upward to the parents is not a problem)

[GerdP]

I still don't understand how the conflict resolver is involved here. Please attach a screen shot that helps me to understand what is going on.

[anonymous]

updating members never ends (thousands API calls, filling the Java resources cache on disk, allocating several gigabytes of memory until JVM freezes and the GC fails with a fatal error)

[GerdP]

No need to repeat your problem, that is clearly understood. The screenshot doesn't show a conflict dialog, so you started the "download members" manually? In that case we just have to change the recursion as already agreed in ticket:18566#comment:36.

[anonymous]

I don't know why the conflict resolver wanted to download everything, it may be related to the fact that when I performed a local edit that modified one member in the Spanish relation, that relation was reverted administratively from another edit made elsewhere in Spain by someone else a few days before. There's some triggering condition (inconsistant datetimes caused by the adminsitrative revert? hidden/redacted edits in the OSM database?) that triggers downloading everything (even objects that were not loaded before and not involved at all in my edit). This was triggered by a single conflict on that relation on a single way that was splitted or merged by another user or adminsitrator.

I have then passed a lot of time (after the JVM crashed) to check many objects that were submitted halfway but not autosaved in the .osm file with their commited status, then I had to play with many edit conflicts and the last one I could not solve was on the country relation itself.
I could not use JOSM to do that check for a couple of days, then another user could repair that hole that I should have been able to repair... if JOSM had correctly loaded the country relation and only this one but not all the thousands subrelations everywhere in the country.

Note: I cannot reproduce it now, but my .osm file (which was initially very small with a few kilobytes, exploded to more than 4 GB in the last autosaved file; there's still some edits in it that were not submitted, and the data is now inconsistant because the small edit was partly submitted, but I have difficulty to find my real edits in the tons of data added by the recurive download.

anyway, we should have a way to dismiss the "downloading" dialog. But pressing the "Cancel" button or the [X] close box has no effect: the dialog closes but reopens instantly, as JOSM continues recursing to perform a long list of additional downloads (the green progress bar continues to show a fast 100% completion for each of them, falshing constantly)

[GerdP]

Planned changes:

• "download members" should download members and nodes of way members, not further members of sub relations.
• make cancel button work when sub relations are downloaded

[anonymous]

also I cannot submit a screenshot of the conflict resolver that occured, because its dialog was already closed when I clicked the button to update the object in conflict, and then JOSM started to perform the full recursion (until the JVM crashed, because this could never be stopped before and the CPU jumped to 100% with extreme activity of the GC, then a fatail error of the GC and an empty "Error" dialog appeared and everything was frozen, leaving the current OSM file unsaved, with only partial data in the autosaved file)

[anonymous]

re-add the 'regression' keyword (I did not remove it, it was removed implicitly when I replied and there was another comment posted between)

[GerdP]

In 16008/josm:

fix #18835: unexpected full recursive download of children relations makes some edits impossible

• This is a partly revert of r15811 which changed the meaning of "download members" to "download complete members". Now - as before r15811 - only way and node members are completed. This also applies when overpass api is used. For some relations, e.g. 1311341 (the boundary of Spain), the full recursion has to download thousands of sub relations, which is likely to fail.
• Cancel button did not work while relation members were downloaded.

[simon04]

Two tests are failing, see https://josm.openstreetmap.de/jenkins/job/JOSM/lastCompletedBuild/testReport/:

• org.openstreetmap.josm.io.MultiFetchOverpassObjectReaderTest.testBuildRequestRelationsString
• org.openstreetmap.josm.io.MultiFetchOverpassObjectReaderTest.testBuildComplexString

[GerdP]

In 16009/josm:

see #18835: adapt unit tests to different "recurse down" behaviour

[anonymous]

Thanks at least for fixing this regression and admitting this was a real blocking severe bug (which also had a severe impact on the OSM server API, delivering mich more data than expected, and polluting small .osm edits with tons of unnecessary data, making also JOSM then very slow to work with files transformed from a few kilobytes to several gigabytes, plus adding tens of gigabytes stored in the Java HTTP resource cache (including tons of disk I/O), creating a severe problem for users whose Java temporary cache is on a SSD with limited capacity.

---

Note that I had to clear the Java resource cache (which could not even be displayed with the Java control panel, because it contained tens of thousands files, and this Javacontrol panel runs in a small VM size that cannot fit all: this is a separate bug of this Java panel: we have to set a maximum storage size for this cache not larger than about 2 Gigabytes to avoid it to pollute the storage space, given the caching expiration date of these OSM API calls which are too long; I submitted a bug about this to Oracle for its JRE tools... but this could affect as well the OpenJDK implementation).

May be the OSM API downloader in JOSM should contain some parameters to avoid using the local Java resource cache, which makes queries unnecessarily slow, as this local caching is in fact undesirable as well: so I suggest making OSM API download requests non-cachable in the local Java resource cache (possibly use HTTP POST requests instead of HTTP GET requests, or add a meta header to the HTTP request).

[GerdP]

Next time when you see that JOSM doesn't stop downloading data just kill the program. There are more places where the cancel button dooesn't work. No idea why you watch this happening for hours.

[anonymous]

Killing JOSM in the middle of a submission when the database has reported an edit conflict dans not preserve the status of what was submitted with success. The autosaved file that is generated just before JOSM starts downloading things for long time does not help in restoring the correct state of the edit. If you start checking what was submitted or not, there will be:

• missing data that was submitted with success (that was part of an actual edit) but not in the autosaved file
• or data in the autosaved files that will pollute the edits.

We have then to manually use the JOSM seearch for all objects with "modified" status (need to check the box "all objects", to include also those that were marked to be deleted). And start updating them selectively with the update tool to verify what is their new state or if there are objects left orphaned in the database (note that the previous submission may be in a chengeset that is left unclosed or would have been automatically closed by the server if enough time passes).
This requires extensive manual cleanup for very long time (hours, if not days if this affects very large objects having lot of shared ways/nodes).
The autosaved file (or restored one when doing this checkup) grows dramatically. Edits become very slow, we need a very large VM to launch JOSM (make sure that the Java control panel has the "-Xms4G -Xmx4G" parameters and that you use the 64-bit version if the 32-bit JRE is also installed, and you launch JOSM from the JNLP (JOSM is installed in the Java deployement cache, the startup command line indicating which JRE you use is part of the shortcut installd on the desktop, but may have been changed by a recent JRE update)
Then when doinf all this, with many more data than expected, the JOSM validator will take much longer, but you can't really iognore what is signaled by it as an "error" because may of them are coming from the incomplete state of the partially submitted changeset: the OSM file is partially out of date including for your own edits partially submitted, and you even have to manage your own "edit conflicts" created by objects in the autosaved OSM file that are styill marked "modified" but were already submitted and may have been modified later by someone else as well.
Autosaved files are not error proof because they are not always generated when there were significant state and status changes for submitted objects.
And if you cannot interrupt a long download and have to kill JOSM, there's no way to force a new intermediate "save" manually. All you have is the autosaved file that you need to check selectively.
What was initially a small edit tht should have kept seconds or minutes, can then take hours or days to resolve completely and correctly.