Ctrl+U "update data" sends too-large overpass query

[zelonewolf@…]

This bug applies when JOSM is configured to use overpass for updates.

When using "Ctrl+U" on a large data set, JOSM can send a request that's larger than overpass's max input size. By default, overpass is limited to a max input size of 16MB. Therefore, Ctrl+U updates that target overpass should break up the request into chunks such that each chunk is less than 16MB.

[GerdP]

Update data normally downloads again the download areas. Only if there is no such area it will download a list of primitives like Update selection. I wonder if it makes sense to start such complex queries at all?

[skyper]

I just used it on a session file with some bus routes and all members downloaded. I had no time and my incomplete changes are lying around in a local file.

[zelonewolf@…]

Note that I use this functionality with sparse editing, where I would load data from overpass, and edit over a period of time. I then desire to update all in case there were any changes in the map in the meantime. If my data set is too big for update-all, I usually go with update-modified. However, there is no update-deleted, which would detect if I've deleted anything from the sparse data set that has undownloaded referrers, and hence why I try to use Ctrl+U.

[GerdP]

Hmm, Update data probably doesn't do what you think. The code that is executed if there is no download area:

            // no bounds defined in the dataset? We update all but the incomplete primitives in the data set
            UpdateSelectionAction.updatePrimitives(editLayer.data.allPrimitives().stream()
                    .filter(p -> !p.isNew() && !p.isIncomplete()).collect(Collectors.toList()));

So, incomplete objects are not updated. See #19451.
I think the only way to update all modified and deleted objects is to select them and use Update selection.

[zelonewolf@…]

Is it even possible to select deleted objects? Ctrl+F searches for "deleted" find nothing.

[GerdP]

Yes, activate the checkbox "all objects" in the search dialog.

[GerdP]

I think it requires quite a lot of changes to split huge overpass queries into chunks. It might also not be the correct thing to do, maybe fall back to OSM API would be better?
Problem is I have no time for complex JOSM development at the moment.

[skyper]

How does JOSM handle this problem with the "main" OSM server? Not downloading incomplete members of relation is ok, but I wonder if we can tweak the update of ways and its children by downloading the ways completely, with its children nodes.

By the way, these nodes downloads/updates are the major cause for my timeouts. Finally, it worked after twice timing out. The already downloaded data seems to be thrown away, all the time, instead of keeping/caching and merging even with a timeout.

[GerdP]

Replying to skyper:

How does JOSM handle this problem with the "main" OSM server?

JOSM sends lot's of rather short queries with max. 170 primitives each and merges the results.

Not downloading incomplete members of relation is ok, but I wonder if we can tweak the update of ways and its children by downloading the ways completely, with its children nodes.

Maybe JOSM should simply download all elements but avoid to add some of the downloaded elements?

I think it would be good to change the download strategy. At the moment JOSM calculates the list of nodes and produces lots of queries to fetch them. Next the list of ways and finally the list of relations.
Maybe it's better to download the complete ways first and then calculate the list of still missing nodes.