Modify

Opened 13 months ago

Last modified 13 months ago

#20869 new defect

Ctrl+U "update data" sends too-large overpass query

Reported by: zelonewolf@… Owned by: team
Priority: normal Milestone:
Component: Core Version:
Keywords: overpass Cc:

Description

This bug applies when JOSM is configured to use overpass for updates.

When using "Ctrl+U" on a large data set, JOSM can send a request that's larger than overpass's max input size. By default, overpass is limited to a max input size of 16MB. Therefore, Ctrl+U updates that target overpass should break up the request into chunks such that each chunk is less than 16MB.

Attachments (0)

Change History (9)

comment:1 Changed 13 months ago by GerdP

Update data normally downloads again the download areas. Only if there is no such area it will download a list of primitives like Update selection. I wonder if it makes sense to start such complex queries at all?

comment:2 Changed 13 months ago by skyper

I just used it on a session file with some bus routes and all members downloaded. I had no time and my incomplete changes are lying around in a local file.

comment:3 Changed 13 months ago by zelonewolf@…

Note that I use this functionality with sparse editing, where I would load data from overpass, and edit over a period of time. I then desire to update all in case there were any changes in the map in the meantime. If my data set is too big for update-all, I usually go with update-modified. However, there is no update-deleted, which would detect if I've deleted anything from the sparse data set that has undownloaded referrers, and hence why I try to use Ctrl+U.

comment:4 Changed 13 months ago by GerdP

Hmm, Update data probably doesn't do what you think. The code that is executed if there is no download area:

            // no bounds defined in the dataset? We update all but the incomplete primitives in the data set
            UpdateSelectionAction.updatePrimitives(editLayer.data.allPrimitives().stream()
                    .filter(p -> !p.isNew() && !p.isIncomplete()).collect(Collectors.toList()));

So, incomplete objects are not updated. See #19451.
I think the only way to update all modified and deleted objects is to select them and use Update selection.

comment:5 Changed 13 months ago by zelonewolf@…

Is it even possible to select deleted objects? Ctrl+F searches for "deleted" find nothing.

comment:6 Changed 13 months ago by GerdP

Yes, activate the checkbox "all objects" in the search dialog.

comment:7 Changed 13 months ago by GerdP

I think it requires quite a lot of changes to split huge overpass queries into chunks. It might also not be the correct thing to do, maybe fall back to OSM API would be better?
Problem is I have no time for complex JOSM development at the moment.

comment:8 Changed 13 months ago by skyper

How does JOSM handle this problem with the "main" OSM server? Not downloading incomplete members of relation is ok, but I wonder if we can tweak the update of ways and its children by downloading the ways completely, with its children nodes.

By the way, these nodes downloads/updates are the major cause for my timeouts. Finally, it worked after twice timing out. The already downloaded data seems to be thrown away, all the time, instead of keeping/caching and merging even with a timeout.

comment:9 in reply to:  8 Changed 13 months ago by GerdP

Replying to skyper:

How does JOSM handle this problem with the "main" OSM server?

JOSM sends lot's of rather short queries with max. 170 primitives each and merges the results.

Not downloading incomplete members of relation is ok, but I wonder if we can tweak the update of ways and its children by downloading the ways completely, with its children nodes.

Maybe JOSM should simply download all elements but avoid to add some of the downloaded elements?

I think it would be good to change the download strategy. At the moment JOSM calculates the list of nodes and produces lots of queries to fetch them. Next the list of ways and finally the list of relations.
Maybe it's better to download the complete ways first and then calculate the list of still missing nodes.

Modify Ticket

Change Properties
Set your email in Preferences
Action
as new The owner will remain team.
as The resolution will be set.
to The owner will be changed from team to the specified user.
The owner will change to zelonewolf@gmail.com
as duplicate The resolution will be set to duplicate.The specified ticket will be cross-referenced with this ticket
The owner will be changed from team to anonymous.

Add Comment


E-mail address and name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.