The `State: all-referrers-downloaded` is not saved to file

[skyper]

What steps will reproduce the problem?

1. Download a way and its parent objects
2. Select the way and open "Advanced Info" Ctrl+I
   • The second line says State: all-referrers-downloaded
3. Save the session (or data layer to file)
4. Exit JOSM
5. Start JOSM and load the session (or file)
6. Select the way and open "Advanced Info" Ctrl+I

What is the expected result?

The way should have State: all-referrers-downloaded

What happens instead?

The way has State: referrers-not-all-downloaded

Please provide any additional information below. Attach a screenshot if possible.

The State should be saved

Relative:URL: ^/trunk
Repository:UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
Last:Changed Date: 2024-06-20 17:24:54 +0200 (Thu, 20 Jun 2024)
Revision:19122
Build-Date:2024-06-21 01:31:16
URL:https://josm.openstreetmap.de/svn/trunk

Last errors/warnings:
- 00106.759 E: java.lang.IllegalStateException: JOSM expected to find primitive [way 324274729] in dataset but it is not there. Please report this at https://josm.openstreetmap.de. This is not a critical error, it should be safe to continue in your work.
- 00106.764 E: java.lang.IllegalStateException: JOSM expected to find primitive [way -917] in dataset but it is not there. Please report this at https://josm.openstreetmap.de. This is not a critical error, it should be safe to continue in your work.

[stoecker]

Saving the state wont work because that information is anyway temporary. We probably need a different handling for online and offline data.

[skyper]

JOSM is an offline editor. All data is always only up to the date it was updated the last time from the server. There is no difference in downloading and have the JOSM instance running for a week or so without updating from server or save the downloaded data to a file and open it after a week. You should run into the identical problems (conflicts) with both workflows.

Note: In my case, the time between downloading the parent objects, saving and loading the session was only a few minutes.

[GerdP]

You should run into the identical problems (conflicts) with both workflows.

I don't fully agree (a lot of things are done differently when a newly started JOSM loads a session file)
With a session file it would be possible to save the status of the flag in a special meta file, but I think there is no way to store it in the OSM xml format.

[stoecker]

Yes, JOSM is an offline editor, but that doesn't mean that it works exactly the same online and offline. We have lots of other differences everywhere and storing the download state of something which may change without notice is not helpful. A server-side changed node, way or relation you will notice on upload, an added referrer will be a silent operation. Handling that as if the state is still "recently downloaded" simply is wrong.

But the warnings will make offline work bothersome, so we should simply not warn for offline data (in most cases, except the very dangerous ones).

[GerdP]

Is any layer that was loaded from file considered offline data as long as no download from a server happened?
That would more or less disable all the tests that we implemented to avoid the corruption of relations or geometry.
At least in the scenario described above where the user stores data on disk to continue working with it later.

[skyper]

Replying to GerdP:

You should run into the identical problems (conflicts) with both workflows.

I don't fully agree (a lot of things are done differently when a newly started JOSM loads a session file)

Do you have any example regarding the data layers? Especially, I am interested in the differences between starting a session by downloading and editing and than have it running a week and starting a session by downloading and editing and then save the session and loading it again after a week.

Replying to stoecker:

Yes, JOSM is an offline editor, but that doesn't mean that it works exactly the same online and offline. We have lots of other differences everywhere and storing the download state of something which may change without notice is not helpful. A server-side changed node, way or relation you will notice on upload, an added referrer will be a silent operation. Handling that as if the state is still "recently downloaded" simply is wrong.

Sorry, I do not get it. In my eyes, JOSM core is not an online editor by any means. Yes, there are plugins (continuous download) and few options which offer to download additional data but even then the download of an object (or/and its children/parents) is only performed once and with the options the downloads are still manual as the user has to accept them (at least once).

Regarding referrer the elapsed time after the last download/update is the only significant number which could lead to a higher percentage to run into situations where new referrers have been created on the server, meanwhile, but even after a successful download we can not be 100% confident that there isn't a referrer created on the server within the next second.
I could even create this problem by myself if I download an area, duplicate the layer, add a relation in one of the layers, upload the layer and delete it and then continue to work in the other layer without updating the data.

Replying to GerdP:

Is any layer that was loaded from file considered offline data as long as no download from a server happened?

I would say all data layers are offline (even those just downloaded). The last download/update of an objects sets a timestamp and after that there is no more communication between JOSM and the server. The only relevant info we currently take into account is the object's version number.

[stoecker]

Replying to GerdP:

Is any layer that was loaded from file considered offline data as long as no download from a server happened?

I though more about offline is offline until you download something new into it. From then on it's "warnings on".

[stoecker]

@skyper:

Sorry, I do not get it. In my eyes, JOSM core is not an online editor by any means.

If you go by this definition there is no online editor. You will always have a caching delay as long as the API does not actively inform the editor about news.

[skyper]

Actually, I only know JOSM well enough and I do not know how other editor software work but automatically updating in the background after some period of time or if the user selects an object are possibilities.

I still miss an answer about the differences between following two workflows:

1. Download some data including referrers
2. Do some edits
3. Go for lunch without closing JOSM
4. Continue working

and b) instead save the data layer before lunch plus close JOSM and continue after lunch by starting JOSM and loading the previously saved data layer to continue working.

[GerdP]

Just a few that come into my mind:
JOSM stores information in caches, Undo/Redo stack is not stored on file, some changes in preferences are not used until a restart, in general any memory leak will be "cleared" with a restart.
None of this is assumed to cause problems, but for sure it can cause differences.

[skyper]

None of the above makes any difference regarding the State: all-referrers-downloaded.

[GerdP]

You can think of this state as being cached in JOSM.

[skyper]

I was really hoping that this new flag would solve the problems we have with inside/outside download area (query). Without saving the flag loaded files will lead to different results in many places and using the flag for validator is probably not a good idea (or we will have lots of different results between downloaded data and data loaded from file which would make it almost impossible to debug problems).
Meanwhile, not saving the flag leads to more (most of the time unneeded) traffic as download parent objects is quite costly.

[GerdP]

Yes, we replaced one problem by another. I think I warned about the problem that we cannot store the flag in files long ago.