Tiles constantly reloading

[Stereo]

What steps will reproduce the problem?

1. Zoom to an area in Luxembourg
2. Activate the 'geoportail.lu ortho latest' layer

This isn't easy to reproduce, and doesn't happen every time. Sometimes, zooming out and in fixes it.

It has been happening to me for a few versions, but has been getting worse.

What is the expected result?

Tiles load and get displayed

What happens instead?

Tiles constantly reload, flashing on the screen.

Please provide any additional information below. Attach a screenshot if possible.

See attached video.

I also get this with other layers hosted on a different server like "openstreetmap.lu "mapper’s delight" hillshade lidar 2019 with unclassified points" from Imagery > Elevation.

See also #19741

Revision:17288
Is-Local-Build:true
Build-Date:2020-10-31 01:23:03

Identification: JOSM/1.5 (17288 SVN en_GB) Mac OS X 10.15.7
OS Build number: Mac OS X 10.15.7 (19H2)
Memory Usage: 8192 MB / 8192 MB (4855 MB allocated, but free)
Java version: 15.0.1+8, Azul Systems, Inc., OpenJDK 64-Bit Server VM
Look and Feel: com.apple.laf.AquaLookAndFeel
Screen: Display 69733632 1680x1050 (scaling 2.0x2.0)
Maximum Screen Size: 1680x1050
Best cursor sizes: 16x16 -> 16x16, 32x32 -> 32x32
VM arguments: [--module-path=/Applications/JOSM.app/Contents/app/mods]

Plugins:
+ AddrInterpolation (35583)
+ CommandLine (35583)
+ Create_grid_of_ways (35583)
+ ImportImagePlugin (35248)
+ ImproveWay (29)
+ Mapillary (1.5.27)
+ OpeningHoursEditor (35579)
+ PicLayer (2a9aa7a)
+ RelationDissolve (0.2.0)
+ alignways (35583)
+ apache-commons (35524)
+ apache-http (35092)
+ areaselector (368)
+ austriaaddresshelper (1597341117)
+ auto_tools (73)
+ buildings_tools (35579)
+ changeset-viewer (22)
+ conflation (0.6.6)
+ continuosDownload (91)
+ contourmerge (v0.1.6)
+ ejml (35313)
+ geotools (35169)
+ gridify (1588746833)
+ imagery-xml-bounds (35546)
+ jaxb (35092)
+ jna (35092)
+ jogl (1.2.3)
+ jts (35122)
+ log4j (35092)
+ opendata (35513)
+ openqa (0.2.0)
+ poly (35248)
+ public_transport (35405)
+ reltoolbox (35602)
+ reverter (35579)
+ shrinkwrap (v1.0.3)
+ splinex (35454)
+ terracer (35579)
+ todo (30306)
+ undelete (35521)
+ utilsplugin2 (35624)
+ wikipedia (1.1.4)

Tagging presets:
+ https://josm.openstreetmap.de/josmfile?page=Presets/NewTags&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Presets/BicycleJunction&preset&zip=1
+ http://osmtools.de/josm/steps.xml
+ https://josm.openstreetmap.de/josmfile?page=Presets/Historic_Stone&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Presets/LaneAttributes&preset&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Presets/Golf_Course&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Presets/PublicTransportOneClick&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Presets/Bus_lanes&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Presets/Historical_Objects&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Presets/public_bookcase&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Presets/Stolpersteine&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Presets/BuildingPreset&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Presets/Playground_Equipment&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Presets/PublicTransportGtfs&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Presets/Heritage&zip=1
+ https://gitlab.com/k127/josm-presets/raw/master/diplomatic.xml

Map paint styles:
- https://josm.openstreetmap.de/josmfile?page=Styles/Lit&style&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/ParkingLanes&style&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/Surface&style&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/Maxspeed&style&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/Modified&style&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/LessObtrusiveNodes&style&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/AddressValidator&style&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/Noname&style&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/Power&style&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/Surface-DataEntry&style&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/PowerMapping&style&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/Lane_and_Road_Attributes&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/BesideTheRoad_Speed&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/TreeCrownDiameter&style&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/SlovakiaBicycleRoutes&style&zip=1
- https://github.com/simon04/coloured-addresses.mapcss/raw/master/dist/coloured-addresses.mapcss
- https://josm.openstreetmap.de/josmfile?page=Styles/Coloured_Streets&style&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/Coloured_Suburb&style&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/Cycleways&style&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/SlovakiaBicycleRoutes&style&zip=1
- https://github.com/GlassOceanos/indoor-JOSM-style/archive/master.zip
- https://josm.openstreetmap.de/josmfile?page=Styles/PublicTransport&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/PTStops&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/SlovakiaBicycleRoutes&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/SimpleRoofTags&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/SimpleBuildingTags&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/HiDPISupport&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/NumberedCycleNodeNetworks&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/NumberedWalkingNodeNetworks&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/AddressValidator&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/NewHighwayColors&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/Noname&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/iD&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/Potlach2_access&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/IconTester&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Styles/Potlatch2&zip=1

Validator rules:
+ http://www.openrailwaymap.org/validator/openrailwaymap.validator.mapcss
+ https://josm.openstreetmap.de/josmfile?page=Rules/GermanySpecific&zip=1
+ http://www.openrailwaymap.org/validator/de-openrailwaymap.validator.mapcss
+ https://josm.openstreetmap.de/josmfile?page=Rules/MissingWeekdays&zip=1
+ https://raw.githubusercontent.com/nlehuby/transport_mapcss/master/transport.validator.mapcss

Last errors/warnings:
- 00025.335 W: Not a single layer for the name 'Bing Sat': []
- 00025.335 W: Not a single layer for the name 'ANA DTM 2017': []
- 00028.517 E: <josm.userdata>/plugins/opendata/resources/org/openstreetmap/josm/plugins/opendata/modules/fr/datagouvfr/datasets/agriculture/RegistreParcellaire.mapcss (No such file or directory)

[Stereo]

Tiles reloading constantly

[Klumbumbus]

Is this with reprojection?

[Stereo]

No, everything in EPSG:3857.

[GerdP]

Duplicate of #20207?

[Stereo]

Some of the log messages I'm getting:

2021-01-17 14:32:11.280 WARNING: Not downloading all tiles because there is more than 18 tiles on an axis!
2021-01-17 14:32:11.280 JOSM[62485:5146735] 2021-01-17 14:32:11.280 WARNING: Not downloading all tiles because there is more than 18 tiles on an axis!

And indeed, in the video, you'll see that at the resolution the tiles are being downloaded, it needs about 22 tiles.

The limit seems to be in AbstractTileSourceLayer.java:

        private boolean tooLarge() {
            return insane() || this.tilesSpanned() > 20;
        }

I'll play with that limit a bit and see if it works better.

[GerdP]

#20207 was fixed, did you try if that solved your problem as well?

[Stereo]

#20207 didn't fix my problem, but raising the tooLarge limit helped.

Adding some logging, I found that this.tilesSpanned() was going all the way up to 22.20360331117452. Doing a bit of quick multiplication, this means that the 'virtual' size of the viewport was 5684 pixels across.

I then found out that because I had my Imagery > Settings > Tile Zoom Offset setting set to 1, JOSM was downloading tiles at twice the display resolution. Divided by two, loading 2842 pixels of tiles across to fill 2560 pixels of width does make sense.

The highest resolution displays you can buy in January 2021 are 8K displays that are 7680 pixels wide. Even without the zoom offset setting changed, someone with a display like that would run into the same issue.

I would suggest bumping the limit in tooLarge() to at least 33 to make sure that JOSM can load tiles when in full screen on a 8K display, and maybe making it an advanced config option to make it easier to work around the problem when we run into it again the day we have 16K displays. I wouldn't suggest a warning message or exception which would get triggered when someone zooms out on a layer with a high min_zoom.

[sano]

I see this problem as well.
MacBook Pro resolution 1440 x 900
MacOS X 10.14.6 Mojave
JOSM v. 17428

I will provide more info, please ask if necessary.

[GerdP]

I don't understand what the magic value 20 in tooLarge() means. A related hardcoded warning message is "Not downloading all tiles because there is more than 18 tiles on an axis!", and the 18 is also not computed :(

I wonder if JOSM users are expected to reduce the viewport size when such a message is printed on the console?

[skyper]

With a slow internet connection it would be useful, too, to limit the number of parallel connections and downloads with an option.

[GerdP]

Do you mean the existing preference cache.jcs.max_threads?

[skyper]

Replying to GerdP:

Do you mean the existing preference cache.jcs.max_threads?

Not sure, what does it do? Is it a general option for all connections or per layer? cache is misleading as I have no problem with cached images but with the parallel download/update of the images (cache).

[GerdP]

Replying to skyper:

Replying to GerdP:

Do you mean the existing preference cache.jcs.max_threads?

Not sure, what does it do? Is it a general option for all connections or per layer? cache is misleading as I have no problem with cached images but with the parallel download/update of the images (cache).

Comment says "maximum download threads that will be started". Code was introduced for #11216

[skyper]

Ok, thanks, this preference has already a GUI but documentation is rather poor.

[GerdP]

I don't see a dialog for cache.jcs.max_threads and I don't yet understand the meaning of the setting.

[skyper]

Arrg, sorry, I mixed it up with imagery.wms.loader.maxjobs.

[GerdP]

In 17486/josm:

see #20014: Tiles constantly reloading

• change synchronization a bit. No idea if it helps.
• simplify code

[Stereo]

Can we still set the limit in tooLarge() and the error message in loadAllTiles() to 40?

[Stereo]

Tentatively setting to 21.02 if it's just a limit to bump?

[GerdP]

In 17492/josm:

see #20014: see #20014: Tiles constantly reloading

• implement named constant MAX_TILES_SPANNED with value 40 instead of 20

[GerdP]

Replying to Stereo:

Can we still set the limit in tooLarge() and the error message in loadAllTiles() to 40?

Well, I hoped for an explanation of this hard coded limit because I don't understand what it is about.

[Stereo]

From digging through the history, it's nothing more than a sanity check. Of course, having an undocumented magic number isn't great. If you're stumbling upon this bug after hours of debugging ten years from now, hi!

[GerdP]

While looking at #20497 I learned that tooLarge() should consider the WMS tile size, preference imagery.wms.imageSize. Default is 512, but values between 1 and 4096 are acepted. The method only works well with the default.

[Stereo]

Ah, indeed! That's a larger change, and I've never seen tiles smaller than 512, and the higher limit won't cause more problems. Should we split that into another ticket, or at least unset the milestone?

[GerdP]

No idea. I feel like I only understand 1% of the code until now. Esp. I don't understand the difference between TMS and WMS yet.
For example I wonder why I see so many calls of method createTileLoaderJob() when I press Ctrl+CursorUp once after waiting for quite a while. It feels as if each pan action requires new tiles. When I see this

WARNING: java.net.SocketTimeoutException: Read timed out. Cause: java.net.SocketTimeoutException: Read timed out

createTileLoaderJob() is also executed, probably to try again to download the tile which just got a time out. No idea if that is a good idea.
I also see strange effects (black tiles, flickering, blurs) from the renderer(?) when all tiles are in the JCS disk cache when using hotkeys 8 and 9 two switch between just two different viewports. It feels as if the rendering of one tile triggers a repaint of the screen which sometimes seems to re-trigger the rendering of the tile.

[GerdP]

I think method overloadTiles() is part of the problem. When I remove the code in that method I see fewer problems with WMS.

[wiktorn]

While at this, I'd like to point you to this part of AbstractTileSourceLayer:

    protected int estimateTileCacheSize() {
        Dimension screenSize = GuiHelper.getMaximumScreenSize();
        int height = screenSize.height;
        int width = screenSize.width;
        int tileSize = 256; // default tile size
        if (tileSource != null) {
            tileSize = tileSource.getTileSize();
        }
        // as we can see part of the tile at the top and at the bottom, use Math.ceil(...) + 1 to accommodate for that
        int visibileTiles = (int) (Math.ceil((double) height / tileSize + 1) * Math.ceil((double) width / tileSize + 1));
        // add 10% for tiles from different zoom levels
        int ret = (int) Math.ceil(
                Math.pow(2d, ZOOM_OFFSET.get()) * visibileTiles // use offset to decide, how many tiles are visible
                * 4);
        Logging.info("AbstractTileSourceLayer: estimated visible tiles: {0}, estimated cache size: {1}", visibileTiles, ret);
        return ret;
    }

It already estimates how many tiles can be visible on screen and sizes the caches accordingly. If more tiles will be visible on screen, then due to tiles going out of MemoryTileCache bad things can happen.

Maybe we should calculate here value for tooLarge() so it's adapted to the screen size of the user as well as it will be checked, if the user has sufficient memory set to work with such large screen (see allocateCacheMemory).

Taking into account, which zoomOffsets do we try, when painting screen (from -5 to 2), I'd change *magic* value of 4 in here to:

zoom offset	tiles
-5	visibileTiles / (25)
-4	visibileTiles / (24)
-3	visibileTiles / (23)
-2	visibileTiles / (22)
-1	visibileTiles / (21)
0	visibileTiles
1	visibileTiles * 21
2	visibileTiles * 22

As sum(1/2^(-x)) converges to 1, we should have:
visibileTiles * (1 + 1 + 2 + 4) = visibileTiles * 8
(1 for negative zoom offsets, 1 for current zoom level, 2 for +1 offset, 4 for +2 offset).

Plus we need to add to visibileTiles +2 on each axis for overloadTiles() - as it adds 2 tiles to load.

So we'll require two times more memory for each imagery layer but we'll be covered even in most pessimistic scenarios

In the end, this means, that we should have following tooLarge body, as insane() checks for MemoryTileCache size:

        private boolean tooLarge() {
            return insane();
        }

[wiktorn]

In 17495/josm:

Extend the size of MemoryTileCache

Include in the computations:

• all zoomOffsets that might be visited during paint
• overloadTiles() method that adds new tiles to be fetched

Merge tooLarge() and insane() method, as only tileCache size limit is sensible and resign from any
artificial limits.

See: #20014, #20497

[wiktorn]

In 17507/josm:

Add documentation for changes from [17495]

Reverse links in code, so when changes will be applied in zoomOffsets or
overloadTiles comments will direct developer to estimateTileCacheSize to update
the computation there.

See: #20014, #20497

[wiktorn]

@Stereo: can you check, if after [17495] the problem persists? At least in #20443 users report, that "disco mode" disappeared.

[wiktorn]

In 17516/josm:

Fine tune cache sizing

Reduce the cache size, by not counting the tiles that are overloaded for other
zoom levels. Overloaded tiles are only counted for current zoom level.

This reduces memory requirements for a layer, depending on the number of the
tiles shown by 20%-50%. Reduction is high for low number of tiles (when large
tiles or small screen is in use) and low for high number of tiles (when
small tiles or large screen is in use)

See: #20014, #20497

[wiktorn]

After all the patches here the status is:

• ✔️ tiles should not be constantly reloading (apart from edge cases mentioned below)
• ❌ tiles will be loaded more than once, if reprojection is active and noticeable flicker will occur on zoom level change (tile visible, tile not visible, tile visible). This is due to following code and I guess we need to live with it. They might give impression that they are loaded from network, but they are (usually) loaded from disk cache, though this takes time

              if (tile instanceof ReprojectionTile && ((ReprojectionTile) tile).needsUpdate(mapView.getScale())) {
                  // This means we have a reprojected tile in memory cache, but not at
                  // current scale. Generally, the positioning of the tile will still
                  // be correct, but for best image quality, the tile should be
                  // reprojected to the target scale. The original tile image should
                  // still be in disk cache, so this is fairly cheap.
                  ((ReprojectionTile) tile).invalidate();
                  loadTile(tile, false);
              }
  

• ❌ estimations of number of visible tiles at current zoom level are wrong, in case of reprojections (esp. when reprojection means tiles are under angle). Anyone can help with math, how to calculate this properly?
• ❌ we are using tileCache.size() to check if TileSet is too large, instead of checking it against visible set size estimation. This means, that we might want to show at most tileCache.size() on currentZoomLevel and there will be no space in tileCache for over-loaded tiles or tiles from different zoom levels if some will be missing at current one. This might lead again to "disco" situation

So currently I think that we'll need two limits:

• how many tiles we want to show on the screen (suggestion: visibileTiles, this means we will not show tiles, if zoomLevel < minZoom)
• what's maximum TileSet we allow to load (suggestion: tileCache.size())

For it to work properly, I need a better calculation for the number of visible tiles. Then I can check all TileSet.tooLarge() calls, and decide, what we actually wanted in each of the places.