Modify

Opened 14 months ago

Closed 5 weeks ago

Last modified 5 weeks ago

#12562 closed defect (fixed)

USGS Topographic Map Tiles Slow or Fail to Load

Reported by: miketho16@… Owned by: team
Priority: normal Milestone:
Component: Core imagery Version:
Keywords: template_report usgs Cc:

Description

What steps will reproduce the problem?

  1. Start JOSM
  2. Imagery -> USGS Topographic Maps

What is the expected result?

All USGS Topographic Map tiles load quickly and without error

What happens instead?

  1. Some tiles load very slowly.
  2. Sometimes an error is displayed "Error loading tile" in place of the tile itself.
  3. Sometimes a big red "X" is displayed in place of the tile itself.
  4. Sometimes and error is displayed "No tiles at this zoom level" in place of the tile itself, even though it may later display after panning / zooming to another area and then returning (indicated that there is a tile at that zoom level).
  5. Sometimes happens with other imagery sources.
  6. Sometimes a tile that did display previously (and thus should be in the cache) will fail to display when I return to its area after just a minute or so.

Please provide any additional information below. Attach a screenshot if possible.

I have tried:

  1. Restarting JOSM
  2. Rebooting PC
  3. Flushing tile cache, both by right clicking on map and clicking "flush tile cache" as well as manually deleting from hard drive.
  4. Right click on map, "Show Tile Info" and copying tile url and pasting in browser address bar always quickly shows tile.
  5. Turning off all other imagery sources seems to help, but doesn't completely solve the problem.
Revision: 8969
Repository Root: http://josm.openstreetmap.de/svn
Relative URL: ^/trunk
Last Changed Author: Don-vip
Last Changed Date: 2015-10-29 22:15:23 +0100 (Thu, 29 Oct 2015)
Build-Date: 2015-10-30 02:32:24
URL: http://josm.openstreetmap.de/svn/trunk
Repository UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
Last Changed Rev: 8969

Identification: JOSM/1.5 (8969 en) Windows 7 64-Bit
Memory Usage: 324 MB / 1776 MB (89 MB allocated, but free)
Java version: 1.8.0_73, Oracle Corporation, Java HotSpot(TM) 64-Bit Server VM
Dataset consistency test: No problems found

Plugins:
- AddrInterpolation (31772)
- InfoMode (31772)
- Mapillary (31802)
- PicLayer (31772)
- apache-commons (31772)
- apache-http (31772)
- buildings_tools (31772)
- ejml (31772)
- geotools (31774)
- jts (31772)
- opendata (31772)
- photo_geotagging (31772)
- photoadjust (31772)
- proj4j (31772)
- undelete (31772)
- utilsplugin2 (31772)

Last errors/warnings:
- W: JCS - Silent failure during download: http://c.tile.openstreetmap.us/usgs_scanned_topos/17/27223/49355.png
- W: JCS - Silent failure during download: http://c.tile.openstreetmap.us/usgs_scanned_topos/17/27227/49353.png
- W: JCS - Silent failure during download: http://a.tile.openstreetmap.us/usgs_scanned_topos/17/27223/49353.png
- W: JCS - Silent failure during download: http://b.tile.openstreetmap.us/usgs_scanned_topos/18/54451/98708.png
- W: JCS - Silent failure during download: http://b.tile.openstreetmap.us/usgs_scanned_topos/18/54450/98709.png

Attachments (0)

Change History (21)

comment:1 Changed 14 months ago by Klumbumbus

Component: CoreCore imagery

comment:2 Changed 14 months ago by wiktorn

I've checked, it looks like a real problem that needs some better logging on JOSM side. But it looks like the real problem is on the server side. When I do some more map browsing, the server starts returning:
`
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body bgcolor="white">
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.1.19</center>
</body>
</html>
`

If you happen to know, what are the limitations of the tile service provider and how we can avoid them, then we can think, how to incorporate them into JOSM.

comment:3 Changed 14 months ago by Don-vip

Keywords: usgs added

comment:4 Changed 14 months ago by miketho16@…

There might be several things going on here. One is that the cache seems to be ignored. Tiles that should be in the cache are not displaying. I did a test. I waited until I had some USGS tiles successfully loaded (it does happen sometimes). I then disconnected from the Internet. I then could pan and zoom around and any tiles that had been displayed, continued to be displayed. When I try the same thing with the Internet connected, frequently tiles that were previously displayed later fail to display. It seems that JOSM has a preference for re-fetching the USGS tiles rather than using the cache.

comment:5 Changed 5 months ago by daveswarthout@…

I am having a similar problem. I'm running JOSM 11223 on a Windows 10 machine. I've been mapping extensively in Alaska during the past few weeks and the USGS Topographic Maps were working fine but since yesterday I get HTTP Response code 502 and many tiles do not load or if they do, the resolution is so low that names cannot be read. Flushing the tile cache doesn't help.

AlaskaDave

comment:6 Changed 5 months ago by wiktorn

If the server returns 502 JOSM tries it's best to show the tiles based on what it had already downloaded on different zoom levels, that's why the names might be unreadable.

What you can try to do is to go to Advanced Preferences and set:
imagery.generic.minimum_expires and imagery.generic.maximum_expires` to extremely large value such as 31536000000 (1 year). This will bump how long a tile will be fetched from disk cache instead of downloading from network, i.e. tiles will be kept fetch from disk for one year.

*warning* this will affect all layers, so for example if you use also OpenStreetMap Mapnik layer that changes very frequently, you will have to remember to flush the cache of this layer manually from time to time.

The problem with USGS service is that it sets short time-to-live for tiles it returns (7 days) when one takes into account the fact, that they actually wont change anytime soon.

comment:7 in reply to:  6 ; Changed 5 months ago by Klumbumbus

Replying to wiktorn:

What you can try to do is to go to Advanced Preferences and set:
imagery.generic.minimum_expires and imagery.generic.maximum_expires` to extremely large value such as 31536000000 (1 year). This will bump how long a tile will be fetched from disk cache instead of downloading from network, i.e. tiles will be kept fetch from disk for one year.

What about this solution: If JOSM was not able to download new tiles it keeps the old tiles even if they expired. This way you can keep imagery.generic.maximum_expires at its default low value without to fear to loose tiles due to a temporary bad connection to the tile server.

comment:8 in reply to:  7 ; Changed 5 months ago by wiktorn

Replying to Klumbumbus:

What about this solution: If JOSM was not able to download new tiles it keeps the old tiles even if they expired. This way you can keep imagery.generic.maximum_expires at its default low value without to fear to loose tiles due to a temporary bad connection to the tile server.

This logic is already implemented in JOSM and that's why user sees some tiles in different resolution.

If HTTP 502 is a sign of "bandwidth exceeded", then what we need is just to do as few requests towards tile source as we could. If we see HTTP 502 it is already too late.

comment:9 in reply to:  8 ; Changed 5 months ago by Klumbumbus

Replying to wiktorn:

This logic is already implemented in JOSM and that's why user sees some tiles in different resolution.

If this logic is working properly then the behavior described in comment:4 shouldn't be possible, if I understand it right. JOSM should display the tile from the cache as long as the tile is not successfully completely redownloaded from the server.

If HTTP 502 is a sign of "bandwidth exceeded", then what we need is just to do as few requests towards tile source as we could. If we see HTTP 502 it is already too late.

How do we know the reason for the 502 error is "bandwidth exceeded"? Couldn't it have other reasons?

comment:10 in reply to:  9 Changed 5 months ago by wiktorn

Replying to Klumbumbus:

Replying to wiktorn:

This logic is already implemented in JOSM and that's why user sees some tiles in different resolution.

If this logic is working properly then the behavior described in comment:4 shouldn't be possible, if I understand it right. JOSM should display the tile from the cache as long as the tile is not successfully completely redownloaded from the server.

I guess that this was fixed in [10545]. So my guess that it was possible at that time.

If HTTP 502 is a sign of "bandwidth exceeded", then what we need is just to do as few requests towards tile source as we could. If we see HTTP 502 it is already too late.

How do we know the reason for the 502 error is "bandwidth exceeded"? Couldn't it have other reasons?

That's my wild guess. I was fetching tiles for some time, and when I started getting 502, I was getting it for all further requests. Once I waited for some time, then I could again download some more tiles. That's how I remember that it was looking in February.

comment:11 Changed 5 months ago by Klumbumbus

OK, thanks for the explanation.

comment:12 Changed 5 months ago by wiktorn

Just to keep the information why this ticket is open - my feeling is that the best way to solve this is to have possibility to set different minimum expires time per imagery.

comment:13 Changed 5 months ago by daveswarthout@…

My tiles are loading properly now. I am happy to see the imagery return but am curious as to why it happened in the first place. I forgot to mention that based information in the original ticket I tried to load a tile using its URL: "4. Right click on map, "Show Tile Info" and copying tile url and pasting in browser address bar", but in my case that did not load the tile. That would seem to nullify the bandwidth exceeded argument.

Thanks for the quick feedback.

AlaskaDave

comment:14 in reply to:  13 ; Changed 5 months ago by wiktorn

Replying to daveswarthout@…:

I forgot to mention that based information in the original ticket I tried to load a tile using its URL: "4. Right click on map, "Show Tile Info" and copying tile url and pasting in browser address bar", but in my case that did not load the tile. That would seem to nullify the bandwidth exceeded argument.

That's exactly what makes me think that 502 error is sign of traffic limitation. If you can't get to load tile in JOSM nor in browser then it may mean that you've just hit limit of transfer based on IP address.

comment:15 in reply to:  14 ; Changed 4 months ago by daveswarthout@…

Replying to wiktorn:

Replying to daveswarthout@…:

I forgot to mention that based information in the original ticket I tried to load a tile using its URL: "4. Right click on map, "Show Tile Info" and copying tile url and pasting in browser address bar", but in my case that did not load the tile. That would seem to nullify the bandwidth exceeded argument.

That's exactly what makes me think that 502 error is sign of traffic limitation. If you can't get to load tile in JOSM nor in browser then it may mean that you've just hit limit of transfer based on IP address.

2016-12-19 I'm back to having the same problem. Only low res imagery showing, cannot load the faulting tiles from their URLs (HTTP error 404). The limitation of transfer you mention — who is imposing it? The tiles are hosted by OSM, correct?

Do you know if there is some good, as in easy, way to load a subset of Topos locally so my Alaska work can continue?

comment:16 in reply to:  15 ; Changed 4 months ago by wiktorn

Replying to daveswarthout@…:

2016-12-19 I'm back to having the same problem. Only low res imagery showing, cannot load the faulting tiles from their URLs (HTTP error 404). The limitation of transfer you mention — who is imposing it? The tiles are hosted by OSM, correct?

I'm not so sure, that the tiles are hosted by OSM. The page:
http://wiki.openstreetmap.org/wiki/Foundation/Local_Chapters/United_States/Servers/Imagery

States, that they are *proxied*. Though the WMS to which USGS Topo layer points doesn't serve any layers. You should try to contact administrators of this tile source. Maybe authors of this wiki page or maybe on US forum you can get more help.

But maybe 404 means, that the scan is not available in the area that you're working.

comment:17 in reply to:  16 Changed 4 months ago by daveswarthout@…

Replying to wiktorn:

Replying to daveswarthout@…:

2016-12-19 I'm back to having the same problem. Only low res imagery showing, cannot load the faulting tiles from their URLs (HTTP error 404). The limitation of transfer you mention — who is imposing it? The tiles are hosted by OSM, correct?

I'm not so sure, that the tiles are hosted by OSM. The page:
http://wiki.openstreetmap.org/wiki/Foundation/Local_Chapters/United_States/Servers/Imagery

States, that they are *proxied*. Though the WMS to which USGS Topo layer points doesn't serve any layers. You should try to contact administrators of this tile source. Maybe authors of this wiki page or maybe on US forum you can get more help.

But maybe 404 means, that the scan is not available in the area that you're working.

Usually, error 404 means file not found. I believe the scans are available for the areas I'm working in because I saw good resolution details earlier. I'll check the Wiki pages you mentioned and take it from there. Thanks.

comment:18 Changed 5 weeks ago by Klumbumbus

USGS topographic maps tiles should load fine again, since we pull them from another source now. More info about this see https://github.com/osmlab/editor-layer-index/issues/151 and https://github.com/osmlab/editor-layer-index/pull/290

I'm not sure if this ticket should be closed or kept open due to comment:12

comment:19 Changed 5 weeks ago by wiktorn

We can close this. Expiration times per imagery I implement together with #13118.

comment:20 Changed 5 weeks ago by anonymous

Resolution: fixed
Status: newclosed

comment:21 Changed 5 weeks ago by AlaskaDave

Fantastic! The new imagery source is beautiful. Many thanks to all who worked on finding a solution to this thorny problem. Up until yesterday, I had been using copies of the 1:100,000 Alaska maps downloaded from the USGS Topo store to my desktop where I was trying to use them in my mapping. It's very difficult and no substitute for a JOSM layer!

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain team.
as The resolution will be set. Next status will be 'closed'.
The resolution will be deleted. Next status will be 'reopened'.

Add Comment


E-mail address and name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.