[PATCH] TMS simultaneous connections

[bastiK]

TMS layer sends 25 requests at the same time. This is quite excessive, it should be 2-3 simultaneous connections per domain at most.

[stoecker]

Hmm, I think this is in sync with what a web browser does. TMS normally is a simple take and deliver. Usually no advanced operation is needed.

[bastiK]

Normal browser limit seems to be at about 6 connections: ​http://stackoverflow.com/questions/985431/max-parallel-http-connections-in-a-browser

[stoecker]

My feeling always was that a web-browser is a little more demanding to my tile servers than JOSM. So there must be other differences as well.

[wiktorn]

After 8168/josm we have a default limit of 25 concurrent downloads (but not constrained by hosts), which is further tweakable from preferences menu.

If we are to do better here, then we need to decide, whether:

• limit the number per host (many TMS providers has multiple hosts, but one host might also provide different TMS layers)
• limit the number per TMS source

I can take this one.

[bastiK]

Replying to wiktorn:

After 8168/josm we have a default limit of 25 concurrent downloads (but not constrained by hosts), which is further tweakable from preferences menu.

If we are to do better here, then we need to decide, whether:

• limit the number per host (many TMS providers has multiple hosts, but one host might also provide different TMS layers)
• limit the number per TMS source

I can take this one.

Please go ahead. 6-8 concurrent downloads per host sounds reasonable to me.

[wiktorn]

patch ready

[bastiK]

In 8307/josm:

applied #10902 - TMS simultaneous connections (patch by wiktorn)

[bastiK]

Great! I'm still confused by the numbers though. 25 concurrent threads for a queue of 5 elements, with a restriction that no more than 6 elements of the 5-element queue can be from the same host? Is the semaphore released when the download is finished or when it starts?

Edit: More documentation would be nice.

[wiktorn]

Indeed, this is not a fully working solution, as this only balances tasks in the queue

[wiktorn]

Different approach to the problem attached, and some additional comments in the code.

Other possible approaches that I rejected (for reference):

1. Create a thread-pool per host (but this will create a lot of threads, and it will be hard to control, how many threads in total we spawned.
2. Implement all the limits in TMSCachedTileLoaderJob (submit + loadingFinished), but then limit would also cover cache accesses

[bastiK]

In 8314/josm:

applied #10902 - TMS simultaneous connections (2nd patch by wiktorn)

[aceman]

I think something is wrong with this patch. Panning the map (to download area) when the tiles are not yet downloaded OR panning the editor when background imagery needs to be downloaded is very slow and takes 100% of CPU. I have 10 downloads maximum set and the new downloads per host is at the default of 6.

[wiktorn]

@aceman:

I can't reproduce the problem. Are you sure, that this is the patch, that introduced the problem?

Indeed, when downloading the tiles, I notice high CPU usage, but profiling discloses, that most of the time is spent in ImageIO.read (conversion JPEG form to native bitmap) and in painting the tiles.

I do not notice also any slowdown of the GUI itself. Can you please provide:

• java version you're running JOSM on
• operating system

[aceman]

NO, I am not sure this is the culprit, but I just upgraded from 8297 to 8315 and this started happening. So it is the most logical candidate for the problem. I think the CPU load is higher when the network is congested. Maybe the threads are dropping some downloads and the resubmitting of requests is too costly?

I am on Linux kernel 3.19, Java 8b45.

[wiktorn]

Ok, when I limited available bandwidth I see, that AWT thread consumes a lot of CPU, and SlippyMabBoxChoser even more. I'll check if 8297 (and ealier) did had the same behaviour. But I'd like to track this in separate defect.

[wiktorn]

@aceman:
See #11404