Modify

Opened 3 years ago

Last modified 4 months ago

#21563 new defect

Opening history of relation with high version number leads to exceeding OSM bandwidth limit

Reported by: skyper Owned by: team
Priority: normal Milestone:
Component: Core Version:
Keywords: template_report history error 509 bandwidth limit download Cc:

Description (last modified by skyper)

What steps will reproduce the problem?

  1. Have a relation with quite some versions, in my case 1281 (relation/2769137)
  2. Open history without noticing the high version number
  3. Cancel download after some seconds as it takes too long
  4. Notice the high version number and open history
  5. Sucessfully, look at the last 20 version
  6. Download a member

What is the expected result?

The member is downloaded and other downloads work as expected

What happens instead?

SEVERE: Bandwidth Limit Exceeded - 
<html>Communication with the OSM server 'api.openstreetmap.org'failed. The server replied
the following error code and the following error message:
<strong>Error code:<strong> 509 <strong>Error message (untranslated)</strong>: You have downloaded too much data. Please try again later.</html>

and the user is temporarly blocked

Please provide any additional information below. Attach a screenshot if possible.

I am not sure what to do here. Maybe, only loading the last 20/50/100 version could help or downloading not all versions from main server.
Anyway, this is annoying as I got blocked without even really noticing that I may exceed the limit and I only wanted to take a look at the versions of the last couple month and not at all versions of the last decade.

Relative:URL: ^/trunk
Repository:UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
Last:Changed Date: 2021-11-12 11:23:59 +0100 (Fri, 12 Nov 2021)
Revision:18323
Build-Date:2021-11-13 02:31:03
URL:https://josm.openstreetmap.de/svn/trunk

Last errors/warnings:
- 12417.014 E: java.util.concurrent.ExecutionException: org.openstreetmap.josm.io.OsmApiException: ResponseCode=509, Error Header=<You have downloaded too much data. Please try again later.>. Cause: org.openstreetmap.josm.io.OsmApiException: ResponseCode=509, Error Header=<You have downloaded too much data. Please try again later.>
- 12417.014 E: org.openstreetmap.josm.io.OsmApiException: ResponseCode=509, Error Header=<You have downloaded too much data. Please try again later.>
- 12417.014 E: org.openstreetmap.josm.io.OsmApiException: ResponseCode=509, Error Header=<You have downloaded too much data. Please try again later.>
- 12417.020 E: Bandwidth Limit Exceeded - <html>Communication with the OSM server 'api.openstreetmap.org'failed. The server replied<br>the following error code and the following error message:<br><strong>Error code:<strong> 509<br><strong>Error message (untranslated)</strong>: You have downloaded too much data. Please try again later.</html>
- 12445.113 E: org.openstreetmap.josm.io.OsmApiException: ResponseCode=509, Error Header=<You have downloaded too much data. Please try again later.>
- 12445.113 E: org.openstreetmap.josm.io.OsmApiException: ResponseCode=509, Error Header=<You have downloaded too much data. Please try again later.>
- 12445.125 E: Bandwidth Limit Exceeded - <html>Communication with the OSM server 'api.openstreetmap.org'failed. The server replied<br>the following error code and the following error message:<br><strong>Error code:<strong> 509<br><strong>Error message (untranslated)</strong>: You have downloaded too much data. Please try again later.</html>
- 12527.828 E: org.openstreetmap.josm.io.OsmApiException: ResponseCode=509, Error Header=<You have downloaded too much data. Please try again later.>
- 12527.829 E: org.openstreetmap.josm.io.OsmApiException: ResponseCode=509, Error Header=<You have downloaded too much data. Please try again later.>
- 12527.837 E: Bandwidth Limit Exceeded - <html>Communication with the OSM server 'api.openstreetmap.org'failed. The server replied<br>the following error code and the following error message:<br><strong>Error code:<strong> 509<br><strong>Error message (untranslated)</strong>: You have downloaded too much data. Please try again later.</html>

Attachments (0)

Change History (6)

comment:1 by GerdP, 3 years ago

+1 for selective download of a few versions only, if that is possible. Maybe tricky since #20879
See also #20405.

comment:2 by skyper, 3 years ago

Description: modified (diff)

comment:3 by taylor.smock, 13 months ago

Looking at the debug output,

GET https://api.openstreetmap.org/api/0.6/relation/2769137/history -> HTTP_2 200 (4.6 s)
RESPONSE HEADERS: {:status=[200], cache-control=[private, max-age=0, must-revalidate], content-encoding=[gzip], content-type=[application/xml; charset=utf-8], date=[Thu, 27 Apr 2023 20:16:57 GMT], server=[Apache/2.4.54 (Ubuntu)], strict-transport-security=[max-age=31536000; includeSubDomains; preload]}
SAX parsing completed in 27.1 s
GET https://api.openstreetmap.org/api/0.6/changesets?changesets=15235306,15237358,17113347,15876341,124125036,28633534,31203799,46088883,70132770,87565620,27756983,60185478,47051481,87657772,88411475,89204038,50529051,75930782,99761642,69637183,88687952,88470879,94469563,37962339,27763075,88472931,52132663,88808805,111122094,88175989,88227190,48372439,100150735,32147922,72971363,100505031,50871092,28631434,44155618,88167687,16629946,24863036,68963419,87647601,36334179,88331528,68627545,81585298,88814878,16138401,89351428,87725410,88630545,25219547,33573467,22608132,88292637,53187403,88368421,57977623,41327122,55030563,124100353,29413799,38064685,70542431,74071046,88636708,20490581,87567702,16793967,23404820,89255200,94121453,17930594,73681936,27759058,23130391,69389403,88515900,52368228,28819912,24064493,88477124,37827268,27844896,83078266,87657908,20179362,15231098,89079261,49629798,37020321,71218344,100363629,48097866,30165332,58608614,74077425,88083931 -> HTTP_2 200 (351 ms; 10.0 kB)
RESPONSE HEADERS: {:status=[200], cache-control=[max-age=0, private, must-revalidate], content-encoding=[gzip], content-length=[10248], content-security-policy=[default-src 'self'; child-src 'self'; connect-src 'self' matomo.openstreetmap.org; font-src 'none'; form-action 'self'; frame-ancestors 'self'; frame-src 'self'; img-src 'self' data: www.gravatar.com *.wp.com tile.openstreetmap.org *.tile.openstreetmap.org *.tile.thunderforest.com tileserver.memomaps.de *.openstreetmap.fr matomo.openstreetmap.org https://openstreetmap-user-avatars.s3.dualstack.eu-west-1.amazonaws.com https://openstreetmap-gps-images.s3.dualstack.eu-west-1.amazonaws.com; manifest-src 'self'; media-src 'none'; object-src 'self'; script-src 'self' matomo.openstreetmap.org; style-src 'self'; worker-src 'none'], content-type=[application/xml; charset=utf-8], date=[Thu, 27 Apr 2023 20:17:29 GMT], etag=[W/"b399ad8462160d9b2881c53c8ca36b1c-gzip"], referrer-policy=[strict-origin-when-cross-origin], server=[Apache/2.4.54 (Ubuntu)], status=[200 OK], strict-transport-security=[max-age=31536000; includeSubDomains; preload], vary=[Origin,Accept-Encoding], x-content-type-options=[nosniff], x-download-options=[noopen], x-frame-options=[sameorigin], x-permitted-cross-domain-policies=[none], x-powered-by=[Phusion Passenger(R) 6.0.17], x-request-id=[ZErYWRo4dMYNK0nyfjfVjgAAqi0], x-runtime=[0.197043], x-xss-protection=[1; mode=block]}
SAX parsing completed in 5 ms
GET https://api.openstreetmap.org/api/0.6/changesets?changesets=87655846,84473256,24988124,83299412,17205664,27767041,88962538,35111615,20695442,88458732,86140302,48185968,26626343,72635623,88811005,43534895,89179623,46772855,58254278,19087772,46406148,87612922,88040951,68619486,88626562,55221126,52919285,101451487,66421536,88528275,94166346,20666860,31336713,34593492,36569842,71634075,79596640,21770669,88511911,16367643,16087056,39127728,65157897,29014358,20169182,50508783,28957014,67261687,61158226,21289406,65698582,44133281,22646872,88151106,33227004,70599986,33540343,50397706,88243273,80903644,19998757,21608541,24512611,85013508,17225745,95351990,100972316,17412112,55186003,88255579,17162264,58305124,46240735,15223287,88351835,20690970,79820263,56380027,72578420,15235521,17352737,72973680,89248888,84964411,18491447,88173676,31236332,38996835,50553394,82496987,96480497,22474850,17205299,58053196,35334944,93330605,77596063,20041840,21248005,27912416 -> HTTP_2 200 (376 ms; 10.3 kB)
RESPONSE HEADERS: {:status=[200], cache-control=[max-age=0, private, must-revalidate], content-encoding=[gzip], content-length=[10552], content-security-policy=[default-src 'self'; child-src 'self'; connect-src 'self' matomo.openstreetmap.org; font-src 'none'; form-action 'self'; frame-ancestors 'self'; frame-src 'self'; img-src 'self' data: www.gravatar.com *.wp.com tile.openstreetmap.org *.tile.openstreetmap.org *.tile.thunderforest.com tileserver.memomaps.de *.openstreetmap.fr matomo.openstreetmap.org https://openstreetmap-user-avatars.s3.dualstack.eu-west-1.amazonaws.com https://openstreetmap-gps-images.s3.dualstack.eu-west-1.amazonaws.com; manifest-src 'self'; media-src 'none'; object-src 'self'; script-src 'self' matomo.openstreetmap.org; style-src 'self'; worker-src 'none'], content-type=[application/xml; charset=utf-8], date=[Thu, 27 Apr 2023 20:17:29 GMT], etag=[W/"81a0e0f3be92f68586a2bf182fc71f32-gzip"], referrer-policy=[strict-origin-when-cross-origin], server=[Apache/2.4.54 (Ubuntu)], status=[200 OK], strict-transport-security=[max-age=31536000; includeSubDomains; preload], vary=[Origin,Accept-Encoding], x-content-type-options=[nosniff], x-download-options=[noopen], x-frame-options=[sameorigin], x-permitted-cross-domain-policies=[none], x-powered-by=[Phusion Passenger(R) 6.0.17], x-request-id=[ZErYWRo4dMYNK0nyfjfVkAAAnC0], x-runtime=[0.229301], x-xss-protection=[1; mode=block]}
SAX parsing completed in 6 ms
GET https://api.openstreetmap.org/api/0.6/changesets?changesets=89312281,101386054,44137454,35578703,15452600,15755687,89072664,94467321,30310551,57973290,72992007,61565684,20420695,88067167,33995623,17688687,101709675,36113216,88982570,99675287,55716411,23191563,36342599,15235463,89351228,28819652,93734117,16679306,18221154,26947823,52150894,88142906,15995279,70206804,87362633,41286410,71073212,85734556,88511686,88282306,17440898,20168891,31985763,37682096,87643304,75640080,88960222,88450257,89070811,28123195,89345221,31785085,88042663,79871323,87725217,88349912,85734530,49959770,29229084,30150766,16171349,88806639,87772318,83807572,89165053,24432859,26040377,15083859,63404670,52114080,18075823,24891588,95677555,85458132,23347360,45677437,88253573,88409231,34147266,91747516,46187277,20177148,79975701,52040398,77892977,88177816,20656359,99542066,36709358,39787393,47569675,85626106,87655643,88528036,100904928,88974505,17854711,78099776,95632468,32956447 -> HTTP_2 200 (366 ms; 10.2 kB)
RESPONSE HEADERS: {:status=[200], cache-control=[max-age=0, private, must-revalidate], content-encoding=[gzip], content-length=[10465], content-security-policy=[default-src 'self'; child-src 'self'; connect-src 'self' matomo.openstreetmap.org; font-src 'none'; form-action 'self'; frame-ancestors 'self'; frame-src 'self'; img-src 'self' data: www.gravatar.com *.wp.com tile.openstreetmap.org *.tile.openstreetmap.org *.tile.thunderforest.com tileserver.memomaps.de *.openstreetmap.fr matomo.openstreetmap.org https://openstreetmap-user-avatars.s3.dualstack.eu-west-1.amazonaws.com https://openstreetmap-gps-images.s3.dualstack.eu-west-1.amazonaws.com; manifest-src 'self'; media-src 'none'; object-src 'self'; script-src 'self' matomo.openstreetmap.org; style-src 'self'; worker-src 'none'], content-type=[application/xml; charset=utf-8], date=[Thu, 27 Apr 2023 20:17:29 GMT], etag=[W/"964a54573583caafb86ed7bd3876bed5-gzip"], referrer-policy=[strict-origin-when-cross-origin], server=[Apache/2.4.54 (Ubuntu)], status=[200 OK], strict-transport-security=[max-age=31536000; includeSubDomains; preload], vary=[Origin,Accept-Encoding], x-content-type-options=[nosniff], x-download-options=[noopen], x-frame-options=[sameorigin], x-permitted-cross-domain-policies=[none], x-powered-by=[Phusion Passenger(R) 6.0.17], x-request-id=[ZErYWRo4dMYNK0nyfjfVkgAAqC0], x-runtime=[0.215565], x-xss-protection=[1; mode=block]}
SAX parsing completed in 3 ms
GET https://api.openstreetmap.org/api/0.6/changesets?changesets=63044136,79820076,88452267,16556307,17176809,42287988,55454399,56142506,25059469,51086072,20979901,114847527,63173059,15829749,87658300,100147197,40042596,88604480,34129920,47038660,89100102,62153151,21764931,50397454,89077568,91801453,88528720,67153431,88661843,15635184,64297419,24732505,53754132,42727594,15338184,60047795,61530509,50374956,99948507,24597317,88326001,46749948,88098679,88614783,100992061,88743795,88475516,58020168,27448290,89083663,38203407,87656304,28298213,85958485,88336136,20593524,86636391,46096624,56185098,31302536,61864441,89286404,74913327,52511096,69639804,64967044,88571674,19088253,88667926,16177833,88278812,73442817,41646104,18678648,89308981,41941013,88096558,24099584,98361267,88495927,69639765,27075563,78415578,16603773,29530947,70526641,53623221,67122823,66277755,46350410,89094086,16593527,72658655,16384623,22389696,49856621,69639857,72693452,89233355,33894558 -> HTTP_2 200 (375 ms; 10.5 kB)
RESPONSE HEADERS: {:status=[200], cache-control=[max-age=0, private, must-revalidate], content-encoding=[gzip], content-length=[10742], content-security-policy=[default-src 'self'; child-src 'self'; connect-src 'self' matomo.openstreetmap.org; font-src 'none'; form-action 'self'; frame-ancestors 'self'; frame-src 'self'; img-src 'self' data: www.gravatar.com *.wp.com tile.openstreetmap.org *.tile.openstreetmap.org *.tile.thunderforest.com tileserver.memomaps.de *.openstreetmap.fr matomo.openstreetmap.org https://openstreetmap-user-avatars.s3.dualstack.eu-west-1.amazonaws.com https://openstreetmap-gps-images.s3.dualstack.eu-west-1.amazonaws.com; manifest-src 'self'; media-src 'none'; object-src 'self'; script-src 'self' matomo.openstreetmap.org; style-src 'self'; worker-src 'none'], content-type=[application/xml; charset=utf-8], date=[Thu, 27 Apr 2023 20:17:30 GMT], etag=[W/"c6c5b19c552c222841eb3df8e7cac943-gzip"], referrer-policy=[strict-origin-when-cross-origin], server=[Apache/2.4.54 (Ubuntu)], status=[200 OK], strict-transport-security=[max-age=31536000; includeSubDomains; preload], vary=[Origin,Accept-Encoding], x-content-type-options=[nosniff], x-download-options=[noopen], x-frame-options=[sameorigin], x-permitted-cross-domain-policies=[none], x-powered-by=[Phusion Passenger(R) 6.0.17], x-request-id=[ZErYWho4dMYNK0nyfjfVlQAAki0], x-runtime=[0.214004], x-xss-protection=[1; mode=block]}
SAX parsing completed in 3 ms
GET https://api.openstreetmap.org/api/0.6/changesets?changesets=46321758,53785000,16622176,17148824,31648639,19125181,23964659,49631338,23952333,124128711,19063684,88778733,43681841,111272970,15350340,56306165,63445350,88414187,40042707,64004450,29424500,88494064,30131059,60725543,29406074,74219223,46186621,35641566,50397632,16663100,114492433,73905828,103015653,22104986,88500109,17386457,51235293,60506447,21179286,88121237,38582427,83121709,88960916,89288587,62624097,39250058,61524343,16952284,50102306,52132293,87414741,88813481,36493514,50397673,74862238,17798118,27743049,75091594,19442652,124128659,61292893,68843233,23071645,124093848,18428441,51015691,51134472,95446713,17205772,44634531,69308205,30132934,88135246,96454317,83324900,72068957,88817239,29760230,95352462,45992348,36317448,88086114,48865738,72966002,16694227,15778782,72380286,70295301,88168052,88528499,89253484,55414899,45220237,84573749,99761843,25008703,74180392,31265429,39178527,88684037 -> HTTP_2 200 (599 ms; 10.4 kB)
RESPONSE HEADERS: {:status=[200], cache-control=[max-age=0, private, must-revalidate], content-encoding=[gzip], content-length=[10680], content-security-policy=[default-src 'self'; child-src 'self'; connect-src 'self' matomo.openstreetmap.org; font-src 'none'; form-action 'self'; frame-ancestors 'self'; frame-src 'self'; img-src 'self' data: www.gravatar.com *.wp.com tile.openstreetmap.org *.tile.openstreetmap.org *.tile.thunderforest.com tileserver.memomaps.de *.openstreetmap.fr matomo.openstreetmap.org https://openstreetmap-user-avatars.s3.dualstack.eu-west-1.amazonaws.com https://openstreetmap-gps-images.s3.dualstack.eu-west-1.amazonaws.com; manifest-src 'self'; media-src 'none'; object-src 'self'; script-src 'self' matomo.openstreetmap.org; style-src 'self'; worker-src 'none'], content-type=[application/xml; charset=utf-8], date=[Thu, 27 Apr 2023 20:17:30 GMT], etag=[W/"ecbbc212779aa157bbd3e5ad3132d8ee-gzip"], referrer-policy=[strict-origin-when-cross-origin], server=[Apache/2.4.54 (Ubuntu)], status=[200 OK], strict-transport-security=[max-age=31536000; includeSubDomains; preload], vary=[Origin,Accept-Encoding], x-content-type-options=[nosniff], x-download-options=[noopen], x-frame-options=[sameorigin], x-permitted-cross-domain-policies=[none], x-powered-by=[Phusion Passenger(R) 6.0.17], x-request-id=[ZErYWho4dMYNK0nyfjfVlgAAhS0], x-runtime=[0.318229], x-xss-protection=[1; mode=block]}
SAX parsing completed in 2 ms
GET https://api.openstreetmap.org/api/0.6/changesets?changesets=16358324,88503819,30016158,34068819,18080325,69932925,28631785,93824713,88051294,17461869,88686112,67628906,37785902,74647324,87658070,86867545,39526702,74143517,40042772,19503697,71305016,61831381,75325191,87656003,50632824,74051348,114879909,52549726,89339603,88821448,88039099,100365951,79404862,87830201,22608593,50374794,68843408,25635339,50397325,35307914,64964687,69449654,88143576,124128510,25002721,88133340,88639189,48996683,74815445,69361539,58118355,88364770,20495007,33053270,83322712,87885452,88157936,80740221,22149862,102573477,15223633,74385366,71340026,22459117,85350061,89247466,85341866,83087181,38511092,16393010,25426506,20683504,66938934,48996646,87920364,15223610,58312875,88405654,88497810,22155911,74233785,69640176,43532623,50374872,74172338,15717173,48859445,88159901,25584219,55242900,88467099,48775479,26153581,19575497,72677298,34572773,70627271,73029545,24865422,52424931 -> HTTP_2 200 (613 ms; 9.91 kB)
RESPONSE HEADERS: {:status=[200], cache-control=[max-age=0, private, must-revalidate], content-encoding=[gzip], content-length=[10150], content-security-policy=[default-src 'self'; child-src 'self'; connect-src 'self' matomo.openstreetmap.org; font-src 'none'; form-action 'self'; frame-ancestors 'self'; frame-src 'self'; img-src 'self' data: www.gravatar.com *.wp.com tile.openstreetmap.org *.tile.openstreetmap.org *.tile.thunderforest.com tileserver.memomaps.de *.openstreetmap.fr matomo.openstreetmap.org https://openstreetmap-user-avatars.s3.dualstack.eu-west-1.amazonaws.com https://openstreetmap-gps-images.s3.dualstack.eu-west-1.amazonaws.com; manifest-src 'self'; media-src 'none'; object-src 'self'; script-src 'self' matomo.openstreetmap.org; style-src 'self'; worker-src 'none'], content-type=[application/xml; charset=utf-8], date=[Thu, 27 Apr 2023 20:17:31 GMT], etag=[W/"706d01c182124281da2a33ccaaede46d-gzip"], referrer-policy=[strict-origin-when-cross-origin], server=[Apache/2.4.54 (Ubuntu)], status=[200 OK], strict-transport-security=[max-age=31536000; includeSubDomains; preload], vary=[Origin,Accept-Encoding], x-content-type-options=[nosniff], x-download-options=[noopen], x-frame-options=[sameorigin], x-permitted-cross-domain-policies=[none], x-powered-by=[Phusion Passenger(R) 6.0.17], x-request-id=[ZErYWxo4dMYNK0nyfjfVlwAAny0], x-runtime=[0.273742], x-xss-protection=[1; mode=block]}
SAX parsing completed in 2 ms
GET https://api.openstreetmap.org/api/0.6/changesets?changesets=42742130,17253637,102949409,25693580,66088952,18607378,20122936,74188903,15078635,64966610,28984756,15496444,88672601,89319746,38162004,26731907,71039023,18609409,102949432,87724321,51484428,32966121,112992931,23637322,74164297,55408488,33469912,44269189,89082216,88258922,61554561,52369202,27637130,24511814,29328783,56379241,67520574,17466675,88410495,97325559,32110034,24519990,88680712,98363806,30342541,39818780,55334671,29306363,88422657,65159083,57513767,42426057,51519300,68567104,69658736,88498463,50374491,26721739,84906323,18081102,30438798,19828047,38362664,88783144,32058766,88168745,28569050,26861043,43796214,31982979,39814708,34371168,71333933,38415931,74074136,38948385,86002004,81471661,88473918,88090938,25292282,38950441,68839521,28630473,16641149,31235420,36228776,88197569,16903552,124099556,89201115,34961048,88822214,23309804,41090813,26737925,31114568,53884834,124130290,88095189 -> HTTP_2 200 (613 ms; 10.3 kB)
RESPONSE HEADERS: {:status=[200], cache-control=[max-age=0, private, must-revalidate], content-encoding=[gzip], content-length=[10545], content-security-policy=[default-src 'self'; child-src 'self'; connect-src 'self' matomo.openstreetmap.org; font-src 'none'; form-action 'self'; frame-ancestors 'self'; frame-src 'self'; img-src 'self' data: www.gravatar.com *.wp.com tile.openstreetmap.org *.tile.openstreetmap.org *.tile.thunderforest.com tileserver.memomaps.de *.openstreetmap.fr matomo.openstreetmap.org https://openstreetmap-user-avatars.s3.dualstack.eu-west-1.amazonaws.com https://openstreetmap-gps-images.s3.dualstack.eu-west-1.amazonaws.com; manifest-src 'self'; media-src 'none'; object-src 'self'; script-src 'self' matomo.openstreetmap.org; style-src 'self'; worker-src 'none'], content-type=[application/xml; charset=utf-8], date=[Thu, 27 Apr 2023 20:17:31 GMT], etag=[W/"f087e57d21e4cc332e59791dbc03863a-gzip"], referrer-policy=[strict-origin-when-cross-origin], server=[Apache/2.4.54 (Ubuntu)], status=[200 OK], strict-transport-security=[max-age=31536000; includeSubDomains; preload], vary=[Origin,Accept-Encoding], x-content-type-options=[nosniff], x-download-options=[noopen], x-frame-options=[sameorigin], x-permitted-cross-domain-policies=[none], x-powered-by=[Phusion Passenger(R) 6.0.17], x-request-id=[ZErYWxo4dMYNK0nyfjfVmQAAoS0], x-runtime=[0.257034], x-xss-protection=[1; mode=block]}
SAX parsing completed in 2 ms
GET https://api.openstreetmap.org/api/0.6/changesets?changesets=101880455,61527869,81078338,26852608,57972717,100991647,86340025,89188812,88074654,89205232,17237413,93442355,88502753,89094631,31960398,45057541,59514663,88805856,30901627,57182149,37557906,88816121,30866787,44043797,88959480,45635086,64970599,88459773,31294823,54030210,61568785,71100550,36329105,66934595,41381576,88379783,88775055,35907300,51486678,91836849,23248296,34911962,106545814,20663787,74055866,88142228,110252614,46210582,15078462,20583905,18451909,88693141,89166221,114108944,66740004,71153884,28845402,41307797,66508565,29547814,15078400,29230422,73377973,74119296,88447407,98521397,25589098,70230211,88529320,93364606,15813633,19989953,50399216,68208865,15305754,54181833,33918705,51824610,17751547,74217626,63375167,17323508,88811959,48373279,76298353,87724485,39833439,100376825,18586655,71813443,96010431,56198737,28933303,72735067,66795243,27444407,20599852,89344068,15957481,36306738 -> HTTP_2 200 (611 ms; 10.2 kB)
RESPONSE HEADERS: {:status=[200], cache-control=[max-age=0, private, must-revalidate], content-encoding=[gzip], content-length=[10468], content-security-policy=[default-src 'self'; child-src 'self'; connect-src 'self' matomo.openstreetmap.org; font-src 'none'; form-action 'self'; frame-ancestors 'self'; frame-src 'self'; img-src 'self' data: www.gravatar.com *.wp.com tile.openstreetmap.org *.tile.openstreetmap.org *.tile.thunderforest.com tileserver.memomaps.de *.openstreetmap.fr matomo.openstreetmap.org https://openstreetmap-user-avatars.s3.dualstack.eu-west-1.amazonaws.com https://openstreetmap-gps-images.s3.dualstack.eu-west-1.amazonaws.com; manifest-src 'self'; media-src 'none'; object-src 'self'; script-src 'self' matomo.openstreetmap.org; style-src 'self'; worker-src 'none'], content-type=[application/xml; charset=utf-8], date=[Thu, 27 Apr 2023 20:17:32 GMT], etag=[W/"803fbd03fa41e93701ef4b4d6157533f-gzip"], referrer-policy=[strict-origin-when-cross-origin], server=[Apache/2.4.54 (Ubuntu)], status=[200 OK], strict-transport-security=[max-age=31536000; includeSubDomains; preload], vary=[Origin,Accept-Encoding], x-content-type-options=[nosniff], x-download-options=[noopen], x-frame-options=[sameorigin], x-permitted-cross-domain-policies=[none], x-powered-by=[Phusion Passenger(R) 6.0.17], x-request-id=[ZErYXBo4dMYNK0nyfjfVmgAAky0], x-runtime=[0.251775], x-xss-protection=[1; mode=block]}
SAX parsing completed in 3 ms
GET https://api.openstreetmap.org/api/0.6/changesets?changesets=106545986,114940869,52020743,16938016,33375455,88780910,54035997,88815725,34961202,53528087,51402297,69171510,72692092,88813690,74006876,50398777,87928839,17151035,40515370,85778527,89313297,28067048,26733778,63702665,69640544,87648383,83167676,17529921,100918089,21406733,18017373,70785399,61574904,23247927,56645173,88416285,88616981,43964353,73033989,88528934,96620708,100858725,88807458,18099326,88545321,30143674,70744390,59182837,23956505,55858720,33816440,87658562,71493950,88748086,49629066,23360742,69431713,88299713,88547522,86040742,89092296,87656624,87050426,18099353,24491256,33804172,102568850,88199376,100575338,124130035,64548431,100128878,20602017,34545556,60689925,17904780,18273462,74101192,23209158,18101436,88799458,21595360,87689367,88746212,90788039,16854194,99762241,15238483,61849106,88584441,18261206,52121307,19451113,88256640,17259713,75121074,17466566,88635527,61388408,33806290 -> HTTP_2 200 (623 ms; 10.4 kB)
RESPONSE HEADERS: {:status=[200], cache-control=[max-age=0, private, must-revalidate], content-encoding=[gzip], content-length=[10613], content-security-policy=[default-src 'self'; child-src 'self'; connect-src 'self' matomo.openstreetmap.org; font-src 'none'; form-action 'self'; frame-ancestors 'self'; frame-src 'self'; img-src 'self' data: www.gravatar.com *.wp.com tile.openstreetmap.org *.tile.openstreetmap.org *.tile.thunderforest.com tileserver.memomaps.de *.openstreetmap.fr matomo.openstreetmap.org https://openstreetmap-user-avatars.s3.dualstack.eu-west-1.amazonaws.com https://openstreetmap-gps-images.s3.dualstack.eu-west-1.amazonaws.com; manifest-src 'self'; media-src 'none'; object-src 'self'; script-src 'self' matomo.openstreetmap.org; style-src 'self'; worker-src 'none'], content-type=[application/xml; charset=utf-8], date=[Thu, 27 Apr 2023 20:17:32 GMT], etag=[W/"3cbdb8241a84fd246da79f88b8fe3573-gzip"], referrer-policy=[strict-origin-when-cross-origin], server=[Apache/2.4.54 (Ubuntu)], status=[200 OK], strict-transport-security=[max-age=31536000; includeSubDomains; preload], vary=[Origin,Accept-Encoding], x-content-type-options=[nosniff], x-download-options=[noopen], x-frame-options=[sameorigin], x-permitted-cross-domain-policies=[none], x-powered-by=[Phusion Passenger(R) 6.0.17], x-request-id=[ZErYXBo4dMYNK0nyfjfVnAAApS0], x-runtime=[0.244171], x-xss-protection=[1; mode=block]}
SAX parsing completed in 1 ms
GET https://api.openstreetmap.org/api/0.6/changesets?changesets=17413341,33824720,87562476,89110682,50374358,41424801,18359490,89071764,58482337,89342093,30067754,56585918,86022337,69663173,86782153,16866535,39159731,28568660,39815093,74011010,22469819,29969446,69659081,88486057,44791643,61566553,88156338,72071615,88152245,19088598,58261135,64243235,15078684,37720005,15236371,33976315,56184155,85797662,100363263,79172286,78121647,19769120,88562506,88210255,18224921,88578903,41804911,70785581,100500456,88288085,18224897,88613727,17205023,88494943,63108574,27174784,38513748,70140431,89101166,18988802,20739871,35316798,48176377,61599105,47035617,87657237,88013586,21247851,54556017,88787832,17327921,88472447,88619892,88818550,74078804,37506051,24581943,30285710,18224983,89293584,46220420,86051686,85902166,24872757,32855995,88820485,89598744,88195861,28645344,93615041,15609529,15750823,88222485,88021862,20600672,94163910,32729000,70642281,83288746,15926957 -> HTTP_2 200 (412 ms; 10.1 kB)
RESPONSE HEADERS: {:status=[200], cache-control=[max-age=0, private, must-revalidate], content-encoding=[gzip], content-length=[10341], content-security-policy=[default-src 'self'; child-src 'self'; connect-src 'self' matomo.openstreetmap.org; font-src 'none'; form-action 'self'; frame-ancestors 'self'; frame-src 'self'; img-src 'self' data: www.gravatar.com *.wp.com tile.openstreetmap.org *.tile.openstreetmap.org *.tile.thunderforest.com tileserver.memomaps.de *.openstreetmap.fr matomo.openstreetmap.org https://openstreetmap-user-avatars.s3.dualstack.eu-west-1.amazonaws.com https://openstreetmap-gps-images.s3.dualstack.eu-west-1.amazonaws.com; manifest-src 'self'; media-src 'none'; object-src 'self'; script-src 'self' matomo.openstreetmap.org; style-src 'self'; worker-src 'none'], content-type=[application/xml; charset=utf-8], date=[Thu, 27 Apr 2023 20:17:33 GMT], etag=[W/"b6b2da38b5631d85fc4b1ba7fcdcacde-gzip"], referrer-policy=[strict-origin-when-cross-origin], server=[Apache/2.4.54 (Ubuntu)], status=[200 OK], strict-transport-security=[max-age=31536000; includeSubDomains; preload], vary=[Origin,Accept-Encoding], x-content-type-options=[nosniff], x-download-options=[noopen], x-frame-options=[sameorigin], x-permitted-cross-domain-policies=[none], x-powered-by=[Phusion Passenger(R) 6.0.17], x-request-id=[ZErYXRo4dMYNK0nyfjfVnQAAgS0], x-runtime=[0.231637], x-xss-protection=[1; mode=block]}
SAX parsing completed in 2 ms
GET https://api.openstreetmap.org/api/0.6/changesets?changesets=89185038,30150547,34280532,89203470,88808235,36228171,88040284,69323332,51901809,17205089,87655249,50400618,36305989,85341050,35490925,19769180,99634052,15550103,88240952,89199405,88304445,15609455,89197520,61994293,85687199,88677325,88034234,42621964,49189992,70480564,96437048,24850419,88097736,18220954,89336799,88656839,85894031,52096393,32147324,58254830,84161426,88507359,52368770,42288155,74199794,46840917,16600669,107516104,21145571,53491089,48063615,27111230,88361952,111122491,44620800,36318336,36062349,89240573,86072204,17921983,18220985,15337050,21135345,87657354,49699918,17205183,43056169,67560124,58049992,82371112,15240747,50400706,76751440,88499073,89303960,28061536,64249112,89471897,67109587,124101046,21522332,46670878,54277611,73681587,88812444,21288861,19087352,27766652,58238361,111339587,49007624,52960715,85806078,21766056,30394153,36334785,80808506,100074268,49189891,26820471 -> HTTP_2 200 (435 ms; 10.3 kB)
RESPONSE HEADERS: {:status=[200], cache-control=[max-age=0, private, must-revalidate], content-encoding=[gzip], content-length=[10590], content-security-policy=[default-src 'self'; child-src 'self'; connect-src 'self' matomo.openstreetmap.org; font-src 'none'; form-action 'self'; frame-ancestors 'self'; frame-src 'self'; img-src 'self' data: www.gravatar.com *.wp.com tile.openstreetmap.org *.tile.openstreetmap.org *.tile.thunderforest.com tileserver.memomaps.de *.openstreetmap.fr matomo.openstreetmap.org https://openstreetmap-user-avatars.s3.dualstack.eu-west-1.amazonaws.com https://openstreetmap-gps-images.s3.dualstack.eu-west-1.amazonaws.com; manifest-src 'self'; media-src 'none'; object-src 'self'; script-src 'self' matomo.openstreetmap.org; style-src 'self'; worker-src 'none'], content-type=[application/xml; charset=utf-8], date=[Thu, 27 Apr 2023 20:17:34 GMT], etag=[W/"f6b29c127c961af5a60ecd614f60c031-gzip"], referrer-policy=[strict-origin-when-cross-origin], server=[Apache/2.4.54 (Ubuntu)], status=[200 OK], strict-transport-security=[max-age=31536000; includeSubDomains; preload], vary=[Origin,Accept-Encoding], x-content-type-options=[nosniff], x-download-options=[noopen], x-frame-options=[sameorigin], x-permitted-cross-domain-policies=[none], x-powered-by=[Phusion Passenger(R) 6.0.17], x-request-id=[ZErYXho4dMYNK0nyfjfVngAAkC0], x-runtime=[0.255654], x-xss-protection=[1; mode=block]}
SAX parsing completed in 2 ms
GET https://api.openstreetmap.org/api/0.6/changesets?changesets=50974203,29781811,17205232,79481418,19466192,63845684,15824396,18157547,88157119,89186896,88044092,100962567,88365632,60046482,112896444,38405453,26392218,124100717,43355550,21210701,68594438,100649709,20043298,65703161,85383685,74277754,86526518,65737972,97546959,31733447,83082192,59716779,53130255,22134393,15724486,99760860,87656982,86155794,89930348,100145865,89080445,17937957,64220389,88463987,15241168,86059544,70286090,68969255,88175228,50562109,20739708,30434960,94540514,15790001,87657080,73448226,17888851,70923129,30893714,75799490,48426405,87614079,75523016,21693954,27743981,74777406,34626908,35515725,87202400,56396815,86090359,37997851,42552784,101798219,17673814,17475156,33539745,16424869,101855602,88402471,28630738,17649252,78853082,15239043,99762841,16230299,15241094,20020831,93428479,18327139,79881143,44193242,50400377,87687746,21253695,88523462,37354938,88453830,70063017,34170252 -> HTTP_2 200 (425 ms; 11.5 kB)
RESPONSE HEADERS: {:status=[200], cache-control=[max-age=0, private, must-revalidate], content-encoding=[gzip], content-length=[11768], content-security-policy=[default-src 'self'; child-src 'self'; connect-src 'self' matomo.openstreetmap.org; font-src 'none'; form-action 'self'; frame-ancestors 'self'; frame-src 'self'; img-src 'self' data: www.gravatar.com *.wp.com tile.openstreetmap.org *.tile.openstreetmap.org *.tile.thunderforest.com tileserver.memomaps.de *.openstreetmap.fr matomo.openstreetmap.org https://openstreetmap-user-avatars.s3.dualstack.eu-west-1.amazonaws.com https://openstreetmap-gps-images.s3.dualstack.eu-west-1.amazonaws.com; manifest-src 'self'; media-src 'none'; object-src 'self'; script-src 'self' matomo.openstreetmap.org; style-src 'self'; worker-src 'none'], content-type=[application/xml; charset=utf-8], date=[Thu, 27 Apr 2023 20:17:34 GMT], etag=[W/"94a108223cca4807a23fd51c0b7269fd-gzip"], referrer-policy=[strict-origin-when-cross-origin], server=[Apache/2.4.54 (Ubuntu)], status=[200 OK], strict-transport-security=[max-age=31536000; includeSubDomains; preload], vary=[Origin,Accept-Encoding], x-content-type-options=[nosniff], x-download-options=[noopen], x-frame-options=[sameorigin], x-permitted-cross-domain-policies=[none], x-powered-by=[Phusion Passenger(R) 6.0.17], x-request-id=[ZErYXho4dMYNK0nyfjfVoAAAnC0], x-runtime=[0.249489], x-xss-protection=[1; mode=block]}
SAX parsing completed in 3 ms
GET https://api.openstreetmap.org/api/0.6/changesets?changesets=88492739,19087013,28630578,29273656,88961733,19451554,17292937,49263969,42708251,85430918,15798115,21104336,67627925,76559108,52368522,15241072,16989849,61791284,74142709,79416101,75246556,15241034,53511315,63120485,89930486,53529756,93825594,50398383,34594237,95640071,64965738,88765180,88941307,96598649,15241041,33402439,19246750,72029175,17344182,89215727,67550150,39648664,88679053,19418862,25566784,76596055,22564498,95853181,33727951,48997668,75641683,107784638,31294985,55207070,15241021,70824942,16617258,36300286,88136346,15241011,74122166,87722724,70599635,17379054,88244907,21452460,49038592,18220795,86002383,15240986,33154572,17694453,88545983,30131761,41805187,54277314,18607844,47842597,78789453 -> HTTP_2 200 (379 ms; 8.56 kB)
RESPONSE HEADERS: {:status=[200], cache-control=[max-age=0, private, must-revalidate], content-encoding=[gzip], content-length=[8762], content-security-policy=[default-src 'self'; child-src 'self'; connect-src 'self' matomo.openstreetmap.org; font-src 'none'; form-action 'self'; frame-ancestors 'self'; frame-src 'self'; img-src 'self' data: www.gravatar.com *.wp.com tile.openstreetmap.org *.tile.openstreetmap.org *.tile.thunderforest.com tileserver.memomaps.de *.openstreetmap.fr matomo.openstreetmap.org https://openstreetmap-user-avatars.s3.dualstack.eu-west-1.amazonaws.com https://openstreetmap-gps-images.s3.dualstack.eu-west-1.amazonaws.com; manifest-src 'self'; media-src 'none'; object-src 'self'; script-src 'self' matomo.openstreetmap.org; style-src 'self'; worker-src 'none'], content-type=[application/xml; charset=utf-8], date=[Thu, 27 Apr 2023 20:17:34 GMT], etag=[W/"2174c29e6e810d78af0103a8bcbae134-gzip"], referrer-policy=[strict-origin-when-cross-origin], server=[Apache/2.4.54 (Ubuntu)], status=[200 OK], strict-transport-security=[max-age=31536000; includeSubDomains; preload], vary=[Origin,Accept-Encoding], x-content-type-options=[nosniff], x-download-options=[noopen], x-frame-options=[sameorigin], x-permitted-cross-domain-policies=[none], x-powered-by=[Phusion Passenger(R) 6.0.17], x-request-id=[ZErYXho4dMYNK0nyfjfVpAAAsC0], x-runtime=[0.199664], x-xss-protection=[1; mode=block]}
SAX parsing completed in 2 ms

I think the problem lies with the first call, https://api.openstreetmap.org/api/0.6/relation/2769137/history. Unfortunately, it doesn't look like we have the size to print to console there, so I had to wait to download it again.

The history for relation 2769137 is 190M uncompressed. When compressed with gzip, it is 22M (22761298 bytes). Which is over the CGIMap default limits (204800, no clue what unit that is in though).

I believe the changesets API calls proceed to work since they are hitting the ruby endpoints instead of the CGIMap endpoints. But the CGIMap endpoints controls the stuff most editors care about.

In order to fix this bug, changes need to be made on the OSM website api. Specifically, history calls must be paginated.

comment:4 by mmd, 13 months ago

For the time being maybe use the multi fetch instead and provide exact version numbers you want to download: https://wiki.openstreetmap.org/wiki/API_v0.6#Multi_fetch:_GET_/api/0.6/%5Bnodes%7Cways%7Crelations%5D?%23parameters

comment:5 by mmd, 13 months ago

Pagination would have to be added to Rails first by the way, before any changes could be done to CGImap.

comment:6 by GerdP, 4 months ago

If I got that right we could replace our api call
https://api.openstreetmap.org/api/0.6/relation/2769137/history by a bunch of calls like

https://api.openstreetmap.org/api/0.6/relations?relations=2769137v1,2769137v2,2769137v3...
https://api.openstreetmap.org/api/0.6/relations?relations=2769137v128,2769137v129,2769137v130...
...
https://api.openstreetmap.org/api/0.6/relations?relations=2769137v1699,2769137v1700
where each call would list maybe a 127 ids or so.
What would be the difference on the server side? Would that really improve something?

Modify Ticket

Change Properties
Set your email in Preferences
Action
as new The owner will remain team.
as The resolution will be set. Next status will be 'closed'.
to The owner will be changed from team to the specified user.
Next status will be 'needinfo'. The owner will be changed from team to skyper.
as duplicate The resolution will be set to duplicate. Next status will be 'closed'. The specified ticket will be cross-referenced with this ticket.
The owner will be changed from team to anonymous. Next status will be 'assigned'.

Add Comment


E-mail address and name can be saved in the Preferences .
 
Note: See TracTickets for help on using tickets.