Modify

Opened 7 years ago

Closed 6 years ago

Last modified 6 years ago

#8581 closed defect (fixed)

Embedded SVG leads to very high memory consumption

Reported by: imagic Owned by: team
Priority: major Milestone: 14.01
Component: Core mappaint Version: latest
Keywords: SVG Cc:

Description

When using embedded SVGs as image/symbols the memory consumption of JOSM rises very fast.

Example of an embedded SVG:
repeat-image: "data:image/svg+xml,<svg><rect x='10' y='10' width='50' height='20' fill='white' stroke='blue' stroke-width='2px'/><line x1='18' y1='37' x2='50' y2='5' stroke='black' stroke-width='1.5px'/></svg>";

Currently if multiple images have the same SVG code they are rendered and cached independently which lead to massive memory consumption. Embedded SVGs should be cached based on their actual SVG code to prevent this.

Attached is a JOSM style that heavily depends on embedded SVGs. A region that quickly lead to the issue is http://www.openstreetmap.org/?lat=47.06554&lon=15.43302&zoom=17&layers=M . When viewing even a small area in this region memory consumption will raise by multiples of 100MB and quickly lead to an out of memory exception.

Attachments (1)

RoadAttributes.mapcss.zip (36.9 KB) - added by imagic 7 years ago.
JOSM style with embedded SVGs

Download all attachments as: .zip

Change History (10)

Changed 7 years ago by imagic

Attachment: RoadAttributes.mapcss.zip added

JOSM style with embedded SVGs

comment:1 Changed 7 years ago by akks

I did not see the code of renderer, but I think that much harder problem with cpu and memory consumption is caused by huge number of evaluated embedded SVGs in the style file. Samples:
turn_svg_slight_left_forward: eval(concat("<g transform='translate(0,50) scale(1.0,-1.0)'>",prop(turn_svg_slight_right_forward),"</g>"));

t_test: eval(regexp_test("^(.*;)*slight_right(;.*)*$",prop(t))); turn_image_forward_3: eval(prop(t_test)?concat(prop(turn_image_forward_3),"<g transform='scale(",prop(t_scale)," 1.0)'>",prop(turn_svg_slight_right_forward),"</g>"):prop(turn_image_forward_3)); t_scale: eval(prop(t_test)?(prop(t_scale)*0.666):prop(t_scale));

I am not sure it can be reworked so that image will not be constructed every time. (Or there is only limited number of images generated?).

Last edited 7 years ago by akks (previous) (diff)

comment:2 in reply to:  1 Changed 7 years ago by imagic

Replying to akks:

(Or there is only limited number of images generated?).

Exactly. Consider the images for turn lanes: the style supports twelve values. Theoretically we could combine all of them which would give us 4095 possible images. Practically the number is much lower. Most road markings are either combination of the seven values through, right, slight_right, sharp_right, left, slight_left and sharp_left or are represented only by one single value (e.g. merge_to_right is usually not used in any combination). So we would end up with (another theoretically) maximum of 127+5=132 images. But even that number is much too high as most markings on turning lanes usually can be represented by one or two values (combined), sometimes three. So if we just take all single values (12) and add the most common combinations (through+left/right, through+slight_left/slight_right) we end up with 16 images which is a rather good estimate for the number of images needed for turning lanes on average.
All the numbers above have to be doubled, because I need images for forward and backward direction. So summing everything up and adding a little buffer I would say we are far below 100 images. All of them 100x50 which are 5,000 pixel and approx 20KB (32bpp) so we should need something around a few megs for those images.

I also use embedded SVGs for traffic signs, which again could be (theoretically) a few thousands. Practically I would guess something around 10-20.

Embedded SVGs have great potential. So fixing this would really be a leap forward for styles in JOSM.

comment:3 Changed 7 years ago by imagic

As I don't have the time to take a look into the code just a few thoughts:
I assume you have somewhere a method which takes the SVG code as input and returns some kind of image object. Then the issue could be solved by implementing a wrapper around that method.

I'll assume the following:

  • A class Image which stores your image data
  • A method createSVG(String) which returns an Image for the given svg code
  • A method getHashcode(String) which implements some kind of hash code. The default .hashCode() is not appropriate in this case IMO, as it would be too short and collisions would occur too often. Maybe simply MD5 from MessageDigest would do the trick.

Now some (more-or-less pseudo) code for the wrapper:

Map<Object,SoftReference<Image>> svgCache=new TreeMap<Object,SoftReference<Image>>();

public Image createCachedSVG(String svgcode)
{

Image image=null;

Try to get the image from the cache
Object hash=getHashcode(svgcode);
SoftReference<Image> ref=svgCache.get(hash);
if (ref!=null)
{

image=ref.get();

Check if the cache was cleared by the garbage collector; if so remove the entry from the map
if (image==null) svgCache.remove(hash);

}

Was the image cached? If not, create it and store it in the cache
if (image==null)
{

image=createSVG(svgcode);
if (image!=null) svgCode.put(hash,new SoftReference<Image>(image));

}

Return the image
return image;

}

Why a hash code instead of the complete svg code? Just to save memory, because the svg code might be really large. Collisions shouldn't be any problem as their occurrence is more theoretically. If the hash code doesn't make sense in your opinion, just take the svg code instead. The hash code would also slow down image creation for embedded svg images.

comment:4 Changed 7 years ago by Manu1400

Keywords: SVG added

comment:5 Changed 6 years ago by anonymous

Would be nice, if this bug would be solved. Waiting for even nicer lane attributes to be released :-)
Holger

comment:6 Changed 6 years ago by bastiK

Resolution: fixed
Status: newclosed

In [6747] - #8581 - Embedded SVG leads to very high memory consumption

comment:7 Changed 6 years ago by bastiK

I'm using the data string directly as cache key, so there may (or may not) be problems with excessively large SVG data URLs.

comment:8 Changed 6 years ago by imagic

Great - thanks a lot! I'll update the style as soon as the next tested version is available.

comment:9 Changed 6 years ago by Don-vip

Milestone: 14.01

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain team.
as The resolution will be set.
The resolution will be deleted.

Add Comment


E-mail address and name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.