Shared Tile Cache

This document describes a common caching layout that can used to share imagery tiles among different applications.

Status

Draft

Author/s

Yuri D'Elia

Last revised

2014-03-23

Premise

We have several desktop applications that are querying various tile servers and cache their imagery locally, however each application is currently storing and managing their cache independently.

This leads to a huge waste of space (especially for aerial imagery) in the form of duplicated caches with essentially the same content, and needless server re-querying.

In this document we describe a common cache format, layout and methods to access, create and modify tile caches that applications may choose to adhere to share their imagery cache.

Cache directory root

Unix/Linux

The cache directory root is located in the $XDG_CACHE_HOME/tiles (usually "~/.cache/tiles"). See the ​XDG Base Directory Specification to locate this directory correctly.

Windows XP

C:\Documents and Settings\%USERNAME%\Local Settings\Application Data\Shared Tile Cache\

Windows 7 or Vista

C:\Users\%USERNAME%\AppData\Local\Shared Tile Cache\

Tile directory layout

Under the cache directory root, each directory contains an independent tile cache directory. Each directory can be any valid arbitrary directory name, but should be an understandable short ID for the tile provider (such as "Mapnik", "Bing", etc). Applications must not rely on the short ID name, but the must instead parse the cache properties as outlined in #CacheDiscovery.

The content of the tile cache is the same as followed by ​Slippy Map, with the difference that each file can be located by the pattern "/zoom/x/y.extension".

Each tile can optionally have additional metadata associated with it, contained in the file following the pattern "/zoom/x/y.extension.ini", as discussed in #TileManagement.

At the root level, each cache must provide a #CacheProperties file, named "/cache.ini". This file provides several know attributes, such as the "extension" required to locate each tile.

An example cache directory layout is outlined below:

~/.cache/tiles/
~/.cache/tiles/Mapnik/
~/.cache/tiles/Mapnik/cache.ini
~/.cache/tiles/Mapnik/14/8709/5792.png
~/.cache/tiles/Mapnik/14/8709/5793.png.ini
~/.cache/tiles/Bing/
~/.cache/tiles/Bing/cache.ini
...

Cache properties

The "Cache properties" file defines data associated with the cache itself. The file is named "cache.ini" an is located at the root level of the cache:

~/.cache/tiles/Mapnik/cache.ini

The file format is a simple text file, UTF-8 encoded, with each line containing a "key=value" pair. Applications must ignore any key that they don't know. Applications must also preserve all keys when the file is being updated. Any other arbitrary key can be included.

The following mandatory keys are defined:

name

A long descriptive name of the tile provider for display purposes.

url

Base URL of the tile provider.

type

URL schema of the tile provider (currently only "TMS")

extension

Image file extension ("png" or "jpg").

size

Maximal size of the cache in bytes, with 0 meaning unlimited.

age

Minimum tile age (in seconds)

An example properties file is shown below:

name=OSM Mapnik
url=http://tile.openstreetmap.org
type=TMS
extension=png
size=0
age=604800

Cache discovery

When initializing cache management, applications should list all available caches inside the shared tile cache, and parse their respective cache properties.

Applications must not depending on the ID (given by the directory name), but should instead locate the cache by matching on "url", "type" and possibly "extension". If a suitable match is found, the existing ID/directory should be used for further access.

If no suitable directory can be found, a new directory can be safely created inside the shared root. The application is free to choose any valid directory name, but should use a short, explicative name.

Tile management

Assuming an empty tile cache, the application is free to create any file by downloading it from the provider and using the appropriate "/zoom/x/y.extension" pattern, creating missing directories.

Additional data can be stored for each tile, using the file named "/zoom/x/y.extension.ini". This file follows the same convention as the #CacheProperties file, and is a simple UTF-8 encoded "key=value" file containing arbitrary keys which are application-defined. Again, applications must ignore unknown keys, can add new keys, but must also preserve any existing one when updating the content. Applications are free to create/update the metadata file at any time.

When probing the cache for an existing tile, the application must check for the presence of the tile image "/zoom/x/y.extension" and check it's modification time. If the tile has been created or modified less than the specified cache age, the application must use the existing tile.

If the tile is older, it can be considered stale and the application is free to query the server again, supplying an "If-Modified-Since" header. For this reason, applications must not modify the file modification time of the tile image except when updating the tile.

When an existing tile is being updated or removed, any associated "/zoom/x/y.extension.ini" file must be deleted as well, irregardless of the contents.

When an existing tile is being deleted, in addition to removing the associated "ini" file, the application must also try to remove the "/zoom/x/" and "/zoom/" directories when empty.

When the cache size is defined as 0, applications must not perform any cache pruning. If size is anything greater than 0, each application can choose his own policy for tile expiration (which should usually be done by least-recently-used tile which is not currently visible), deleting each tile in turn until the size of the entire directory tree is smaller or equal to the defined cache size.

Concurrent access

To enable "opportunistic" concurrent cache access to each tile cache, we define the following simple rules:

• After deciding to create or update a tile, the image file ("/zoom/x/y.extension") must be moved atomically into the directory tree (and not created in-place), replacing any existing file (if any). At attempt can then be made to read the associated "ini" file. The "ini" file must be ignored if it's create/modification time is earlier than the image file and must then be deleted if not replaced.

• When deleting an existing tile, an attempt to remove "ini" file must be made first, followed by the image file.

• Applications must not expect written tiles (or their metadata) to exist on disk or be the same as previously written.

• Updates to the cache properties and tile metadata must be similarly performed using atomic replacement. Since no locking is performed, applications almost must not cache their contents.