Shared Tile Cache

This document describes a common caching layout that can be used to share imagery tiles among different applications.

Status

Draft

Author/s

Yuri D'Elia

Last revised

2014-03-23

Minor revisions

Lists of concerns addressed since the initial draft:

• 2014-03-24: Name of the cache "ID"/directory restricted to 20 characters (Robert Norris, from Viking development)
• 2014-03-24: Spaces removed from Windows/OSX shared tile root (Don-vip, JOSM)
• 2014-03-25: Use an "OSM" prefix for the shared tile root to limit namespace squatting (Kevin Krammer, Marble)

To encourage discussions about the spec, please don't edit this page directly. Use a relevant mailing list or this bug report: #9813

Premise

We have several desktop applications that are querying various tile servers and cache their imagery locally, however each application is currently storing and managing their cache independently.

This leads to a huge waste of space (especially for aerial imagery) in the form of duplicated caches with essentially the same content, and bandwidth due to needless server re-querying.

In this document we describe a common cache format, layout and methods to access, create and modify tile caches that applications may choose to adhere to share their imagery cache.

Cache directory root

Unix/Linux

The cache directory root is located in the $XDG_CACHE_HOME/osm/tiles (usually "~/.cache/osm/tiles"). See the ​XDG Base Directory Specification to locate this directory correctly.

Windows XP

%UserProfile%\Local Settings\Application Data\OSM\SharedTileCache\

Windows Vista, 7, 8

%UserProfile%\AppData\Local\OSM\SharedTileCache\

Mac OS X

NSCachesDirectory/OSM/SharedTileCache (usually "~/Library/Caches/OSM/SharedTileCache"). Use the NSPathUtilities to locate the cache directory using ​NSCachesDirectory.

Tile directory layout

Under the cache directory root, each directory contains an independent tile cache directory. Each directory can be any valid arbitrary directory name, but should be an understandable short ID for the tile provider (such as "Mapnik", "Bing", etc). Applications must not rely on the short ID name, but must instead parse the cache properties as outlined in #CacheDiscovery.

The content of the tile cache is the same as followed by ​Slippy Map, with the difference that each file can be located by the pattern "/zoom/x/y.extension".

Each tile can optionally have additional metadata associated with it, contained in the file following the pattern "/zoom/x/y.extension.ini", as discussed in #TileManagement.

At the root level, each cache must provide a #CacheProperties file, named "/cache.ini". This file provides several know attributes, such as the "extension" required to locate each tile.

An example cache directory layout is outlined below:

~/.cache/osm/tiles/
~/.cache/osm/tiles/Mapnik/
~/.cache/osm/tiles/Mapnik/cache.ini
~/.cache/osm/tiles/Mapnik/14/8709/5792.png
~/.cache/osm/tiles/Mapnik/14/8709/5793.png.ini
~/.cache/osm/tiles/Bing/
~/.cache/osm/tiles/Bing/cache.ini
...

Cache properties

The "Cache properties" file defines data associated with the cache itself. The file is named "cache.ini" an is located at the root level of the cache:

~/.cache/osm/tiles/Mapnik/cache.ini

The file format is a simple text file, UTF-8 encoded, with each line containing a "key=value" pair. Applications must ignore any key that they don't know. Applications must also preserve all keys when the file is being updated. Any other arbitrary key can be included.

The following mandatory keys are defined:

name

A long descriptive name of the tile provider for display purposes.

url

Base URL of the tile provider.

type

URL schema of the tile provider (currently only "TMS")

extension

Image file extension ("png" or "jpg").

size

Maximal size of the cache in bytes, with 0 meaning unlimited, and -1 meaning no appending.

age

Minimum tile age (in seconds)

An example properties file is shown below:

name=OSM Mapnik
url=http://tile.openstreetmap.org
type=TMS
extension=png
size=0
age=604800

Cache discovery

When initializing cache management, applications willing to use the shared tile cache should list all available caches inside the shared tile cache, and parse their respective properties files.

Applications must not depend on the ID (given by the directory name), but should instead locate the cache by matching on "url", "type" and possibly "extension". If a suitable match is found, the existing ID/directory should be used for further access.

If no suitable directory can be found, a new directory can be safely created inside the shared root. The application is free to choose any valid directory name, but should use a short, explicative name which must be less or equal to 20 characters.

Tile management

Assuming an empty tile cache, the application is free to create any file by downloading it from the provider and using the appropriate "/zoom/x/y.extension" pattern, creating missing directories.

Additional data can be stored for each tile, using the file named "/zoom/x/y.extension.ini". This file follows the same convention as the #CacheProperties file, and is a simple UTF-8 encoded "key=value" file containing arbitrary keys which are application-defined. Again, applications must ignore unknown keys, can add new keys, but must also preserve any existing one when updating the content. Applications are free to create/update the metadata file at any time.

When probing the cache for an existing tile, the application must check for the presence of the tile image "/zoom/x/y.extension" and check it's modification time. If the tile has been created or modified less than the specified cache age, the application must use the existing tile.

If the tile is older, it can be considered stale and the application is free to query the server again, supplying an "If-Modified-Since" header. For this reason, applications must not modify the file modification time of the tile image except when updating the tile.

When an existing tile is being updated or removed, any associated "/zoom/x/y.extension.ini" file must be deleted as well, irregardless of the contents.

When an existing tile is being deleted, in addition to removing the associated "ini" file, the application must also try to remove the "/zoom/x/" and "/zoom/" directories when empty.

When the cache size is defined as 0, applications must not perform any cache pruning. If size is anything greater than 0, each application can choose his own policy and schedule for tile expiration, deleting each tile in turn until the size of the entire directory tree is smaller or equal to the defined cache size.

When the cache size is defined as -1, applications must not include any additional content into the cache, but are free to query it.

Tile metadata

Tile metadata can be used for any purpose where storing additional information for each tile can be useful. As such, the following keys are defined to be known:

etag

ETag as returned by the server (used for querying servers that don't support If-Modified-Since).

Tile metadata can also be useful to store information about tile usage (view count) to improve usage-based cache pruning, though no keys are standardized for this behavior yet.

Concurrent access

To enable "opportunistic" concurrent cache access to each tile cache, we define the following simple rules:

• After deciding to create or update a tile, the image file ("/zoom/x/y.extension") must be moved atomically into the directory tree (and not created in-place), replacing any existing file (if any). An attempt can then be made to read the associated "ini" file. The "ini" file must be ignored if it's create/modification time is earlier than the image file and must then be deleted if not replaced.

• When deleting an existing tile, the image file must be removed first, followed by an attempt to remove the "ini" file, and the x/zoom directories in turn.

• Applications must not expect written tiles (or their metadata) to exist on disk or be the same as previously written.

• Updates to the cache properties and tile metadata must be similarly performed using atomic replacement. Since no locking is performed, applications must not cache their contents.

With this approach, a tile being updated concurrently may remove existing metadata, however doesn't require any locking or sophisticated metadata syncronization. Moreover, it doesn't break the assumption that metadata is optional while guaranteeing that any data (when present) is not stale.