Browser storage limits and eviction criteria

There are a number of web technologies that store data of one kind or another on the client-side (i.e. on your local disk.) The process by which the browser works out how much space to allocate to web data storage and what to delete when that limit is reached is not simple, and differs between browsers. This article attempts to explain how this all works.

Different types of data storage

Even in the same browser, using the same storage method, there are different classes of data storage to understand. This section discusses the different ones you might find in different browsers.

Generally, the main two types of storage are as follows:

• Persistent: This is data that is intended to be kept around for a long time. This will only be evicted if the user chooses to (for example in Firefox there is a "clear storage" button on the page info dialog for each page.)
• Temporary: This is data that doesn't need to persist for such long time . This will be evicted under a least recently used (LRU policy) when storage limits are reached.

By default, temporary storage will be used for most usage contexts (e.g. standard web apps), and persistent will be used for installed apps (e.g. Firefox apps installed on Firefox OS/Firefox desktop, Chrome apps.)

Firefox specifics

In Firefox you can choose which type of storage you want to use by including a proprietary option — storage — when you create an IndexedDB database using open():

• JavaScript

  Copy Code

  var request = indexedDB.open("myDatabase", { version: 1, storage: "persistent" });

• JavaScript

  Copy Code

  var request = indexedDB.open("myDatabase", { version: 1, storage: "temporary" });

In Firefox, when persistent storage is used the user is given a UI popup to alert them that this data will persist, and ask if they are happy with that.

Temporary data storage does not elicit any user prompts, but there are Storage limits.

Default storage in Firefox

There is a third type of storage to consider in Firefox — Default storage. This is a default option, used when you don't specify the storage parameter shown above. Default storage data behaves differently in different circumstances — it behaves as persistent storage for installed Firefox OS apps, and temporary storage for any other type of use.

Where is the data stored?

Each storage type represents a separate repository, here's the actual mapping to directories under a user's Firefox profile (other browsers may differ slightly):

• <profile>/storage — the main top level directory for storages maintained by the quota manager (see below.)
• <profile>/storage/permanent — persistent data storage repository
• <profile>/storage/temporary — temporary data storage repository
• <profile>/storage/default — default data storage repository

Storage limits

The maximum browser storage space is dynamic — it is based on your hard drive size. The global limit is calculated as 50% of free disk space. In Firefox, an internal browser tool called the Quota Manager keeps track of how much disk space each origin is using up, and deletes data if necessary.

So if your hard drive is 500GB, then the total storage for a browser is 250GB. If this is exceeded, a process called origin eviction comes into play, deleting entire origin's worth of data until the storage amount goes under the limit again. There is no trimming effect put in place, to delete parts of origins — deleting one database of an origin could cause problems with inconsistency.

There's also another limit called group limit — this is defined as 20% of the global limit. Each origin is part of a group (group of origins). There's one group for each eTLD+1 domain.

For example:

• mozilla.org — group1, origin1
• www.mozilla.org — group1, origin2
• joe.blogs.mozilla.org — group1, origin3
• firefox.com — group2, origin4

In this group, mozilla.org, www.mozilla.org, and joe.blogs.mozilla.org can aggregately use a maximum of 20% of the global limit. firefox.com has a separate maximum of 20%.

The two limits react differently to limits being reached:

• The group limit is also called the "hard limit": it doesn't trigger origin eviction.
• The global limit is a "soft limit" since there's a chance that some space will be freed and the operation can continue.

LRU policy

When the available disk space is filled up, the quota manager will start clearing out data based on a LRU policy — the least recently used origin will be deleted first, then the next one, until the browser is no longer over the limit.

We track the "last access time" for each origin using temporary storage. Once the global limit for temporary storage is reached (more on the limit later), we try to find all currently unused origins (i.e. ones with no tabs/apps open that are keeping open datastores). These are then sorted according to "last access time". The least recently used origins are then deleted until there's enough space to fulfill the request that triggered this origin eviction.

What technologies use browser data storage?

In Firefox, the following technologies make use of browser data storage to store data when required. We term them "quota clients" in this context:

• IndexedDB
• asm.js caching
• Cache API

The "last access time" of origins is updated when any of these are activated/deactivated origin eviction will delete data for all these quota clients.

In Chrome/Opera, the Quota Management API handles quota management for AppCache, IndexedDB, WebSQL and File System API.

See also

• Working with quota on mobile browsers, by Eiji Kitamura. A detailed analysis of client-side storage on mobile browsers.
• Quota Management API : Fast Facts, by Eiji Kitamura. A look at the Quota Management API in Chrome/Blink (which should include Opera too.)