Caching

Overview

The Fusion Registry provides a number of caching solutions to help ensure the performance of both server and client side solutions . The various cache layers include:

• Caching layer before the Registry via a reverse proxy (Varnish Cache)
• Caching on the Registry web service (If-Not-Modified,
• Caching on server for data responses (pre-cached datasets)
• Caching with SDMX Queries (updatedAfter parameter)

Varnish Cache

Varnish is an HTTP accelerator allowing for caching of HTTP requests. The Varnish server acts as a reverse proxy accepting a clients HTTP request and then passing it onto the target server (Fusion Registry). If Varnish has precached a response, then the response to the client will be server from the Varnish cache, and the request will not be passed onto the Fusion Registry.

Varnish Cache can be used for both Data and Structural Metadata queries via the REST API. Varnish can be used to cache other web services or HTML pages of the Fusion Registry, however the Fusion Registry will not automatically send requests to Varnish to purge these caches, and therefore it must be managed via another process

Enabling Varnish Caching

The Varnish Cahce server is a 3rd party software solution, and must be configured by folloing the vendor's documentation. The Fusion Registry 'integrates' with Varnish by knowing:

• That Varnish is being used as a front end caching solution
• The URL of the Varnish server, so that the Fusion Registry can tell varnish when and which parts of it's cache to purge

The Fusion Registry is told of Varnish through the Settings -> Cache page. The Fusion Registry only purges cache requests for Data and Structural Metadata queries, and as such Varnish should only be used to cache these requests, unless there is provision for another purge solution in place.

Purging the Cache

The Fusion Registry is responsible for purging the varnish cache when structural metadata or data changes. A BAN request is sent to the varnish server URL. Varnish will consume this request and remove the appropriate values from its cache.

When Fusion Registry structures are changed, a BAN request is sent to Varnish to BAN all previously cached structure queries, regardless of the structure. When FUsion Registry data are changed, a BAN request is sent to only the data queries for the Dataflow whose data changed

Example

1. Request for data https://yoursite.org/sdmx/data/ECB,EXR/A.UK+FR.../
2. New data is loaded or Registered for EXR
3. Fusion Registry send BAN requests to:

• https://yoursite.org/sdmx/data/ECB,EXR
• https://yoursite.org/sdmx/data/EXR

Note: For Fusion Registry to know the URL of your public server, please configure the Registry Server Settings -> Reverse Proxy Mapping.

HTTP Cache Headers

HTTP Cache Headers of If-Modified-Since and If-None-Match are supported for Structural Metadata queries (NOT Data queries).

Enabling HTTP Caching

The Fusion Registry cache settings must be set to enable HTTP 304 Header.

If-Modified-Since

The Fusion Registry maintains a record of all the timestamps a structure has changed. This information is persisted to the database, so on application startup the Fusion Registry is able to lookup the timestamps of when structures were last modified. When the HTTP Header If-Modified-Since is used on a structure query, the Fusion Registry processses the query to determine which structures make up the response. If any of the structures in the query response have been modified since the time passed in the If-Modified-Since header, then the user will get the full query response. If none of the structures have been modified, then a HTTP 304 (not mofified) response is sent back to the client.

If-None-Match

The If-None-Match request makes use of a hashing function. When the Fusion Registry responds to a structure query, it will add a HTTP Header called ETag. The value of the ETag header is a hash which represents the content of the response. When the client requests the same resource, it can use the If-None-Match HTTP header, which uses the same hash value from the ETag of the previous request. The Fusion Registry will process the structure query request, hash the response, and then check it against the hash passed inthe If-None-Match HTTP Header. If both client and server hashes match, then a HTTP 304 response is sent, otherwise the full query response is sent back to the client with a HTTP 200 status and a new ETag hash to respresent the state of the response.

Fusion Registry Cache

The Fusion Registry provides a caching solution for data queries only. This cache is called the Fusion Cache. The Fusion Cache and Varnish cache are mutually exclusive caching solutions. If varnish Cache is being used, there is no need for the Fusion Cache.

The Fusion Cache is a file system cache, where the configuration of the cache tells the Registry which folder to use, and how much space it is entitled to use. The Fusion Registry will cache the response to all data queries in this cache. The cache contains the response dataset in gzip format, in the format that the user requested. If the user queries for data in SDMX-ML Generic format, then the cached dataset will be in this format. If a new request comes in for the same dataset in a different format, then the Fusion Registry will convert the pre-cached data into the format requested, and then cache the response. If the HTTP request Accepts gzip response, then the Fusion Cache provides a very fast caching strategy for data queries. If the HTTP request does not Accept gzip, then the cache is still fast, but must unzip the cached dataset before writing it to the client