Difference between revisions of "Caching"

From Fusion Registry Wiki
Jump to navigation Jump to search
(→‎SDMX Data Query Updated After Request)
(→‎Fusion Registry Cache)
Line 86: Line 86:
 
The If-None-Match request makes use of a hashing function.  When the Fusion Registry responds to a structure query, it will add a HTTP Header called <strong>ETag</strong>.  The value of the ETag header is a hash which represents the content of the response.  When the client requests the same resource, it can use the If-None-Match HTTP header, which uses the same hash value from the ETag of the previous request.  The Fusion Registry will process the structure query request, hash the response, and then check it against the hash passed in the If-None-Match HTTP Header.  If both client and server hashes match, then a HTTP 304 response is sent, otherwise the full query response is sent back to the client with a HTTP 200 status and a new ETag hash to represent the state of the response.
 
The If-None-Match request makes use of a hashing function.  When the Fusion Registry responds to a structure query, it will add a HTTP Header called <strong>ETag</strong>.  The value of the ETag header is a hash which represents the content of the response.  When the client requests the same resource, it can use the If-None-Match HTTP header, which uses the same hash value from the ETag of the previous request.  The Fusion Registry will process the structure query request, hash the response, and then check it against the hash passed in the If-None-Match HTTP Header.  If both client and server hashes match, then a HTTP 304 response is sent, otherwise the full query response is sent back to the client with a HTTP 200 status and a new ETag hash to represent the state of the response.
  
−
= Fusion Registry Cache =
+
= Fusion Cache =
−
<p>The Fusion Registry provides a caching solution for data queries only.  This cache is called the <b>Fusion Cache</b>.  The Fusion Cache and Varnish cache are mutually exclusive caching solutions.  If varnish Cache is being used, there is no need for the Fusion Cache.</p>
+
<p>The Fusion Registry provides a caching solution supporting a subset of HTTP request, in particular data queries and data availability queries.  The Fusion Cache uses the local file system to cache information, and it manages the purge of information periodically based on a pre-defined cache size.</p>
  
−
<p>The Fusion Cache is a file system cache, where the configuration of the cache tells the Registry which folder to use, and how much space it is entitled to use.  The Fusion Registry will cache the response to all data queries in this cache.  The cache contains the response dataset in gzip format, in the format that the user requested.  If the user queries for data in SDMX-ML Generic format, then the cached dataset will be in this format.  If a new request comes in for the same dataset in a different format, then the Fusion Registry will convert the pre-cached data into the format requested, and then cache the response. If the HTTP request Accepts gzip response, then the Fusion Cache provides a very fast caching strategy for data queries.  If the HTTP request does not Accept gzip, then the cache is still fast, but must unzip the cached dataset before writing it to the client</p>
+
<p>The Fusion Cache and Varnish cache are mutually exclusive caching solutions.  If varnish Cache is being used, there is no need for the Fusion Cache. </p>
 +
 
 +
== Where on the file system is Information Cached? ==
 +
<p>The Fusion Cache uses the local file system cache, it uses the Registry temporary directory as a base folder, and creates sub-folders under this directory for each category of cached information.  An example sub directory is '''FusionDataCache''' which is a category for storing the response to data queries.</p>  
 +
 
 +
== How is Information Cached? ==
 +
Information is cached in '''gzip''' format, this makes it much faster to respond to HTTP queries with the HTTP Accept-Encoding Header set to gzip, most web browsers will provide this header by default, however if opening up REST APIs for public consumption, we recommend the web server enforces clients set this header if you are looking to get fast response times for large concurrent queries.
 +
 
 +
== Managing the herd ==
 +
In Fusion Registry v10.6.12 and higher the Fusion Cache is used to manage a large number of concurrent requests for the same resource, by only executing a single query and then responding to the others from the cached information. 
  
 
== Enabling the Cache ==
 
== Enabling the Cache ==
Line 95: Line 104:
  
 
== Purging the Cache ==
 
== Purging the Cache ==
−
The Fusion Cache is purged for a dataset, when a new dataset is loaded, or new data is registered with the Fusion Registry.
+
The Fusion Cache is automatically deleted on Fusion Registry start-up, so it is not idea as a shared caching solution.  A cache limit is also set, which is configurable, and when the cache reaches 90% of its limit, the Fusion Cache will start purging files.  The order in which the files are purged is based on which file was requested last, meaning cached files that have not been required are the first to go.  Files from the cache are purged in short bursts until the required space is recovered, without bottlenecking the CPU or IO.
 +
 
 +
== Stale Cache ==
 +
The Fusion Cache will never contain stale information, as the system is designed to automatically purge parts of the cache as soon as the information in the database is updated.  For example, if a dataset is updated, the data cache will be purged automatically
 +
 
 +
== Caching and Security ==
 +
The Fusion Cache can be used in combination with security rules as the cache key takes into account current user and their access rights.
  
 
== Fusion Cache and Locale ==
 
== Fusion Cache and Locale ==
 
Some datasets can change depending on the client's Locale settings.  The Fusion Cache takes the Locale request into account, each cached item is stored against the data query, the response format, and client locale.
 
Some datasets can change depending on the client's Locale settings.  The Fusion Cache takes the Locale request into account, each cached item is stored against the data query, the response format, and client locale.
−
 
−
== Fusion Cache and Data Format ==
 
−
The Fusion Cache is data format specific.  If the user requests a the same dataset for a different format (using either a query parameter, or the Accept Header) then the Fusion Cache will be used to get the dataset, and the dataset will be converted to the requested format.
 
−
 
−
== Fusion Cache and Security ==
 
−
If the Fusion Registry has security rules on specific datasets, then the Fusion Registry re-writes the data query, and therefore once it hits the cache, it will only hit on a dataset that the user has permission to see.  This strategy enables cached data to be reused by users with different security privileges.
 
  
 
= SDMX Data Query Updated After Request =
 
= SDMX Data Query Updated After Request =

Revision as of 13:20, 12 April 2021


Overview

The Fusion Registry provides a number of caching solutions to help ensure the performance of both server and client side solutions . The various cache layers include:

  • Caching layer before the Registry via a reverse proxy (Varnish Cache)
  • Caching on the Registry web service (If-Not-Modified),
  • Caching on server for data responses (pre-cached datasets)
  • Caching with SDMX Queries (updatedAfter parameter)

Cache Purge General

Regardless of cache types, the Fusion Registry is responsible for purging the cache.

The Fusion Registry knows when structures have changed, and will issue a purge request.

For data, the Fusion Registry only knows that data has changed if there is a new Data Registration event. This event is automatic when data are loaded into a Fusion Registry managed data store (Fusion Store or Registry managed database). For a data store that is not managed by the Fusion Registry (external database or URL) then it is up to the user to Re-Register the dataset with the Registry which will result in a cache purge.

Force Purge

To force the Fusion Registry to purge all caches use the following web service. The method is HTTP POST but no POSTed content is required.

Entry Point /ws/secure/settings/sendPurgeRequest
Access Private (admin only)
Http Method POST
Accepts N/A
Compression N/A
Content-Type N/A
Response Format application/json
Response Statuses

200 - Purge Cache granted

401 - Unauthorized (if access has been restricted)

500 - Server Error

Varnish Cache

Varnish is an HTTP accelerator allowing for caching of HTTP requests. The Varnish server acts as a reverse proxy accepting a clients HTTP request and then passing it onto the target server (Fusion Registry). If Varnish has pre-cached a response, then the response to the client will be server from the Varnish cache, and the request will not be passed onto the Fusion Registry.

Varnish Cache can be used for both Data and Structural Metadata queries via the REST API. Varnish can be used to cache other web services or HTML pages of the Fusion Registry, however the Fusion Registry will not automatically send requests to Varnish to purge these caches, and therefore it must be managed via another process

Enabling Varnish Caching

The Varnish Cache server is a 3rd party software solution, and must be configured by following the vendor's documentation. The Fusion Registry 'integrates' with Varnish by knowing:

  • That Varnish is being used as a front end caching solution
  • The URL of the Varnish server, so that the Fusion Registry can tell varnish when and which parts of it's cache to purge

Varnish should be configured to cache any requests to the data or structure web service, and preserve any Accept-Language HTTP Headers. An example configuration is provided based on Varnish Cache 4.1.2

The Fusion Registry is told of Varnish through the Settings -> Cache page. The Fusion Registry only purges cache requests for Data and Structural Metadata queries, and as such Varnish should only be used to cache these requests, unless there is provision for another purge solution in place.

Varnish Cache and Security

f the Fusion Registry has security rules on specific datasets, Varnish Cache will be unaware of this. Therefore we do not recommend Varnish Cache as a solution for a Fusion Registry which enforces different data access levels.

Varnish Cache and Locale

The Fusion Registry adds a VARY Http Header to all responses to indicate that if the client's Accept-Language changes from the cached version, then the cache should not be used. This is relevant for all structure requests, and some datasets which include code names.

Varnish Cache and Data Format

The Fusion Registry adds a VARY Http Header to all responses to indicate that if the client's Accept Header changes from the cached version, then the cache should not be used.

Purging the Cache

The Fusion Registry is responsible for purging the varnish cache when structural metadata or data changes. A BAN request is sent to the varnish server URL. Varnish will consume this request and remove the appropriate values from its cache.

When Fusion Registry structures are changed, a BAN request is sent to Varnish to BAN all previously cached structure queries, regardless of the structure. When FUsion Registry data are changed, a BAN request is sent to only the data queries for the Dataflow whose data changed

Example

  1. Request for data https://yoursite.org/sdmx/data/ECB,EXR/A.UK+FR.../
  2. New data is loaded or Registered for EXR
  3. Fusion Registry send BAN requests to:

Note: For Fusion Registry to know the URL of your public server, please configure the Registry Server Settings -> Reverse Proxy Mapping.

HTTP Cache Headers

HTTP Cache Headers of If-Modified-Since and If-None-Match are supported for Structural Metadata queries (NOT Data queries).

Enabling HTTP Caching

The Fusion Registry cache settings must be set to enable HTTP 304 Header.

If-Modified-Since

The Fusion Registry maintains a record of all the timestamps a structure has changed. This information is persisted to the database, so on application start-up the Fusion Registry is able to lookup the timestamps of when structures were last modified. When the HTTP Header If-Modified-Since is used on a structure query, the Fusion Registry processes the query to determine which structures make up the response. If any of the structures in the query response have been modified since the time passed in the If-Modified-Since header, then the user will get the full query response. If none of the structures have been modified, then a HTTP 304 (not modified) response is sent back to the client.

If-None-Match

The If-None-Match request makes use of a hashing function. When the Fusion Registry responds to a structure query, it will add a HTTP Header called ETag. The value of the ETag header is a hash which represents the content of the response. When the client requests the same resource, it can use the If-None-Match HTTP header, which uses the same hash value from the ETag of the previous request. The Fusion Registry will process the structure query request, hash the response, and then check it against the hash passed in the If-None-Match HTTP Header. If both client and server hashes match, then a HTTP 304 response is sent, otherwise the full query response is sent back to the client with a HTTP 200 status and a new ETag hash to represent the state of the response.

Fusion Cache

The Fusion Registry provides a caching solution supporting a subset of HTTP request, in particular data queries and data availability queries. The Fusion Cache uses the local file system to cache information, and it manages the purge of information periodically based on a pre-defined cache size.

The Fusion Cache and Varnish cache are mutually exclusive caching solutions. If varnish Cache is being used, there is no need for the Fusion Cache.

Where on the file system is Information Cached?

The Fusion Cache uses the local file system cache, it uses the Registry temporary directory as a base folder, and creates sub-folders under this directory for each category of cached information. An example sub directory is FusionDataCache which is a category for storing the response to data queries.

How is Information Cached?

Information is cached in gzip format, this makes it much faster to respond to HTTP queries with the HTTP Accept-Encoding Header set to gzip, most web browsers will provide this header by default, however if opening up REST APIs for public consumption, we recommend the web server enforces clients set this header if you are looking to get fast response times for large concurrent queries.

Managing the herd

In Fusion Registry v10.6.12 and higher the Fusion Cache is used to manage a large number of concurrent requests for the same resource, by only executing a single query and then responding to the others from the cached information.

Enabling the Cache

The Fusion Cache is enabled in the Settings -> Cache page.

Purging the Cache

The Fusion Cache is automatically deleted on Fusion Registry start-up, so it is not idea as a shared caching solution. A cache limit is also set, which is configurable, and when the cache reaches 90% of its limit, the Fusion Cache will start purging files. The order in which the files are purged is based on which file was requested last, meaning cached files that have not been required are the first to go. Files from the cache are purged in short bursts until the required space is recovered, without bottlenecking the CPU or IO.

Stale Cache

The Fusion Cache will never contain stale information, as the system is designed to automatically purge parts of the cache as soon as the information in the database is updated. For example, if a dataset is updated, the data cache will be purged automatically

Caching and Security

The Fusion Cache can be used in combination with security rules as the cache key takes into account current user and their access rights.

Fusion Cache and Locale

Some datasets can change depending on the client's Locale settings. The Fusion Cache takes the Locale request into account, each cached item is stored against the data query, the response format, and client locale.

SDMX Data Query Updated After Request

The SDMX data query supports an updatedAfter query parameter, which tells the server to only return data that matches the query, and has been added or updated after the given time period. The time can be provided in any valid SDMX Time Format, the time will always be taken as the start of period in the GMT time zone, i.e updateAfter=2009 will resolve to 2009-01-01T00:00:00GMT

How this request is handled depends on the data store that contains the data.

For Registry Managed Data Stores, which includes the Fusion Store and Registry managed Databases, the Registry records when each observation is updated, and when each series is updated. The updatedAfter is then used to remove series from the response if they have not been updated after the specified period (true for both series only and series with observation queries), if observations are in the response, these too will be removed if they have not been updated after the specified time.

For Externally managed SDMX web services which are registered, the query parameter will be passed onto the service, and it is up to the service provider to ensure the query parameters are honoured

For Externally managed Databases if there is a column for last updated date, then this will be used in the data query, otherwise the updatedAfter parameter will be ignored

Retrieved from "https://wiki.sdmxcloud.org/index.php?title=Caching&oldid=3514"