Difference between revisions of "Caching"
| (21 intermediate revisions by 3 users not shown) | |||
| Line 1: | Line 1: | ||
[[Category:Functions]] | [[Category:Functions]] | ||
| − | + | [[Category:How_To]] | |
| + | [[Category:Fusion Registry Install]] | ||
= Overview = | = Overview = | ||
<p>The Fusion Registry provides a number of caching solutions to help ensure the performance of both server and client side solutions . The various cache layers include:</p> | <p>The Fusion Registry provides a number of caching solutions to help ensure the performance of both server and client side solutions . The various cache layers include:</p> | ||
* Caching layer before the Registry via a reverse proxy (Varnish Cache) | * Caching layer before the Registry via a reverse proxy (Varnish Cache) | ||
| − | * Caching on the Registry web service (If-Not-Modified, | + | * Caching on the Registry web service (If-Not-Modified), |
* Caching on server for data responses (pre-cached datasets) | * Caching on server for data responses (pre-cached datasets) | ||
* Caching with SDMX Queries (updatedAfter parameter) | * Caching with SDMX Queries (updatedAfter parameter) | ||
= Cache Purge General = | = Cache Purge General = | ||
| − | <p> | + | <p>@see [[Cache_Management_Web_Service#Purge_Cache|Purge Cache Web Service]]</p> |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
= HTTP Cache Headers = | = HTTP Cache Headers = | ||
| Line 81: | Line 19: | ||
== If-Modified-Since == | == If-Modified-Since == | ||
| − | The Fusion Registry maintains a record of all the timestamps a structure has changed. This information is persisted to the database, so on application | + | The Fusion Registry maintains a record of all the timestamps a structure has changed. This information is persisted to the database, so on application start-up the Fusion Registry is able to lookup the timestamps of when structures were last modified. When the HTTP Header If-Modified-Since is used on a structure query, the Fusion Registry processes the query to determine which structures make up the response. If any of the structures in the query response have been modified since the time passed in the If-Modified-Since header, then the user will get the full query response. If none of the structures have been modified, then a HTTP 304 (not modified) response is sent back to the client. |
== If-None-Match == | == If-None-Match == | ||
| − | The If-None-Match request makes use of a hashing function. When the Fusion Registry responds to a structure query, it will add a HTTP Header called <strong>ETag</strong>. The value of the ETag header is a hash which represents the content of the response. When the client requests the same resource, it can use the If-None-Match HTTP header, which uses the same hash value from the ETag of the previous request. The Fusion Registry will process the structure query request, hash the response, and then check it against the hash passed | + | The If-None-Match request makes use of a hashing function. When the Fusion Registry responds to a structure query, it will add a HTTP Header called <strong>ETag</strong>. The value of the ETag header is a hash which represents the content of the response. When the client requests the same resource, it can use the If-None-Match HTTP header, which uses the same hash value from the ETag of the previous request. The Fusion Registry will process the structure query request, hash the response, and then check it against the hash passed in the If-None-Match HTTP Header. If both client and server hashes match, then a HTTP 304 response is sent, otherwise the full query response is sent back to the client with a HTTP 200 status and a new ETag hash to represent the state of the response. |
| − | = Fusion | + | = Fusion Cache = |
| − | <p>The Fusion Registry provides a caching solution | + | <p>The Fusion Registry provides a caching solution supporting a subset of HTTP request, in particular data queries and data availability queries. The Fusion Cache uses the local file system to cache information, and it manages the purge of information periodically based on a pre-defined cache size.</p> |
| − | <p>The Fusion Cache is | + | <p>The Fusion Cache and Varnish cache are mutually exclusive caching solutions. If varnish Cache is being used, there is no need for the Fusion Cache. </p> |
| + | |||
| + | == Where on the file system is Information Cached? == | ||
| + | <p>The Fusion Cache uses the local file system cache, it uses the Registry temporary directory as a base folder, and creates sub-folders under this directory for each category of cached information. An example sub directory is '''FusionDataCache''' which is a category for storing the response to data queries.</p> | ||
| + | |||
| + | == How is Information Cached? == | ||
| + | Information is cached in '''gzip''' format, this makes it much faster to respond to HTTP queries with the HTTP Accept-Encoding Header set to gzip, most web browsers will provide this header by default, however if opening up REST APIs for public consumption, we recommend the web server enforces clients set this header if you are looking to get fast response times for large concurrent queries. | ||
| + | |||
| + | == Managing the herd == | ||
| + | In Fusion Registry v10.6.12 and higher the Fusion Cache is used to manage a large number of concurrent requests for the same resource, by only executing a single query and then responding to the others from the cached information. | ||
== Enabling the Cache == | == Enabling the Cache == | ||
| Line 95: | Line 42: | ||
== Purging the Cache == | == Purging the Cache == | ||
| − | The Fusion Cache is purged | + | The Fusion Cache is automatically deleted on Fusion Registry start-up, so it is not idea as a shared caching solution. A cache limit is also set, which is configurable, and when the cache reaches 90% of its limit, the Fusion Cache will start purging files. The order in which the files are purged is based on which file was requested last, meaning the last accessed cached file will be the first to be deleted. Files from the cache are purged in short bursts until the required space is recovered, this is to ensure CPU and IO does not become a bottleneck due to a long running purge process. |
| + | |||
| + | == Stale Cache == | ||
| + | The Fusion Cache will never contain stale information, as the system is designed to automatically purge parts of the cache as soon as the information in the database is updated. For example, if a dataset is updated, the data cache will be purged automatically | ||
| + | |||
| + | == Fusion Cache and Security == | ||
| + | The Fusion Cache can be used in combination with security rules as the cache key takes into account current user and their access rights. | ||
== Fusion Cache and Locale == | == Fusion Cache and Locale == | ||
Some datasets can change depending on the client's Locale settings. The Fusion Cache takes the Locale request into account, each cached item is stored against the data query, the response format, and client locale. | Some datasets can change depending on the client's Locale settings. The Fusion Cache takes the Locale request into account, each cached item is stored against the data query, the response format, and client locale. | ||
| − | = | + | = SDMX Data Query Updated After Request = |
| − | The | + | <p>The SDMX data query supports an updatedAfter query parameter, which tells the server to only return data that matches the query, and has been added or updated after the given time period. The time can be provided in any valid [[SDMX_Time_Formats|SDMX Time Format]], the time will always be taken as the start of period in the GMT time zone, i.e updateAfter=2009 will resolve to 2009-01-01T00:00:00GMT</p> |
| − | + | <p>How this request is handled depends on the data store that contains the data.</p> | |
| − | + | <p> For <b>[[Data_Stores#Registry_Managed_Data_Store|Registry Managed Data Stores]]</b>, which includes the <b>[[Data_Stores#Fusion_Data_Store|Fusion Store]]</b> and <b>[[Data_Stores#Registry_Managed_Database|Registry managed Databases]]</b>, the Registry records when each observation is updated, and when each series is updated. The updatedAfter is then used to remove series from the response if they have not been updated after the specified period (true for both series only and series with observation queries), if observations are in the response, these too will be removed if they have not been updated after the specified time.</p> | |
| + | <p>For <b>Externally managed SDMX web services</b> which are registered, the query parameter will be passed onto the service, and it is up to the service provider to ensure the query parameters are honoured</p> | ||
| + | <p>For <b>[[Data_Stores#Externally_Managed_Database|Externally managed Databases]]</b> if there is a column for last updated date, then this will be used in the data query, otherwise the updatedAfter parameter will be ignored</p> | ||
Latest revision as of 04:43, 13 September 2023
Contents
Overview
The Fusion Registry provides a number of caching solutions to help ensure the performance of both server and client side solutions . The various cache layers include:
- Caching layer before the Registry via a reverse proxy (Varnish Cache)
- Caching on the Registry web service (If-Not-Modified),
- Caching on server for data responses (pre-cached datasets)
- Caching with SDMX Queries (updatedAfter parameter)
Cache Purge General
HTTP Cache Headers
HTTP Cache Headers of If-Modified-Since and If-None-Match are supported for Structural Metadata queries (NOT Data queries).
Enabling HTTP Caching
The Fusion Registry cache settings must be set to enable HTTP 304 Header.
If-Modified-Since
The Fusion Registry maintains a record of all the timestamps a structure has changed. This information is persisted to the database, so on application start-up the Fusion Registry is able to lookup the timestamps of when structures were last modified. When the HTTP Header If-Modified-Since is used on a structure query, the Fusion Registry processes the query to determine which structures make up the response. If any of the structures in the query response have been modified since the time passed in the If-Modified-Since header, then the user will get the full query response. If none of the structures have been modified, then a HTTP 304 (not modified) response is sent back to the client.
If-None-Match
The If-None-Match request makes use of a hashing function. When the Fusion Registry responds to a structure query, it will add a HTTP Header called ETag. The value of the ETag header is a hash which represents the content of the response. When the client requests the same resource, it can use the If-None-Match HTTP header, which uses the same hash value from the ETag of the previous request. The Fusion Registry will process the structure query request, hash the response, and then check it against the hash passed in the If-None-Match HTTP Header. If both client and server hashes match, then a HTTP 304 response is sent, otherwise the full query response is sent back to the client with a HTTP 200 status and a new ETag hash to represent the state of the response.
Fusion Cache
The Fusion Registry provides a caching solution supporting a subset of HTTP request, in particular data queries and data availability queries. The Fusion Cache uses the local file system to cache information, and it manages the purge of information periodically based on a pre-defined cache size.
The Fusion Cache and Varnish cache are mutually exclusive caching solutions. If varnish Cache is being used, there is no need for the Fusion Cache.
Where on the file system is Information Cached?
The Fusion Cache uses the local file system cache, it uses the Registry temporary directory as a base folder, and creates sub-folders under this directory for each category of cached information. An example sub directory is FusionDataCache which is a category for storing the response to data queries.
How is Information Cached?
Information is cached in gzip format, this makes it much faster to respond to HTTP queries with the HTTP Accept-Encoding Header set to gzip, most web browsers will provide this header by default, however if opening up REST APIs for public consumption, we recommend the web server enforces clients set this header if you are looking to get fast response times for large concurrent queries.
Managing the herd
In Fusion Registry v10.6.12 and higher the Fusion Cache is used to manage a large number of concurrent requests for the same resource, by only executing a single query and then responding to the others from the cached information.
Enabling the Cache
The Fusion Cache is enabled in the Settings -> Cache page.
Purging the Cache
The Fusion Cache is automatically deleted on Fusion Registry start-up, so it is not idea as a shared caching solution. A cache limit is also set, which is configurable, and when the cache reaches 90% of its limit, the Fusion Cache will start purging files. The order in which the files are purged is based on which file was requested last, meaning the last accessed cached file will be the first to be deleted. Files from the cache are purged in short bursts until the required space is recovered, this is to ensure CPU and IO does not become a bottleneck due to a long running purge process.
Stale Cache
The Fusion Cache will never contain stale information, as the system is designed to automatically purge parts of the cache as soon as the information in the database is updated. For example, if a dataset is updated, the data cache will be purged automatically
Fusion Cache and Security
The Fusion Cache can be used in combination with security rules as the cache key takes into account current user and their access rights.
Fusion Cache and Locale
Some datasets can change depending on the client's Locale settings. The Fusion Cache takes the Locale request into account, each cached item is stored against the data query, the response format, and client locale.
SDMX Data Query Updated After Request
The SDMX data query supports an updatedAfter query parameter, which tells the server to only return data that matches the query, and has been added or updated after the given time period. The time can be provided in any valid SDMX Time Format, the time will always be taken as the start of period in the GMT time zone, i.e updateAfter=2009 will resolve to 2009-01-01T00:00:00GMT
How this request is handled depends on the data store that contains the data.
For Registry Managed Data Stores, which includes the Fusion Store and Registry managed Databases, the Registry records when each observation is updated, and when each series is updated. The updatedAfter is then used to remove series from the response if they have not been updated after the specified period (true for both series only and series with observation queries), if observations are in the response, these too will be removed if they have not been updated after the specified time.
For Externally managed SDMX web services which are registered, the query parameter will be passed onto the service, and it is up to the service provider to ensure the query parameters are honoured
For Externally managed Databases if there is a column for last updated date, then this will be used in the data query, otherwise the updatedAfter parameter will be ignored
