Fusion Registry Wiki - User contributions [en]

Data Stores

2025-01-14T13:23:54Z

Lythhishmeh:

[[Category:Functions]]
[[Category:How_To]]
= Overview =

The Fusion Registry can act as a Virtual Data Store. This means that the Registry is able to link to datasets accross multiple physical data stores, and each data store can be of a diffent type (database, web service, file). The Fusion Registry is able to query multiple data stores for a single data query and then combine the data at the end, before writing it back to the user in a response format of their choice.

== Registry Managed Data Store ==
A Registry Managed Data Store is one which the Registry maintains. This means as data is published to the Fusion Registry, the Registry wil write the data to the store and manage the transaction. Registry managed data stores come in two flavours, a relational database (MySQL or SQL Server) store, or a Fusion Store which has been custom built by Metadata Technology for rapid storage and dissemination of data.

=== Fusion Data Store ===
The Fusion Data Store is an in-memory database, custom built by Metadata Technology for storing time series SDMX datasets. The data is ultimately persisted to the Registry database, so that when the Fusion Registry server is terminated, and re-launched, the Fusion Data Stores will be read back into memory. For the duration of the server up-time, the data store will live in memory.

=== Registry Managed Database ===
A Registry Managed Database includes support for SQL Server and MySQL database platforms, MariaDB*. The Fusion Registry is responsible for creating the database tables when new datasets are loaded. The Fusion Registry will also update table definitions if the [[Data_Structure_Definition|Data Structure Definition]] is changed.

*Note Maria DB re-writes a query for both lastNObservation and firstNObservations to [https://dba.stackexchange.com/questions/117954/is-a-ranking-querys-order-by-in-from-subquery-is-preserved optimise away] a required part of the SQL query. MariaDB can be configured to run in MySQL compatability mode, where this error does not occur.

The database table structure is a two or three tables per Dataflow. The table names are based on the Dataflow, but can be changed if required. The tables include:
* Table for Series Definitions
* Table for Observations
* Table data dataset attributes (only created if Dataset Attributes exist on the Data Structure Definition)

==== Column Names ====
The Series Table will have the following Columns created.

{| class="wikitable"
|-
! Column Name !! Column Type !! Purpose
|-
| ID || Varchar || A concatenation of the Dimension values that make up the series. The width of the ID column is a combination of all the Dimension Widths plus the number of Dimensions.
An example ID is M:FR:EMP. It is important that the column width for each Dimension of the Data Structure Definition is low, i.e if the Dimenison is data type String then place a Max Width restriction on the Dimension's definition so that the database table can be constructed with a primary key that does not exceed the limit set by the database vendor.
|-
| FR_GROUP_ID || varchar|| (optional) Created on if the Data Structure Definition defines a Groups. This coulmn enable the storage of Group level Attributes
|-
| FR_PROVIDER_ACY || varchar || Data Provider's Agency, used if the data query includes Data Provider filters
|-
| FR_PROVIDER_ID || varchar|| Data Provider's Id, used if the data query includes Data Provider filters
|-
| FR_PROVISION || varchar || Provision Agreement short URN e.g ECB:EXR(1.0)
|-
| FR_ERROR || varchar || Internal Use only
|-
| FR_UPDATED_DATE || varchar|| Last updated date for the series
|-
| DIM ID/ATT_ID || varchar|text|| A Column for each Dimension and Series Level Attribute. The Column Name is the ID of the Dimension/Attribute and the Column Type is dependant on the Text Format
|}

The Observation Table will have the following Columns created.
{| class="wikitable"
|-
! Column Name !! Column Type !! Purpose
|-
| ID || int || Auto increment primary key
|-
| FR_KEY_ID || varchar|| Foreign key to the series ID column
|-
| FR_OBS_DATE || datetime || A datetime representaion of the TIME_PERIOD column
|-
| OBS_VALUE|| varchar || The Observation Value
|-
| TIME_PERIOD || varchar || [[SDMX_Time_Formats|Reported Time Period]]
|-
| FR_ERROR || varchar || Internal use only
|-
| FR_EMBARGO || int || Deprecated
|-
| FR_IS_PUBLIC || int || Deprecated
|-
| FR_UPDATED_DATE_OBS || int || Last time the observation was inserted/updated
|-
| FR_VALID_FROM || datetime || Used to support the includeHistory query parameter. A time stamp of when the observation is valid from, either taken from the Header of the dataset, or set to the time the observation was inserted
|-
| FR_VALID_TO|| datetime || Used to support the includeHistory query parameter. A time stamp of when the observation is valid to (if the observation is updated, this is used to capture previous revisions)
|-
| FR_CHKSUM || varchar|| a checksum on the column (internal use)
|-
| ATT_ID || varchar|text|| A Column for each Observation Level Attribute. The Column Name is the ID of the Attribute and the Column Type is dependant on the Text Format
|}

The Dataset Attribute Table, if required, will have the following Columns created.

{| class="wikitable"
|-
! Column Name !! Column Type !! Purpose
|-
| ID || Varchar || The ID of the Dataset Level Attribute.
|-
| VALUE || varchar|| The reported value for the attribute
|-
| FR_ERROR || varchar|| internal use only
|}

==== Column Data Types ====
The column data types that are based on the [[Data_Structure_Definition|Data Structure Definitions]] Dimensions and Attributes are set to Varchar with the width restrictions based on the table below. If the width restriction exceeds 250 characters the column data type is set to TEXT.

Width restrictions are calculated as follows:
{| class="wikitable"
|-
! Component's Representation !! Column Width !! Example Value
|-
| String || Max Length defined on Component or 250 if unspecified || This is a series title
|-
| Codelist || Maximum length of the longest Code Id (e.g UK would be length 2) || UK
|-
| Boolean || 5 || true
|-
| Short || 5 || 32767
|-
| Integer || 10 || 123
|-
| Long || 19 || 123456
|-
| Float || 12 || 12.20
|-
| Double || 22 || 12.20
|-
| Double || 22 || 12.20
|-
| Date || 30 || 2002-01-01T12:15:00.000GMT
|-
| Date Time || 30 || 2002-01-01T12:15:00.000GMT
|-
| Time Period || 30 || 2002-Q1 (any valid [[SDMX_Time_Formats|Time Format]])
|-
| Day || 2 || 31
|-
| Month || 2 || 12
|-
| Year_Month || 7 || 2002-01
|-
| Year || 4 || 2001
|-
| Time || 10 || 12:15:00
|}

Note If the dimenison or attribute is numerical and a maximum and minimum value is set, then this will be used to determine the column width. For example if the Attribute is of type Double, with a min value of 0 and a max value of 10 with a Decimals of 3, then the maximum length is 6, allowing for the values such as 10.123 to be reported.

== Externally Managed Database ==
An Externally Managed Database can be MySQL, SQLServer, or Oracle.

There are two options for the database table design when linking to an externally managed database, these are:
# Mirror the structure used by the Fusion Registry Managed database table structure (Series / Observation).
# A single database table for a Dataflow, where the Series and Observations are in one table

'''Note''' The Table and column names must all be in UPPER CASE.

When the Provision Agreement is linked to an externally managed data store, the user interface will provide the ability to specify what the database tables/views are used for the Series table and the Observation table. The Series table is optional, and if it is not provided then the Single Table Design will be used to query the database table. If the Series table is provided, the Series/Obs table design will be used.

=== Series/Obs Table Design ===
See the [[#Registry_Managed_Database|registry table design]] section for the full structure. The full table structure is not required, but there are some rules required for this table/view design, these are:

# The '''Series table''' must have a primary key OR a Column called ID. If there is a column called ID it will be expected to be the Primary Key column.
# If the '''Series table''' has an ID column AND it is of type Varchar, the values will be expected to be the concatenated series key (A:UK:EMP for example).
# The '''Obs table''' must either have a foreign key to the Series Table, OR a column called FR_KEY_ID which can be used to join to the primary key of the Series Table
# The '''Series table''' is expected to have a Column per Dimension, attribute columns are all optional
# The '''Obs table''' is expected to have a Column for Obs Value and Time Period, attribute columns are all optional
# All columns that are prefixed with 'FR_' are optional

=== Single Table Design ===
The flat table structure is a denormalised structure where the Series and Observations are in one table. This does lead to a repetition of dimension values where a series consists of multiple observations. The flat table structure does not support all the features of a SDMX Query including firstNObservations, lastNObservations, includeHistory.

The table structure is as follows:

{| class="wikitable"
|-
! Column Name !! Column Type !! Required !! Purpose
|-
| DIM _ID || varchar || Required || Column per Dimension where the column name is that of the Dimension Id
|-
| ATTR_ID || varchar || Optional|| Column per Series or Observation Attribute where the column name is that of the Attribute Id
|-
| TIME_PERIOD || varchar || Required || [[SDMX_Time_Formats|Reported Time Period]]
|-
| FR_VALID_FROM || datetime || Optional || Used to support startPeriod, endPeriod, lastNObservations, firstNObservations query parameters
|-
| FR_UPDATED_DATE || datetime || Optional || to support the updatedAfter query parameter
|}

'''Note:''' NVARCHAR and VARCHAR are not interchangeable and columns must be defined as VARCHAR and not NVARCHAR to prevent errors.

== Custom Table and Column Names ==
It is possible to override the Registrys own table nameing and column naming convention. To do this via a web service, use the following:

{| class="wikitable"
|-
|style="background-color:#eaecf0"|Entry Point|| /ws/secure/datasource/tableMapping
|-
|style="background-color:#eaecf0"|Access|| Private (admin)
|-
|style="background-color:#eaecf0"|Http Method|| POST
|-
|style="background-color:#eaecf0"|Accepts|| application/json
|-
|style="background-color:#eaecf0"|Compression || N/A
|-
|style="background-color:#eaecf0"|Content-Type || application/json
|-
|style="background-color:#eaecf0"|Response Format|| application/json
|-
|style="background-color:#eaecf0"|Response Statuses|| 200 - success 
400 - Bad JSON syntax
401 - Unauthorized
500 - Server Error
|}

The JSON request has the following structure:
{
"URN" : "urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=WB:GCI(1.0)",
"DataSourceId" : "MY_STORE",
"KeyTable" : "GCI_KEY",
"ObsTable" : "GCI_OBS",
"DSAttributeTable" : null,
"MappedColumns" : {
"REF_AREA" : "COUNTRY",
"FR_UPDATE_DATE" : "UPDATED_DATE"
}
}

The URN field is the URN of the Dataflow to map the table/colmn names for. The Mapped Columns field is optional, and only required if column names require mapping as well as table names. The same web service can be used to update a mapping, the Dataflow URN is the key for the mapping.

To delete a mapping call the same web service with method DELETE, and post the following JSON:

{
"URN" : "urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=WB:GCI(1.0)",
"DataSourceId" : "MY_STORE",
}

To retrieve a mapping call the same web service with the following query parameters:

{| class="wikitable"
|-
|style="background-color:#eaecf0"|urn|| urn of the Dataflow to get the table mapping description for
|-
|style="background-color:#eaecf0"|dataSourceId|| Id of the data store to get the table mapping description for
|}

==Example Using Oracle==
The following is a short example demonstrating how to use Oracle and set up 2 externally managed tables and connect them to the Fusion Registry, allowing the Registry to disseminate data.

The steps to follow are:
# Using the database tools of your choice, create the Key Table and Obs Table and populate both with data
# Load the structures (attached below) into your Fusion Registry
# Navigate to the Data Sources Manager page and:
#* Set up an Oracle Connection where the Connection Type is '''"Externally Managed"''' and the Database Platform is '''"Oracle"'''
#* Select this Oracle data source in the table and click the cogs at top-right. Select '''"Link Provision Agreement"''' and select the Provision Agreement '''"WB_WDI_ECON_POL_DEBT"'''
#* If everything has gone correctly, then the table will show 3 Series (and an unknown amount of Observations)
# The data can now be queried from the Registry using the appropriate Web Service and the dataflow '''"WDI_ECON_POL_DEBT"'''. E.g.:
<pre>
http://localhost:8080/FusionRegistry/sdmx/v2/data/dataflow/WB/WDI_ECON_POL_DEBT/1.0/?format=sdmx-compact-2.1
</pre>

====Key Table:====
The following creates the Key Table: '''WB_WDI_ECON_POL_DEBT_1_0_KEY'''
<pre>
CREATE TABLE WB_WDI_ECON_POL_DEBT_1_0_KEY (
ID VARCHAR2(250 BYTE),
FREQ VARCHAR2(5 BYTE),
SERIES VARCHAR2(25 BYTE),
REF_AREA VARCHAR2(3 BYTE)
)
</pre>

====Sample Data for Key Table:====
<pre>
INSERT INTO WB_WDI_ECON_POL_DEBT_1_0_KEY values ('A:AG_AGR_TRAC_NO:ABW', 'A', 'AG_AGR_TRAC_NO', 'ABW')
INSERT INTO WB_WDI_ECON_POL_DEBT_1_0_KEY values ('A:AG_AGR_TRAC_NO:ADO', 'A', 'AG_AGR_TRAC_NO', 'ADO')
INSERT INTO WB_WDI_ECON_POL_DEBT_1_0_KEY values ('A:AG_AGR_TRAC_NO:AFG', 'A', 'AG_AGR_TRAC_NO', 'AFG')
</pre>

====Observation Table:====
The following creates the Observation Table: '''WB_WDI_ECON_POL_DEBT_1_0_OBS'''
<pre>
CREATE TABLE WB_WDI_ECON_POL_DEBT_1_0_OBS (
FR_KEY_ID VARCHAR2(50 BYTE),
FR_OBS_DATE TIMESTAMP (6) NOT NULL,
TIME_PERIOD VARCHAR2(50 BYTE) NOT NULL,
OBS_VALUE VARCHAR2(10 BYTE) NOT NULL,
UNIT_MULT VARCHAR2(2 BYTE) NOT NULL,
FR_VALID_FROM TIMESTAMP (6) DEFAULT TO_TIMESTAMP( '2015-01-01 00:00:00', 'YYYY-MM-DD HH24:MI:SS') NOT NULL,
FR_VALID_TO TIMESTAMP (6) DEFAULT TO_TIMESTAMP( '2999-12-31 00:00:00', 'YYYY-MM-DD HH24:MI:SS') NOT NULL,
FR_UPDATED_DATE_OBS TIMESTAMP (6) DEFAULT TO_TIMESTAMP( '2014-01-01 00:00:00', 'YYYY-MM-DD HH24:MI:SS') NOT NULL
)
</pre>

====Sample Data for Obs Table:====
<pre>
INSERT INTO WB_WDI_ECON_POL_DEBT_1_0_OBS (FR_KEY_ID, FR_OBS_DATE, TIME_PERIOD, OBS_VALUE, UNIT_MULT) values ('A:AG_AGR_TRAC_NO:ABW', '01-JAN-1950', '1950', '1950.5001', 'C')
INSERT INTO WB_WDI_ECON_POL_DEBT_1_0_OBS (FR_KEY_ID, FR_OBS_DATE, TIME_PERIOD, OBS_VALUE, UNIT_MULT) values ('A:AG_AGR_TRAC_NO:ABW', '01-JAN-1951', '1951', '1951.5152', 'C')
INSERT INTO WB_WDI_ECON_POL_DEBT_1_0_OBS (FR_KEY_ID, FR_OBS_DATE, TIME_PERIOD, OBS_VALUE, UNIT_MULT) values ('A:AG_AGR_TRAC_NO:ABW', '01-JAN-1952', '1952', '1952.52', 'C')
INSERT INTO WB_WDI_ECON_POL_DEBT_1_0_OBS (FR_KEY_ID, FR_OBS_DATE, TIME_PERIOD, OBS_VALUE, UNIT_MULT) values ('A:AG_AGR_TRAC_NO:ADO', '01-JAN-1951', '1951', '19.51', 'C')
INSERT INTO WB_WDI_ECON_POL_DEBT_1_0_OBS (FR_KEY_ID, FR_OBS_DATE, TIME_PERIOD, OBS_VALUE, UNIT_MULT) values ('A:AG_AGR_TRAC_NO:AFG', '01-JAN-1950', '1950', '11950.50', 'C')
INSERT INTO WB_WDI_ECON_POL_DEBT_1_0_OBS (FR_KEY_ID, FR_OBS_DATE, TIME_PERIOD, OBS_VALUE, UNIT_MULT) values ('A:AG_AGR_TRAC_NO:AFG', '01-JAN-1951', '1951', '11951.51', 'C')
INSERT INTO WB_WDI_ECON_POL_DEBT_1_0_OBS (FR_KEY_ID, FR_OBS_DATE, TIME_PERIOD, OBS_VALUE, UNIT_MULT) values ('A:AG_AGR_TRAC_NO:AFG', '01-JAN-1952', '1952', '11952.999', 'C'
</pre>

====Dataflow, DSD and other related structures:====
The following file ( [[File:ExampleStructures.zip|thumb|Structures For Oracle]] ) contains the structures required by this demo. It includes:
* The Data Structure: '''WB:WDI(1.0)'''
* The Dataflow: '''WB:WDI_ECON_POL_DEBT(1.0)'''
* The Provision Agreement: '''WB:WB_WDI_ECON_POL_DEBT(1.0)'''
* Supporting structures such as Codelists and Concept Schemes

Recommended Tomcat Setup

2024-06-13T21:09:55Z

Lythhishmeh:

[[Category:How_To]]
[[Category:Fusion Registry Install]]

= Recommended Apache Tomcat Setup =

Fusion Registry must be run within a Java Servlet Container. Regnology recommends using Apache Tomcat as the Java Servlet Container, as this has been used during the testing lifecycle of Fusion Registry. The Fusion Registry has only been tested in Apache Tomcat and therefore we cannot guarantee that the Fusion Registry will work with other Java Servlet Containers.

There are a few things to consider when running Fusion Registry on Apache Tomcat.

== Security ==

It's worth noting that this is not an exhaustive list of all needed security settings, nor is it a substitute for on-team infrastructure expertise, but for enhanced security, we recommend the following settings:

=== server.xml File ===

==== Hiding Stack Traces ====

By default, Tomcat will display Stack Traces in certain circumstances. We recommend the following change be made to the file: '''server.xml'''. This file is located in the Apache Tomcat Directory ''<Apache Root>/conf''

Edit the file and and locate the "Host" section. At the end of this section, just before the closing </Host> tag, insert the following:
<Valve className="org.apache.catalina.valves.ErrorReportValve" showReport="false" showServerInfo="false"/>

This will prevent stack traces from being displayed to the user with HTTP error codes. Tomcat must be restarted for this change to take effect.

=== context.xml File ===

==== Same Site Cookies ====

Since release 11.13 of the Fusion Registry, the product now ships with a default ''context.xml'' file which is located in the META-INF directory in the Fusion Registry directory. This file overrides Tomcat's cookie pre-processor settings and enforces a strict SameSite cookies policy. More information on SameSite cookie security policies can be found [https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie here]. This prevents potential vulnerabilities related to Cross-Site Request Forgery (CSRF).

No action is required on the deployer's side, but if you want to disable this setting, either change or delete the file.

=== web.xml File ===

==== HTTP Strict Transport Security (HSTS) ====

We recommend the following change be made to the file: '''web.xml'''. This file is located in the Apache Tomcat Directory ''<Apache Root>/conf''. HTTP Strict Transport Security (HSTS) is a standard which forces those connecting to the registry over the internet to only use HTTPs. This can prevent sniffing attacks and MITM (Man in the middle) attacks which could compromise data security. We don't enable HSTS by default in the registry to allow for flexibility in configuration, but we do encourage our users to enable it.

In Tomcat 9.0, HSTS can be enabled by doing the following:

Modify/Update "httpHeaderSecurity" filter within Tomcat web.xml file with following values:
<filter>
<filter-name>httpHeaderSecurity</filter-name>
<filter-class>org.apache.catalina.filters.HttpHeaderSecurityFilter</filter-class>
<async-supported>true</async-supported>
<init-param>
<param-name>hstsEnabled</param-name>
<param-value>true</param-value>
</init-param>
<init-param>
<param-name>hstsMaxAgeSeconds</param-name>
<param-value>31536000</param-value>
</init-param>
<init-param>
<param-name>hstsIncludeSubDomains</param-name>
<param-value>true</param-value>
</init-param>
</filter>
and uncomment the below section which can be found further down in the web.xml file:
<filter-mapping>
<filter-name>httpHeaderSecurity</filter-name>
<url-pattern>/*</url-pattern>
<dispatcher>REQUEST</dispatcher>
</filter-mapping>

The above filter values represent:

'''hstsEnabled (true)''': HTTP Strict Transport Security (HSTS) header to be added to the response. 
'''hstsMaxAgeSeconds (31536000)''': The one year age value that should be used in the HSTS header. 
'''hstsIncludeSubDomains (true)''': The includeSubDomains parameter to be included in the HSTS header. 

==== Setting a Maximum HTTP Request Size ====

We recommend our customers set a limit on the maximum size of the HTTP requests they allow. This is to prevent malicious actors from sending overly large requests with the intent of crashing the registry. This can be done by modifying the '''web.xml''' file.

Under the <code><webapp></code> tag, create a new servlet with the following values:

<servlet>
<multipart-config>
<file-size-threshold>0</file-size-threshold>
<max-file-size>209715200</max-file-size>
<max-request-size>209715200</max-request-size>
</multipart-config>
</servlet>

The above filter values represent:

'''file-size-threshold''': Determines the size threshold over which a file will be flushed to disk instead of being held in memory. 
'''max-file-size''': Determines the largest size the server will accept for an uploaded file. 
'''max-request-size''': The threshold for the total size of a POST request. 

The values are measured in bytes.

Recommended Tomcat Setup

2024-06-13T21:07:51Z

Lythhishmeh:

[[Category:How_To]]
[[Category:Fusion Registry Install]]

= Recommended Apache Tomcat Setup =

Fusion Registry must be run within a Java Servlet Container. Regnology recommends using Apache Tomcat as the Java Servlet Container, as this has been used during the testing lifecycle of Fusion Registry. The Fusion Registry has only been tested in Apache Tomcat and therefore we cannot guarantee that the Fusion Registry will work with other Java Servlet Containers.

There are a few things to consider when running Fusion Registry on Apache Tomcat.

== Security ==

For enhanced security, we recommend the following:

=== server.xml File ===

==== Hiding Stack Traces ====

By default, Tomcat will display Stack Traces in certain circumstances. We recommend the following change be made to the file: '''server.xml'''. This file is located in the Apache Tomcat Directory ''<Apache Root>/conf''

Edit the file and and locate the "Host" section. At the end of this section, just before the closing </Host> tag, insert the following:
<Valve className="org.apache.catalina.valves.ErrorReportValve" showReport="false" showServerInfo="false"/>

This will prevent stack traces from being displayed to the user with HTTP error codes. Tomcat must be restarted for this change to take effect.

=== context.xml File ===

==== Same Site Cookies ====

Since release 11.13 of the Fusion Registry, the product now ships with a default ''context.xml'' file which is located in the META-INF directory in the Fusion Registry directory. This file overrides Tomcat's cookie pre-processor settings and enforces a strict SameSite cookies policy. More information on SameSite cookie security policies can be found [https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie here]. This prevents potential vulnerabilities related to Cross-Site Request Forgery (CSRF).

No action is required on the deployer's side, but if you want to disable this setting, either change or delete the file.

=== web.xml File ===

==== HTTP Strict Transport Security (HSTS) ====

We recommend the following change be made to the file: '''web.xml'''. This file is located in the Apache Tomcat Directory ''<Apache Root>/conf''. HTTP Strict Transport Security (HSTS) is a standard which forces those connecting to the registry over the internet to only use HTTPs. This can prevent sniffing attacks and MITM (Man in the middle) attacks which could compromise data security. We don't enable HSTS by default in the registry to allow for flexibility in configuration, but we do encourage our users to enable it.

In Tomcat 9.0, HSTS can be enabled by doing the following:

Modify/Update "httpHeaderSecurity" filter within Tomcat web.xml file with following values:
<filter>
<filter-name>httpHeaderSecurity</filter-name>
<filter-class>org.apache.catalina.filters.HttpHeaderSecurityFilter</filter-class>
<async-supported>true</async-supported>
<init-param>
<param-name>hstsEnabled</param-name>
<param-value>true</param-value>
</init-param>
<init-param>
<param-name>hstsMaxAgeSeconds</param-name>
<param-value>31536000</param-value>
</init-param>
<init-param>
<param-name>hstsIncludeSubDomains</param-name>
<param-value>true</param-value>
</init-param>
</filter>
and uncomment the below section which can be found further down in the web.xml file:
<filter-mapping>
<filter-name>httpHeaderSecurity</filter-name>
<url-pattern>/*</url-pattern>
<dispatcher>REQUEST</dispatcher>
</filter-mapping>

The above filter values represent:

'''hstsEnabled (true)''': HTTP Strict Transport Security (HSTS) header to be added to the response. 
'''hstsMaxAgeSeconds (31536000)''': The one year age value that should be used in the HSTS header. 
'''hstsIncludeSubDomains (true)''': The includeSubDomains parameter to be included in the HSTS header. 

==== Setting a Maximum HTTP Request Size ====

We recommend our customers set a limit on the maximum size of the HTTP requests they allow. This is to prevent malicious actors from sending overly large requests with the intent of crashing the registry. This can be done by modifying the '''web.xml''' file.

Under the <code><webapp></code> tag, create a new servlet with the following values:

<servlet>
<multipart-config>
<file-size-threshold>0</file-size-threshold>
<max-file-size>209715200</max-file-size>
<max-request-size>209715200</max-request-size>
</multipart-config>
</servlet>

The above filter values represent:

'''file-size-threshold''': Determines the size threshold over which a file will be flushed to disk instead of being held in memory. 
'''max-file-size''': Determines the largest size the server will accept for an uploaded file. 
'''max-request-size''': The threshold for the total size of a POST request. 

The values are measured in bytes.

System Properties

2024-05-13T12:21:40Z

Lythhishmeh: /* Properties which may be Dynamically changed */

[[Category:How_To]]
[[Category:Fusion Registry Install]]
= Overview =

System Properties are generally set when the Java virual machine that is running Fusion Registry is launched. However the Registry provides a way to list all of the System properties and also provides the ability to set their values. Note that setting some of the system properties will have no effect on the running Registry as these must be supplied at startup time . The properties that can be changed and for the change to have an effect are listed below.

= Obtaining System Properties =
A root or an admin user of Fusion Registry can access the System Properites, returned in JSON form, by performing a GET request to the url:

<code>
http://[server]:[port]/FusionRegistry/ws/secure/settings/getSystemProperties
</code>

As this is a secure Web Service the user must be logged on, or supply correct credentials. The result is an array of JSON objects, where each object has a Key and a Value For example:

<pre>
{
Key: "edi.lenient.enabled",
Value: "true"
}
</pre>

= Setting System Properties =

A root or an admin user of Fusion Registry can set a System Properites by performing a POST request to the url:

<code>
http://[server]:[port]/FusionRegistry/ws/secure/settings/setSystemProperty
</code>

Appropriate credential must be supplied. The input format is JSON of the form
<pre>
{
"Key": "edi.lenient.enabled",
"Value": "false"
}
</pre>

If the "value" specified is blank this will have the effect of removing the property. For example the following will remove the property "ENFORCE_STRICT_SDMX_2.1":

<pre>
{
"Key": "ENFORCE_STRICT_SDMX_2.1",
"Value": ""
}
</pre>

'''IMPORTANT NOTE:''' Values set by using the Web Service will '''not''' be preserved on a Registry reboot. It is recommended to supply the System properties as part of the Java Server startup process (for example by the use of [[Tomcat_setenv_file | setenv]] in Apache Tomcat).

= Properties which may be Dynamically changed =

Changing a property whilst the Registry is running may not have any effect. The following properties can be changed and their effect will be immediate:

{| class="wikitable"
|-
! Property !! Default Value !! Effect
|-
| edi.lenient.enabled || false || [[Edi_Lenient_Mode|Sets the EDI Leniency of the Registry]]
|-
| edi.enforce.primarymeasure.enabled || false || [[Edi_Enforce_Primary_Measure|Sets whether a Primary Measure should be enforced in EDI]]
|-
| edi.forceTimeFormatSeriesAttr.enabled || false || [[Edi_Create_Time_Format|Sets whether a series attribute of TIME_FORMAT should be created]]
|-
| edi.skipagencycheck.enabled || false || When writing an EDI data file, does not check that the Agencies of the Dataflow and DSD match. When writing an EDI structure file, does not check that cross-referenced components are of the same agency as the DSD. This feature is part of EDI Leniency mode. [[EDI_skipagencycheck_enabled|Ignores the Agency check when evaluating the DSI file]]
|-
| edi.dsi.dataflow.enabled || false || When reading an EDI file, determines whether the DSI segment should be used to refer to a Dataflow[[EDI_dsi.dataflow_enabled|Switches the DSI from refering to the Datafllow]]
|-
|edi.datagroup.identifier || unset || [[Edi_DataGroup_Identifier|Sets how the Registry sets the Group name when reading an EDI data file]]
|-
| ENFORCE_STRICT_SDMX_2.1 || false || Puts the Registry into Enforcing Strict values for SDMX 2.1
 Week data is of the format: YYYY-Www
 Monthly data is of the format: YYYY-Mmm
 Half-Yearly data is of the format: YYYY-S[1-2]
|-
| server.secondary || false || [[Secondary_Server|Sets the Registry into secondary server mode]]
|-
| TF_ALTERNATE_HALF_YEAR || false || [[Time_Format_Alternate_Half_Year|Sets the Registry into Alternate Half Year mode]]
|-
| aggregation.series.threshold || 500000 || Sets the limit for aggregated series. Keep in mind that increasing this value can have an impact on performance during data load/hierarchy association or hierarchy change.
|-
| HCL_OUTPUT_PERMIT_ALIAS_REF || false || Determine how Hierarchical Codelists should be output.Fusion Registry version 10 only and since version 10.9.4.[[Hierarchical_Codelists_Output_Format| Hierarchical Codelist Output]]
|}

Temporary Files

2024-04-16T15:24:23Z

Lythhishmeh:

== RFS Temporary Files ==

RFS creates and uses temporary files for its operation. These files are created during the process of loading, conversion, and general manipulation of structures and data in the registry. These files are created because the buffers in RFS that manipulate structures and data '''overflow to disk when the amount of storage per buffer exceeds 30 Megabytes in size.''' These files are created in Apache Tomcat's temp directory by default, but this location can be modified either by changing the '''java.tmp''' key in RFS' properties file (see: https://wiki.sdmxcloud.org/Registry_Properties_File) or by going to '''Admin -> Server Settings -> Caching''' and then changing the ''''Temporary Directory'''' setting to the desired value.

The files are usually named '''tmpFile''*random number here*''sdmxsource_tmp'''.
 

[[File:Temp file settings2.png|400px|Temporary files settings in RFS]]

== Temporary Files Lifetime ==

'''All temporary files have a maximum lifetime of 15 minutes once RFS is no longer using them.''' The files are normally deleted after this and '''users should not be attempting to delete them manually.''' This could lead to errors in RFS if successful. If temporary files persist for longer than this, it indicates that either RFS is still processing them, if the files contain an exceptionally large amount of data of structures, or that RFS has failed to release the lock on those files after being done with them. This should not normally occur, but if it does, the optimal approach is to report this issue to customer support, stop RFS, then attempt to delete them before restarting RFS. It's worth noting that RFS can automatically delete these files on start-up if the correct setting is enabled ('''Admin -> Server Settings -> Caching, Clear Temporary Directory on Startup''').

Temporary Files

2024-04-16T15:22:09Z

Lythhishmeh: Created page with "== RFS Temporary Files == RFS creates and uses temporary files for its operation. These files are created during the process of loading, conversion, and general manipulation..."

== RFS Temporary Files ==

RFS creates and uses temporary files for its operation. These files are created during the process of loading, conversion, and general manipulation of structures and data in the registry. These files are created because the buffers in RFS that manipulate structures and data '''overflow to disk when the amount of storage per buffer exceeds 30 Megabytes in size.''' These files are created in Apache Tomcat's temp directory by default, but this location can be modified either by changing the java.tmp key in RFS' properties file (see: https://wiki.sdmxcloud.org/Registry_Properties_File) or by going to '''Admin -> Server Settings -> Caching''' and then changing the ''''Temporary Directory'''' setting to the desired value.

The files are usually named '''tmpFile''*random number here*''sdmxsource_tmp'''.
 

[[File:Temp file settings2.png|400px|Temporary files settings in RFS]]

== Temporary Files Lifetime ==

'''All temporary files have a maximum lifetime of 15 minutes once RFS is no longer using them.''' The files are normally deleted after this and '''users should not be attempting to delete them manually.''' This could lead to errors in RFS if successful. If temporary files persist for longer than this, it indicates that either RFS is still processing them, if the files contain an exceptionally large amount of data of structures, or that RFS has failed to release the lock on those files after being done with them. This should not normally occur, but if it does, the optimal approach is to report this issue to customer support, stop RFS, then attempt to delete them before restarting RFS. It's worth noting that RFS can automatically delete these files on start-up if the correct setting is enabled ('''Admin -> Server Settings -> Caching, Clear Temporary Directory on Startup''').

File:Screenshot 2024-04-16 at 16.12.57.png

2024-04-16T15:18:19Z

Lythhishmeh:

Test

File:Temp file settings2.png

2024-04-16T15:14:29Z

Lythhishmeh:

File:Temp file settings.png

2024-04-16T15:13:36Z

Lythhishmeh:

Recommended Tomcat Setup

2024-01-12T16:23:03Z

Lythhishmeh: Created page with " == Recommended Apache Tomcat Setup == Fusion Registry 11 is designed to run on a variety of web servers and platforms, although we recommend and support Fusion Registry runn..."

== Recommended Apache Tomcat Setup ==

Fusion Registry 11 is designed to run on a variety of web servers and platforms, although we recommend and support Fusion Registry running on Apache Tomcat.

There are a few things to consider when running Fusion Registry on Apache Tomcat.

=== Security ===

For enhanced security, we recommend the following:

==== server.xml file ====

To prevent Tomcat related stack traces from being displayed in certain circumstances, we recommend the following changes be made to the server.xml file. This file can be found in (Apache Tomcat Directory)/conf.

Open the file in a text editor, and just before the </Host> tag, insert

'''<Valve className="org.apache.catalina.valves.ErrorReportValve" showReport="false" showServerInfo="false"/>
'''

and then save the file.

This prevents stack traces from being displayed to the user with HTTP error codes.

==== context.xml file ====

This is not a change you are required to make, but we present the following for informational purposes. To prevent potential vulnerabilities related to Cross-Site Request Forgery (CSRF), we have included a context.xml file, which is located in the META-INF directory in the Fusion Registry directory. This file overwrites Tomcat's cookie pre-processor settings and enforces a strict SameSite cookies policy. More information on SameSite cookie security policies can be found [https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie here].