Difference between revisions of "Fusion Edge Server Properties"

From Fusion Registry Wiki
Jump to navigation Jump to search
(→‎Limiting Supported Data Formats)
(→‎Overview)
 
(40 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 +
[[Category:Fusion Edge Server]]
 +
[[Category:FESInstall]]
 +
 
= Overview =
 
= Overview =
−
The Fusion Edge Server provides configuration options on various aspects of the Edge Server. To set these configurations, create a file under the Edge Server Directory with the name: '''edgeserver.properties'''.
+
The Fusion Edge Server properties file provides configuration options on various aspects of the Edge Server. To set these configurations, create a file under the [[Fusion_Edge_Server_Directory|Edge Server Directory]] with the name: '''edgeserver.properties'''.
 +
 
 +
This file is '''required'''. At a minimum it should be given the ledger.secret property, this is the secret string used when the environment was compiled. Without this the environment can not be read.
  
 
= Public URL =
 
= Public URL =
Line 11: Line 16:
  
 
The above setting would make the SDMX web service entry point: https://myserver.myorg.org/Edge/sdmx
 
The above setting would make the SDMX web service entry point: https://myserver.myorg.org/Edge/sdmx
 +
 +
= Sender Id =
 +
SDMX Messages contain a Header section which includes the identity of the sender.  The identity is mandatory in a SDMX message, and the default ID is set to "UNKNOWN".  To change the Id, set the sender Id property
 +
 +
sender.id=YOUR_ID
 +
 +
= Secret Key =
 +
The Fusion Edge server must be given a secret key, and it must match that used by the [[Fusion_Edge_Compiler|Fusion Edge Compiler]] in the compilation of the [[Edge_Server_Environment|Environment]].  If the secret key does not match the secret key of the [[Edge_Server_Environment|Environment]], the Fusion Edge Server will fail to load the [[Edge_Server_Environment|Environment]].
 +
 +
ledger.secret=MySecret
 +
 +
= Static Mode - File Name =
 +
In [[Edge_Server_-_Publish_Content#.E2.80.8BStatic_Mode|static mode]] the Fusion Edge Server will read the properties file to know where the environment is. If nothing is set it will look for a file called node.zip.
 +
 +
environment.file=LiveEnvironment
 +
 +
= Embargo Data =
 +
The Fusion Edge server will not release an [[Edge_Server_Environment|Environment]] with an [[Fusion_Edge_Compiler#Embargo|Embargo]] until the time on the server is later then or equal to the timestamp of the Embargo. 
 +
 +
The Fusion Edge Server can be configured to pre-prepare and [[Edge_Server_Environment|Environment]] for release prior to the embargo time. For example if an Environment is scheduled for release at 12.00pm, and the Environment is moved to a secure Amazon S3 file system at 11.50, the Fusion Edge Server can be configured to pull these files into its local file system 5 minutes before go live, to eliminate any risk of network lag pulling the files. The Fusion Ede Server can also be set to pre-load the Environment into memory 30 seconds before go live, the Environment is now ready for dissemination but disconnected from any process which can get to the data, it is still fully secure. The Fusion Edge Server is able to then make the Environment go live by simply swapping the old Environment with the new, making it go live with millisecond precision. 
 +
 +
To preload the Environment from a remote location to the local file system prior to the Embargo time, set the preload filesystem property to define the number of milliseconds before the embargo time that it should initiate the preload process.
 +
 +
ledger.preload.filesystem=40000
 +
 +
To preload the Environment into memory, set the preload memory property to define the number of milliseconds before the embargo time that it should start reading the Environment into memory.
 +
 +
ledger.preload.memory=20000
 +
 +
In this example the Fusion Edge Server will pull an Environment from the remote location to the local file system 40 seconds prior to the go live time, and then it will pre load the Environment from the file system into memory 20 seconds before the go live time.  The Environment will be made live on the go live (embargo) time.
 +
 +
= Disable Audit =
 +
The Fusion Edge maintains an audit log of events in its local [[Fusion_Edge_Server_Directory|Edge Server Directory]]. Auditing can be disabled with the following property
 +
 +
audit.disabled=true
  
 
= Limiting Series and Observations =
 
= Limiting Series and Observations =
Line 21: Line 61:
  
 
If the limit is exceeded an HTTP status code of 413 (Payload Too Large) will be returned to the client.
 
If the limit is exceeded an HTTP status code of 413 (Payload Too Large) will be returned to the client.
 +
 +
= Default Reponse Formats =
 +
''Since v4.7.0''
 +
 +
The Fusion Edge Server supports a number of different response formats for structures, data, and metadata.  When the client does not specify which format to return the response in, the Fusion Edge Server will respond in the default format.  The properties file can be used to override the default formats by using the vnd header of the format to return.
 +
 +
{| class="wikitable"
 +
|-
 +
! REST Resource !! Default Format !! Override Property
 +
|-
 +
| data|| application/vnd.sdmx.structurespecificdata+xml;version=2.1 || default.format.data
 +
|-
 +
| structure|| application/vnd.sdmx.structure+json;version=2.0.0 || default.format.structure
 +
|-
 +
| availability|| application/vnd.sdmx.structure+json;version=2.0.0|| default.format.availability
 +
|-
 +
| tableavailability|| application/vnd.sdmx.structure+json;version=2.0.0 || default.format.tableavailability
 +
|-
 +
| table|| application/json || default.format.table
 +
|-
 +
| metadata|| application/vnd.sdmx.metadata+json;version=2.0.0 || default.format.metadata
 +
|}
  
 
= Limiting Supported Data Formats =
 
= Limiting Supported Data Formats =
−
To enforce a restriction on which data formats are supported use the format.data property as follows:
+
To enforce a restriction on which data formats are supported from the REST API of the Fusion Edge Server, use the '''format.data''' property as follows:
  
 
  format.data=[format1],[format2]
 
  format.data=[format1],[format2]
  
−
The supported formats are defined by the [[Data Formats]] VND header.
+
This property defines which formats are supported, if this property is not provided all formats will be supported. If this property is provided, only the formats in this list will be supported.
 +
 
 +
VND Headers are listed in the [[Data Formats]] section.
  
 
'''Example:'''<br/>
 
'''Example:'''<br/>
−
format.data=application/vnd.sdmx.generictimeseriesdata+xml;version=2.1
+
format.data=application/vnd.sdmx.generictimeseriesdata+xml;version=2.1,application/vnd.sdmx.data+csv
  
 
= Limiting Supported Structure Formats =
 
= Limiting Supported Structure Formats =
Line 56: Line 120:
 
| application/vnd.fusion.json || Fusion JSON
 
| application/vnd.fusion.json || Fusion JSON
 
|-
 
|-
−
| application/vnd.sdmx.json || SDMX JSON
+
| application/vnd.sdmx.structure+json;version=1.0.0 || SDMX JSON v1.0.0
 +
|-
 +
| application/vnd.sdmx.structure+json;version=2.0.0 || SDMX JSON v2.0.0
 
|}
 
|}
  
 
'''Example:''' <br/>
 
'''Example:''' <br/>
 
format.structure=application/vnd.sdmx.structure;version=edi,application/vnd.sdmx.json
 
format.structure=application/vnd.sdmx.structure;version=edi,application/vnd.sdmx.json
 +
 +
= Limiting Query Functionality =
 +
The [[Data_Query_Web_Service#Extended_HTTP_Query_Parameters|extended data query API]] can have some features disabled by using the following properties:
 +
rest.query.attribute = false
 +
rest.query.aggregate = false
 +
rest.query.interpolate = false
 +
rest.query.calculatemeasure = false
 +
rest.query.normalisefreq = false
 +
rest.query.round = false
 +
 +
= Caching =
 +
The Fusion Edge Server caches the response to data queries in the tmp folder of the [[Fusion_Edge_Server_Directory|Edge Directory]].  The response is stored in gzip format, which ensures clients who request data in gzip format receive a rapid response when the data has been cached due to previous requests for the same data. 
 +
 +
The following are important to note:
 +
* The cache is NOT read on application startup.  The cache is generated during the operational running time of the Fusion Edge Server.  The cache is a combination of in memory pointers to files in the file system.  Once the application server is terminated, it is not re-read on startup.  This means any cached files are redundant after the Fusion Edge Server is terminated.
 +
* The cache is NOT cleared on application startup. This is for a number of reasons, but principally deleting files from a running Edge Server is a slower process then could be achieved via an external mechanism. The delete process consumes CPU and therefore should be performed via a more efficient process before server startup.
 +
* The cache can be manually cleared by deleting all the files in the tmp directory.
 +
* The Fusion Edge Server will clear the cache if the Environment is updated in [[Edge_Server_-_Publish_Content#Dynamic_Mode|Dynamic mode]].  The in memory cache is first cleared to ensure clearance with immediate effect. A separate thread is initiated to clear the cached files, this can take time to complete if there are lots of cached files.  Files are cleared in batches of 5000 with a 5 second delay per batch, this is to ensure the CPU required to run the deletion process does not block external traffic to the Edge Server.
 +
 +
The cache can be disabled by supplying the following property (Fusion Edge Server v2.3.5 an higher):
 +
 +
rest.query.cache=false
  
 
= CSV Data Format Defaults =  
 
= CSV Data Format Defaults =  
Line 75: Line 163:
 
The values for 'Labels' must be one of the following: ID; NAME; ID_AND_NAME (defaults to ID_AND_NAME).  
 
The values for 'Labels' must be one of the following: ID; NAME; ID_AND_NAME (defaults to ID_AND_NAME).  
 
Show Series defines whether to output the series key in the output (defaults to true).
 
Show Series defines whether to output the series key in the output (defaults to true).
 +
 +
= Cross-Origin Resource Sharing (CORS) =
 +
By default the [https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS CORS] policy of the Fusion Edge server is to reject cross domain requests.  To provide CORS policy to certain or all domains the cors.policy property should be set.
 +
 +
cors.policy=[policy 1]@[policy 2]@[policy 3]
 +
 +
Each policy is separated by an '@' symbol, within each policy are the following properties, each separated by a '$' symbol:
 +
 +
# Access-Control-Allow-Origin
 +
# Access-Control-Allow-Headers (comma separated)
 +
# Access-Control-Expose-Headers (comma separated)
 +
# Access-Control-Allow-Credentials (true/false)
 +
 +
'''Example'''
 +
cors.policy=www.metadatatechnology.com$GET,HEAD$*$Content-Disposition$true
 +
 +
'''Example All Domains'''
 +
cors.policy=*$GET,HEAD,POST$*$Content-Disposition$false
 +
 +
= Password Protect Excel Workbooks =
 +
Generated [[Reporting_Template|Reporting Template]] data formats can be set to be password protected using the following property
 +
 +
reporttemplate.password=[password]

Latest revision as of 04:14, 8 November 2023


Overview

The Fusion Edge Server properties file provides configuration options on various aspects of the Edge Server. To set these configurations, create a file under the Edge Server Directory with the name: edgeserver.properties.

This file is required. At a minimum it should be given the ledger.secret property, this is the secret string used when the environment was compiled. Without this the environment can not be read.

Public URL

The Edge Server application and web service URL should be defined in the edge server properties file. These URLs are used in responses to structure queries which include links to associated reference metadata, and stub structures which link back to the full artefact.

The URLs are split into two parts, the application URL, which should be a reference to the public application server, and the rest url which is the POSTfix URL to the web service, for example: 

app.url=https://myserver.myorg.org/Edge
rest.url=/sdmx

The above setting would make the SDMX web service entry point: https://myserver.myorg.org/Edge/sdmx

Sender Id

SDMX Messages contain a Header section which includes the identity of the sender. The identity is mandatory in a SDMX message, and the default ID is set to "UNKNOWN". To change the Id, set the sender Id property 

sender.id=YOUR_ID

Secret Key

The Fusion Edge server must be given a secret key, and it must match that used by the Fusion Edge Compiler in the compilation of the Environment. If the secret key does not match the secret key of the Environment, the Fusion Edge Server will fail to load the Environment. 

ledger.secret=MySecret

Static Mode - File Name

In static mode the Fusion Edge Server will read the properties file to know where the environment is. If nothing is set it will look for a file called node.zip. 

environment.file=LiveEnvironment

Embargo Data

The Fusion Edge server will not release an Environment with an Embargo until the time on the server is later then or equal to the timestamp of the Embargo.

The Fusion Edge Server can be configured to pre-prepare and Environment for release prior to the embargo time. For example if an Environment is scheduled for release at 12.00pm, and the Environment is moved to a secure Amazon S3 file system at 11.50, the Fusion Edge Server can be configured to pull these files into its local file system 5 minutes before go live, to eliminate any risk of network lag pulling the files. The Fusion Ede Server can also be set to pre-load the Environment into memory 30 seconds before go live, the Environment is now ready for dissemination but disconnected from any process which can get to the data, it is still fully secure. The Fusion Edge Server is able to then make the Environment go live by simply swapping the old Environment with the new, making it go live with millisecond precision.

To preload the Environment from a remote location to the local file system prior to the Embargo time, set the preload filesystem property to define the number of milliseconds before the embargo time that it should initiate the preload process. 

ledger.preload.filesystem=40000

To preload the Environment into memory, set the preload memory property to define the number of milliseconds before the embargo time that it should start reading the Environment into memory. 

ledger.preload.memory=20000

In this example the Fusion Edge Server will pull an Environment from the remote location to the local file system 40 seconds prior to the go live time, and then it will pre load the Environment from the file system into memory 20 seconds before the go live time. The Environment will be made live on the go live (embargo) time.

Disable Audit

The Fusion Edge maintains an audit log of events in its local Edge Server Directory. Auditing can be disabled with the following property 

audit.disabled=true

Limiting Series and Observations

To place a limit on the number of series and the number of observations that can be queried via the web service the following two properties are supported: 

limit.obs=12345
limit.series=1234

The numeric value indicates the maximum number of series or observations that can be returned from a single data query.

If the limit is exceeded an HTTP status code of 413 (Payload Too Large) will be returned to the client.

Default Reponse Formats

Since v4.7.0

The Fusion Edge Server supports a number of different response formats for structures, data, and metadata. When the client does not specify which format to return the response in, the Fusion Edge Server will respond in the default format. The properties file can be used to override the default formats by using the vnd header of the format to return.

REST Resource Default Format Override Property
data application/vnd.sdmx.structurespecificdata+xml;version=2.1 default.format.data
structure application/vnd.sdmx.structure+json;version=2.0.0 default.format.structure
availability application/vnd.sdmx.structure+json;version=2.0.0 default.format.availability
tableavailability application/vnd.sdmx.structure+json;version=2.0.0 default.format.tableavailability
table application/json default.format.table
metadata application/vnd.sdmx.metadata+json;version=2.0.0 default.format.metadata

Limiting Supported Data Formats

To enforce a restriction on which data formats are supported from the REST API of the Fusion Edge Server, use the format.data property as follows: 

format.data=[format1],[format2]

This property defines which formats are supported, if this property is not provided all formats will be supported. If this property is provided, only the formats in this list will be supported.

VND Headers are listed in the Data Formats section.

Example:
format.data=application/vnd.sdmx.generictimeseriesdata+xml;version=2.1,application/vnd.sdmx.data+csv

Limiting Supported Structure Formats

To enforce a restriction on which structure formats are supported from the REST API of the Fusion Edge Server, use the format.structure property as follows: 

format.structure=[format1],[format2]

This property defines which formats are supported, if this property is not provided all formats will be supported. If this property is provided, only the formats in this list will be supported.

Each [format] is a VND header the supported formats are:

Structure Format VND Header
SDMX EDI application/vnd.sdmx.structure;version=edi
application/vnd.sdmx.structure+xml;version=1.0 SDMX v1.0
application/vnd.sdmx.structure+xml;version=2.0 SDMX v2.0
application/vnd.sdmx.structure+xml;version=2.1 SDMX v2.1
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet Excel
application/vnd.fusion.json Fusion JSON
application/vnd.sdmx.structure+json;version=1.0.0 SDMX JSON v1.0.0
application/vnd.sdmx.structure+json;version=2.0.0 SDMX JSON v2.0.0

Example:
format.structure=application/vnd.sdmx.structure;version=edi,application/vnd.sdmx.json

Limiting Query Functionality

The extended data query API can have some features disabled by using the following properties: 

rest.query.attribute = false
rest.query.aggregate = false
rest.query.interpolate = false
rest.query.calculatemeasure = false
rest.query.normalisefreq = false
rest.query.round = false

Caching

The Fusion Edge Server caches the response to data queries in the tmp folder of the Edge Directory. The response is stored in gzip format, which ensures clients who request data in gzip format receive a rapid response when the data has been cached due to previous requests for the same data.

The following are important to note:

  • The cache is NOT read on application startup. The cache is generated during the operational running time of the Fusion Edge Server. The cache is a combination of in memory pointers to files in the file system. Once the application server is terminated, it is not re-read on startup. This means any cached files are redundant after the Fusion Edge Server is terminated.
  • The cache is NOT cleared on application startup. This is for a number of reasons, but principally deleting files from a running Edge Server is a slower process then could be achieved via an external mechanism. The delete process consumes CPU and therefore should be performed via a more efficient process before server startup.
  • The cache can be manually cleared by deleting all the files in the tmp directory.
  • The Fusion Edge Server will clear the cache if the Environment is updated in Dynamic mode. The in memory cache is first cleared to ensure clearance with immediate effect. A separate thread is initiated to clear the cached files, this can take time to complete if there are lots of cached files. Files are cleared in batches of 5000 with a 5 second delay per batch, this is to ensure the CPU required to run the deletion process does not block external traffic to the Edge Server.

The cache can be disabled by supplying the following property (Fusion Edge Server v2.3.5 an higher): 

rest.query.cache=false

CSV Data Format Defaults

To modify the default CSV output is returned, the following properties are supported: 

csv.delimiter=comma|tab|space|semicolon
csv.labels=ID|NAME|ID_AND_NAME
csv.showSeries=true|false
csv.includeBom=true|false
csv.keySeparator=:

The delimiter is the value of the character that will be output between the elements (defaults to comma).

The values for 'Labels' must be one of the following: ID; NAME; ID_AND_NAME (defaults to ID_AND_NAME). Show Series defines whether to output the series key in the output (defaults to true).

Cross-Origin Resource Sharing (CORS)

By default the CORS policy of the Fusion Edge server is to reject cross domain requests. To provide CORS policy to certain or all domains the cors.policy property should be set. 

cors.policy=[policy 1]@[policy 2]@[policy 3]

Each policy is separated by an '@' symbol, within each policy are the following properties, each separated by a '$' symbol:

  1. Access-Control-Allow-Origin
  2. Access-Control-Allow-Headers (comma separated)
  3. Access-Control-Expose-Headers (comma separated)
  4. Access-Control-Allow-Credentials (true/false)

Example cors.policy=www.metadatatechnology.com$GET,HEAD$*$Content-Disposition$true

Example All Domains cors.policy=*$GET,HEAD,POST$*$Content-Disposition$false

Password Protect Excel Workbooks

Generated Reporting Template data formats can be set to be password protected using the following property 

reporttemplate.password=[password]
Retrieved from "https://wiki.sdmxcloud.org/index.php?title=Fusion_Edge_Server_Properties&oldid=6540"