Edge Server Data Preperation Web Service
Contents
Overview
Fusion Edge Server is a high speed secure server for data and metadata dissemination. Publication of data from the Fusion Registry to the Fusion Edge Server is achieved by a number of steps, which include:
- Defining a Edge Environment - this defines information such as what Dataflows to include data for in the published content.
- Generating the Edge content - this is an asynchronous service which runs in the background after it receives instruction to launch
- Exporting Edge Content - this exports the edge content generated in step 2
Web services exist for all these steps, the Fusion Registry User Interface can also be used to perform the same actions, the UI also uses these same web services.
Manage Edge Environments
An Edge Environment is a container for a description of what gets put into the edge content. For example if there are 2 Edge Server clusters, one for public data and one for internal use within the organisation, the there would be 2 Edge Environment definitions in the Fusion Registry - where the Dataflows in each environment differ based on what data is to be made public and which data is to be disseminated internally. The Identity and description of the environment is just for local use, there is no connection between this information and what is published in the edge content.
Add Environment
To add a new Environment, the Id and description need to be provided - after an environment is created, it is possible to use other web services to define what is published to the environment.
| Entry Point | /ws/secure/edge/addEnvironment |
| Access | Secure (Admin) |
| Http Method | POST |
| Compression | gzip supported |
| Response Format | JSON |
| Response Statuses | 200 - Ok 400 - Bad URL Syntax 401 - Unauthorised (authenticated no permission to perform action) 403 - Forbidden (not authenticated) 500 - Server Error |
POST Syntax
{
"Id" : "EnvironmentId",
"Description" : "Environment Description"
}
Edit Environment
This web service allows for the Id and/or description of the environment to be updated
| Entry Point | /ws/secure/edge/editEnvironment |
| Access | Secure (Admin) |
| Http Method | POST |
| Compression | gzip supported |
| Response Format | JSON |
| Response Statuses | 200 - Ok 400 - Bad URL Syntax 401 - Unauthorised (authenticated no permission to perform action) 403 - Forbidden (not authenticated) 500 - Server Error |
POST Syntax
{
"Id" : "EnvironmentId_New",
"OldId" : "EnvironmentId_Old"
"Description" : "Environment Description Updated"
}
Environment Dataflows
This web service is used to supply the list of Dataflows to be included in the publication for the environment, the list overwrites any list that was previously provided
| Entry Point | /ws/secure/edge/setIncludedDataflows |
| Access | Secure (Admin) |
| Http Method | POST |
| Compression | gzip supported |
| Response Format | JSON |
| Response Statuses | 200 - Ok 400 - Bad URL Syntax 401 - Unauthorised (authenticated no permission to perform action) 403 - Forbidden (not authenticated) 500 - Server Error |
POST Syntax
{
"Id" : "EnvironmentId",
"IncludedDataflows" : ["urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=ECB:EXR(1.0)", "urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=ECB:TRD(1.0)"]
}
Environment Metadata
This web service is used to define if reference metadata is to be included in the content or not
| Entry Point | /ws/secure/edge/setIncludedMetadata |
| Access | Secure (Admin) |
| Http Method | POST |
| Compression | gzip supported |
| Response Format | JSON |
| Response Statuses | 200 - Ok 400 - Bad URL Syntax 401 - Unauthorised (authenticated no permission to perform action) 403 - Forbidden (not authenticated) 500 - Server Error |
POST Syntax
{
"Id" : "EnvironmentId",
"IncludeMetadata" : true
}
Environment Provision Agreements
This web service is used to define if Provision Agreements are to be included in the content, if they are included it will be possible to query a dataset by both Dataflow and Data Provider, if they are not included then all data for a Dataflow will be stored at the level of the dataflow - the data provider information will be lost.
| Entry Point | /ws/secure/edge/setIncludedProvisions |
| Access | Secure (Admin) |
| Http Method | POST |
| Compression | gzip supported |
| Response Format | JSON |
| Response Statuses | 200 - Ok 400 - Bad URL Syntax 401 - Unauthorised (authenticated no permission to perform action) 403 - Forbidden (not authenticated) 500 - Server Error |
POST Syntax
{
"Id" : "EnvironmentId",
"IncludeProvisions" : false
}
Environment Security
In generating the Edge Server content, the Fusion Registry will treat the Edge Server Environment as a user, by default the user will be anonymous and will be grated the same permissions on structures and data as a user who has not logged in. Any secure content will not be included in the published content. It is possible to link the Fusion Edge Server to a specific Organisation (SDMX Agency, Data Provider, Data Consumer) in order to grant the edge server access to secure content. The Fusion Registry security rules should then be used to grant permissions to the Organisation that the Fusion Edge Environment is linked to.
| Entry Point | /ws/secure/edge/setOrganisation |
| Access | Secure (Admin) |
| Http Method | POST |
| Compression | gzip supported |
| Response Format | JSON |
| Response Statuses | 200 - Ok 400 - Bad URL Syntax 401 - Unauthorised (authenticated no permission to perform action) 403 - Forbidden (not authenticated) 500 - Server Error |
POST Syntax
{
"Id" : "EnvironmentId",
"OrganisationUrn" : "urn:sdmx:org.sdmx.infomodel.base.DataProvider=ORG1:DATA_CONSUMERS(1.0).EDGE_PUBLIC"
}
Build Publication Content
These web services are used to request the edge content is built from the environment description, and exported. There are three steps:
- Build Request
- Poll Build Status
- Export node.zip file, used to publish to the Fusion Edge Server
Build Request
The build request is an asynchronous request, which starts off the build process.
| Entry Point | /ws/secure/edge/buildEdgeServerPublication |
| Access | Secure (Admin) |
| Http Method | POST |
| Compression | gzip supported |
| Response Format | JSON |
| Response Statuses | 200 - Ok 400 - Bad URL Syntax 401 - Unauthorised (authenticated no permission to perform action) 403 - Forbidden (not authenticated) 500 - Server Error |
POST Syntax
{
"Id" : "EnvironmentId",
"PublicationType" : "FullReplace" | "Replace"
"EmbargoDate" : 2345679987 //Unix time millliseconds
}
A publication type of FullReplace will generate a node.zip file which will replace all Fusion Edge server content, a Replace action will produce a delta, which only includes data that have changed since the last publication time. A delta publication can create new series and observations, or replace existing ones that have been modified.
The Embargo Date is a Unix timestamp (number of milliseconds since 1970). The content of the node.zip file does not change in any way (there is no encryption) the timestamp is simply an indicator for the Fusion Edge Server not to make the data live until the server time has passed the data indicated by the given timestamp. The Fusion Edge server does this by importing the node.zip file and building the new environment in memory, but only links the environment up when the time has passed. This allows for the data to be released exactly on the time specified.
Poll Build Status
The build request is an asynchronous request, which starts off the build process.
| Entry Point | /ws/secure/edge/environment/status/{ENVIRONMENT_ID} |
| Access | Secure (Admin) |
| Http Method | GET |
| Compression | gzip supported |
| Response Format | JSON |
| Response Statuses | 200 - Ok 400 - Bad URL Syntax 401 - Unauthorised (authenticated no permission to perform action) 403 - Forbidden (not authenticated) 500 - Server Error |
Response Syntax
{
"id" : 1,
"processorStartDate" : 123456,
"environmentId" : "PUBLIC_EDGE",
"status" : "PROCESSING" | "WRITING_STRUCTURES" | "WRITING_DATA" | "SUCCESSFUL" | "ERRORED" | "CANCELLED",
"auditTxId" : "uuid"
}
Export Edge Content
The request pulls down the latest node.zip file that was successfully generated for the given environment
| Entry Point | /ws/secure/edge/export/{ENVIRONMENT_ID} |
| Access | Secure (Admin) |
| Http Method | GET |
| Compression | gzip supported |
| Response Format | octet-stream |
| Response Statuses | 200 - Ok 400 - Bad URL Syntax 401 - Unauthorised (authenticated no permission to perform action) 403 - Forbidden (not authenticated) 500 - Server Error |
