Edge Server - Publish Content

Overview

Moving information to the Fusion Edge server requires three steps:

1. Source Data and Metadata Files - A file system is required with source information (datasets and related metadata) which is to be published. The files are in SDMX format.
2. Compiled Data and Metadata Files. A file system which contains the information from the SDMX files, compiled into an Environment. The files are in a format which can be read by the Fusion Edge Server.
3. Moving the Environment. This step makes the Environment available to the Fusion Edge Server

The Fusion Edge Compiler is a command line application which can be used for all 3 steps.

An [Example Architecture] shows how these steps in a set up which contains multiple servers each with a copy of an Environment.

Part 1 - Source Data and Metadata Files

Content is published to the Fusion Edge Server by compiling datasets, structure files, and reference metadata files that are present in a local file system. The compilation process is run using the Fusion Edge Compiler. The Fusion Edge Compiler is given the root folder as an argument and it expects to find the following folder structure under the root folder:

|- data
|-- [agency id]
|---- [dataflow id]
|------ [dataflow version]  (data files are placed in this folder)
|- structure (structure files are placed in this folder)
|- metadata (metadata files are placed in this folder)

Where agency id, dataflow id, and dataflow version are specific to the Dataflows that the data are for.

The content can be in any SDMX format. Each folder can contain multiple files, the compiler will merge the information where required.

An example folder/file content is given below:

|- data
|-- WB
|---- POVERTY
|------ 1.0 
|-------- PovertyData.zip
|-------- PovertyUpdate.xml
|---- EDUCATION
|------ 1.0 
|-------- EduData_1990_2010.json
|-------- EduData2010_2020.xml
|- structure 
|-- corestructures.zip
|-- categories.xml
|-- msds.xml
|- metadata 
|--metadataset1.zip
|--metadataset2.zip

It is possible to manually build this file system, however the Fusion Edge Compiler can be used to obtain information directly from an SDMX REST API to automate the process of building the File system.

Part 2 - Compiled Environment

The Fusion Edge Compiler compiles the source file system into an Environment.

Part 3 - Publish

There are 2 main modes of publication, static mode and dynamic mode.

​Static Mode

Static mode is termed as such because the Fusion Edge Server loads the Environment on application startup, and it never changes the Environment after this. The contents of the Fusion Edge Server are static, and can only be changed by restarting the web application server.

To use static mode, the contents of the Environment folder is zipped to producing a single zip file. The zip file is placed on the file system in the home directory of the Fusion Edge Server. The edgeserver.properties file is configured to define the name of the Environment zip file that the Fusion Edge Server should load on application startup.

The Fusion Edge Server will read the zip file on application startup and load the Environment into memory. The only way a new Environment can be deployed is by restarting the web application server.

Dynamic Mode

Dynamic mode is termed as such because the Fusion Edge Server can update its Environment without an application restart. It does so by periodically polling the Environment's ledger file to check for a change in version number. When a change is detected, the new Environment is loaded.

To run in dynamic mode, the Environment file system must be moved to a location which can be read by the Fusion Edge Server. The environment must not be zipped, and the folder structure must no be changed from the compiled output.

The Fusion Edge Server can read an Environment from either:

1. a file system local to the Fusion Edge Server
2. A location where the files can be obtained via a URL
3. Amazon S3

File System

The Fusion Edge Server is given the root folder of the Environment.

URL Environment

The Fusion Edge Server is given the URL to the root folder, for example https://mydomain.org/subfolder/edge-conent. The Fusion Edge Server will expect to find all files and folders as a subpath to this URL, for example:

https://mydomain.org/subfolder/edge-conent/fes_ledger.json
https://mydomain.org/subfolder/edge-conent/data/[data file]

Amazon S3

The Fusion Edge Server is given the Amazon S3 bucket, region, secret key, and access key. The contents of the bucket are expected to be the files and folders of the Environment.

Moving an Environment

Static Mode

In static Mode an Environment can be moved to the target server without problem because the file it overwrites is not read until web application startup.

Dynamic Mode

In dynamic mode it is important to move the fes_ledger.json file last, after the Environment has been moved, because it is this file that informs the Fusion Edge Server to update its contents. It is possible to delete the existing Environment first, the Fusion Edge Server will not be able to poll the fes_ledger.json file if the live Environment is deleted, but it will continue to operate as normal. The new Environment can be copied to the folder, with the file file being the new fes_ledger.json file.

Another technique for copying files is to delete any files in the target Environment that do not exist in the new Environment, and then copy any files which are only in the new Environment. Then finally copy the new fes_ledger.json file.

Automating the Process

The Fusion Edge Compiler does not provide any support for moving an Environment to a file system or URL. However it does provide support for publishing content to Amazon S3. It is recommended to set up an script to automate the process of publishing a new Environment to the Fusion Edge Server.