Difference between revisions of "Edge Server - Publish Content"

From Fusion Registry Wiki
Jump to navigation Jump to search
(→‎Part 1 - Source Data and Metadata Files)
 
(20 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 +
[[Category:Fusion Edge Server]]
 +
[[Category:How_to_FES]]
 
== Overview ==
 
== Overview ==
  
−
There are three parts to publishing content
+
Moving information to the Fusion Edge server requires three steps:
  
−
# Source Data and Metadata Files
+
# [[#Part_1_-_Source_Data_and_Metadata_Files|Source Data and Metadata Files]] - A file system is required with source information (datasets and related metadata) which is to be published. The files are in SDMX format.
−
# Compiled Data and Metadata Files (the [[Edge_Server_Environment|Environment]])
+
# [[#Part_2_-_Compiled_Environment|Compiled Data and Metadata Files]].  A file system which contains the information from the SDMX files, compiled into an [[Edge_Server_Environment|Environment]]. The files are in a format which can be read by the Fusion Edge Server.
−
# Published ([[Edge_Server_Environment|Environment]])
+
# [[#Moving_an_Environment|Moving the Environment]] . This step makes the [[Edge_Server_Environment|Environment]] available to the Fusion Edge Server
 +
 
 +
The [[Fusion_Edge_Compiler|Fusion Edge Compiler]] is a command line application which can be used for all 3 steps.
 +
 
 +
An [https://wiki.sdmxcloud.org/images/2/28/FusionEdgeServer_Deployment_Architecture.png Example Architecture] shows how these steps in a set up which contains multiple servers each with a copy of an Environment.
  
 
== Part 1 - Source Data and Metadata Files ==
 
== Part 1 - Source Data and Metadata Files ==
Line 17: Line 23:
 
  |- metadata (metadata files are placed in this folder)
 
  |- metadata (metadata files are placed in this folder)
  
−
Where agency id, dataflow id, and dataflow version are specific to the Dataflows that the data are for.  The content can be in any SDMX format, each folder can contain multiple files, the compiler will merge the information where required.
+
Where agency id, dataflow id, and dataflow version are specific to the Dataflows that the data are for.   
  
−
'''Note''': The Fusion Edge Compiler can build this local file system automatically from content pulled from compliant SDMX web services such as those provided by Fusion Registry. Information is provided later about how this is achieved.
+
The content can be in any SDMX format. Each folder can contain multiple files, the compiler will merge the information where required.
  
 
An example folder/file content is given below:
 
An example folder/file content is given below:
Line 41: Line 47:
 
  |--metadataset2.zip
 
  |--metadataset2.zip
  
−
The files in the file system must be in SDMX format, and may be individually zipped.  Each folder may contain multiple files.  The compilation process will combine all the files in each folder to create a consolidated output.  For example a dataflow folder may contain multiple dataset instances with different series or time periods, the output will be a single compiled dataset instance built from all the dataset files.
 
  
 +
It is possible to manually build this file system, however the Fusion Edge Compiler can be used to [[Fusion_Edge_Compiler#Pull_Content|obtain information directly]] from an SDMX REST API to automate the process of building the File system.
  
 
== Part 2 - Compiled Environment ==
 
== Part 2 - Compiled Environment ==
 +
The Fusion Edge Compiler [[Fusion_Edge_Compiler#Compile_Content|compiles]] the source file system into an [[Edge_Server_Environment|Environment]].
 +
 +
== Part 3 - Publish ==
 +
There are 2 main modes of publication, static mode and dynamic mode. 
 +
 +
=== ​Static Mode ===
 +
Static mode is termed as such because the Fusion Edge Server loads the [[Edge_Server_Environment|Environment]] on application startup, and the Fusion Edge Server never checks for Environment updates after launch.  The contents of the Fusion Edge Server are static, and can only be changed by restarting the web application server.
 +
 +
To use static mode,the '''environment.file''' property should be set in the [[Fusion_Edge_Server_Properties|'''edgeserver.properties''']] file.  The properties file is located in the Fusion Edge Server [[Fusion_Edge_Server_Directory|home directory]].  The [[Edge_Server_Environment|Environment]] is accessed as a local file or folder, the path to the file or folder can be relative to the home directory.
 +
 +
Example:
 +
 +
environment.file=LiveEnvironment
 +
 +
Note: in Fusion Edge Server 4.4.0 and later the Environment can be referenced as a folder, previous versions required the Environment to be zipped and the environment.file had to reference the zip file.
 +
 +
The Fusion Edge Server will read the Environment on application startup and load the Environment into memory.  The only way a new Environment can be deployed is by restarting the web application server.
 +
 +
=== Dynamic Mode ===
 +
Dynamic mode is termed as such because the Fusion Edge Server can update its Environment without an application restart.  It does so by periodically polling the Environment's [[Edge_Server_Environment#Fusion_Edge_Server_Ledger|ledger file]] to check for a change in version number.  When a change is detected, the new Environment is loaded.
 +
 +
To run in dynamic mode, the Environment file system must be moved to a location which can be read by the Fusion Edge Server.  The environment must not be zipped, and the folder structure must no be changed from the compiled output.
 +
 +
The Fusion Edge Server can read an Environment from either:
 +
 +
# a file system local to the Fusion Edge Server
 +
# A location where the files can be obtained via a URL
 +
# Amazon S3
 +
 +
 +
==== File System ====
 +
The Fusion Edge Server is given the root folder of the Environment.
 +
 +
==== URL Environment ====
 +
The Fusion Edge Server is given the URL to the root folder, for example https://mydomain.org/subfolder/edge-conent.  The Fusion Edge Server will expect to find all files and folders as a subpath to this URL, for example:
 +
 +
https://mydomain.org/subfolder/edge-conent/fes_ledger.json
 +
https://mydomain.org/subfolder/edge-conent/data/[data file]
 +
 +
==== Amazon S3 ====
 +
The Fusion Edge Server is given the Amazon S3 bucket, region, secret key, and access key.  The contents of the bucket are expected to be the files and folders of the Environment.  The Fusion Edge Compiler can be used to [[Fusion_Edge_Compiler#Publish_Content|publish an Environment to Amazon S3]].
 +
 +
== Moving an Environment ==
 +
=== Static Mode ===
 +
In static Mode an Environment can be moved to the target server without problem because the file it overwrites is not read until web application startup. 
 +
 +
=== Dynamic Mode ===
 +
In dynamic mode it is important to move the fes_ledger.json file last, after the Environment has been moved, because it is this file that informs the Fusion Edge Server to update its contents.  It is possible to delete the existing Environment first, the Fusion Edge Server will not be able to poll the fes_ledger.json file if the live Environment is deleted, but it will continue to operate as normal.  The new Environment can be copied to the folder, with the file file being the new fes_ledger.json file.
 +
 +
Another technique for copying files is to delete any files in the target Environment that do not exist in the new Environment, and then copy any files which are only in the new Environment.  Then finally copy the new fes_ledger.json file.
 +
 +
=== Automating the Process ===
 +
The Fusion Edge Compiler does not provide any support for moving an Environment to a file system or URL.  However it does provide support for [[Fusion_Edge_Compiler#Publish_Content|publishing content to Amazon S3]]. It is recommended to set up an script to automate the process of publishing a new Environment to the Fusion Edge Server.

Latest revision as of 07:32, 11 September 2023

Overview

Moving information to the Fusion Edge server requires three steps:

  1. Source Data and Metadata Files - A file system is required with source information (datasets and related metadata) which is to be published. The files are in SDMX format.
  2. Compiled Data and Metadata Files. A file system which contains the information from the SDMX files, compiled into an Environment. The files are in a format which can be read by the Fusion Edge Server.
  3. Moving the Environment . This step makes the Environment available to the Fusion Edge Server

The Fusion Edge Compiler is a command line application which can be used for all 3 steps.

An Example Architecture shows how these steps in a set up which contains multiple servers each with a copy of an Environment.

Part 1 - Source Data and Metadata Files

Content is published to the Fusion Edge Server by compiling datasets, structure files, and reference metadata files that are present in a local file system. The compilation process is run using the Fusion Edge Compiler. The Fusion Edge Compiler is given the root folder as an argument and it expects to find the following folder structure under the root folder: 

|- data
|-- [agency id]
|---- [dataflow id]
|------ [dataflow version]  (data files are placed in this folder)
|- structure (structure files are placed in this folder)
|- metadata (metadata files are placed in this folder)

Where agency id, dataflow id, and dataflow version are specific to the Dataflows that the data are for.

The content can be in any SDMX format. Each folder can contain multiple files, the compiler will merge the information where required.

An example folder/file content is given below: 

|- data
|-- WB
|---- POVERTY
|------ 1.0 
|-------- PovertyData.zip
|-------- PovertyUpdate.xml
|---- EDUCATION
|------ 1.0 
|-------- EduData_1990_2010.json
|-------- EduData2010_2020.xml
|- structure 
|-- corestructures.zip
|-- categories.xml
|-- msds.xml
|- metadata 
|--metadataset1.zip
|--metadataset2.zip


It is possible to manually build this file system, however the Fusion Edge Compiler can be used to obtain information directly from an SDMX REST API to automate the process of building the File system.

Part 2 - Compiled Environment

The Fusion Edge Compiler compiles the source file system into an Environment.

Part 3 - Publish

There are 2 main modes of publication, static mode and dynamic mode.

​Static Mode

Static mode is termed as such because the Fusion Edge Server loads the Environment on application startup, and the Fusion Edge Server never checks for Environment updates after launch. The contents of the Fusion Edge Server are static, and can only be changed by restarting the web application server.

To use static mode,the environment.file property should be set in the edgeserver.properties file. The properties file is located in the Fusion Edge Server home directory. The Environment is accessed as a local file or folder, the path to the file or folder can be relative to the home directory.

Example: 

environment.file=LiveEnvironment

Note: in Fusion Edge Server 4.4.0 and later the Environment can be referenced as a folder, previous versions required the Environment to be zipped and the environment.file had to reference the zip file.

The Fusion Edge Server will read the Environment on application startup and load the Environment into memory. The only way a new Environment can be deployed is by restarting the web application server.

Dynamic Mode

Dynamic mode is termed as such because the Fusion Edge Server can update its Environment without an application restart. It does so by periodically polling the Environment's ledger file to check for a change in version number. When a change is detected, the new Environment is loaded.

To run in dynamic mode, the Environment file system must be moved to a location which can be read by the Fusion Edge Server. The environment must not be zipped, and the folder structure must no be changed from the compiled output.

The Fusion Edge Server can read an Environment from either:

  1. a file system local to the Fusion Edge Server
  2. A location where the files can be obtained via a URL
  3. Amazon S3


File System

The Fusion Edge Server is given the root folder of the Environment.

URL Environment

The Fusion Edge Server is given the URL to the root folder, for example https://mydomain.org/subfolder/edge-conent. The Fusion Edge Server will expect to find all files and folders as a subpath to this URL, for example: 

https://mydomain.org/subfolder/edge-conent/fes_ledger.json
https://mydomain.org/subfolder/edge-conent/data/[data file]

Amazon S3

The Fusion Edge Server is given the Amazon S3 bucket, region, secret key, and access key. The contents of the bucket are expected to be the files and folders of the Environment. The Fusion Edge Compiler can be used to publish an Environment to Amazon S3.

Moving an Environment

Static Mode

In static Mode an Environment can be moved to the target server without problem because the file it overwrites is not read until web application startup.

Dynamic Mode

In dynamic mode it is important to move the fes_ledger.json file last, after the Environment has been moved, because it is this file that informs the Fusion Edge Server to update its contents. It is possible to delete the existing Environment first, the Fusion Edge Server will not be able to poll the fes_ledger.json file if the live Environment is deleted, but it will continue to operate as normal. The new Environment can be copied to the folder, with the file file being the new fes_ledger.json file.

Another technique for copying files is to delete any files in the target Environment that do not exist in the new Environment, and then copy any files which are only in the new Environment. Then finally copy the new fes_ledger.json file.

Automating the Process

The Fusion Edge Compiler does not provide any support for moving an Environment to a file system or URL. However it does provide support for publishing content to Amazon S3. It is recommended to set up an script to automate the process of publishing a new Environment to the Fusion Edge Server.

Retrieved from "https://wiki.sdmxcloud.org/index.php?title=Edge_Server_-_Publish_Content&oldid=6116"