Difference between revisions of "Edge Server - Publish Content"
(→Part 1 - Source Data and Metadata Files) |
|||
| (9 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
| + | [[Category:Fusion Edge Server]] | ||
| + | [[Category:How_to_FES]] | ||
== Overview == | == Overview == | ||
Moving information to the Fusion Edge server requires three steps: | Moving information to the Fusion Edge server requires three steps: | ||
| − | # Source Data and Metadata Files - A file system is required with source information (datasets and related metadata) which is to be published. The files are in SDMX format. | + | # [[#Part_1_-_Source_Data_and_Metadata_Files|Source Data and Metadata Files]] - A file system is required with source information (datasets and related metadata) which is to be published. The files are in SDMX format. |
| − | # Compiled Data and Metadata Files. A file system which contains the information from the SDMX files, compiled into an [[Edge_Server_Environment|Environment]]. The files are in a format which can be read by the Fusion Edge Server. | + | # [[#Part_2_-_Compiled_Environment|Compiled Data and Metadata Files]]. A file system which contains the information from the SDMX files, compiled into an [[Edge_Server_Environment|Environment]]. The files are in a format which can be read by the Fusion Edge Server. |
| − | # | + | # [[#Moving_an_Environment|Moving the Environment]] . This step makes the [[Edge_Server_Environment|Environment]] available to the Fusion Edge Server |
The [[Fusion_Edge_Compiler|Fusion Edge Compiler]] is a command line application which can be used for all 3 steps. | The [[Fusion_Edge_Compiler|Fusion Edge Compiler]] is a command line application which can be used for all 3 steps. | ||
| − | An | + | An [https://wiki.sdmxcloud.org/images/2/28/FusionEdgeServer_Deployment_Architecture.png Example Architecture] shows how these steps in a set up which contains multiple servers each with a copy of an Environment. |
== Part 1 - Source Data and Metadata Files == | == Part 1 - Source Data and Metadata Files == | ||
| Line 55: | Line 57: | ||
=== Static Mode === | === Static Mode === | ||
| − | Static mode is termed as such because the Fusion Edge Server loads the Environment on application startup, and | + | Static mode is termed as such because the Fusion Edge Server loads the [[Edge_Server_Environment|Environment]] on application startup, and the Fusion Edge Server never checks for Environment updates after launch. The contents of the Fusion Edge Server are static, and can only be changed by restarting the web application server. |
| − | To use static mode, the | + | To use static mode,the '''environment.file''' property should be set in the [[Fusion_Edge_Server_Properties|'''edgeserver.properties''']] file. The properties file is located in the Fusion Edge Server [[Fusion_Edge_Server_Directory|home directory]]. The [[Edge_Server_Environment|Environment]] is accessed as a local file or folder, the path to the file or folder can be relative to the home directory. |
| − | The Fusion Edge Server will read the | + | Example: |
| + | |||
| + | environment.file=LiveEnvironment | ||
| + | |||
| + | Note: in Fusion Edge Server 4.4.0 and later the Environment can be referenced as a folder, previous versions required the Environment to be zipped and the environment.file had to reference the zip file. | ||
| + | |||
| + | The Fusion Edge Server will read the Environment on application startup and load the Environment into memory. The only way a new Environment can be deployed is by restarting the web application server. | ||
=== Dynamic Mode === | === Dynamic Mode === | ||
| Line 83: | Line 91: | ||
==== Amazon S3 ==== | ==== Amazon S3 ==== | ||
| − | The Fusion Edge Server is given the Amazon S3 bucket, region, secret key, and access key. The contents of the bucket are expected to be the files and folders of the Environment. | + | The Fusion Edge Server is given the Amazon S3 bucket, region, secret key, and access key. The contents of the bucket are expected to be the files and folders of the Environment. The Fusion Edge Compiler can be used to [[Fusion_Edge_Compiler#Publish_Content|publish an Environment to Amazon S3]]. |
== Moving an Environment == | == Moving an Environment == | ||
| Line 94: | Line 102: | ||
Another technique for copying files is to delete any files in the target Environment that do not exist in the new Environment, and then copy any files which are only in the new Environment. Then finally copy the new fes_ledger.json file. | Another technique for copying files is to delete any files in the target Environment that do not exist in the new Environment, and then copy any files which are only in the new Environment. Then finally copy the new fes_ledger.json file. | ||
| − | == Automating the Process == | + | === Automating the Process === |
| − | The Fusion Edge Compiler does not provide any support for moving an Environment to a file system or URL. However it does provide support for publishing content to Amazon S3. It is recommended to set up an script to automate the process of publishing a new Environment to the Fusion Edge Server. | + | The Fusion Edge Compiler does not provide any support for moving an Environment to a file system or URL. However it does provide support for [[Fusion_Edge_Compiler#Publish_Content|publishing content to Amazon S3]]. It is recommended to set up an script to automate the process of publishing a new Environment to the Fusion Edge Server. |
Latest revision as of 07:32, 11 September 2023
Contents
Overview
Moving information to the Fusion Edge server requires three steps:
- Source Data and Metadata Files - A file system is required with source information (datasets and related metadata) which is to be published. The files are in SDMX format.
- Compiled Data and Metadata Files. A file system which contains the information from the SDMX files, compiled into an Environment. The files are in a format which can be read by the Fusion Edge Server.
- Moving the Environment . This step makes the Environment available to the Fusion Edge Server
The Fusion Edge Compiler is a command line application which can be used for all 3 steps.
An Example Architecture shows how these steps in a set up which contains multiple servers each with a copy of an Environment.
Part 1 - Source Data and Metadata Files
Content is published to the Fusion Edge Server by compiling datasets, structure files, and reference metadata files that are present in a local file system. The compilation process is run using the Fusion Edge Compiler. The Fusion Edge Compiler is given the root folder as an argument and it expects to find the following folder structure under the root folder:
|- data |-- [agency id] |---- [dataflow id] |------ [dataflow version] (data files are placed in this folder) |- structure (structure files are placed in this folder) |- metadata (metadata files are placed in this folder)
Where agency id, dataflow id, and dataflow version are specific to the Dataflows that the data are for.
The content can be in any SDMX format. Each folder can contain multiple files, the compiler will merge the information where required.
An example folder/file content is given below:
|- data |-- WB |---- POVERTY |------ 1.0 |-------- PovertyData.zip |-------- PovertyUpdate.xml |---- EDUCATION |------ 1.0 |-------- EduData_1990_2010.json |-------- EduData2010_2020.xml |- structure |-- corestructures.zip |-- categories.xml |-- msds.xml |- metadata |--metadataset1.zip |--metadataset2.zip
It is possible to manually build this file system, however the Fusion Edge Compiler can be used to obtain information directly from an SDMX REST API to automate the process of building the File system.
Part 2 - Compiled Environment
The Fusion Edge Compiler compiles the source file system into an Environment.
Part 3 - Publish
There are 2 main modes of publication, static mode and dynamic mode.
Static Mode
Static mode is termed as such because the Fusion Edge Server loads the Environment on application startup, and the Fusion Edge Server never checks for Environment updates after launch. The contents of the Fusion Edge Server are static, and can only be changed by restarting the web application server.
To use static mode,the environment.file property should be set in the edgeserver.properties file. The properties file is located in the Fusion Edge Server home directory. The Environment is accessed as a local file or folder, the path to the file or folder can be relative to the home directory.
Example:
environment.file=LiveEnvironment
Note: in Fusion Edge Server 4.4.0 and later the Environment can be referenced as a folder, previous versions required the Environment to be zipped and the environment.file had to reference the zip file.
The Fusion Edge Server will read the Environment on application startup and load the Environment into memory. The only way a new Environment can be deployed is by restarting the web application server.
Dynamic Mode
Dynamic mode is termed as such because the Fusion Edge Server can update its Environment without an application restart. It does so by periodically polling the Environment's ledger file to check for a change in version number. When a change is detected, the new Environment is loaded.
To run in dynamic mode, the Environment file system must be moved to a location which can be read by the Fusion Edge Server. The environment must not be zipped, and the folder structure must no be changed from the compiled output.
The Fusion Edge Server can read an Environment from either:
- a file system local to the Fusion Edge Server
- A location where the files can be obtained via a URL
- Amazon S3
File System
The Fusion Edge Server is given the root folder of the Environment.
URL Environment
The Fusion Edge Server is given the URL to the root folder, for example https://mydomain.org/subfolder/edge-conent. The Fusion Edge Server will expect to find all files and folders as a subpath to this URL, for example:
https://mydomain.org/subfolder/edge-conent/fes_ledger.json https://mydomain.org/subfolder/edge-conent/data/[data file]
Amazon S3
The Fusion Edge Server is given the Amazon S3 bucket, region, secret key, and access key. The contents of the bucket are expected to be the files and folders of the Environment. The Fusion Edge Compiler can be used to publish an Environment to Amazon S3.
Moving an Environment
Static Mode
In static Mode an Environment can be moved to the target server without problem because the file it overwrites is not read until web application startup.
Dynamic Mode
In dynamic mode it is important to move the fes_ledger.json file last, after the Environment has been moved, because it is this file that informs the Fusion Edge Server to update its contents. It is possible to delete the existing Environment first, the Fusion Edge Server will not be able to poll the fes_ledger.json file if the live Environment is deleted, but it will continue to operate as normal. The new Environment can be copied to the folder, with the file file being the new fes_ledger.json file.
Another technique for copying files is to delete any files in the target Environment that do not exist in the new Environment, and then copy any files which are only in the new Environment. Then finally copy the new fes_ledger.json file.
Automating the Process
The Fusion Edge Compiler does not provide any support for moving an Environment to a file system or URL. However it does provide support for publishing content to Amazon S3. It is recommended to set up an script to automate the process of publishing a new Environment to the Fusion Edge Server.
