Difference between revisions of "Edge Server Environment"

From Fusion Registry Wiki
Jump to navigation Jump to search
(→‎Overview)
(→‎Environment Contents)
 
(6 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 +
[[Category:Fusion Edge Server]]
 
== Overview ==
 
== Overview ==
−
The Fusion Edge Server is responsible for exposing data and supporting metadata via a number of read only web service APIs.  The data and metadata is the Environment, changes to the information in the data or metadata can only be made by supplying a new Environment.  
+
The Fusion Edge Server is responsible for exposing data and supporting metadata via a number of read only web service APIs.  The data and metadata is the Environment, changes to the information in the data or metadata can only be made by supplying a new Environment. The Environment can be thought of as an immutable (can not be changed) copy of everything that is to be disseminated. As an Environment can not be modified once built, the only way to update information is to build a new Environment.  New Environments can be built either from a complete set of source data and metadata files, or by merging new information (data/metadata) into an existing Environment, which results in a new Environment. 
  
−
This design means the Fusion Edge Server, when running in dynamic mode, will only ever reapply a complete Environment (in totality).
+
The Environment is made up of a collection of files, which are generated the output of the compile process of the [[Fusion_Edge_Compiler|Fusion Edge Compiler]]. These files contain prebuilt datastores for structures, data, metadata, prebuilt free text search indexes, and other pre-cached information which is used in dissemination.  
  
−
The Environment is made up of a collection of files, which are generated the output of the compile process of the Fusion Edge Compiler. These files contain prebuilt datastores for structures, data, metadata, prebuilt free text search indexes, and other pre-cached information which is used in dissemination. 
+
The only way to modify an Environment is to re-run the compile process of the [[Fusion_Edge_Compiler|Fusion Edge Compiler]].  A built Environment can not be manually modified as it will result in corruption due to the mismatch in the digital [[Fusion_Edge_Compiler#Signing_Content|signature]] applied to the Environment at compile time.
−
 
 
−
The only way to modify an Environment is to re-run the compile process of the Fusion Edge Compiler.  A built Environment can not be manually modified as it will result in corruption due to the mismatch in the digital signature applied to the Environment at compile time.
 
−
 
 
−
The Fusion Edge Compiler can create a new Environment by merging in new information into an existing Environment - for example when compiling a few additional Observations into an existing dataset.  However, the output is always a new Environment with the new information added in.  In this way Environments are fully immutable, they can only be changed by taking a copy of a previous Environment and adding to it.
 
  
 
== Environment Contents ==
 
== Environment Contents ==
Line 23: Line 20:
 
  '''xref'''      holds a prebuilt cache of cross reference information
 
  '''xref'''      holds a prebuilt cache of cross reference information
 
  fes_ledger.json  describes the release history of environments, holds embargo timestamps
 
  fes_ledger.json  describes the release history of environments, holds embargo timestamps
 +
 +
== Fusion Edge Server Ledger ==
 +
The fes_ledger.json file is important when running in [[Edge_Server_-_Publish_Content#Dynamic_Mode|Dynamic mode]], it is the file which is polled by the Fusion Edge Server when checking for updates.  The ledger file contains information such as:
 +
 +
* Unique Identity
 +
* The time the Environment was prepared
 +
* The embargo time (go live time)
 +
* The version of the Environment
 +
* History of previous Environments from which this Environment was built
 +
 +
== Environment Version ==
 +
The results of a compilation is an Environment, the very first Environment that is built has version 1.0.0.  If an Environment is built by referencing a previous Environment to use as a base (i.e. when merging in new observations for release), the new Environment will increase the version number, and keep a history of previous versions in the main ledger file.  This enables a user to see the history of an Environment (was it a new build, or was it a result of merging information from previous Environments).
 +
 +
When an Environment is merged, it is possible for some files to be exactly the same for the old and new Environment.  For example if there are 10 Datasets in an Environment and only 1 is changed, then the new Environment will contain 9 unmodified Datastores with only 1 modified. 
 +
 
 +
The version is given as a 3 part version syntax, starting at version 1.0.0.  The three parts are known as the major version, minor version, and patch version. 
 +
 +
* The patch version is updated if  the compiler was run in update mode and the only change was to one or more datasets
 +
* The minor version is updated if the structural metadata changed since the last compilation, this could for example be due to new time series requiring new classifications which were not in previous environments.  * * The major version is updated if there are new datasets for Dataflows that previously did not exist or had no data.  The major version is also updated if new reference metadata are released.
 +
 +
'''Example'''
 +
Initial Release Version 1.0.0
 +
Modify a dataset Version 1.0.1
 +
Modify another dataset Version 1.0.2
 +
Export a new Codelist Version 1.1.0
 +
Add a new dataflow with data Version 2.0.0

Latest revision as of 06:20, 14 August 2024

Overview

The Fusion Edge Server is responsible for exposing data and supporting metadata via a number of read only web service APIs. The data and metadata is the Environment, changes to the information in the data or metadata can only be made by supplying a new Environment. The Environment can be thought of as an immutable (can not be changed) copy of everything that is to be disseminated. As an Environment can not be modified once built, the only way to update information is to build a new Environment. New Environments can be built either from a complete set of source data and metadata files, or by merging new information (data/metadata) into an existing Environment, which results in a new Environment.

The Environment is made up of a collection of files, which are generated the output of the compile process of the Fusion Edge Compiler. These files contain prebuilt datastores for structures, data, metadata, prebuilt free text search indexes, and other pre-cached information which is used in dissemination.

The only way to modify an Environment is to re-run the compile process of the Fusion Edge Compiler. A built Environment can not be manually modified as it will result in corruption due to the mismatch in the digital signature applied to the Environment at compile time.

Environment Contents

The Environment is a collection of files and folders (in bold) which are 

cache
dab
data   holds a prebuilt data store for each Dataflow
datasearch   holds a prebuilt free text search dictionary
languages  details about multilingual metadata
ledger_indexes  details about previous Environments from which this was built
metadata   holds a prebuilt reference metadata database
structure  holds a prebuilt structural metadata database
xref       holds a prebuilt cache of cross reference information
fes_ledger.json  describes the release history of environments, holds embargo timestamps

Fusion Edge Server Ledger

The fes_ledger.json file is important when running in Dynamic mode, it is the file which is polled by the Fusion Edge Server when checking for updates. The ledger file contains information such as:

  • Unique Identity
  • The time the Environment was prepared
  • The embargo time (go live time)
  • The version of the Environment
  • History of previous Environments from which this Environment was built

Environment Version

The results of a compilation is an Environment, the very first Environment that is built has version 1.0.0. If an Environment is built by referencing a previous Environment to use as a base (i.e. when merging in new observations for release), the new Environment will increase the version number, and keep a history of previous versions in the main ledger file. This enables a user to see the history of an Environment (was it a new build, or was it a result of merging information from previous Environments).

When an Environment is merged, it is possible for some files to be exactly the same for the old and new Environment. For example if there are 10 Datasets in an Environment and only 1 is changed, then the new Environment will contain 9 unmodified Datastores with only 1 modified.

The version is given as a 3 part version syntax, starting at version 1.0.0. The three parts are known as the major version, minor version, and patch version.

  • The patch version is updated if the compiler was run in update mode and the only change was to one or more datasets
  • The minor version is updated if the structural metadata changed since the last compilation, this could for example be due to new time series requiring new classifications which were not in previous environments. * * The major version is updated if there are new datasets for Dataflows that previously did not exist or had no data. The major version is also updated if new reference metadata are released.

Example 

Initial Release			Version 1.0.0
Modify a dataset		Version 1.0.1
Modify another dataset 		Version 1.0.2
Export a new Codelist		Version 1.1.0
Add a new dataflow with data 	Version 2.0.0
Retrieved from "https://wiki.sdmxcloud.org/index.php?title=Edge_Server_Environment&oldid=6948"