Fusion Edge Compiler

Overview

The Fusion Edge Compiler is a command line client, written in Java and can run on Windows or UNIX operating systems. Its responsibility is to compile SDMX data, structure, and metadata files for dissemination by the Fusion Edge Server. The Fusion Edge Compiler provides three functions:

1. To pull content from SDMX web services (example Fusion Registry web services) in order to populate a local file system of content to publish
2. To compile content in the local file system to create a new Environment which can be consumed by the Fusion Edge Server
3. To publish the Environment to an Amazon S3 bucket from which distributed Fusion Edge Servers can take their content, if configured to do so

The second function, compile, is the main function of the compiler as it is the only function which must be done by the Fusion Edge Compiler. The other two functions could be performed manually or via a custom automated process.

Pull Content

buildFileSystem.sh (UNIX) or buildFileSystem.bat (Windows)

The Fusion Edge Compiler queries and SDMX web service for structural metadata, data, and reference metadata content based on what it has been requested to pull. It can work against a Fusion Registry web service as well as any other SDMX web service that complies with the SDMX specification.

The Fusion Edge Compiler pulls the content to build a target directory of files in the correct structure for the compile process to operate. The Fusion Edge Compiler command line arguments focus on which Datasets to pull from the target web service. When a dataset is pulled, the corresponding metadata (Dataflow, DSD, Concepts, Codelists) will also be pulled.

It is possible to pull only updates to datasets into an existing file system, by using the -lgr or -upd property.

Argument	Example	Description
prop	-prop "/home/props.json"	Optional. A reference to one or more properties files (separated by a space)	api	-api "https://stats.bis.org/api/v1"	Required. The URL of the web service to pull the content from
tgt	-tgt "/home/compiler/target"	Required. The target directory to write the files and folders to
lgr	Example	Optional. The location of an Environment folder. If this is provided then the last compile time of the Environment will be used as the updated after time to use when pulling data
df	-df "ECB:CPI(1.0) ECB:EXR(1.0)"	Optional. A reference to one or more Dataflows to pull data for (separated by a space). The keyword all can be used to pull data for all Dataflows.
upd	Example	This applies an updatedAfter query parameter against the target web service when querying for data (only retrieve data updated after a point in time). This is an alternative to using the lgr argument which dynamically determines the updatedAfter parameter based on the time that Environment was built.
replace	-replace	If present, all the files in the the target directory will be deleted before the pull content is run
usr	-usr "myusername"	Username to authenticate with the REST API, if using the Fusion Registry it should correspond to a user account in the Fusion Registry
pwd	-pwd "mypassword"	Password to authenticate with the REST API
s3rgn	-s3rgn "us-east-1"	Amazon S3 region – required if the Ledger is hosted on Amazon S3
s3sec	-s3sec "azxzcvbnm"	Amazon S3 Secret – required if the Ledger is hosted on Amazon S3
s3acc	s3acc "azxzcvbnm"	Amazon S3 Access Key – required if the Ledger is hosted on Amazon S3
h	-h	Display help information

Properties File

It is possible to provide all the arguments to the Fusion Edge Compiler via a properties file, referenced by the -prop argument. The properties file has additional support for pulling specific structures and describing sub-sets of a dataset, both of which can not be done via the simple command line arguments.

An example Properties file is given below:

{
  "Ledger" : 	"s3:mybucket",
  "TgtDir" :  	"/home/compiler/target",
  "SdmxAPI" : 	"https://demo.metadatatechnology.com/FusionRegistry/ws/public/sdmxapi/rest",
  "UpdatedAfter" :	"2010",
  "Username" : 	"myuser",
  "Password" : 	"pwd",
  "AllData" : 	true,
  "FullReplace" : 	true,
  "Zip" : 		true,
  "Metadata" : 	true,
  "S3Region":	"us-east-1",
  "S3SecretKey":	"azxasdasfcvbn",
  "S3AccessKey":	"sxcvbnmu",
  "SubCubes":{
     "ECB:EXR(1.0)" : {
        "SubCube1" : {
           "Include" : {
              "FREQ":["A","M"],
              "REF_AREA":["UK"]
           }
        }
     },
     "WB:POVERTY(1.0)":{ }
  },
  "Structures":{
     "Codelist": ["ECB,EXR,1.0"]
     "HierarchicalCodelist": ["ECB", "BIS"]
     "all": ["SDMX"]
     ]
  }
}

Structures
The Structures section of the properties file defines which structural metadata should be included in the outputs.

Note, when outputting data for a Dataflow, the Dataflow and all descendants (DSD, Codelist, Concept Scheme, Agency Scheme) will be automatically included in the structure metadata that is generated and do not need to be explicitly specified. This is also true for the Structure section, specifying a specific structure such as a HierarchicalCodelist, will automatically include its descendant structures.

The arguments are:

1. The structure type, this is the same as the path parameter on the REST API, i.e. Codelist.
2. An array of structure filters in the format AgencyId,Id,Version. Each filter is optional, the absence of which meaning all. The keyword all can be used as a structure type to indicate all structures, which can also take the filters for agency, id and version. Example ["ECB,EXR,1.0", "SDMX", "BIS,all,2.1"]