Fusion Registry Wiki - User contributions [en]

Fusion Edge Compiler

2024-02-23T11:34:44Z

Mnelson: /* Refresh Content */

[[Category:Fusion Edge Server]]
== Overview ==
The Fusion Edge Compiler is a command line client, written in Java and can run on Windows or UNIX operating systems. Its responsibility is to compile SDMX data, structure, and metadata files for dissemination by the Fusion Edge Server. The Fusion Edge Compiler provides the following functions:

# To '''[[#Pull_Content|pull content]]''' from SDMX web services (example Fusion Registry web services) in order to populate a [[Edge_Server_-_Publish_Content#Part_1_-_Source_Data_and_Metadata_Files|local file system]] of content to publish
# To '''[[#Compile_Content|compile content]]''' in the local file system to create a new [[Edge_Server_Environment|Environment]] which can be consumed by the Fusion Edge Server
# To '''[[#Publish_Content|publish]]''' the [[Edge_Server_Environment|Environment]] to an Amazon S3 bucket from which distributed Fusion Edge Servers can take their content, if configured to do so
# To '''[[#Print_Content|print content]]''' writes the contents of an Environment to a CSV file as a report of what the compiled Environment contains

The compile function is the main function of the compiler as it is the only function which must be performed by the Fusion Edge Compiler in order to have a deployable Environment for the Fusion Edge Server. Organising SDMX files in a file system, and moving an Environment to the Edge Server can both be performed manually or via a custom automated process.

== Common Arguments ==
The following table contains arguments which are used for more then one Compile function.

=== Help ===
Every compile function will output a list of the available function arguments by passing -h to the function.

'''Example'''

buildFileSystem.sh -h

=== Properties File ===
Every compile function can read a JSON file to get the values, this can be used instead of, or in addition to adding command line arguments.

Each command line argument has a corresponding JSON property.

{| class="wikitable sortable"
|-
! Argument !! Example !! Description
|-
| prop || -prop "/home/props.json" || Optional. A reference to one or more properties files (separated by a space)
|}

'''Example'''

buildFileSystem.sh -prop "MyConfig.json"

'''Example MyConfig.json'''

{
"SdmxAPI" : "https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2",
"TgtDir" : "DemoServerFiles",
"AllData" : true
}

=== Web Service Arguments ===
The web service arguments are used by '''buildFileSystem''' and the '''refreshContent''' functions.

{| class="wikitable sortable"
|-
! Argument !! Example !! Description
|-
| api || -api "https://stats.bis.org/api/v1" || Required. The URL of the web service to pull the content from
|-
| apiv since 4.4.0 || -apiv "1.5.0" || Optional. The version of the SDMX API to query, it defaults to version 2.0.0 (entry point in Fusion Registry is ''/sdmx/v2'') the other option is 1.5.0 (Fusion Registry entry point is ''/ws/public/sdmxapi/rest'')
|-
| apict || -apict 60 || API Connect Timeout (seconds)
|-
| apirt || -apirt 600 || API Read Timeout (seconds)
|-
| apiua || -apiua EdgeServer || API User-Agent
|-
| apiformat || -apiformat sdmx-json || Format to get structures back from API, default is fusion-json (compatible with Fusion APIs and included non-sdmx structure types) - alternatives are sdmx-json, sdmx-ml-21, sdmx-ml-3
|-
| usr || -usr "myusername" || Username to authenticate with the REST API, if using the Fusion Registry it should correspond to a user account in the Fusion Registry
|-
| pwd || -pwd "mypassword" || Password to authenticate with the REST API
|}

=== Current Environment ===
The '''compileFileSystem''' and '''refreshContent''' functions can reference an existing Environment in order to merge new content into that Environment.

{| class="wikitable sortable"
|-
! Argument !! Example !! Description
|-
| prop || -prop "/home/props.json" || Optional. A reference to one or more properties files (separated by a space)
|-
| tgt || -tgt "/home/compiler/compiled" || Required. The target directory to write the new Environment to
|-
| lgr || -lgr "/home/compiler/live_environment" || The location of the current Environment
|-
| sgn || -sgn "my_signature" || Required. A secret signature to sign the new Environment files with. This signature must match the one used to sign the current Environment.
|-
| s3rgn || -s3rgn "us-east-1" || Amazon S3 region – required if the Ledger is hosted on Amazon S3
|-
| s3sec || -s3sec "azxzcvbnm" || Amazon S3 Secret – required if the Ledger is hosted on Amazon S3
|-
| s3acc || s3acc "azxzcvbnm" || Amazon S3 Access Key – required if the Ledger is hosted on Amazon S3
|}

== Pull Content ==
buildFileSystem.sh (UNIX) or buildFileSystem.bat (Windows)

=== Overview ===
The Fusion Edge Compiler queries and SDMX web service for structural metadata, data, and reference metadata content based on what it has been requested to pull. It can work against a Fusion Registry web service as well as any other SDMX web service that complies with the SDMX specification.

The Fusion Edge Compiler pulls the content to build a target directory of files in the correct structure for the compile process to operate. The Fusion Edge Compiler command line arguments focus on which Datasets to pull from the target web service. When a dataset is pulled, the corresponding metadata (Dataflow, DSD, Concepts, Codelists) will also be pulled.

It is possible to pull only updates to datasets into an existing file system, by using the -lgr or -upd property.

===Command Line Arguments===

The minimum arguments are the api to pull the data from, and the target folder to write the files to, in addition instruction on which datasets or structures to pull.

'''Example (minimum arguments)'''

buildFileSystem.sh -api "https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2" -df "all" -tgt "SourceFiles"

'''Arguments'''
* [[#Help|Help]]
* [[#Properties_File|Properties File]]
* [[#Web_Service_Arguments|Web service arguments]] - required as the '''buildFileSystem''' function pulls information from an SDMX Compliant web service
* [[#Current_Environment|Environment Arguments]] - optional, if provided then the last compile time of the Environment will be used as the updated after time to use when pulling data

'''Additional Arguments'''

{| class="wikitable sortable"
|-
! Argument !! Example !! Description
|-
| audit since 4.2.0 || -audit || If present, will output an Audit folder with a json file containing information about the REST API request and corresponding response.
|-
| exempty since 4.2.1 || -exempty || Optional. If present Dataflows from the '''-df''' parameter will be excluded from the output if they have no data pulled from the server
|-
| datastruct since 4.2.1 || -datastruct || Optional. If present each Dataflow in the '''-df''' argument will be processed to include the following related structures in the output: Pre-defined Queries, [[Publication_Table|Publication Tables]], and Category Schemes +Categorisations
|-
| df || -df "ECB:CPI(1.0)" "ECB:EXR(1.0)" || Optional. A reference to one or more Dataflows to pull data for (separated by a space). The keyword '''all''' can be used to pull data for all Dataflows. A '*' can be used to mean all versions, e.g. ECB:EXR(*)
|-
| lenient || -lenient || If present the pull process will skip failed data queries (they will not prevent the remaining pull to complete). The failure will be noted in the report for the Dataset.
|-
| metadata since 4.5.0 || -metadata incref || Issues a request for all metadata from the web service, will also include the Metadataflows and Metadata Structures which the Metadata Sets conform to. The argument '''incref''' will ensure that all target structures (the structures the metadata is authored against) and their descendants are also included in the output, '''excref''' will not include the target structures. '''Note:''' If the target structures are not present in the file system by some other means then the reference metadata will be excluded during the Compile process.
|-
| replace || -replace || If present, all the files in the the target directory will be deleted before the pull content is run
|-
| report || -report || If present, the report will be written to a file, if not present the report will be written to the System.out
|-
| str || -str "codelist=SDMX" || Structures to query (in addition to dataset related metadata). Syntax [structure type]=[agencyid],[id],[version], the only required part is the structure type with other parameters defaulting to 'all'
|-
| tgt || -tgt "/home/compiler/target" || Required. The target directory to write the files and folders to
|-
| upd || Example || This applies an updatedAfter query parameter against the target web service when querying for data (only retrieve data updated after a point in time). This is an alternative to using the '''lgr''' argument which dynamically determines the updatedAfter parameter based on the time that Environment was built.
|}

'''Pulling Data'''
When building an Edge Server for data dissemination, the compiler must be told which Dataflows to pull data for, this is achieved using the '''-df''' argument, for example:

buildFileSystem.sh -api "https://myregistry/ws/public/sdmxapi/rest" '''-df "all"''' -tgt "/home/user/EdgeServer/RawFiles"

In this example all Dataflows will be pulled from the web service, and the data for each Dataflow will be pulled.

It may be the case that the api contains Dataflows which have no data. These Dataflows will be pulled into the Edge Sever file system, meaning they will be available via the Edge Server structure web service, even though the Edge Server has no data available (in the same way the Dataflow was pulled from the source api with no corresponding dataset). This behaviour can be changed so that the Dataflow is excluded from the structure file if it has no data. This is achieved using the '''-exempty''' argument

buildFileSystem.sh -api "https://myregistry/ws/public/sdmxapi/rest" -df "all" -tgt "/home/user/EdgeServer/RawFiles" '''-exempty'''

The file system will now only contain the Dataflows where there is a corresponding dataset for the Dataflow.

Whilst the '''-str''' argument can be used to list additional structures to pull from the api, in the case of data dissemination, it is likely that a core set of additional 'data related' structures are also required. These structures are the Predefined Query, Publication Table, Categorisations and related target Structure. By adding the '''-datastruct''' argument the compiler will include, for each Dataflow, these related structures.

buildFileSystem.sh -api "https://myregistry/ws/public/sdmxapi/rest" -df "all" -tgt "/home/user/EdgeServer/RawFiles" -exempty '''-datastruct'''

The above example will pull all Dataflows that have a corresponding dataset, and for each Dataflow it will obtain related the related structures for the Dataflow.

===Properties File===
It is possible to provide all the arguments to the Fusion Edge Compiler via a properties file, referenced by the -prop argument.


An example Properties file is given below:

{
"Ledger" : "s3:mybucket",
"TgtDir" : "/home/compiler/target",
"SdmxAPI" : "https://demo.metadatatechnology.com/FusionRegistry/ws/public/sdmxapi/rest",
"UpdatedAfter" : "2010",
"Username" : "myuser",
"Password" : "pwd",
"AllData" : true,
"FullReplace" : true,
"Zip" : true,
"Metadata" : true,
"S3Region": "us-east-1",
"S3SecretKey": "azxasdasfcvbn",
"S3AccessKey": "sxcvbnmu",
"SubCubes":{
"ECB:EXR(1.0)" : {
"SubCube1" : {
"Include" : {
"FREQ":["A","M"],
"REF_AREA":["UK"]
}
}
},
"WB:POVERTY(1.0)":{ }
},
"Structures":{
"Codelist": ["ECB,EXR,1.0"]
"HierarchicalCodelist": ["ECB", "BIS"]
"all": ["SDMX"]
]
}
}

'''Structures'''
The Structures section of the properties file defines which structural metadata should be included in the outputs.

Note, when outputting data for a Dataflow, the Dataflow and all descendants (DSD, Codelist, Concept Scheme, Agency Scheme) will be automatically included in the structure metadata that is generated and do not need to be explicitly specified. This is also true for the Structure section, specifying a specific structure such as a HierarchicalCodelist, will automatically include its descendant structures.

The arguments are:
# The structure type, this is the same as the path parameter on the REST API, i.e. Codelist.
# An array of structure filters in the format AgencyId,Id,Version. Each filter is optional, the absence of which meaning all. The keyword all can be used as a structure type to indicate all structures, which can also take the filters for agency, id and version. Example ["ECB,EXR,1.0", "SDMX", "BIS,all,2.1"]

== Compile Content ==
compileFileSystem.sh (UNIX) or compileFileSystem.bat (Windows)

=== Overview ===
The compile script reads the files in from the source directory and compiles them into the target directory. The result of the compile process is an [[Edge_Server_Environment|Environment]] that can be published to the Fusion Edge Server.

If the target directory does not exist it will be created, if it does exist the contents will be overwritten.

=== Signing Content ===
All compiled Environments are digitally signed using the a secret key provided using the '''-sgn''' command line argument. This signature ensures that the Environment files are not tampered with after they have been generated. It also ensures that the Fusion Edge Server knows that the Environment was generated by a trusted source, as the Fusion Edge Server must also know that the secret key is in order to verify the source.

=== Embargo ===
The generated Environment contains the timestamp of Compilation, it also contains a timestamp for release, which unless other specified is the same as the timestamp of compilation. The timestamp for release can be modified, using the '''-liv''' argument, to be a point in time in the future. When this argument is provided the Environment will not be made live by the Fusion Edge Server until this time has passed. It is possible configure the Fusion Edge Server in the [[Fusion_Edge_Server_Properties#Embargo_Data| properties file]] to pre-load the Environment into memory so it is ready for release.

=== Static Mode Compile ===
When running in [[Edge_Server_-_Publish_Content#Static_Mode|Static mode]] the Fusion Edge Server does not make use of the [[Edge_Server_Environment#Fusion_Edge_Server_Ledger|ledger file]], it always reads in the Environment zip file regardless of what version is in the ledger.json file. In this way a static mode Compile does not need to use the -lgr argument, as long as the source file system contains the full Environment source files for dissemination.

The compile process is:
# The Fusion Edge Compiler reads all files from the -src directory
# The Fusion Edge Compiler compiles all files
# The Fusion Edge Compiler writes the compiled files to the -tgt directory

It is possible to make use of the lgr argument if the intention is to merge new information into a previous compile. Details of merging Environments are provided in the next section.

=== Dynamic Mode Compile ===
When running in [[Edge_Server_-_Publish_Content#Dynamic_Mode|Dynamic mode]] it is important to include the location of the last (current) Environment using the '''-lgr''' argument. The [[Edge_Server_Environment#Fusion_Edge_Server_Ledger|ledger]] of the new Environment will be appended to in the new Environment, creating a new version to describe the new Environment. This is critical as the Fusion Edge Server will only update its content if it detects a change in the version of the ledger file.

By default, when using the -lgr argument, the compile process will append and/or modify information in an existing Environment. Information will not be deleted (although there are ways to force a delete, covered later).

When providing a -lgr argument, the compile process is:
# The Fusion Edge Compiler reads the referenced Environment of the -lgr argument to determine the timestamp it was built
# The Fusion Edge Compiler reads files from the source file system whose timestamps are later then the build time of the Environment (to get the deltas)
# The Fusion Edge Compiler copies all previously built data and structure stores into the new Environment - only making modifications if it found data/metadata files whose timestamps were later then the referenced Environment compiled timestamp.

In essence, the default behaviour is to copy the previous Environment, and to merge in deltas. However, this can result in undesirable results when the intention is to delete information such as datasets, series, observations, codelists, etc. If structural changes are made to a Data Structure which makes previously compiled datasets invalid, the merge process will result in an unstable system. Therefore the following arguments are provided to enable a finer level of control over what is read from the file system and what is copied from the old Environment.

==== Ignore Timestamp on File ====
The '''-f''' argument can be provided which forces all source files in the file system to be read regardless of the timestamp.

==== Replace Dataset ====
the '''-rd''' argument (replace dataset) informs the compile process to replace the dataset in the current Environment with those found in the file system. This argument has 2 possible actions '''remove''' or '''keep'''.

The '''keep''' action informs the compiler to keep any datasets that are in the Environment and not in the File System. For example if the Environment contains a dataset for Exchange Rates, Interest Rates and Employment and local file system has a dataset for Exchange Rates, using the option '''-rd keep''' will output a new Environment with the Exchange Rates dataset from the file system, and the Interest Rates and Employment datasets from the previous Environment.

The '''remove''' action informs the compiler to remove any datasets that are in the Environment and not in the File System. For example if the Environment contains a dataset for Exchange Rates, Interest Rates and Employment and local file system has a dataset for Exchange Rates, using the option '''-rd remove''' will output a new Environment with only the Exchange Rates dataset from the file system - the other two datasets will no longer exist in the new Environment.

'''Note''' prior to version 4.4.0 the -rd parameter took no arguments, the behaviour was always '''remove'''.

'''Note''' the -f argument should still be used if the intention is to read all data files in the file system, otherwise the default behaviour will be to read only the data files that have changed since the last compile.

==== Full Replace ====
Compiles all files in the source file system, ignoring any timestamp on the referenced Environment. No information is copied from the referenced Environment, with exception of the fes_ledger.json file, which is used as a base for creating the new fes_ledger.json with an updated version and history of previous versions.

===Command Line Arguments===
The minimum arguments are the source folder which contains the files to compile, and the target folder to write the compiled Environment to, along with the signature on the Environment

'''Example (minimum arguments)'''

compileFileSystem.sh -src "SourceFiles" -tgt "CompiledFiles" -sgn "password"

'''Arguments'''
* [[#Help|Help]]
* [[#Properties_File|Properties File]]
* [[#Current_Environment|Environment Arguments]] - optional, if provided then the Environment will be used as a source Environment which the newly compiled files will be merged into

'''Additional Arguments'''

{| class="wikitable sortable"
|-
! Argument !! Example !! Description
|-
| f || -f || Optional. Force all files in the source file system to be read regardless of the previous Environment compiled timestamp
|-
| fr || -fr || Optional. Full Replace. Read all files in the source file system. Do not merge any metadata or data stores from previous Environment.
|-
| liv || -liv "2020-01-30T00:00.00" || Optional. The go live time (embargo time). The generated Environment will include this timestamp so that it is not released by the Fusion Edge Server until the time. The format is yyyy-mm-ddThh:mm:ss and the timezone is GMT
|-
| ra since 4.4.0 || -ra "UNIT" -ra "OBS_CONF" ||Optional. Remove Attribute. This argument removes attributes from all Data Structures that contain it, and all corresponding datasets. This will NOT remove attributes from existing datasets in previously built environments - it only works on newly compiled datasets.
|-
| rd || -rd keep || Optional. Do not merge any data stores from previous Environment. Replace all Datasets with those read in from the file system.
 Note: From v4.4.0 this argument takes an addition value of 'keep' or 'remove', keep indicates that any datasets in the current Environment but not in the file system should be kept in the new Environment, remove indicates they should be removed with only the datasets in the file system being included in the new Environment.
|-
| src || -src "/home/compiler/source" || Required. The source directory that contains the files to be compiled (this is the tgt directory in the build file system script)
|-
| tgt || -tgt "/home/compiler/compiled" || Required. The target directory to write the files and folders to
|}

=== Properties File ===
It is possible to provide all the arguments to the Fusion Edge Compiler via a properties file, referenced by the '''-prop''' argument.

An example Properties file is given below:

{
"Ledger" : "s3:mybucket",
"SrcDir" : "/home/compiler/source",
"TgtDir" : "/home/compiler/compiled",
"ForceRebuild" : false,
"Signature" : "myuser",
"LiveTime" : "2020-01-30T00:00.00",
"S3Region": "us-east-1",
"S3SecretKey": "azxasdasfcvbn",
"S3AccessKey": "sxcvbnmu",
"ForceRebuild": false,
"ReplaceData": "keep",
"FullReplace": false,
}

== Refresh Content ==
refreshContent.sh (UNIX) or refreshContent.bat (Windows)

'''Note:''' this feature was introduced in version 4.4.0

=== Overview ===
The Refresh command updates the Environment directly, by pulling SDMX content from a web service, comparing it with what is in the Environment, and refreshing the information. Critically the refresh command will only update existing information in the Environment, for example a refresh on structures will compare the structures in the Environment against those from a web service to ensure the Environment's structures are up to date - any structure differences from the web service will be written into the Environment - structures found from the web service will NOT be written into the Environment if those structures do not already exist in the Environment.

===Command Line Arguments===
The minimum arguments are the Environment to refresh (along with the signature), what part(s) if of the Environment to refresh, where to built the new Environment, and the web service details which will be used as the source of information which will be written into the new Environment

'''Note''': If Dataflows are removed from the source API service, these will be removed from the Compiled Environment if the '-remove' argument is used, and associated data will also be removed from the Compiled Environment.

'''Note''': If using Fusion Registry as the source service, it is possible to 'remove' a structure by making it private to the compiler using the [[Main_Page#Fusion_Security_Manager_Topics|Fusion Security Manager]].

'''Example (minimum arguments)'''

refreshContent.sh -api "https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2" -src "CompiledFiles" -tgt "NewCompiled" -sgn "password" -structures

'''Arguments'''
* [[#Help|Help]]
* [[#Properties_File|Properties File]]
* [[#Web_Service_Arguments|Web service arguments]] - required as the '''refreshContent''' function pulls information from an SDMX Compliant web service into the new Environment
* [[#Current_Environment|Environment Arguments]] - required as the '''refreshContent''' must have an existing Environment to refresh

'''Additional Arguments'''

{| class="wikitable sortable"
|-
! Argument !! Example !! Description
|-
| structures || -structures || Optional. If provided the structures in the Environment will be refreshed
|-
| remove || -remove || Optional. The default behaviour is to keep structures that are in the Environment, even if those structures could not be found in the web service. This argument overrides this behaviour by requesting that structures are removed from the Environment if those structures are not found in the web service. Corresponding data and metadata will also be removed if the underlying structures upon which it relies are removed.
|}

== Refresh File System==
refreshFileSystem.sh (UNIX) or refreshFileSystem.bat (Windows)

'''Note:''' this feature was introduced in version 4.9.0 (Jan 2024)

=== Overview ===
The Refresh file system command updates all structure files in the structures folder of the Edge Server file system with those from the REST API.

The process is as follows:

# The compiler reads each file in turn for Sdmx Structural Metadata
# The compiler renames the file to postfix it with .bak (backup)
# The compiler creates a new file with the same name as the original file
# For each [[Maintainable]] structure that was read from the original file, the compiler will request the corresponding Maintainable from the REST API
# The Maintainable structure from the server is written to the new file, if the file postfix was .xml the format is SDMX-ML v3.0.0, otherwise the structure(s) will be written in fusion-json format
# If there is no structure returned from the server, the original Maintainable structure is written to the new file
# If there is an error reading the original file or an error reading the server response, the process will skip the file entirely and record the error in the audited output
# The audited output contains an array of Maintainable structures that were read from the server which differed from the original structure file

===Command Line Arguments===
The minimum arguments is the directory to read the structure file(s) from (note it will check in the 'structures' subdirectory of the location it is given), and the URL of the REST API to refresh against.

'''Example (minimum arguments)'''

refreshFileSystem.sh -api "https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2" -tgt "File"

'''Arguments'''
* [[#Help|Help]]
* [[#Web_Service_Arguments|Web service arguments]] - required as the '''refreshContent''' function pulls information from an SDMX Compliant web service into the new Environment

'''Additional Arguments'''

{| class="wikitable sortable"
|-
! Argument !! Example !! Description
|-
| tgt || -tgt "/home/files/target" || Required. The target directory to read the files and folders from and updated the files in, note this will check structures in a 'structures' subfolder
|}

== Publish Content ==
publishContent.sh (UNIX) or publishContent.bat (Windows)

===Overview===
The Publish Content script is used to move an [[Edge_Server_Environment|Environment]] from the local file system to Amazon S3. If the [[Edge_Server_Environment|Environment]] is hosted elsewhere, for example a private web server, or a file system local to the Fusion Edge Server, then a custom process must be put in place to move the [[Edge_Server_Environment|Environment]] .

The Publish Content will ensure the Amazon S3 bucket is updated to reflect the new Environment in the most efficient way, for example if it detects a file is unchanged in Environment it will not be moved. The ledger file is always moved last which ensures the Fusion Edge Server does not attempt to process the Environment until all the files are moved.

===Command Line Arguments===
The minimum arguments are the source folder of the current Environment, and the location of the Amazon S3 bucket to publish the Environment to, along with the required credentials

'''Example (minimum arguments)'''

publishContent.sh -src "CompiledFiles" -lgr “s3:mybucket” -s3rgn "us-east-1" -s3sec "azxzcvbnm" -s3acc "azxzcvbnm"

'''Arguments'''
* [[#Help|Help]]
* [[#Properties_File|Properties File]]
* [[#Current_Environment|Environment Arguments]] - required, the ledger must resolve to the S3 bucket and the s3rgn, s3sec, and s3acc must be provided

{| class="wikitable sortable"
|-
! Argument !! Example !! Description
|-
| src || -src "/home/compiler/environment" || Required. The directory that contains the Environment to be published (this is the tgt folder in the compile process)
|}

=== Properties File ===
It is possible to provide all the arguments to the Fusion Edge Compiler via a properties file, referenced by the '''-prop''' argument.

An example Properties file is given below:

{
"Ledger" : "s3:mybucket",
"SrcDir" : "/home/compiler/compiled",
"S3Region": "us-east-1",
"S3SecretKey": "azxasdasfcvbn",
"S3AccessKey": "sxcvbnmu",
}

== Print Content ==
printContent.sh (UNIX) or printContent.bat (Windows)

===Overview===
The Print Content script outputs a CSV report on what is contained in an Environment.

===Command Line Arguments===
The minimum arguments are the Environment to print along with the signature on that Environment

'''Example (minimum arguments)'''

printContent.sh -lgr "CompiledFiles" -sgn "password"

'''Arguments'''
* [[#Help|Help]]
* [[#Properties_File|Properties File]]
* [[#Current_Environment|Environment Arguments]] - required, the ledger must resolve to the Environment folder

{| class="wikitable sortable"
|-
! Argument !! Example !! Description
|-
| out || -out "/home/compiler/report.csv" || Optional. If provided this is the file that the report will be written to, otherwise the report will be written to the console
|-
| loc || -loc "fr" || Optional. If provided this is the default locale to use when printing the name of the structures - unresolvable names will be written in an alternative locale, defaulting to "en". If not provided "en" will be the default locale.
|}

Fusion Edge Compiler

2024-02-23T11:34:15Z

Mnelson: /* Command Line Arguments */

[[Category:Fusion Edge Server]]
== Overview ==
The Fusion Edge Compiler is a command line client, written in Java and can run on Windows or UNIX operating systems. Its responsibility is to compile SDMX data, structure, and metadata files for dissemination by the Fusion Edge Server. The Fusion Edge Compiler provides the following functions:

# To '''[[#Pull_Content|pull content]]''' from SDMX web services (example Fusion Registry web services) in order to populate a [[Edge_Server_-_Publish_Content#Part_1_-_Source_Data_and_Metadata_Files|local file system]] of content to publish
# To '''[[#Compile_Content|compile content]]''' in the local file system to create a new [[Edge_Server_Environment|Environment]] which can be consumed by the Fusion Edge Server
# To '''[[#Publish_Content|publish]]''' the [[Edge_Server_Environment|Environment]] to an Amazon S3 bucket from which distributed Fusion Edge Servers can take their content, if configured to do so
# To '''[[#Print_Content|print content]]''' writes the contents of an Environment to a CSV file as a report of what the compiled Environment contains

The compile function is the main function of the compiler as it is the only function which must be performed by the Fusion Edge Compiler in order to have a deployable Environment for the Fusion Edge Server. Organising SDMX files in a file system, and moving an Environment to the Edge Server can both be performed manually or via a custom automated process.

== Common Arguments ==
The following table contains arguments which are used for more then one Compile function.

=== Help ===
Every compile function will output a list of the available function arguments by passing -h to the function.

'''Example'''

buildFileSystem.sh -h

=== Properties File ===
Every compile function can read a JSON file to get the values, this can be used instead of, or in addition to adding command line arguments.

Each command line argument has a corresponding JSON property.

{| class="wikitable sortable"
|-
! Argument !! Example !! Description
|-
| prop || -prop "/home/props.json" || Optional. A reference to one or more properties files (separated by a space)
|}

'''Example'''

buildFileSystem.sh -prop "MyConfig.json"

'''Example MyConfig.json'''

{
"SdmxAPI" : "https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2",
"TgtDir" : "DemoServerFiles",
"AllData" : true
}

=== Web Service Arguments ===
The web service arguments are used by '''buildFileSystem''' and the '''refreshContent''' functions.

{| class="wikitable sortable"
|-
! Argument !! Example !! Description
|-
| api || -api "https://stats.bis.org/api/v1" || Required. The URL of the web service to pull the content from
|-
| apiv since 4.4.0 || -apiv "1.5.0" || Optional. The version of the SDMX API to query, it defaults to version 2.0.0 (entry point in Fusion Registry is ''/sdmx/v2'') the other option is 1.5.0 (Fusion Registry entry point is ''/ws/public/sdmxapi/rest'')
|-
| apict || -apict 60 || API Connect Timeout (seconds)
|-
| apirt || -apirt 600 || API Read Timeout (seconds)
|-
| apiua || -apiua EdgeServer || API User-Agent
|-
| apiformat || -apiformat sdmx-json || Format to get structures back from API, default is fusion-json (compatible with Fusion APIs and included non-sdmx structure types) - alternatives are sdmx-json, sdmx-ml-21, sdmx-ml-3
|-
| usr || -usr "myusername" || Username to authenticate with the REST API, if using the Fusion Registry it should correspond to a user account in the Fusion Registry
|-
| pwd || -pwd "mypassword" || Password to authenticate with the REST API
|}

=== Current Environment ===
The '''compileFileSystem''' and '''refreshContent''' functions can reference an existing Environment in order to merge new content into that Environment.

{| class="wikitable sortable"
|-
! Argument !! Example !! Description
|-
| prop || -prop "/home/props.json" || Optional. A reference to one or more properties files (separated by a space)
|-
| tgt || -tgt "/home/compiler/compiled" || Required. The target directory to write the new Environment to
|-
| lgr || -lgr "/home/compiler/live_environment" || The location of the current Environment
|-
| sgn || -sgn "my_signature" || Required. A secret signature to sign the new Environment files with. This signature must match the one used to sign the current Environment.
|-
| s3rgn || -s3rgn "us-east-1" || Amazon S3 region – required if the Ledger is hosted on Amazon S3
|-
| s3sec || -s3sec "azxzcvbnm" || Amazon S3 Secret – required if the Ledger is hosted on Amazon S3
|-
| s3acc || s3acc "azxzcvbnm" || Amazon S3 Access Key – required if the Ledger is hosted on Amazon S3
|}

== Pull Content ==
buildFileSystem.sh (UNIX) or buildFileSystem.bat (Windows)

=== Overview ===
The Fusion Edge Compiler queries and SDMX web service for structural metadata, data, and reference metadata content based on what it has been requested to pull. It can work against a Fusion Registry web service as well as any other SDMX web service that complies with the SDMX specification.

The Fusion Edge Compiler pulls the content to build a target directory of files in the correct structure for the compile process to operate. The Fusion Edge Compiler command line arguments focus on which Datasets to pull from the target web service. When a dataset is pulled, the corresponding metadata (Dataflow, DSD, Concepts, Codelists) will also be pulled.

It is possible to pull only updates to datasets into an existing file system, by using the -lgr or -upd property.

===Command Line Arguments===

The minimum arguments are the api to pull the data from, and the target folder to write the files to, in addition instruction on which datasets or structures to pull.

'''Example (minimum arguments)'''

buildFileSystem.sh -api "https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2" -df "all" -tgt "SourceFiles"

'''Arguments'''
* [[#Help|Help]]
* [[#Properties_File|Properties File]]
* [[#Web_Service_Arguments|Web service arguments]] - required as the '''buildFileSystem''' function pulls information from an SDMX Compliant web service
* [[#Current_Environment|Environment Arguments]] - optional, if provided then the last compile time of the Environment will be used as the updated after time to use when pulling data

'''Additional Arguments'''

{| class="wikitable sortable"
|-
! Argument !! Example !! Description
|-
| audit since 4.2.0 || -audit || If present, will output an Audit folder with a json file containing information about the REST API request and corresponding response.
|-
| exempty since 4.2.1 || -exempty || Optional. If present Dataflows from the '''-df''' parameter will be excluded from the output if they have no data pulled from the server
|-
| datastruct since 4.2.1 || -datastruct || Optional. If present each Dataflow in the '''-df''' argument will be processed to include the following related structures in the output: Pre-defined Queries, [[Publication_Table|Publication Tables]], and Category Schemes +Categorisations
|-
| df || -df "ECB:CPI(1.0)" "ECB:EXR(1.0)" || Optional. A reference to one or more Dataflows to pull data for (separated by a space). The keyword '''all''' can be used to pull data for all Dataflows. A '*' can be used to mean all versions, e.g. ECB:EXR(*)
|-
| lenient || -lenient || If present the pull process will skip failed data queries (they will not prevent the remaining pull to complete). The failure will be noted in the report for the Dataset.
|-
| metadata since 4.5.0 || -metadata incref || Issues a request for all metadata from the web service, will also include the Metadataflows and Metadata Structures which the Metadata Sets conform to. The argument '''incref''' will ensure that all target structures (the structures the metadata is authored against) and their descendants are also included in the output, '''excref''' will not include the target structures. '''Note:''' If the target structures are not present in the file system by some other means then the reference metadata will be excluded during the Compile process.
|-
| replace || -replace || If present, all the files in the the target directory will be deleted before the pull content is run
|-
| report || -report || If present, the report will be written to a file, if not present the report will be written to the System.out
|-
| str || -str "codelist=SDMX" || Structures to query (in addition to dataset related metadata). Syntax [structure type]=[agencyid],[id],[version], the only required part is the structure type with other parameters defaulting to 'all'
|-
| tgt || -tgt "/home/compiler/target" || Required. The target directory to write the files and folders to
|-
| upd || Example || This applies an updatedAfter query parameter against the target web service when querying for data (only retrieve data updated after a point in time). This is an alternative to using the '''lgr''' argument which dynamically determines the updatedAfter parameter based on the time that Environment was built.
|}

'''Pulling Data'''
When building an Edge Server for data dissemination, the compiler must be told which Dataflows to pull data for, this is achieved using the '''-df''' argument, for example:

buildFileSystem.sh -api "https://myregistry/ws/public/sdmxapi/rest" '''-df "all"''' -tgt "/home/user/EdgeServer/RawFiles"

In this example all Dataflows will be pulled from the web service, and the data for each Dataflow will be pulled.

It may be the case that the api contains Dataflows which have no data. These Dataflows will be pulled into the Edge Sever file system, meaning they will be available via the Edge Server structure web service, even though the Edge Server has no data available (in the same way the Dataflow was pulled from the source api with no corresponding dataset). This behaviour can be changed so that the Dataflow is excluded from the structure file if it has no data. This is achieved using the '''-exempty''' argument

buildFileSystem.sh -api "https://myregistry/ws/public/sdmxapi/rest" -df "all" -tgt "/home/user/EdgeServer/RawFiles" '''-exempty'''

The file system will now only contain the Dataflows where there is a corresponding dataset for the Dataflow.

Whilst the '''-str''' argument can be used to list additional structures to pull from the api, in the case of data dissemination, it is likely that a core set of additional 'data related' structures are also required. These structures are the Predefined Query, Publication Table, Categorisations and related target Structure. By adding the '''-datastruct''' argument the compiler will include, for each Dataflow, these related structures.

buildFileSystem.sh -api "https://myregistry/ws/public/sdmxapi/rest" -df "all" -tgt "/home/user/EdgeServer/RawFiles" -exempty '''-datastruct'''

The above example will pull all Dataflows that have a corresponding dataset, and for each Dataflow it will obtain related the related structures for the Dataflow.

===Properties File===
It is possible to provide all the arguments to the Fusion Edge Compiler via a properties file, referenced by the -prop argument.


An example Properties file is given below:

{
"Ledger" : "s3:mybucket",
"TgtDir" : "/home/compiler/target",
"SdmxAPI" : "https://demo.metadatatechnology.com/FusionRegistry/ws/public/sdmxapi/rest",
"UpdatedAfter" : "2010",
"Username" : "myuser",
"Password" : "pwd",
"AllData" : true,
"FullReplace" : true,
"Zip" : true,
"Metadata" : true,
"S3Region": "us-east-1",
"S3SecretKey": "azxasdasfcvbn",
"S3AccessKey": "sxcvbnmu",
"SubCubes":{
"ECB:EXR(1.0)" : {
"SubCube1" : {
"Include" : {
"FREQ":["A","M"],
"REF_AREA":["UK"]
}
}
},
"WB:POVERTY(1.0)":{ }
},
"Structures":{
"Codelist": ["ECB,EXR,1.0"]
"HierarchicalCodelist": ["ECB", "BIS"]
"all": ["SDMX"]
]
}
}

'''Structures'''
The Structures section of the properties file defines which structural metadata should be included in the outputs.

Note, when outputting data for a Dataflow, the Dataflow and all descendants (DSD, Codelist, Concept Scheme, Agency Scheme) will be automatically included in the structure metadata that is generated and do not need to be explicitly specified. This is also true for the Structure section, specifying a specific structure such as a HierarchicalCodelist, will automatically include its descendant structures.

The arguments are:
# The structure type, this is the same as the path parameter on the REST API, i.e. Codelist.
# An array of structure filters in the format AgencyId,Id,Version. Each filter is optional, the absence of which meaning all. The keyword all can be used as a structure type to indicate all structures, which can also take the filters for agency, id and version. Example ["ECB,EXR,1.0", "SDMX", "BIS,all,2.1"]

== Compile Content ==
compileFileSystem.sh (UNIX) or compileFileSystem.bat (Windows)

=== Overview ===
The compile script reads the files in from the source directory and compiles them into the target directory. The result of the compile process is an [[Edge_Server_Environment|Environment]] that can be published to the Fusion Edge Server.

If the target directory does not exist it will be created, if it does exist the contents will be overwritten.

=== Signing Content ===
All compiled Environments are digitally signed using the a secret key provided using the '''-sgn''' command line argument. This signature ensures that the Environment files are not tampered with after they have been generated. It also ensures that the Fusion Edge Server knows that the Environment was generated by a trusted source, as the Fusion Edge Server must also know that the secret key is in order to verify the source.

=== Embargo ===
The generated Environment contains the timestamp of Compilation, it also contains a timestamp for release, which unless other specified is the same as the timestamp of compilation. The timestamp for release can be modified, using the '''-liv''' argument, to be a point in time in the future. When this argument is provided the Environment will not be made live by the Fusion Edge Server until this time has passed. It is possible configure the Fusion Edge Server in the [[Fusion_Edge_Server_Properties#Embargo_Data| properties file]] to pre-load the Environment into memory so it is ready for release.

=== Static Mode Compile ===
When running in [[Edge_Server_-_Publish_Content#Static_Mode|Static mode]] the Fusion Edge Server does not make use of the [[Edge_Server_Environment#Fusion_Edge_Server_Ledger|ledger file]], it always reads in the Environment zip file regardless of what version is in the ledger.json file. In this way a static mode Compile does not need to use the -lgr argument, as long as the source file system contains the full Environment source files for dissemination.

The compile process is:
# The Fusion Edge Compiler reads all files from the -src directory
# The Fusion Edge Compiler compiles all files
# The Fusion Edge Compiler writes the compiled files to the -tgt directory

It is possible to make use of the lgr argument if the intention is to merge new information into a previous compile. Details of merging Environments are provided in the next section.

=== Dynamic Mode Compile ===
When running in [[Edge_Server_-_Publish_Content#Dynamic_Mode|Dynamic mode]] it is important to include the location of the last (current) Environment using the '''-lgr''' argument. The [[Edge_Server_Environment#Fusion_Edge_Server_Ledger|ledger]] of the new Environment will be appended to in the new Environment, creating a new version to describe the new Environment. This is critical as the Fusion Edge Server will only update its content if it detects a change in the version of the ledger file.

By default, when using the -lgr argument, the compile process will append and/or modify information in an existing Environment. Information will not be deleted (although there are ways to force a delete, covered later).

When providing a -lgr argument, the compile process is:
# The Fusion Edge Compiler reads the referenced Environment of the -lgr argument to determine the timestamp it was built
# The Fusion Edge Compiler reads files from the source file system whose timestamps are later then the build time of the Environment (to get the deltas)
# The Fusion Edge Compiler copies all previously built data and structure stores into the new Environment - only making modifications if it found data/metadata files whose timestamps were later then the referenced Environment compiled timestamp.

In essence, the default behaviour is to copy the previous Environment, and to merge in deltas. However, this can result in undesirable results when the intention is to delete information such as datasets, series, observations, codelists, etc. If structural changes are made to a Data Structure which makes previously compiled datasets invalid, the merge process will result in an unstable system. Therefore the following arguments are provided to enable a finer level of control over what is read from the file system and what is copied from the old Environment.

==== Ignore Timestamp on File ====
The '''-f''' argument can be provided which forces all source files in the file system to be read regardless of the timestamp.

==== Replace Dataset ====
the '''-rd''' argument (replace dataset) informs the compile process to replace the dataset in the current Environment with those found in the file system. This argument has 2 possible actions '''remove''' or '''keep'''.

The '''keep''' action informs the compiler to keep any datasets that are in the Environment and not in the File System. For example if the Environment contains a dataset for Exchange Rates, Interest Rates and Employment and local file system has a dataset for Exchange Rates, using the option '''-rd keep''' will output a new Environment with the Exchange Rates dataset from the file system, and the Interest Rates and Employment datasets from the previous Environment.

The '''remove''' action informs the compiler to remove any datasets that are in the Environment and not in the File System. For example if the Environment contains a dataset for Exchange Rates, Interest Rates and Employment and local file system has a dataset for Exchange Rates, using the option '''-rd remove''' will output a new Environment with only the Exchange Rates dataset from the file system - the other two datasets will no longer exist in the new Environment.

'''Note''' prior to version 4.4.0 the -rd parameter took no arguments, the behaviour was always '''remove'''.

'''Note''' the -f argument should still be used if the intention is to read all data files in the file system, otherwise the default behaviour will be to read only the data files that have changed since the last compile.

==== Full Replace ====
Compiles all files in the source file system, ignoring any timestamp on the referenced Environment. No information is copied from the referenced Environment, with exception of the fes_ledger.json file, which is used as a base for creating the new fes_ledger.json with an updated version and history of previous versions.

===Command Line Arguments===
The minimum arguments are the source folder which contains the files to compile, and the target folder to write the compiled Environment to, along with the signature on the Environment

'''Example (minimum arguments)'''

compileFileSystem.sh -src "SourceFiles" -tgt "CompiledFiles" -sgn "password"

'''Arguments'''
* [[#Help|Help]]
* [[#Properties_File|Properties File]]
* [[#Current_Environment|Environment Arguments]] - optional, if provided then the Environment will be used as a source Environment which the newly compiled files will be merged into

'''Additional Arguments'''

{| class="wikitable sortable"
|-
! Argument !! Example !! Description
|-
| f || -f || Optional. Force all files in the source file system to be read regardless of the previous Environment compiled timestamp
|-
| fr || -fr || Optional. Full Replace. Read all files in the source file system. Do not merge any metadata or data stores from previous Environment.
|-
| liv || -liv "2020-01-30T00:00.00" || Optional. The go live time (embargo time). The generated Environment will include this timestamp so that it is not released by the Fusion Edge Server until the time. The format is yyyy-mm-ddThh:mm:ss and the timezone is GMT
|-
| ra since 4.4.0 || -ra "UNIT" -ra "OBS_CONF" ||Optional. Remove Attribute. This argument removes attributes from all Data Structures that contain it, and all corresponding datasets. This will NOT remove attributes from existing datasets in previously built environments - it only works on newly compiled datasets.
|-
| rd || -rd keep || Optional. Do not merge any data stores from previous Environment. Replace all Datasets with those read in from the file system.
 Note: From v4.4.0 this argument takes an addition value of 'keep' or 'remove', keep indicates that any datasets in the current Environment but not in the file system should be kept in the new Environment, remove indicates they should be removed with only the datasets in the file system being included in the new Environment.
|-
| src || -src "/home/compiler/source" || Required. The source directory that contains the files to be compiled (this is the tgt directory in the build file system script)
|-
| tgt || -tgt "/home/compiler/compiled" || Required. The target directory to write the files and folders to
|}

=== Properties File ===
It is possible to provide all the arguments to the Fusion Edge Compiler via a properties file, referenced by the '''-prop''' argument.

An example Properties file is given below:

{
"Ledger" : "s3:mybucket",
"SrcDir" : "/home/compiler/source",
"TgtDir" : "/home/compiler/compiled",
"ForceRebuild" : false,
"Signature" : "myuser",
"LiveTime" : "2020-01-30T00:00.00",
"S3Region": "us-east-1",
"S3SecretKey": "azxasdasfcvbn",
"S3AccessKey": "sxcvbnmu",
"ForceRebuild": false,
"ReplaceData": "keep",
"FullReplace": false,
}

== Refresh Content ==
refreshContent.sh (UNIX) or refreshContent.bat (Windows)

'''Note:''' this feature was introduced in version 4.4.0

=== Overview ===
The Refresh command updates the Environment directly, by pulling SDMX content from a web service, comparing it with what is in the Environment, and refreshing the information. Critically the refresh command will only update existing information in the Environment, for example a refresh on structures will compare the structures in the Environment against those from a web service to ensure the Environment's structures are up to date - any structure differences from the web service will be written into the Environment - structures found from the web service will NOT be written into the Environment if those structures do not already exist in the Environment.

===Command Line Arguments===
The minimum arguments are the Environment to refresh (along with the signature), what part(s) if of the Environment to refresh, where to built the new Environment, and the web service details which will be used as the source of information which will be written into the new Environment

'''Note''': If Dataflows are removed from the source API service, these will be removed from the Compiled Environment if the '-remove' argument is used, and associated data will also be removed from the Compiled Environment.
'''Note''': If using Fusion Registry as the source service, it is possible to 'remove' a structure by making it private to the compiler using the [[Main_Page#Fusion_Security_Manager_Topics|Fusion Security Manager]].

'''Example (minimum arguments)'''

refreshContent.sh -api "https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2" -src "CompiledFiles" -tgt "NewCompiled" -sgn "password" -structures

'''Arguments'''
* [[#Help|Help]]
* [[#Properties_File|Properties File]]
* [[#Web_Service_Arguments|Web service arguments]] - required as the '''refreshContent''' function pulls information from an SDMX Compliant web service into the new Environment
* [[#Current_Environment|Environment Arguments]] - required as the '''refreshContent''' must have an existing Environment to refresh

'''Additional Arguments'''

{| class="wikitable sortable"
|-
! Argument !! Example !! Description
|-
| structures || -structures || Optional. If provided the structures in the Environment will be refreshed
|-
| remove || -remove || Optional. The default behaviour is to keep structures that are in the Environment, even if those structures could not be found in the web service. This argument overrides this behaviour by requesting that structures are removed from the Environment if those structures are not found in the web service. Corresponding data and metadata will also be removed if the underlying structures upon which it relies are removed.
|}

== Refresh File System==
refreshFileSystem.sh (UNIX) or refreshFileSystem.bat (Windows)

'''Note:''' this feature was introduced in version 4.9.0 (Jan 2024)

=== Overview ===
The Refresh file system command updates all structure files in the structures folder of the Edge Server file system with those from the REST API.

The process is as follows:

# The compiler reads each file in turn for Sdmx Structural Metadata
# The compiler renames the file to postfix it with .bak (backup)
# The compiler creates a new file with the same name as the original file
# For each [[Maintainable]] structure that was read from the original file, the compiler will request the corresponding Maintainable from the REST API
# The Maintainable structure from the server is written to the new file, if the file postfix was .xml the format is SDMX-ML v3.0.0, otherwise the structure(s) will be written in fusion-json format
# If there is no structure returned from the server, the original Maintainable structure is written to the new file
# If there is an error reading the original file or an error reading the server response, the process will skip the file entirely and record the error in the audited output
# The audited output contains an array of Maintainable structures that were read from the server which differed from the original structure file

===Command Line Arguments===
The minimum arguments is the directory to read the structure file(s) from (note it will check in the 'structures' subdirectory of the location it is given), and the URL of the REST API to refresh against.

'''Example (minimum arguments)'''

refreshFileSystem.sh -api "https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2" -tgt "File"

'''Arguments'''
* [[#Help|Help]]
* [[#Web_Service_Arguments|Web service arguments]] - required as the '''refreshContent''' function pulls information from an SDMX Compliant web service into the new Environment

'''Additional Arguments'''

{| class="wikitable sortable"
|-
! Argument !! Example !! Description
|-
| tgt || -tgt "/home/files/target" || Required. The target directory to read the files and folders from and updated the files in, note this will check structures in a 'structures' subfolder
|}

== Publish Content ==
publishContent.sh (UNIX) or publishContent.bat (Windows)

===Overview===
The Publish Content script is used to move an [[Edge_Server_Environment|Environment]] from the local file system to Amazon S3. If the [[Edge_Server_Environment|Environment]] is hosted elsewhere, for example a private web server, or a file system local to the Fusion Edge Server, then a custom process must be put in place to move the [[Edge_Server_Environment|Environment]] .

The Publish Content will ensure the Amazon S3 bucket is updated to reflect the new Environment in the most efficient way, for example if it detects a file is unchanged in Environment it will not be moved. The ledger file is always moved last which ensures the Fusion Edge Server does not attempt to process the Environment until all the files are moved.

===Command Line Arguments===
The minimum arguments are the source folder of the current Environment, and the location of the Amazon S3 bucket to publish the Environment to, along with the required credentials

'''Example (minimum arguments)'''

publishContent.sh -src "CompiledFiles" -lgr “s3:mybucket” -s3rgn "us-east-1" -s3sec "azxzcvbnm" -s3acc "azxzcvbnm"

'''Arguments'''
* [[#Help|Help]]
* [[#Properties_File|Properties File]]
* [[#Current_Environment|Environment Arguments]] - required, the ledger must resolve to the S3 bucket and the s3rgn, s3sec, and s3acc must be provided

{| class="wikitable sortable"
|-
! Argument !! Example !! Description
|-
| src || -src "/home/compiler/environment" || Required. The directory that contains the Environment to be published (this is the tgt folder in the compile process)
|}

=== Properties File ===
It is possible to provide all the arguments to the Fusion Edge Compiler via a properties file, referenced by the '''-prop''' argument.

An example Properties file is given below:

{
"Ledger" : "s3:mybucket",
"SrcDir" : "/home/compiler/compiled",
"S3Region": "us-east-1",
"S3SecretKey": "azxasdasfcvbn",
"S3AccessKey": "sxcvbnmu",
}

== Print Content ==
printContent.sh (UNIX) or printContent.bat (Windows)

===Overview===
The Print Content script outputs a CSV report on what is contained in an Environment.

===Command Line Arguments===
The minimum arguments are the Environment to print along with the signature on that Environment

'''Example (minimum arguments)'''

printContent.sh -lgr "CompiledFiles" -sgn "password"

'''Arguments'''
* [[#Help|Help]]
* [[#Properties_File|Properties File]]
* [[#Current_Environment|Environment Arguments]] - required, the ledger must resolve to the Environment folder

{| class="wikitable sortable"
|-
! Argument !! Example !! Description
|-
| out || -out "/home/compiler/report.csv" || Optional. If provided this is the file that the report will be written to, otherwise the report will be written to the console
|-
| loc || -loc "fr" || Optional. If provided this is the default locale to use when printing the name of the structures - unresolvable names will be written in an alternative locale, defaulting to "en". If not provided "en" will be the default locale.
|}

Fusion Edge Compiler

2024-01-24T12:58:58Z

Mnelson: /* Command Line Arguments */

[[Category:Fusion Edge Server]]
== Overview ==
The Fusion Edge Compiler is a command line client, written in Java and can run on Windows or UNIX operating systems. Its responsibility is to compile SDMX data, structure, and metadata files for dissemination by the Fusion Edge Server. The Fusion Edge Compiler provides the following functions:

# To '''[[#Pull_Content|pull content]]''' from SDMX web services (example Fusion Registry web services) in order to populate a [[Edge_Server_-_Publish_Content#Part_1_-_Source_Data_and_Metadata_Files|local file system]] of content to publish
# To '''[[#Compile_Content|compile content]]''' in the local file system to create a new [[Edge_Server_Environment|Environment]] which can be consumed by the Fusion Edge Server
# To '''[[#Publish_Content|publish]]''' the [[Edge_Server_Environment|Environment]] to an Amazon S3 bucket from which distributed Fusion Edge Servers can take their content, if configured to do so
# To '''[[#Print_Content|print content]]''' writes the contents of an Environment to a CSV file as a report of what the compiled Environment contains

The compile function is the main function of the compiler as it is the only function which must be performed by the Fusion Edge Compiler in order to have a deployable Environment for the Fusion Edge Server. Organising SDMX files in a file system, and moving an Environment to the Edge Server can both be performed manually or via a custom automated process.

== Common Arguments ==
The following table contains arguments which are used for more then one Compile function.

=== Help ===
Every compile function will output a list of the available function arguments by passing -h to the function.

'''Example'''

buildFileSystem.sh -h

=== Properties File ===
Every compile function can read a JSON file to get the values, this can be used instead of, or in addition to adding command line arguments.

Each command line argument has a corresponding JSON property.

{| class="wikitable sortable"
|-
! Argument !! Example !! Description
|-
| prop || -prop "/home/props.json" || Optional. A reference to one or more properties files (separated by a space)
|}

'''Example'''

buildFileSystem.sh -prop "MyConfig.json"

'''Example MyConfig.json'''

{
"SdmxAPI" : "https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2",
"TgtDir" : "DemoServerFiles",
"AllData" : true
}

=== Web Service Arguments ===
The web service arguments are used by '''buildFileSystem''' and the '''refreshContent''' functions.

{| class="wikitable sortable"
|-
! Argument !! Example !! Description
|-
| api || -api "https://stats.bis.org/api/v1" || Required. The URL of the web service to pull the content from
|-
| apiv since 4.4.0 || -apiv "1.5.0" || Optional. The version of the SDMX API to query, it defaults to version 2.0.0 (entry point in Fusion Registry is ''/sdmx/v2'') the other option is 1.5.0 (Fusion Registry entry point is ''/ws/public/sdmxapi/rest'')
|-
| apict || -apict 60 || API Connect Timeout (seconds)
|-
| apirt || -apirt 600 || API Read Timeout (seconds)
|-
| apiua || -apiua EdgeServer || API User-Agent
|-
| apiformat || -apiformat sdmx-json || Format to get structures back from API, default is fusion-json (compatible with Fusion APIs and included non-sdmx structure types) - alternatives are sdmx-json, sdmx-ml-21, sdmx-ml-3
|-
| usr || -usr "myusername" || Username to authenticate with the REST API, if using the Fusion Registry it should correspond to a user account in the Fusion Registry
|-
| pwd || -pwd "mypassword" || Password to authenticate with the REST API
|}

=== Current Environment ===
The '''compileFileSystem''' and '''refreshContent''' functions can reference an existing Environment in order to merge new content into that Environment.

{| class="wikitable sortable"
|-
! Argument !! Example !! Description
|-
| prop || -prop "/home/props.json" || Optional. A reference to one or more properties files (separated by a space)
|-
| tgt || -tgt "/home/compiler/compiled" || Required. The target directory to write the new Environment to
|-
| lgr || -lgr "/home/compiler/live_environment" || The location of the current Environment
|-
| sgn || -sgn "my_signature" || Required. A secret signature to sign the new Environment files with. This signature must match the one used to sign the current Environment.
|-
| s3rgn || -s3rgn "us-east-1" || Amazon S3 region – required if the Ledger is hosted on Amazon S3
|-
| s3sec || -s3sec "azxzcvbnm" || Amazon S3 Secret – required if the Ledger is hosted on Amazon S3
|-
| s3acc || s3acc "azxzcvbnm" || Amazon S3 Access Key – required if the Ledger is hosted on Amazon S3
|}

== Pull Content ==
buildFileSystem.sh (UNIX) or buildFileSystem.bat (Windows)

=== Overview ===
The Fusion Edge Compiler queries and SDMX web service for structural metadata, data, and reference metadata content based on what it has been requested to pull. It can work against a Fusion Registry web service as well as any other SDMX web service that complies with the SDMX specification.

The Fusion Edge Compiler pulls the content to build a target directory of files in the correct structure for the compile process to operate. The Fusion Edge Compiler command line arguments focus on which Datasets to pull from the target web service. When a dataset is pulled, the corresponding metadata (Dataflow, DSD, Concepts, Codelists) will also be pulled.

It is possible to pull only updates to datasets into an existing file system, by using the -lgr or -upd property.

===Command Line Arguments===

The minimum arguments are the api to pull the data from, and the target folder to write the files to, in addition instruction on which datasets or structures to pull.

'''Example (minimum arguments)'''

buildFileSystem.sh -api "https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2" -df "all" -tgt "SourceFiles"

'''Arguments'''
* [[#Help|Help]]
* [[#Properties_File|Properties File]]
* [[#Web_Service_Arguments|Web service arguments]] - required as the '''buildFileSystem''' function pulls information from an SDMX Compliant web service
* [[#Current_Environment|Environment Arguments]] - optional, if provided then the last compile time of the Environment will be used as the updated after time to use when pulling data

'''Additional Arguments'''

{| class="wikitable sortable"
|-
! Argument !! Example !! Description
|-
| audit since 4.2.0 || -audit || If present, will output an Audit folder with a json file containing information about the REST API request and corresponding response.
|-
| exempty since 4.2.1 || -exempty || Optional. If present Dataflows from the '''-df''' parameter will be excluded from the output if they have no data pulled from the server
|-
| datastruct since 4.2.1 || -datastruct || Optional. If present each Dataflow in the '''-df''' argument will be processed to include the following related structures in the output: Pre-defined Queries, [[Publication_Table|Publication Tables]], and Category Schemes +Categorisations
|-
| df || -df "ECB:CPI(1.0)" "ECB:EXR(1.0)" || Optional. A reference to one or more Dataflows to pull data for (separated by a space). The keyword '''all''' can be used to pull data for all Dataflows. A '*' can be used to mean all versions, e.g. ECB:EXR(*)
|-
| lenient || -lenient || If present the pull process will skip failed data queries (they will not prevent the remaining pull to complete). The failure will be noted in the report for the Dataset.
|-
| metadata since 4.5.0 || -metadata incref || Issues a request for all metadata from the web service, will also include the Metadataflows and Metadata Structures which the Metadata Sets conform to. The argument '''incref''' will ensure that all target structures (the structures the metadata is authored against) and their descendants are also included in the output, '''excref''' will not include the target structures. '''Note:''' If the target structures are not present in the file system by some other means then the reference metadata will be excluded during the Compile process.
|-
| replace || -replace || If present, all the files in the the target directory will be deleted before the pull content is run
|-
| report || -report || If present, the report will be written to a file, if not present the report will be written to the System.out
|-
| str || -str "codelist=SDMX" || Structures to query (in addition to dataset related metadata). Syntax [structure type]=[agencyid],[id],[version], the only required part is the structure type with other parameters defaulting to 'all'
|-
| tgt || -tgt "/home/compiler/target" || Required. The target directory to write the files and folders to
|-
| upd || Example || This applies an updatedAfter query parameter against the target web service when querying for data (only retrieve data updated after a point in time). This is an alternative to using the '''lgr''' argument which dynamically determines the updatedAfter parameter based on the time that Environment was built.
|}

'''Pulling Data'''
When building an Edge Server for data dissemination, the compiler must be told which Dataflows to pull data for, this is achieved using the '''-df''' argument, for example:

buildFileSystem.sh -api "https://myregistry/ws/public/sdmxapi/rest" '''-df "all"''' -tgt "/home/user/EdgeServer/RawFiles"

In this example all Dataflows will be pulled from the web service, and the data for each Dataflow will be pulled.

It may be the case that the api contains Dataflows which have no data. These Dataflows will be pulled into the Edge Sever file system, meaning they will be available via the Edge Server structure web service, even though the Edge Server has no data available (in the same way the Dataflow was pulled from the source api with no corresponding dataset). This behaviour can be changed so that the Dataflow is excluded from the structure file if it has no data. This is achieved using the '''-exempty''' argument

buildFileSystem.sh -api "https://myregistry/ws/public/sdmxapi/rest" -df "all" -tgt "/home/user/EdgeServer/RawFiles" '''-exempty'''

The file system will now only contain the Dataflows where there is a corresponding dataset for the Dataflow.

Whilst the '''-str''' argument can be used to list additional structures to pull from the api, in the case of data dissemination, it is likely that a core set of additional 'data related' structures are also required. These structures are the Predefined Query, Publication Table, Categorisations and related target Structure. By adding the '''-datastruct''' argument the compiler will include, for each Dataflow, these related structures.

buildFileSystem.sh -api "https://myregistry/ws/public/sdmxapi/rest" -df "all" -tgt "/home/user/EdgeServer/RawFiles" -exempty '''-datastruct'''

The above example will pull all Dataflows that have a corresponding dataset, and for each Dataflow it will obtain related the related structures for the Dataflow.

===Properties File===
It is possible to provide all the arguments to the Fusion Edge Compiler via a properties file, referenced by the -prop argument.


An example Properties file is given below:

{
"Ledger" : "s3:mybucket",
"TgtDir" : "/home/compiler/target",
"SdmxAPI" : "https://demo.metadatatechnology.com/FusionRegistry/ws/public/sdmxapi/rest",
"UpdatedAfter" : "2010",
"Username" : "myuser",
"Password" : "pwd",
"AllData" : true,
"FullReplace" : true,
"Zip" : true,
"Metadata" : true,
"S3Region": "us-east-1",
"S3SecretKey": "azxasdasfcvbn",
"S3AccessKey": "sxcvbnmu",
"SubCubes":{
"ECB:EXR(1.0)" : {
"SubCube1" : {
"Include" : {
"FREQ":["A","M"],
"REF_AREA":["UK"]
}
}
},
"WB:POVERTY(1.0)":{ }
},
"Structures":{
"Codelist": ["ECB,EXR,1.0"]
"HierarchicalCodelist": ["ECB", "BIS"]
"all": ["SDMX"]
]
}
}

'''Structures'''
The Structures section of the properties file defines which structural metadata should be included in the outputs.

Note, when outputting data for a Dataflow, the Dataflow and all descendants (DSD, Codelist, Concept Scheme, Agency Scheme) will be automatically included in the structure metadata that is generated and do not need to be explicitly specified. This is also true for the Structure section, specifying a specific structure such as a HierarchicalCodelist, will automatically include its descendant structures.

The arguments are:
# The structure type, this is the same as the path parameter on the REST API, i.e. Codelist.
# An array of structure filters in the format AgencyId,Id,Version. Each filter is optional, the absence of which meaning all. The keyword all can be used as a structure type to indicate all structures, which can also take the filters for agency, id and version. Example ["ECB,EXR,1.0", "SDMX", "BIS,all,2.1"]

== Compile Content ==
compileFileSystem.sh (UNIX) or compileFileSystem.bat (Windows)

=== Overview ===
The compile script reads the files in from the source directory and compiles them into the target directory. The result of the compile process is an [[Edge_Server_Environment|Environment]] that can be published to the Fusion Edge Server.

If the target directory does not exist it will be created, if it does exist the contents will be overwritten.

=== Signing Content ===
All compiled Environments are digitally signed using the a secret key provided using the '''-sgn''' command line argument. This signature ensures that the Environment files are not tampered with after they have been generated. It also ensures that the Fusion Edge Server knows that the Environment was generated by a trusted source, as the Fusion Edge Server must also know that the secret key is in order to verify the source.

=== Embargo ===
The generated Environment contains the timestamp of Compilation, it also contains a timestamp for release, which unless other specified is the same as the timestamp of compilation. The timestamp for release can be modified, using the '''-liv''' argument, to be a point in time in the future. When this argument is provided the Environment will not be made live by the Fusion Edge Server until this time has passed. It is possible configure the Fusion Edge Server in the [[Fusion_Edge_Server_Properties#Embargo_Data| properties file]] to pre-load the Environment into memory so it is ready for release.

=== Static Mode Compile ===
When running in [[Edge_Server_-_Publish_Content#Static_Mode|Static mode]] the Fusion Edge Server does not make use of the [[Edge_Server_Environment#Fusion_Edge_Server_Ledger|ledger file]], it always reads in the Environment zip file regardless of what version is in the ledger.json file. In this way a static mode Compile does not need to use the -lgr argument, as long as the source file system contains the full Environment source files for dissemination.

The compile process is:
# The Fusion Edge Compiler reads all files from the -src directory
# The Fusion Edge Compiler compiles all files
# The Fusion Edge Compiler writes the compiled files to the -tgt directory

It is possible to make use of the lgr argument if the intention is to merge new information into a previous compile. Details of merging Environments are provided in the next section.

=== Dynamic Mode Compile ===
When running in [[Edge_Server_-_Publish_Content#Dynamic_Mode|Dynamic mode]] it is important to include the location of the last (current) Environment using the '''-lgr''' argument. The [[Edge_Server_Environment#Fusion_Edge_Server_Ledger|ledger]] of the new Environment will be appended to in the new Environment, creating a new version to describe the new Environment. This is critical as the Fusion Edge Server will only update its content if it detects a change in the version of the ledger file.

By default, when using the -lgr argument, the compile process will append and/or modify information in an existing Environment. Information will not be deleted (although there are ways to force a delete, covered later).

When providing a -lgr argument, the compile process is:
# The Fusion Edge Compiler reads the referenced Environment of the -lgr argument to determine the timestamp it was built
# The Fusion Edge Compiler reads files from the source file system whose timestamps are later then the build time of the Environment (to get the deltas)
# The Fusion Edge Compiler copies all previously built data and structure stores into the new Environment - only making modifications if it found data/metadata files whose timestamps were later then the referenced Environment compiled timestamp.

In essence, the default behaviour is to copy the previous Environment, and to merge in deltas. However, this can result in undesirable results when the intention is to delete information such as datasets, series, observations, codelists, etc. If structural changes are made to a Data Structure which makes previously compiled datasets invalid, the merge process will result in an unstable system. Therefore the following arguments are provided to enable a finer level of control over what is read from the file system and what is copied from the old Environment.

==== Ignore Timestamp on File ====
The '''-f''' argument can be provided which forces all source files in the file system to be read regardless of the timestamp.

==== Replace Dataset ====
the '''-rd''' argument (replace dataset) informs the compile process to replace the dataset in the current Environment with those found in the file system. This argument has 2 possible actions '''remove''' or '''keep'''.

The '''keep''' action informs the compiler to keep any datasets that are in the Environment and not in the File System. For example if the Environment contains a dataset for Exchange Rates, Interest Rates and Employment and local file system has a dataset for Exchange Rates, using the option '''-rd keep''' will output a new Environment with the Exchange Rates dataset from the file system, and the Interest Rates and Employment datasets from the previous Environment.

The '''remove''' action informs the compiler to remove any datasets that are in the Environment and not in the File System. For example if the Environment contains a dataset for Exchange Rates, Interest Rates and Employment and local file system has a dataset for Exchange Rates, using the option '''-rd remove''' will output a new Environment with only the Exchange Rates dataset from the file system - the other two datasets will no longer exist in the new Environment.

'''Note''' prior to version 4.4.0 the -rd parameter took no arguments, the behaviour was always '''remove'''.

'''Note''' the -f argument should still be used if the intention is to read all data files in the file system, otherwise the default behaviour will be to read only the data files that have changed since the last compile.

==== Full Replace ====
Compiles all files in the source file system, ignoring any timestamp on the referenced Environment. No information is copied from the referenced Environment, with exception of the fes_ledger.json file, which is used as a base for creating the new fes_ledger.json with an updated version and history of previous versions.

===Command Line Arguments===
The minimum arguments are the source folder which contains the files to compile, and the target folder to write the compiled Environment to, along with the signature on the Environment

'''Example (minimum arguments)'''

compileFileSystem.sh -src "SourceFiles" -tgt "CompiledFiles" -sgn "password"

'''Arguments'''
* [[#Help|Help]]
* [[#Properties_File|Properties File]]
* [[#Current_Environment|Environment Arguments]] - optional, if provided then the Environment will be used as a source Environment which the newly compiled files will be merged into

'''Additional Arguments'''

{| class="wikitable sortable"
|-
! Argument !! Example !! Description
|-
| f || -f || Optional. Force all files in the source file system to be read regardless of the previous Environment compiled timestamp
|-
| fr || -fr || Optional. Full Replace. Read all files in the source file system. Do not merge any metadata or data stores from previous Environment.
|-
| liv || -liv "2020-01-30T00:00.00" || Optional. The go live time (embargo time). The generated Environment will include this timestamp so that it is not released by the Fusion Edge Server until the time. The format is yyyy-mm-ddThh:mm:ss and the timezone is GMT
|-
| ra since 4.4.0 || -ra "UNIT" -ra "OBS_CONF" ||Optional. Remove Attribute. This argument removes attributes from all Data Structures that contain it, and all corresponding datasets. This will NOT remove attributes from existing datasets in previously built environments - it only works on newly compiled datasets.
|-
| rd || -rd keep || Optional. Do not merge any data stores from previous Environment. Replace all Datasets with those read in from the file system.
 Note: From v4.4.0 this argument takes an addition value of 'keep' or 'remove', keep indicates that any datasets in the current Environment but not in the file system should be kept in the new Environment, remove indicates they should be removed with only the datasets in the file system being included in the new Environment.
|-
| src || -src "/home/compiler/source" || Required. The source directory that contains the files to be compiled (this is the tgt directory in the build file system script)
|-
| tgt || -tgt "/home/compiler/compiled" || Required. The target directory to write the files and folders to
|}

=== Properties File ===
It is possible to provide all the arguments to the Fusion Edge Compiler via a properties file, referenced by the '''-prop''' argument.

An example Properties file is given below:

{
"Ledger" : "s3:mybucket",
"SrcDir" : "/home/compiler/source",
"TgtDir" : "/home/compiler/compiled",
"ForceRebuild" : false,
"Signature" : "myuser",
"LiveTime" : "2020-01-30T00:00.00",
"S3Region": "us-east-1",
"S3SecretKey": "azxasdasfcvbn",
"S3AccessKey": "sxcvbnmu",
"ForceRebuild": false,
"ReplaceData": "keep",
"FullReplace": false,
}

== Refresh Content ==
refreshContent.sh (UNIX) or refreshContent.bat (Windows)

'''Note:''' this feature was introduced in version 4.4.0

=== Overview ===
The Refresh command updates the Environment directly, by pulling SDMX content from a web service, comparing it with what is in the Environment, and refreshing the information. Critically the refresh command will only update existing information in the Environment, for example a refresh on structures will compare the structures in the Environment against those from a web service to ensure the Environment's structures are up to date - any structure differences from the web service will be written into the Environment - structures found from the web service will NOT be written into the Environment if those structures do not already exist in the Environment.

===Command Line Arguments===
The minimum arguments are the Environment to refresh (along with the signature), what part(s) if of the Environment to refresh, where to built the new Environment, and the web service details which will be used as the source of information which will be written into the new Environment

'''Example (minimum arguments)'''

refreshContent.sh -api "https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2" -src "CompiledFiles" -tgt "NewCompiled" -sgn "password" -structures

'''Arguments'''
* [[#Help|Help]]
* [[#Properties_File|Properties File]]
* [[#Web_Service_Arguments|Web service arguments]] - required as the '''refreshContent''' function pulls information from an SDMX Compliant web service into the new Environment
* [[#Current_Environment|Environment Arguments]] - required as the '''refreshContent''' must have an existing Environment to refresh

'''Additional Arguments'''

{| class="wikitable sortable"
|-
! Argument !! Example !! Description
|-
| structures || -structures || Optional. If provided the structures in the Environment will be refreshed
|-
| remove || -remove || Optional. The default behaviour is to keep structures that are in the Environment, even if those structures could not be found in the web service. This argument overrides this behaviour by requesting that structures are removed from the Environment if those structures are not found in the web service. Corresponding data and metadata will also be removed if the underlying structures upon which it relies are removed.
|}

== Refresh File System==
refreshFileSystem.sh (UNIX) or refreshFileSystem.bat (Windows)

'''Note:''' this feature was introduced in version 4.9.0 (Jan 2024)

=== Overview ===
The Refresh file system command updates all structure files in the structures folder of the Edge Server file system with those from the REST API.

The process is as follows:

# The compiler reads each file in turn for Sdmx Structural Metadata
# The compiler renames the file to postfix it with .bak (backup)
# The compiler creates a new file with the same name as the original file
# For each [[Maintainable]] structure that was read from the original file, the compiler will request the corresponding Maintainable from the REST API
# The Maintainable structure from the server is written to the new file, if the file postfix was .xml the format is SDMX-ML v3.0.0, otherwise the structure(s) will be written in fusion-json format
# If there is no structure returned from the server, the original Maintainable structure is written to the new file
# If there is an error reading the original file or an error reading the server response, the process will skip the file entirely and record the error in the audited output
# The audited output contains an array of Maintainable structures that were read from the server which differed from the original structure file

===Command Line Arguments===
The minimum arguments is the directory to read the structure file(s) from (note it will check in the 'structures' subdirectory of the location it is given), and the URL of the REST API to refresh against.

'''Example (minimum arguments)'''

refreshFileSystem.sh -api "https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2" -tgt "File"

'''Arguments'''
* [[#Help|Help]]
* [[#Web_Service_Arguments|Web service arguments]] - required as the '''refreshContent''' function pulls information from an SDMX Compliant web service into the new Environment

'''Additional Arguments'''

{| class="wikitable sortable"
|-
! Argument !! Example !! Description
|-
| tgt || -tgt "/home/files/target" || Required. The target directory to read the files and folders from and updated the files in, note this will check structures in a 'structures' subfolder
|}

== Publish Content ==
publishContent.sh (UNIX) or publishContent.bat (Windows)

===Overview===
The Publish Content script is used to move an [[Edge_Server_Environment|Environment]] from the local file system to Amazon S3. If the [[Edge_Server_Environment|Environment]] is hosted elsewhere, for example a private web server, or a file system local to the Fusion Edge Server, then a custom process must be put in place to move the [[Edge_Server_Environment|Environment]] .

The Publish Content will ensure the Amazon S3 bucket is updated to reflect the new Environment in the most efficient way, for example if it detects a file is unchanged in Environment it will not be moved. The ledger file is always moved last which ensures the Fusion Edge Server does not attempt to process the Environment until all the files are moved.

===Command Line Arguments===
The minimum arguments are the source folder of the current Environment, and the location of the Amazon S3 bucket to publish the Environment to, along with the required credentials

'''Example (minimum arguments)'''

publishContent.sh -src "CompiledFiles" -lgr “s3:mybucket” -s3rgn "us-east-1" -s3sec "azxzcvbnm" -s3acc "azxzcvbnm"

'''Arguments'''
* [[#Help|Help]]
* [[#Properties_File|Properties File]]
* [[#Current_Environment|Environment Arguments]] - required, the ledger must resolve to the S3 bucket and the s3rgn, s3sec, and s3acc must be provided

{| class="wikitable sortable"
|-
! Argument !! Example !! Description
|-
| src || -src "/home/compiler/environment" || Required. The directory that contains the Environment to be published (this is the tgt folder in the compile process)
|}

=== Properties File ===
It is possible to provide all the arguments to the Fusion Edge Compiler via a properties file, referenced by the '''-prop''' argument.

An example Properties file is given below:

{
"Ledger" : "s3:mybucket",
"SrcDir" : "/home/compiler/compiled",
"S3Region": "us-east-1",
"S3SecretKey": "azxasdasfcvbn",
"S3AccessKey": "sxcvbnmu",
}

== Print Content ==
printContent.sh (UNIX) or printContent.bat (Windows)

===Overview===
The Print Content script outputs a CSV report on what is contained in an Environment.

===Command Line Arguments===
The minimum arguments are the Environment to print along with the signature on that Environment

'''Example (minimum arguments)'''

printContent.sh -lgr "CompiledFiles" -sgn "password"

'''Arguments'''
* [[#Help|Help]]
* [[#Properties_File|Properties File]]
* [[#Current_Environment|Environment Arguments]] - required, the ledger must resolve to the Environment folder

{| class="wikitable sortable"
|-
! Argument !! Example !! Description
|-
| out || -out "/home/compiler/report.csv" || Optional. If provided this is the file that the report will be written to, otherwise the report will be written to the console
|-
| loc || -loc "fr" || Optional. If provided this is the default locale to use when printing the name of the structures - unresolvable names will be written in an alternative locale, defaulting to "en". If not provided "en" will be the default locale.
|}

Fusion Edge Compiler

2024-01-24T12:57:04Z

Mnelson: /* Refresh Content */

[[Category:Fusion Edge Server]]
== Overview ==
The Fusion Edge Compiler is a command line client, written in Java and can run on Windows or UNIX operating systems. Its responsibility is to compile SDMX data, structure, and metadata files for dissemination by the Fusion Edge Server. The Fusion Edge Compiler provides the following functions:

# To '''[[#Pull_Content|pull content]]''' from SDMX web services (example Fusion Registry web services) in order to populate a [[Edge_Server_-_Publish_Content#Part_1_-_Source_Data_and_Metadata_Files|local file system]] of content to publish
# To '''[[#Compile_Content|compile content]]''' in the local file system to create a new [[Edge_Server_Environment|Environment]] which can be consumed by the Fusion Edge Server
# To '''[[#Publish_Content|publish]]''' the [[Edge_Server_Environment|Environment]] to an Amazon S3 bucket from which distributed Fusion Edge Servers can take their content, if configured to do so
# To '''[[#Print_Content|print content]]''' writes the contents of an Environment to a CSV file as a report of what the compiled Environment contains

The compile function is the main function of the compiler as it is the only function which must be performed by the Fusion Edge Compiler in order to have a deployable Environment for the Fusion Edge Server. Organising SDMX files in a file system, and moving an Environment to the Edge Server can both be performed manually or via a custom automated process.

== Common Arguments ==
The following table contains arguments which are used for more then one Compile function.

=== Help ===
Every compile function will output a list of the available function arguments by passing -h to the function.

'''Example'''

buildFileSystem.sh -h

=== Properties File ===
Every compile function can read a JSON file to get the values, this can be used instead of, or in addition to adding command line arguments.

Each command line argument has a corresponding JSON property.

{| class="wikitable sortable"
|-
! Argument !! Example !! Description
|-
| prop || -prop "/home/props.json" || Optional. A reference to one or more properties files (separated by a space)
|}

'''Example'''

buildFileSystem.sh -prop "MyConfig.json"

'''Example MyConfig.json'''

{
"SdmxAPI" : "https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2",
"TgtDir" : "DemoServerFiles",
"AllData" : true
}

=== Web Service Arguments ===
The web service arguments are used by '''buildFileSystem''' and the '''refreshContent''' functions.

{| class="wikitable sortable"
|-
! Argument !! Example !! Description
|-
| api || -api "https://stats.bis.org/api/v1" || Required. The URL of the web service to pull the content from
|-
| apiv since 4.4.0 || -apiv "1.5.0" || Optional. The version of the SDMX API to query, it defaults to version 2.0.0 (entry point in Fusion Registry is ''/sdmx/v2'') the other option is 1.5.0 (Fusion Registry entry point is ''/ws/public/sdmxapi/rest'')
|-
| apict || -apict 60 || API Connect Timeout (seconds)
|-
| apirt || -apirt 600 || API Read Timeout (seconds)
|-
| apiua || -apiua EdgeServer || API User-Agent
|-
| apiformat || -apiformat sdmx-json || Format to get structures back from API, default is fusion-json (compatible with Fusion APIs and included non-sdmx structure types) - alternatives are sdmx-json, sdmx-ml-21, sdmx-ml-3
|-
| usr || -usr "myusername" || Username to authenticate with the REST API, if using the Fusion Registry it should correspond to a user account in the Fusion Registry
|-
| pwd || -pwd "mypassword" || Password to authenticate with the REST API
|}

=== Current Environment ===
The '''compileFileSystem''' and '''refreshContent''' functions can reference an existing Environment in order to merge new content into that Environment.

{| class="wikitable sortable"
|-
! Argument !! Example !! Description
|-
| prop || -prop "/home/props.json" || Optional. A reference to one or more properties files (separated by a space)
|-
| tgt || -tgt "/home/compiler/compiled" || Required. The target directory to write the new Environment to
|-
| lgr || -lgr "/home/compiler/live_environment" || The location of the current Environment
|-
| sgn || -sgn "my_signature" || Required. A secret signature to sign the new Environment files with. This signature must match the one used to sign the current Environment.
|-
| s3rgn || -s3rgn "us-east-1" || Amazon S3 region – required if the Ledger is hosted on Amazon S3
|-
| s3sec || -s3sec "azxzcvbnm" || Amazon S3 Secret – required if the Ledger is hosted on Amazon S3
|-
| s3acc || s3acc "azxzcvbnm" || Amazon S3 Access Key – required if the Ledger is hosted on Amazon S3
|}

== Pull Content ==
buildFileSystem.sh (UNIX) or buildFileSystem.bat (Windows)

=== Overview ===
The Fusion Edge Compiler queries and SDMX web service for structural metadata, data, and reference metadata content based on what it has been requested to pull. It can work against a Fusion Registry web service as well as any other SDMX web service that complies with the SDMX specification.

The Fusion Edge Compiler pulls the content to build a target directory of files in the correct structure for the compile process to operate. The Fusion Edge Compiler command line arguments focus on which Datasets to pull from the target web service. When a dataset is pulled, the corresponding metadata (Dataflow, DSD, Concepts, Codelists) will also be pulled.

It is possible to pull only updates to datasets into an existing file system, by using the -lgr or -upd property.

===Command Line Arguments===

The minimum arguments are the api to pull the data from, and the target folder to write the files to, in addition instruction on which datasets or structures to pull.

'''Example (minimum arguments)'''

buildFileSystem.sh -api "https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2" -df "all" -tgt "SourceFiles"

'''Arguments'''
* [[#Help|Help]]
* [[#Properties_File|Properties File]]
* [[#Web_Service_Arguments|Web service arguments]] - required as the '''buildFileSystem''' function pulls information from an SDMX Compliant web service
* [[#Current_Environment|Environment Arguments]] - optional, if provided then the last compile time of the Environment will be used as the updated after time to use when pulling data

'''Additional Arguments'''

{| class="wikitable sortable"
|-
! Argument !! Example !! Description
|-
| audit since 4.2.0 || -audit || If present, will output an Audit folder with a json file containing information about the REST API request and corresponding response.
|-
| exempty since 4.2.1 || -exempty || Optional. If present Dataflows from the '''-df''' parameter will be excluded from the output if they have no data pulled from the server
|-
| datastruct since 4.2.1 || -datastruct || Optional. If present each Dataflow in the '''-df''' argument will be processed to include the following related structures in the output: Pre-defined Queries, [[Publication_Table|Publication Tables]], and Category Schemes +Categorisations
|-
| df || -df "ECB:CPI(1.0)" "ECB:EXR(1.0)" || Optional. A reference to one or more Dataflows to pull data for (separated by a space). The keyword '''all''' can be used to pull data for all Dataflows. A '*' can be used to mean all versions, e.g. ECB:EXR(*)
|-
| lenient || -lenient || If present the pull process will skip failed data queries (they will not prevent the remaining pull to complete). The failure will be noted in the report for the Dataset.
|-
| metadata since 4.5.0 || -metadata incref || Issues a request for all metadata from the web service, will also include the Metadataflows and Metadata Structures which the Metadata Sets conform to. The argument '''incref''' will ensure that all target structures (the structures the metadata is authored against) and their descendants are also included in the output, '''excref''' will not include the target structures. '''Note:''' If the target structures are not present in the file system by some other means then the reference metadata will be excluded during the Compile process.
|-
| replace || -replace || If present, all the files in the the target directory will be deleted before the pull content is run
|-
| report || -report || If present, the report will be written to a file, if not present the report will be written to the System.out
|-
| str || -str "codelist=SDMX" || Structures to query (in addition to dataset related metadata). Syntax [structure type]=[agencyid],[id],[version], the only required part is the structure type with other parameters defaulting to 'all'
|-
| tgt || -tgt "/home/compiler/target" || Required. The target directory to write the files and folders to
|-
| upd || Example || This applies an updatedAfter query parameter against the target web service when querying for data (only retrieve data updated after a point in time). This is an alternative to using the '''lgr''' argument which dynamically determines the updatedAfter parameter based on the time that Environment was built.
|}

'''Pulling Data'''
When building an Edge Server for data dissemination, the compiler must be told which Dataflows to pull data for, this is achieved using the '''-df''' argument, for example:

buildFileSystem.sh -api "https://myregistry/ws/public/sdmxapi/rest" '''-df "all"''' -tgt "/home/user/EdgeServer/RawFiles"

In this example all Dataflows will be pulled from the web service, and the data for each Dataflow will be pulled.

It may be the case that the api contains Dataflows which have no data. These Dataflows will be pulled into the Edge Sever file system, meaning they will be available via the Edge Server structure web service, even though the Edge Server has no data available (in the same way the Dataflow was pulled from the source api with no corresponding dataset). This behaviour can be changed so that the Dataflow is excluded from the structure file if it has no data. This is achieved using the '''-exempty''' argument

buildFileSystem.sh -api "https://myregistry/ws/public/sdmxapi/rest" -df "all" -tgt "/home/user/EdgeServer/RawFiles" '''-exempty'''

The file system will now only contain the Dataflows where there is a corresponding dataset for the Dataflow.

Whilst the '''-str''' argument can be used to list additional structures to pull from the api, in the case of data dissemination, it is likely that a core set of additional 'data related' structures are also required. These structures are the Predefined Query, Publication Table, Categorisations and related target Structure. By adding the '''-datastruct''' argument the compiler will include, for each Dataflow, these related structures.

buildFileSystem.sh -api "https://myregistry/ws/public/sdmxapi/rest" -df "all" -tgt "/home/user/EdgeServer/RawFiles" -exempty '''-datastruct'''

The above example will pull all Dataflows that have a corresponding dataset, and for each Dataflow it will obtain related the related structures for the Dataflow.

===Properties File===
It is possible to provide all the arguments to the Fusion Edge Compiler via a properties file, referenced by the -prop argument.


An example Properties file is given below:

{
"Ledger" : "s3:mybucket",
"TgtDir" : "/home/compiler/target",
"SdmxAPI" : "https://demo.metadatatechnology.com/FusionRegistry/ws/public/sdmxapi/rest",
"UpdatedAfter" : "2010",
"Username" : "myuser",
"Password" : "pwd",
"AllData" : true,
"FullReplace" : true,
"Zip" : true,
"Metadata" : true,
"S3Region": "us-east-1",
"S3SecretKey": "azxasdasfcvbn",
"S3AccessKey": "sxcvbnmu",
"SubCubes":{
"ECB:EXR(1.0)" : {
"SubCube1" : {
"Include" : {
"FREQ":["A","M"],
"REF_AREA":["UK"]
}
}
},
"WB:POVERTY(1.0)":{ }
},
"Structures":{
"Codelist": ["ECB,EXR,1.0"]
"HierarchicalCodelist": ["ECB", "BIS"]
"all": ["SDMX"]
]
}
}

'''Structures'''
The Structures section of the properties file defines which structural metadata should be included in the outputs.

Note, when outputting data for a Dataflow, the Dataflow and all descendants (DSD, Codelist, Concept Scheme, Agency Scheme) will be automatically included in the structure metadata that is generated and do not need to be explicitly specified. This is also true for the Structure section, specifying a specific structure such as a HierarchicalCodelist, will automatically include its descendant structures.

The arguments are:
# The structure type, this is the same as the path parameter on the REST API, i.e. Codelist.
# An array of structure filters in the format AgencyId,Id,Version. Each filter is optional, the absence of which meaning all. The keyword all can be used as a structure type to indicate all structures, which can also take the filters for agency, id and version. Example ["ECB,EXR,1.0", "SDMX", "BIS,all,2.1"]

== Compile Content ==
compileFileSystem.sh (UNIX) or compileFileSystem.bat (Windows)

=== Overview ===
The compile script reads the files in from the source directory and compiles them into the target directory. The result of the compile process is an [[Edge_Server_Environment|Environment]] that can be published to the Fusion Edge Server.

If the target directory does not exist it will be created, if it does exist the contents will be overwritten.

=== Signing Content ===
All compiled Environments are digitally signed using the a secret key provided using the '''-sgn''' command line argument. This signature ensures that the Environment files are not tampered with after they have been generated. It also ensures that the Fusion Edge Server knows that the Environment was generated by a trusted source, as the Fusion Edge Server must also know that the secret key is in order to verify the source.

=== Embargo ===
The generated Environment contains the timestamp of Compilation, it also contains a timestamp for release, which unless other specified is the same as the timestamp of compilation. The timestamp for release can be modified, using the '''-liv''' argument, to be a point in time in the future. When this argument is provided the Environment will not be made live by the Fusion Edge Server until this time has passed. It is possible configure the Fusion Edge Server in the [[Fusion_Edge_Server_Properties#Embargo_Data| properties file]] to pre-load the Environment into memory so it is ready for release.

=== Static Mode Compile ===
When running in [[Edge_Server_-_Publish_Content#Static_Mode|Static mode]] the Fusion Edge Server does not make use of the [[Edge_Server_Environment#Fusion_Edge_Server_Ledger|ledger file]], it always reads in the Environment zip file regardless of what version is in the ledger.json file. In this way a static mode Compile does not need to use the -lgr argument, as long as the source file system contains the full Environment source files for dissemination.

The compile process is:
# The Fusion Edge Compiler reads all files from the -src directory
# The Fusion Edge Compiler compiles all files
# The Fusion Edge Compiler writes the compiled files to the -tgt directory

It is possible to make use of the lgr argument if the intention is to merge new information into a previous compile. Details of merging Environments are provided in the next section.

=== Dynamic Mode Compile ===
When running in [[Edge_Server_-_Publish_Content#Dynamic_Mode|Dynamic mode]] it is important to include the location of the last (current) Environment using the '''-lgr''' argument. The [[Edge_Server_Environment#Fusion_Edge_Server_Ledger|ledger]] of the new Environment will be appended to in the new Environment, creating a new version to describe the new Environment. This is critical as the Fusion Edge Server will only update its content if it detects a change in the version of the ledger file.

By default, when using the -lgr argument, the compile process will append and/or modify information in an existing Environment. Information will not be deleted (although there are ways to force a delete, covered later).

When providing a -lgr argument, the compile process is:
# The Fusion Edge Compiler reads the referenced Environment of the -lgr argument to determine the timestamp it was built
# The Fusion Edge Compiler reads files from the source file system whose timestamps are later then the build time of the Environment (to get the deltas)
# The Fusion Edge Compiler copies all previously built data and structure stores into the new Environment - only making modifications if it found data/metadata files whose timestamps were later then the referenced Environment compiled timestamp.

In essence, the default behaviour is to copy the previous Environment, and to merge in deltas. However, this can result in undesirable results when the intention is to delete information such as datasets, series, observations, codelists, etc. If structural changes are made to a Data Structure which makes previously compiled datasets invalid, the merge process will result in an unstable system. Therefore the following arguments are provided to enable a finer level of control over what is read from the file system and what is copied from the old Environment.

==== Ignore Timestamp on File ====
The '''-f''' argument can be provided which forces all source files in the file system to be read regardless of the timestamp.

==== Replace Dataset ====
the '''-rd''' argument (replace dataset) informs the compile process to replace the dataset in the current Environment with those found in the file system. This argument has 2 possible actions '''remove''' or '''keep'''.

The '''keep''' action informs the compiler to keep any datasets that are in the Environment and not in the File System. For example if the Environment contains a dataset for Exchange Rates, Interest Rates and Employment and local file system has a dataset for Exchange Rates, using the option '''-rd keep''' will output a new Environment with the Exchange Rates dataset from the file system, and the Interest Rates and Employment datasets from the previous Environment.

The '''remove''' action informs the compiler to remove any datasets that are in the Environment and not in the File System. For example if the Environment contains a dataset for Exchange Rates, Interest Rates and Employment and local file system has a dataset for Exchange Rates, using the option '''-rd remove''' will output a new Environment with only the Exchange Rates dataset from the file system - the other two datasets will no longer exist in the new Environment.

'''Note''' prior to version 4.4.0 the -rd parameter took no arguments, the behaviour was always '''remove'''.

'''Note''' the -f argument should still be used if the intention is to read all data files in the file system, otherwise the default behaviour will be to read only the data files that have changed since the last compile.

==== Full Replace ====
Compiles all files in the source file system, ignoring any timestamp on the referenced Environment. No information is copied from the referenced Environment, with exception of the fes_ledger.json file, which is used as a base for creating the new fes_ledger.json with an updated version and history of previous versions.

===Command Line Arguments===
The minimum arguments are the source folder which contains the files to compile, and the target folder to write the compiled Environment to, along with the signature on the Environment

'''Example (minimum arguments)'''

compileFileSystem.sh -src "SourceFiles" -tgt "CompiledFiles" -sgn "password"

'''Arguments'''
* [[#Help|Help]]
* [[#Properties_File|Properties File]]
* [[#Current_Environment|Environment Arguments]] - optional, if provided then the Environment will be used as a source Environment which the newly compiled files will be merged into

'''Additional Arguments'''

{| class="wikitable sortable"
|-
! Argument !! Example !! Description
|-
| f || -f || Optional. Force all files in the source file system to be read regardless of the previous Environment compiled timestamp
|-
| fr || -fr || Optional. Full Replace. Read all files in the source file system. Do not merge any metadata or data stores from previous Environment.
|-
| liv || -liv "2020-01-30T00:00.00" || Optional. The go live time (embargo time). The generated Environment will include this timestamp so that it is not released by the Fusion Edge Server until the time. The format is yyyy-mm-ddThh:mm:ss and the timezone is GMT
|-
| ra since 4.4.0 || -ra "UNIT" -ra "OBS_CONF" ||Optional. Remove Attribute. This argument removes attributes from all Data Structures that contain it, and all corresponding datasets. This will NOT remove attributes from existing datasets in previously built environments - it only works on newly compiled datasets.
|-
| rd || -rd keep || Optional. Do not merge any data stores from previous Environment. Replace all Datasets with those read in from the file system.
 Note: From v4.4.0 this argument takes an addition value of 'keep' or 'remove', keep indicates that any datasets in the current Environment but not in the file system should be kept in the new Environment, remove indicates they should be removed with only the datasets in the file system being included in the new Environment.
|-
| src || -src "/home/compiler/source" || Required. The source directory that contains the files to be compiled (this is the tgt directory in the build file system script)
|-
| tgt || -tgt "/home/compiler/compiled" || Required. The target directory to write the files and folders to
|}

=== Properties File ===
It is possible to provide all the arguments to the Fusion Edge Compiler via a properties file, referenced by the '''-prop''' argument.

An example Properties file is given below:

{
"Ledger" : "s3:mybucket",
"SrcDir" : "/home/compiler/source",
"TgtDir" : "/home/compiler/compiled",
"ForceRebuild" : false,
"Signature" : "myuser",
"LiveTime" : "2020-01-30T00:00.00",
"S3Region": "us-east-1",
"S3SecretKey": "azxasdasfcvbn",
"S3AccessKey": "sxcvbnmu",
"ForceRebuild": false,
"ReplaceData": "keep",
"FullReplace": false,
}

== Refresh Content ==
refreshContent.sh (UNIX) or refreshContent.bat (Windows)

'''Note:''' this feature was introduced in version 4.4.0

=== Overview ===
The Refresh command updates the Environment directly, by pulling SDMX content from a web service, comparing it with what is in the Environment, and refreshing the information. Critically the refresh command will only update existing information in the Environment, for example a refresh on structures will compare the structures in the Environment against those from a web service to ensure the Environment's structures are up to date - any structure differences from the web service will be written into the Environment - structures found from the web service will NOT be written into the Environment if those structures do not already exist in the Environment.

===Command Line Arguments===
The minimum arguments are the Environment to refresh (along with the signature), what part(s) if of the Environment to refresh, where to built the new Environment, and the web service details which will be used as the source of information which will be written into the new Environment

'''Example (minimum arguments)'''

refreshContent.sh -api "https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2" -src "CompiledFiles" -tgt "NewCompiled" -sgn "password" -structures

'''Arguments'''
* [[#Help|Help]]
* [[#Properties_File|Properties File]]
* [[#Web_Service_Arguments|Web service arguments]] - required as the '''refreshContent''' function pulls information from an SDMX Compliant web service into the new Environment
* [[#Current_Environment|Environment Arguments]] - required as the '''refreshContent''' must have an existing Environment to refresh

'''Additional Arguments'''

{| class="wikitable sortable"
|-
! Argument !! Example !! Description
|-
| structures || -structures || Optional. If provided the structures in the Environment will be refreshed
|-
| remove || -remove || Optional. The default behaviour is to keep structures that are in the Environment, even if those structures could not be found in the web service. This argument overrides this behaviour by requesting that structures are removed from the Environment if those structures are not found in the web service. Corresponding data and metadata will also be removed if the underlying structures upon which it relies are removed.
|}

== Refresh File System==
refreshFileSystem.sh (UNIX) or refreshFileSystem.bat (Windows)

'''Note:''' this feature was introduced in version 4.9.0 (Jan 2024)

=== Overview ===
The Refresh file system command updates all structure files in the structures folder of the Edge Server file system with those from the REST API.

The process is as follows:

# The compiler reads each file in turn for Sdmx Structural Metadata
# The compiler renames the file to postfix it with .bak (backup)
# The compiler creates a new file with the same name as the original file
# For each [[Maintainable]] structure that was read from the original file, the compiler will request the corresponding Maintainable from the REST API
# The Maintainable structure from the server is written to the new file, if the file postfix was .xml the format is SDMX-ML v3.0.0, otherwise the structure(s) will be written in fusion-json format
# If there is no structure returned from the server, the original Maintainable structure is written to the new file
# If there is an error reading the original file or an error reading the server response, the process will skip the file entirely and record the error in the audited output
# The audited output contains an array of Maintainable structures that were read from the server which differed from the original structure file

===Command Line Arguments===
The minimum arguments is the directory to read the structure file(s) from (note it will check in the 'structures' subdirectory of the location it is given), and the URL of the REST API to refresh against.

'''Example (minimum arguments)'''

refreshFileSystem.sh -api "https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2" -tgt "File"

'''Arguments'''
* [[#Help|Help]]
* [[#Web_Service_Arguments|Web service arguments]] - required as the '''refreshContent''' function pulls information from an SDMX Compliant web service into the new Environment

== Publish Content ==
publishContent.sh (UNIX) or publishContent.bat (Windows)

===Overview===
The Publish Content script is used to move an [[Edge_Server_Environment|Environment]] from the local file system to Amazon S3. If the [[Edge_Server_Environment|Environment]] is hosted elsewhere, for example a private web server, or a file system local to the Fusion Edge Server, then a custom process must be put in place to move the [[Edge_Server_Environment|Environment]] .

The Publish Content will ensure the Amazon S3 bucket is updated to reflect the new Environment in the most efficient way, for example if it detects a file is unchanged in Environment it will not be moved. The ledger file is always moved last which ensures the Fusion Edge Server does not attempt to process the Environment until all the files are moved.

===Command Line Arguments===
The minimum arguments are the source folder of the current Environment, and the location of the Amazon S3 bucket to publish the Environment to, along with the required credentials

'''Example (minimum arguments)'''

publishContent.sh -src "CompiledFiles" -lgr “s3:mybucket” -s3rgn "us-east-1" -s3sec "azxzcvbnm" -s3acc "azxzcvbnm"

'''Arguments'''
* [[#Help|Help]]
* [[#Properties_File|Properties File]]
* [[#Current_Environment|Environment Arguments]] - required, the ledger must resolve to the S3 bucket and the s3rgn, s3sec, and s3acc must be provided

{| class="wikitable sortable"
|-
! Argument !! Example !! Description
|-
| src || -src "/home/compiler/environment" || Required. The directory that contains the Environment to be published (this is the tgt folder in the compile process)
|}

=== Properties File ===
It is possible to provide all the arguments to the Fusion Edge Compiler via a properties file, referenced by the '''-prop''' argument.

An example Properties file is given below:

{
"Ledger" : "s3:mybucket",
"SrcDir" : "/home/compiler/compiled",
"S3Region": "us-east-1",
"S3SecretKey": "azxasdasfcvbn",
"S3AccessKey": "sxcvbnmu",
}

== Print Content ==
printContent.sh (UNIX) or printContent.bat (Windows)

===Overview===
The Print Content script outputs a CSV report on what is contained in an Environment.

===Command Line Arguments===
The minimum arguments are the Environment to print along with the signature on that Environment

'''Example (minimum arguments)'''

printContent.sh -lgr "CompiledFiles" -sgn "password"

'''Arguments'''
* [[#Help|Help]]
* [[#Properties_File|Properties File]]
* [[#Current_Environment|Environment Arguments]] - required, the ledger must resolve to the Environment folder

{| class="wikitable sortable"
|-
! Argument !! Example !! Description
|-
| out || -out "/home/compiler/report.csv" || Optional. If provided this is the file that the report will be written to, otherwise the report will be written to the console
|-
| loc || -loc "fr" || Optional. If provided this is the default locale to use when printing the name of the structures - unresolvable names will be written in an alternative locale, defaulting to "en". If not provided "en" will be the default locale.
|}

Install Apache Tomcat

2024-01-23T15:18:11Z

Mnelson: /* GMT */

[[Category:How_To]]
[[Category:Fusion Registry Install]]
=Overview=
Fusion Registry Core, Fusion Edge Server and other modules are Java applications which run under a Java Web Application Server.

A number of different open source and commerical options are available, but this guide explains how to install the free to use Apache Tomcat.

=Installing the Apache Tomcat=
Download and install [http://tomcat.apache.org/ Apache Tomcat].

Choose either Tomcat 8.5 or 9.0.

=Set the Java runtime options=

Tomcat needs to be configured to set certain Java options when it starts. The most important is to set the amount of memory that the Fusion Registry service will be allocated. A minimum of 4GB is recommended and the following explains how to do that.

====Windows====
* Go to <code>tomcat\bin</code> folder
* Create a file called <code>setenv.bat</code> with the following content
set "JAVA_OPTS=-Xmx4G"

====Linux====
* Go to the <code>tomcat/bin</code> folder
* Create a file called <code>setenv.sh</code> with the following content (it does not need to have 'x' executable permissions, but must be readable 'r')
export JAVA_OPTS="-Xmx4G"

==== GMT ====
It is also reccomended to set the timezone to GMT

-Duser.timezone=GMT

e.g.

set "JAVA_OPTS=-Xmx4G -Duser.timezone=GMT"

====Special purpose Java runtime options====
Other Java runtime options that can be set for special purposes and are explained in [[Tomcat setenv file]] article.

=Changing the Tomcat port number (optional)=
Tomcat's default configuration is to serve its applications on port 8080. That means, by default, the Fusion Registry user interface will be available on the following URL: 
<code>http://localhost:8080/FusionRegistry</code>

You will need to choose a different port if other software on the machine is already using port 8080.

* Go to <code>tomcat/conf</code> folder
* Edit <code>server.xml</code>
* Search "Connector port"
* Replace "8080" by your port number (8081 for instance)
* Restart tomcat server.

Install Apache Tomcat

2024-01-23T15:17:49Z

Mnelson: /* GMT */

[[Category:How_To]]
[[Category:Fusion Registry Install]]
=Overview=
Fusion Registry Core, Fusion Edge Server and other modules are Java applications which run under a Java Web Application Server.

A number of different open source and commerical options are available, but this guide explains how to install the free to use Apache Tomcat.

=Installing the Apache Tomcat=
Download and install [http://tomcat.apache.org/ Apache Tomcat].

Choose either Tomcat 8.5 or 9.0.

=Set the Java runtime options=

Tomcat needs to be configured to set certain Java options when it starts. The most important is to set the amount of memory that the Fusion Registry service will be allocated. A minimum of 4GB is recommended and the following explains how to do that.

====Windows====
* Go to <code>tomcat\bin</code> folder
* Create a file called <code>setenv.bat</code> with the following content
set "JAVA_OPTS=-Xmx4G"

====Linux====
* Go to the <code>tomcat/bin</code> folder
* Create a file called <code>setenv.sh</code> with the following content (it does not need to have 'x' executable permissions, but must be readable 'r')
export JAVA_OPTS="-Xmx4G"

==== GMT ====
It is also reccomended to set the timezone to GMT

-Duser.timezone=GMT

====Special purpose Java runtime options====
Other Java runtime options that can be set for special purposes and are explained in [[Tomcat setenv file]] article.

=Changing the Tomcat port number (optional)=
Tomcat's default configuration is to serve its applications on port 8080. That means, by default, the Fusion Registry user interface will be available on the following URL: 
<code>http://localhost:8080/FusionRegistry</code>

You will need to choose a different port if other software on the machine is already using port 8080.

* Go to <code>tomcat/conf</code> folder
* Edit <code>server.xml</code>
* Search "Connector port"
* Replace "8080" by your port number (8081 for instance)
* Restart tomcat server.

Install Apache Tomcat

2024-01-23T15:17:30Z

Mnelson: /* Set the Java runtime options */

[[Category:How_To]]
[[Category:Fusion Registry Install]]
=Overview=
Fusion Registry Core, Fusion Edge Server and other modules are Java applications which run under a Java Web Application Server.

A number of different open source and commerical options are available, but this guide explains how to install the free to use Apache Tomcat.

=Installing the Apache Tomcat=
Download and install [http://tomcat.apache.org/ Apache Tomcat].

Choose either Tomcat 8.5 or 9.0.

=Set the Java runtime options=

Tomcat needs to be configured to set certain Java options when it starts. The most important is to set the amount of memory that the Fusion Registry service will be allocated. A minimum of 4GB is recommended and the following explains how to do that.

====Windows====
* Go to <code>tomcat\bin</code> folder
* Create a file called <code>setenv.bat</code> with the following content
set "JAVA_OPTS=-Xmx4G"

====Linux====
* Go to the <code>tomcat/bin</code> folder
* Create a file called <code>setenv.sh</code> with the following content (it does not need to have 'x' executable permissions, but must be readable 'r')
export JAVA_OPTS="-Xmx4G"

== GMT ==
It is also reccomended to set the timezone to GMT

-Duser.timezone=GMT

====Special purpose Java runtime options====
Other Java runtime options that can be set for special purposes and are explained in [[Tomcat setenv file]] article.

=Changing the Tomcat port number (optional)=
Tomcat's default configuration is to serve its applications on port 8080. That means, by default, the Fusion Registry user interface will be available on the following URL: 
<code>http://localhost:8080/FusionRegistry</code>

You will need to choose a different port if other software on the machine is already using port 8080.

* Go to <code>tomcat/conf</code> folder
* Edit <code>server.xml</code>
* Search "Connector port"
* Replace "8080" by your port number (8081 for instance)
* Restart tomcat server.

Publication Table

2023-11-08T16:22:48Z

Mnelson: /* Missing Values */

[[Category:Functions]]
[[Category:How_To]]

@See [[Publication_Table_Web_Services|Publication Table Web Services]]

= Publication Tables =
'''Introduced to Fusion Registry Enterprise v11.0.0.17'''

== Overview ==
[[File:Publication Table Definition.png|thumb|Using SDMX concepts of Agency, Id, Name, Description]]

[[File:Publication Table.png|thumb|Table Layout and Content Design]]

[[File:MaterialisedTable Preview.png|thumb|Showing the Table Preview feature which materialises the table with the given variables]]

A Publication Table is a curated table of data, where the values in the table rows, columns and table body cells can be individually specified. This enables a single table to pull data from multiple sources and piece the data together in a way which is not possible when dealing with standard cubes of data, whose table model is typically based on a Pivot model, with columns and rows built around the Dimension values to which the column or row is assigned. A Publication Table breaks out of this restrictive model by allowing cells of data to reference observations which do not need to share any relationship, in terms of dimensionality or even data source, with adjacent cells,.

Data in a Publication Table may be sourced from a single [[Dataflow_-_Structural_Metadata_Management|Dataflow]] or it may be sourced from multiple Dataflows enabling cross comparison of data.

Publication Tables can be static (every cell is a defined Observation) or dynamic, where a cell includes [[#Variable Dimension Members|variables Dimensions]] in the [[#Observation Key|Observation Key]], which can be provided at [[#Metadata vs Materialised|table build time]].

[[#Time Variables|Time arithmetic and frequency conversion]] is supported, enabling a single table to pull data from different time periods and frequencies which are all derived from the base period on which the table is built.

Publication Tables support [[#Calculated Observation Cells|calculations]], where each calculation can take input from one or more Observations over one or more [[Dataflow_-_Structural_Metadata_Management|Dataflows]].

Variable values also play a big role in Publication Tables, and includes standard variable placeholders – as well as advanced [[#Conditional Variables|conditional variables]], whose list of variable values change based on the value of another Dimension for which the table is built.

== Metadata vs Materialised ==

A Publication Table is stored in Fusion Registry the same way any other piece of structural metadata is stored. In this way it has an owner (Agency), and identity (ID) and a version. The change history is tracked using the same services, and the table definition can be [[Publication_Table_Web_Services|queried]] using the same SDMX web services for Structural Metadata.

The Publication Table metadata contains information about which Dataflows are used by the Table, which additional variables are being imported, and a definition of the table structure, and which Observations to import by referencing them by their unique [[#Observation Key|Observation Key]].

A Publication Table definition can be '''materialised''' into a table of Data by from the [[Publication_Table_Web_Services|Publication Table Web Service]], which combines the information of the Table Identity, and Variable Values to rewrite the [[#Observation Key|Observation Key]] into actual observation values. If building a web User Interface, then response from this service can be a packet of JSON, which describes the layout and content of the final Table in a way which is quick and simple to render for the User to see. The process of materialising the table is known as ‘'''build time'''’.

Additional web services are provided to enable clients to ask questions about the table such as ‘what are the earliest and latest time periods for which there is data for this table’ or ‘what possible variable values are there for this table’. This enables user interfaces to provide a user with the option to change the variables ‘on the fly’ ensuing they are able to build a table which is known to have data.

= Publication Table Cell Content =
== Overview ==
[[File:PublicationTable Layout.png|thumb|Publication Table Layout]]
A Publication Table is made up of a '''Table Head Rows''' and '''Table Body Rows'''.

The Table Head is made up of 1 or more table rows. Each Table Head cell may span 1 or more columns and 1 or more rows. The Table Head content is made up of free text and / or variable placeholders whose values are resolved when the table is built. Table Head Cells can be left empty.

The Table Body is made up of 1 or more Table Rows. The Table Rows provide the Publication Table with its data, by referencing Observations from the Dataflow.

At table build time all variables are resolved, and the corresponding data are retrieved from the data stores which hold the data.

== Table Rows ==
=== Overview ===
A Table Row consist of a Table Row heading, this can be a mix of free text and [[#Table Variables|variable text]]. The Table Row Heading is always the first Table Row Cell, and can not span multiple columns or rows. A Table Row can not have more then one heading. The remaining Table Row Cells are the Observation Cells, which may contain free text and [[#Table Variables|variable text]] but will typically contain a reference to an Observation by an [[#Observation Key|Obsevation Keys]] or a [[#Calculated Observation Cells|Calculation]] which can combine multiple [[#Observation Key|Obsevation Keys]].

A Table Row which results in no Observation data will not be included in materialised table at build time, unless the Table Row is a grouping row (see [[#Row Levels|Row Levels]] for details on grouping rows).

=== Row Levels ===
[[File:PublicationTabel Levels.png|thumb|Showing Levels in the Table Rows]]
The Table Row Heading can be used to define a row level. The level indicates to the client responsible for rendering the table the row's position in a hierarchy. Row levels are depicted using one or more leading chevrons ‘>‘ before the text. The number of hyphens indicates the level, with one leading hypen indicating level 1, two indicating level 2, and so on. It is possible to skip levels if required, i.e. Row 1 is level 1 with Row 2 assigned to level 3, skipping out level 2 entirely.

An example of row headings using levels is as follows:

Countries
> Europe
>> France
>> Germany
> Asia
>> China

Levels can be an important way to '''group data'''. A Table Row which acts as a parent row in a hierarchy, can be used as grouping mechanism for the rows that follow. This enables Parent Rows to be included in the output table, even if the Table Row contains no Observation data. The default behaviour for a Table Row with no data is to exclude it from the built table.

=== Looping Rows ===

Instead of explicitly defining a row for each Dimension member, it is possible to define a row as a loop (to iterate over all, or a subset of Dimension members). For example, if a publication table contains a Dimension for countries with an ID of REF_AREA then a loop can be set up to loop through all REF_AREA values, or a subset of the values - for which there is data. If the values are backed by a Codelist which has a hierarchy, the hierarchy can be preserved in the final table. It is also possible to nest a loop within another loop, for example for each REF_AREA output each INDICATOR.

====Syntax====
The loop is defined by giving the table row a label which starts with a syntax of :=LOOP([args]), like in the following example
:=LOOP([dimension id], [preserve hierarchy], [loop order], [explicit values to include])

'''Argument Definition'''
''dimension id'' = Required. The ID of the Dimension to loop over. For example CURRENCY would loop over all the Currency values which have data in the given context. The row label is the Currency, for example USD.
''preserve hierarchy'' = Optional, default false. If true the hierarchy of the Codes in the underlying Codelist will be built into the row hierarchy
''loop order'' = Optional, default 1. 1=Natural Order (as they appear in the Codelist), 2=By Code ID, 3=By Code Label
''explicit values to include'' = Optional, default include all values. A var arg list of values to loop over.

'''Full Example'''

:=LOOP(REF_AREA, true, 3, EUR,FR,DE,UK,ES)

====Simple Loop====
The loop is defined by giving the table row a label which starts with a syntax of :=LOOP([args]), like in the following example
:=LOOP(REF_AREA)

This loop definition will loop through all the REF_AREA values, building a row for each value. The Observation keys for the row can refer to the loop variable using a $(DIM_ID) placeholder in the Observation Key, for example:

Employed Student
:=LOOP(REF_AREA) A:$(REF_AREA):EMP A:$(REF_AREA):STU

The output table would look like the following:

Employed Student
United Kingdom 20 12
France 22 11
Germany 25 15

====Nested Loop====
If a loop row is followed by a row (or rows) which is an extra level deep, the child row(s) will be included for each loop, for example

Employed Student
> :=LOOP(REF_AREA)
>> Total A:$(REF_AREA):'''T''':EMP A:$(REF_AREA):'''T''':STU
>>> Male A:$(REF_AREA):'''M''':EMP A:$(REF_AREA):'''M''':STU
>>> Female A:$(REF_AREA):'''F''':EMP A:$(REF_AREA):'''F''':STU

Because the '''Total''' row is a child row of the LOOP (denoted by the number of leading '>') and '''Male'''/'''Female''' are children of '''Total''' - for each REF_AREA there will be a Total, Male and Female, outputting a table like the following:

Employed Student
United Kingdom
Total
Male 5 1
Female 4 2
France
Total
Male 6 2
Female 7 3

It is also possible to define a subloop as a loop. The previous example can be rewritten as:

Employed Student
> :=LOOP(REF_AREA)
>> :=LOOP(SEX) A:$(REF_AREA):'''$(SEX)''':EMP A:$(REF_AREA):'''$(SEX)''':STU

There is no limit to the number of subloops that can be defined.

====Loop Hierarchical Values====
If the underlying Codelist has a hierarchy, a subsequent argument can be provided to ask for the hierarchy to be preserved

Countries
> :=LOOP(REF_AREA, true)

'''Note:''' A hierarchy will keep rows with no data if they are acting as a parent node for rows that do have data (e.g. there may not be data for Europe, but the row exists because there are child rows of France and Germany with data).

Nested loops can also preserve hierarchies for example:

:=LOOP(REF_AREA, true)
>:=LOOP(SEX, true)

With an example output (assuming no data exists for the hierarical parents (Continent, Europe)

Continent
Europe
France
Total (M+F)
Male
Female
Germany
Total (M+F)
Male
Female

====Order Loop Rows====
It is possible to order the labels by their natural order (that of the Codelist), by id or by label, using the numbers 1, 2, or 3 respectively. The following outputs the rows ordered by label.

Countries
> :=LOOP(REF_AREA, true, 3)

====Fixed Loop List====
It is also possible to define a subset of the values to loop over. For example, if REF_AREA has data for 100 countries, but only 3 are required in the table. In this scenario, the natural order is that of the provided list, not of the underlying Codelist.

Countries
> :=LOOP(REF_AREA, true, 3, UK, FR, DE)

====Loop over External Variable====
Instead of looping the values of the Dimension's Codelist, it is possible to loop another Codelist - the purpose for this is to support alternative labels (if the labels for the publication table need to be different from those of the Dimension members) - or a subset (as an alternative to passing the subset of values into the LOOP argument), or to define a hierarchy (if the Dimension's Codelist does not have a hierarchy, or an alternative is required).

In order to loop over an external Variable, the Publication Table must be linked to an external Codelist/Valuelist against an Alias. For example EXT=My Country Codelist with Hierarchy. The corresponding loop then loops the Codes in the external list, and stores these the corresponding Dimension ID variable, for example:

Countries
> '''REF_AREA''':=LOOP('''EXT''')

Noting that the loop is over the variable EXT which resolves to an external Codelist. The loop is assigned to the REF_AREA Dimension.

=== Anchor Rows ===
The default behaviour is for a row to be omitted if the row has no data and it has no child rows with data. However, if the desired result is to always output the row, then this can be achieved by using the [anchor] tag in the row header label.

Consider the following Publication Table definition:

Male Female
Employed A:M:EMP A:F:EMP
Student A:M:STU A:F:STU
Retired A:M:RET A:F:RET

If there are no data for Student, then the materialised table would look like this

Male Female
Employed 12 13
Retired 3 5

==== Anchor Single Row [anchor] ====
To anchor a single row, add the '''[anchor]''' tag at the start of the row

Male Female
Employed A:M:EMP A:F:EMP
[anchor]Student A:M:STU A:F:STU
Retired A:M:RET A:F:RET

Then the output table would preserve the 'Student' row

Male Female
Employed 12 13
Student
Retired 3 5

This rule can be coupled with [[Publication_Table#Formatting_Missing_Observations|formatting for missing observations]], then the table could be enhanced to look something like this

Male Female
Employed 12 13
Student - -
Retired 3 5

==== Anchor Child Rows [anchor+] ====
For a row with a hierarchy, an '''[anchor+]''' tag can be placed on a parent node to anchor the parent and all of its children, for example:

Continent
[anchor+]>Europe
>>France
>>>Total (M+F)
>>>>Male
>>>>Female
>>Germany
>>>Total (M+F)
>>>>Male
>>>>Female

The above rule would always output Europe and all of its descendants (France, Total, Male, Female, Germany).

==== Anchor Siblings and Child Rows [anchor*]====
The '''[anchor*]''' tag can be used as a way to anchor all following siblings and child nodes, this can be a useful mechanism to anchor all rows for the whole table by placing the anchor[*] on the first row of the table.

==== Anchor Loops ====
If a loop is preceeded by an '''[anchor]''' tag then the system will have the following behaviour:

* If an external Codelist variable is being looped, all codes will be ouput regardless of what data is stored in the system
* If a Dimension is being looped, all codes in the Codelist used by the Dimension will be output regardless of what data is stored in the system
* If a Dimension is being looped and it is not coded (i.e it is free text or numerical or other) then the [anchor] has no effect, as the looped values are based on what data exists

== Observation Key ==
=== Overview ===
The purpose of the Observation Key is to reference a Observation whose value will be resolved at table build time. The Observation Key can explicitly resolve to 1 observation value, if all parts of the Key are provided. However, it is possible to make parts of the key varaibles, allowing the values to be set at table build time.

An Observation Key is made up of 4 parts:

* Dataflow Reference
* Series Reference
* Time Period Reference
* Measure Reference

An example Observation Key with all of these parameters in place is:

DF_REG.M:UK:SUR:2008:BIRTHS

The key is broken down into the following parts:

{| class="wikitable"
|-
! Observation Key Part !! Name !! Description !! Required
|-
| DF_REG|| Dataflow Alias|| This identifies the Dataflow which has the data. The identification uses the alias, which is assiged to the Dataflow in the Publication Table definition. The Dataflow Alias is only required if the Publication Table references more then one Dataflow. If the Publication Table references more then one Dataflow, it can assign a Dataflow to a Default Alias, which means that it is required as part of the Observation Key. || No (if only 1 Dataflow or a default alias is set)
|-
| M:UK:SUR || Series Key || This identifies the Series of Data for the Dataflow, by providing a single value for each of the Dimension Members of the Dataflow, separated by a ‘:’ character. In this example the Dataflow has 3 Dimensions (excluding time), the value for the first is 'M', the second 'UK' and the last 'SUR'. As the Dataflow in this example has a Time Dimension this Series Key will uniquely identify a Series of Observations where each Observation refers to a unique Time Period || Yes (but can omit values)
|-
| 2008 || Time Period || This identifies the Time Period in the Series of Data identified by the previous section. The Combination of the Series and Time Period for a Time Series Dataflow will uniquely identify the Observation || No. Omit to make it variable for Time Series Data (defaults to last period). Not required if there is no Time Dimension for the DSD.
|-
| BIRTHS || Measure Dimension || If the Dataflow uses a Data Structure with multiple Measures, then the Measure ID must be included as the last part of the Observation Key. In this example the Dataflow has multiple measures (BIRTHS/DEATHS/MARRIAGES) so the Observation Key needs to include this value. || When there is only 1 measure it is not required, when there are multiple measures, it is required.
|}

Observation Keys are included on the materialised table in JSON format enabling the observation value to be tied back to the key. Observation Keys also provide a mechanism by which [[#Formatting_Cells|formatting rules]] can be applied to a subset of table cells.

=== Variable Dimension Members ===
It is possible to omit Dimension Values from the Observation Key, when this is the case that Dimension will become a variable – whose value can be set at table build time. For example, the second Dimension value of ‘UK’, from the previous example, can be omitted like this:

M::SUR:2008

The Publication Table can now be built for any Country by passing in the variable at table build time - i.e. UK, FR, DE. By passing in different variable values, the Observation Key will change, and as such it will reference a different Series in the Dataset.

When an Observation Key has a variable Dimension member, the ID of the Dimension can be referenced in text cells to create variable headings. For example, if the ID of the 2nd Dimension in the DSD is '''REF_AREA''', the Table Column could be given a title of $(REF_AREA) which would resolve to 'United Kingdom', 'France', or 'Germany' at build time depending on which varaible value is used to build the table. [[#Dimension Variables|#See Dimension Variables]].

=== Variable Time Period ===
For Time Series dataset, the Time Dimension can form part of the Observation Key, for example M:UK:SUR:'''2008'''. The 2008 part of the key refers to the year 2008.

It is possible to omit the value for Time Period in the Observation Key. Omitting Time Period will make the Time value a variable which can be provided at table build time, defaulting to the last period for which data exists if no value is supplied.

Extending the above example to omit the Time value results in an Observation Key whose Country and Time values can be provided to the web service at table build time.

M::SUR

'''Note:''' The above key is exactly the same as using the variable Time Period value 'P' in the Time Dimension, as follows:

M::SUR:'''P'''

Additional functions can be performed on the Time Dimension, such as moving forwards or backwards from the table base period, [[#Time Variables|#see Time Variables]] for more details.

=== Free Text Observation Keys ===
Whilst typically a cell in the body of the Publication Table would resolve to an Observation value, it is possible to use it for free text. To include free text cells, start the cell value with a leading apostrophe character, for example:

'Cell Text

'''Note''': An Observation Key can '''not''' contain a mix of free text as well as an Observation value

== Calculated Observation Cells ==
[[File:TableDefinition Calculations Variables.png|thumb|Publication Table with Calculated Cells and Variable Dimensions]]

An Observation Cell may contain calculated values, which can consist of a mix of simple algebra (+, -, *, /) and a reference to one or more Observations which may come from multiple Dataflows.

To define a Calculated Observation the Cell the following rules apply:

* The Observation Key must start with :=
* Any Observation Keys must appear within enclosing square brackets '[' and ']'
* An Observation key may embed calculations on a single Dimension by using parenthesis '(' and ')' in the key part
* Numbers in Observation Keys are treated as Code IDs, unless prefixed with a '#'
* The same rules for variable Dimensions and Time Period apply when resolving the Observation Keys used in the calculation

Examples are shown in the table below

{| class="wikitable"
|-
! Example Syntax !! Description
|-
| :=[A:UK:EMP]+10 || Add 10 to the observation value retrieved from the key A:UK:EMP
|-
| :=[A:(UK+FR+DE):EMP] || Add the series A:UK:EMP + A:FR:EMP + A:DE:EMP
|-
| :=[A:(UK+FR+12):EMP] || Add the series A:UK:EMP + A:FR:EMP + A:DE:12
|-
| :=[A:(UK+FR+#12):EMP] || Add the series A:UK:EMP + A:FR:EMP and add the Constant '12'
|-
| :=[A:UK:EMP]+[A:FR:EMP]+[A:DE:EMP] || Add the series A:UK:EMP + A:FR:EMP + A:DE:EMP
|-
| :=[A:(UK+FR):EMP]/[A:(UK+FR):UMP] || Adds the series A:UK:EMP + A:FR:EMP and divides by the result of A:UK:UMP + A:FR:UMP
|-
| :=((100/[A:UK:EMP:(M+F):P-1])*[A:UK:EMP:(M+F)])-100 || Demonstrates the ability to use parenthesis outside of the Observation Keys to control the order of mathematical operations
|-
| :=[POP.M:UK:(M+F)] / [LAB.M:UK:EMP] || Demonstrates the ability to calculate data from two Dataflows (defined by alias POP and LAB)
|-
| :=[M:UK:EMP:M(P)] + [A:UK:EMP:A(P)] || Demonstrates the ability to calculate data where the period is converted from the base period
|}

=== Missing Values ===
If any input Observations are missing in the calculation, the final result of the calculation is an empty value. For example:

:=[A:(UK+FR+DE):EMP]

This would not output a value if A:UK:EMP, A:FR:EMP or A:DE:EMP have no corresponding Observation value for the time period the table is being built for.

RFS version 11.12.0 and higher allow for assignment of default value(s) to missing components of the calculation.

The syntax uses the double pipe '||' symbol, the semantic meaning is 'use this value || or this value if missing'.
:=[A:UK:EMP||0] + [A:FR:EMP||0] + [A:DE:EMP||0]

The above expression substitutes each component with a zero value if the corresponding Observation is missing.

When using inline expressions, i.e. A:UK+FR+DE:EMP, the single pipe '|' can be used after each value in the expresion, for example:

:=[A:(UK|1+FR|1+DE|0):EMP]

This expression substitutes the UK for the value of 1 if there is no corresponding Observation for A:UK:EMP. The Observation A:FR:EMP is substituted for 2 if missing, and A:DE:EMP is substituted for 0. Therefore is all 3 Observations are missing, the final value for the expression = 1+1+0 = 2.

It is also possible to substitute the entire block when using inline expressions. The following example results in a zero value if any input Observation (UK, FR, DE) value is missing:

:=[A:(UK+FR+DE):EMP||0]

=== Calculated Cell Observation Key ===
Calculated Cells have implicit [[#Observation_Key|Observation Keys]] which are derived based on the inputs series in the calculation. The [[#Observation_Key|Observation Key]] can be used to assign [[#Formatting_Cells|Formatting Rules]] in the same way as a 'standard key' can be used. It is possible to assign an explicit key to the calculation, which will override the implicit key assigned. The following table shows how the key assignment works.

For example
{| class="wikitable"
|-
! Calculation !! Output Key || Explanation
|-
|:=[A:(UK+FR+DE):EMP] || A:#:EMP:2009 || The first and last dimension have a fixed value, the second dimension has a variable value and is replaced by '#'. The time period is added to the end of the key.
|-
|:=((100/[A:UK:EMP:(M+F):P-1])*[A:UK:EMP:(M+F)])-100 || A:UK:EMP:#:# || The first three dimensions have a fixed value, the last dimension has a variable value and is replaced by '#'. The time period is a variable as there are 2 time inputs to the calculation (P and P-1)
|-
|:=PCTOF([A:M:UK:P],[A:TOT:UK:P]) || A:#:UK:2009 || The first and last dimension have a fixed value, the second dimension has a variable value and is replaced by '#'. The time period is added to the end of the key.
|-
|:=[A:UK:EMP]+10 || A:UK:EMP || The input key is the same as the output key as none of the dimensions have a variable value. The time period is added to the end of the key.
|-
|A:UK10:EMP:=[A:UK:EMP]+10 || A:UK10:EMP:2009 || The output key is explicitly assigned. The time period is added to the end of the key.
|}

The output key conforms to the same Dimension breakdown as the underlying Data Structure. If a Observation Key is explicitly assigned, it must also honour the same Dimension breakdown. This allows formatting rules to be applied using the key component syntax, for example COUNTRY=UK10 would search for Observation Keys whose Country Dimension is reporting the value UK10.

== Percentage Of ==
The PCTOF (Percentage Of) function can be used to calculate 'x' as a percentage of 'y' following the syntax:

PCTOF(x,y)

This is reolved to the following mathmatical expression: '''(100/y) * x'''

x and y must either be a numerical value, and observation key, or an expression using a combination of the two. For example

:=PCTOF([A:M:UK:P],[A:TOT:UK:P])

The Observation Key follows the same standard rules with respect to varaibles and inline calculations - the following example shows an Observation Key where Male + Female values are added with the result being used in the Percentage Of function

:=PCTOF([A:M:UK],[A:('''M+F'''):UK])

A more complex example shows calculations around the PCTOF function:
:=(10 + PCTOF([A:M:UK:A(P)-1],[A:(M+F):UK:A(P)])) / 2

== Footnotes ==

[[File:Footnotes3.png|thumb|A table with footnotes]]
[[File:Footnotes2.png|thumb|Footnotes in the table builder]]

=== Overview ===
Footnotes can be added to any part of the publication table (table column/row header, or observation cells). The output table is generated with a footnote number, with the list of footnotes at the end of the table.

Footnotes need to be thought about in 2 parts: Part 1 are the list of footnotes which are defined at the Publication Table level. Part 2 are the references to the footnotes from within the table. This separation of allows footnotes to be reused within the same table, and when working with [[#Footnotes_with_Placeholders|placeholders]] footnote templates can be set up where the text changes based on the placeholder values defined in the table builder.

To add a footnote, add it to the Publication Table, for example: 
Footnote 1 
Footnote 2

Then in the table builder, add a reference to which footnote to use, $(1) references '''Footnote 1''', and $(2) references '''Footnote 2'''.

'''Note:''' The order of the list of footnotes in the table definition may not be reflected in the order they are output. As such a footnote referenced by $(1) may be output as the second footnote.

=== Footnotes with Variable ===
A footnote can reference variables, using the [[#Table_Variables|variable syntax]],

A footnote for $(REF_AREA.UK)

A footnote can even output an observation value, observation attribute or series attribute.

Measure Value
A footnote with obs value $(A:UK:EMP:2008:OBS_VALUE)

Observation Attribute
A footnote with obs attribute $(A:UK:EMP:2008:OBS_COMMENT)

Series Attribute
A footnote with obs attribute $(A:UK:EMP:P:SERIES_TITLE)

When combining variable footnotes with placeholder values and loops, it is possible to dynamically create references to observation attributes.

'''Note:''' the use of P in the final cell key which is a reference to the [[#Time_Variables|time period]] for which the table is being built.

=== Footnotes with Placeholders ===
It is possible to write a generic footnote for the publication table, which includes placeholders. When the footnote is referenced, the placeholder values are provided. The syntax for a placeholder is {n} where n is a positive integer, for example {1}.

Footnote Example

A footnote for $(REF_AREA.{1}) and $(REF_AREA.{2})

In the table builder, the footnote is referenced as usual, but with arguments for the placeholders:

Footnote reference in Table builder

My Table Row '''$(1,UK,FR)'''

The footnote reference in the above example is in bold, and consists of which footnote text to use (in this case the first one in the list, denoted by '1'), and 2 arguments are passed for the placeholders, placeholder 1 takes the value of UK and placeholder 2 takes the value of FR. During the materialise process the intermediate text is generated:

A footnote for $(REF_AREA.UK) and $(REF_AREA.FR)

Which then goes through the variable resolve process, in this case both variables, '''$(REF_AREA.UK) and $(REF_AREA.FR)''', match the [[#Coded_Variables|Coded Variable]] syntax and would resolve to:

Final Output
A footnote for United Kingdom and France

=== Footnotes in Observation Cell ===
Any number of footnotes can appear in an Observation cell, simply put the footnote(s) at the end of the [[#Observation_Key|Observation Key]].

Standard Footnote Reference 
DF_REG.M:UK:SUR:2008:BIRTHS $(1) $(2)

Footnote with Placeholder 
DF_REG.M:UK:SUR:2008:BIRTHS $(1,UK)

=== Footnotes in Loops ===
A row [[#Looping_Rows|loop]] contains a locally scoped variable, and as such it can be used when calling a footnote with a placeholder.

Example Footnotes
$(A:T:{1}:EMP:P:SERIES_TITLE)
$(A:T:{1}:EMP:P:TITLE_COMPL)
$(A:{1}:{2}:EMP:P:OBS_COMMENT)

Example Table Definition
Employed Student
> :=LOOP(REF_AREA) '''$(1,$(REF_AREA[id]))'''
>> Total '''$(2,$(REF_AREA[id]))''' A:$(REF_AREA):T:EMP A:$(REF_AREA):T:STU
>> :=LOOP(SEX) '''$(3,$(REF_AREA[id]),$(SEX[id]))''' A:$(REF_AREA):$(SEX):EMP A:$(REF_AREA):$(SEX):STU

Footnote Intermediate Resolution
Row 1 (REF_AREA=UK)
$(A:T:UK:EMP:P:SERIES_TITLE)

Row 2 (REF_AREA=UK)
$(A:T:UK:EMP:P:TITLE_COMPL)

Row 3 (REF_AREA=UK, SEX=M)
$(A:UK:M:EMP:P:OBS_COMMENT)

Final Footnote output
Row 1 (REF_AREA=UK)
This is a series for the United Kingdom

Row 2 (REF_AREA=UK)
This is a title compliment for the United Kingdom

Row 3 (REF_AREA=UK, SEX=M)
This is an observation comment for Males in the United Kingdom

= Formatting Cells =
== Formatting Numerical Observation Values ==
[[File:Obs formatting.png|thumb|Number Formatting Options]]
Numerical observation values can be formatted in the following way

# '''Number of Decimal Places''' or '''Significant Figures''' can be defined
# For Decimal places this can be set to a maximum rule or a fixed rule, i.e the value 1.3 to 2 decimal places is 1.3 using the maximum rule and 1.30 using the fixed rule
# '''Rounding mode''' can be specified
# '''Scaling Factor''' this is a multiplier applied to the observation value, the multiplier is x10^[scaling factor]. Example, a value of 1.1 with a scaling factor of 1 is 11, a scaling factor of 2 is 110, a scaling factor of 3 is 1100. If the Observation or related Series has a UNIT_MULT concept the difference between the unit multiplier and scaling factor will be taken. Example, an Observation with a value of 1.1 and a unit multiplier of 9 (Billions) is reported, the publication table has a scaling factor of 6 (millions) - the number 1.1 is scaled using 10^3 (calculated by 9-6), the 1.1 billion is converted to 1100 million.
# '''Scientific notation''' can be requested on output, i.e 123456 to 2 significant figures is 120000 in standard form or 1.2E4 in scientific form
# '''Decimal formatting''' for long numbers, i.e. 12345.8 there is not thousands separator by default, to include a thousand separator set the decimal formatting rule to either a single Language (i.e. French Formatting) or Dynamic formatting, which uses the Accept-Langauge locale/language sent be the web browser (this can be overridden using the locale=[loc] query parameter. The formatting rules can also influence which characters the numerical characters used.
# Rules can be applied to all series, or specific series

=== Match Series ===
When creating a formatting rule, it is possible to define which series the rule applies to. If left blank, the rule will apply to all observation values.

The syntax for defining specific series is shown in the following table:

{| class="wikitable"
|-
! Rule Type !! Syntax !! Description
|-
| Series Match || =A:UK:EMP || Match a specific series from the default Dataflow
|-
| Series Match Multiple Values || =A:UK+FR+DE:EMP || Multiple codes in 2nd Dimension (UK, FR and DE) any can match for the rule to be valid
|-
| Series Match by Dataflow Alias || =FLOW2.A:UK:EMP || Match a specific series for the Dataflow with alias FLOW2
|-
| Wildcard Series Match || =A::EMP || Match all series with the given dimension values (2nd Dimension is wildcarded)
|-
| Not Rule || !=A:UK:EMP || Match all series which do not match this series (wildcard series also supported)
|-
| Dimension Rule || CURRENCY=GBP || Match all series which have CURRENCY of GBP
|-
| Not Dimension Rule || !CURRENCY=GBP || Match all series do not have a CURRENCY of GBP
|-
| Multiple Rules || CURRENCY=GBP,FREQ=M || Comma is used to include multiple matches (also applies to series match)
|}

== Formatting Missing Observations ==

Observation Values can be missing for 2 reasons:
# There is no Observation in the database for the given Observation Key
# There is an Observation in the database but the measure value is either: NaN, null, or an empty string

In both cases it is possible to define an output value using formatting rules. For the first case, the observation does not exist, it is possible to define a default output value, for example ‘-‘. For the second case, where the observation does exist, it is possible to define a set of rules based on another Component of the Observation, for example the Observation output may be related to the OBS_STATUS Attribute.

When the formatted value is based on a Component, a set of rules must be defined which state ‘when component=x output y’, where x may be an fixed value or a pattern to match (using a regular expression). The rules for mapping component values are defined in a [https://fmrwiki.sdmxcloud.org/Representation_Map Representation Map], and the Publication Table simply provides the connection to the Representation Map to use. This enables reuse of the mapping rules across multiple Publication Tables. The Publication Table must define two properties in order to support Component Mapping rules:

# The Component to obtain the value from i.e OBS_STATUS
# A link to the [https://fmrwiki.sdmxcloud.org/Representation_Map Representation Map], which holds the mapping rules. Note, the Representation Map is expected to have 1 source mapping, this could be of any type (string, codelist, other)

An example Reprsentation Map for OBS_STATUS is shown below:

{| class="wikitable"
|-
! OBS_STATUS !! Is Reg Ex !! Missing Value !! Description
|-
| M || false || mv || When OBS_STATUS is M output mv
|-
| L || false || np || When OBS_STATUS is L output np
|-
| .* || true || - || When OBS_STATUS is anything else output -
|}

== Frequency Formatting ==
[[File:PublicationTable FreqMap.png|thumb|Representation Map with frequency mapping rules]]
[[File:PublicationTable Materialised FreqMap.png|thumb|Materialised table with formatted time headings ]]
Time Variables always resolve themselves to an [[SDMX_Time_Formats|SDMX Date Format]] when forming the [[#Observation Key|Observation Key]], this ensures the correct observation value is returned from the data store. When a time variable is used as a label, in a table heading for example, the same formatting will be applied by default but a custom format can be provided. Custom display formats for a Time Period enable a time period such as 2007-Q1 to be resolved to custom text string, for example '31 Mar 2007'.

Custom Time formats rules are defined by providing an association between an SDMX Frequency ID (#see [[SDMX_Time_Formats|SDMX Date Format]]) and the output format template. The output format template is a string that defines which components of time are output an in which order and style. For example to output a date as 'March 2001' the format template would look like this 'MMMM yyyy' to indicate a full month name (MMMM) and a 4 character year (yyyy). format rules make use of the [https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html Simple Date Format] which is used by Java as such all of the documentation on Simple Date Format applies.

Frequency Format Mapping is not defined as part of a Publication Table, instead the rules are defined in a [https://fmrwiki.sdmxcloud.org/Representation_Map Representation Map] and the Publication Table references the Representation Map as the source of its frequency conversion rules. This enables the same set of rules to be used for multiple Publication Tables. The Representation Map must map 1 source (Codelist or String) to 1 target (String), it can contain as many mapped frequencies as required. If the Representation Map contains a mapping for a value that is not a valid SDMX Frequency, then it will be ignored.

It is important to note that all time periods resolve to a single point in time, which is the end of the period, for example an Annual period of 2008 resolves to 2008-12-31:T23:59:59. The resolution of frequency to a point in time enables the frequency formatting rules to use all aspects of time, even when converting a low frequency, for example 2008 formatted using the pattern dd MMM yy would resolve to '31 Dec 2008'.

== Variables Summary ==

{| class="wikitable"
|-
! Variable Type !! Description !! Syntax !! Example !! Example Output
|-
| '''Dataflow Name''' || Displays the name of a Dataflow referenced by the Publication Table in the locale in which the Publication Table is built || $([dataflow alias]) || $(EDU) || World Bank, Education
|-
| '''Dataflow Description''' || Displays the description of a Dataflow referenced by the Publication Table in the locale in which the Publication Table is built || $([dataflow alias].desc) || $(EDU.desc) || World Bank, Education
|-
| '''Dataflow Id''' || Displays the ID of a Dataflow referenced by the Publication Table || $([dataflow alias].id) || $(EDU.id) || DF_EDU
|-
| '''Variable Dimension Value''' || Displays the name of the variable used when an Observation key has a wildcard member || $([dimension id]) || $(REF_AREA) || United Kingdom
|-
| '''Specific Component Value''' || Displays the name of the Component Value || $([component id].[component value]) || $(OBS_STATUS.C) || Confidential
|-
| '''Component Name''' || Displays the name of the DSD Component (this is the name of the Concept used by the Component) in the locale in which the Publication Table is built || $([component id].name) || $(OBS_STATUS) || Confidentiality Status
|-
| ''' Enumeration Value ''' || Displays the name of a Code in a Codelist or Value in a Valuelist || $([list alias].[list item ID] || $(SEX.M) || Male
|-
| '''Conditional Variable''' || Displays the variable value, which is conditional on the value of a variable Dimension || $([conditional variable id][variable id]) || $(BANKNOTES[NOTE1]) || £5
|-
| ''' Time ''' || Displays the base time period for which the table is built, or an offset from this time period / cast to a different frequency || see [[#Time Functions|Time Functions]] || $(P) || 2008
|}

= Table Variables =
== Overview ==
Variables are a very important part of Publication Tables. They can be used to make the content of the table dynamic, setting and changing the value at run time to change the table content. Variables can be used in Observation Keys, but the values of variables can also be used anywhere where there is text, for example the table heading, subheading, column or row headings, footnotes, and Observation Cells with free text.

The general syntax for a variable is

$(''variable id''.''sub part''[''detail''])

Where variable id is always required, the subpart and detail are optional. Subpart is used if the variable resolves to a Component and a coded value is required, the detail array part obtains the id, name, or description of the underlying SDMX type.

This section describes the various variables that exist, along with the syntax of how to use them.

== Dataflow Variables ==
Publication Tables define data cells across one or more Dataflows. Each ‘imported’ Dataflow is given an alias, which can be used to refer to the Dataflow in the [[#Observation Key|Observation Key]].

It is possible to refer to the Dataflow's Name, Description, and Id using the following variable syntax.

'''Default Dataflow''' 
Dataflow Id
$([id])

Dataflow Name
$() or $([name))

Dataflow Description
$([desc])

'''Dataflow with Alias=EXR''' 
Dataflow Id
$(EXR[id])

Dataflow Name
$(EXR) or $(EXR[name))

Dataflow Description
$(EXR[desc])

== Dimension Variables ==
When an [[#Observation Key|Observation Key]] has [[#Variable Dimension Members|variable Dimension Values]], the table will be built by replacing this variable with a value to complete the Observation Key. The variable value is either passed explicitly to the web service, or defaulted by the system if not provided. The Dimension Variable can be referenced in the table using the syntax $([dimension id]), for example:

Dimension Value Name
$(INDICATOR) resolves to 'Employment'

This would resolve to the '''name''' of the Indicator value used in the table, for example '''EMP''' may resolve to the label '''Employment'''. The label for the Dimension Value is obtained from the Codelist or Valuelist which the Dimension uses. If the Dimension does not use a list (i.e. it is free text or numerical) then the Dimension value is simply used in place of the variable placeholder.

To resolve to the '''id''' of the value for the Component, for example '''EMP''', use the following syntax

Dimension Value Id
$(INDICATOR[id]) resolves to 'EMP'

== Dependent Dimension Variable ==
It is possible that the value for one Dimension is dependent on the value of another Dimension, for example CURRENCY may be dependent on the COUNTRY and COUNTRY is a variable in the [[#Observation Key|Observation Key]] - it is possible to specify that the value to use for the CURRENCY Dimension can be derived using a lookup table. The lookup table in this instance is maintained as an [https://fmrwiki.sdmxcloud.org/Representation_Map SDMX Representation Map] which is used to map COUNTRY to CURRENCY. The Publication Table specifies that the two Dimensions are related and provides a reference to the Representation Map to use which has the rules.

When working with a dependent variable, both the dependent variable and the Dimension on which it depends should be omitted from the [[#Observation Key|Observation Key]] - for example consider a full key of:

A:UK:GBP:2007

Where UK is the COUNTRY and GBP is the Currency, to make the COUNTRY a variable Dimension, it should be omitted from the key, as follows:

A::GBP:2007

To then make the CURRENCY a variable it is also omitted , as follows:

A:::2007

The Publication Table is given the rule that CURRENCY depends on COUNTRY and should use the COUNTRY_2_CURRENCY [https://fmrwiki.sdmxcloud.org/Representation_Map Representation Map], which in this example looks like the following:

{| class="wikitable"
|-
! COUNTRY !! Is Reg Ex !! CURRENCY
|-
| UK || false || GBP
|-
| US || false || USD
|-
| .* || true || EUR
|}

When the Publication Table is materialised the [[Publication_Table_Web_Service|web service]] only requires the COUNTRY information, and it will complete the Observation key using the representation map lookup table.

'''Note''' it is possible to override the mapped value by passing in an explicit override to the web service. For example passing to the web service both COUNTRY=UK with CURRENCY=EUR will override the mapping behaviour for CURRENCY.

== Component Values ==
Components of the underlying Data Structure can be explicitly referenced using the following syntax '''$([component id])''', for example:

Component Name
$(OBS_STATUS) or $(OBS_STATUS[name]) resolves to 'Observation Status'

Component Description
$(OBS_STATUS[desc]) resolves to 'Information on the quality of a value or an unusual or missing value.'

== Coded Variables ==
To reference a Code value, the syntax is $([component id / codelist alias].[item id]). The first variable part is the ID of a Component in the underlying Data Structure, or alternativly the alias of an external variable defined on the Publication Table. An external variable is a link to an SDMX Codelists or Valuelist. Example are:

Name of Code
$(CURRENCY.$) resolving to the text '''Dollar''

Description of Code
$(DWELLING.SM[desc])

== Conditional Variables ==
A Conditional Variable is one whose value is conditional on the value of a [[#Dimension Variables|Dimension Variable]].

An example use case is Banknotes, where the values for NOTE1, NOTE2, or NOTE3 is dependant on the Country Dimension. A Conditional Variable could be used to set a values for each variable (NOTE1, NOTE2, NOTE3) based on the value used for the current Country Dimension i.e. US NOTE1=$5, UK NOTE1=£5.

The syntax of referring to a Conditional Variable is $([conditional variable id][variable id]), for example
$(BANKNOTE[NOTE1])

== Reported Attribute for an Observation or Series ==
A variable can be a reference to a part of an Observation, for example the Unit Multiplier (UNIT_MULT). A Table Row header, for example, could include variable text such as 'Number of payment accounts (''thousands'')' where the word '''thousands''' is a variable because it depends on other variables which build the key.

To achieve this, use the syntax:

$(A:UK::2008:UNIT_MULT[name])

The observation key in this case is A:UK::2008, where the third Dimension is a variable as it has no value (in the main body of the table this dimension would probably also be a variable, which means the table can be built with different values, each value may change the output of this UNIT_MULT value). The last part of the key is the Attribute ID (UNIT_MULT) - this can be a reference to a series attribute, or an observation attribute. The [name] indicates that it should resolve to the name of the reported attribute value, alternative values are '''id''' for the reported value or '''desc''' for the description of the reported value.

If time is also a variable, then the key can change accordingly, using the variable syntax for time (P):

$(A:UK::P:UNIT_MULT[name])

== Time Variables ==
[[File:PublicationTable MovingTime.png|thumb|Publication Table Design, moving and converting the base time period]]

[[File:PublicationTable Materialised MovingTime.png|thumb|Moving and converting periods - showing the materialised table]]

When the data is time series (i.e. the [[Data_Structure_Definition_(DSD)_-_Structural_Metadata_Management|DSD]] has a Time Dimension), the [[#Observation_Key|Observation Key]] can include the time period, for example:

A:UK:EMP:'''2008'''

Where 2008 is the Observation Time Period to use. '''The format of the Time Period must always be the same format as the frequency for which the data are reported'''. For example if the data is annual, and an observation is reported for 2008, then the observation key must use 2008, not another frequency such as 2008-12-31. It is typical in a [[Data_Structure_Definition_(DSD)_-_Structural_Metadata_Management|DSD]] for the Frequency of the Series to be the first Dimension, in all the examples on this page this convention is applied, and as such the above key A:UK:EMP refers to Annual data, depicted by the initial dimension value 'A'.

If the key omits the time period, then the '''time period becomes a variable''' which can be set at table build time, if not set it defaults to the last period for which there is data.

A:UK:EMP

When a Publication Table has a variable time period, the resolved variable time value for this varaible is known as the '''base period'''. There can only be 1 base period for a Publication Table, however other Observation Keys (and corresponding headings) can perform time arithmetic from the base period for which the table is built.

The supported time functions include walking forwards and backwards in increments of the given frequency, converting to another frequency of data, and a combination of the two (convert and move, move and convert, move convert and move again). Ths supported functions are shown in the table below.

{| class="wikitable"
|-
! Syntax !! Description !! Example Observation Key !! Example Resolved Key
|-
| P || The base period (default if not provided) || A:UK:EMP:P || A:UK:EMP:2008 (base period=2008)
|-
| P-''n'' || Base Period minus 'n' Periods where n is a positive Integer || A:UK:EMP:P-1 || A:UK:EMP:2007 (base period=2008)
|-
| P+''n'' || Base Period plus 'n' Periods where n is a positive Integer || A:UK:EMP:P+1 || A:UK:EMP:2008-Q2 (base period=2008-Q1)
|-
| A(P) || Base Period converted to '''Annual''' || A:UK:EMP:A(P) || A:UK:EMP:2008 (base period=2008-Q2)
|-
| S(P) || Base Period converted to '''Bi-Annual''' (Semester) || S:UK:EMP:S(P) || S:UK:EMP:2008-S2 (base period=2008-Q3)
|-
| T(P) || Base Period converted to '''Tri-Annual''' || T:UK:EMP:T(P) || T:UK:EMP:2008-T3 (base period=2008)
|-
| Q(P) || Base Period converted to '''Quarterly''' || Q:UK:EMP:Q(P) || Q:UK:EMP:2008-Q4 (base period=2008)
|-
| M(P) || Base Period converted to '''Monthly''' || M:UK:EMP:M(P) || M:UK:EMP:2008-03 (base period=2008-Q1)
|-
| W(P) || Base Period converted to '''Weekly''' || W:UK:EMP:W(P) || W:UK:EMP:2008-04 (base period=2008-01-01)
|-
| D(P) || Base Period converted to '''Daily''' || D:UK:EMP:D(P) || D:UK:EMP:2008-12-31 (base period=2008)
|-
| H(P) || Base Period converted to '''Hourly''' || H:UK:EMP:D(P) || H:UK:EMP:2008-12-31T12 (base period=2008-02-02T12:00:03)
|-
| A(P-n) || Base Period minus (or plus) 'n' periods, converted to another frequency 'A' || A:UK:EMP:A(P-4) || A:UK:EMP:2007 (base period=2008-Q4)
|-
| A(P)-n || Base Period, converted to another frequency 'A' minus (or plus) n' periods || A:UK:EMP:A(P)-4 || A:UK:EMP:2004 (base period=2008-Q4)
|-
| A(P-n1)-n2 || Base Period, minus (or plus) 'n1' periods, converted to another frequency 'A' minus (or plus) 'n2' periods || A:UK:EMP:A(P-4)-4 || A:UK:EMP:2003 (base period=2008-Q4)
|-
| FA(P) || Subtracts 1 Period from the Converted Period IF the resolved Convert From end of period does not match the same point in time as the Convert To period.

'''Examples:'''
* Base Period =2008-Q4 Rule= FA(P) Output = 2008
* Base Period =2008-Q3 Rule= FA(P) Output = 2007 - the base period 2008-Q3 is not the same time as 2008 (end of period, so move back 1 year)
* Base Period =2008-Q1 Rule= FA(P) Output = 2007
* Base Period =2008-Q1 Rule= FA(P+3) Output = 2008
* Base Period =2008-Q1 Rule= FA(P+3)-1 Output = 2007

This works for ALL frequencies where the convert to frequency is lower then the base Period P, for example Quarter converted to Annual.

|| A:UK:EMP:FA(P) || A:UK:EMP:2007 (base period=2008-Q3)
|}

'''Compound expressions''' 
From Fusion Registry v11.5.1 and later it is possible to cast a period more the once in the same expression, for example

Q(FA(P-1))-1

If the input time period P=2019-Q4, then P-1 = 2019-Q3, the expression is rewritten as Q(FA(2019-Q3))-1. The Full Annual (FA) for 2019-Q3 is 2018, as the end of period 2019 does not match the end of period 2019-Q3, so the annual period moves back an additional period. The expression is rewritten as Q(2018)-1. 2018 to Quarterly moves to 2018-Q4 and then the last part of the expression steps back 1 period, the final period is 2018-Q3.

'''Notes on Time Period''' 
When converting from a lower frequency to a higher frequency (Weekly to Daily for example), the cast date always resolves to the end of the Period.

It is also important to note that as the [[#Observation Key|Observation Keys]] in these examples all have Frequency as the first Dimension, when converting to another frequency (Quarterly to Annual for example) the Observation Key changes the value for the Frequency Dimension, ensuring the correct series of data is retrieved.

'''Time Period variable in free text''' 
When refering to the Time Period as a variable in free text, for example in the table heading or a column heading, the variable is placed inside the variable syntax '''$()'''. For example $(P) would resolve to base period, and $(A(P)) would resolve to 'base period converted to annual frequency'. An example usage is shown below:

Table 1A showing data for $(P) to $(P+3)

Publication Table

2023-11-08T16:07:57Z

Mnelson: /* Calculated Cell Missing Values */

[[Category:Functions]]
[[Category:How_To]]

@See [[Publication_Table_Web_Services|Publication Table Web Services]]

= Publication Tables =
'''Introduced to Fusion Registry Enterprise v11.0.0.17'''

== Overview ==
[[File:Publication Table Definition.png|thumb|Using SDMX concepts of Agency, Id, Name, Description]]

[[File:Publication Table.png|thumb|Table Layout and Content Design]]

[[File:MaterialisedTable Preview.png|thumb|Showing the Table Preview feature which materialises the table with the given variables]]

A Publication Table is a curated table of data, where the values in the table rows, columns and table body cells can be individually specified. This enables a single table to pull data from multiple sources and piece the data together in a way which is not possible when dealing with standard cubes of data, whose table model is typically based on a Pivot model, with columns and rows built around the Dimension values to which the column or row is assigned. A Publication Table breaks out of this restrictive model by allowing cells of data to reference observations which do not need to share any relationship, in terms of dimensionality or even data source, with adjacent cells,.

Data in a Publication Table may be sourced from a single [[Dataflow_-_Structural_Metadata_Management|Dataflow]] or it may be sourced from multiple Dataflows enabling cross comparison of data.

Publication Tables can be static (every cell is a defined Observation) or dynamic, where a cell includes [[#Variable Dimension Members|variables Dimensions]] in the [[#Observation Key|Observation Key]], which can be provided at [[#Metadata vs Materialised|table build time]].

[[#Time Variables|Time arithmetic and frequency conversion]] is supported, enabling a single table to pull data from different time periods and frequencies which are all derived from the base period on which the table is built.

Publication Tables support [[#Calculated Observation Cells|calculations]], where each calculation can take input from one or more Observations over one or more [[Dataflow_-_Structural_Metadata_Management|Dataflows]].

Variable values also play a big role in Publication Tables, and includes standard variable placeholders – as well as advanced [[#Conditional Variables|conditional variables]], whose list of variable values change based on the value of another Dimension for which the table is built.

== Metadata vs Materialised ==

A Publication Table is stored in Fusion Registry the same way any other piece of structural metadata is stored. In this way it has an owner (Agency), and identity (ID) and a version. The change history is tracked using the same services, and the table definition can be [[Publication_Table_Web_Services|queried]] using the same SDMX web services for Structural Metadata.

The Publication Table metadata contains information about which Dataflows are used by the Table, which additional variables are being imported, and a definition of the table structure, and which Observations to import by referencing them by their unique [[#Observation Key|Observation Key]].

A Publication Table definition can be '''materialised''' into a table of Data by from the [[Publication_Table_Web_Services|Publication Table Web Service]], which combines the information of the Table Identity, and Variable Values to rewrite the [[#Observation Key|Observation Key]] into actual observation values. If building a web User Interface, then response from this service can be a packet of JSON, which describes the layout and content of the final Table in a way which is quick and simple to render for the User to see. The process of materialising the table is known as ‘'''build time'''’.

Additional web services are provided to enable clients to ask questions about the table such as ‘what are the earliest and latest time periods for which there is data for this table’ or ‘what possible variable values are there for this table’. This enables user interfaces to provide a user with the option to change the variables ‘on the fly’ ensuing they are able to build a table which is known to have data.

= Publication Table Cell Content =
== Overview ==
[[File:PublicationTable Layout.png|thumb|Publication Table Layout]]
A Publication Table is made up of a '''Table Head Rows''' and '''Table Body Rows'''.

The Table Head is made up of 1 or more table rows. Each Table Head cell may span 1 or more columns and 1 or more rows. The Table Head content is made up of free text and / or variable placeholders whose values are resolved when the table is built. Table Head Cells can be left empty.

The Table Body is made up of 1 or more Table Rows. The Table Rows provide the Publication Table with its data, by referencing Observations from the Dataflow.

At table build time all variables are resolved, and the corresponding data are retrieved from the data stores which hold the data.

== Table Rows ==
=== Overview ===
A Table Row consist of a Table Row heading, this can be a mix of free text and [[#Table Variables|variable text]]. The Table Row Heading is always the first Table Row Cell, and can not span multiple columns or rows. A Table Row can not have more then one heading. The remaining Table Row Cells are the Observation Cells, which may contain free text and [[#Table Variables|variable text]] but will typically contain a reference to an Observation by an [[#Observation Key|Obsevation Keys]] or a [[#Calculated Observation Cells|Calculation]] which can combine multiple [[#Observation Key|Obsevation Keys]].

A Table Row which results in no Observation data will not be included in materialised table at build time, unless the Table Row is a grouping row (see [[#Row Levels|Row Levels]] for details on grouping rows).

=== Row Levels ===
[[File:PublicationTabel Levels.png|thumb|Showing Levels in the Table Rows]]
The Table Row Heading can be used to define a row level. The level indicates to the client responsible for rendering the table the row's position in a hierarchy. Row levels are depicted using one or more leading chevrons ‘>‘ before the text. The number of hyphens indicates the level, with one leading hypen indicating level 1, two indicating level 2, and so on. It is possible to skip levels if required, i.e. Row 1 is level 1 with Row 2 assigned to level 3, skipping out level 2 entirely.

An example of row headings using levels is as follows:

Countries
> Europe
>> France
>> Germany
> Asia
>> China

Levels can be an important way to '''group data'''. A Table Row which acts as a parent row in a hierarchy, can be used as grouping mechanism for the rows that follow. This enables Parent Rows to be included in the output table, even if the Table Row contains no Observation data. The default behaviour for a Table Row with no data is to exclude it from the built table.

=== Looping Rows ===

Instead of explicitly defining a row for each Dimension member, it is possible to define a row as a loop (to iterate over all, or a subset of Dimension members). For example, if a publication table contains a Dimension for countries with an ID of REF_AREA then a loop can be set up to loop through all REF_AREA values, or a subset of the values - for which there is data. If the values are backed by a Codelist which has a hierarchy, the hierarchy can be preserved in the final table. It is also possible to nest a loop within another loop, for example for each REF_AREA output each INDICATOR.

====Syntax====
The loop is defined by giving the table row a label which starts with a syntax of :=LOOP([args]), like in the following example
:=LOOP([dimension id], [preserve hierarchy], [loop order], [explicit values to include])

'''Argument Definition'''
''dimension id'' = Required. The ID of the Dimension to loop over. For example CURRENCY would loop over all the Currency values which have data in the given context. The row label is the Currency, for example USD.
''preserve hierarchy'' = Optional, default false. If true the hierarchy of the Codes in the underlying Codelist will be built into the row hierarchy
''loop order'' = Optional, default 1. 1=Natural Order (as they appear in the Codelist), 2=By Code ID, 3=By Code Label
''explicit values to include'' = Optional, default include all values. A var arg list of values to loop over.

'''Full Example'''

:=LOOP(REF_AREA, true, 3, EUR,FR,DE,UK,ES)

====Simple Loop====
The loop is defined by giving the table row a label which starts with a syntax of :=LOOP([args]), like in the following example
:=LOOP(REF_AREA)

This loop definition will loop through all the REF_AREA values, building a row for each value. The Observation keys for the row can refer to the loop variable using a $(DIM_ID) placeholder in the Observation Key, for example:

Employed Student
:=LOOP(REF_AREA) A:$(REF_AREA):EMP A:$(REF_AREA):STU

The output table would look like the following:

Employed Student
United Kingdom 20 12
France 22 11
Germany 25 15

====Nested Loop====
If a loop row is followed by a row (or rows) which is an extra level deep, the child row(s) will be included for each loop, for example

Employed Student
> :=LOOP(REF_AREA)
>> Total A:$(REF_AREA):'''T''':EMP A:$(REF_AREA):'''T''':STU
>>> Male A:$(REF_AREA):'''M''':EMP A:$(REF_AREA):'''M''':STU
>>> Female A:$(REF_AREA):'''F''':EMP A:$(REF_AREA):'''F''':STU

Because the '''Total''' row is a child row of the LOOP (denoted by the number of leading '>') and '''Male'''/'''Female''' are children of '''Total''' - for each REF_AREA there will be a Total, Male and Female, outputting a table like the following:

Employed Student
United Kingdom
Total
Male 5 1
Female 4 2
France
Total
Male 6 2
Female 7 3

It is also possible to define a subloop as a loop. The previous example can be rewritten as:

Employed Student
> :=LOOP(REF_AREA)
>> :=LOOP(SEX) A:$(REF_AREA):'''$(SEX)''':EMP A:$(REF_AREA):'''$(SEX)''':STU

There is no limit to the number of subloops that can be defined.

====Loop Hierarchical Values====
If the underlying Codelist has a hierarchy, a subsequent argument can be provided to ask for the hierarchy to be preserved

Countries
> :=LOOP(REF_AREA, true)

'''Note:''' A hierarchy will keep rows with no data if they are acting as a parent node for rows that do have data (e.g. there may not be data for Europe, but the row exists because there are child rows of France and Germany with data).

Nested loops can also preserve hierarchies for example:

:=LOOP(REF_AREA, true)
>:=LOOP(SEX, true)

With an example output (assuming no data exists for the hierarical parents (Continent, Europe)

Continent
Europe
France
Total (M+F)
Male
Female
Germany
Total (M+F)
Male
Female

====Order Loop Rows====
It is possible to order the labels by their natural order (that of the Codelist), by id or by label, using the numbers 1, 2, or 3 respectively. The following outputs the rows ordered by label.

Countries
> :=LOOP(REF_AREA, true, 3)

====Fixed Loop List====
It is also possible to define a subset of the values to loop over. For example, if REF_AREA has data for 100 countries, but only 3 are required in the table. In this scenario, the natural order is that of the provided list, not of the underlying Codelist.

Countries
> :=LOOP(REF_AREA, true, 3, UK, FR, DE)

====Loop over External Variable====
Instead of looping the values of the Dimension's Codelist, it is possible to loop another Codelist - the purpose for this is to support alternative labels (if the labels for the publication table need to be different from those of the Dimension members) - or a subset (as an alternative to passing the subset of values into the LOOP argument), or to define a hierarchy (if the Dimension's Codelist does not have a hierarchy, or an alternative is required).

In order to loop over an external Variable, the Publication Table must be linked to an external Codelist/Valuelist against an Alias. For example EXT=My Country Codelist with Hierarchy. The corresponding loop then loops the Codes in the external list, and stores these the corresponding Dimension ID variable, for example:

Countries
> '''REF_AREA''':=LOOP('''EXT''')

Noting that the loop is over the variable EXT which resolves to an external Codelist. The loop is assigned to the REF_AREA Dimension.

=== Anchor Rows ===
The default behaviour is for a row to be omitted if the row has no data and it has no child rows with data. However, if the desired result is to always output the row, then this can be achieved by using the [anchor] tag in the row header label.

Consider the following Publication Table definition:

Male Female
Employed A:M:EMP A:F:EMP
Student A:M:STU A:F:STU
Retired A:M:RET A:F:RET

If there are no data for Student, then the materialised table would look like this

Male Female
Employed 12 13
Retired 3 5

==== Anchor Single Row [anchor] ====
To anchor a single row, add the '''[anchor]''' tag at the start of the row

Male Female
Employed A:M:EMP A:F:EMP
[anchor]Student A:M:STU A:F:STU
Retired A:M:RET A:F:RET

Then the output table would preserve the 'Student' row

Male Female
Employed 12 13
Student
Retired 3 5

This rule can be coupled with [[Publication_Table#Formatting_Missing_Observations|formatting for missing observations]], then the table could be enhanced to look something like this

Male Female
Employed 12 13
Student - -
Retired 3 5

==== Anchor Child Rows [anchor+] ====
For a row with a hierarchy, an '''[anchor+]''' tag can be placed on a parent node to anchor the parent and all of its children, for example:

Continent
[anchor+]>Europe
>>France
>>>Total (M+F)
>>>>Male
>>>>Female
>>Germany
>>>Total (M+F)
>>>>Male
>>>>Female

The above rule would always output Europe and all of its descendants (France, Total, Male, Female, Germany).

==== Anchor Siblings and Child Rows [anchor*]====
The '''[anchor*]''' tag can be used as a way to anchor all following siblings and child nodes, this can be a useful mechanism to anchor all rows for the whole table by placing the anchor[*] on the first row of the table.

==== Anchor Loops ====
If a loop is preceeded by an '''[anchor]''' tag then the system will have the following behaviour:

* If an external Codelist variable is being looped, all codes will be ouput regardless of what data is stored in the system
* If a Dimension is being looped, all codes in the Codelist used by the Dimension will be output regardless of what data is stored in the system
* If a Dimension is being looped and it is not coded (i.e it is free text or numerical or other) then the [anchor] has no effect, as the looped values are based on what data exists

== Observation Key ==
=== Overview ===
The purpose of the Observation Key is to reference a Observation whose value will be resolved at table build time. The Observation Key can explicitly resolve to 1 observation value, if all parts of the Key are provided. However, it is possible to make parts of the key varaibles, allowing the values to be set at table build time.

An Observation Key is made up of 4 parts:

* Dataflow Reference
* Series Reference
* Time Period Reference
* Measure Reference

An example Observation Key with all of these parameters in place is:

DF_REG.M:UK:SUR:2008:BIRTHS

The key is broken down into the following parts:

{| class="wikitable"
|-
! Observation Key Part !! Name !! Description !! Required
|-
| DF_REG|| Dataflow Alias|| This identifies the Dataflow which has the data. The identification uses the alias, which is assiged to the Dataflow in the Publication Table definition. The Dataflow Alias is only required if the Publication Table references more then one Dataflow. If the Publication Table references more then one Dataflow, it can assign a Dataflow to a Default Alias, which means that it is required as part of the Observation Key. || No (if only 1 Dataflow or a default alias is set)
|-
| M:UK:SUR || Series Key || This identifies the Series of Data for the Dataflow, by providing a single value for each of the Dimension Members of the Dataflow, separated by a ‘:’ character. In this example the Dataflow has 3 Dimensions (excluding time), the value for the first is 'M', the second 'UK' and the last 'SUR'. As the Dataflow in this example has a Time Dimension this Series Key will uniquely identify a Series of Observations where each Observation refers to a unique Time Period || Yes (but can omit values)
|-
| 2008 || Time Period || This identifies the Time Period in the Series of Data identified by the previous section. The Combination of the Series and Time Period for a Time Series Dataflow will uniquely identify the Observation || No. Omit to make it variable for Time Series Data (defaults to last period). Not required if there is no Time Dimension for the DSD.
|-
| BIRTHS || Measure Dimension || If the Dataflow uses a Data Structure with multiple Measures, then the Measure ID must be included as the last part of the Observation Key. In this example the Dataflow has multiple measures (BIRTHS/DEATHS/MARRIAGES) so the Observation Key needs to include this value. || When there is only 1 measure it is not required, when there are multiple measures, it is required.
|}

Observation Keys are included on the materialised table in JSON format enabling the observation value to be tied back to the key. Observation Keys also provide a mechanism by which [[#Formatting_Cells|formatting rules]] can be applied to a subset of table cells.

=== Variable Dimension Members ===
It is possible to omit Dimension Values from the Observation Key, when this is the case that Dimension will become a variable – whose value can be set at table build time. For example, the second Dimension value of ‘UK’, from the previous example, can be omitted like this:

M::SUR:2008

The Publication Table can now be built for any Country by passing in the variable at table build time - i.e. UK, FR, DE. By passing in different variable values, the Observation Key will change, and as such it will reference a different Series in the Dataset.

When an Observation Key has a variable Dimension member, the ID of the Dimension can be referenced in text cells to create variable headings. For example, if the ID of the 2nd Dimension in the DSD is '''REF_AREA''', the Table Column could be given a title of $(REF_AREA) which would resolve to 'United Kingdom', 'France', or 'Germany' at build time depending on which varaible value is used to build the table. [[#Dimension Variables|#See Dimension Variables]].

=== Variable Time Period ===
For Time Series dataset, the Time Dimension can form part of the Observation Key, for example M:UK:SUR:'''2008'''. The 2008 part of the key refers to the year 2008.

It is possible to omit the value for Time Period in the Observation Key. Omitting Time Period will make the Time value a variable which can be provided at table build time, defaulting to the last period for which data exists if no value is supplied.

Extending the above example to omit the Time value results in an Observation Key whose Country and Time values can be provided to the web service at table build time.

M::SUR

'''Note:''' The above key is exactly the same as using the variable Time Period value 'P' in the Time Dimension, as follows:

M::SUR:'''P'''

Additional functions can be performed on the Time Dimension, such as moving forwards or backwards from the table base period, [[#Time Variables|#see Time Variables]] for more details.

=== Free Text Observation Keys ===
Whilst typically a cell in the body of the Publication Table would resolve to an Observation value, it is possible to use it for free text. To include free text cells, start the cell value with a leading apostrophe character, for example:

'Cell Text

'''Note''': An Observation Key can '''not''' contain a mix of free text as well as an Observation value

== Calculated Observation Cells ==
[[File:TableDefinition Calculations Variables.png|thumb|Publication Table with Calculated Cells and Variable Dimensions]]

An Observation Cell may contain calculated values, which can consist of a mix of simple algebra (+, -, *, /) and a reference to one or more Observations which may come from multiple Dataflows.

To define a Calculated Observation the Cell the following rules apply:

* The Observation Key must start with :=
* Any Observation Keys must appear within enclosing square brackets '[' and ']'
* An Observation key may embed calculations on a single Dimension by using parenthesis '(' and ')' in the key part
* Numbers in Observation Keys are treated as Code IDs, unless prefixed with a '#'
* The same rules for variable Dimensions and Time Period apply when resolving the Observation Keys used in the calculation

Examples are shown in the table below

{| class="wikitable"
|-
! Example Syntax !! Description
|-
| :=[A:UK:EMP]+10 || Add 10 to the observation value retrieved from the key A:UK:EMP
|-
| :=[A:(UK+FR+DE):EMP] || Add the series A:UK:EMP + A:FR:EMP + A:DE:EMP
|-
| :=[A:(UK+FR+12):EMP] || Add the series A:UK:EMP + A:FR:EMP + A:DE:12
|-
| :=[A:(UK+FR+#12):EMP] || Add the series A:UK:EMP + A:FR:EMP and add the Constant '12'
|-
| :=[A:UK:EMP]+[A:FR:EMP]+[A:DE:EMP] || Add the series A:UK:EMP + A:FR:EMP + A:DE:EMP
|-
| :=[A:(UK+FR):EMP]/[A:(UK+FR):UMP] || Adds the series A:UK:EMP + A:FR:EMP and divides by the result of A:UK:UMP + A:FR:UMP
|-
| :=((100/[A:UK:EMP:(M+F):P-1])*[A:UK:EMP:(M+F)])-100 || Demonstrates the ability to use parenthesis outside of the Observation Keys to control the order of mathematical operations
|-
| :=[POP.M:UK:(M+F)] / [LAB.M:UK:EMP] || Demonstrates the ability to calculate data from two Dataflows (defined by alias POP and LAB)
|-
| :=[M:UK:EMP:M(P)] + [A:UK:EMP:A(P)] || Demonstrates the ability to calculate data where the period is converted from the base period
|}

=== Missing Values ===
If any input Observations are missing in the calculation, the final result of the calculation is an empty value. For example:

:=[A:(UK+FR+DE):EMP]

This would not output a value if A:UK:EMP, A:FR:EMP or A:DE:EMP have no corresponding Observation value for the time period the table is being built for.

RFS version 11.12.0 and higher allow for assignment of default value(s) to missing components of the calculation. The syntax uses the pipe '|' symbol, the semantic meaning is 'use this value | or this value if missing'. An example of the pipe symbol being used in a calculation expression is shown below:

:=[A:(UK|1+FR|1+DE|0):EMP]

This expression substitutes the UK for the value of 1 if there is no corresponding Observation for A:UK:EMP. The Observation A:FR:EMP is substituted for 2 if missing, and A:DE:EMP is substituted for 0. Therefore is all 3 Observations are missing, the final value for the expression = 1+1+0 = 2.

It is possible to substitute the entire block if any part is missing by appending the double pipe '||' to the end of the expression. The following example results in a zero value if any input Observation (UK, FR, DE) value is missing:

:=[A:(UK+FR+DE):EMP||0]

The above expression is equivalent to the following expression:

:=[A:UK:EMP||0] + [A:FR:EMP||0] + [A:DE:EMP||0]

=== Calculated Cell Observation Key ===
Calculated Cells have implicit [[#Observation_Key|Observation Keys]] which are derived based on the inputs series in the calculation. The [[#Observation_Key|Observation Key]] can be used to assign [[#Formatting_Cells|Formatting Rules]] in the same way as a 'standard key' can be used. It is possible to assign an explicit key to the calculation, which will override the implicit key assigned. The following table shows how the key assignment works.

For example
{| class="wikitable"
|-
! Calculation !! Output Key || Explanation
|-
|:=[A:(UK+FR+DE):EMP] || A:#:EMP:2009 || The first and last dimension have a fixed value, the second dimension has a variable value and is replaced by '#'. The time period is added to the end of the key.
|-
|:=((100/[A:UK:EMP:(M+F):P-1])*[A:UK:EMP:(M+F)])-100 || A:UK:EMP:#:# || The first three dimensions have a fixed value, the last dimension has a variable value and is replaced by '#'. The time period is a variable as there are 2 time inputs to the calculation (P and P-1)
|-
|:=PCTOF([A:M:UK:P],[A:TOT:UK:P]) || A:#:UK:2009 || The first and last dimension have a fixed value, the second dimension has a variable value and is replaced by '#'. The time period is added to the end of the key.
|-
|:=[A:UK:EMP]+10 || A:UK:EMP || The input key is the same as the output key as none of the dimensions have a variable value. The time period is added to the end of the key.
|-
|A:UK10:EMP:=[A:UK:EMP]+10 || A:UK10:EMP:2009 || The output key is explicitly assigned. The time period is added to the end of the key.
|}

The output key conforms to the same Dimension breakdown as the underlying Data Structure. If a Observation Key is explicitly assigned, it must also honour the same Dimension breakdown. This allows formatting rules to be applied using the key component syntax, for example COUNTRY=UK10 would search for Observation Keys whose Country Dimension is reporting the value UK10.

== Percentage Of ==
The PCTOF (Percentage Of) function can be used to calculate 'x' as a percentage of 'y' following the syntax:

PCTOF(x,y)

This is reolved to the following mathmatical expression: '''(100/y) * x'''

x and y must either be a numerical value, and observation key, or an expression using a combination of the two. For example

:=PCTOF([A:M:UK:P],[A:TOT:UK:P])

The Observation Key follows the same standard rules with respect to varaibles and inline calculations - the following example shows an Observation Key where Male + Female values are added with the result being used in the Percentage Of function

:=PCTOF([A:M:UK],[A:('''M+F'''):UK])

A more complex example shows calculations around the PCTOF function:
:=(10 + PCTOF([A:M:UK:A(P)-1],[A:(M+F):UK:A(P)])) / 2

== Footnotes ==

[[File:Footnotes3.png|thumb|A table with footnotes]]
[[File:Footnotes2.png|thumb|Footnotes in the table builder]]

=== Overview ===
Footnotes can be added to any part of the publication table (table column/row header, or observation cells). The output table is generated with a footnote number, with the list of footnotes at the end of the table.

Footnotes need to be thought about in 2 parts: Part 1 are the list of footnotes which are defined at the Publication Table level. Part 2 are the references to the footnotes from within the table. This separation of allows footnotes to be reused within the same table, and when working with [[#Footnotes_with_Placeholders|placeholders]] footnote templates can be set up where the text changes based on the placeholder values defined in the table builder.

To add a footnote, add it to the Publication Table, for example: 
Footnote 1 
Footnote 2

Then in the table builder, add a reference to which footnote to use, $(1) references '''Footnote 1''', and $(2) references '''Footnote 2'''.

'''Note:''' The order of the list of footnotes in the table definition may not be reflected in the order they are output. As such a footnote referenced by $(1) may be output as the second footnote.

=== Footnotes with Variable ===
A footnote can reference variables, using the [[#Table_Variables|variable syntax]],

A footnote for $(REF_AREA.UK)

A footnote can even output an observation value, observation attribute or series attribute.

Measure Value
A footnote with obs value $(A:UK:EMP:2008:OBS_VALUE)

Observation Attribute
A footnote with obs attribute $(A:UK:EMP:2008:OBS_COMMENT)

Series Attribute
A footnote with obs attribute $(A:UK:EMP:P:SERIES_TITLE)

When combining variable footnotes with placeholder values and loops, it is possible to dynamically create references to observation attributes.

'''Note:''' the use of P in the final cell key which is a reference to the [[#Time_Variables|time period]] for which the table is being built.

=== Footnotes with Placeholders ===
It is possible to write a generic footnote for the publication table, which includes placeholders. When the footnote is referenced, the placeholder values are provided. The syntax for a placeholder is {n} where n is a positive integer, for example {1}.

Footnote Example

A footnote for $(REF_AREA.{1}) and $(REF_AREA.{2})

In the table builder, the footnote is referenced as usual, but with arguments for the placeholders:

Footnote reference in Table builder

My Table Row '''$(1,UK,FR)'''

The footnote reference in the above example is in bold, and consists of which footnote text to use (in this case the first one in the list, denoted by '1'), and 2 arguments are passed for the placeholders, placeholder 1 takes the value of UK and placeholder 2 takes the value of FR. During the materialise process the intermediate text is generated:

A footnote for $(REF_AREA.UK) and $(REF_AREA.FR)

Which then goes through the variable resolve process, in this case both variables, '''$(REF_AREA.UK) and $(REF_AREA.FR)''', match the [[#Coded_Variables|Coded Variable]] syntax and would resolve to:

Final Output
A footnote for United Kingdom and France

=== Footnotes in Observation Cell ===
Any number of footnotes can appear in an Observation cell, simply put the footnote(s) at the end of the [[#Observation_Key|Observation Key]].

Standard Footnote Reference 
DF_REG.M:UK:SUR:2008:BIRTHS $(1) $(2)

Footnote with Placeholder 
DF_REG.M:UK:SUR:2008:BIRTHS $(1,UK)

=== Footnotes in Loops ===
A row [[#Looping_Rows|loop]] contains a locally scoped variable, and as such it can be used when calling a footnote with a placeholder.

Example Footnotes
$(A:T:{1}:EMP:P:SERIES_TITLE)
$(A:T:{1}:EMP:P:TITLE_COMPL)
$(A:{1}:{2}:EMP:P:OBS_COMMENT)

Example Table Definition
Employed Student
> :=LOOP(REF_AREA) '''$(1,$(REF_AREA[id]))'''
>> Total '''$(2,$(REF_AREA[id]))''' A:$(REF_AREA):T:EMP A:$(REF_AREA):T:STU
>> :=LOOP(SEX) '''$(3,$(REF_AREA[id]),$(SEX[id]))''' A:$(REF_AREA):$(SEX):EMP A:$(REF_AREA):$(SEX):STU

Footnote Intermediate Resolution
Row 1 (REF_AREA=UK)
$(A:T:UK:EMP:P:SERIES_TITLE)

Row 2 (REF_AREA=UK)
$(A:T:UK:EMP:P:TITLE_COMPL)

Row 3 (REF_AREA=UK, SEX=M)
$(A:UK:M:EMP:P:OBS_COMMENT)

Final Footnote output
Row 1 (REF_AREA=UK)
This is a series for the United Kingdom

Row 2 (REF_AREA=UK)
This is a title compliment for the United Kingdom

Row 3 (REF_AREA=UK, SEX=M)
This is an observation comment for Males in the United Kingdom

= Formatting Cells =
== Formatting Numerical Observation Values ==
[[File:Obs formatting.png|thumb|Number Formatting Options]]
Numerical observation values can be formatted in the following way

# '''Number of Decimal Places''' or '''Significant Figures''' can be defined
# For Decimal places this can be set to a maximum rule or a fixed rule, i.e the value 1.3 to 2 decimal places is 1.3 using the maximum rule and 1.30 using the fixed rule
# '''Rounding mode''' can be specified
# '''Scaling Factor''' this is a multiplier applied to the observation value, the multiplier is x10^[scaling factor]. Example, a value of 1.1 with a scaling factor of 1 is 11, a scaling factor of 2 is 110, a scaling factor of 3 is 1100. If the Observation or related Series has a UNIT_MULT concept the difference between the unit multiplier and scaling factor will be taken. Example, an Observation with a value of 1.1 and a unit multiplier of 9 (Billions) is reported, the publication table has a scaling factor of 6 (millions) - the number 1.1 is scaled using 10^3 (calculated by 9-6), the 1.1 billion is converted to 1100 million.
# '''Scientific notation''' can be requested on output, i.e 123456 to 2 significant figures is 120000 in standard form or 1.2E4 in scientific form
# '''Decimal formatting''' for long numbers, i.e. 12345.8 there is not thousands separator by default, to include a thousand separator set the decimal formatting rule to either a single Language (i.e. French Formatting) or Dynamic formatting, which uses the Accept-Langauge locale/language sent be the web browser (this can be overridden using the locale=[loc] query parameter. The formatting rules can also influence which characters the numerical characters used.
# Rules can be applied to all series, or specific series

=== Match Series ===
When creating a formatting rule, it is possible to define which series the rule applies to. If left blank, the rule will apply to all observation values.

The syntax for defining specific series is shown in the following table:

{| class="wikitable"
|-
! Rule Type !! Syntax !! Description
|-
| Series Match || =A:UK:EMP || Match a specific series from the default Dataflow
|-
| Series Match Multiple Values || =A:UK+FR+DE:EMP || Multiple codes in 2nd Dimension (UK, FR and DE) any can match for the rule to be valid
|-
| Series Match by Dataflow Alias || =FLOW2.A:UK:EMP || Match a specific series for the Dataflow with alias FLOW2
|-
| Wildcard Series Match || =A::EMP || Match all series with the given dimension values (2nd Dimension is wildcarded)
|-
| Not Rule || !=A:UK:EMP || Match all series which do not match this series (wildcard series also supported)
|-
| Dimension Rule || CURRENCY=GBP || Match all series which have CURRENCY of GBP
|-
| Not Dimension Rule || !CURRENCY=GBP || Match all series do not have a CURRENCY of GBP
|-
| Multiple Rules || CURRENCY=GBP,FREQ=M || Comma is used to include multiple matches (also applies to series match)
|}

== Formatting Missing Observations ==

Observation Values can be missing for 2 reasons:
# There is no Observation in the database for the given Observation Key
# There is an Observation in the database but the measure value is either: NaN, null, or an empty string

In both cases it is possible to define an output value using formatting rules. For the first case, the observation does not exist, it is possible to define a default output value, for example ‘-‘. For the second case, where the observation does exist, it is possible to define a set of rules based on another Component of the Observation, for example the Observation output may be related to the OBS_STATUS Attribute.

When the formatted value is based on a Component, a set of rules must be defined which state ‘when component=x output y’, where x may be an fixed value or a pattern to match (using a regular expression). The rules for mapping component values are defined in a [https://fmrwiki.sdmxcloud.org/Representation_Map Representation Map], and the Publication Table simply provides the connection to the Representation Map to use. This enables reuse of the mapping rules across multiple Publication Tables. The Publication Table must define two properties in order to support Component Mapping rules:

# The Component to obtain the value from i.e OBS_STATUS
# A link to the [https://fmrwiki.sdmxcloud.org/Representation_Map Representation Map], which holds the mapping rules. Note, the Representation Map is expected to have 1 source mapping, this could be of any type (string, codelist, other)

An example Reprsentation Map for OBS_STATUS is shown below:

{| class="wikitable"
|-
! OBS_STATUS !! Is Reg Ex !! Missing Value !! Description
|-
| M || false || mv || When OBS_STATUS is M output mv
|-
| L || false || np || When OBS_STATUS is L output np
|-
| .* || true || - || When OBS_STATUS is anything else output -
|}

== Frequency Formatting ==
[[File:PublicationTable FreqMap.png|thumb|Representation Map with frequency mapping rules]]
[[File:PublicationTable Materialised FreqMap.png|thumb|Materialised table with formatted time headings ]]
Time Variables always resolve themselves to an [[SDMX_Time_Formats|SDMX Date Format]] when forming the [[#Observation Key|Observation Key]], this ensures the correct observation value is returned from the data store. When a time variable is used as a label, in a table heading for example, the same formatting will be applied by default but a custom format can be provided. Custom display formats for a Time Period enable a time period such as 2007-Q1 to be resolved to custom text string, for example '31 Mar 2007'.

Custom Time formats rules are defined by providing an association between an SDMX Frequency ID (#see [[SDMX_Time_Formats|SDMX Date Format]]) and the output format template. The output format template is a string that defines which components of time are output an in which order and style. For example to output a date as 'March 2001' the format template would look like this 'MMMM yyyy' to indicate a full month name (MMMM) and a 4 character year (yyyy). format rules make use of the [https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html Simple Date Format] which is used by Java as such all of the documentation on Simple Date Format applies.

Frequency Format Mapping is not defined as part of a Publication Table, instead the rules are defined in a [https://fmrwiki.sdmxcloud.org/Representation_Map Representation Map] and the Publication Table references the Representation Map as the source of its frequency conversion rules. This enables the same set of rules to be used for multiple Publication Tables. The Representation Map must map 1 source (Codelist or String) to 1 target (String), it can contain as many mapped frequencies as required. If the Representation Map contains a mapping for a value that is not a valid SDMX Frequency, then it will be ignored.

It is important to note that all time periods resolve to a single point in time, which is the end of the period, for example an Annual period of 2008 resolves to 2008-12-31:T23:59:59. The resolution of frequency to a point in time enables the frequency formatting rules to use all aspects of time, even when converting a low frequency, for example 2008 formatted using the pattern dd MMM yy would resolve to '31 Dec 2008'.

== Variables Summary ==

{| class="wikitable"
|-
! Variable Type !! Description !! Syntax !! Example !! Example Output
|-
| '''Dataflow Name''' || Displays the name of a Dataflow referenced by the Publication Table in the locale in which the Publication Table is built || $([dataflow alias]) || $(EDU) || World Bank, Education
|-
| '''Dataflow Description''' || Displays the description of a Dataflow referenced by the Publication Table in the locale in which the Publication Table is built || $([dataflow alias].desc) || $(EDU.desc) || World Bank, Education
|-
| '''Dataflow Id''' || Displays the ID of a Dataflow referenced by the Publication Table || $([dataflow alias].id) || $(EDU.id) || DF_EDU
|-
| '''Variable Dimension Value''' || Displays the name of the variable used when an Observation key has a wildcard member || $([dimension id]) || $(REF_AREA) || United Kingdom
|-
| '''Specific Component Value''' || Displays the name of the Component Value || $([component id].[component value]) || $(OBS_STATUS.C) || Confidential
|-
| '''Component Name''' || Displays the name of the DSD Component (this is the name of the Concept used by the Component) in the locale in which the Publication Table is built || $([component id].name) || $(OBS_STATUS) || Confidentiality Status
|-
| ''' Enumeration Value ''' || Displays the name of a Code in a Codelist or Value in a Valuelist || $([list alias].[list item ID] || $(SEX.M) || Male
|-
| '''Conditional Variable''' || Displays the variable value, which is conditional on the value of a variable Dimension || $([conditional variable id][variable id]) || $(BANKNOTES[NOTE1]) || £5
|-
| ''' Time ''' || Displays the base time period for which the table is built, or an offset from this time period / cast to a different frequency || see [[#Time Functions|Time Functions]] || $(P) || 2008
|}

= Table Variables =
== Overview ==
Variables are a very important part of Publication Tables. They can be used to make the content of the table dynamic, setting and changing the value at run time to change the table content. Variables can be used in Observation Keys, but the values of variables can also be used anywhere where there is text, for example the table heading, subheading, column or row headings, footnotes, and Observation Cells with free text.

The general syntax for a variable is

$(''variable id''.''sub part''[''detail''])

Where variable id is always required, the subpart and detail are optional. Subpart is used if the variable resolves to a Component and a coded value is required, the detail array part obtains the id, name, or description of the underlying SDMX type.

This section describes the various variables that exist, along with the syntax of how to use them.

== Dataflow Variables ==
Publication Tables define data cells across one or more Dataflows. Each ‘imported’ Dataflow is given an alias, which can be used to refer to the Dataflow in the [[#Observation Key|Observation Key]].

It is possible to refer to the Dataflow's Name, Description, and Id using the following variable syntax.

'''Default Dataflow''' 
Dataflow Id
$([id])

Dataflow Name
$() or $([name))

Dataflow Description
$([desc])

'''Dataflow with Alias=EXR''' 
Dataflow Id
$(EXR[id])

Dataflow Name
$(EXR) or $(EXR[name))

Dataflow Description
$(EXR[desc])

== Dimension Variables ==
When an [[#Observation Key|Observation Key]] has [[#Variable Dimension Members|variable Dimension Values]], the table will be built by replacing this variable with a value to complete the Observation Key. The variable value is either passed explicitly to the web service, or defaulted by the system if not provided. The Dimension Variable can be referenced in the table using the syntax $([dimension id]), for example:

Dimension Value Name
$(INDICATOR) resolves to 'Employment'

This would resolve to the '''name''' of the Indicator value used in the table, for example '''EMP''' may resolve to the label '''Employment'''. The label for the Dimension Value is obtained from the Codelist or Valuelist which the Dimension uses. If the Dimension does not use a list (i.e. it is free text or numerical) then the Dimension value is simply used in place of the variable placeholder.

To resolve to the '''id''' of the value for the Component, for example '''EMP''', use the following syntax

Dimension Value Id
$(INDICATOR[id]) resolves to 'EMP'

== Dependent Dimension Variable ==
It is possible that the value for one Dimension is dependent on the value of another Dimension, for example CURRENCY may be dependent on the COUNTRY and COUNTRY is a variable in the [[#Observation Key|Observation Key]] - it is possible to specify that the value to use for the CURRENCY Dimension can be derived using a lookup table. The lookup table in this instance is maintained as an [https://fmrwiki.sdmxcloud.org/Representation_Map SDMX Representation Map] which is used to map COUNTRY to CURRENCY. The Publication Table specifies that the two Dimensions are related and provides a reference to the Representation Map to use which has the rules.

When working with a dependent variable, both the dependent variable and the Dimension on which it depends should be omitted from the [[#Observation Key|Observation Key]] - for example consider a full key of:

A:UK:GBP:2007

Where UK is the COUNTRY and GBP is the Currency, to make the COUNTRY a variable Dimension, it should be omitted from the key, as follows:

A::GBP:2007

To then make the CURRENCY a variable it is also omitted , as follows:

A:::2007

The Publication Table is given the rule that CURRENCY depends on COUNTRY and should use the COUNTRY_2_CURRENCY [https://fmrwiki.sdmxcloud.org/Representation_Map Representation Map], which in this example looks like the following:

{| class="wikitable"
|-
! COUNTRY !! Is Reg Ex !! CURRENCY
|-
| UK || false || GBP
|-
| US || false || USD
|-
| .* || true || EUR
|}

When the Publication Table is materialised the [[Publication_Table_Web_Service|web service]] only requires the COUNTRY information, and it will complete the Observation key using the representation map lookup table.

'''Note''' it is possible to override the mapped value by passing in an explicit override to the web service. For example passing to the web service both COUNTRY=UK with CURRENCY=EUR will override the mapping behaviour for CURRENCY.

== Component Values ==
Components of the underlying Data Structure can be explicitly referenced using the following syntax '''$([component id])''', for example:

Component Name
$(OBS_STATUS) or $(OBS_STATUS[name]) resolves to 'Observation Status'

Component Description
$(OBS_STATUS[desc]) resolves to 'Information on the quality of a value or an unusual or missing value.'

== Coded Variables ==
To reference a Code value, the syntax is $([component id / codelist alias].[item id]). The first variable part is the ID of a Component in the underlying Data Structure, or alternativly the alias of an external variable defined on the Publication Table. An external variable is a link to an SDMX Codelists or Valuelist. Example are:

Name of Code
$(CURRENCY.$) resolving to the text '''Dollar''

Description of Code
$(DWELLING.SM[desc])

== Conditional Variables ==
A Conditional Variable is one whose value is conditional on the value of a [[#Dimension Variables|Dimension Variable]].

An example use case is Banknotes, where the values for NOTE1, NOTE2, or NOTE3 is dependant on the Country Dimension. A Conditional Variable could be used to set a values for each variable (NOTE1, NOTE2, NOTE3) based on the value used for the current Country Dimension i.e. US NOTE1=$5, UK NOTE1=£5.

The syntax of referring to a Conditional Variable is $([conditional variable id][variable id]), for example
$(BANKNOTE[NOTE1])

== Reported Attribute for an Observation or Series ==
A variable can be a reference to a part of an Observation, for example the Unit Multiplier (UNIT_MULT). A Table Row header, for example, could include variable text such as 'Number of payment accounts (''thousands'')' where the word '''thousands''' is a variable because it depends on other variables which build the key.

To achieve this, use the syntax:

$(A:UK::2008:UNIT_MULT[name])

The observation key in this case is A:UK::2008, where the third Dimension is a variable as it has no value (in the main body of the table this dimension would probably also be a variable, which means the table can be built with different values, each value may change the output of this UNIT_MULT value). The last part of the key is the Attribute ID (UNIT_MULT) - this can be a reference to a series attribute, or an observation attribute. The [name] indicates that it should resolve to the name of the reported attribute value, alternative values are '''id''' for the reported value or '''desc''' for the description of the reported value.

If time is also a variable, then the key can change accordingly, using the variable syntax for time (P):

$(A:UK::P:UNIT_MULT[name])

== Time Variables ==
[[File:PublicationTable MovingTime.png|thumb|Publication Table Design, moving and converting the base time period]]

[[File:PublicationTable Materialised MovingTime.png|thumb|Moving and converting periods - showing the materialised table]]

When the data is time series (i.e. the [[Data_Structure_Definition_(DSD)_-_Structural_Metadata_Management|DSD]] has a Time Dimension), the [[#Observation_Key|Observation Key]] can include the time period, for example:

A:UK:EMP:'''2008'''

Where 2008 is the Observation Time Period to use. '''The format of the Time Period must always be the same format as the frequency for which the data are reported'''. For example if the data is annual, and an observation is reported for 2008, then the observation key must use 2008, not another frequency such as 2008-12-31. It is typical in a [[Data_Structure_Definition_(DSD)_-_Structural_Metadata_Management|DSD]] for the Frequency of the Series to be the first Dimension, in all the examples on this page this convention is applied, and as such the above key A:UK:EMP refers to Annual data, depicted by the initial dimension value 'A'.

If the key omits the time period, then the '''time period becomes a variable''' which can be set at table build time, if not set it defaults to the last period for which there is data.

A:UK:EMP

When a Publication Table has a variable time period, the resolved variable time value for this varaible is known as the '''base period'''. There can only be 1 base period for a Publication Table, however other Observation Keys (and corresponding headings) can perform time arithmetic from the base period for which the table is built.

The supported time functions include walking forwards and backwards in increments of the given frequency, converting to another frequency of data, and a combination of the two (convert and move, move and convert, move convert and move again). Ths supported functions are shown in the table below.

{| class="wikitable"
|-
! Syntax !! Description !! Example Observation Key !! Example Resolved Key
|-
| P || The base period (default if not provided) || A:UK:EMP:P || A:UK:EMP:2008 (base period=2008)
|-
| P-''n'' || Base Period minus 'n' Periods where n is a positive Integer || A:UK:EMP:P-1 || A:UK:EMP:2007 (base period=2008)
|-
| P+''n'' || Base Period plus 'n' Periods where n is a positive Integer || A:UK:EMP:P+1 || A:UK:EMP:2008-Q2 (base period=2008-Q1)
|-
| A(P) || Base Period converted to '''Annual''' || A:UK:EMP:A(P) || A:UK:EMP:2008 (base period=2008-Q2)
|-
| S(P) || Base Period converted to '''Bi-Annual''' (Semester) || S:UK:EMP:S(P) || S:UK:EMP:2008-S2 (base period=2008-Q3)
|-
| T(P) || Base Period converted to '''Tri-Annual''' || T:UK:EMP:T(P) || T:UK:EMP:2008-T3 (base period=2008)
|-
| Q(P) || Base Period converted to '''Quarterly''' || Q:UK:EMP:Q(P) || Q:UK:EMP:2008-Q4 (base period=2008)
|-
| M(P) || Base Period converted to '''Monthly''' || M:UK:EMP:M(P) || M:UK:EMP:2008-03 (base period=2008-Q1)
|-
| W(P) || Base Period converted to '''Weekly''' || W:UK:EMP:W(P) || W:UK:EMP:2008-04 (base period=2008-01-01)
|-
| D(P) || Base Period converted to '''Daily''' || D:UK:EMP:D(P) || D:UK:EMP:2008-12-31 (base period=2008)
|-
| H(P) || Base Period converted to '''Hourly''' || H:UK:EMP:D(P) || H:UK:EMP:2008-12-31T12 (base period=2008-02-02T12:00:03)
|-
| A(P-n) || Base Period minus (or plus) 'n' periods, converted to another frequency 'A' || A:UK:EMP:A(P-4) || A:UK:EMP:2007 (base period=2008-Q4)
|-
| A(P)-n || Base Period, converted to another frequency 'A' minus (or plus) n' periods || A:UK:EMP:A(P)-4 || A:UK:EMP:2004 (base period=2008-Q4)
|-
| A(P-n1)-n2 || Base Period, minus (or plus) 'n1' periods, converted to another frequency 'A' minus (or plus) 'n2' periods || A:UK:EMP:A(P-4)-4 || A:UK:EMP:2003 (base period=2008-Q4)
|-
| FA(P) || Subtracts 1 Period from the Converted Period IF the resolved Convert From end of period does not match the same point in time as the Convert To period.

'''Examples:'''
* Base Period =2008-Q4 Rule= FA(P) Output = 2008
* Base Period =2008-Q3 Rule= FA(P) Output = 2007 - the base period 2008-Q3 is not the same time as 2008 (end of period, so move back 1 year)
* Base Period =2008-Q1 Rule= FA(P) Output = 2007
* Base Period =2008-Q1 Rule= FA(P+3) Output = 2008
* Base Period =2008-Q1 Rule= FA(P+3)-1 Output = 2007

This works for ALL frequencies where the convert to frequency is lower then the base Period P, for example Quarter converted to Annual.

|| A:UK:EMP:FA(P) || A:UK:EMP:2007 (base period=2008-Q3)
|}

'''Compound expressions''' 
From Fusion Registry v11.5.1 and later it is possible to cast a period more the once in the same expression, for example

Q(FA(P-1))-1

If the input time period P=2019-Q4, then P-1 = 2019-Q3, the expression is rewritten as Q(FA(2019-Q3))-1. The Full Annual (FA) for 2019-Q3 is 2018, as the end of period 2019 does not match the end of period 2019-Q3, so the annual period moves back an additional period. The expression is rewritten as Q(2018)-1. 2018 to Quarterly moves to 2018-Q4 and then the last part of the expression steps back 1 period, the final period is 2018-Q3.

'''Notes on Time Period''' 
When converting from a lower frequency to a higher frequency (Weekly to Daily for example), the cast date always resolves to the end of the Period.

It is also important to note that as the [[#Observation Key|Observation Keys]] in these examples all have Frequency as the first Dimension, when converting to another frequency (Quarterly to Annual for example) the Observation Key changes the value for the Frequency Dimension, ensuring the correct series of data is retrieved.

'''Time Period variable in free text''' 
When refering to the Time Period as a variable in free text, for example in the table heading or a column heading, the variable is placed inside the variable syntax '''$()'''. For example $(P) would resolve to base period, and $(A(P)) would resolve to 'base period converted to annual frequency'. An example usage is shown below:

Table 1A showing data for $(P) to $(P+3)

Publication Table

2023-11-08T16:07:37Z

Mnelson: /* Calculated Cell Observation Key */

[[Category:Functions]]
[[Category:How_To]]

@See [[Publication_Table_Web_Services|Publication Table Web Services]]

= Publication Tables =
'''Introduced to Fusion Registry Enterprise v11.0.0.17'''

== Overview ==
[[File:Publication Table Definition.png|thumb|Using SDMX concepts of Agency, Id, Name, Description]]

[[File:Publication Table.png|thumb|Table Layout and Content Design]]

[[File:MaterialisedTable Preview.png|thumb|Showing the Table Preview feature which materialises the table with the given variables]]

A Publication Table is a curated table of data, where the values in the table rows, columns and table body cells can be individually specified. This enables a single table to pull data from multiple sources and piece the data together in a way which is not possible when dealing with standard cubes of data, whose table model is typically based on a Pivot model, with columns and rows built around the Dimension values to which the column or row is assigned. A Publication Table breaks out of this restrictive model by allowing cells of data to reference observations which do not need to share any relationship, in terms of dimensionality or even data source, with adjacent cells,.

Data in a Publication Table may be sourced from a single [[Dataflow_-_Structural_Metadata_Management|Dataflow]] or it may be sourced from multiple Dataflows enabling cross comparison of data.

Publication Tables can be static (every cell is a defined Observation) or dynamic, where a cell includes [[#Variable Dimension Members|variables Dimensions]] in the [[#Observation Key|Observation Key]], which can be provided at [[#Metadata vs Materialised|table build time]].

[[#Time Variables|Time arithmetic and frequency conversion]] is supported, enabling a single table to pull data from different time periods and frequencies which are all derived from the base period on which the table is built.

Publication Tables support [[#Calculated Observation Cells|calculations]], where each calculation can take input from one or more Observations over one or more [[Dataflow_-_Structural_Metadata_Management|Dataflows]].

Variable values also play a big role in Publication Tables, and includes standard variable placeholders – as well as advanced [[#Conditional Variables|conditional variables]], whose list of variable values change based on the value of another Dimension for which the table is built.

== Metadata vs Materialised ==

A Publication Table is stored in Fusion Registry the same way any other piece of structural metadata is stored. In this way it has an owner (Agency), and identity (ID) and a version. The change history is tracked using the same services, and the table definition can be [[Publication_Table_Web_Services|queried]] using the same SDMX web services for Structural Metadata.

The Publication Table metadata contains information about which Dataflows are used by the Table, which additional variables are being imported, and a definition of the table structure, and which Observations to import by referencing them by their unique [[#Observation Key|Observation Key]].

A Publication Table definition can be '''materialised''' into a table of Data by from the [[Publication_Table_Web_Services|Publication Table Web Service]], which combines the information of the Table Identity, and Variable Values to rewrite the [[#Observation Key|Observation Key]] into actual observation values. If building a web User Interface, then response from this service can be a packet of JSON, which describes the layout and content of the final Table in a way which is quick and simple to render for the User to see. The process of materialising the table is known as ‘'''build time'''’.

Additional web services are provided to enable clients to ask questions about the table such as ‘what are the earliest and latest time periods for which there is data for this table’ or ‘what possible variable values are there for this table’. This enables user interfaces to provide a user with the option to change the variables ‘on the fly’ ensuing they are able to build a table which is known to have data.

= Publication Table Cell Content =
== Overview ==
[[File:PublicationTable Layout.png|thumb|Publication Table Layout]]
A Publication Table is made up of a '''Table Head Rows''' and '''Table Body Rows'''.

The Table Head is made up of 1 or more table rows. Each Table Head cell may span 1 or more columns and 1 or more rows. The Table Head content is made up of free text and / or variable placeholders whose values are resolved when the table is built. Table Head Cells can be left empty.

The Table Body is made up of 1 or more Table Rows. The Table Rows provide the Publication Table with its data, by referencing Observations from the Dataflow.

At table build time all variables are resolved, and the corresponding data are retrieved from the data stores which hold the data.

== Table Rows ==
=== Overview ===
A Table Row consist of a Table Row heading, this can be a mix of free text and [[#Table Variables|variable text]]. The Table Row Heading is always the first Table Row Cell, and can not span multiple columns or rows. A Table Row can not have more then one heading. The remaining Table Row Cells are the Observation Cells, which may contain free text and [[#Table Variables|variable text]] but will typically contain a reference to an Observation by an [[#Observation Key|Obsevation Keys]] or a [[#Calculated Observation Cells|Calculation]] which can combine multiple [[#Observation Key|Obsevation Keys]].

A Table Row which results in no Observation data will not be included in materialised table at build time, unless the Table Row is a grouping row (see [[#Row Levels|Row Levels]] for details on grouping rows).

=== Row Levels ===
[[File:PublicationTabel Levels.png|thumb|Showing Levels in the Table Rows]]
The Table Row Heading can be used to define a row level. The level indicates to the client responsible for rendering the table the row's position in a hierarchy. Row levels are depicted using one or more leading chevrons ‘>‘ before the text. The number of hyphens indicates the level, with one leading hypen indicating level 1, two indicating level 2, and so on. It is possible to skip levels if required, i.e. Row 1 is level 1 with Row 2 assigned to level 3, skipping out level 2 entirely.

An example of row headings using levels is as follows:

Countries
> Europe
>> France
>> Germany
> Asia
>> China

Levels can be an important way to '''group data'''. A Table Row which acts as a parent row in a hierarchy, can be used as grouping mechanism for the rows that follow. This enables Parent Rows to be included in the output table, even if the Table Row contains no Observation data. The default behaviour for a Table Row with no data is to exclude it from the built table.

=== Looping Rows ===

Instead of explicitly defining a row for each Dimension member, it is possible to define a row as a loop (to iterate over all, or a subset of Dimension members). For example, if a publication table contains a Dimension for countries with an ID of REF_AREA then a loop can be set up to loop through all REF_AREA values, or a subset of the values - for which there is data. If the values are backed by a Codelist which has a hierarchy, the hierarchy can be preserved in the final table. It is also possible to nest a loop within another loop, for example for each REF_AREA output each INDICATOR.

====Syntax====
The loop is defined by giving the table row a label which starts with a syntax of :=LOOP([args]), like in the following example
:=LOOP([dimension id], [preserve hierarchy], [loop order], [explicit values to include])

'''Argument Definition'''
''dimension id'' = Required. The ID of the Dimension to loop over. For example CURRENCY would loop over all the Currency values which have data in the given context. The row label is the Currency, for example USD.
''preserve hierarchy'' = Optional, default false. If true the hierarchy of the Codes in the underlying Codelist will be built into the row hierarchy
''loop order'' = Optional, default 1. 1=Natural Order (as they appear in the Codelist), 2=By Code ID, 3=By Code Label
''explicit values to include'' = Optional, default include all values. A var arg list of values to loop over.

'''Full Example'''

:=LOOP(REF_AREA, true, 3, EUR,FR,DE,UK,ES)

====Simple Loop====
The loop is defined by giving the table row a label which starts with a syntax of :=LOOP([args]), like in the following example
:=LOOP(REF_AREA)

This loop definition will loop through all the REF_AREA values, building a row for each value. The Observation keys for the row can refer to the loop variable using a $(DIM_ID) placeholder in the Observation Key, for example:

Employed Student
:=LOOP(REF_AREA) A:$(REF_AREA):EMP A:$(REF_AREA):STU

The output table would look like the following:

Employed Student
United Kingdom 20 12
France 22 11
Germany 25 15

====Nested Loop====
If a loop row is followed by a row (or rows) which is an extra level deep, the child row(s) will be included for each loop, for example

Employed Student
> :=LOOP(REF_AREA)
>> Total A:$(REF_AREA):'''T''':EMP A:$(REF_AREA):'''T''':STU
>>> Male A:$(REF_AREA):'''M''':EMP A:$(REF_AREA):'''M''':STU
>>> Female A:$(REF_AREA):'''F''':EMP A:$(REF_AREA):'''F''':STU

Because the '''Total''' row is a child row of the LOOP (denoted by the number of leading '>') and '''Male'''/'''Female''' are children of '''Total''' - for each REF_AREA there will be a Total, Male and Female, outputting a table like the following:

Employed Student
United Kingdom
Total
Male 5 1
Female 4 2
France
Total
Male 6 2
Female 7 3

It is also possible to define a subloop as a loop. The previous example can be rewritten as:

Employed Student
> :=LOOP(REF_AREA)
>> :=LOOP(SEX) A:$(REF_AREA):'''$(SEX)''':EMP A:$(REF_AREA):'''$(SEX)''':STU

There is no limit to the number of subloops that can be defined.

====Loop Hierarchical Values====
If the underlying Codelist has a hierarchy, a subsequent argument can be provided to ask for the hierarchy to be preserved

Countries
> :=LOOP(REF_AREA, true)

'''Note:''' A hierarchy will keep rows with no data if they are acting as a parent node for rows that do have data (e.g. there may not be data for Europe, but the row exists because there are child rows of France and Germany with data).

Nested loops can also preserve hierarchies for example:

:=LOOP(REF_AREA, true)
>:=LOOP(SEX, true)

With an example output (assuming no data exists for the hierarical parents (Continent, Europe)

Continent
Europe
France
Total (M+F)
Male
Female
Germany
Total (M+F)
Male
Female

====Order Loop Rows====
It is possible to order the labels by their natural order (that of the Codelist), by id or by label, using the numbers 1, 2, or 3 respectively. The following outputs the rows ordered by label.

Countries
> :=LOOP(REF_AREA, true, 3)

====Fixed Loop List====
It is also possible to define a subset of the values to loop over. For example, if REF_AREA has data for 100 countries, but only 3 are required in the table. In this scenario, the natural order is that of the provided list, not of the underlying Codelist.

Countries
> :=LOOP(REF_AREA, true, 3, UK, FR, DE)

====Loop over External Variable====
Instead of looping the values of the Dimension's Codelist, it is possible to loop another Codelist - the purpose for this is to support alternative labels (if the labels for the publication table need to be different from those of the Dimension members) - or a subset (as an alternative to passing the subset of values into the LOOP argument), or to define a hierarchy (if the Dimension's Codelist does not have a hierarchy, or an alternative is required).

In order to loop over an external Variable, the Publication Table must be linked to an external Codelist/Valuelist against an Alias. For example EXT=My Country Codelist with Hierarchy. The corresponding loop then loops the Codes in the external list, and stores these the corresponding Dimension ID variable, for example:

Countries
> '''REF_AREA''':=LOOP('''EXT''')

Noting that the loop is over the variable EXT which resolves to an external Codelist. The loop is assigned to the REF_AREA Dimension.

=== Anchor Rows ===
The default behaviour is for a row to be omitted if the row has no data and it has no child rows with data. However, if the desired result is to always output the row, then this can be achieved by using the [anchor] tag in the row header label.

Consider the following Publication Table definition:

Male Female
Employed A:M:EMP A:F:EMP
Student A:M:STU A:F:STU
Retired A:M:RET A:F:RET

If there are no data for Student, then the materialised table would look like this

Male Female
Employed 12 13
Retired 3 5

==== Anchor Single Row [anchor] ====
To anchor a single row, add the '''[anchor]''' tag at the start of the row

Male Female
Employed A:M:EMP A:F:EMP
[anchor]Student A:M:STU A:F:STU
Retired A:M:RET A:F:RET

Then the output table would preserve the 'Student' row

Male Female
Employed 12 13
Student
Retired 3 5

This rule can be coupled with [[Publication_Table#Formatting_Missing_Observations|formatting for missing observations]], then the table could be enhanced to look something like this

Male Female
Employed 12 13
Student - -
Retired 3 5

==== Anchor Child Rows [anchor+] ====
For a row with a hierarchy, an '''[anchor+]''' tag can be placed on a parent node to anchor the parent and all of its children, for example:

Continent
[anchor+]>Europe
>>France
>>>Total (M+F)
>>>>Male
>>>>Female
>>Germany
>>>Total (M+F)
>>>>Male
>>>>Female

The above rule would always output Europe and all of its descendants (France, Total, Male, Female, Germany).

==== Anchor Siblings and Child Rows [anchor*]====
The '''[anchor*]''' tag can be used as a way to anchor all following siblings and child nodes, this can be a useful mechanism to anchor all rows for the whole table by placing the anchor[*] on the first row of the table.

==== Anchor Loops ====
If a loop is preceeded by an '''[anchor]''' tag then the system will have the following behaviour:

* If an external Codelist variable is being looped, all codes will be ouput regardless of what data is stored in the system
* If a Dimension is being looped, all codes in the Codelist used by the Dimension will be output regardless of what data is stored in the system
* If a Dimension is being looped and it is not coded (i.e it is free text or numerical or other) then the [anchor] has no effect, as the looped values are based on what data exists

== Observation Key ==
=== Overview ===
The purpose of the Observation Key is to reference a Observation whose value will be resolved at table build time. The Observation Key can explicitly resolve to 1 observation value, if all parts of the Key are provided. However, it is possible to make parts of the key varaibles, allowing the values to be set at table build time.

An Observation Key is made up of 4 parts:

* Dataflow Reference
* Series Reference
* Time Period Reference
* Measure Reference

An example Observation Key with all of these parameters in place is:

DF_REG.M:UK:SUR:2008:BIRTHS

The key is broken down into the following parts:

{| class="wikitable"
|-
! Observation Key Part !! Name !! Description !! Required
|-
| DF_REG|| Dataflow Alias|| This identifies the Dataflow which has the data. The identification uses the alias, which is assiged to the Dataflow in the Publication Table definition. The Dataflow Alias is only required if the Publication Table references more then one Dataflow. If the Publication Table references more then one Dataflow, it can assign a Dataflow to a Default Alias, which means that it is required as part of the Observation Key. || No (if only 1 Dataflow or a default alias is set)
|-
| M:UK:SUR || Series Key || This identifies the Series of Data for the Dataflow, by providing a single value for each of the Dimension Members of the Dataflow, separated by a ‘:’ character. In this example the Dataflow has 3 Dimensions (excluding time), the value for the first is 'M', the second 'UK' and the last 'SUR'. As the Dataflow in this example has a Time Dimension this Series Key will uniquely identify a Series of Observations where each Observation refers to a unique Time Period || Yes (but can omit values)
|-
| 2008 || Time Period || This identifies the Time Period in the Series of Data identified by the previous section. The Combination of the Series and Time Period for a Time Series Dataflow will uniquely identify the Observation || No. Omit to make it variable for Time Series Data (defaults to last period). Not required if there is no Time Dimension for the DSD.
|-
| BIRTHS || Measure Dimension || If the Dataflow uses a Data Structure with multiple Measures, then the Measure ID must be included as the last part of the Observation Key. In this example the Dataflow has multiple measures (BIRTHS/DEATHS/MARRIAGES) so the Observation Key needs to include this value. || When there is only 1 measure it is not required, when there are multiple measures, it is required.
|}

Observation Keys are included on the materialised table in JSON format enabling the observation value to be tied back to the key. Observation Keys also provide a mechanism by which [[#Formatting_Cells|formatting rules]] can be applied to a subset of table cells.

=== Variable Dimension Members ===
It is possible to omit Dimension Values from the Observation Key, when this is the case that Dimension will become a variable – whose value can be set at table build time. For example, the second Dimension value of ‘UK’, from the previous example, can be omitted like this:

M::SUR:2008

The Publication Table can now be built for any Country by passing in the variable at table build time - i.e. UK, FR, DE. By passing in different variable values, the Observation Key will change, and as such it will reference a different Series in the Dataset.

When an Observation Key has a variable Dimension member, the ID of the Dimension can be referenced in text cells to create variable headings. For example, if the ID of the 2nd Dimension in the DSD is '''REF_AREA''', the Table Column could be given a title of $(REF_AREA) which would resolve to 'United Kingdom', 'France', or 'Germany' at build time depending on which varaible value is used to build the table. [[#Dimension Variables|#See Dimension Variables]].

=== Variable Time Period ===
For Time Series dataset, the Time Dimension can form part of the Observation Key, for example M:UK:SUR:'''2008'''. The 2008 part of the key refers to the year 2008.

It is possible to omit the value for Time Period in the Observation Key. Omitting Time Period will make the Time value a variable which can be provided at table build time, defaulting to the last period for which data exists if no value is supplied.

Extending the above example to omit the Time value results in an Observation Key whose Country and Time values can be provided to the web service at table build time.

M::SUR

'''Note:''' The above key is exactly the same as using the variable Time Period value 'P' in the Time Dimension, as follows:

M::SUR:'''P'''

Additional functions can be performed on the Time Dimension, such as moving forwards or backwards from the table base period, [[#Time Variables|#see Time Variables]] for more details.

=== Free Text Observation Keys ===
Whilst typically a cell in the body of the Publication Table would resolve to an Observation value, it is possible to use it for free text. To include free text cells, start the cell value with a leading apostrophe character, for example:

'Cell Text

'''Note''': An Observation Key can '''not''' contain a mix of free text as well as an Observation value

== Calculated Observation Cells ==
[[File:TableDefinition Calculations Variables.png|thumb|Publication Table with Calculated Cells and Variable Dimensions]]

An Observation Cell may contain calculated values, which can consist of a mix of simple algebra (+, -, *, /) and a reference to one or more Observations which may come from multiple Dataflows.

To define a Calculated Observation the Cell the following rules apply:

* The Observation Key must start with :=
* Any Observation Keys must appear within enclosing square brackets '[' and ']'
* An Observation key may embed calculations on a single Dimension by using parenthesis '(' and ')' in the key part
* Numbers in Observation Keys are treated as Code IDs, unless prefixed with a '#'
* The same rules for variable Dimensions and Time Period apply when resolving the Observation Keys used in the calculation

Examples are shown in the table below

{| class="wikitable"
|-
! Example Syntax !! Description
|-
| :=[A:UK:EMP]+10 || Add 10 to the observation value retrieved from the key A:UK:EMP
|-
| :=[A:(UK+FR+DE):EMP] || Add the series A:UK:EMP + A:FR:EMP + A:DE:EMP
|-
| :=[A:(UK+FR+12):EMP] || Add the series A:UK:EMP + A:FR:EMP + A:DE:12
|-
| :=[A:(UK+FR+#12):EMP] || Add the series A:UK:EMP + A:FR:EMP and add the Constant '12'
|-
| :=[A:UK:EMP]+[A:FR:EMP]+[A:DE:EMP] || Add the series A:UK:EMP + A:FR:EMP + A:DE:EMP
|-
| :=[A:(UK+FR):EMP]/[A:(UK+FR):UMP] || Adds the series A:UK:EMP + A:FR:EMP and divides by the result of A:UK:UMP + A:FR:UMP
|-
| :=((100/[A:UK:EMP:(M+F):P-1])*[A:UK:EMP:(M+F)])-100 || Demonstrates the ability to use parenthesis outside of the Observation Keys to control the order of mathematical operations
|-
| :=[POP.M:UK:(M+F)] / [LAB.M:UK:EMP] || Demonstrates the ability to calculate data from two Dataflows (defined by alias POP and LAB)
|-
| :=[M:UK:EMP:M(P)] + [A:UK:EMP:A(P)] || Demonstrates the ability to calculate data where the period is converted from the base period
|}

=== Calculated Cell Missing Values ===
If any input Observations are missing in the calculation, the final result of the calculation is an empty value. For example:

:=[A:(UK+FR+DE):EMP]

This would not output a value if A:UK:EMP, A:FR:EMP or A:DE:EMP have no corresponding Observation value for the time period the table is being built for.

RFS version 11.12.0 and higher allow for assignment of default value(s) to missing components of the calculation. The syntax uses the pipe '|' symbol, the semantic meaning is 'use this value | or this value if missing'. An example of the pipe symbol being used in a calculation expression is shown below:

:=[A:(UK|1+FR|1+DE|0):EMP]

This expression substitutes the UK for the value of 1 if there is no corresponding Observation for A:UK:EMP. The Observation A:FR:EMP is substituted for 2 if missing, and A:DE:EMP is substituted for 0. Therefore is all 3 Observations are missing, the final value for the expression = 1+1+0 = 2.

It is possible to substitute the entire block if any part is missing by appending the double pipe '||' to the end of the expression. The following example results in a zero value if any input Observation (UK, FR, DE) value is missing:

:=[A:(UK+FR+DE):EMP||0]

The above expression is equivalent to the following expression:

:=[A:UK:EMP||0] + [A:FR:EMP||0] + [A:DE:EMP||0]

=== Calculated Cell Observation Key ===
Calculated Cells have implicit [[#Observation_Key|Observation Keys]] which are derived based on the inputs series in the calculation. The [[#Observation_Key|Observation Key]] can be used to assign [[#Formatting_Cells|Formatting Rules]] in the same way as a 'standard key' can be used. It is possible to assign an explicit key to the calculation, which will override the implicit key assigned. The following table shows how the key assignment works.

For example
{| class="wikitable"
|-
! Calculation !! Output Key || Explanation
|-
|:=[A:(UK+FR+DE):EMP] || A:#:EMP:2009 || The first and last dimension have a fixed value, the second dimension has a variable value and is replaced by '#'. The time period is added to the end of the key.
|-
|:=((100/[A:UK:EMP:(M+F):P-1])*[A:UK:EMP:(M+F)])-100 || A:UK:EMP:#:# || The first three dimensions have a fixed value, the last dimension has a variable value and is replaced by '#'. The time period is a variable as there are 2 time inputs to the calculation (P and P-1)
|-
|:=PCTOF([A:M:UK:P],[A:TOT:UK:P]) || A:#:UK:2009 || The first and last dimension have a fixed value, the second dimension has a variable value and is replaced by '#'. The time period is added to the end of the key.
|-
|:=[A:UK:EMP]+10 || A:UK:EMP || The input key is the same as the output key as none of the dimensions have a variable value. The time period is added to the end of the key.
|-
|A:UK10:EMP:=[A:UK:EMP]+10 || A:UK10:EMP:2009 || The output key is explicitly assigned. The time period is added to the end of the key.
|}

The output key conforms to the same Dimension breakdown as the underlying Data Structure. If a Observation Key is explicitly assigned, it must also honour the same Dimension breakdown. This allows formatting rules to be applied using the key component syntax, for example COUNTRY=UK10 would search for Observation Keys whose Country Dimension is reporting the value UK10.

== Percentage Of ==
The PCTOF (Percentage Of) function can be used to calculate 'x' as a percentage of 'y' following the syntax:

PCTOF(x,y)

This is reolved to the following mathmatical expression: '''(100/y) * x'''

x and y must either be a numerical value, and observation key, or an expression using a combination of the two. For example

:=PCTOF([A:M:UK:P],[A:TOT:UK:P])

The Observation Key follows the same standard rules with respect to varaibles and inline calculations - the following example shows an Observation Key where Male + Female values are added with the result being used in the Percentage Of function

:=PCTOF([A:M:UK],[A:('''M+F'''):UK])

A more complex example shows calculations around the PCTOF function:
:=(10 + PCTOF([A:M:UK:A(P)-1],[A:(M+F):UK:A(P)])) / 2

== Footnotes ==

[[File:Footnotes3.png|thumb|A table with footnotes]]
[[File:Footnotes2.png|thumb|Footnotes in the table builder]]

=== Overview ===
Footnotes can be added to any part of the publication table (table column/row header, or observation cells). The output table is generated with a footnote number, with the list of footnotes at the end of the table.

Footnotes need to be thought about in 2 parts: Part 1 are the list of footnotes which are defined at the Publication Table level. Part 2 are the references to the footnotes from within the table. This separation of allows footnotes to be reused within the same table, and when working with [[#Footnotes_with_Placeholders|placeholders]] footnote templates can be set up where the text changes based on the placeholder values defined in the table builder.

To add a footnote, add it to the Publication Table, for example: 
Footnote 1 
Footnote 2

Then in the table builder, add a reference to which footnote to use, $(1) references '''Footnote 1''', and $(2) references '''Footnote 2'''.

'''Note:''' The order of the list of footnotes in the table definition may not be reflected in the order they are output. As such a footnote referenced by $(1) may be output as the second footnote.

=== Footnotes with Variable ===
A footnote can reference variables, using the [[#Table_Variables|variable syntax]],

A footnote for $(REF_AREA.UK)

A footnote can even output an observation value, observation attribute or series attribute.

Measure Value
A footnote with obs value $(A:UK:EMP:2008:OBS_VALUE)

Observation Attribute
A footnote with obs attribute $(A:UK:EMP:2008:OBS_COMMENT)

Series Attribute
A footnote with obs attribute $(A:UK:EMP:P:SERIES_TITLE)

When combining variable footnotes with placeholder values and loops, it is possible to dynamically create references to observation attributes.

'''Note:''' the use of P in the final cell key which is a reference to the [[#Time_Variables|time period]] for which the table is being built.

=== Footnotes with Placeholders ===
It is possible to write a generic footnote for the publication table, which includes placeholders. When the footnote is referenced, the placeholder values are provided. The syntax for a placeholder is {n} where n is a positive integer, for example {1}.

Footnote Example

A footnote for $(REF_AREA.{1}) and $(REF_AREA.{2})

In the table builder, the footnote is referenced as usual, but with arguments for the placeholders:

Footnote reference in Table builder

My Table Row '''$(1,UK,FR)'''

The footnote reference in the above example is in bold, and consists of which footnote text to use (in this case the first one in the list, denoted by '1'), and 2 arguments are passed for the placeholders, placeholder 1 takes the value of UK and placeholder 2 takes the value of FR. During the materialise process the intermediate text is generated:

A footnote for $(REF_AREA.UK) and $(REF_AREA.FR)

Which then goes through the variable resolve process, in this case both variables, '''$(REF_AREA.UK) and $(REF_AREA.FR)''', match the [[#Coded_Variables|Coded Variable]] syntax and would resolve to:

Final Output
A footnote for United Kingdom and France

=== Footnotes in Observation Cell ===
Any number of footnotes can appear in an Observation cell, simply put the footnote(s) at the end of the [[#Observation_Key|Observation Key]].

Standard Footnote Reference 
DF_REG.M:UK:SUR:2008:BIRTHS $(1) $(2)

Footnote with Placeholder 
DF_REG.M:UK:SUR:2008:BIRTHS $(1,UK)

=== Footnotes in Loops ===
A row [[#Looping_Rows|loop]] contains a locally scoped variable, and as such it can be used when calling a footnote with a placeholder.

Example Footnotes
$(A:T:{1}:EMP:P:SERIES_TITLE)
$(A:T:{1}:EMP:P:TITLE_COMPL)
$(A:{1}:{2}:EMP:P:OBS_COMMENT)

Example Table Definition
Employed Student
> :=LOOP(REF_AREA) '''$(1,$(REF_AREA[id]))'''
>> Total '''$(2,$(REF_AREA[id]))''' A:$(REF_AREA):T:EMP A:$(REF_AREA):T:STU
>> :=LOOP(SEX) '''$(3,$(REF_AREA[id]),$(SEX[id]))''' A:$(REF_AREA):$(SEX):EMP A:$(REF_AREA):$(SEX):STU

Footnote Intermediate Resolution
Row 1 (REF_AREA=UK)
$(A:T:UK:EMP:P:SERIES_TITLE)

Row 2 (REF_AREA=UK)
$(A:T:UK:EMP:P:TITLE_COMPL)

Row 3 (REF_AREA=UK, SEX=M)
$(A:UK:M:EMP:P:OBS_COMMENT)

Final Footnote output
Row 1 (REF_AREA=UK)
This is a series for the United Kingdom

Row 2 (REF_AREA=UK)
This is a title compliment for the United Kingdom

Row 3 (REF_AREA=UK, SEX=M)
This is an observation comment for Males in the United Kingdom

= Formatting Cells =
== Formatting Numerical Observation Values ==
[[File:Obs formatting.png|thumb|Number Formatting Options]]
Numerical observation values can be formatted in the following way

# '''Number of Decimal Places''' or '''Significant Figures''' can be defined
# For Decimal places this can be set to a maximum rule or a fixed rule, i.e the value 1.3 to 2 decimal places is 1.3 using the maximum rule and 1.30 using the fixed rule
# '''Rounding mode''' can be specified
# '''Scaling Factor''' this is a multiplier applied to the observation value, the multiplier is x10^[scaling factor]. Example, a value of 1.1 with a scaling factor of 1 is 11, a scaling factor of 2 is 110, a scaling factor of 3 is 1100. If the Observation or related Series has a UNIT_MULT concept the difference between the unit multiplier and scaling factor will be taken. Example, an Observation with a value of 1.1 and a unit multiplier of 9 (Billions) is reported, the publication table has a scaling factor of 6 (millions) - the number 1.1 is scaled using 10^3 (calculated by 9-6), the 1.1 billion is converted to 1100 million.
# '''Scientific notation''' can be requested on output, i.e 123456 to 2 significant figures is 120000 in standard form or 1.2E4 in scientific form
# '''Decimal formatting''' for long numbers, i.e. 12345.8 there is not thousands separator by default, to include a thousand separator set the decimal formatting rule to either a single Language (i.e. French Formatting) or Dynamic formatting, which uses the Accept-Langauge locale/language sent be the web browser (this can be overridden using the locale=[loc] query parameter. The formatting rules can also influence which characters the numerical characters used.
# Rules can be applied to all series, or specific series

=== Match Series ===
When creating a formatting rule, it is possible to define which series the rule applies to. If left blank, the rule will apply to all observation values.

The syntax for defining specific series is shown in the following table:

{| class="wikitable"
|-
! Rule Type !! Syntax !! Description
|-
| Series Match || =A:UK:EMP || Match a specific series from the default Dataflow
|-
| Series Match Multiple Values || =A:UK+FR+DE:EMP || Multiple codes in 2nd Dimension (UK, FR and DE) any can match for the rule to be valid
|-
| Series Match by Dataflow Alias || =FLOW2.A:UK:EMP || Match a specific series for the Dataflow with alias FLOW2
|-
| Wildcard Series Match || =A::EMP || Match all series with the given dimension values (2nd Dimension is wildcarded)
|-
| Not Rule || !=A:UK:EMP || Match all series which do not match this series (wildcard series also supported)
|-
| Dimension Rule || CURRENCY=GBP || Match all series which have CURRENCY of GBP
|-
| Not Dimension Rule || !CURRENCY=GBP || Match all series do not have a CURRENCY of GBP
|-
| Multiple Rules || CURRENCY=GBP,FREQ=M || Comma is used to include multiple matches (also applies to series match)
|}

== Formatting Missing Observations ==

Observation Values can be missing for 2 reasons:
# There is no Observation in the database for the given Observation Key
# There is an Observation in the database but the measure value is either: NaN, null, or an empty string

In both cases it is possible to define an output value using formatting rules. For the first case, the observation does not exist, it is possible to define a default output value, for example ‘-‘. For the second case, where the observation does exist, it is possible to define a set of rules based on another Component of the Observation, for example the Observation output may be related to the OBS_STATUS Attribute.

When the formatted value is based on a Component, a set of rules must be defined which state ‘when component=x output y’, where x may be an fixed value or a pattern to match (using a regular expression). The rules for mapping component values are defined in a [https://fmrwiki.sdmxcloud.org/Representation_Map Representation Map], and the Publication Table simply provides the connection to the Representation Map to use. This enables reuse of the mapping rules across multiple Publication Tables. The Publication Table must define two properties in order to support Component Mapping rules:

# The Component to obtain the value from i.e OBS_STATUS
# A link to the [https://fmrwiki.sdmxcloud.org/Representation_Map Representation Map], which holds the mapping rules. Note, the Representation Map is expected to have 1 source mapping, this could be of any type (string, codelist, other)

An example Reprsentation Map for OBS_STATUS is shown below:

{| class="wikitable"
|-
! OBS_STATUS !! Is Reg Ex !! Missing Value !! Description
|-
| M || false || mv || When OBS_STATUS is M output mv
|-
| L || false || np || When OBS_STATUS is L output np
|-
| .* || true || - || When OBS_STATUS is anything else output -
|}

== Frequency Formatting ==
[[File:PublicationTable FreqMap.png|thumb|Representation Map with frequency mapping rules]]
[[File:PublicationTable Materialised FreqMap.png|thumb|Materialised table with formatted time headings ]]
Time Variables always resolve themselves to an [[SDMX_Time_Formats|SDMX Date Format]] when forming the [[#Observation Key|Observation Key]], this ensures the correct observation value is returned from the data store. When a time variable is used as a label, in a table heading for example, the same formatting will be applied by default but a custom format can be provided. Custom display formats for a Time Period enable a time period such as 2007-Q1 to be resolved to custom text string, for example '31 Mar 2007'.

Custom Time formats rules are defined by providing an association between an SDMX Frequency ID (#see [[SDMX_Time_Formats|SDMX Date Format]]) and the output format template. The output format template is a string that defines which components of time are output an in which order and style. For example to output a date as 'March 2001' the format template would look like this 'MMMM yyyy' to indicate a full month name (MMMM) and a 4 character year (yyyy). format rules make use of the [https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html Simple Date Format] which is used by Java as such all of the documentation on Simple Date Format applies.

Frequency Format Mapping is not defined as part of a Publication Table, instead the rules are defined in a [https://fmrwiki.sdmxcloud.org/Representation_Map Representation Map] and the Publication Table references the Representation Map as the source of its frequency conversion rules. This enables the same set of rules to be used for multiple Publication Tables. The Representation Map must map 1 source (Codelist or String) to 1 target (String), it can contain as many mapped frequencies as required. If the Representation Map contains a mapping for a value that is not a valid SDMX Frequency, then it will be ignored.

It is important to note that all time periods resolve to a single point in time, which is the end of the period, for example an Annual period of 2008 resolves to 2008-12-31:T23:59:59. The resolution of frequency to a point in time enables the frequency formatting rules to use all aspects of time, even when converting a low frequency, for example 2008 formatted using the pattern dd MMM yy would resolve to '31 Dec 2008'.

== Variables Summary ==

{| class="wikitable"
|-
! Variable Type !! Description !! Syntax !! Example !! Example Output
|-
| '''Dataflow Name''' || Displays the name of a Dataflow referenced by the Publication Table in the locale in which the Publication Table is built || $([dataflow alias]) || $(EDU) || World Bank, Education
|-
| '''Dataflow Description''' || Displays the description of a Dataflow referenced by the Publication Table in the locale in which the Publication Table is built || $([dataflow alias].desc) || $(EDU.desc) || World Bank, Education
|-
| '''Dataflow Id''' || Displays the ID of a Dataflow referenced by the Publication Table || $([dataflow alias].id) || $(EDU.id) || DF_EDU
|-
| '''Variable Dimension Value''' || Displays the name of the variable used when an Observation key has a wildcard member || $([dimension id]) || $(REF_AREA) || United Kingdom
|-
| '''Specific Component Value''' || Displays the name of the Component Value || $([component id].[component value]) || $(OBS_STATUS.C) || Confidential
|-
| '''Component Name''' || Displays the name of the DSD Component (this is the name of the Concept used by the Component) in the locale in which the Publication Table is built || $([component id].name) || $(OBS_STATUS) || Confidentiality Status
|-
| ''' Enumeration Value ''' || Displays the name of a Code in a Codelist or Value in a Valuelist || $([list alias].[list item ID] || $(SEX.M) || Male
|-
| '''Conditional Variable''' || Displays the variable value, which is conditional on the value of a variable Dimension || $([conditional variable id][variable id]) || $(BANKNOTES[NOTE1]) || £5
|-
| ''' Time ''' || Displays the base time period for which the table is built, or an offset from this time period / cast to a different frequency || see [[#Time Functions|Time Functions]] || $(P) || 2008
|}

= Table Variables =
== Overview ==
Variables are a very important part of Publication Tables. They can be used to make the content of the table dynamic, setting and changing the value at run time to change the table content. Variables can be used in Observation Keys, but the values of variables can also be used anywhere where there is text, for example the table heading, subheading, column or row headings, footnotes, and Observation Cells with free text.

The general syntax for a variable is

$(''variable id''.''sub part''[''detail''])

Where variable id is always required, the subpart and detail are optional. Subpart is used if the variable resolves to a Component and a coded value is required, the detail array part obtains the id, name, or description of the underlying SDMX type.

This section describes the various variables that exist, along with the syntax of how to use them.

== Dataflow Variables ==
Publication Tables define data cells across one or more Dataflows. Each ‘imported’ Dataflow is given an alias, which can be used to refer to the Dataflow in the [[#Observation Key|Observation Key]].

It is possible to refer to the Dataflow's Name, Description, and Id using the following variable syntax.

'''Default Dataflow''' 
Dataflow Id
$([id])

Dataflow Name
$() or $([name))

Dataflow Description
$([desc])

'''Dataflow with Alias=EXR''' 
Dataflow Id
$(EXR[id])

Dataflow Name
$(EXR) or $(EXR[name))

Dataflow Description
$(EXR[desc])

== Dimension Variables ==
When an [[#Observation Key|Observation Key]] has [[#Variable Dimension Members|variable Dimension Values]], the table will be built by replacing this variable with a value to complete the Observation Key. The variable value is either passed explicitly to the web service, or defaulted by the system if not provided. The Dimension Variable can be referenced in the table using the syntax $([dimension id]), for example:

Dimension Value Name
$(INDICATOR) resolves to 'Employment'

This would resolve to the '''name''' of the Indicator value used in the table, for example '''EMP''' may resolve to the label '''Employment'''. The label for the Dimension Value is obtained from the Codelist or Valuelist which the Dimension uses. If the Dimension does not use a list (i.e. it is free text or numerical) then the Dimension value is simply used in place of the variable placeholder.

To resolve to the '''id''' of the value for the Component, for example '''EMP''', use the following syntax

Dimension Value Id
$(INDICATOR[id]) resolves to 'EMP'

== Dependent Dimension Variable ==
It is possible that the value for one Dimension is dependent on the value of another Dimension, for example CURRENCY may be dependent on the COUNTRY and COUNTRY is a variable in the [[#Observation Key|Observation Key]] - it is possible to specify that the value to use for the CURRENCY Dimension can be derived using a lookup table. The lookup table in this instance is maintained as an [https://fmrwiki.sdmxcloud.org/Representation_Map SDMX Representation Map] which is used to map COUNTRY to CURRENCY. The Publication Table specifies that the two Dimensions are related and provides a reference to the Representation Map to use which has the rules.

When working with a dependent variable, both the dependent variable and the Dimension on which it depends should be omitted from the [[#Observation Key|Observation Key]] - for example consider a full key of:

A:UK:GBP:2007

Where UK is the COUNTRY and GBP is the Currency, to make the COUNTRY a variable Dimension, it should be omitted from the key, as follows:

A::GBP:2007

To then make the CURRENCY a variable it is also omitted , as follows:

A:::2007

The Publication Table is given the rule that CURRENCY depends on COUNTRY and should use the COUNTRY_2_CURRENCY [https://fmrwiki.sdmxcloud.org/Representation_Map Representation Map], which in this example looks like the following:

{| class="wikitable"
|-
! COUNTRY !! Is Reg Ex !! CURRENCY
|-
| UK || false || GBP
|-
| US || false || USD
|-
| .* || true || EUR
|}

When the Publication Table is materialised the [[Publication_Table_Web_Service|web service]] only requires the COUNTRY information, and it will complete the Observation key using the representation map lookup table.

'''Note''' it is possible to override the mapped value by passing in an explicit override to the web service. For example passing to the web service both COUNTRY=UK with CURRENCY=EUR will override the mapping behaviour for CURRENCY.

== Component Values ==
Components of the underlying Data Structure can be explicitly referenced using the following syntax '''$([component id])''', for example:

Component Name
$(OBS_STATUS) or $(OBS_STATUS[name]) resolves to 'Observation Status'

Component Description
$(OBS_STATUS[desc]) resolves to 'Information on the quality of a value or an unusual or missing value.'

== Coded Variables ==
To reference a Code value, the syntax is $([component id / codelist alias].[item id]). The first variable part is the ID of a Component in the underlying Data Structure, or alternativly the alias of an external variable defined on the Publication Table. An external variable is a link to an SDMX Codelists or Valuelist. Example are:

Name of Code
$(CURRENCY.$) resolving to the text '''Dollar''

Description of Code
$(DWELLING.SM[desc])

== Conditional Variables ==
A Conditional Variable is one whose value is conditional on the value of a [[#Dimension Variables|Dimension Variable]].

An example use case is Banknotes, where the values for NOTE1, NOTE2, or NOTE3 is dependant on the Country Dimension. A Conditional Variable could be used to set a values for each variable (NOTE1, NOTE2, NOTE3) based on the value used for the current Country Dimension i.e. US NOTE1=$5, UK NOTE1=£5.

The syntax of referring to a Conditional Variable is $([conditional variable id][variable id]), for example
$(BANKNOTE[NOTE1])

== Reported Attribute for an Observation or Series ==
A variable can be a reference to a part of an Observation, for example the Unit Multiplier (UNIT_MULT). A Table Row header, for example, could include variable text such as 'Number of payment accounts (''thousands'')' where the word '''thousands''' is a variable because it depends on other variables which build the key.

To achieve this, use the syntax:

$(A:UK::2008:UNIT_MULT[name])

The observation key in this case is A:UK::2008, where the third Dimension is a variable as it has no value (in the main body of the table this dimension would probably also be a variable, which means the table can be built with different values, each value may change the output of this UNIT_MULT value). The last part of the key is the Attribute ID (UNIT_MULT) - this can be a reference to a series attribute, or an observation attribute. The [name] indicates that it should resolve to the name of the reported attribute value, alternative values are '''id''' for the reported value or '''desc''' for the description of the reported value.

If time is also a variable, then the key can change accordingly, using the variable syntax for time (P):

$(A:UK::P:UNIT_MULT[name])

== Time Variables ==
[[File:PublicationTable MovingTime.png|thumb|Publication Table Design, moving and converting the base time period]]

[[File:PublicationTable Materialised MovingTime.png|thumb|Moving and converting periods - showing the materialised table]]

When the data is time series (i.e. the [[Data_Structure_Definition_(DSD)_-_Structural_Metadata_Management|DSD]] has a Time Dimension), the [[#Observation_Key|Observation Key]] can include the time period, for example:

A:UK:EMP:'''2008'''

Where 2008 is the Observation Time Period to use. '''The format of the Time Period must always be the same format as the frequency for which the data are reported'''. For example if the data is annual, and an observation is reported for 2008, then the observation key must use 2008, not another frequency such as 2008-12-31. It is typical in a [[Data_Structure_Definition_(DSD)_-_Structural_Metadata_Management|DSD]] for the Frequency of the Series to be the first Dimension, in all the examples on this page this convention is applied, and as such the above key A:UK:EMP refers to Annual data, depicted by the initial dimension value 'A'.

If the key omits the time period, then the '''time period becomes a variable''' which can be set at table build time, if not set it defaults to the last period for which there is data.

A:UK:EMP

When a Publication Table has a variable time period, the resolved variable time value for this varaible is known as the '''base period'''. There can only be 1 base period for a Publication Table, however other Observation Keys (and corresponding headings) can perform time arithmetic from the base period for which the table is built.

The supported time functions include walking forwards and backwards in increments of the given frequency, converting to another frequency of data, and a combination of the two (convert and move, move and convert, move convert and move again). Ths supported functions are shown in the table below.

{| class="wikitable"
|-
! Syntax !! Description !! Example Observation Key !! Example Resolved Key
|-
| P || The base period (default if not provided) || A:UK:EMP:P || A:UK:EMP:2008 (base period=2008)
|-
| P-''n'' || Base Period minus 'n' Periods where n is a positive Integer || A:UK:EMP:P-1 || A:UK:EMP:2007 (base period=2008)
|-
| P+''n'' || Base Period plus 'n' Periods where n is a positive Integer || A:UK:EMP:P+1 || A:UK:EMP:2008-Q2 (base period=2008-Q1)
|-
| A(P) || Base Period converted to '''Annual''' || A:UK:EMP:A(P) || A:UK:EMP:2008 (base period=2008-Q2)
|-
| S(P) || Base Period converted to '''Bi-Annual''' (Semester) || S:UK:EMP:S(P) || S:UK:EMP:2008-S2 (base period=2008-Q3)
|-
| T(P) || Base Period converted to '''Tri-Annual''' || T:UK:EMP:T(P) || T:UK:EMP:2008-T3 (base period=2008)
|-
| Q(P) || Base Period converted to '''Quarterly''' || Q:UK:EMP:Q(P) || Q:UK:EMP:2008-Q4 (base period=2008)
|-
| M(P) || Base Period converted to '''Monthly''' || M:UK:EMP:M(P) || M:UK:EMP:2008-03 (base period=2008-Q1)
|-
| W(P) || Base Period converted to '''Weekly''' || W:UK:EMP:W(P) || W:UK:EMP:2008-04 (base period=2008-01-01)
|-
| D(P) || Base Period converted to '''Daily''' || D:UK:EMP:D(P) || D:UK:EMP:2008-12-31 (base period=2008)
|-
| H(P) || Base Period converted to '''Hourly''' || H:UK:EMP:D(P) || H:UK:EMP:2008-12-31T12 (base period=2008-02-02T12:00:03)
|-
| A(P-n) || Base Period minus (or plus) 'n' periods, converted to another frequency 'A' || A:UK:EMP:A(P-4) || A:UK:EMP:2007 (base period=2008-Q4)
|-
| A(P)-n || Base Period, converted to another frequency 'A' minus (or plus) n' periods || A:UK:EMP:A(P)-4 || A:UK:EMP:2004 (base period=2008-Q4)
|-
| A(P-n1)-n2 || Base Period, minus (or plus) 'n1' periods, converted to another frequency 'A' minus (or plus) 'n2' periods || A:UK:EMP:A(P-4)-4 || A:UK:EMP:2003 (base period=2008-Q4)
|-
| FA(P) || Subtracts 1 Period from the Converted Period IF the resolved Convert From end of period does not match the same point in time as the Convert To period.

'''Examples:'''
* Base Period =2008-Q4 Rule= FA(P) Output = 2008
* Base Period =2008-Q3 Rule= FA(P) Output = 2007 - the base period 2008-Q3 is not the same time as 2008 (end of period, so move back 1 year)
* Base Period =2008-Q1 Rule= FA(P) Output = 2007
* Base Period =2008-Q1 Rule= FA(P+3) Output = 2008
* Base Period =2008-Q1 Rule= FA(P+3)-1 Output = 2007

This works for ALL frequencies where the convert to frequency is lower then the base Period P, for example Quarter converted to Annual.

|| A:UK:EMP:FA(P) || A:UK:EMP:2007 (base period=2008-Q3)
|}

'''Compound expressions''' 
From Fusion Registry v11.5.1 and later it is possible to cast a period more the once in the same expression, for example

Q(FA(P-1))-1

If the input time period P=2019-Q4, then P-1 = 2019-Q3, the expression is rewritten as Q(FA(2019-Q3))-1. The Full Annual (FA) for 2019-Q3 is 2018, as the end of period 2019 does not match the end of period 2019-Q3, so the annual period moves back an additional period. The expression is rewritten as Q(2018)-1. 2018 to Quarterly moves to 2018-Q4 and then the last part of the expression steps back 1 period, the final period is 2018-Q3.

'''Notes on Time Period''' 
When converting from a lower frequency to a higher frequency (Weekly to Daily for example), the cast date always resolves to the end of the Period.

It is also important to note that as the [[#Observation Key|Observation Keys]] in these examples all have Frequency as the first Dimension, when converting to another frequency (Quarterly to Annual for example) the Observation Key changes the value for the Frequency Dimension, ensuring the correct series of data is retrieved.

'''Time Period variable in free text''' 
When refering to the Time Period as a variable in free text, for example in the table heading or a column heading, the variable is placed inside the variable syntax '''$()'''. For example $(P) would resolve to base period, and $(A(P)) would resolve to 'base period converted to annual frequency'. An example usage is shown below:

Table 1A showing data for $(P) to $(P+3)

Audit Web Services

2023-09-29T09:07:34Z

Mnelson: /* Overview */

[[Category:WebService]]
= Overview =

Fusion Registry 11 provides 2 main audit functions

# Audit of structure events, including data registrations
# System audits, example Fusion Portal events

= Audited Structure Events =

== Search Transaction ==
Search structure transactions

{| class="wikitable"
|-
|style="background-color:#eaecf0"|Entry Point|| /ws/registry/tx/searchAudits
|-
|style="background-color:#eaecf0"|Access|| Restricted to '''Audit Manager, Structure Manager, Portal Manager'''
|-
|style="background-color:#eaecf0"|Http Method|| GET
|}

{| class="wikitable"
|+ Query Parameters
|-
! Parameter !! Required !! Description
|-
| dateFrom || no || transaction date from in [[SDMX_Time_Formats|SDMX date format]]
|-
| dateTo || no || transaction date to in [[SDMX_Time_Formats|SDMX date format]]
|-
| max || no || the maximum number of results
|-
| user || no || Username of user which initiated the transaction
|-
| orderDesc || no || Order results in date descending order
|}

=== Response ===
A JSON array of audited transactions
[
{
"Date": 1645027254245,
"Id": 53,
"User": "root",
"Action": "Replace",
"Type": "Structure Submission"
},...
]

== Audit Details for Transaction ==
Return the audit details for a transaction. The transaction ID can be obtained from the RSS feed, or from the results of the '''searchTransactions''' API

{| class="wikitable"
|-
|style="background-color:#eaecf0"|Entry Point|| /ws/secure/audit/getDetailsByTransactionId
|-
|style="background-color:#eaecf0"|Access|| Restricted to '''Audit Manager, Structure Manager, Portal Manager'''
|-
|style="background-color:#eaecf0"|Http Method|| GET
|}

{| class="wikitable"
|+ Query Parameters
|-
! Parameter !! Required !! Description
|-
| id || yes || Transaction Id
|}

== Response ==
A JSON response in the following format:

{
"UID": "6d6ef80c-d19f-4506-94cb-05e53ebd06e1",
"TxId": 1,
"Type": "Structure Submission",
"ServerName": "localhost",
"VMID": "g6ee3edeb60c835d:55ba23af:19df811c3be:-8000",
"MachineId": "DESKTOP-E6MP45C/192.168.4.70",
"ServerContext": "/FusionRegistry",
"ServletPath": "/ws/registry",
"ServerPort": 8080,
"PathInfo": "/json/save",
"Parameters": "",
"Protocol": "HTTP/1.1",
"AcceptHeaders": "application/json, text/javascript, */*; q=0.01",
"AcceptLanguage": "en-US,en;q=0.9",
"SubmissionMethod": "POST",
"HttpResponseStatus": 200,
"RequestTime": 1644827120293,
"Duration": 385,
"Username": "root",
"RequestIP": "0:0:0:0:0:0:0:1",
"ProductVersion": "11.0.0.13",
"UserAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.82 Safari/537.36",
"BrowserName": "Chrome",
"BrowserVersion": "98",
"OS": "Windows 10"
}

= Audited System Events =

== Overview ==
Audited system events are categorised by the process id (which process initiated the event) and the event type for that process. For example a process id of STRUCTURE may have a sub-event of SAVE to declare it is a save request, or AFTER_COMMIT to declare it is a process which is initiated after a save event. Each audit event is uniquely identified by its UID, and can be retrieved by this UID.

A single event (for example a Portal Sync) is likely to consist of a chain of audited events, where one process and event pair PORTAL.MANUAL_SYNC and likely to call other services which capture their own audits, such as SEND_API_REQUEST.GET which captures the information about the HTTP Get Request from a web service. This chaining of events enables users to search for specific events (find all audits which call a web service) or search for a chain of events (find a PORTAL.MANUAL_SYNC and include all children). If an event is found in a chain, for example a specific web service call event, it is possible to request the parent event that led to that call, or all the ancestors - or for the full chain, the ancestors and children events.

Each audit event captures the same high level information (unique id, process id, event type, username, thread, start time, duration, status, machine identification, and software version), when a specific event is retrieved by its unique identification any event specific properties will also be retrieved. An example of event specific property is a for a REST GET event, which may include the HTTP Request and response Headers as well as the URL called.

'''Note:''' Properties specific to the event are only returned if a specific audit event is queried, (may include child or parent/ancestors for the audit).

== Search Audits ==
Search all system audits

{| class="wikitable"
|-
|style="background-color:#eaecf0"|Entry Point|| /ws/secure/audit
|-
|style="background-color:#eaecf0"|Access|| Restricted to '''Audit Manager, Structure Manager, Portal Manager'''
|-
|style="background-color:#eaecf0"|Http Method|| GET
|}

{| class="wikitable"
|+ Query Parameters
|-
! Parameter !! Required !! Description
|-
| min_duration || no || minimum processing duration in milliseconds
|-
| max_duration || no || maximum processing duration in milliseconds
|-
| event_type || no || desc
|-
| ancestors || no || true/false to include ancestor audit events (parent events, and their parents, and so on)
|-
| children || no || true/false to include child audit events (includes all decendants)
|-
| limit || no || maximum number of results
|-
| offset || no || used to page the results, the offset skips an [offset] number of results before writing
|-
| parent || no || retrieve any parent audit events of this event
|-
| process_id || no || filter by the audited process
|-
| process_start|| no || filter by the timestamp of the event (no sooner then process start)
|-
| process_end|| no || filter by the timestamp of the event (no later then process end)
|-
| software_version || no || the version of the software that was running when the even was captured
|-
| status|| no || the response status for the event
|-
| username|| no || the username associated (that caused) the event
|}

=== Response ===
A JSON Array of matched events

[
{
"uid": "6fa6c67c-37ac-48e5-b687-da2ebb316857",
"process_id": "APPLICATION_START",
"thread": "main",
"event_type": "START",
"username": "SYSTEM",
"process_start": 1644827100088,
"process_end": 1644827102046,
"duration": 1958,
"status": 200,
"vmid": "f5da5cfeb60c837d:44ff22ec:17ef755c3be:-8000",
"machine_id": "DESKTOP-E6MP45C/192.168.4.70",
"software_version": "11.0.0.13",
"properties": {}
},...
]

=== Search Structure Queries ===
Set process_id to SDMX_GET and event_type to structure
ws/secure/audit/?process_id=SDMX_GET&event_type=structure

=== Search Data Queries ===
Set process_id to SDMX_GET and event_type to data
ws/secure/audit/?process_id=SDMX_GET&event_type=data

== Audit Details ==
Retrieve the audit details for a specific audit event (and optionally child/ancestor events). This web service call includes the UID which can be found in the audit response of the search audits, this UID is also reported in error events (such as an emailed error notification).

{| class="wikitable"
|-
|style="background-color:#eaecf0"|Entry Point|| /ws/secure/audit/{uid}
|-
|style="background-color:#eaecf0"|Access|| Restricted to '''Audit Manager, Structure Manager, Portal Manager'''
|-
|style="background-color:#eaecf0"|Http Method|| GET
|}

{| class="wikitable"
|+ Query Parameters
|-
! Parameter !! Required !! Description
|-
| ancestors || no || true/false to include ancestor audit events (parent events, and their parents, and so on)
|-
| children || no || true/false to include child audit events (includes all decendants)
|}

=== Response ===
A JSON Array of matched events, this is the same as the response to search audits, but with the properties object complete with event specific properties.

[
{
"uid": "6fa6c67c-37ac-48e5-b687-da2ebb316857",
"process_id": "APPLICATION_START",
"thread": "main",
"event_type": "START",
"username": "SYSTEM",
"process_start": 1644827100088,
"process_end": 1644827102046,
"duration": 1958,
"status": 200,
"vmid": "f5da5cfeb60c837d:44ff22ec:17ef755c3be:-8000",
"machine_id": "DESKTOP-E6MP45C/192.168.4.70",
"software_version": "11.0.0.13",
"properties": {
'''"MEM_FREE_START": "8.468993E9",'''
'''"java.specification.version": "16",'''
'''"sun.cpu.isalist": "amd64",'''
'''"sun.jnu.encoding": "Cp1252", ....'''
}
},...
]

== Logs for Audit ==
Retrieve the logs for a specific audit event (and optionally child & ancestor events). The granularity of the log events are dependent on the system log level, which can be modified under the admin pages.

{| class="wikitable"
|-
|style="background-color:#eaecf0"|Entry Point|| /ws/secure/audit/log/{uid}
|-
|style="background-color:#eaecf0"|Access|| Restricted to '''Audit Manager, Structure Manager, Portal Manager'''
|-
|style="background-color:#eaecf0"|Http Method|| GET
|}

{| class="wikitable"
|+ Query Parameters
|-
! Parameter !! Required !! Description
|-
| ancestors || no || true/false to include ancestor audit events (parent events, and their parents, and so on)
|-
| children || no || true/false to include child audit events (includes all decendants)
|}

=== Response ===
A JSON Array of log events

[
{
"AuditId": "6fa6c67c-37ac-48e5-b687-da2ebb316857",
"Level": 2,
"Logger": "org.bis.fusion.utils.spring.ApplicationStarterImpl",
"Message": "Starting class org.bis.fusion.metadataregistry.manager.settings.RegistryAuditSettingsManagerImpl",
"Thread": "main",
"LogTime": 1644827100277
},...
]

Fusion Edge Server Properties

2023-08-31T15:49:08Z

Mnelson: /* Default Reponse Formats */

[[Category:Fusion Edge Server]]
[[Category:EdgeServerConfigure]]

= Overview =
The Fusion Edge Server provides configuration options on various aspects of the Edge Server. To set these configurations, create a file under the [[Fusion_Edge_Server_Directory|Edge Server Directory]] with the name: '''edgeserver.properties'''.

= Public URL =
The Edge Server application and web service URL should be defined in the edge server properties file. These URLs are used in responses to structure queries which include links to associated reference metadata, and stub structures which link back to the full artefact.

The URLs are split into two parts, the application URL, which should be a reference to the public application server, and the rest url which is the POSTfix URL to the web service, for example:

app.url=https://myserver.myorg.org/Edge
rest.url=/sdmx

The above setting would make the SDMX web service entry point: https://myserver.myorg.org/Edge/sdmx

= Sender Id =
SDMX Messages contain a Header section which includes the identity of the sender. The identity is mandatory in a SDMX message, and the default ID is set to "UNKNOWN". To change the Id, set the sender Id property

sender.id=YOUR_ID

= Secret Key =
The Fusion Edge server must be given a secret key, and it must match that used by the [[Fusion_Edge_Compiler|Fusion Edge Compiler]] in the compilation of the [[Edge_Server_Environment|Environment]]. If the secret key does not match the secret key of the [[Edge_Server_Environment|Environment]], the Fusion Edge Server will fail to load the [[Edge_Server_Environment|Environment]].

ledger.secret=MySecret

= Static Mode - File Name =
In [[Edge_Server_-_Publish_Content#.E2.80.8BStatic_Mode|static mode]] the Fusion Edge Server will read the properties file to know where the environment is. If nothing is set it will look for a file called node.zip.

environment.file=LiveEnvironment

= Embargo Data =
The Fusion Edge server will not release an [[Edge_Server_Environment|Environment]] with an [[Fusion_Edge_Compiler#Embargo|Embargo]] until the time on the server is later then or equal to the timestamp of the Embargo.

The Fusion Edge Server can be configured to pre-prepare and [[Edge_Server_Environment|Environment]] for release prior to the embargo time. For example if an Environment is scheduled for release at 12.00pm, and the Environment is moved to a secure Amazon S3 file system at 11.50, the Fusion Edge Server can be configured to pull these files into its local file system 5 minutes before go live, to eliminate any risk of network lag pulling the files. The Fusion Ede Server can also be set to pre-load the Environment into memory 30 seconds before go live, the Environment is now ready for dissemination but disconnected from any process which can get to the data, it is still fully secure. The Fusion Edge Server is able to then make the Environment go live by simply swapping the old Environment with the new, making it go live with millisecond precision.

To preload the Environment from a remote location to the local file system prior to the Embargo time, set the preload filesystem property to define the number of milliseconds before the embargo time that it should initiate the preload process.

ledger.preload.filesystem=40000

To preload the Environment into memory, set the preload memory property to define the number of milliseconds before the embargo time that it should start reading the Environment into memory.

ledger.preload.memory=20000

In this example the Fusion Edge Server will pull an Environment from the remote location to the local file system 40 seconds prior to the go live time, and then it will pre load the Environment from the file system into memory 20 seconds before the go live time. The Environment will be made live on the go live (embargo) time.

= Disable Audit =
The Fusion Edge maintains an audit log of events in its local [[Fusion_Edge_Server_Directory|Edge Server Directory]]. Auditing can be disabled with the following property

audit.disabled=true

= Limiting Series and Observations =
To place a limit on the number of series and the number of observations that can be queried via the web service the following two properties are supported:

limit.obs=12345
limit.series=1234

The numeric value indicates the maximum number of series or observations that can be returned from a single data query.

If the limit is exceeded an HTTP status code of 413 (Payload Too Large) will be returned to the client.

= Default Reponse Formats =
''Since v4.7.0''

The Fusion Edge Server supports a number of different response formats for structures, data, and metadata. When the client does not specify which format to return the response in, the Fusion Edge Server will respond in the default format. The properties file can be used to override the default formats by using the vnd header of the format to return.

{| class="wikitable"
|-
! REST Resource !! Default Format !! Override Property
|-
| data|| application/vnd.sdmx.structurespecificdata+xml;version=2.1 || default.format.data
|-
| structure|| application/vnd.sdmx.structure+json;version=2.0.0 || default.format.structure
|-
| availability|| application/vnd.sdmx.structure+json;version=2.0.0|| default.format.availability
|-
| tableavailability|| application/vnd.sdmx.structure+json;version=2.0.0 || default.format.tableavailability
|-
| table|| application/json || default.format.table
|-
| metadata|| application/vnd.sdmx.metadata+json;version=2.0.0 || default.format.metadata
|}

= Limiting Supported Data Formats =
To enforce a restriction on which data formats are supported from the REST API of the Fusion Edge Server, use the '''format.data''' property as follows:

format.data=[format1],[format2]

This property defines which formats are supported, if this property is not provided all formats will be supported. If this property is provided, only the formats in this list will be supported.

VND Headers are listed in the [[Data Formats]] section.

'''Example:''' 
format.data=application/vnd.sdmx.generictimeseriesdata+xml;version=2.1,application/vnd.sdmx.data+csv

= Limiting Supported Structure Formats =
To enforce a restriction on which structure formats are supported from the REST API of the Fusion Edge Server, use the '''format.structure''' property as follows:

format.structure=[format1],[format2]

This property defines which formats are supported, if this property is not provided all formats will be supported. If this property is provided, only the formats in this list will be supported.

Each [format] is a VND header the supported formats are:
{| class="wikitable"
|-
! Structure Format !! VND Header
|-
| SDMX EDI || application/vnd.sdmx.structure;version=edi
|-
| application/vnd.sdmx.structure+xml;version=1.0 || SDMX v1.0
|-
| application/vnd.sdmx.structure+xml;version=2.0 || SDMX v2.0
|-
| application/vnd.sdmx.structure+xml;version=2.1 || SDMX v2.1
|-
| application/vnd.openxmlformats-officedocument.spreadsheetml.sheet|| Excel
|-
| application/vnd.fusion.json || Fusion JSON
|-
| application/vnd.sdmx.structure+json;version=1.0.0 || SDMX JSON v1.0.0
|-
| application/vnd.sdmx.structure+json;version=2.0.0 || SDMX JSON v2.0.0
|}

'''Example:''' 
format.structure=application/vnd.sdmx.structure;version=edi,application/vnd.sdmx.json

= Limiting Query Functionality =
The [[Data_Query_Web_Service#Extended_HTTP_Query_Parameters|extended data query API]] can have some features disabled by using the following properties:
rest.query.attribute = false
rest.query.aggregate = false
rest.query.interpolate = false
rest.query.calculatemeasure = false
rest.query.normalisefreq = false
rest.query.round = false

= Caching =
The Fusion Edge Server caches the response to data queries in the tmp folder of the [[Fusion_Edge_Server_Directory|Edge Directory]]. The response is stored in gzip format, which ensures clients who request data in gzip format receive a rapid response when the data has been cached due to previous requests for the same data.

The following are important to note:
* The cache is NOT read on application startup. The cache is generated during the operational running time of the Fusion Edge Server. The cache is a combination of in memory pointers to files in the file system. Once the application server is terminated, it is not re-read on startup. This means any cached files are redundant after the Fusion Edge Server is terminated.
* The cache is NOT cleared on application startup. This is for a number of reasons, but principally deleting files from a running Edge Server is a slower process then could be achieved via an external mechanism. The delete process consumes CPU and therefore should be performed via a more efficient process before server startup.
* The cache can be manually cleared by deleting all the files in the tmp directory.
* The Fusion Edge Server will clear the cache if the Environment is updated in [[Edge_Server_-_Publish_Content#Dynamic_Mode|Dynamic mode]]. The in memory cache is first cleared to ensure clearance with immediate effect. A separate thread is initiated to clear the cached files, this can take time to complete if there are lots of cached files. Files are cleared in batches of 5000 with a 5 second delay per batch, this is to ensure the CPU required to run the deletion process does not block external traffic to the Edge Server.

The cache can be disabled by supplying the following property (Fusion Edge Server v2.3.5 an higher):

rest.query.cache=false

= CSV Data Format Defaults =
To modify the default CSV output is returned, the following properties are supported:

csv.delimiter=comma|tab|space|semicolon
csv.labels=ID|NAME|ID_AND_NAME
csv.showSeries=true|false
csv.includeBom=true|false
csv.keySeparator=:

The delimiter is the value of the character that will be output between the elements (defaults to comma).

The values for 'Labels' must be one of the following: ID; NAME; ID_AND_NAME (defaults to ID_AND_NAME).
Show Series defines whether to output the series key in the output (defaults to true).

= Cross-Origin Resource Sharing (CORS) =
By default the [https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS CORS] policy of the Fusion Edge server is to reject cross domain requests. To provide CORS policy to certain or all domains the cors.policy property should be set.

cors.policy=[policy 1]@[policy 2]@[policy 3]

Each policy is separated by an '@' symbol, within each policy are the following properties, each separated by a '$' symbol:

# Access-Control-Allow-Origin
# Access-Control-Allow-Headers (comma separated)
# Access-Control-Expose-Headers (comma separated)
# Access-Control-Allow-Credentials (true/false)

'''Example'''
cors.policy=www.metadatatechnology.com$GET,HEAD$*$Content-Disposition$true

'''Example All Domains'''
cors.policy=*$GET,HEAD,POST$*$Content-Disposition$false

= Password Protect Excel Workbooks =
Generated [[Reporting_Template|Reporting Template]] data formats can be set to be password protected using the following property

reporttemplate.password=[password]

Fusion Edge Server Properties

2023-08-31T15:48:24Z

Mnelson: /* Limiting Supported Data Formats */

[[Category:Fusion Edge Server]]
[[Category:EdgeServerConfigure]]

= Overview =
The Fusion Edge Server provides configuration options on various aspects of the Edge Server. To set these configurations, create a file under the [[Fusion_Edge_Server_Directory|Edge Server Directory]] with the name: '''edgeserver.properties'''.

= Public URL =
The Edge Server application and web service URL should be defined in the edge server properties file. These URLs are used in responses to structure queries which include links to associated reference metadata, and stub structures which link back to the full artefact.

The URLs are split into two parts, the application URL, which should be a reference to the public application server, and the rest url which is the POSTfix URL to the web service, for example:

app.url=https://myserver.myorg.org/Edge
rest.url=/sdmx

The above setting would make the SDMX web service entry point: https://myserver.myorg.org/Edge/sdmx

= Sender Id =
SDMX Messages contain a Header section which includes the identity of the sender. The identity is mandatory in a SDMX message, and the default ID is set to "UNKNOWN". To change the Id, set the sender Id property

sender.id=YOUR_ID

= Secret Key =
The Fusion Edge server must be given a secret key, and it must match that used by the [[Fusion_Edge_Compiler|Fusion Edge Compiler]] in the compilation of the [[Edge_Server_Environment|Environment]]. If the secret key does not match the secret key of the [[Edge_Server_Environment|Environment]], the Fusion Edge Server will fail to load the [[Edge_Server_Environment|Environment]].

ledger.secret=MySecret

= Static Mode - File Name =
In [[Edge_Server_-_Publish_Content#.E2.80.8BStatic_Mode|static mode]] the Fusion Edge Server will read the properties file to know where the environment is. If nothing is set it will look for a file called node.zip.

environment.file=LiveEnvironment

= Embargo Data =
The Fusion Edge server will not release an [[Edge_Server_Environment|Environment]] with an [[Fusion_Edge_Compiler#Embargo|Embargo]] until the time on the server is later then or equal to the timestamp of the Embargo.

The Fusion Edge Server can be configured to pre-prepare and [[Edge_Server_Environment|Environment]] for release prior to the embargo time. For example if an Environment is scheduled for release at 12.00pm, and the Environment is moved to a secure Amazon S3 file system at 11.50, the Fusion Edge Server can be configured to pull these files into its local file system 5 minutes before go live, to eliminate any risk of network lag pulling the files. The Fusion Ede Server can also be set to pre-load the Environment into memory 30 seconds before go live, the Environment is now ready for dissemination but disconnected from any process which can get to the data, it is still fully secure. The Fusion Edge Server is able to then make the Environment go live by simply swapping the old Environment with the new, making it go live with millisecond precision.

To preload the Environment from a remote location to the local file system prior to the Embargo time, set the preload filesystem property to define the number of milliseconds before the embargo time that it should initiate the preload process.

ledger.preload.filesystem=40000

To preload the Environment into memory, set the preload memory property to define the number of milliseconds before the embargo time that it should start reading the Environment into memory.

ledger.preload.memory=20000

In this example the Fusion Edge Server will pull an Environment from the remote location to the local file system 40 seconds prior to the go live time, and then it will pre load the Environment from the file system into memory 20 seconds before the go live time. The Environment will be made live on the go live (embargo) time.

= Disable Audit =
The Fusion Edge maintains an audit log of events in its local [[Fusion_Edge_Server_Directory|Edge Server Directory]]. Auditing can be disabled with the following property

audit.disabled=true

= Limiting Series and Observations =
To place a limit on the number of series and the number of observations that can be queried via the web service the following two properties are supported:

limit.obs=12345
limit.series=1234

The numeric value indicates the maximum number of series or observations that can be returned from a single data query.

If the limit is exceeded an HTTP status code of 413 (Payload Too Large) will be returned to the client.

= Default Reponse Formats =
The Fusion Edge Server supports a number of different response formats for structures, data, and metadata. When the client does not specify which format to return the response in, the Fusion Edge Server will respond in the default format. The properties file can be used to override the default formats by using the vnd header of the format to return.

{| class="wikitable"
|-
! REST Resource !! Default Format !! Override Property
|-
| data|| application/vnd.sdmx.structurespecificdata+xml;version=2.1 || default.format.data
|-
| structure|| application/vnd.sdmx.structure+json;version=2.0.0 || default.format.structure
|-
| availability|| application/vnd.sdmx.structure+json;version=2.0.0|| default.format.availability
|-
| tableavailability|| application/vnd.sdmx.structure+json;version=2.0.0 || default.format.tableavailability
|-
| table|| application/json || default.format.table
|-
| metadata|| application/vnd.sdmx.metadata+json;version=2.0.0 || default.format.metadata
|}

= Limiting Supported Data Formats =
To enforce a restriction on which data formats are supported from the REST API of the Fusion Edge Server, use the '''format.data''' property as follows:

format.data=[format1],[format2]

This property defines which formats are supported, if this property is not provided all formats will be supported. If this property is provided, only the formats in this list will be supported.

VND Headers are listed in the [[Data Formats]] section.

'''Example:''' 
format.data=application/vnd.sdmx.generictimeseriesdata+xml;version=2.1,application/vnd.sdmx.data+csv

= Limiting Supported Structure Formats =
To enforce a restriction on which structure formats are supported from the REST API of the Fusion Edge Server, use the '''format.structure''' property as follows:

format.structure=[format1],[format2]

This property defines which formats are supported, if this property is not provided all formats will be supported. If this property is provided, only the formats in this list will be supported.

Each [format] is a VND header the supported formats are:
{| class="wikitable"
|-
! Structure Format !! VND Header
|-
| SDMX EDI || application/vnd.sdmx.structure;version=edi
|-
| application/vnd.sdmx.structure+xml;version=1.0 || SDMX v1.0
|-
| application/vnd.sdmx.structure+xml;version=2.0 || SDMX v2.0
|-
| application/vnd.sdmx.structure+xml;version=2.1 || SDMX v2.1
|-
| application/vnd.openxmlformats-officedocument.spreadsheetml.sheet|| Excel
|-
| application/vnd.fusion.json || Fusion JSON
|-
| application/vnd.sdmx.structure+json;version=1.0.0 || SDMX JSON v1.0.0
|-
| application/vnd.sdmx.structure+json;version=2.0.0 || SDMX JSON v2.0.0
|}

'''Example:''' 
format.structure=application/vnd.sdmx.structure;version=edi,application/vnd.sdmx.json

= Limiting Query Functionality =
The [[Data_Query_Web_Service#Extended_HTTP_Query_Parameters|extended data query API]] can have some features disabled by using the following properties:
rest.query.attribute = false
rest.query.aggregate = false
rest.query.interpolate = false
rest.query.calculatemeasure = false
rest.query.normalisefreq = false
rest.query.round = false

= Caching =
The Fusion Edge Server caches the response to data queries in the tmp folder of the [[Fusion_Edge_Server_Directory|Edge Directory]]. The response is stored in gzip format, which ensures clients who request data in gzip format receive a rapid response when the data has been cached due to previous requests for the same data.

The following are important to note:
* The cache is NOT read on application startup. The cache is generated during the operational running time of the Fusion Edge Server. The cache is a combination of in memory pointers to files in the file system. Once the application server is terminated, it is not re-read on startup. This means any cached files are redundant after the Fusion Edge Server is terminated.
* The cache is NOT cleared on application startup. This is for a number of reasons, but principally deleting files from a running Edge Server is a slower process then could be achieved via an external mechanism. The delete process consumes CPU and therefore should be performed via a more efficient process before server startup.
* The cache can be manually cleared by deleting all the files in the tmp directory.
* The Fusion Edge Server will clear the cache if the Environment is updated in [[Edge_Server_-_Publish_Content#Dynamic_Mode|Dynamic mode]]. The in memory cache is first cleared to ensure clearance with immediate effect. A separate thread is initiated to clear the cached files, this can take time to complete if there are lots of cached files. Files are cleared in batches of 5000 with a 5 second delay per batch, this is to ensure the CPU required to run the deletion process does not block external traffic to the Edge Server.

The cache can be disabled by supplying the following property (Fusion Edge Server v2.3.5 an higher):

rest.query.cache=false

= CSV Data Format Defaults =
To modify the default CSV output is returned, the following properties are supported:

csv.delimiter=comma|tab|space|semicolon
csv.labels=ID|NAME|ID_AND_NAME
csv.showSeries=true|false
csv.includeBom=true|false
csv.keySeparator=:

The delimiter is the value of the character that will be output between the elements (defaults to comma).

The values for 'Labels' must be one of the following: ID; NAME; ID_AND_NAME (defaults to ID_AND_NAME).
Show Series defines whether to output the series key in the output (defaults to true).

= Cross-Origin Resource Sharing (CORS) =
By default the [https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS CORS] policy of the Fusion Edge server is to reject cross domain requests. To provide CORS policy to certain or all domains the cors.policy property should be set.

cors.policy=[policy 1]@[policy 2]@[policy 3]

Each policy is separated by an '@' symbol, within each policy are the following properties, each separated by a '$' symbol:

# Access-Control-Allow-Origin
# Access-Control-Allow-Headers (comma separated)
# Access-Control-Expose-Headers (comma separated)
# Access-Control-Allow-Credentials (true/false)

'''Example'''
cors.policy=www.metadatatechnology.com$GET,HEAD$*$Content-Disposition$true

'''Example All Domains'''
cors.policy=*$GET,HEAD,POST$*$Content-Disposition$false

= Password Protect Excel Workbooks =
Generated [[Reporting_Template|Reporting Template]] data formats can be set to be password protected using the following property

reporttemplate.password=[password]

Publication Table

2023-08-30T11:58:40Z

Mnelson: /* Dependent Dimension Variable */

[[Category:Functions]]

@See [[Publication_Table_Web_Services|Publication Table Web Services]]

= Publication Tables =
'''Introduced to Fusion Registry Enterprise v11.0.0.17'''

== Overview ==
[[File:Publication Table Definition.png|thumb|Using SDMX concepts of Agency, Id, Name, Description]]

[[File:Publication Table.png|thumb|Table Layout and Content Design]]

[[File:MaterialisedTable Preview.png|thumb|Showing the Table Preview feature which materialises the table with the given variables]]

A Publication Table is a curated table of data, where the values in the table rows, columns and table body cells can be individually specified. This enables a single table to pull data from multiple sources and piece the data together in a way which is not possible when dealing with standard cubes of data, whose table model is typically based on a Pivot model, with columns and rows built around the Dimension values to which the column or row is assigned. A Publication Table breaks out of this restrictive model by allowing cells of data to reference observations which do not need to share any relationship, in terms of dimensionality or even data source, with adjacent cells,.

Data in a Publication Table may be sourced from a single [[Dataflow_-_Structural_Metadata_Management|Dataflow]] or it may be sourced from multiple Dataflows enabling cross comparison of data.

Publication Tables can be static (every cell is a defined Observation) or dynamic, where a cell includes [[#Variable Dimension Members|variables Dimensions]] in the [[#Observation Key|Observation Key]], which can be provided at [[#Metadata vs Materialised|table build time]].

[[#Time Variables|Time arithmetic and frequency conversion]] is supported, enabling a single table to pull data from different time periods and frequencies which are all derived from the base period on which the table is built.

Publication Tables support [[#Calculated Observation Cells|calculations]], where each calculation can take input from one or more Observations over one or more [[Dataflow_-_Structural_Metadata_Management|Dataflows]].

Variable values also play a big role in Publication Tables, and includes standard variable placeholders – as well as advanced [[#Conditional Variables|conditional variables]], whose list of variable values change based on the value of another Dimension for which the table is built.

== Metadata vs Materialised ==

A Publication Table is stored in Fusion Registry the same way any other piece of structural metadata is stored. In this way it has an owner (Agency), and identity (ID) and a version. The change history is tracked using the same services, and the table definition can be [[Publication_Table_Web_Services|queried]] using the same SDMX web services for Structural Metadata.

The Publication Table metadata contains information about which Dataflows are used by the Table, which additional variables are being imported, and a definition of the table structure, and which Observations to import by referencing them by their unique [[#Observation Key|Observation Key]].

A Publication Table definition can be '''materialised''' into a table of Data by from the [[Publication_Table_Web_Services|Publication Table Web Service]], which combines the information of the Table Identity, and Variable Values to rewrite the [[#Observation Key|Observation Key]] into actual observation values. If building a web User Interface, then response from this service can be a packet of JSON, which describes the layout and content of the final Table in a way which is quick and simple to render for the User to see. The process of materialising the table is known as ‘'''build time'''’.

Additional web services are provided to enable clients to ask questions about the table such as ‘what are the earliest and latest time periods for which there is data for this table’ or ‘what possible variable values are there for this table’. This enables user interfaces to provide a user with the option to change the variables ‘on the fly’ ensuing they are able to build a table which is known to have data.

= Publication Table Cell Content =
== Overview ==
[[File:PublicationTable Layout.png|thumb|Publication Table Layout]]
A Publication Table is made up of a '''Table Head Rows''' and '''Table Body Rows'''.

The Table Head is made up of 1 or more table rows. Each Table Head cell may span 1 or more columns and 1 or more rows. The Table Head content is made up of free text and / or variable placeholders whose values are resolved when the table is built. Table Head Cells can be left empty.

The Table Body is made up of 1 or more Table Rows. The Table Rows provide the Publication Table with its data, by referencing Observations from the Dataflow.

At table build time all variables are resolved, and the corresponding data are retrieved from the data stores which hold the data.

== Table Rows ==
=== Overview ===
A Table Row consist of a Table Row heading, this can be a mix of free text and [[#Table Variables|variable text]]. The Table Row Heading is always the first Table Row Cell, and can not span multiple columns or rows. A Table Row can not have more then one heading. The remaining Table Row Cells are the Observation Cells, which may contain free text and [[#Table Variables|variable text]] but will typically contain a reference to an Observation by an [[#Observation Key|Obsevation Keys]] or a [[#Calculated Observation Cells|Calculation]] which can combine multiple [[#Observation Key|Obsevation Keys]].

A Table Row which results in no Observation data will not be included in materialised table at build time, unless the Table Row is a grouping row (see [[#Row Levels|Row Levels]] for details on grouping rows).

=== Row Levels ===
[[File:PublicationTabel Levels.png|thumb|Showing Levels in the Table Rows]]
The Table Row Heading can be used to define a row level. The level indicates to the client responsible for rendering the table the row's position in a hierarchy. Row levels are depicted using one or more leading chevrons ‘>‘ before the text. The number of hyphens indicates the level, with one leading hypen indicating level 1, two indicating level 2, and so on. It is possible to skip levels if required, i.e. Row 1 is level 1 with Row 2 assigned to level 3, skipping out level 2 entirely.

An example of row headings using levels is as follows:

Countries
> Europe
>> France
>> Germany
> Asia
>> China

Levels can be an important way to '''group data'''. A Table Row which acts as a parent row in a hierarchy, can be used as grouping mechanism for the rows that follow. This enables Parent Rows to be included in the output table, even if the Table Row contains no Observation data. The default behaviour for a Table Row with no data is to exclude it from the built table.

=== Looping Rows ===

Instead of explicitly defining a row for each Dimension member, it is possible to define a row as a loop (to iterate over all, or a subset of Dimension members). For example, if a publication table contains a Dimension for countries with an ID of REF_AREA then a loop can be set up to loop through all REF_AREA values, or a subset of the values - for which there is data. If the values are backed by a Codelist which has a hierarchy, the hierarchy can be preserved in the final table. It is also possible to nest a loop within another loop, for example for each REF_AREA output each INDICATOR.

====Syntax====
The loop is defined by giving the table row a label which starts with a syntax of :=LOOP([args]), like in the following example
:=LOOP([dimension id], [preserve hierarchy], [loop order], [explicit values to include])

'''Argument Definition'''
''dimension id'' = Required. The ID of the Dimension to loop over. For example CURRENCY would loop over all the Currency values which have data in the given context. The row label is the Currency, for example USD.
''preserve hierarchy'' = Optional, default false. If true the hierarchy of the Codes in the underlying Codelist will be built into the row hierarchy
''loop order'' = Optional, default 1. 1=Natural Order (as they appear in the Codelist), 2=By Code ID, 3=By Code Label
''explicit values to include'' = Optional, default include all values. A var arg list of values to loop over.

'''Full Example'''

:=LOOP(REF_AREA, true, 3, EUR,FR,DE,UK,ES)

====Simple Loop====
The loop is defined by giving the table row a label which starts with a syntax of :=LOOP([args]), like in the following example
:=LOOP(REF_AREA)

This loop definition will loop through all the REF_AREA values, building a row for each value. The Observation keys for the row can refer to the loop variable using a $(DIM_ID) placeholder in the Observation Key, for example:

Employed Student
:=LOOP(REF_AREA) A:$(REF_AREA):EMP A:$(REF_AREA):STU

The output table would look like the following:

Employed Student
United Kingdom 20 12
France 22 11
Germany 25 15

====Nested Loop====
If a loop row is followed by a row (or rows) which is an extra level deep, the child row(s) will be included for each loop, for example

Employed Student
> :=LOOP(REF_AREA)
>> Total A:$(REF_AREA):'''T''':EMP A:$(REF_AREA):'''T''':STU
>>> Male A:$(REF_AREA):'''M''':EMP A:$(REF_AREA):'''M''':STU
>>> Female A:$(REF_AREA):'''F''':EMP A:$(REF_AREA):'''F''':STU

Because the '''Total''' row is a child row of the LOOP (denoted by the number of leading '>') and '''Male'''/'''Female''' are children of '''Total''' - for each REF_AREA there will be a Total, Male and Female, outputting a table like the following:

Employed Student
United Kingdom
Total
Male 5 1
Female 4 2
France
Total
Male 6 2
Female 7 3

It is also possible to define a subloop as a loop. The previous example can be rewritten as:

Employed Student
> :=LOOP(REF_AREA)
>> :=LOOP(SEX) A:$(REF_AREA):'''$(SEX)''':EMP A:$(REF_AREA):'''$(SEX)''':STU

There is no limit to the number of subloops that can be defined.

====Loop Hierarchical Values====
If the underlying Codelist has a hierarchy, a subsequent argument can be provided to ask for the hierarchy to be preserved

Countries
> :=LOOP(REF_AREA, true)

'''Note:''' A hierarchy will keep rows with no data if they are acting as a parent node for rows that do have data (e.g. there may not be data for Europe, but the row exists because there are child rows of France and Germany with data).

Nested loops can also preserve hierarchies for example:

:=LOOP(REF_AREA, true)
>:=LOOP(SEX, true)

With an example output (assuming no data exists for the hierarical parents (Continent, Europe)

Continent
Europe
France
Total (M+F)
Male
Female
Germany
Total (M+F)
Male
Female

====Order Loop Rows====
It is possible to order the labels by their natural order (that of the Codelist), by id or by label, using the numbers 1, 2, or 3 respectively. The following outputs the rows ordered by label.

Countries
> :=LOOP(REF_AREA, true, 3)

====Fixed Loop List====
It is also possible to define a subset of the values to loop over. For example, if REF_AREA has data for 100 countries, but only 3 are required in the table. In this scenario, the natural order is that of the provided list, not of the underlying Codelist.

Countries
> :=LOOP(REF_AREA, true, 3, UK, FR, DE)

====Loop over External Variable====
Instead of looping the values of the Dimension's Codelist, it is possible to loop another Codelist - the purpose for this is to support alternative labels (if the labels for the publication table need to be different from those of the Dimension members) - or a subset (as an alternative to passing the subset of values into the LOOP argument), or to define a hierarchy (if the Dimension's Codelist does not have a hierarchy, or an alternative is required).

In order to loop over an external Variable, the Publication Table must be linked to an external Codelist/Valuelist against an Alias. For example EXT=My Country Codelist with Hierarchy. The corresponding loop then loops the Codes in the external list, and stores these the corresponding Dimension ID variable, for example:

Countries
> '''REF_AREA''':=LOOP('''EXT''')

Noting that the loop is over the variable EXT which resolves to an external Codelist. The loop is assigned to the REF_AREA Dimension.

=== Anchor Rows ===
The default behaviour is for a row to be omitted if the row has no data and it has no child rows with data. However, if the desired result is to always output the row, then this can be achieved by using the [anchor] tag in the row header label.

Consider the following Publication Table definition:

Male Female
Employed A:M:EMP A:F:EMP
Student A:M:STU A:F:STU
Retired A:M:RET A:F:RET

If there are no data for Student, then the materialised table would look like this

Male Female
Employed 12 13
Retired 3 5

==== Anchor Single Row [anchor] ====
To anchor a single row, add the '''[anchor]''' tag at the start of the row

Male Female
Employed A:M:EMP A:F:EMP
[anchor]Student A:M:STU A:F:STU
Retired A:M:RET A:F:RET

Then the output table would preserve the 'Student' row

Male Female
Employed 12 13
Student
Retired 3 5

This rule can be coupled with [[Publication_Table#Formatting_Missing_Observations|formatting for missing observations]], then the table could be enhanced to look something like this

Male Female
Employed 12 13
Student - -
Retired 3 5

==== Anchor Child Rows [anchor+] ====
For a row with a hierarchy, an '''[anchor+]''' tag can be placed on a parent node to anchor the parent and all of its children, for example:

Continent
[anchor+]>Europe
>>France
>>>Total (M+F)
>>>>Male
>>>>Female
>>Germany
>>>Total (M+F)
>>>>Male
>>>>Female

The above rule would always output Europe and all of its descendants (France, Total, Male, Female, Germany).

==== Anchor Siblings and Child Rows [anchor*]====
The '''[anchor*]''' tag can be used as a way to anchor all following siblings and child nodes, this can be a useful mechanism to anchor all rows for the whole table by placing the anchor[*] on the first row of the table.

==== Anchor Loops ====
If a loop is preceeded by an '''[anchor]''' tag then the system will have the following behaviour:

* If an external Codelist variable is being looped, all codes will be ouput regardless of what data is stored in the system
* If a Dimension is being looped, all codes in the Codelist used by the Dimension will be output regardless of what data is stored in the system
* If a Dimension is being looped and it is not coded (i.e it is free text or numerical or other) then the [anchor] has no effect, as the looped values are based on what data exists

== Observation Key ==
=== Overview ===
The purpose of the Observation Key is to reference a Observation whose value will be resolved at table build time. The Observation Key can explicitly resolve to 1 observation value, if all parts of the Key are provided. However, it is possible to make parts of the key varaibles, allowing the values to be set at table build time.

An Observation Key is made up of 4 parts:

* Dataflow Reference
* Series Reference
* Time Period Reference
* Measure Reference

An example Observation Key with all of these parameters in place is:

DF_REG.M:UK:SUR:2008:BIRTHS

The key is broken down into the following parts:

{| class="wikitable"
|-
! Observation Key Part !! Name !! Description !! Required
|-
| DF_REG|| Dataflow Alias|| This identifies the Dataflow which has the data. The identification uses the alias, which is assiged to the Dataflow in the Publication Table definition. The Dataflow Alias is only required if the Publication Table references more then one Dataflow. If the Publication Table references more then one Dataflow, it can assign a Dataflow to a Default Alias, which means that it is required as part of the Observation Key. || No (if only 1 Dataflow or a default alias is set)
|-
| M:UK:SUR || Series Key || This identifies the Series of Data for the Dataflow, by providing a single value for each of the Dimension Members of the Dataflow, separated by a ‘:’ character. In this example the Dataflow has 3 Dimensions (excluding time), the value for the first is 'M', the second 'UK' and the last 'SUR'. As the Dataflow in this example has a Time Dimension this Series Key will uniquely identify a Series of Observations where each Observation refers to a unique Time Period || Yes (but can omit values)
|-
| 2008 || Time Period || This identifies the Time Period in the Series of Data identified by the previous section. The Combination of the Series and Time Period for a Time Series Dataflow will uniquely identify the Observation || No. Omit to make it variable for Time Series Data (defaults to last period). Not required if there is no Time Dimension for the DSD.
|-
| BIRTHS || Measure Dimension || If the Dataflow uses a Data Structure with multiple Measures, then the Measure ID must be included as the last part of the Observation Key. In this example the Dataflow has multiple measures (BIRTHS/DEATHS/MARRIAGES) so the Observation Key needs to include this value. || When there is only 1 measure it is not required, when there are multiple measures, it is required.
|}

Observation Keys are included on the materialised table in JSON format enabling the observation value to be tied back to the key. Observation Keys also provide a mechanism by which [[#Formatting_Cells|formatting rules]] can be applied to a subset of table cells.

=== Variable Dimension Members ===
It is possible to omit Dimension Values from the Observation Key, when this is the case that Dimension will become a variable – whose value can be set at table build time. For example, the second Dimension value of ‘UK’, from the previous example, can be omitted like this:

M::SUR:2008

The Publication Table can now be built for any Country by passing in the variable at table build time - i.e. UK, FR, DE. By passing in different variable values, the Observation Key will change, and as such it will reference a different Series in the Dataset.

When an Observation Key has a variable Dimension member, the ID of the Dimension can be referenced in text cells to create variable headings. For example, if the ID of the 2nd Dimension in the DSD is '''REF_AREA''', the Table Column could be given a title of $(REF_AREA) which would resolve to 'United Kingdom', 'France', or 'Germany' at build time depending on which varaible value is used to build the table. [[#Dimension Variables|#See Dimension Variables]].

=== Variable Time Period ===
For Time Series dataset, the Time Dimension can form part of the Observation Key, for example M:UK:SUR:'''2008'''. The 2008 part of the key refers to the year 2008.

It is possible to omit the value for Time Period in the Observation Key. Omitting Time Period will make the Time value a variable which can be provided at table build time, defaulting to the last period for which data exists if no value is supplied.

Extending the above example to omit the Time value results in an Observation Key whose Country and Time values can be provided to the web service at table build time.

M::SUR

'''Note:''' The above key is exactly the same as using the variable Time Period value 'P' in the Time Dimension, as follows:

M::SUR:'''P'''

Additional functions can be performed on the Time Dimension, such as moving forwards or backwards from the table base period, [[#Time Variables|#see Time Variables]] for more details.

=== Free Text Observation Keys ===
Whilst typically a cell in the body of the Publication Table would resolve to an Observation value, it is possible to use it for free text. To include free text cells, start the cell value with a leading apostrophe character, for example:

'Cell Text

'''Note''': An Observation Key can '''not''' contain a mix of free text as well as an Observation value

== Calculated Observation Cells ==
[[File:TableDefinition Calculations Variables.png|thumb|Publication Table with Calculated Cells and Variable Dimensions]]

An Observation Cell may contain calculated values, which can consist of a mix of simple algebra (+, -, *, /) and a reference to one or more Observations which may come from multiple Dataflows.

To define a Calculated Observation the Cell the following rules apply:

* The Observation Key must start with :=
* Any Observation Keys must appear within enclosing square brackets '[' and ']'
* An Observation key may embed calculations on a single Dimension by using parenthesis '(' and ')' in the key part
* Numbers in Observation Keys are treated as Code IDs, unless prefixed with a '#'
* The same rules for variable Dimensions and Time Period apply when resolving the Observation Keys used in the calculation

Examples are shown in the table below

{| class="wikitable"
|-
! Example Syntax !! Description
|-
| :=[A:UK:EMP]+10 || Add 10 to the observation value retrieved from the key A:UK:EMP
|-
| :=[A:(UK+FR+DE):EMP] || Add the series A:UK:EMP + A:FR:EMP + A:DE:EMP
|-
| :=[A:(UK+FR+12):EMP] || Add the series A:UK:EMP + A:FR:EMP + A:DE:12
|-
| :=[A:(UK+FR+#12):EMP] || Add the series A:UK:EMP + A:FR:EMP and add the Constant '12'
|-
| :=[A:UK:EMP]+[A:FR:EMP]+[A:DE:EMP] || Add the series A:UK:EMP + A:FR:EMP + A:DE:EMP
|-
| :=[A:(UK+FR):EMP]/[A:(UK+FR):UMP] || Adds the series A:UK:EMP + A:FR:EMP and divides by the result of A:UK:UMP + A:FR:UMP
|-
| :=((100/[A:UK:EMP:(M+F):P-1])*[A:UK:EMP:(M+F)])-100 || Demonstrates the ability to use parenthesis outside of the Observation Keys to control the order of mathematical operations
|-
| :=[POP.M:UK:(M+F)] / [LAB.M:UK:EMP] || Demonstrates the ability to calculate data from two Dataflows (defined by alias POP and LAB)
|-
| :=[M:UK:EMP:M(P)] + [A:UK:EMP:A(P)] || Demonstrates the ability to calculate data where the period is converted from the base period
|}

=== Calculated Cell Observation Key ===
Calculated Cells have implicit [[#Observation_Key|Observation Keys]] which are derived based on the inputs series in the calculation. The [[#Observation_Key|Observation Key]] can be used to assign [[#Formatting_Cells|Formatting Rules]] in the same way as a 'standard key' can be used. It is possible to assign an explicit key to the calculation, which will override the implicit key assigned. The following table shows how the key assignment works.

For example
{| class="wikitable"
|-
! Calculation !! Output Key || Explanation
|-
|:=[A:(UK+FR+DE):EMP] || A:#:EMP:2009 || The first and last dimension have a fixed value, the second dimension has a variable value and is replaced by '#'. The time period is added to the end of the key.
|-
|:=((100/[A:UK:EMP:(M+F):P-1])*[A:UK:EMP:(M+F)])-100 || A:UK:EMP:#:# || The first three dimensions have a fixed value, the last dimension has a variable value and is replaced by '#'. The time period is a variable as there are 2 time inputs to the calculation (P and P-1)
|-
|:=PCTOF([A:M:UK:P],[A:TOT:UK:P]) || A:#:UK:2009 || The first and last dimension have a fixed value, the second dimension has a variable value and is replaced by '#'. The time period is added to the end of the key.
|-
|:=[A:UK:EMP]+10 || A:UK:EMP || The input key is the same as the output key as none of the dimensions have a variable value. The time period is added to the end of the key.
|-
|A:UK10:EMP:=[A:UK:EMP]+10 || A:UK10:EMP:2009 || The output key is explicitly assigned. The time period is added to the end of the key.
|}

The output key conforms to the same Dimension breakdown as the underlying Data Structure. If a Observation Key is explicitly assigned, it must also honour the same Dimension breakdown. This allows formatting rules to be applied using the key component syntax, for example COUNTRY=UK10 would search for Observation Keys whose Country Dimension is reporting the value UK10.

== Percentage Of ==
The PCTOF (Percentage Of) function can be used to calculate 'x' as a percentage of 'y' following the syntax:

PCTOF(x,y)

This is reolved to the following mathmatical expression: '''(100/y) * x'''

x and y must either be a numerical value, and observation key, or an expression using a combination of the two. For example

:=PCTOF([A:M:UK:P],[A:TOT:UK:P])

The Observation Key follows the same standard rules with respect to varaibles and inline calculations - the following example shows an Observation Key where Male + Female values are added with the result being used in the Percentage Of function

:=PCTOF([A:M:UK],[A:('''M+F'''):UK])

A more complex example shows calculations around the PCTOF function:
:=(10 + PCTOF([A:M:UK:A(P)-1],[A:(M+F):UK:A(P)])) / 2

== Footnotes ==

[[File:Footnotes3.png|thumb|A table with footnotes]]
[[File:Footnotes2.png|thumb|Footnotes in the table builder]]

=== Overview ===
Footnotes can be added to any part of the publication table (table column/row header, or observation cells). The output table is generated with a footnote number, with the list of footnotes at the end of the table.

Footnotes need to be thought about in 2 parts: Part 1 are the list of footnotes which are defined at the Publication Table level. Part 2 are the references to the footnotes from within the table. This separation of allows footnotes to be reused within the same table, and when working with [[#Footnotes_with_Placeholders|placeholders]] footnote templates can be set up where the text changes based on the placeholder values defined in the table builder.

To add a footnote, add it to the Publication Table, for example: 
Footnote 1 
Footnote 2

Then in the table builder, add a reference to which footnote to use, $(1) references '''Footnote 1''', and $(2) references '''Footnote 2'''.

'''Note:''' The order of the list of footnotes in the table definition may not be reflected in the order they are output. As such a footnote referenced by $(1) may be output as the second footnote.

=== Footnotes with Variable ===
A footnote can reference variables, using the [[#Table_Variables|variable syntax]],

A footnote for $(REF_AREA.UK)

A footnote can even output an observation value, observation attribute or series attribute.

Measure Value
A footnote with obs value $(A:UK:EMP:2008:OBS_VALUE)

Observation Attribute
A footnote with obs attribute $(A:UK:EMP:2008:OBS_COMMENT)

Series Attribute
A footnote with obs attribute $(A:UK:EMP:P:SERIES_TITLE)

When combining variable footnotes with placeholder values and loops, it is possible to dynamically create references to observation attributes.

'''Note:''' the use of P in the final cell key which is a reference to the [[#Time_Variables|time period]] for which the table is being built.

=== Footnotes with Placeholders ===
It is possible to write a generic footnote for the publication table, which includes placeholders. When the footnote is referenced, the placeholder values are provided. The syntax for a placeholder is {n} where n is a positive integer, for example {1}.

Footnote Example

A footnote for $(REF_AREA.{1}) and $(REF_AREA.{2})

In the table builder, the footnote is referenced as usual, but with arguments for the placeholders:

Footnote reference in Table builder

My Table Row '''$(1,UK,FR)'''

The footnote reference in the above example is in bold, and consists of which footnote text to use (in this case the first one in the list, denoted by '1'), and 2 arguments are passed for the placeholders, placeholder 1 takes the value of UK and placeholder 2 takes the value of FR. During the materialise process the intermediate text is generated:

A footnote for $(REF_AREA.UK) and $(REF_AREA.FR)

Which then goes through the variable resolve process, in this case both variables, '''$(REF_AREA.UK) and $(REF_AREA.FR)''', match the [[#Coded_Variables|Coded Variable]] syntax and would resolve to:

Final Output
A footnote for United Kingdom and France

=== Footnotes in Observation Cell ===
Any number of footnotes can appear in an Observation cell, simply put the footnote(s) at the end of the [[#Observation_Key|Observation Key]].

Standard Footnote Reference 
DF_REG.M:UK:SUR:2008:BIRTHS $(1) $(2)

Footnote with Placeholder 
DF_REG.M:UK:SUR:2008:BIRTHS $(1,UK)

=== Footnotes in Loops ===
A row [[#Looping_Rows|loop]] contains a locally scoped variable, and as such it can be used when calling a footnote with a placeholder.

Example Footnotes
$(A:T:{1}:EMP:P:SERIES_TITLE)
$(A:T:{1}:EMP:P:TITLE_COMPL)
$(A:{1}:{2}:EMP:P:OBS_COMMENT)

Example Table Definition
Employed Student
> :=LOOP(REF_AREA) '''$(1,$(REF_AREA[id]))'''
>> Total '''$(2,$(REF_AREA[id]))''' A:$(REF_AREA):T:EMP A:$(REF_AREA):T:STU
>> :=LOOP(SEX) '''$(3,$(REF_AREA[id]),$(SEX[id]))''' A:$(REF_AREA):$(SEX):EMP A:$(REF_AREA):$(SEX):STU

Footnote Intermediate Resolution
Row 1 (REF_AREA=UK)
$(A:T:UK:EMP:P:SERIES_TITLE)

Row 2 (REF_AREA=UK)
$(A:T:UK:EMP:P:TITLE_COMPL)

Row 3 (REF_AREA=UK, SEX=M)
$(A:UK:M:EMP:P:OBS_COMMENT)

Final Footnote output
Row 1 (REF_AREA=UK)
This is a series for the United Kingdom

Row 2 (REF_AREA=UK)
This is a title compliment for the United Kingdom

Row 3 (REF_AREA=UK, SEX=M)
This is an observation comment for Males in the United Kingdom

= Formatting Cells =
== Formatting Numerical Observation Values ==
[[File:Obs formatting.png|thumb|Number Formatting Options]]
Numerical observation values can be formatted in the following way

# '''Number of Decimal Places''' or '''Significant Figures''' can be defined
# For Decimal places this can be set to a maximum rule or a fixed rule, i.e the value 1.3 to 2 decimal places is 1.3 using the maximum rule and 1.30 using the fixed rule
# '''Rounding mode''' can be specified
# '''Scaling Factor''' this is a multiplier applied to the observation value, the multiplier is x10^[scaling factor]. Example, a value of 1.1 with a scaling factor of 1 is 11, a scaling factor of 2 is 110, a scaling factor of 3 is 1100. If the Observation or related Series has a UNIT_MULT concept the difference between the unit multiplier and scaling factor will be taken. Example, an Observation with a value of 1.1 and a unit multiplier of 9 (Billions) is reported, the publication table has a scaling factor of 6 (millions) - the number 1.1 is scaled using 10^3 (calculated by 9-6), the 1.1 billion is converted to 1100 million.
# '''Scientific notation''' can be requested on output, i.e 123456 to 2 significant figures is 120000 in standard form or 1.2E4 in scientific form
# '''Decimal formatting''' for long numbers, i.e. 12345.8 there is not thousands separator by default, to include a thousand separator set the decimal formatting rule to either a single Language (i.e. French Formatting) or Dynamic formatting, which uses the Accept-Langauge locale/language sent be the web browser (this can be overridden using the locale=[loc] query parameter. The formatting rules can also influence which characters the numerical characters used.
# Rules can be applied to all series, or specific series

=== Match Series ===
When creating a formatting rule, it is possible to define which series the rule applies to. If left blank, the rule will apply to all observation values.

The syntax for defining specific series is shown in the following table:

{| class="wikitable"
|-
! Rule Type !! Syntax !! Description
|-
| Series Match || =A:UK:EMP || Match a specific series from the default Dataflow
|-
| Series Match Multiple Values || =A:UK+FR+DE:EMP || Multiple codes in 2nd Dimension (UK, FR and DE) any can match for the rule to be valid
|-
| Series Match by Dataflow Alias || =FLOW2.A:UK:EMP || Match a specific series for the Dataflow with alias FLOW2
|-
| Wildcard Series Match || =A::EMP || Match all series with the given dimension values (2nd Dimension is wildcarded)
|-
| Not Rule || !=A:UK:EMP || Match all series which do not match this series (wildcard series also supported)
|-
| Dimension Rule || CURRENCY=GBP || Match all series which have CURRENCY of GBP
|-
| Not Dimension Rule || !CURRENCY=GBP || Match all series do not have a CURRENCY of GBP
|-
| Multiple Rules || CURRENCY=GBP,FREQ=M || Comma is used to include multiple matches (also applies to series match)
|}

== Formatting Missing Observations ==

Observation Values can be missing for 2 reasons:
# There is no Observation in the database for the given Observation Key
# There is an Observation in the database but the measure value is either: NaN, null, or an empty string

In both cases it is possible to define an output value using formatting rules. For the first case, the observation does not exist, it is possible to define a default output value, for example ‘-‘. For the second case, where the observation does exist, it is possible to define a set of rules based on another Component of the Observation, for example the Observation output may be related to the OBS_STATUS Attribute.

When the formatted value is based on a Component, a set of rules must be defined which state ‘when component=x output y’, where x may be an fixed value or a pattern to match (using a regular expression). The rules for mapping component values are defined in a [https://fmrwiki.sdmxcloud.org/Representation_Map Representation Map], and the Publication Table simply provides the connection to the Representation Map to use. This enables reuse of the mapping rules across multiple Publication Tables. The Publication Table must define two properties in order to support Component Mapping rules:

# The Component to obtain the value from i.e OBS_STATUS
# A link to the [https://fmrwiki.sdmxcloud.org/Representation_Map Representation Map], which holds the mapping rules. Note, the Representation Map is expected to have 1 source mapping, this could be of any type (string, codelist, other)

An example Reprsentation Map for OBS_STATUS is shown below:

{| class="wikitable"
|-
! OBS_STATUS !! Is Reg Ex !! Missing Value !! Description
|-
| M || false || mv || When OBS_STATUS is M output mv
|-
| L || false || np || When OBS_STATUS is L output np
|-
| .* || true || - || When OBS_STATUS is anything else output -
|}

== Frequency Formatting ==
[[File:PublicationTable FreqMap.png|thumb|Representation Map with frequency mapping rules]]
[[File:PublicationTable Materialised FreqMap.png|thumb|Materialised table with formatted time headings ]]
Time Variables always resolve themselves to an [[SDMX_Time_Formats|SDMX Date Format]] when forming the [[#Observation Key|Observation Key]], this ensures the correct observation value is returned from the data store. When a time variable is used as a label, in a table heading for example, the same formatting will be applied by default but a custom format can be provided. Custom display formats for a Time Period enable a time period such as 2007-Q1 to be resolved to custom text string, for example '31 Mar 2007'.

Custom Time formats rules are defined by providing an association between an SDMX Frequency ID (#see [[SDMX_Time_Formats|SDMX Date Format]]) and the output format template. The output format template is a string that defines which components of time are output an in which order and style. For example to output a date as 'March 2001' the format template would look like this 'MMMM yyyy' to indicate a full month name (MMMM) and a 4 character year (yyyy). format rules make use of the [https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html Simple Date Format] which is used by Java as such all of the documentation on Simple Date Format applies.

Frequency Format Mapping is not defined as part of a Publication Table, instead the rules are defined in a [https://fmrwiki.sdmxcloud.org/Representation_Map Representation Map] and the Publication Table references the Representation Map as the source of its frequency conversion rules. This enables the same set of rules to be used for multiple Publication Tables. The Representation Map must map 1 source (Codelist or String) to 1 target (String), it can contain as many mapped frequencies as required. If the Representation Map contains a mapping for a value that is not a valid SDMX Frequency, then it will be ignored.

It is important to note that all time periods resolve to a single point in time, which is the end of the period, for example an Annual period of 2008 resolves to 2008-12-31:T23:59:59. The resolution of frequency to a point in time enables the frequency formatting rules to use all aspects of time, even when converting a low frequency, for example 2008 formatted using the pattern dd MMM yy would resolve to '31 Dec 2008'.

== Variables Summary ==

{| class="wikitable"
|-
! Variable Type !! Description !! Syntax !! Example !! Example Output
|-
| '''Dataflow Name''' || Displays the name of a Dataflow referenced by the Publication Table in the locale in which the Publication Table is built || $([dataflow alias]) || $(EDU) || World Bank, Education
|-
| '''Dataflow Description''' || Displays the description of a Dataflow referenced by the Publication Table in the locale in which the Publication Table is built || $([dataflow alias].desc) || $(EDU.desc) || World Bank, Education
|-
| '''Dataflow Id''' || Displays the ID of a Dataflow referenced by the Publication Table || $([dataflow alias].id) || $(EDU.id) || DF_EDU
|-
| '''Variable Dimension Value''' || Displays the name of the variable used when an Observation key has a wildcard member || $([dimension id]) || $(REF_AREA) || United Kingdom
|-
| '''Specific Component Value''' || Displays the name of the Component Value || $([component id].[component value]) || $(OBS_STATUS.C) || Confidential
|-
| '''Component Name''' || Displays the name of the DSD Component (this is the name of the Concept used by the Component) in the locale in which the Publication Table is built || $([component id].name) || $(OBS_STATUS) || Confidentiality Status
|-
| ''' Enumeration Value ''' || Displays the name of a Code in a Codelist or Value in a Valuelist || $([list alias].[list item ID] || $(SEX.M) || Male
|-
| '''Conditional Variable''' || Displays the variable value, which is conditional on the value of a variable Dimension || $([conditional variable id][variable id]) || $(BANKNOTES[NOTE1]) || £5
|-
| ''' Time ''' || Displays the base time period for which the table is built, or an offset from this time period / cast to a different frequency || see [[#Time Functions|Time Functions]] || $(P) || 2008
|}

= Table Variables =
== Overview ==
Variables are a very important part of Publication Tables. They can be used to make the content of the table dynamic, setting and changing the value at run time to change the table content. Variables can be used in Observation Keys, but the values of variables can also be used anywhere where there is text, for example the table heading, subheading, column or row headings, footnotes, and Observation Cells with free text.

The general syntax for a variable is

$(''variable id''.''sub part''[''detail''])

Where variable id is always required, the subpart and detail are optional. Subpart is used if the variable resolves to a Component and a coded value is required, the detail array part obtains the id, name, or description of the underlying SDMX type.

This section describes the various variables that exist, along with the syntax of how to use them.

== Dataflow Variables ==
Publication Tables define data cells across one or more Dataflows. Each ‘imported’ Dataflow is given an alias, which can be used to refer to the Dataflow in the [[#Observation Key|Observation Key]].

It is possible to refer to the Dataflow's Name, Description, and Id using the following variable syntax.

'''Default Dataflow''' 
Dataflow Id
$([id])

Dataflow Name
$() or $([name))

Dataflow Description
$([desc])

'''Dataflow with Alias=EXR''' 
Dataflow Id
$(EXR[id])

Dataflow Name
$(EXR) or $(EXR[name))

Dataflow Description
$(EXR[desc])

== Dimension Variables ==
When an [[#Observation Key|Observation Key]] has [[#Variable Dimension Members|variable Dimension Values]], the table will be built by replacing this variable with a value to complete the Observation Key. The variable value is either passed explicitly to the web service, or defaulted by the system if not provided. The Dimension Variable can be referenced in the table using the syntax $([dimension id]), for example:

Dimension Value Name
$(INDICATOR) resolves to 'Employment'

This would resolve to the '''name''' of the Indicator value used in the table, for example '''EMP''' may resolve to the label '''Employment'''. The label for the Dimension Value is obtained from the Codelist or Valuelist which the Dimension uses. If the Dimension does not use a list (i.e. it is free text or numerical) then the Dimension value is simply used in place of the variable placeholder.

To resolve to the '''id''' of the value for the Component, for example '''EMP''', use the following syntax

Dimension Value Id
$(INDICATOR[id]) resolves to 'EMP'

== Dependent Dimension Variable ==
It is possible that the value for one Dimension is dependent on the value of another Dimension, for example CURRENCY may be dependent on the COUNTRY and COUNTRY is a variable in the [[#Observation Key|Observation Key]] - it is possible to specify that the value to use for the CURRENCY Dimension can be derived using a lookup table. The lookup table in this instance is maintained as an [https://fmrwiki.sdmxcloud.org/Representation_Map SDMX Representation Map] which is used to map COUNTRY to CURRENCY. The Publication Table specifies that the two Dimensions are related and provides a reference to the Representation Map to use which has the rules.

When working with a dependent variable, both the dependent variable and the Dimension on which it depends should be omitted from the [[#Observation Key|Observation Key]] - for example consider a full key of:

A:UK:GBP:2007

Where UK is the COUNTRY and GBP is the Currency, to make the COUNTRY a variable Dimension, it should be omitted from the key, as follows:

A::GBP:2007

To then make the CURRENCY a variable it is also omitted , as follows:

A:::2007

The Publication Table is given the rule that CURRENCY depends on COUNTRY and should use the COUNTRY_2_CURRENCY [https://fmrwiki.sdmxcloud.org/Representation_Map Representation Map], which in this example looks like the following:

{| class="wikitable"
|-
! COUNTRY !! Is Reg Ex !! CURRENCY
|-
| UK || false || GBP
|-
| US || false || USD
|-
| .* || true || EUR
|}

When the Publication Table is materialised the [[Publication_Table_Web_Service|web service]] only requires the COUNTRY information, and it will complete the Observation key using the representation map lookup table.

'''Note''' it is possible to override the mapped value by passing in an explicit override to the web service. For example passing to the web service both COUNTRY=UK with CURRENCY=EUR will override the mapping behaviour for CURRENCY.

== Component Values ==
Components of the underlying Data Structure can be explicitly referenced using the following syntax '''$([component id])''', for example:

Component Name
$(OBS_STATUS) or $(OBS_STATUS[name]) resolves to 'Observation Status'

Component Description
$(OBS_STATUS[desc]) resolves to 'Information on the quality of a value or an unusual or missing value.'

== Coded Variables ==
To reference a Code value, the syntax is $([component id / codelist alias].[item id]). The first variable part is the ID of a Component in the underlying Data Structure, or alternativly the alias of an external variable defined on the Publication Table. An external variable is a link to an SDMX Codelists or Valuelist. Example are:

Name of Code
$(CURRENCY.$) resolving to the text '''Dollar''

Description of Code
$(DWELLING.SM[desc])

== Conditional Variables ==
A Conditional Variable is one whose value is conditional on the value of a [[#Dimension Variables|Dimension Variable]].

An example use case is Banknotes, where the values for NOTE1, NOTE2, or NOTE3 is dependant on the Country Dimension. A Conditional Variable could be used to set a values for each variable (NOTE1, NOTE2, NOTE3) based on the value used for the current Country Dimension i.e. US NOTE1=$5, UK NOTE1=£5.

The syntax of referring to a Conditional Variable is $([conditional variable id][variable id]), for example
$(BANKNOTE[NOTE1])

== Reported Attribute for an Observation or Series ==
A variable can be a reference to a part of an Observation, for example the Unit Multiplier (UNIT_MULT). A Table Row header, for example, could include variable text such as 'Number of payment accounts (''thousands'')' where the word '''thousands''' is a variable because it depends on other variables which build the key.

To achieve this, use the syntax:

$(A:UK::2008:UNIT_MULT[name])

The observation key in this case is A:UK::2008, where the third Dimension is a variable as it has no value (in the main body of the table this dimension would probably also be a variable, which means the table can be built with different values, each value may change the output of this UNIT_MULT value). The last part of the key is the Attribute ID (UNIT_MULT) - this can be a reference to a series attribute, or an observation attribute. The [name] indicates that it should resolve to the name of the reported attribute value, alternative values are '''id''' for the reported value or '''desc''' for the description of the reported value.

If time is also a variable, then the key can change accordingly, using the variable syntax for time (P):

$(A:UK::P:UNIT_MULT[name])

== Time Variables ==
[[File:PublicationTable MovingTime.png|thumb|Publication Table Design, moving and converting the base time period]]

[[File:PublicationTable Materialised MovingTime.png|thumb|Moving and converting periods - showing the materialised table]]

When the data is time series (i.e. the [[Data_Structure_Definition_(DSD)_-_Structural_Metadata_Management|DSD]] has a Time Dimension), the [[#Observation_Key|Observation Key]] can include the time period, for example:

A:UK:EMP:'''2008'''

Where 2008 is the Observation Time Period to use. '''The format of the Time Period must always be the same format as the frequency for which the data are reported'''. For example if the data is annual, and an observation is reported for 2008, then the observation key must use 2008, not another frequency such as 2008-12-31. It is typical in a [[Data_Structure_Definition_(DSD)_-_Structural_Metadata_Management|DSD]] for the Frequency of the Series to be the first Dimension, in all the examples on this page this convention is applied, and as such the above key A:UK:EMP refers to Annual data, depicted by the initial dimension value 'A'.

If the key omits the time period, then the '''time period becomes a variable''' which can be set at table build time, if not set it defaults to the last period for which there is data.

A:UK:EMP

When a Publication Table has a variable time period, the resolved variable time value for this varaible is known as the '''base period'''. There can only be 1 base period for a Publication Table, however other Observation Keys (and corresponding headings) can perform time arithmetic from the base period for which the table is built.

The supported time functions include walking forwards and backwards in increments of the given frequency, converting to another frequency of data, and a combination of the two (convert and move, move and convert, move convert and move again). Ths supported functions are shown in the table below.

{| class="wikitable"
|-
! Syntax !! Description !! Example Observation Key !! Example Resolved Key
|-
| P || The base period (default if not provided) || A:UK:EMP:P || A:UK:EMP:2008 (base period=2008)
|-
| P-''n'' || Base Period minus 'n' Periods where n is a positive Integer || A:UK:EMP:P-1 || A:UK:EMP:2007 (base period=2008)
|-
| P+''n'' || Base Period plus 'n' Periods where n is a positive Integer || A:UK:EMP:P+1 || A:UK:EMP:2008-Q2 (base period=2008-Q1)
|-
| A(P) || Base Period converted to '''Annual''' || A:UK:EMP:A(P) || A:UK:EMP:2008 (base period=2008-Q2)
|-
| S(P) || Base Period converted to '''Bi-Annual''' (Semester) || S:UK:EMP:S(P) || S:UK:EMP:2008-S2 (base period=2008-Q3)
|-
| T(P) || Base Period converted to '''Tri-Annual''' || T:UK:EMP:T(P) || T:UK:EMP:2008-T3 (base period=2008)
|-
| Q(P) || Base Period converted to '''Quarterly''' || Q:UK:EMP:Q(P) || Q:UK:EMP:2008-Q4 (base period=2008)
|-
| M(P) || Base Period converted to '''Monthly''' || M:UK:EMP:M(P) || M:UK:EMP:2008-03 (base period=2008-Q1)
|-
| W(P) || Base Period converted to '''Weekly''' || W:UK:EMP:W(P) || W:UK:EMP:2008-04 (base period=2008-01-01)
|-
| D(P) || Base Period converted to '''Daily''' || D:UK:EMP:D(P) || D:UK:EMP:2008-12-31 (base period=2008)
|-
| H(P) || Base Period converted to '''Hourly''' || H:UK:EMP:D(P) || H:UK:EMP:2008-12-31T12 (base period=2008-02-02T12:00:03)
|-
| A(P-n) || Base Period minus (or plus) 'n' periods, converted to another frequency 'A' || A:UK:EMP:A(P-4) || A:UK:EMP:2007 (base period=2008-Q4)
|-
| A(P)-n || Base Period, converted to another frequency 'A' minus (or plus) n' periods || A:UK:EMP:A(P)-4 || A:UK:EMP:2004 (base period=2008-Q4)
|-
| A(P-n1)-n2 || Base Period, minus (or plus) 'n1' periods, converted to another frequency 'A' minus (or plus) 'n2' periods || A:UK:EMP:A(P-4)-4 || A:UK:EMP:2003 (base period=2008-Q4)
|-
| FA(P) || Subtracts 1 Period from the Converted Period IF the resolved Convert From end of period does not match the same point in time as the Convert To period.

'''Examples:'''
* Base Period =2008-Q4 Rule= FA(P) Output = 2008
* Base Period =2008-Q3 Rule= FA(P) Output = 2007 - the base period 2008-Q3 is not the same time as 2008 (end of period, so move back 1 year)
* Base Period =2008-Q1 Rule= FA(P) Output = 2007
* Base Period =2008-Q1 Rule= FA(P+3) Output = 2008
* Base Period =2008-Q1 Rule= FA(P+3)-1 Output = 2007

This works for ALL frequencies where the convert to frequency is lower then the base Period P, for example Quarter converted to Annual.

|| A:UK:EMP:FA(P) || A:UK:EMP:2007 (base period=2008-Q3)
|}

'''Compound expressions''' 
From Fusion Registry v11.5.1 and later it is possible to cast a period more the once in the same expression, for example

Q(FA(P-1))-1

If the input time period P=2019-Q4, then P-1 = 2019-Q3, the expression is rewritten as Q(FA(2019-Q3))-1. The Full Annual (FA) for 2019-Q3 is 2018, as the end of period 2019 does not match the end of period 2019-Q3, so the annual period moves back an additional period. The expression is rewritten as Q(2018)-1. 2018 to Quarterly moves to 2018-Q4 and then the last part of the expression steps back 1 period, the final period is 2018-Q3.

'''Notes on Time Period''' 
When converting from a lower frequency to a higher frequency (Weekly to Daily for example), the cast date always resolves to the end of the Period.

It is also important to note that as the [[#Observation Key|Observation Keys]] in these examples all have Frequency as the first Dimension, when converting to another frequency (Quarterly to Annual for example) the Observation Key changes the value for the Frequency Dimension, ensuring the correct series of data is retrieved.

'''Time Period variable in free text''' 
When refering to the Time Period as a variable in free text, for example in the table heading or a column heading, the variable is placed inside the variable syntax '''$()'''. For example $(P) would resolve to base period, and $(A(P)) would resolve to 'base period converted to annual frequency'. An example usage is shown below:

Table 1A showing data for $(P) to $(P+3)

Fusion Registry Command Line

2023-03-10T09:19:09Z

Mnelson: /* Data */

= Overview =
The Fusion Registry command line client (FusionRegistryCL) runs on a Windows or UNIX command line, to support loading content to the Fusion Registry via the Fusion Registry secure web services.

FusionRegistryCL is distributed with a single properties file fusionregistrycl.properties. This file needs to be modified to point to the server URL of the Fusion Registry to communicate with.

Batch files (Windows) and sh files (UNIX) are provided to support loading (and deleting) structures, data, and reference metadata.

= Publishing Content =

== Structures ==
To Publish structures call the '''publishStructures.bat''' (or sh) with the following commands:

{| class="wikitable"
|-
! Argument !! Required !! Purpose !! Example
|-
| uri || true || The location of the Structures to load. This can resolve to a web URL or local file system. The format of the structure file can be any supported Fusion Registry format. The file may also be in zip format.||

'''File''' -uri "file:///C:/Sample/Structures.xml" -uri "file:///C:/Sample/Structures.zip" '''URL''' -uri "http://myserver/rest/conceptscheme" -uri "http://myserver/file/structures.xml"
|-
| ds || false || The Data Source argument will create a link between all submitted Provision Agreements and a Data Source in the Fusion Registry. See Data Source Manager of Fusion Registry.
If not provided the Provision Agreements will default to using the REST or File Data Registration mechanism for data.
|| '''Fusion Store''' 
-ds FUSION 
'''Custom Data Store''' 
-ds MY_SQL_STORE
|-
| a || false || Action – allowable actions are:
* Append
* Replace
* FullReplace
* Delete
The publish action, defaults to Replace.
||
'''Delete''' 
-a DELETE 
'''REPLACE''' 
-a REPLACE 
'''FullReplace''' 
-a FullReplace

|-
| u || false || Username of a Fusion Registry user with permissions to load the structure file. 
Not required if using Certificates for Authentication.
|| -u myuser
|-
| p || false || Password of a Fusion Registry user with permissions to load the structure file. Not required if using Certificates for Authentication. || -p mypass
|-
| o || false || An output file to record which structures were added, modified or removed. || -o "C:/temp/Result.csv"
|-
| cert || false || File of Certificate to be used to authenticate with server (if username and password not used). Not required if using Username/Password for Authentication. || -cert "C:/Cert/mycert.p12"
|-
| certPwd || false || The Certificate Password. || -certPwd "password"
|-
| certType|| false || The Certificate Type to use. Defaults to PKCS12 if not specified || -certType "PKCS12"
|}

== Data ==
To Publish data call the '''publishData.bat''' (or sh) with the following commands:

{| class="wikitable"
|-
! Argument !! Required !! Purpose !! Example
|-
| uri || true || The location of the Data to load. This can resolve to a web URL or local file system. The format of the data file can be any supported Fusion Registry format. The file may also be in zip format. If this URI resolves to a folder on the file system, all the files in the folder will be submitted for import via a single zip file. It is possible to load multiple files/folder’s by supplying multiple Uris each separated by a semicolon
||
'''File''' 
-uri "file:///C:/Sample/Data.xml" 
-uri "file:///C:/Sample/Data.zip" 
'''Folder''' 
-uri "file:///C:/SampleDir" 
'''URL''' 
-uri "http://myserver/rest/data" 
-uri "http://myserver/file/data.xml" 
'''Multiple''' 
-uri "http://myserver/rest/data; http://myserver/file/data.xml; file:///C:/SampleDir"
|-
| ref|| false || Optionally a Provision Agreement reference can be supplied, so the server can identify the Dataflow and Data Provider.
This is not required if the data file contains this information. 
The format is Agency Id, Provision Id, Provision Version.
|| -ref "ACY,PROV_ID,1.0"
|-
| u || false || Username of a Fusion Registry user with permissions to load the structure file. 
Not required if using Certificates for Authentication.
|| -u myuser
|-
| p || false || Password of a Fusion Registry user with permissions to load the structure file. Not required if using Certificates for Authentication. || -p mypass
|-
| o || false || An output file to record which structures were added, modified or removed. || -o "C:/temp/Result.csv"
|-
| cert || false || File of Certificate to be used to authenticate with server (if username and password not used). Not required if using Username/Password for Authentication. || -cert "C:/Cert/mycert.p12"
|-
| certPwd || false || The Certificate Password. || -certPwd "password"
|-
| certType|| false || The Certificate Type to use. Defaults to PKCS12 if not specified || -certType "PKCS12"
|}

== Reference Metadata ==
'''Note''' Discontinued at version 2.0.0

To Publish metadata call the '''publishMetadata.bat''' (or sh) with the following commands:

{| class="wikitable"
|-
! Argument !! Required !! Purpose !! Example
|-
| uri || true || TThe location of the Metadata to load. This can resolve to a web URL or local file system. The format of the file can be any supported Fusion Registry format. The file may also be in zip format.
||
'''File''' 
-uri "file:///C:/Sample/Metadata.xml" 
-uri "file:///C:/Sample/Metadata.zip" 
'''URL''' 
-uri "http://myserver/rest/metadata" 
-uri "http://myserver/file/metadata.xml"
|-
| a || false || The action. If the action is DELETE and a metadata file is provided, then the submission will delete the metadata.
If the action is DELETE and a metadata file is not provided, then an id argument is expected.
|| -a "DELETE"
|-
| id || false || If the action is DELETE and a URI is not provided, then this argument can be used to identify the metadata set to delete by its id.
|| -id "MY_METADATASET"
|-
| u || false || Username of a Fusion Registry user with permissions to load the structure file. 
Not required if using Certificates for Authentication.
|| -u myuser
|-
| p || false || Password of a Fusion Registry user with permissions to load the structure file. Not required if using Certificates for Authentication. || -p mypass
|-
| o || false || An output file to record which structures were added, modified or removed. || -o "C:/temp/Result.csv"
|-
| cert || false || File of Certificate to be used to authenticate with server (if username and password not used). Not required if using Username/Password for Authentication. || -cert "C:/Cert/mycert.p12"
|-
| certPwd || false || The Certificate Password. || -certPwd "password"
|-
| certType|| false || The Certificate Type to use. Defaults to PKCS12 if not specified || -certType "PKCS12"
|}

= Exporting Content =
== Data ==
A full data export can be run against a Dataflow, in a specified format. The export is written to a file, whose name is specified as an argument to the export function.

To export data call the '''exportData.bat''' (or sh) with the following commands:

{| class="wikitable"
|-
! Argument !! Required !! Purpose !! Example
|-
| ex || true || The dataflow to export all the data for, the format is Agency Id, Dataflow Id, Dataflow Version || -ex "ACY,DF_ID,1.0"
|-
| exfmt || true || The Export format. Allowed values vary by format, please refer to the [https://wiki.sdmxcloud.org/Category:SdmxDataFormat Formats] page to select a data format and check the parameters for that format || -exfmt "[[SDMX-ML_Generic_Data|sdmx-generic-2.1]]" -exfmt "[[SDMX-CSV|sdmx-csv&labels=both&serieskey=exclude]]"
|-
| exf || true ||
Export file, the file will be created if it does not exist, or deleted and recreated if it does exist.

To export as a compressed zip file, ensure the file name ends in .zip

If the file postfix is not provided, it will be determined by the server based on the export format

The file name may include placeholders for Dataflow Agency Id (%agency%), Dataflow Id (%id%), and Dataflow version (%version%).

The placeholders will be replaced with the values from the dataflow
||
'''XML''' 
-exf "genricdata.xml" 
'''ZIP''' 
-exf "genericdata.zip" 
'''Placeholders''' 
-exf "%agency%_%id%_%version%" 
'''No Postfix''' 
-exf "myexport"
|-
| s|| false || Timeout time in seconds, default is 120 || -s 600
|-
| pp || false || Pretty Print. Only required for SDMX-ML data formats || -pp true
|-
| params || false || Query Parameters. Additional REST query parameters to use in the request || -params "param1=value1&param2=value2"
|-
|}

= Deleting Content =
== Data ==
{| class="wikitable"
|-
! Argument !! Required !! Purpose !! Example
|-
| df || df or pa is required || The dataflow to delete all the data for, the format is Agency Id, Dataflow Id, Dataflow Version || -df "ACY,DF_ID,1.0"
|-
| pa || df or pa is required || The provision agreement to delete all the data for, the format is Agency Id, Dataflow Id, Dataflow Version || -pa "ACY,PA_ID,1.0"
|-
| ds || true || The Id of the Data Source to delete the data from. (see Data Source Manager of Fusion Registry) ||
'''Fusion Store''' 
-ds FUSION 
'''Custom Data Store''' 
-ds MY_SQL_STORE
|-
| u || false || Username of a Fusion Registry user with permissions to load the structure file. 
Not required if using Certificates for Authentication.
|| -u myuser
|-
| p || false || Password of a Fusion Registry user with permissions to load the structure file. Not required if using Certificates for Authentication. || -p mypass
|-
| o || false || An output file to record which structures were added, modified or removed. || -o "C:/temp/Result.csv"
|-
| cert || false || File of Certificate to be used to authenticate with server (if username and password not used). Not required if using Username/Password for Authentication. || -cert "C:/Cert/mycert.p12"
|-
| certPwd || false || The Certificate Password. || -certPwd "password"
|-
| certType|| false || The Certificate Type to use. Defaults to PKCS12 if not specified || -certType "PKCS12"
|}

= Data Validation =
To validate data call the '''validateData.bat''' (or sh) with the following commands:

The output is the same as [[Data_Validation_Web_Service#Response|calling the web service directly]].

{| class="wikitable"
|-
! Argument !! Required !! Purpose !! Example
|-
| dsd || false || The Data Sructure to validate the dataset against. '''Note:''' dsd, df, and pa are mutually exclusive || -dsd "ACY,DSD_ID,1.0"
|-
| df || false || The Dataflow to validate the dataset against. '''Note:''' dsd, df, and pa are mutually exclusive || -df "ACY,DF_ID,1.0"
|-
| pa || false || The Provision Agreement to validate the dataset against. '''Note:''' dsd, df, and pa are mutually exclusive || -pa "ACY,PA_ID,1.0"
|-
| uri || true || The data file to validate ||
'''File''' 
-uri "file:///C:/Sample/Data.xml" 
-uri "file:///C:/Sample/Data.zip" 
'''URL''' 
-uri "http://myserver/rest/data" 
-uri "http://myserver/file/data.xml"
|-
| f || true || Denotes the format and delimiter. Only required if data format is CSV. See data-format in Fusion Registry web services guide (validation section) || csv;delimiter=[delimiter] Where [delimiter] is either:
* comma
* tab
* semicolon
'''Comma Separated''' 
-f csv:delimiter=comm
|-
| o || false|| The output file, if not present then the validation report will be written to the System.out ||
'''Windows''' 
-o c:/mydir/myFile.json
'''Unix''' 
-o /mydir/myFile.json
|-
| u || false || Username of a Fusion Registry user with permissions to load the structure file. 
Only required if Fusion Registry is enforcing user login and if not using Certificate for Authentication
|| -u myuser
|-
| p || false || Password of a Fusion Registry user with permissions to load the structure file. Only required if Fusion Registry is enforcing user login and if not using Certificate for Authentication. || -p mypass
|-
| cert || false || File of Certificate to be used to authenticate with server (if username and password not used). Only required if Fusion Registry is enforcing user login and not required if using Username/Password for Authentication. || -cert "C:/Cert/mycert.p12"
|-
| certPwd || false || The Certificate Password. || -certPwd "password"
|-
| certType|| false || The Certificate Type to use. Defaults to PKCS12 if not specified || -certType "PKCS12"
|}

Data Formats

2023-03-10T09:15:30Z

Mnelson: /* CSV Formats */

= Overview =
Fusion Registry accepts and outputs datasets in a number of formats. When consuming data, the Fusion Registry will analyse the dataset to try to determine what data format it has received, so that it is able to direct it to the right reader. All datasets are read by the Fusion Registry in excatly the same way, so any data processing performed on a Dataset is the same, regardless of the input Data Format.

When querying for data from the Registry web service, or performing a Data Transformation via the Web service, the output data format is described using the [https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html HTTP Accept Header] which describes the required data format.

= HTTP Accept Headers =

==SDMX Formats==
SDMX Fomats are supported as described by the [https://github.com/sdmx-twg/sdmx-rest/blob/master/v2_1/ws/rest/docs/4_6_conneg.md web services specification]

Accept Headers
{| class="wikitable"
|-
! Accept Header !! Format
|-
| application/vnd.sdmx.data+json;version=1.0.0|| [[SDMX-JSON Data|SDMX JSON]]
|-
| application/vnd.sdmx.data+csv;version=1.0.0|| [[SDMX-CSV|SDMX CSV]]
|-
| application/vnd.sdmx.data+edi|| [[SDMX-EDI Data|SDMX EDI]]
|-
| application/vnd.sdmx.structurespecificdata+xml;version=2.1|| [[SDMX-ML Structure Specific Data|Structure Specific]] (2.1)
|-
| application/vnd.sdmx.structurespecificdata+xml;version=2.0|| Compact (2.0/1.0)
|-
| application/vnd.sdmx.genericdata+xml;version=2.1|| [[SDMX-ML Generic Data|Generic]] (2.1/2.0/1.0)
|}

==CSV Formats==
There are a number of CSV 'flavours' supported by Fusion Software, including the [[SDMX-CSV]] format which is an official SDMX data format.

The following Formats and correspoding VND Headers are supported: 
* [[SDMX-CSV]] application/vnd.sdmx.data+csv
* [[Fusion-CSV]] application/vnd.csv
* [[Fusion-CSV-TS]] application/vnd.csv-ts

Each VND Header can then take the additional arguments of, note SDMX-CSV only supports a subset of these arguments or supported values.
{| class="wikitable"
|-
! Argument !! Description !! Supported Values !! Default !! Supported Formats
|-
| version || the version of the format|| 1.0.0, 2.0.0 [[SDMX-CSV]] only || 1.0.0 || [[SDMX-CSV]], [[Fusion-CSV]], [[Fusion-CSV-TS]]
|-
| timeFormat || values are converted to the most granular ISO 8601 representation taking into account the highest frequency of the data in the message|| original or normalized || original || [[SDMX-CSV]]
|-
| labels || output both code/concept ids and the respective labels in the specified language || both/id/name || id || [[SDMX-CSV]] (with the exception of labels=name), [[Fusion-CSV]], [[Fusion-CSV-TS]]
|-
| delimiter || the delimiter to use || comma/tab/semicolon/space || comma || [[Fusion-CSV]], [[Fusion-CSV-TS]]
|-
| serieskey || include the series key as a column A series key is the concatenation of the dimension values for example A:UK:EMPLOYMENT || include/exclude || exclude || [[Fusion-CSV]], [[Fusion-CSV-TS]]
|-
| bom (since 10.3.1) || Include or Exclude the [https://en.wikipedia.org/wiki/Byte_order_mark '''B'''yte '''O'''rder '''M'''ark] (BOM). The BOM helps Excel interpret non Latin characters when opening a CSV file || include/exclude || exclude || [[SDMX-CSV]], [[Fusion-CSV]], [[Fusion-CSV-TS]]
|-
|}

Note:The Labels parameter can be used in conjuntion with the [https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.4 HTTP Accept-Language] Header to indicate which language to resolve the labels in. If the labels are not available in the requested language, another language will be selected, defaulting to English. 

===Examples===
application/vnd.sdmx.data+csv 
application/vnd.sdmx.data+csv;version=1.0.0; 
application/vnd.sdmx.data+csv;version=2.0.0; 
application/vnd.sdmx.data+csv;version=1.0.0;timeFormat=normalized 
application/vnd.sdmx.data+csv;version=1.0.0;timeFormat=normalized;labels=both 

application/vnd.csv 
application/vnd.csv;delimiter=tab 
application/vnd.csv;timeFormat=normalized;serieskey=include 
application/vnd.csv;version=1.0.0;labels=both 

application/vnd.csv-ts 
application/vnd.csv-ts;version=1.0.0; 
application/vnd.csv-ts;version=1.0.0;labels=name 

==[[Reporting Template|Data Reporting Template]]==

There are two types of output when converting data to a Data Reporting Template format. The first is where the Data reporting template is constructed in the usual way, with the [[Data_Reporting_Template#Defining_the_Universe_of_Data|Universe of Data]] being derived from the [[Dataflow]], and related [[Content Constraint|Content Constraints]]. The second is where the [[Data_Reporting_Template#Defining_the_Universe_of_Data|Universe of Data]] is derived from the dataset being written into the Excel workbook. The default output is to base the Report Template Universe on the constraints, to change this behaviour, use the '''+partial''' indicator in the VND Header.

{| class="wikitable"
|-
! Accept Header !! Description
|-
| application/vnd.reporttemplate
|| Excel Report Template pre-populated with the data from a dataset.
 The dataset should contain the Provision Agreement reference, to enable the Fusion Registry to determine the Data Provider
 The excel file will be the same as a Report Template generated via the [[Data_Reporting_Template_Web_Services|Reporting Template Web Service]], but it will be pre-populated with observation and attribute values from the dataset.

|-
| application/vnd.reporttemplate.ACY:BANKING(1.0)
|| This is an extension of application/vnd.reporttemplate, it tells the Fusion Registry which Reporting Template to use. Only required if there is more then one Reporting Template for the Dataflow(s) being written.
|-
| application/vnd.reporttemplate;DATA_PROVIDER=ONS
|| This is an extension of application/vnd.reporttemplate, it tells the Fusion Registry who the Data Provider is.

The Data Provider's Agency defaults to SDMX. If this is not true, use the syntax '''DATA_PROVIDER=ACY_ID.ONS'''

Can be used in conjunction with other VND arguments such as the Report Template identifer.
|-
| application/vnd.reporttemplate+partial
|| This outputs the dataset conforming to the layout of the Report Template, but includes only the worksheets, and observation cells for which there is data in the dataset. There is no main worksheet.

A Data Provider reference is not necessary, however information about which Report Template to use can be provided using the .ACY:TEMPLATE_ID(1.0) syntax.

|}

Data Formats

2023-03-10T09:14:35Z

Mnelson: /* Examples */

= Overview =
Fusion Registry accepts and outputs datasets in a number of formats. When consuming data, the Fusion Registry will analyse the dataset to try to determine what data format it has received, so that it is able to direct it to the right reader. All datasets are read by the Fusion Registry in excatly the same way, so any data processing performed on a Dataset is the same, regardless of the input Data Format.

When querying for data from the Registry web service, or performing a Data Transformation via the Web service, the output data format is described using the [https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html HTTP Accept Header] which describes the required data format.

= HTTP Accept Headers =

==SDMX Formats==
SDMX Fomats are supported as described by the [https://github.com/sdmx-twg/sdmx-rest/blob/master/v2_1/ws/rest/docs/4_6_conneg.md web services specification]

Accept Headers
{| class="wikitable"
|-
! Accept Header !! Format
|-
| application/vnd.sdmx.data+json;version=1.0.0|| [[SDMX-JSON Data|SDMX JSON]]
|-
| application/vnd.sdmx.data+csv;version=1.0.0|| [[SDMX-CSV|SDMX CSV]]
|-
| application/vnd.sdmx.data+edi|| [[SDMX-EDI Data|SDMX EDI]]
|-
| application/vnd.sdmx.structurespecificdata+xml;version=2.1|| [[SDMX-ML Structure Specific Data|Structure Specific]] (2.1)
|-
| application/vnd.sdmx.structurespecificdata+xml;version=2.0|| Compact (2.0/1.0)
|-
| application/vnd.sdmx.genericdata+xml;version=2.1|| [[SDMX-ML Generic Data|Generic]] (2.1/2.0/1.0)
|}

==CSV Formats==
There are a number of CSV 'flavours' supported by Fusion Software, including the [[SDMX-CSV]] format which is an official SDMX data format.

The following Formats and correspoding VND Headers are supported: 
* [[SDMX-CSV]] application/vnd.sdmx.data+csv
* [[Fusion-CSV]] application/vnd.csv
* [[Fusion-CSV-TS]] application/vnd.csv-ts

Each VND Header can then take the additional arguments of, note SDMX-CSV only supports a subset of these arguments or supported values.
{| class="wikitable"
|-
! Argument !! Description !! Supported Values !! Default !! Supported Formats
|-
| version || the version of the format|| 1.0.0 || 1.0.0 || [[SDMX-CSV]], [[Fusion-CSV]], [[Fusion-CSV-TS]]
|-
| timeFormat || values are converted to the most granular ISO 8601 representation taking into account the highest frequency of the data in the message|| original or normalized || original || [[SDMX-CSV]]
|-
| labels || output both code/concept ids and the respective labels in the specified language || both/id/name || id || [[SDMX-CSV]] (with the exception of labels=name), [[Fusion-CSV]], [[Fusion-CSV-TS]]
|-
| delimiter || the delimiter to use || comma/tab/semicolon/space || comma || [[Fusion-CSV]], [[Fusion-CSV-TS]]
|-
| serieskey || include the series key as a column A series key is the concatenation of the dimension values for example A:UK:EMPLOYMENT || include/exclude || exclude || [[Fusion-CSV]], [[Fusion-CSV-TS]]
|-
| bom (since 10.3.1) || Include or Exclude the [https://en.wikipedia.org/wiki/Byte_order_mark '''B'''yte '''O'''rder '''M'''ark] (BOM). The BOM helps Excel interpret non Latin characters when opening a CSV file || include/exclude || exclude || [[SDMX-CSV]], [[Fusion-CSV]], [[Fusion-CSV-TS]]
|-
|}

Note:The Labels parameter can be used in conjuntion with the [https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.4 HTTP Accept-Language] Header to indicate which language to resolve the labels in. If the labels are not available in the requested language, another language will be selected, defaulting to English. 

===Examples===
application/vnd.sdmx.data+csv 
application/vnd.sdmx.data+csv;version=1.0.0; 
application/vnd.sdmx.data+csv;version=2.0.0; 
application/vnd.sdmx.data+csv;version=1.0.0;timeFormat=normalized 
application/vnd.sdmx.data+csv;version=1.0.0;timeFormat=normalized;labels=both 

application/vnd.csv 
application/vnd.csv;delimiter=tab 
application/vnd.csv;timeFormat=normalized;serieskey=include 
application/vnd.csv;version=1.0.0;labels=both 

application/vnd.csv-ts 
application/vnd.csv-ts;version=1.0.0; 
application/vnd.csv-ts;version=1.0.0;labels=name 

==[[Reporting Template|Data Reporting Template]]==

There are two types of output when converting data to a Data Reporting Template format. The first is where the Data reporting template is constructed in the usual way, with the [[Data_Reporting_Template#Defining_the_Universe_of_Data|Universe of Data]] being derived from the [[Dataflow]], and related [[Content Constraint|Content Constraints]]. The second is where the [[Data_Reporting_Template#Defining_the_Universe_of_Data|Universe of Data]] is derived from the dataset being written into the Excel workbook. The default output is to base the Report Template Universe on the constraints, to change this behaviour, use the '''+partial''' indicator in the VND Header.

{| class="wikitable"
|-
! Accept Header !! Description
|-
| application/vnd.reporttemplate
|| Excel Report Template pre-populated with the data from a dataset.
 The dataset should contain the Provision Agreement reference, to enable the Fusion Registry to determine the Data Provider
 The excel file will be the same as a Report Template generated via the [[Data_Reporting_Template_Web_Services|Reporting Template Web Service]], but it will be pre-populated with observation and attribute values from the dataset.

|-
| application/vnd.reporttemplate.ACY:BANKING(1.0)
|| This is an extension of application/vnd.reporttemplate, it tells the Fusion Registry which Reporting Template to use. Only required if there is more then one Reporting Template for the Dataflow(s) being written.
|-
| application/vnd.reporttemplate;DATA_PROVIDER=ONS
|| This is an extension of application/vnd.reporttemplate, it tells the Fusion Registry who the Data Provider is.

The Data Provider's Agency defaults to SDMX. If this is not true, use the syntax '''DATA_PROVIDER=ACY_ID.ONS'''

Can be used in conjunction with other VND arguments such as the Report Template identifer.
|-
| application/vnd.reporttemplate+partial
|| This outputs the dataset conforming to the layout of the Report Template, but includes only the worksheets, and observation cells for which there is data in the dataset. There is no main worksheet.

A Data Provider reference is not necessary, however information about which Report Template to use can be provided using the .ACY:TEMPLATE_ID(1.0) syntax.

|}

Data Publication Web Service

2023-02-14T13:50:45Z

Mnelson: /* Overview */

[[Category:WebService]]
= Overview =
Importing data is an asynchronous request to the Fusion Registry, which receives a token in the response which can be used to track the progress of import. This page describes both the submission web service, and the progress report web service.

{| class="wikitable"
|-
|style="background-color:#eaecf0"|Entry Point|| /ws/secure/data/publish
|-
|style="background-color:#eaecf0"|Access|| Secure Admin or Data Provider
|-
|style="background-color:#eaecf0"|Http Method|| POST
|-
|style="background-color:#eaecf0"|Accepts|| CSV, XLSX, SDMX-ML, SDMX-EDI (any format for which there is a Data Reader)
|-
|style="background-color:#eaecf0"|Compression || Zip files supported, if loading from URL gzip responses supported
|-
|style="background-color:#eaecf0"|Content-Type|| 1. multipart/form-data (if attaching file) – the attached file must be in field name of uploadFile
2. application/text or application/xml (if submitting data in the body of the POST)
|-
|style="background-color:#eaecf0"|Response Format|| Determined by [[Data Formats|Accept Header]] - default SDMX 2.1 Structure Specific
|-
|style="background-color:#eaecf0"|Response Statuses|| 200 - Submisson Accepted
400 - Submission could not be performed (either an unreadable dataset, or unresolvable reference to a required structure)
401 - Unauthorized (if access has been restricted)
500 - Server Error
|}

= HTTP Headers =
The Accept Header is used to define the output format, to transform the data to. The supported accept header values are

In addition, the following optional header parameters can be used to provide further details on the incoming dataset. If these details are not provided, the Fusion Registry will interrogate the dataset header to get the information. If the dataset is a non-SDMX format, or does not contain the required information in the header, then an error response will be returned.

{| class="wikitable"
|-
!|HTTP Header || Purpose || Allowed Values
|-
|style="background-color:#eaecf0"|ProvisionURN ||This can be used to tell the Fusion Registry which Provision Agreement to use for Data Import. This will override the Dataset Header information if it exists. ||
A valid Provision Agreement URN
|-
|style="background-color:#eaecf0"|DatasetAction || Specifies the action to perform when submitting the dataset. If this value is specified it will override any action specified in the dataset.
|| Append, Replace, Delete, or FullReplace
|}

== Server Response ==
The server will respond with a JSON object containing a token to track the progress of the import.

{
"Id" : "unique-token-as-a-UUID"
}

== Error Response ==
If the server rejects the submission request, it will reply with an error reponse in JSON format:

{
"Error": [
"Error while attempting to import data",
"Data Publishing is not supported for Provision Agreement 'urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=WB:WDI_ECON_POL_DEBT(1.0)'. Provision Agreement must be configured to using a Registry Managed or Fusion Data Store. Provision Agreement has been configured to use data from the following source: 'Registration (FILE)'"
]}

= Check Import Progress =
The progress of the import is checked by posting the Id response from the data submission to poll the following service

{| class="wikitable"
|-
|style="background-color:#eaecf0"|Entry Point|| /ws/secure/data/status/find
|-
|style="background-color:#eaecf0"|Access|| Secure Admin or Data Provider
|-
|style="background-color:#eaecf0"|Http Method|| GET
|-
|style="background-color:#eaecf0"|Reponst Format|| application/json
|-
|style="background-color:#eaecf0"|Response Statuses|| 200 - Import Check Success
401 - Unauthorized (if access has been restricted)
500 - Server Error
|}

= HTTP Query Parameters =

{| class="wikitable"
|-
!| Request Parameter || Required || Purpose
|-
|style="background-color:#eaecf0"|id|| Yes || This is the token that was passed back from the Fusion registry server on the publication request
|-
|}

== Progress Check Response ==

The response status, any errors, import action, and general details about the Dataset.

{
"Status": "SUCCESS",
"Username": "user_id",
"Start": 1464781420000,
"Completion": 1464781422000,
"Errors": [],
"Keys": 960,
"Obs": 11220,
"Action": "Information",
"DataflowUrn": "urn…",
"ProvisionUrn": "urn…",
"DatasetId": "2401cfbb-1783-4ee2-ad4f-d900332e0034"
}

Status returned can be one of the following:
* '''PENDING''' – the data is undergoing initial validation
* '''QUEUED''' – the data has been validated and is awaiting database imoport
* '''PROCESSING''' – the data is being loaded into the database
* '''PROCESSED''' – the data has been imported but not yet indexed
* '''INDEXING''' – the fusion registry is reindexing the data
* '''SUCCESS''' – the data import has completed with no errors
* '''ERROR''' – the import resulted in error

If the status is PROCESSING then the response Object will also contain PercentComplete as a number, for example:
{
"Status": "PROCESSING",
"PercentComplete": 72, ...
}

The DatasetId is the unique identifier for the dataset, and will match the 'id' query parameter.

'''NOTE''': Start and End Dates is the measured in number of milliseconds since January 1st 1970.

== Publish Error ==
If the data publication resulted in error the Errors feild is populated with the error message(s). The data publish is set to fail on first error, so the array of errors are only relevent if the error message contains a number of levels of detail. For a more comprehensive error report, use the [[Data_Validation_Web_Service|Data Validation web service]], or the [[Asynchronous_Data_Validation_and_Transformation|Asynchronous equivalent]].

{
Status: "ERROR",
Username: "root",
Start: 1589832922000,
Completion: 1589832922000,
Errors: [ "C=Constraint,P=SERIES,DS=0,K=A:BN_KLT_DINV_CD:AUS,Comp=REF_AREA:AUS,M=Disallowed Dimension Value: REF_AREA=AUS"],
Action: "Information",
DataflowUrn: "urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=WB:WDI_ECON_POL_DEBT(1.0)",
ProvisionUrn: "urn:sdmx:org.sdmx.infomodel.registry.ProvisionAgreement=WB:WB_ECON_DP1(1.0)",
DatasetId: "1af3cb4f-979c-4956-b595-a107f6926ff7"
}

Data Stores

2023-02-14T11:38:28Z

Mnelson: /* Externally Managed Database */

[[Category:Functions]]
= Overview =

The Fusion Registry can act as a Virtual Data Store. This means that the Registry is able to link to datasets accross multiple physical data stores, and each data store can be of a diffent type (database, web service, file). The Fusion Registry is able to query multiple data stores for a single data query and then combine the data at the end, before writing it back to the user in a response format of their choice.

== Registry Managed Data Store ==
A Registry Managed Data Store is one which the Registry maintains. This means as data is published to the Fusion Registry, the Registry wil write the data to the store and manage the transaction. Registry managed data stores come in two flavours, a relational database (MySQL or SQL Server) store, or a Fusion Store which has been custom built by Metadata Technology for rapid storage and dissemination of data.

=== Fusion Data Store ===
The Fusion Data Store is an in-memory database, custom built by Metadata Technology for storing time series SDMX datasets. The data is ultimately persisted to the Registry database, so that when the Fusion Registry server is terminated, and re-launched, the Fusion Data Stores will be read back into memory. For the duration of the server up-time, the data store will live in memory.

=== Registry Managed Database ===
A Registry Managed Database includes support for SQL Server and MySQL database platforms, MariaDB*. The Fusion Registry is responsible for creating the database tables when new datasets are loaded. The Fusion Registry will also update table definitions if the [[Data_Structure_Definition|Data Structure Definition]] is changed.

*Note Maria DB re-writes a query for both lastNObservation and firstNObservations to [https://dba.stackexchange.com/questions/117954/is-a-ranking-querys-order-by-in-from-subquery-is-preserved optimise away] a required part of the SQL query. MariaDB can be configured to run in MySQL compatability mode, where this error does not occur.

The database table structure is a two or three tables per Dataflow. The table names are based on the Dataflow, but can be changed if required. The tables include:
* Table for Series Definitions
* Table for Observations
* Table data dataset attributes (only created if Dataset Attributes exist on the Data Structure Definition)

==== Column Names ====
The Series Table will have the following Columns created.

{| class="wikitable"
|-
! Column Name !! Column Type !! Purpose
|-
| ID || Varchar || A concatenation of the Dimension values that make up the series. The width of the ID column is a combination of all the Dimension Widths plus the number of Dimensions.
An example ID is M:FR:EMP. It is important that the column width for each Dimension of the Data Structure Definition is low, i.e if the Dimenison is data type String then place a Max Width restriction on the Dimension's definition so that the database table can be constructed with a primary key that does not exceed the limit set by the database vendor.
|-
| FR_GROUP_ID || varchar|| (optional) Created on if the Data Structure Definition defines a Groups. This coulmn enable the storage of Group level Attributes
|-
| FR_PROVIDER_ACY || varchar || Data Provider's Agency, used if the data query includes Data Provider filters
|-
| FR_PROVIDER_ID || varchar|| Data Provider's Id, used if the data query includes Data Provider filters
|-
| FR_PROVISION || varchar || Provision Agreement short URN e.g ECB:EXR(1.0)
|-
| FR_ERROR || varchar || Internal Use only
|-
| FR_UPDATED_DATE || varchar|| Last updated date for the series
|-
| DIM ID/ATT_ID || varchar|text|| A Column for each Dimension and Series Level Attribute. The Column Name is the ID of the Dimension/Attribute and the Column Type is dependant on the Text Format
|}

The Observation Table will have the following Columns created.
{| class="wikitable"
|-
! Column Name !! Column Type !! Purpose
|-
| ID || int || Auto increment primary key
|-
| FR_KEY_ID || varchar|| Foreign key to the series ID column
|-
| FR_OBS_DATE || datetime || A datetime representaion of the TIME_PERIOD column
|-
| OBS_VALUE|| varchar || The Observation Value
|-
| TIME_PERIOD || varchar || [[SDMX_Time_Formats|Reported Time Period]]
|-
| FR_ERROR || varchar || Internal use only
|-
| FR_EMBARGO || int || Deprecated
|-
| FR_IS_PUBLIC || int || Deprecated
|-
| FR_UPDATED_DATE_OBS || int || Last time the observation was inserted/updated
|-
| FR_VALID_FROM || datetime || Used to support the includeHistory query parameter. A time stamp of when the observation is valid from, either taken from the Header of the dataset, or set to the time the observation was inserted
|-
| FR_VALID_TO|| datetime || Used to support the includeHistory query parameter. A time stamp of when the observation is valid to (if the observation is updated, this is used to capture previous revisions)
|-
| FR_CHKSUM || varchar|| a checksum on the column (internal use)
|-
| ATT_ID || varchar|text|| A Column for each Observation Level Attribute. The Column Name is the ID of the Attribute and the Column Type is dependant on the Text Format
|}

The Dataset Attribute Table, if required, will have the following Columns created.

{| class="wikitable"
|-
! Column Name !! Column Type !! Purpose
|-
| ID || Varchar || The ID of the Dataset Level Attribute.
|-
| VALUE || varchar|| The reported value for the attribute
|-
| FR_ERROR || varchar|| internal use only
|}

==== Column Data Types ====
The column data types that are based on the [[Data_Structure_Definition|Data Structure Definitions]] Dimensions and Attributes are set to Varchar with the width restrictions based on the table below. If the width restriction exceeds 250 characters the column data type is set to TEXT.

Width restrictions are calculated as follows:
{| class="wikitable"
|-
! Component's Representation !! Column Width !! Example Value
|-
| String || Max Length defined on Component or 250 if unspecified || This is a series title
|-
| Codelist || Maximum length of the longest Code Id (e.g UK would be length 2) || UK
|-
| Boolean || 5 || true
|-
| Short || 5 || 32767
|-
| Integer || 10 || 123
|-
| Long || 19 || 123456
|-
| Float || 12 || 12.20
|-
| Double || 22 || 12.20
|-
| Double || 22 || 12.20
|-
| Date || 30 || 2002-01-01T12:15:00.000GMT
|-
| Date Time || 30 || 2002-01-01T12:15:00.000GMT
|-
| Time Period || 30 || 2002-Q1 (any valid [[SDMX_Time_Formats|Time Format]])
|-
| Day || 2 || 31
|-
| Month || 2 || 12
|-
| Year_Month || 7 || 2002-01
|-
| Year || 4 || 2001
|-
| Time || 10 || 12:15:00
|}

Note If the dimenison or attribute is numerical and a maximum and minimum value is set, then this will be used to determine the column width. For example if the Attribute is of type Double, with a min value of 0 and a max value of 10 with a Decimals of 3, then the maximum length is 6, allowing for the values such as 10.123 to be reported.

== Externally Managed Database ==
An Externally Managed Database can be MySQL, SQLServer, or Oracle.

There are two options for the database table design when linking to an externally managed database, these are:
# Mirror the structure used by the Fusion Registry Managed database table structure (Series / Observation).
# A single database table for a Dataflow, where the Series and Observations are in one table

'''Note''' The Table and column names must all be in UPPER CASE.

When the Provision Agreement is linked to an externally managed data store, the user interface will provide the ability to specify what the database tables/views are used for the Series table and the Observation table. The Series table is optional, and if it is not provided then the Single Table Design will be used to query the database table. If the Series table is provided, the Series/Obs table design will be used.

=== Series/Obs Table Design ===
See the [[#Registry_Managed_Database|registry table design]] section for the full structure. The full table structure is not required, but there are some rules required for this table/view design, these are:

# The '''Series table''' must have a primary key OR a Column called ID. If there is a column called ID it will be expected to be the Primary Key column.
# If the '''Series table''' has an ID column AND it is of type Varchar, the values will be expected to be the concatenated series key (A:UK:EMP for example).
# The '''Obs table''' must either have a foreign key to the Series Table, OR a column called FR_KEY_ID which can be used to join to the primary key of the Series Table
# The '''Series table''' is expected to have a Column per Dimension, attribute columns are all optional
# The '''Obs table''' is expected to have a Column for Obs Value and Time Period, attribute columns are all optional
# All columns that are prefixed with 'FR_' are optional

=== Single Table Design ===
The flat table structure is a denormalised structure where the Series and Observations are in one table. This does lead to a repetition of dimension values where a series consists of multiple observations. The flat table structure does not support all the features of a SDMX Query including firstNObservations, lastNObservations, includeHistory.

The table structure is as follows:

{| class="wikitable"
|-
! Column Name !! Column Type !! Required !! Purpose
|-
| DIM _ID || varchar || Required || Column per Dimension where the column name is that of the Dimension Id
|-
| ATTR_ID || varchar || Optional|| Column per Series or Observation Attribute where the column name is that of the Attribute Id
|-
| TIME_PERIOD || varchar || Required || [[SDMX_Time_Formats|Reported Time Period]]
|-
| FR_VALID_FROM || datetime || Optional || Used to support startPeriod, endPeriod, lastNObservations, firstNObservations query parameters
|-
| FR_UPDATED_DATE || datetime || Optional || to support the updatedAfter query parameter
|}

== Custom Table and Column Names ==
It is possible to override the Registrys own table nameing and column naming convention. To do this via a web service, use the following:

{| class="wikitable"
|-
|style="background-color:#eaecf0"|Entry Point|| /ws/secure/datasource/tableMapping
|-
|style="background-color:#eaecf0"|Access|| Private (admin)
|-
|style="background-color:#eaecf0"|Http Method|| POST
|-
|style="background-color:#eaecf0"|Accepts|| application/json
|-
|style="background-color:#eaecf0"|Compression || N/A
|-
|style="background-color:#eaecf0"|Content-Type || application/json
|-
|style="background-color:#eaecf0"|Response Format|| application/json
|-
|style="background-color:#eaecf0"|Response Statuses|| 200 - success 
400 - Bad JSON syntax
401 - Unauthorized
500 - Server Error
|}

The JSON request has the following structure:
{
"URN" : "urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=WB:GCI(1.0)",
"DataSourceId" : "MY_STORE",
"KeyTable" : "GCI_KEY",
"ObsTable" : "GCI_OBS",
"DSAttributeTable" : null,
"MappedColumns" : {
"REF_AREA" : "COUNTRY",
"FR_UPDATE_DATE" : "UPDATED_DATE"
}
}

The URN field is the URN of the Dataflow to map the table/colmn names for. The Mapped Columns field is optional, and only required if column names require mapping as well as table names. The same web service can be used to update a mapping, the Dataflow URN is the key for the mapping.

To delete a mapping call the same web service with method DELETE, and post the following JSON:

{
"URN" : "urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=WB:GCI(1.0)",
"DataSourceId" : "MY_STORE",
}

To retrieve a mapping call the same web service with the following query parameters:

{| class="wikitable"
|-
|style="background-color:#eaecf0"|urn|| urn of the Dataflow to get the table mapping description for
|-
|style="background-color:#eaecf0"|dataSourceId|| Id of the data store to get the table mapping description for
|}

Data Stores

2023-02-14T11:36:25Z

Mnelson: /* Multi Table */

[[Category:Functions]]
= Overview =

The Fusion Registry can act as a Virtual Data Store. This means that the Registry is able to link to datasets accross multiple physical data stores, and each data store can be of a diffent type (database, web service, file). The Fusion Registry is able to query multiple data stores for a single data query and then combine the data at the end, before writing it back to the user in a response format of their choice.

== Registry Managed Data Store ==
A Registry Managed Data Store is one which the Registry maintains. This means as data is published to the Fusion Registry, the Registry wil write the data to the store and manage the transaction. Registry managed data stores come in two flavours, a relational database (MySQL or SQL Server) store, or a Fusion Store which has been custom built by Metadata Technology for rapid storage and dissemination of data.

=== Fusion Data Store ===
The Fusion Data Store is an in-memory database, custom built by Metadata Technology for storing time series SDMX datasets. The data is ultimately persisted to the Registry database, so that when the Fusion Registry server is terminated, and re-launched, the Fusion Data Stores will be read back into memory. For the duration of the server up-time, the data store will live in memory.

=== Registry Managed Database ===
A Registry Managed Database includes support for SQL Server and MySQL database platforms, MariaDB*. The Fusion Registry is responsible for creating the database tables when new datasets are loaded. The Fusion Registry will also update table definitions if the [[Data_Structure_Definition|Data Structure Definition]] is changed.

*Note Maria DB re-writes a query for both lastNObservation and firstNObservations to [https://dba.stackexchange.com/questions/117954/is-a-ranking-querys-order-by-in-from-subquery-is-preserved optimise away] a required part of the SQL query. MariaDB can be configured to run in MySQL compatability mode, where this error does not occur.

The database table structure is a two or three tables per Dataflow. The table names are based on the Dataflow, but can be changed if required. The tables include:
* Table for Series Definitions
* Table for Observations
* Table data dataset attributes (only created if Dataset Attributes exist on the Data Structure Definition)

==== Column Names ====
The Series Table will have the following Columns created.

{| class="wikitable"
|-
! Column Name !! Column Type !! Purpose
|-
| ID || Varchar || A concatenation of the Dimension values that make up the series. The width of the ID column is a combination of all the Dimension Widths plus the number of Dimensions.
An example ID is M:FR:EMP. It is important that the column width for each Dimension of the Data Structure Definition is low, i.e if the Dimenison is data type String then place a Max Width restriction on the Dimension's definition so that the database table can be constructed with a primary key that does not exceed the limit set by the database vendor.
|-
| FR_GROUP_ID || varchar|| (optional) Created on if the Data Structure Definition defines a Groups. This coulmn enable the storage of Group level Attributes
|-
| FR_PROVIDER_ACY || varchar || Data Provider's Agency, used if the data query includes Data Provider filters
|-
| FR_PROVIDER_ID || varchar|| Data Provider's Id, used if the data query includes Data Provider filters
|-
| FR_PROVISION || varchar || Provision Agreement short URN e.g ECB:EXR(1.0)
|-
| FR_ERROR || varchar || Internal Use only
|-
| FR_UPDATED_DATE || varchar|| Last updated date for the series
|-
| DIM ID/ATT_ID || varchar|text|| A Column for each Dimension and Series Level Attribute. The Column Name is the ID of the Dimension/Attribute and the Column Type is dependant on the Text Format
|}

The Observation Table will have the following Columns created.
{| class="wikitable"
|-
! Column Name !! Column Type !! Purpose
|-
| ID || int || Auto increment primary key
|-
| FR_KEY_ID || varchar|| Foreign key to the series ID column
|-
| FR_OBS_DATE || datetime || A datetime representaion of the TIME_PERIOD column
|-
| OBS_VALUE|| varchar || The Observation Value
|-
| TIME_PERIOD || varchar || [[SDMX_Time_Formats|Reported Time Period]]
|-
| FR_ERROR || varchar || Internal use only
|-
| FR_EMBARGO || int || Deprecated
|-
| FR_IS_PUBLIC || int || Deprecated
|-
| FR_UPDATED_DATE_OBS || int || Last time the observation was inserted/updated
|-
| FR_VALID_FROM || datetime || Used to support the includeHistory query parameter. A time stamp of when the observation is valid from, either taken from the Header of the dataset, or set to the time the observation was inserted
|-
| FR_VALID_TO|| datetime || Used to support the includeHistory query parameter. A time stamp of when the observation is valid to (if the observation is updated, this is used to capture previous revisions)
|-
| FR_CHKSUM || varchar|| a checksum on the column (internal use)
|-
| ATT_ID || varchar|text|| A Column for each Observation Level Attribute. The Column Name is the ID of the Attribute and the Column Type is dependant on the Text Format
|}

The Dataset Attribute Table, if required, will have the following Columns created.

{| class="wikitable"
|-
! Column Name !! Column Type !! Purpose
|-
| ID || Varchar || The ID of the Dataset Level Attribute.
|-
| VALUE || varchar|| The reported value for the attribute
|-
| FR_ERROR || varchar|| internal use only
|}

==== Column Data Types ====
The column data types that are based on the [[Data_Structure_Definition|Data Structure Definitions]] Dimensions and Attributes are set to Varchar with the width restrictions based on the table below. If the width restriction exceeds 250 characters the column data type is set to TEXT.

Width restrictions are calculated as follows:
{| class="wikitable"
|-
! Component's Representation !! Column Width !! Example Value
|-
| String || Max Length defined on Component or 250 if unspecified || This is a series title
|-
| Codelist || Maximum length of the longest Code Id (e.g UK would be length 2) || UK
|-
| Boolean || 5 || true
|-
| Short || 5 || 32767
|-
| Integer || 10 || 123
|-
| Long || 19 || 123456
|-
| Float || 12 || 12.20
|-
| Double || 22 || 12.20
|-
| Double || 22 || 12.20
|-
| Date || 30 || 2002-01-01T12:15:00.000GMT
|-
| Date Time || 30 || 2002-01-01T12:15:00.000GMT
|-
| Time Period || 30 || 2002-Q1 (any valid [[SDMX_Time_Formats|Time Format]])
|-
| Day || 2 || 31
|-
| Month || 2 || 12
|-
| Year_Month || 7 || 2002-01
|-
| Year || 4 || 2001
|-
| Time || 10 || 12:15:00
|}

Note If the dimenison or attribute is numerical and a maximum and minimum value is set, then this will be used to determine the column width. For example if the Attribute is of type Double, with a min value of 0 and a max value of 10 with a Decimals of 3, then the maximum length is 6, allowing for the values such as 10.123 to be reported.

== Externally Managed Database ==
An Externally Managed Database can be MySQL, SQLServer, or Oracle.

There are two options for the database table design when linking to an externally managed database, these are:
# Mirror the structure used by the Fusion Registry Managed database table structure (Series / Observation).
# A single database table for a Dataflow, where the Series and Observations are in one table

'''Note''' The Table and column names must all be in UPPER CASE.

=== Series/Obs Table Design ===
See the [[#Registry_Managed_Database|registry table design]] section for the full structure. The full table structure is not required, but there are some rules required for this table/view design, these are:

# The '''Series table''' must have a primary key OR a Column called ID. If there is a column called ID it will be expected to be the Primary Key column.
# If the '''Series table''' has an ID column AND it is of type Varchar, the values will be expected to be the concatenated series key (A:UK:EMP for example).
# The '''Obs table''' must either have a foreign key to the Series Table, OR a column called FR_KEY_ID which can be used to join to the primary key of the Series Table
# The '''Series table''' is expected to have a Column per Dimension, attribute columns are all optional
# The '''Obs table''' is expected to have a Column for Obs Value and Time Period, attribute columns are all optional
# All columns that are prefixed with 'FR_' are optional

=== Single Table Design ===
The flat table structure is a denormalised structure where the Series and Observations are in one table. This does lead to a repetition of dimension values where a series consists of multiple observations. The flat table structure does not support all the features of a SDMX Query including firstNObservations, lastNObservations, includeHistory.

The table structure is as follows:

{| class="wikitable"
|-
! Column Name !! Column Type !! Required !! Purpose
|-
| DIM _ID || varchar || Required || Column per Dimension where the column name is that of the Dimension Id
|-
| ATTR_ID || varchar || Optional|| Column per Series or Observation Attribute where the column name is that of the Attribute Id
|-
| TIME_PERIOD || varchar || Required || [[SDMX_Time_Formats|Reported Time Period]]
|-
| FR_VALID_FROM || datetime || Optional || Used to support startPeriod, endPeriod, lastNObservations, firstNObservations query parameters
|-
| FR_UPDATED_DATE || datetime || Optional || to support the updatedAfter query parameter
|}

== Custom Table and Column Names ==
It is possible to override the Registrys own table nameing and column naming convention. To do this via a web service, use the following:

{| class="wikitable"
|-
|style="background-color:#eaecf0"|Entry Point|| /ws/secure/datasource/tableMapping
|-
|style="background-color:#eaecf0"|Access|| Private (admin)
|-
|style="background-color:#eaecf0"|Http Method|| POST
|-
|style="background-color:#eaecf0"|Accepts|| application/json
|-
|style="background-color:#eaecf0"|Compression || N/A
|-
|style="background-color:#eaecf0"|Content-Type || application/json
|-
|style="background-color:#eaecf0"|Response Format|| application/json
|-
|style="background-color:#eaecf0"|Response Statuses|| 200 - success 
400 - Bad JSON syntax
401 - Unauthorized
500 - Server Error
|}

The JSON request has the following structure:
{
"URN" : "urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=WB:GCI(1.0)",
"DataSourceId" : "MY_STORE",
"KeyTable" : "GCI_KEY",
"ObsTable" : "GCI_OBS",
"DSAttributeTable" : null,
"MappedColumns" : {
"REF_AREA" : "COUNTRY",
"FR_UPDATE_DATE" : "UPDATED_DATE"
}
}

The URN field is the URN of the Dataflow to map the table/colmn names for. The Mapped Columns field is optional, and only required if column names require mapping as well as table names. The same web service can be used to update a mapping, the Dataflow URN is the key for the mapping.

To delete a mapping call the same web service with method DELETE, and post the following JSON:

{
"URN" : "urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=WB:GCI(1.0)",
"DataSourceId" : "MY_STORE",
}

To retrieve a mapping call the same web service with the following query parameters:

{| class="wikitable"
|-
|style="background-color:#eaecf0"|urn|| urn of the Dataflow to get the table mapping description for
|-
|style="background-color:#eaecf0"|dataSourceId|| Id of the data store to get the table mapping description for
|}

Data Stores

2023-02-14T09:51:55Z

Mnelson: /* Externally Managed Database */

[[Category:Functions]]
= Overview =

The Fusion Registry can act as a Virtual Data Store. This means that the Registry is able to link to datasets accross multiple physical data stores, and each data store can be of a diffent type (database, web service, file). The Fusion Registry is able to query multiple data stores for a single data query and then combine the data at the end, before writing it back to the user in a response format of their choice.

== Registry Managed Data Store ==
A Registry Managed Data Store is one which the Registry maintains. This means as data is published to the Fusion Registry, the Registry wil write the data to the store and manage the transaction. Registry managed data stores come in two flavours, a relational database (MySQL or SQL Server) store, or a Fusion Store which has been custom built by Metadata Technology for rapid storage and dissemination of data.

=== Fusion Data Store ===
The Fusion Data Store is an in-memory database, custom built by Metadata Technology for storing time series SDMX datasets. The data is ultimately persisted to the Registry database, so that when the Fusion Registry server is terminated, and re-launched, the Fusion Data Stores will be read back into memory. For the duration of the server up-time, the data store will live in memory.

=== Registry Managed Database ===
A Registry Managed Database includes support for SQL Server and MySQL database platforms, MariaDB*. The Fusion Registry is responsible for creating the database tables when new datasets are loaded. The Fusion Registry will also update table definitions if the [[Data_Structure_Definition|Data Structure Definition]] is changed.

*Note Maria DB re-writes a query for both lastNObservation and firstNObservations to [https://dba.stackexchange.com/questions/117954/is-a-ranking-querys-order-by-in-from-subquery-is-preserved optimise away] a required part of the SQL query. MariaDB can be configured to run in MySQL compatability mode, where this error does not occur.

The database table structure is a two or three tables per Dataflow. The table names are based on the Dataflow, but can be changed if required. The tables include:
* Table for Series Definitions
* Table for Observations
* Table data dataset attributes (only created if Dataset Attributes exist on the Data Structure Definition)

==== Column Names ====
The Series Table will have the following Columns created.

{| class="wikitable"
|-
! Column Name !! Column Type !! Purpose
|-
| ID || Varchar || A concatenation of the Dimension values that make up the series. The width of the ID column is a combination of all the Dimension Widths plus the number of Dimensions.
An example ID is M:FR:EMP. It is important that the column width for each Dimension of the Data Structure Definition is low, i.e if the Dimenison is data type String then place a Max Width restriction on the Dimension's definition so that the database table can be constructed with a primary key that does not exceed the limit set by the database vendor.
|-
| FR_GROUP_ID || varchar|| (optional) Created on if the Data Structure Definition defines a Groups. This coulmn enable the storage of Group level Attributes
|-
| FR_PROVIDER_ACY || varchar || Data Provider's Agency, used if the data query includes Data Provider filters
|-
| FR_PROVIDER_ID || varchar|| Data Provider's Id, used if the data query includes Data Provider filters
|-
| FR_PROVISION || varchar || Provision Agreement short URN e.g ECB:EXR(1.0)
|-
| FR_ERROR || varchar || Internal Use only
|-
| FR_UPDATED_DATE || varchar|| Last updated date for the series
|-
| DIM ID/ATT_ID || varchar|text|| A Column for each Dimension and Series Level Attribute. The Column Name is the ID of the Dimension/Attribute and the Column Type is dependant on the Text Format
|}

The Observation Table will have the following Columns created.
{| class="wikitable"
|-
! Column Name !! Column Type !! Purpose
|-
| ID || int || Auto increment primary key
|-
| FR_KEY_ID || varchar|| Foreign key to the series ID column
|-
| FR_OBS_DATE || datetime || A datetime representaion of the TIME_PERIOD column
|-
| OBS_VALUE|| varchar || The Observation Value
|-
| TIME_PERIOD || varchar || [[SDMX_Time_Formats|Reported Time Period]]
|-
| FR_ERROR || varchar || Internal use only
|-
| FR_EMBARGO || int || Deprecated
|-
| FR_IS_PUBLIC || int || Deprecated
|-
| FR_UPDATED_DATE_OBS || int || Last time the observation was inserted/updated
|-
| FR_VALID_FROM || datetime || Used to support the includeHistory query parameter. A time stamp of when the observation is valid from, either taken from the Header of the dataset, or set to the time the observation was inserted
|-
| FR_VALID_TO|| datetime || Used to support the includeHistory query parameter. A time stamp of when the observation is valid to (if the observation is updated, this is used to capture previous revisions)
|-
| FR_CHKSUM || varchar|| a checksum on the column (internal use)
|-
| ATT_ID || varchar|text|| A Column for each Observation Level Attribute. The Column Name is the ID of the Attribute and the Column Type is dependant on the Text Format
|}

The Dataset Attribute Table, if required, will have the following Columns created.

{| class="wikitable"
|-
! Column Name !! Column Type !! Purpose
|-
| ID || Varchar || The ID of the Dataset Level Attribute.
|-
| VALUE || varchar|| The reported value for the attribute
|-
| FR_ERROR || varchar|| internal use only
|}

==== Column Data Types ====
The column data types that are based on the [[Data_Structure_Definition|Data Structure Definitions]] Dimensions and Attributes are set to Varchar with the width restrictions based on the table below. If the width restriction exceeds 250 characters the column data type is set to TEXT.

Width restrictions are calculated as follows:
{| class="wikitable"
|-
! Component's Representation !! Column Width !! Example Value
|-
| String || Max Length defined on Component or 250 if unspecified || This is a series title
|-
| Codelist || Maximum length of the longest Code Id (e.g UK would be length 2) || UK
|-
| Boolean || 5 || true
|-
| Short || 5 || 32767
|-
| Integer || 10 || 123
|-
| Long || 19 || 123456
|-
| Float || 12 || 12.20
|-
| Double || 22 || 12.20
|-
| Double || 22 || 12.20
|-
| Date || 30 || 2002-01-01T12:15:00.000GMT
|-
| Date Time || 30 || 2002-01-01T12:15:00.000GMT
|-
| Time Period || 30 || 2002-Q1 (any valid [[SDMX_Time_Formats|Time Format]])
|-
| Day || 2 || 31
|-
| Month || 2 || 12
|-
| Year_Month || 7 || 2002-01
|-
| Year || 4 || 2001
|-
| Time || 10 || 12:15:00
|}

Note If the dimenison or attribute is numerical and a maximum and minimum value is set, then this will be used to determine the column width. For example if the Attribute is of type Double, with a min value of 0 and a max value of 10 with a Decimals of 3, then the maximum length is 6, allowing for the values such as 10.123 to be reported.

== Externally Managed Database ==
An Externally Managed Database can be MySQL, SQLServer, or Oracle.

There are two options for the database table design when linking to an externally managed database, these are:
# Mirror the structure used by the Fusion Registry Managed database table structure (Series / Observation).
# A single database table for a Dataflow, where the Series and Observations are in one table

'''Note''' The Table and column names must all be in UPPER CASE.

=== Series/Obs Table Design ===
See the [[#Registry_Managed_Database|registry table design]] section for the full structure. The full table structure is not required, but there are some rules required for this table/view design, these are:

# The '''Series table''' must have a primary key OR a Column called ID. If there is a column called ID it will be expected to be the Primary Key column.
# If the '''Series table''' has an ID column AND it is of type Varchar, the values will be expected to be the concatenated series key (A:UK:EMP for example).
# The '''Obs table''' must either have a foreign key to the Series Table, OR a column called FR_KEY_ID which can be used to join to the primary key of the Series Table
# The '''Series table''' is expected to have a Column per Dimension, attribute columns are all optional
# The '''Obs table''' is expected to have a Column for Obs Value and Time Period, attribute columns are all optional
# All columns that are prefixed with 'FR_' are optional

=== Single Table Design ===
The flat table structure is a denormalised structure where the Series and Observations are in one table. This does lead to a repetition of dimension values where a series consists of multiple observations. The flat table structure does not support all the features of a SDMX Query including firstNObservations, lastNObservations, includeHistory.

The table structure is as follows:

{| class="wikitable"
|-
! Column Name !! Column Type !! Required !! Purpose
|-
| DIM _ID || varchar || Required || Column per Dimension where the column name is that of the Dimension Id
|-
| ATTR_ID || varchar || Optional|| Column per Series or Observation Attribute where the column name is that of the Attribute Id
|-
| TIME_PERIOD || varchar || Required || [[SDMX_Time_Formats|Reported Time Period]]
|-
| FR_VALID_FROM || datetime || Optional || Used to support startPeriod, endPeriod, lastNObservations, firstNObservations query parameters
|-
| FR_UPDATED_DATE || datetime || Optional || to support the updatedAfter query parameter
|}

=== Multi Table ===
The Series and Observation table structure is based on the Fusion Registry's own table structure. Most the columns are option, however it is advised to include all the Dimesion and Attribute Columns, the Primary key of the Series Table (ID), and Foreign Key of the Observation table (FR_KEY_ID). Other columns are optional, however ommiting a column can result in reduced support for Data Queries. For example not including the column FR_OBS_DATE will result in the data query not supporting the startPeriod or endPeriod query parameters.

The default naming convention is:
{| class="wikitable"
|-
! Table Type !! Table Name !! Example
|-
| Dataset Attributes|| [dataflow agency]_[dataflow id]_[dataflow version]_ds || ecb_exr_1_0_ds
|-
| Series || [dataflow agency]_[dataflow id]_[dataflow version]_key || ecb_exr_1_0_key
|-
| Observation || [dataflow agency]_[dataflow id]_[dataflow version]_obs || ecb_exr_1_0_obs
|}

Note 1: The Version number is converted from dot (period) separators to underscore separators.
Note 2: If the name exceeds 26 characters, a hashed name will be used where the hash is built from the long name. In this instance it is reccomended to use the Registry Custom Table Name feature, which can be accessed from the UI Data Store Manager, or the web service (described below).

== Custom Table and Column Names ==
It is possible to override the Registrys own table nameing and column naming convention. To do this via a web service, use the following:

{| class="wikitable"
|-
|style="background-color:#eaecf0"|Entry Point|| /ws/secure/datasource/tableMapping
|-
|style="background-color:#eaecf0"|Access|| Private (admin)
|-
|style="background-color:#eaecf0"|Http Method|| POST
|-
|style="background-color:#eaecf0"|Accepts|| application/json
|-
|style="background-color:#eaecf0"|Compression || N/A
|-
|style="background-color:#eaecf0"|Content-Type || application/json
|-
|style="background-color:#eaecf0"|Response Format|| application/json
|-
|style="background-color:#eaecf0"|Response Statuses|| 200 - success 
400 - Bad JSON syntax
401 - Unauthorized
500 - Server Error
|}

The JSON request has the following structure:
{
"URN" : "urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=WB:GCI(1.0)",
"DataSourceId" : "MY_STORE",
"KeyTable" : "GCI_KEY",
"ObsTable" : "GCI_OBS",
"DSAttributeTable" : null,
"MappedColumns" : {
"REF_AREA" : "COUNTRY",
"FR_UPDATE_DATE" : "UPDATED_DATE"
}
}

The URN field is the URN of the Dataflow to map the table/colmn names for. The Mapped Columns field is optional, and only required if column names require mapping as well as table names. The same web service can be used to update a mapping, the Dataflow URN is the key for the mapping.

To delete a mapping call the same web service with method DELETE, and post the following JSON:

{
"URN" : "urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=WB:GCI(1.0)",
"DataSourceId" : "MY_STORE",
}

To retrieve a mapping call the same web service with the following query parameters:

{| class="wikitable"
|-
|style="background-color:#eaecf0"|urn|| urn of the Dataflow to get the table mapping description for
|-
|style="background-color:#eaecf0"|dataSourceId|| Id of the data store to get the table mapping description for
|}

Fusion Security Manager (FR V11)

2023-02-08T11:52:29Z

Mnelson: /* Datapoint Read Rules */

[[Category:RegistrySecurity]]

= Overview =
The Security model in Fusion Registry 11 was enhanced to reflect the need to support more complex use cases around securing information in the Registry.

Authorization permissions are now maintained in a separate web User Interface called '''Fusion Security Manager'''. This User Interface enables the creations of Groups, and the creation of authorization rules which explicitly permit or deny a security group access to a resource.

The default behaviour of Fusion Registry 11 is to make everything private. In order to make structures and data public, specific security rules must be defined to grant access.

The following topics discuss each aspect of security in Fusion Registry 11.

= Security Groups =
The Fusion Security Manager enables Groups to be defined. Fusion Registry no longer supports assigning users to Groups, instead it will get the Group information from the Authentication service, for example Active Directory supports assigning Groups to users, and the Fusion Registry will map any Group that the AD User belongs to, with it's own Groups. In short, if the AD Group ID matches a Group ID in the Fusion Security Manager, then the Fusion Registry user assumes that Group. The user can be long to one or more Groups.

By default there is a Group called 'Public'. This Group can not be deleted, and all users belong to the Public Group, even if they are not logged into the system. The Public Group provides a mechanism for granting unrestricted access to resources.

When security rules are defined, they either grant an Allow permission, or a Deny permission. It is important to note that a Deny permission implicitly allows all other Groups. When rules are checked against a specific user, the user will be granted access if they belong to at least one Group that has permission

The following table describes this behaviour with specific examples:

{| class="wikitable"
|-
! User Group(s) !! Security Rule(s) !! Outcome
|-
| Public || - none defined - || The user can not see the resource as the default behaviour is to deny access
|-
| Public || Allow Public || The user can see the resource
|-
| Public, G1 || Deny Public || The user can see the resource. Deny Public implicitly allows all other Groups, the user belongs to Group G1 which grants them access
|-
| Public, G1 || Deny G1 || The user can see the resource. Deny G1 implicitly allows all other Groups, the user belongs to Group Public which grants them access
|-
| Public, G1 || Allow G1 || The user can see the resource as they belong to one of the Groups that has been given permission to the resource
|-
| Public || Allow G1 || The user can not see the resource as they do not belong to the Group which is allowed to see the resource
|-
| Public, G1 || Allow G1, Deny Public || The can see the resource as belong to the Group G1 which is allowed to see the resource
|}

= Security Admin =
The Security Admin page of Fusion Security Manager enables a user to be granted specific permissions for administering parts of the Fusion Registry.

The Admin functions are as follows:

{| class="wikitable"
|-
! Admin Function !! Access Rights
|-
| Audit Manager || Ability to query for audit and log information via the Audit web service
|-
| Data Manager || Unrestricted access to view, publish, and delete data
|-
| Error Subscriber || Ability to view, add, and delete subscriptions to error conditions
|-
| Kafka Manager || Ability to view, add. and remove Kafka connections
|-
| Portal Manager || Ability to use '''Fusion Portal''' for automating the synchronization of datasets from external web services
|-
| Reference Metadata Admin || Unrestricted access to view, add, and delete SDMX Reference Metadata
|-
| Server Admin || Unrestricted access to server administration functions
|-
| Structure Admin || Unrestricted access to view, add, and delete SDMX Structural Metadata. Ability to add, delete, and run Environment Synchronisation Ability to view structure transaction details, and to roll back to a transaction
|-
| User Admin || Ability to perform all functions in the Fusion Security Manager
|}

= Structure Security =
== Read Rules ==
TODO

== Write Rules ==
TODO

= Reference Metadata Security =

== Read Rules ==
TODO

== Write Rules ==
TODO

= Data Security =
== Dataset Read Rules ==
[[File:DataSetRead.png|thumb|Allowing Group DEMO_METATECH the ability to read data for any Dataflow maintained by METATECH]]

=== Rule Description ===
Allows access to query for data that matches the rule definition. When a user has access to a Dataset, it will be visible in any User Interface which supports the querying of data, including the web service page, Fusion Data Browser, Tableau, Excel, and so on.

=== Rule Information ===
The Dataset Read Rule can be applied to a Dataflow, Data Provider, or Provision Agreement. All values in the form are optional and as such it acts as a filter. If no values are provided, the rule will be applied to all Datasets in the system. If values are provided, the structures in the Fusion Registry (Dataflow, Data Provider, or Provision Agreement) will be filtered based on the value(s) provided to determine which datasets the rule applies to.

=== Rule Behaviour ===
If the rule is an Allow rule and the user belongs to at least one of the Groups, they will be able to see the Dataset. If the rule is a Deny rule and the user belongs to ALL Deny Groups, the user will not be able to see the Dataset.

If multiple rules match then the most specific rule will apply. Rules are ordered by the following criteria:

# Non empty Agency, Id, and Version takes priority
# A non-empty Agency ID takes priority over a rule with an empty Agency ID
# A non-empty ID takes priority over a rule with an empty ID
# A non-empty Version ID takes priority over a rule with an empty Version

For example:
* Rule: Agency=BIS, ID=BOP, Version=1.0 will take priority over Rule: Agency=BIS, Version=1.0
* Rule: Agency=BIS will take priority over Rule: ID=BOP

== Dataset Write Rules ==
See the description for Dataset Read Rule. The dataset write rule follows the same behaviour, but is applicable when authorising a data publication, as opposed to a data read.

'''Note:''' A user who has Dataset Write access does not automatically get Dataset Read access, the two access types are separate and have to be managed as separate rules.

== Data Point Read Rules ==
=== Rule Description ===
[[File:Datapoint Read Rule.png|thumb|Datapoint Read Rule denying access to OBS_CONF=C]]

When a user has been granted permission to read a dataset using the Dataset Read rules, the behaviour is for the user to have unrestricted access to the Dataset. The Datapoint Read Rules enable filters to be placed on the Dataset which restrict access to subsets of data. The rules are based on the reported Dimension or Attribute (collectively termed Component) values. For example COUNTRY=UK would be a rule that matches on all Series/Observations where the reported value for COUNTRY is UK,

=== Rule Information ===
The Data Point Read Rule requires the Component ID (this refers to either the Attribute ID or Dimension ID) and the Component Value (the reported value to Deny or Allow access to). An example of a restriction is: Component ID=OBS_CONF, Component Value=C. This rule would match any Series or Observation which is reporting OBS_CONF="C".

Other optional values on this rule are the Dataflow Agency, Id, and Version which this rule is applicable for. If the rule is applicable for all Dataflows, these values do not need to be reported. The Agency, Id, and Version act as a filter on Dataflows, for example it is possible to just fill in the Agency ID, which would ensure the rule was only applicable for data reported against Dataflows whose Agency matches the value defined in the rule.

=== Rule Behaviour ===
The behaviour of the rule is as follows:
* If the rule matches on a Series Attribute or Dimension - the rule is applicable to all observations in the series. For example, if the rule is a Deny rule, the user will not be able to see the Series or any of its Observations.
* If the rule matches on an Observation Attribute - the rule is applicable to the single observations. For example, if the rule is a Deny rule, the user will not be able to see the Observation in the Series.
* If the Attribute has no value in the Series or Observation (i.e. it is an optional Attribute) - the rule will not apply and the user '''will''' be able to see the Series or Observation to which the attribute relates

== Data Attribute Read Rules ==
[[File:DataAttributeRead2.png|thumb|Restricting access to an Attribute with ID SYSTEM_ID]]

=== Rule Description ===
The rules for Data Attributes do not filter data, like the Datapoint Read rules, instead they remove the Attributes from the output. This is useful if there are some data attributes which are used for internal purposes and not intended to be seen or used by external parties.

=== Rule Information ===
The Datapoint Read Rule requires the Attribute ID. An example of a restriction is: Attribute ID=SYSTEM_PROCESS_ID. This rule would match any any Attribute with this ID.

Other optional values on this rule are the Dataflow Agency, Id, and Version which this rule is applicable for. If the rule is applicable for all Dataflows, these values do not need to be reported. The Agency, Id, and Version act as a filter on Dataflows, for example it is possible to just fill in the Agency ID, which would ensure the rule was only applicable for data reported against Dataflows whose Agency matches the value defined in the rule.

=== Rule Behaviour ===
When writing the Dataset to the user, if the Series Attribute or Observation Attribute matches the rule, and the user does not have permission to read it, the Attribute will not be included in the output dataset.

Fusion Security Manager (FR V11)

2023-02-08T11:52:06Z

Mnelson: /* Dataset Write Rules */

[[Category:RegistrySecurity]]

= Overview =
The Security model in Fusion Registry 11 was enhanced to reflect the need to support more complex use cases around securing information in the Registry.

Authorization permissions are now maintained in a separate web User Interface called '''Fusion Security Manager'''. This User Interface enables the creations of Groups, and the creation of authorization rules which explicitly permit or deny a security group access to a resource.

The default behaviour of Fusion Registry 11 is to make everything private. In order to make structures and data public, specific security rules must be defined to grant access.

The following topics discuss each aspect of security in Fusion Registry 11.

= Security Groups =
The Fusion Security Manager enables Groups to be defined. Fusion Registry no longer supports assigning users to Groups, instead it will get the Group information from the Authentication service, for example Active Directory supports assigning Groups to users, and the Fusion Registry will map any Group that the AD User belongs to, with it's own Groups. In short, if the AD Group ID matches a Group ID in the Fusion Security Manager, then the Fusion Registry user assumes that Group. The user can be long to one or more Groups.

By default there is a Group called 'Public'. This Group can not be deleted, and all users belong to the Public Group, even if they are not logged into the system. The Public Group provides a mechanism for granting unrestricted access to resources.

When security rules are defined, they either grant an Allow permission, or a Deny permission. It is important to note that a Deny permission implicitly allows all other Groups. When rules are checked against a specific user, the user will be granted access if they belong to at least one Group that has permission

The following table describes this behaviour with specific examples:

{| class="wikitable"
|-
! User Group(s) !! Security Rule(s) !! Outcome
|-
| Public || - none defined - || The user can not see the resource as the default behaviour is to deny access
|-
| Public || Allow Public || The user can see the resource
|-
| Public, G1 || Deny Public || The user can see the resource. Deny Public implicitly allows all other Groups, the user belongs to Group G1 which grants them access
|-
| Public, G1 || Deny G1 || The user can see the resource. Deny G1 implicitly allows all other Groups, the user belongs to Group Public which grants them access
|-
| Public, G1 || Allow G1 || The user can see the resource as they belong to one of the Groups that has been given permission to the resource
|-
| Public || Allow G1 || The user can not see the resource as they do not belong to the Group which is allowed to see the resource
|-
| Public, G1 || Allow G1, Deny Public || The can see the resource as belong to the Group G1 which is allowed to see the resource
|}

= Security Admin =
The Security Admin page of Fusion Security Manager enables a user to be granted specific permissions for administering parts of the Fusion Registry.

The Admin functions are as follows:

{| class="wikitable"
|-
! Admin Function !! Access Rights
|-
| Audit Manager || Ability to query for audit and log information via the Audit web service
|-
| Data Manager || Unrestricted access to view, publish, and delete data
|-
| Error Subscriber || Ability to view, add, and delete subscriptions to error conditions
|-
| Kafka Manager || Ability to view, add. and remove Kafka connections
|-
| Portal Manager || Ability to use '''Fusion Portal''' for automating the synchronization of datasets from external web services
|-
| Reference Metadata Admin || Unrestricted access to view, add, and delete SDMX Reference Metadata
|-
| Server Admin || Unrestricted access to server administration functions
|-
| Structure Admin || Unrestricted access to view, add, and delete SDMX Structural Metadata. Ability to add, delete, and run Environment Synchronisation Ability to view structure transaction details, and to roll back to a transaction
|-
| User Admin || Ability to perform all functions in the Fusion Security Manager
|}

= Structure Security =
== Read Rules ==
TODO

== Write Rules ==
TODO

= Reference Metadata Security =

== Read Rules ==
TODO

== Write Rules ==
TODO

= Data Security =
== Dataset Read Rules ==
[[File:DataSetRead.png|thumb|Allowing Group DEMO_METATECH the ability to read data for any Dataflow maintained by METATECH]]

=== Rule Description ===
Allows access to query for data that matches the rule definition. When a user has access to a Dataset, it will be visible in any User Interface which supports the querying of data, including the web service page, Fusion Data Browser, Tableau, Excel, and so on.

=== Rule Information ===
The Dataset Read Rule can be applied to a Dataflow, Data Provider, or Provision Agreement. All values in the form are optional and as such it acts as a filter. If no values are provided, the rule will be applied to all Datasets in the system. If values are provided, the structures in the Fusion Registry (Dataflow, Data Provider, or Provision Agreement) will be filtered based on the value(s) provided to determine which datasets the rule applies to.

=== Rule Behaviour ===
If the rule is an Allow rule and the user belongs to at least one of the Groups, they will be able to see the Dataset. If the rule is a Deny rule and the user belongs to ALL Deny Groups, the user will not be able to see the Dataset.

If multiple rules match then the most specific rule will apply. Rules are ordered by the following criteria:

# Non empty Agency, Id, and Version takes priority
# A non-empty Agency ID takes priority over a rule with an empty Agency ID
# A non-empty ID takes priority over a rule with an empty ID
# A non-empty Version ID takes priority over a rule with an empty Version

For example:
* Rule: Agency=BIS, ID=BOP, Version=1.0 will take priority over Rule: Agency=BIS, Version=1.0
* Rule: Agency=BIS will take priority over Rule: ID=BOP

== Dataset Write Rules ==
See the description for Dataset Read Rule. The dataset write rule follows the same behaviour, but is applicable when authorising a data publication, as opposed to a data read.

'''Note:''' A user who has Dataset Write access does not automatically get Dataset Read access, the two access types are separate and have to be managed as separate rules.

== Datapoint Read Rules ==
=== Rule Description ===
[[File:Datapoint Read Rule.png|thumb|Datapoint Read Rule denying access to OBS_CONF=C]]

When a user has been granted permission to read a dataset using the Dataset Read rules, the behaviour is for the user to have unrestricted access to the Dataset. The Datapoint Read Rules enable filters to be placed on the Dataset which restrict access to subsets of data. The rules are based on the reported Dimension or Attribute (collectively termed Component) values. For example COUNTRY=UK would be a rule that matches on all Series/Observations where the reported value for COUNTRY is UK,

=== Rule Information ===
The Datapoint Read Rule requires the Component ID (this refers to either the Attribute ID or Dimension ID) and the Component Value (the reported value to Deny or Allow access to). An example of a restriction is: Component ID=OBS_CONF, Component Value=C. This rule would match any Series or Observation which is reporting OBS_CONF="C".

Other optional values on this rule are the Dataflow Agency, Id, and Version which this rule is applicable for. If the rule is applicable for all Dataflows, these values do not need to be reported. The Agency, Id, and Version act as a filter on Dataflows, for example it is possible to just fill in the Agency ID, which would ensure the rule was only applicable for data reported against Dataflows whose Agency matches the value defined in the rule.

=== Rule Behaviour ===
The behaviour of the rule is as follows:
* If the rule matches on a Series Attribute or Dimension - the rule is applicable to all observations in the series. For example, if the rule is a Deny rule, the user will not be able to see the Series or any of its Observations.
* If the rule matches on an Observation Attribute - the rule is applicable to the single observations. For example, if the rule is a Deny rule, the user will not be able to see the Observation in the Series.
* If the Attribute has no value in the Series or Observation (i.e. it is an optional Attribute) - the rule will not apply and the user '''will''' be able to see the Series or Observation to which the attribute relates

== Data Attribute Read Rules ==
[[File:DataAttributeRead2.png|thumb|Restricting access to an Attribute with ID SYSTEM_ID]]

=== Rule Description ===
The rules for Data Attributes do not filter data, like the Datapoint Read rules, instead they remove the Attributes from the output. This is useful if there are some data attributes which are used for internal purposes and not intended to be seen or used by external parties.

=== Rule Information ===
The Datapoint Read Rule requires the Attribute ID. An example of a restriction is: Attribute ID=SYSTEM_PROCESS_ID. This rule would match any any Attribute with this ID.

Other optional values on this rule are the Dataflow Agency, Id, and Version which this rule is applicable for. If the rule is applicable for all Dataflows, these values do not need to be reported. The Agency, Id, and Version act as a filter on Dataflows, for example it is possible to just fill in the Agency ID, which would ensure the rule was only applicable for data reported against Dataflows whose Agency matches the value defined in the rule.

=== Rule Behaviour ===
When writing the Dataset to the user, if the Series Attribute or Observation Attribute matches the rule, and the user does not have permission to read it, the Attribute will not be included in the output dataset.

Fusion Security Manager (FR V11)

2023-02-08T11:49:57Z

Mnelson: /* Dataset Read Rules */

[[Category:RegistrySecurity]]

= Overview =
The Security model in Fusion Registry 11 was enhanced to reflect the need to support more complex use cases around securing information in the Registry.

Authorization permissions are now maintained in a separate web User Interface called '''Fusion Security Manager'''. This User Interface enables the creations of Groups, and the creation of authorization rules which explicitly permit or deny a security group access to a resource.

The default behaviour of Fusion Registry 11 is to make everything private. In order to make structures and data public, specific security rules must be defined to grant access.

The following topics discuss each aspect of security in Fusion Registry 11.

= Security Groups =
The Fusion Security Manager enables Groups to be defined. Fusion Registry no longer supports assigning users to Groups, instead it will get the Group information from the Authentication service, for example Active Directory supports assigning Groups to users, and the Fusion Registry will map any Group that the AD User belongs to, with it's own Groups. In short, if the AD Group ID matches a Group ID in the Fusion Security Manager, then the Fusion Registry user assumes that Group. The user can be long to one or more Groups.

By default there is a Group called 'Public'. This Group can not be deleted, and all users belong to the Public Group, even if they are not logged into the system. The Public Group provides a mechanism for granting unrestricted access to resources.

When security rules are defined, they either grant an Allow permission, or a Deny permission. It is important to note that a Deny permission implicitly allows all other Groups. When rules are checked against a specific user, the user will be granted access if they belong to at least one Group that has permission

The following table describes this behaviour with specific examples:

{| class="wikitable"
|-
! User Group(s) !! Security Rule(s) !! Outcome
|-
| Public || - none defined - || The user can not see the resource as the default behaviour is to deny access
|-
| Public || Allow Public || The user can see the resource
|-
| Public, G1 || Deny Public || The user can see the resource. Deny Public implicitly allows all other Groups, the user belongs to Group G1 which grants them access
|-
| Public, G1 || Deny G1 || The user can see the resource. Deny G1 implicitly allows all other Groups, the user belongs to Group Public which grants them access
|-
| Public, G1 || Allow G1 || The user can see the resource as they belong to one of the Groups that has been given permission to the resource
|-
| Public || Allow G1 || The user can not see the resource as they do not belong to the Group which is allowed to see the resource
|-
| Public, G1 || Allow G1, Deny Public || The can see the resource as belong to the Group G1 which is allowed to see the resource
|}

= Security Admin =
The Security Admin page of Fusion Security Manager enables a user to be granted specific permissions for administering parts of the Fusion Registry.

The Admin functions are as follows:

{| class="wikitable"
|-
! Admin Function !! Access Rights
|-
| Audit Manager || Ability to query for audit and log information via the Audit web service
|-
| Data Manager || Unrestricted access to view, publish, and delete data
|-
| Error Subscriber || Ability to view, add, and delete subscriptions to error conditions
|-
| Kafka Manager || Ability to view, add. and remove Kafka connections
|-
| Portal Manager || Ability to use '''Fusion Portal''' for automating the synchronization of datasets from external web services
|-
| Reference Metadata Admin || Unrestricted access to view, add, and delete SDMX Reference Metadata
|-
| Server Admin || Unrestricted access to server administration functions
|-
| Structure Admin || Unrestricted access to view, add, and delete SDMX Structural Metadata. Ability to add, delete, and run Environment Synchronisation Ability to view structure transaction details, and to roll back to a transaction
|-
| User Admin || Ability to perform all functions in the Fusion Security Manager
|}

= Structure Security =
== Read Rules ==
TODO

== Write Rules ==
TODO

= Reference Metadata Security =

== Read Rules ==
TODO

== Write Rules ==
TODO

= Data Security =
== Dataset Read Rules ==
[[File:DataSetRead.png|thumb|Allowing Group DEMO_METATECH the ability to read data for any Dataflow maintained by METATECH]]

=== Rule Description ===
Allows access to query for data that matches the rule definition. When a user has access to a Dataset, it will be visible in any User Interface which supports the querying of data, including the web service page, Fusion Data Browser, Tableau, Excel, and so on.

=== Rule Information ===
The Dataset Read Rule can be applied to a Dataflow, Data Provider, or Provision Agreement. All values in the form are optional and as such it acts as a filter. If no values are provided, the rule will be applied to all Datasets in the system. If values are provided, the structures in the Fusion Registry (Dataflow, Data Provider, or Provision Agreement) will be filtered based on the value(s) provided to determine which datasets the rule applies to.

=== Rule Behaviour ===
If the rule is an Allow rule and the user belongs to at least one of the Groups, they will be able to see the Dataset. If the rule is a Deny rule and the user belongs to ALL Deny Groups, the user will not be able to see the Dataset.

If multiple rules match then the most specific rule will apply. Rules are ordered by the following criteria:

# Non empty Agency, Id, and Version takes priority
# A non-empty Agency ID takes priority over a rule with an empty Agency ID
# A non-empty ID takes priority over a rule with an empty ID
# A non-empty Version ID takes priority over a rule with an empty Version

For example:
* Rule: Agency=BIS, ID=BOP, Version=1.0 will take priority over Rule: Agency=BIS, Version=1.0
* Rule: Agency=BIS will take priority over Rule: ID=BOP

== Dataset Write Rules ==
TODO

== Datapoint Read Rules ==
=== Rule Description ===
[[File:Datapoint Read Rule.png|thumb|Datapoint Read Rule denying access to OBS_CONF=C]]

When a user has been granted permission to read a dataset using the Dataset Read rules, the behaviour is for the user to have unrestricted access to the Dataset. The Datapoint Read Rules enable filters to be placed on the Dataset which restrict access to subsets of data. The rules are based on the reported Dimension or Attribute (collectively termed Component) values. For example COUNTRY=UK would be a rule that matches on all Series/Observations where the reported value for COUNTRY is UK,

=== Rule Information ===
The Datapoint Read Rule requires the Component ID (this refers to either the Attribute ID or Dimension ID) and the Component Value (the reported value to Deny or Allow access to). An example of a restriction is: Component ID=OBS_CONF, Component Value=C. This rule would match any Series or Observation which is reporting OBS_CONF="C".

Other optional values on this rule are the Dataflow Agency, Id, and Version which this rule is applicable for. If the rule is applicable for all Dataflows, these values do not need to be reported. The Agency, Id, and Version act as a filter on Dataflows, for example it is possible to just fill in the Agency ID, which would ensure the rule was only applicable for data reported against Dataflows whose Agency matches the value defined in the rule.

=== Rule Behaviour ===
The behaviour of the rule is as follows:
* If the rule matches on a Series Attribute or Dimension - the rule is applicable to all observations in the series. For example, if the rule is a Deny rule, the user will not be able to see the Series or any of its Observations.
* If the rule matches on an Observation Attribute - the rule is applicable to the single observations. For example, if the rule is a Deny rule, the user will not be able to see the Observation in the Series.
* If the Attribute has no value in the Series or Observation (i.e. it is an optional Attribute) - the rule will not apply and the user '''will''' be able to see the Series or Observation to which the attribute relates

== Data Attribute Read Rules ==
[[File:DataAttributeRead2.png|thumb|Restricting access to an Attribute with ID SYSTEM_ID]]

=== Rule Description ===
The rules for Data Attributes do not filter data, like the Datapoint Read rules, instead they remove the Attributes from the output. This is useful if there are some data attributes which are used for internal purposes and not intended to be seen or used by external parties.

=== Rule Information ===
The Datapoint Read Rule requires the Attribute ID. An example of a restriction is: Attribute ID=SYSTEM_PROCESS_ID. This rule would match any any Attribute with this ID.

Other optional values on this rule are the Dataflow Agency, Id, and Version which this rule is applicable for. If the rule is applicable for all Dataflows, these values do not need to be reported. The Agency, Id, and Version act as a filter on Dataflows, for example it is possible to just fill in the Agency ID, which would ensure the rule was only applicable for data reported against Dataflows whose Agency matches the value defined in the rule.

=== Rule Behaviour ===
When writing the Dataset to the user, if the Series Attribute or Observation Attribute matches the rule, and the user does not have permission to read it, the Attribute will not be included in the output dataset.

Fusion Security Manager (FR V11)

2023-02-08T11:35:31Z

Mnelson: /* Dataset Read Rules */

[[Category:RegistrySecurity]]

= Overview =
The Security model in Fusion Registry 11 was enhanced to reflect the need to support more complex use cases around securing information in the Registry.

Authorization permissions are now maintained in a separate web User Interface called '''Fusion Security Manager'''. This User Interface enables the creations of Groups, and the creation of authorization rules which explicitly permit or deny a security group access to a resource.

The default behaviour of Fusion Registry 11 is to make everything private. In order to make structures and data public, specific security rules must be defined to grant access.

The following topics discuss each aspect of security in Fusion Registry 11.

= Security Groups =
The Fusion Security Manager enables Groups to be defined. Fusion Registry no longer supports assigning users to Groups, instead it will get the Group information from the Authentication service, for example Active Directory supports assigning Groups to users, and the Fusion Registry will map any Group that the AD User belongs to, with it's own Groups. In short, if the AD Group ID matches a Group ID in the Fusion Security Manager, then the Fusion Registry user assumes that Group. The user can be long to one or more Groups.

By default there is a Group called 'Public'. This Group can not be deleted, and all users belong to the Public Group, even if they are not logged into the system. The Public Group provides a mechanism for granting unrestricted access to resources.

When security rules are defined, they either grant an Allow permission, or a Deny permission. It is important to note that a Deny permission implicitly allows all other Groups. When rules are checked against a specific user, the user will be granted access if they belong to at least one Group that has permission

The following table describes this behaviour with specific examples:

{| class="wikitable"
|-
! User Group(s) !! Security Rule(s) !! Outcome
|-
| Public || - none defined - || The user can not see the resource as the default behaviour is to deny access
|-
| Public || Allow Public || The user can see the resource
|-
| Public, G1 || Deny Public || The user can see the resource. Deny Public implicitly allows all other Groups, the user belongs to Group G1 which grants them access
|-
| Public, G1 || Deny G1 || The user can see the resource. Deny G1 implicitly allows all other Groups, the user belongs to Group Public which grants them access
|-
| Public, G1 || Allow G1 || The user can see the resource as they belong to one of the Groups that has been given permission to the resource
|-
| Public || Allow G1 || The user can not see the resource as they do not belong to the Group which is allowed to see the resource
|-
| Public, G1 || Allow G1, Deny Public || The can see the resource as belong to the Group G1 which is allowed to see the resource
|}

= Security Admin =
The Security Admin page of Fusion Security Manager enables a user to be granted specific permissions for administering parts of the Fusion Registry.

The Admin functions are as follows:

{| class="wikitable"
|-
! Admin Function !! Access Rights
|-
| Audit Manager || Ability to query for audit and log information via the Audit web service
|-
| Data Manager || Unrestricted access to view, publish, and delete data
|-
| Error Subscriber || Ability to view, add, and delete subscriptions to error conditions
|-
| Kafka Manager || Ability to view, add. and remove Kafka connections
|-
| Portal Manager || Ability to use '''Fusion Portal''' for automating the synchronization of datasets from external web services
|-
| Reference Metadata Admin || Unrestricted access to view, add, and delete SDMX Reference Metadata
|-
| Server Admin || Unrestricted access to server administration functions
|-
| Structure Admin || Unrestricted access to view, add, and delete SDMX Structural Metadata. Ability to add, delete, and run Environment Synchronisation Ability to view structure transaction details, and to roll back to a transaction
|-
| User Admin || Ability to perform all functions in the Fusion Security Manager
|}

= Structure Security =
== Read Rules ==
TODO

== Write Rules ==
TODO

= Reference Metadata Security =

== Read Rules ==
TODO

== Write Rules ==
TODO

= Data Security =
== Dataset Read Rules ==
[[File:DataSetRead.png|thumb|Allowing Group DEMO_METATECH the ability to read data for any Dataflow maintained by METATECH]]
TODO

== Dataset Write Rules ==
TODO

== Datapoint Read Rules ==
=== Rule Description ===
[[File:Datapoint Read Rule.png|thumb|Datapoint Read Rule denying access to OBS_CONF=C]]

When a user has been granted permission to read a dataset using the Dataset Read rules, the behaviour is for the user to have unrestricted access to the Dataset. The Datapoint Read Rules enable filters to be placed on the Dataset which restrict access to subsets of data. The rules are based on the reported Dimension or Attribute (collectively termed Component) values. For example COUNTRY=UK would be a rule that matches on all Series/Observations where the reported value for COUNTRY is UK,

=== Rule Information ===
The Datapoint Read Rule requires the Component ID (this refers to either the Attribute ID or Dimension ID) and the Component Value (the reported value to Deny or Allow access to). An example of a restriction is: Component ID=OBS_CONF, Component Value=C. This rule would match any Series or Observation which is reporting OBS_CONF="C".

Other optional values on this rule are the Dataflow Agency, Id, and Version which this rule is applicable for. If the rule is applicable for all Dataflows, these values do not need to be reported. The Agency, Id, and Version act as a filter on Dataflows, for example it is possible to just fill in the Agency ID, which would ensure the rule was only applicable for data reported against Dataflows whose Agency matches the value defined in the rule.

=== Rule Behaviour ===
The behaviour of the rule is as follows:
* If the rule matches on a Series Attribute or Dimension - the rule is applicable to all observations in the series. For example, if the rule is a Deny rule, the user will not be able to see the Series or any of its Observations.
* If the rule matches on an Observation Attribute - the rule is applicable to the single observations. For example, if the rule is a Deny rule, the user will not be able to see the Observation in the Series.
* If the Attribute has no value in the Series or Observation (i.e. it is an optional Attribute) - the rule will not apply and the user '''will''' be able to see the Series or Observation to which the attribute relates

== Data Attribute Read Rules ==
[[File:DataAttributeRead2.png|thumb|Restricting access to an Attribute with ID SYSTEM_ID]]

=== Rule Description ===
The rules for Data Attributes do not filter data, like the Datapoint Read rules, instead they remove the Attributes from the output. This is useful if there are some data attributes which are used for internal purposes and not intended to be seen or used by external parties.

=== Rule Information ===
The Datapoint Read Rule requires the Attribute ID. An example of a restriction is: Attribute ID=SYSTEM_PROCESS_ID. This rule would match any any Attribute with this ID.

Other optional values on this rule are the Dataflow Agency, Id, and Version which this rule is applicable for. If the rule is applicable for all Dataflows, these values do not need to be reported. The Agency, Id, and Version act as a filter on Dataflows, for example it is possible to just fill in the Agency ID, which would ensure the rule was only applicable for data reported against Dataflows whose Agency matches the value defined in the rule.

=== Rule Behaviour ===
When writing the Dataset to the user, if the Series Attribute or Observation Attribute matches the rule, and the user does not have permission to read it, the Attribute will not be included in the output dataset.

File:DataSetRead.png

2023-02-08T11:34:50Z

Mnelson:

DataSetRead

Fusion Security Manager (FR V11)

2023-02-08T11:33:45Z

Mnelson: /* Data Attribute Read Rules */

[[Category:RegistrySecurity]]

= Overview =
The Security model in Fusion Registry 11 was enhanced to reflect the need to support more complex use cases around securing information in the Registry.

Authorization permissions are now maintained in a separate web User Interface called '''Fusion Security Manager'''. This User Interface enables the creations of Groups, and the creation of authorization rules which explicitly permit or deny a security group access to a resource.

The default behaviour of Fusion Registry 11 is to make everything private. In order to make structures and data public, specific security rules must be defined to grant access.

The following topics discuss each aspect of security in Fusion Registry 11.

= Security Groups =
The Fusion Security Manager enables Groups to be defined. Fusion Registry no longer supports assigning users to Groups, instead it will get the Group information from the Authentication service, for example Active Directory supports assigning Groups to users, and the Fusion Registry will map any Group that the AD User belongs to, with it's own Groups. In short, if the AD Group ID matches a Group ID in the Fusion Security Manager, then the Fusion Registry user assumes that Group. The user can be long to one or more Groups.

By default there is a Group called 'Public'. This Group can not be deleted, and all users belong to the Public Group, even if they are not logged into the system. The Public Group provides a mechanism for granting unrestricted access to resources.

When security rules are defined, they either grant an Allow permission, or a Deny permission. It is important to note that a Deny permission implicitly allows all other Groups. When rules are checked against a specific user, the user will be granted access if they belong to at least one Group that has permission

The following table describes this behaviour with specific examples:

{| class="wikitable"
|-
! User Group(s) !! Security Rule(s) !! Outcome
|-
| Public || - none defined - || The user can not see the resource as the default behaviour is to deny access
|-
| Public || Allow Public || The user can see the resource
|-
| Public, G1 || Deny Public || The user can see the resource. Deny Public implicitly allows all other Groups, the user belongs to Group G1 which grants them access
|-
| Public, G1 || Deny G1 || The user can see the resource. Deny G1 implicitly allows all other Groups, the user belongs to Group Public which grants them access
|-
| Public, G1 || Allow G1 || The user can see the resource as they belong to one of the Groups that has been given permission to the resource
|-
| Public || Allow G1 || The user can not see the resource as they do not belong to the Group which is allowed to see the resource
|-
| Public, G1 || Allow G1, Deny Public || The can see the resource as belong to the Group G1 which is allowed to see the resource
|}

= Security Admin =
The Security Admin page of Fusion Security Manager enables a user to be granted specific permissions for administering parts of the Fusion Registry.

The Admin functions are as follows:

{| class="wikitable"
|-
! Admin Function !! Access Rights
|-
| Audit Manager || Ability to query for audit and log information via the Audit web service
|-
| Data Manager || Unrestricted access to view, publish, and delete data
|-
| Error Subscriber || Ability to view, add, and delete subscriptions to error conditions
|-
| Kafka Manager || Ability to view, add. and remove Kafka connections
|-
| Portal Manager || Ability to use '''Fusion Portal''' for automating the synchronization of datasets from external web services
|-
| Reference Metadata Admin || Unrestricted access to view, add, and delete SDMX Reference Metadata
|-
| Server Admin || Unrestricted access to server administration functions
|-
| Structure Admin || Unrestricted access to view, add, and delete SDMX Structural Metadata. Ability to add, delete, and run Environment Synchronisation Ability to view structure transaction details, and to roll back to a transaction
|-
| User Admin || Ability to perform all functions in the Fusion Security Manager
|}

= Structure Security =
== Read Rules ==
TODO

== Write Rules ==
TODO

= Reference Metadata Security =

== Read Rules ==
TODO

== Write Rules ==
TODO

= Data Security =
== Dataset Read Rules ==
TODO

== Dataset Write Rules ==
TODO

== Datapoint Read Rules ==
=== Rule Description ===
[[File:Datapoint Read Rule.png|thumb|Datapoint Read Rule denying access to OBS_CONF=C]]

When a user has been granted permission to read a dataset using the Dataset Read rules, the behaviour is for the user to have unrestricted access to the Dataset. The Datapoint Read Rules enable filters to be placed on the Dataset which restrict access to subsets of data. The rules are based on the reported Dimension or Attribute (collectively termed Component) values. For example COUNTRY=UK would be a rule that matches on all Series/Observations where the reported value for COUNTRY is UK,

=== Rule Information ===
The Datapoint Read Rule requires the Component ID (this refers to either the Attribute ID or Dimension ID) and the Component Value (the reported value to Deny or Allow access to). An example of a restriction is: Component ID=OBS_CONF, Component Value=C. This rule would match any Series or Observation which is reporting OBS_CONF="C".

Other optional values on this rule are the Dataflow Agency, Id, and Version which this rule is applicable for. If the rule is applicable for all Dataflows, these values do not need to be reported. The Agency, Id, and Version act as a filter on Dataflows, for example it is possible to just fill in the Agency ID, which would ensure the rule was only applicable for data reported against Dataflows whose Agency matches the value defined in the rule.

=== Rule Behaviour ===
The behaviour of the rule is as follows:
* If the rule matches on a Series Attribute or Dimension - the rule is applicable to all observations in the series. For example, if the rule is a Deny rule, the user will not be able to see the Series or any of its Observations.
* If the rule matches on an Observation Attribute - the rule is applicable to the single observations. For example, if the rule is a Deny rule, the user will not be able to see the Observation in the Series.
* If the Attribute has no value in the Series or Observation (i.e. it is an optional Attribute) - the rule will not apply and the user '''will''' be able to see the Series or Observation to which the attribute relates

== Data Attribute Read Rules ==
[[File:DataAttributeRead2.png|thumb|Restricting access to an Attribute with ID SYSTEM_ID]]

=== Rule Description ===
The rules for Data Attributes do not filter data, like the Datapoint Read rules, instead they remove the Attributes from the output. This is useful if there are some data attributes which are used for internal purposes and not intended to be seen or used by external parties.

=== Rule Information ===
The Datapoint Read Rule requires the Attribute ID. An example of a restriction is: Attribute ID=SYSTEM_PROCESS_ID. This rule would match any any Attribute with this ID.

Other optional values on this rule are the Dataflow Agency, Id, and Version which this rule is applicable for. If the rule is applicable for all Dataflows, these values do not need to be reported. The Agency, Id, and Version act as a filter on Dataflows, for example it is possible to just fill in the Agency ID, which would ensure the rule was only applicable for data reported against Dataflows whose Agency matches the value defined in the rule.

=== Rule Behaviour ===
When writing the Dataset to the user, if the Series Attribute or Observation Attribute matches the rule, and the user does not have permission to read it, the Attribute will not be included in the output dataset.

File:DataAttributeRead2.png

2023-02-08T11:33:37Z

Mnelson:

Restricting access to an Attribute with ID SYSTEM_ID

Fusion Security Manager (FR V11)

2023-02-08T11:32:32Z

Mnelson: /* Data Attribute Read Rules */

[[Category:RegistrySecurity]]

= Overview =
The Security model in Fusion Registry 11 was enhanced to reflect the need to support more complex use cases around securing information in the Registry.

Authorization permissions are now maintained in a separate web User Interface called '''Fusion Security Manager'''. This User Interface enables the creations of Groups, and the creation of authorization rules which explicitly permit or deny a security group access to a resource.

The default behaviour of Fusion Registry 11 is to make everything private. In order to make structures and data public, specific security rules must be defined to grant access.

The following topics discuss each aspect of security in Fusion Registry 11.

= Security Groups =
The Fusion Security Manager enables Groups to be defined. Fusion Registry no longer supports assigning users to Groups, instead it will get the Group information from the Authentication service, for example Active Directory supports assigning Groups to users, and the Fusion Registry will map any Group that the AD User belongs to, with it's own Groups. In short, if the AD Group ID matches a Group ID in the Fusion Security Manager, then the Fusion Registry user assumes that Group. The user can be long to one or more Groups.

By default there is a Group called 'Public'. This Group can not be deleted, and all users belong to the Public Group, even if they are not logged into the system. The Public Group provides a mechanism for granting unrestricted access to resources.

When security rules are defined, they either grant an Allow permission, or a Deny permission. It is important to note that a Deny permission implicitly allows all other Groups. When rules are checked against a specific user, the user will be granted access if they belong to at least one Group that has permission

The following table describes this behaviour with specific examples:

{| class="wikitable"
|-
! User Group(s) !! Security Rule(s) !! Outcome
|-
| Public || - none defined - || The user can not see the resource as the default behaviour is to deny access
|-
| Public || Allow Public || The user can see the resource
|-
| Public, G1 || Deny Public || The user can see the resource. Deny Public implicitly allows all other Groups, the user belongs to Group G1 which grants them access
|-
| Public, G1 || Deny G1 || The user can see the resource. Deny G1 implicitly allows all other Groups, the user belongs to Group Public which grants them access
|-
| Public, G1 || Allow G1 || The user can see the resource as they belong to one of the Groups that has been given permission to the resource
|-
| Public || Allow G1 || The user can not see the resource as they do not belong to the Group which is allowed to see the resource
|-
| Public, G1 || Allow G1, Deny Public || The can see the resource as belong to the Group G1 which is allowed to see the resource
|}

= Security Admin =
The Security Admin page of Fusion Security Manager enables a user to be granted specific permissions for administering parts of the Fusion Registry.

The Admin functions are as follows:

{| class="wikitable"
|-
! Admin Function !! Access Rights
|-
| Audit Manager || Ability to query for audit and log information via the Audit web service
|-
| Data Manager || Unrestricted access to view, publish, and delete data
|-
| Error Subscriber || Ability to view, add, and delete subscriptions to error conditions
|-
| Kafka Manager || Ability to view, add. and remove Kafka connections
|-
| Portal Manager || Ability to use '''Fusion Portal''' for automating the synchronization of datasets from external web services
|-
| Reference Metadata Admin || Unrestricted access to view, add, and delete SDMX Reference Metadata
|-
| Server Admin || Unrestricted access to server administration functions
|-
| Structure Admin || Unrestricted access to view, add, and delete SDMX Structural Metadata. Ability to add, delete, and run Environment Synchronisation Ability to view structure transaction details, and to roll back to a transaction
|-
| User Admin || Ability to perform all functions in the Fusion Security Manager
|}

= Structure Security =
== Read Rules ==
TODO

== Write Rules ==
TODO

= Reference Metadata Security =

== Read Rules ==
TODO

== Write Rules ==
TODO

= Data Security =
== Dataset Read Rules ==
TODO

== Dataset Write Rules ==
TODO

== Datapoint Read Rules ==
=== Rule Description ===
[[File:Datapoint Read Rule.png|thumb|Datapoint Read Rule denying access to OBS_CONF=C]]

When a user has been granted permission to read a dataset using the Dataset Read rules, the behaviour is for the user to have unrestricted access to the Dataset. The Datapoint Read Rules enable filters to be placed on the Dataset which restrict access to subsets of data. The rules are based on the reported Dimension or Attribute (collectively termed Component) values. For example COUNTRY=UK would be a rule that matches on all Series/Observations where the reported value for COUNTRY is UK,

=== Rule Information ===
The Datapoint Read Rule requires the Component ID (this refers to either the Attribute ID or Dimension ID) and the Component Value (the reported value to Deny or Allow access to). An example of a restriction is: Component ID=OBS_CONF, Component Value=C. This rule would match any Series or Observation which is reporting OBS_CONF="C".

Other optional values on this rule are the Dataflow Agency, Id, and Version which this rule is applicable for. If the rule is applicable for all Dataflows, these values do not need to be reported. The Agency, Id, and Version act as a filter on Dataflows, for example it is possible to just fill in the Agency ID, which would ensure the rule was only applicable for data reported against Dataflows whose Agency matches the value defined in the rule.

=== Rule Behaviour ===
The behaviour of the rule is as follows:
* If the rule matches on a Series Attribute or Dimension - the rule is applicable to all observations in the series. For example, if the rule is a Deny rule, the user will not be able to see the Series or any of its Observations.
* If the rule matches on an Observation Attribute - the rule is applicable to the single observations. For example, if the rule is a Deny rule, the user will not be able to see the Observation in the Series.
* If the Attribute has no value in the Series or Observation (i.e. it is an optional Attribute) - the rule will not apply and the user '''will''' be able to see the Series or Observation to which the attribute relates

== Data Attribute Read Rules ==
[[File:DataAttributeRead.png|thumb|Restricting access to an Attribute with ID SYSTEM_ID]]
=== Rule Description ===
The rules for Data Attributes do not filter data, like the Datapoint Read rules, instead they remove the Attributes from the output. This is useful if there are some data attributes which are used for internal purposes and not intended to be seen or used by external parties.

=== Rule Information ===
The Datapoint Read Rule requires the Attribute ID. An example of a restriction is: Attribute ID=SYSTEM_PROCESS_ID. This rule would match any any Attribute with this ID.

Other optional values on this rule are the Dataflow Agency, Id, and Version which this rule is applicable for. If the rule is applicable for all Dataflows, these values do not need to be reported. The Agency, Id, and Version act as a filter on Dataflows, for example it is possible to just fill in the Agency ID, which would ensure the rule was only applicable for data reported against Dataflows whose Agency matches the value defined in the rule.

=== Rule Behaviour ===
When writing the Dataset to the user, if the Series Attribute or Observation Attribute matches the rule, and the user does not have permission to read it, the Attribute will not be included in the output dataset.

File:DataAttributeRead.png

2023-02-08T11:32:05Z

Mnelson:

DataAttributeRead

Fusion Security Manager (FR V11)

2023-02-08T11:30:38Z

Mnelson: /* Rule Information */

[[Category:RegistrySecurity]]

= Overview =
The Security model in Fusion Registry 11 was enhanced to reflect the need to support more complex use cases around securing information in the Registry.

Authorization permissions are now maintained in a separate web User Interface called '''Fusion Security Manager'''. This User Interface enables the creations of Groups, and the creation of authorization rules which explicitly permit or deny a security group access to a resource.

The default behaviour of Fusion Registry 11 is to make everything private. In order to make structures and data public, specific security rules must be defined to grant access.

The following topics discuss each aspect of security in Fusion Registry 11.

= Security Groups =
The Fusion Security Manager enables Groups to be defined. Fusion Registry no longer supports assigning users to Groups, instead it will get the Group information from the Authentication service, for example Active Directory supports assigning Groups to users, and the Fusion Registry will map any Group that the AD User belongs to, with it's own Groups. In short, if the AD Group ID matches a Group ID in the Fusion Security Manager, then the Fusion Registry user assumes that Group. The user can be long to one or more Groups.

By default there is a Group called 'Public'. This Group can not be deleted, and all users belong to the Public Group, even if they are not logged into the system. The Public Group provides a mechanism for granting unrestricted access to resources.

When security rules are defined, they either grant an Allow permission, or a Deny permission. It is important to note that a Deny permission implicitly allows all other Groups. When rules are checked against a specific user, the user will be granted access if they belong to at least one Group that has permission

The following table describes this behaviour with specific examples:

{| class="wikitable"
|-
! User Group(s) !! Security Rule(s) !! Outcome
|-
| Public || - none defined - || The user can not see the resource as the default behaviour is to deny access
|-
| Public || Allow Public || The user can see the resource
|-
| Public, G1 || Deny Public || The user can see the resource. Deny Public implicitly allows all other Groups, the user belongs to Group G1 which grants them access
|-
| Public, G1 || Deny G1 || The user can see the resource. Deny G1 implicitly allows all other Groups, the user belongs to Group Public which grants them access
|-
| Public, G1 || Allow G1 || The user can see the resource as they belong to one of the Groups that has been given permission to the resource
|-
| Public || Allow G1 || The user can not see the resource as they do not belong to the Group which is allowed to see the resource
|-
| Public, G1 || Allow G1, Deny Public || The can see the resource as belong to the Group G1 which is allowed to see the resource
|}

= Security Admin =
The Security Admin page of Fusion Security Manager enables a user to be granted specific permissions for administering parts of the Fusion Registry.

The Admin functions are as follows:

{| class="wikitable"
|-
! Admin Function !! Access Rights
|-
| Audit Manager || Ability to query for audit and log information via the Audit web service
|-
| Data Manager || Unrestricted access to view, publish, and delete data
|-
| Error Subscriber || Ability to view, add, and delete subscriptions to error conditions
|-
| Kafka Manager || Ability to view, add. and remove Kafka connections
|-
| Portal Manager || Ability to use '''Fusion Portal''' for automating the synchronization of datasets from external web services
|-
| Reference Metadata Admin || Unrestricted access to view, add, and delete SDMX Reference Metadata
|-
| Server Admin || Unrestricted access to server administration functions
|-
| Structure Admin || Unrestricted access to view, add, and delete SDMX Structural Metadata. Ability to add, delete, and run Environment Synchronisation Ability to view structure transaction details, and to roll back to a transaction
|-
| User Admin || Ability to perform all functions in the Fusion Security Manager
|}

= Structure Security =
== Read Rules ==
TODO

== Write Rules ==
TODO

= Reference Metadata Security =

== Read Rules ==
TODO

== Write Rules ==
TODO

= Data Security =
== Dataset Read Rules ==
TODO

== Dataset Write Rules ==
TODO

== Datapoint Read Rules ==
=== Rule Description ===
[[File:Datapoint Read Rule.png|thumb|Datapoint Read Rule denying access to OBS_CONF=C]]

When a user has been granted permission to read a dataset using the Dataset Read rules, the behaviour is for the user to have unrestricted access to the Dataset. The Datapoint Read Rules enable filters to be placed on the Dataset which restrict access to subsets of data. The rules are based on the reported Dimension or Attribute (collectively termed Component) values. For example COUNTRY=UK would be a rule that matches on all Series/Observations where the reported value for COUNTRY is UK,

=== Rule Information ===
The Datapoint Read Rule requires the Component ID (this refers to either the Attribute ID or Dimension ID) and the Component Value (the reported value to Deny or Allow access to). An example of a restriction is: Component ID=OBS_CONF, Component Value=C. This rule would match any Series or Observation which is reporting OBS_CONF="C".

Other optional values on this rule are the Dataflow Agency, Id, and Version which this rule is applicable for. If the rule is applicable for all Dataflows, these values do not need to be reported. The Agency, Id, and Version act as a filter on Dataflows, for example it is possible to just fill in the Agency ID, which would ensure the rule was only applicable for data reported against Dataflows whose Agency matches the value defined in the rule.

=== Rule Behaviour ===
The behaviour of the rule is as follows:
* If the rule matches on a Series Attribute or Dimension - the rule is applicable to all observations in the series. For example, if the rule is a Deny rule, the user will not be able to see the Series or any of its Observations.
* If the rule matches on an Observation Attribute - the rule is applicable to the single observations. For example, if the rule is a Deny rule, the user will not be able to see the Observation in the Series.
* If the Attribute has no value in the Series or Observation (i.e. it is an optional Attribute) - the rule will not apply and the user '''will''' be able to see the Series or Observation to which the attribute relates

== Data Attribute Read Rules ==
=== Rule Description ===
The rules for Data Attributes do not filter data, like the Datapoint Read rules, instead they remove the Attributes from the output. This is useful if there are some data attributes which are used for internal purposes and not intended to be seen or used by external parties.

=== Rule Information ===
The Datapoint Read Rule requires the Attribute ID. An example of a restriction is: Attribute ID=SYSTEM_PROCESS_ID. This rule would match any any Attribute with this ID.

Other optional values on this rule are the Dataflow Agency, Id, and Version which this rule is applicable for. If the rule is applicable for all Dataflows, these values do not need to be reported. The Agency, Id, and Version act as a filter on Dataflows, for example it is possible to just fill in the Agency ID, which would ensure the rule was only applicable for data reported against Dataflows whose Agency matches the value defined in the rule.

=== Rule Behaviour ===
When writing the Dataset to the user, if the Series Attribute or Observation Attribute matches the rule, and the user does not have permission to read it, the Attribute will not be included in the output dataset.

Fusion Security Manager (FR V11)

2023-02-08T11:30:26Z

Mnelson: /* Data Attribute Read Rules */

[[Category:RegistrySecurity]]

= Overview =
The Security model in Fusion Registry 11 was enhanced to reflect the need to support more complex use cases around securing information in the Registry.

Authorization permissions are now maintained in a separate web User Interface called '''Fusion Security Manager'''. This User Interface enables the creations of Groups, and the creation of authorization rules which explicitly permit or deny a security group access to a resource.

The default behaviour of Fusion Registry 11 is to make everything private. In order to make structures and data public, specific security rules must be defined to grant access.

The following topics discuss each aspect of security in Fusion Registry 11.

= Security Groups =
The Fusion Security Manager enables Groups to be defined. Fusion Registry no longer supports assigning users to Groups, instead it will get the Group information from the Authentication service, for example Active Directory supports assigning Groups to users, and the Fusion Registry will map any Group that the AD User belongs to, with it's own Groups. In short, if the AD Group ID matches a Group ID in the Fusion Security Manager, then the Fusion Registry user assumes that Group. The user can be long to one or more Groups.

By default there is a Group called 'Public'. This Group can not be deleted, and all users belong to the Public Group, even if they are not logged into the system. The Public Group provides a mechanism for granting unrestricted access to resources.

When security rules are defined, they either grant an Allow permission, or a Deny permission. It is important to note that a Deny permission implicitly allows all other Groups. When rules are checked against a specific user, the user will be granted access if they belong to at least one Group that has permission

The following table describes this behaviour with specific examples:

{| class="wikitable"
|-
! User Group(s) !! Security Rule(s) !! Outcome
|-
| Public || - none defined - || The user can not see the resource as the default behaviour is to deny access
|-
| Public || Allow Public || The user can see the resource
|-
| Public, G1 || Deny Public || The user can see the resource. Deny Public implicitly allows all other Groups, the user belongs to Group G1 which grants them access
|-
| Public, G1 || Deny G1 || The user can see the resource. Deny G1 implicitly allows all other Groups, the user belongs to Group Public which grants them access
|-
| Public, G1 || Allow G1 || The user can see the resource as they belong to one of the Groups that has been given permission to the resource
|-
| Public || Allow G1 || The user can not see the resource as they do not belong to the Group which is allowed to see the resource
|-
| Public, G1 || Allow G1, Deny Public || The can see the resource as belong to the Group G1 which is allowed to see the resource
|}

= Security Admin =
The Security Admin page of Fusion Security Manager enables a user to be granted specific permissions for administering parts of the Fusion Registry.

The Admin functions are as follows:

{| class="wikitable"
|-
! Admin Function !! Access Rights
|-
| Audit Manager || Ability to query for audit and log information via the Audit web service
|-
| Data Manager || Unrestricted access to view, publish, and delete data
|-
| Error Subscriber || Ability to view, add, and delete subscriptions to error conditions
|-
| Kafka Manager || Ability to view, add. and remove Kafka connections
|-
| Portal Manager || Ability to use '''Fusion Portal''' for automating the synchronization of datasets from external web services
|-
| Reference Metadata Admin || Unrestricted access to view, add, and delete SDMX Reference Metadata
|-
| Server Admin || Unrestricted access to server administration functions
|-
| Structure Admin || Unrestricted access to view, add, and delete SDMX Structural Metadata. Ability to add, delete, and run Environment Synchronisation Ability to view structure transaction details, and to roll back to a transaction
|-
| User Admin || Ability to perform all functions in the Fusion Security Manager
|}

= Structure Security =
== Read Rules ==
TODO

== Write Rules ==
TODO

= Reference Metadata Security =

== Read Rules ==
TODO

== Write Rules ==
TODO

= Data Security =
== Dataset Read Rules ==
TODO

== Dataset Write Rules ==
TODO

== Datapoint Read Rules ==
=== Rule Description ===
[[File:Datapoint Read Rule.png|thumb|Datapoint Read Rule denying access to OBS_CONF=C]]

When a user has been granted permission to read a dataset using the Dataset Read rules, the behaviour is for the user to have unrestricted access to the Dataset. The Datapoint Read Rules enable filters to be placed on the Dataset which restrict access to subsets of data. The rules are based on the reported Dimension or Attribute (collectively termed Component) values. For example COUNTRY=UK would be a rule that matches on all Series/Observations where the reported value for COUNTRY is UK,

=== Rule Information ===
The Datapoint Read Rule requires the Component ID (this refers to either the Attribute ID or Dimension ID) and the Component Value (the reported value to Deny or Allow access to). An example of a restriction is: Component ID=OBS_CONF, Component Value=C. This rule would match any Series or Observation which is reporting OBS_CONF="C". Other optional values on this rule are the Dataflow Agency, Id, and Version which this rule is applicable for. If the rule is applicable for all Dataflows, these values do not need to be reported. The Agency, Id, and Version act as a filter on Dataflows, for example it is possible to just fill in the Agency ID, which would ensure the rule was only applicable for data reported against Dataflows whose Agency matches the value defined in the rule.

=== Rule Behaviour ===
The behaviour of the rule is as follows:
* If the rule matches on a Series Attribute or Dimension - the rule is applicable to all observations in the series. For example, if the rule is a Deny rule, the user will not be able to see the Series or any of its Observations.
* If the rule matches on an Observation Attribute - the rule is applicable to the single observations. For example, if the rule is a Deny rule, the user will not be able to see the Observation in the Series.
* If the Attribute has no value in the Series or Observation (i.e. it is an optional Attribute) - the rule will not apply and the user '''will''' be able to see the Series or Observation to which the attribute relates

== Data Attribute Read Rules ==
=== Rule Description ===
The rules for Data Attributes do not filter data, like the Datapoint Read rules, instead they remove the Attributes from the output. This is useful if there are some data attributes which are used for internal purposes and not intended to be seen or used by external parties.

=== Rule Information ===
The Datapoint Read Rule requires the Attribute ID. An example of a restriction is: Attribute ID=SYSTEM_PROCESS_ID. This rule would match any any Attribute with this ID.

Other optional values on this rule are the Dataflow Agency, Id, and Version which this rule is applicable for. If the rule is applicable for all Dataflows, these values do not need to be reported. The Agency, Id, and Version act as a filter on Dataflows, for example it is possible to just fill in the Agency ID, which would ensure the rule was only applicable for data reported against Dataflows whose Agency matches the value defined in the rule.

=== Rule Behaviour ===
When writing the Dataset to the user, if the Series Attribute or Observation Attribute matches the rule, and the user does not have permission to read it, the Attribute will not be included in the output dataset.

Fusion Security Manager (FR V11)

2023-02-08T11:26:11Z

Mnelson: /* Datapoint Read Rules */

Fusion Security Manager (FR V11)

2023-02-08T11:25:18Z

Mnelson: /* Data Attribute Read Rules */

Fusion Security Manager (FR V11)

2023-02-08T11:17:49Z

Mnelson: /* Datapoint Read Rules */

Fusion Security Manager (FR V11)

2023-02-08T11:08:02Z

Mnelson: /* Datapoint Read Rules */

File:Datapoint Read Rule.png

2023-02-08T10:43:05Z

Mnelson:

Deny public access to OBS_CONF=C

Fusion Security Manager (FR V11)

2023-02-08T10:41:13Z

Mnelson: /* Security Admin */

Fusion Security Manager (FR V11)

2023-02-08T10:38:10Z

Mnelson: /* Security Admin */

Fusion Security Manager (FR V11)

2023-02-08T10:16:57Z

Mnelson: Created page with "Category:RegistrySecurity = Overview = The Security model in Fusion Registry 11 was enhanced to reflect the need to support more complex use cases around securing informa..."

Fusion Edge Server Properties

2023-02-07T12:20:08Z

Mnelson: /* Caching */

[[Category:EdgeServerConfigure]]

= Overview =
The Fusion Edge Server provides configuration options on various aspects of the Edge Server. To set these configurations, create a file under the [[Fusion_Edge_Server_Directory|Edge Server Directory]] with the name: '''edgeserver.properties'''.

= Public URL =
The Edge Server application and web service URL should be defined in the edge server properties file. These URLs are used in responses to structure queries which include links to associated reference metadata, and stub structures which link back to the full artefact.

The URLs are split into two parts, the application URL, which should be a reference to the public application server, and the rest url which is the POSTfix URL to the web service, for example:

app.url=https://myserver.myorg.org/Edge
rest.url=/sdmx

The above setting would make the SDMX web service entry point: https://myserver.myorg.org/Edge/sdmx

= Sender Id =
SDMX Messages contain a Header section which includes the identity of the sender. The identity is mandatory in a SDMX message, and the default ID is set to "UNKNOWN". To change the Id, set the sender Id property

sender.id=YOUR_ID

= Secret Key =
The Fusion Edge server must be given a secret key, and it must match that used by the [[Fusion_Edge_Compiler|Fusion Edge Compiler]] in the compilation of the [[Edge_Server_Environment|Environment]]. If the secret key does not match the secret key of the [[Edge_Server_Environment|Environment]], the Fusion Edge Server will fail to load the [[Edge_Server_Environment|Environment]].

ledger.secret=MySecret

= Static Mode - File Name =
In [[Edge_Server_-_Publish_Content#.E2.80.8BStatic_Mode|static mode]] the Fusion Edge Server will read the properties file to know where the environment is. If nothing is set it will look for a file called node.zip.

environment.file=LiveEnvironment

= Embargo Data =
The Fusion Edge server will not release an [[Edge_Server_Environment|Environment]] with an [[Fusion_Edge_Compiler#Embargo|Embargo]] until the time on the server is later then or equal to the timestamp of the Embargo.

The Fusion Edge Server can be configured to pre-prepare and [[Edge_Server_Environment|Environment]] for release prior to the embargo time. For example if an Environment is scheduled for release at 12.00pm, and the Environment is moved to a secure Amazon S3 file system at 11.50, the Fusion Edge Server can be configured to pull these files into its local file system 5 minutes before go live, to eliminate any risk of network lag pulling the files. The Fusion Ede Server can also be set to pre-load the Environment into memory 30 seconds before go live, the Environment is now ready for dissemination but disconnected from any process which can get to the data, it is still fully secure. The Fusion Edge Server is able to then make the Environment go live by simply swapping the old Environment with the new, making it go live with millisecond precision.

To preload the Environment from a remote location to the local file system prior to the Embargo time, set the preload filesystem property to define the number of milliseconds before the embargo time that it should initiate the preload process.

ledger.preload.filesystem=40000

To preload the Environment into memory, set the preload memory property to define the number of milliseconds before the embargo time that it should start reading the Environment into memory.

ledger.preload.memory=20000

In this example the Fusion Edge Server will pull an Environment from the remote location to the local file system 40 seconds prior to the go live time, and then it will pre load the Environment from the file system into memory 20 seconds before the go live time. The Environment will be made live on the go live (embargo) time.

= Disable Audit =
The Fusion Edge maintains an audit log of events in its local [[Fusion_Edge_Server_Directory|Edge Server Directory]]. Auditing can be disabled with the following property

audit.disabled=true

= Limiting Series and Observations =
To place a limit on the number of series and the number of observations that can be queried via the web service the following two properties are supported:

limit.obs=12345
limit.series=1234

The numeric value indicates the maximum number of series or observations that can be returned from a single data query.

If the limit is exceeded an HTTP status code of 413 (Payload Too Large) will be returned to the client.

= Limiting Supported Data Formats =
To enforce a restriction on which data formats are supported from the REST API of the Fusion Edge Server, use the '''format.data''' property as follows:

format.data=[format1],[format2]

This property defines which formats are supported, if this property is not provided all formats will be supported. If this property is provided, only the formats in this list will be supported.

VND Headers are listed in the [[Data Formats]] section.

'''Example:''' 
format.data=application/vnd.sdmx.generictimeseriesdata+xml;version=2.1,application/vnd.sdmx.data+csv

= Limiting Supported Structure Formats =
To enforce a restriction on which structure formats are supported from the REST API of the Fusion Edge Server, use the '''format.structure''' property as follows:

format.structure=[format1],[format2]

This property defines which formats are supported, if this property is not provided all formats will be supported. If this property is provided, only the formats in this list will be supported.

Each [format] is a VND header the supported formats are:
{| class="wikitable"
|-
! Structure Format !! VND Header
|-
| SDMX EDI || application/vnd.sdmx.structure;version=edi
|-
| application/vnd.sdmx.structure+xml;version=1.0 || SDMX v1.0
|-
| application/vnd.sdmx.structure+xml;version=2.0 || SDMX v2.0
|-
| application/vnd.sdmx.structure+xml;version=2.1 || SDMX v2.1
|-
| application/vnd.openxmlformats-officedocument.spreadsheetml.sheet|| Excel
|-
| application/vnd.fusion.json || Fusion JSON
|-
| application/vnd.sdmx.structure+json;version=1.0.0 || SDMX JSON v1.0.0
|-
| application/vnd.sdmx.structure+json;version=2.0.0 || SDMX JSON v2.0.0
|}

'''Example:''' 
format.structure=application/vnd.sdmx.structure;version=edi,application/vnd.sdmx.json

= Limiting Query Functionality =
The [[Data_Query_Web_Service#Extended_HTTP_Query_Parameters|extended data query API]] can have some features disabled by using the following properties:
rest.query.attribute = false
rest.query.aggregate = false
rest.query.interpolate = false
rest.query.calculatemeasure = false
rest.query.normalisefreq = false
rest.query.round = false

= Caching =
The Fusion Edge Server caches the response to data queries in the tmp folder of the [[Fusion_Edge_Server_Directory|Edge Directory]]. The response is stored in gzip format, which ensures clients who request data in gzip format receive a rapid response when the data has been cached due to previous requests for the same data.

The following are important to note:
* The cache is NOT read on application startup. The cache is generated during the operational running time of the Fusion Edge Server. The cache is a combination of in memory pointers to files in the file system. Once the application server is terminated, it is not re-read on startup. This means any cached files are redundant after the Fusion Edge Server is terminated.
* The cache is NOT cleared on application startup. This is for a number of reasons, but principally deleting files from a running Edge Server is a slower process then could be achieved via an external mechanism. The delete process consumes CPU and therefore should be performed via a more efficient process before server startup.
* The cache can be manually cleared by deleting all the files in the tmp directory.
* The Fusion Edge Server will clear the cache if the Environment is updated in [[Edge_Server_-_Publish_Content#Dynamic_Mode|Dynamic mode]]. The in memory cache is first cleared to ensure clearance with immediate effect. A separate thread is initiated to clear the cached files, this can take time to complete if there are lots of cached files. Files are cleared in batches of 5000 with a 5 second delay per batch, this is to ensure the CPU required to run the deletion process does not block external traffic to the Edge Server.

The cache can be disabled by supplying the following property (Fusion Edge Server v2.3.5 an higher):

rest.query.cache=false

= CSV Data Format Defaults =
To modify the default CSV output is returned, the following properties are supported:

csv.delimiter=comma|tab|space|semicolon
csv.labels=ID|NAME|ID_AND_NAME
csv.showSeries=true|false
csv.includeBom=true|false
csv.keySeparator=:

The delimiter is the value of the character that will be output between the elements (defaults to comma).

The values for 'Labels' must be one of the following: ID; NAME; ID_AND_NAME (defaults to ID_AND_NAME).
Show Series defines whether to output the series key in the output (defaults to true).

= Cross-Origin Resource Sharing (CORS) =
By default the [https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS CORS] policy of the Fusion Edge server is to reject cross domain requests. To provide CORS policy to certain or all domains the cors.policy property should be set.

cors.policy=[policy 1]@[policy 2]@[policy 3]

Each policy is separated by an '@' symbol, within each policy are the following properties, each separated by a '$' symbol:

# Access-Control-Allow-Origin
# Access-Control-Allow-Headers (comma separated)
# Access-Control-Expose-Headers (comma separated)
# Access-Control-Allow-Credentials (true/false)

'''Example'''
cors.policy=www.metadatatechnology.com$GET,HEAD$*$Content-Disposition$true

'''Example All Domains'''
cors.policy=*$GET,HEAD,POST$*$Content-Disposition$false

= Password Protect Excel Workbooks =
Generated [[Reporting_Template|Reporting Template]] data formats can be set to be password protected using the following property

reporttemplate.password=[password]

Fusion Edge Cache

2023-02-07T12:02:00Z

Mnelson:

[[Category:Fusion Edge Functions]]

@see [[Fusion_Edge_Server_Properties#Caching|Caching Properties]]

Fusion Edge Cache

2023-02-07T12:01:40Z

Mnelson:

[[Category:Fusion Edge Functions]]

@see Fusion_Edge_Server_Properties#Caching

Fusion Edge Cache

2023-02-07T11:59:58Z

Mnelson: Created page with "Category:Fusion Edge Functions"

[[Category:Fusion Edge Functions]]

Main Page

2023-02-07T11:59:17Z

Mnelson: /* Fusion Edge Server */

{{DISPLAYTITLE:{{FULLPAGENAME}}}}
__NOTOC__
Fusion Registry Knowledge Base

SDMX Statistical Data Warehouse

Fusion Registry is for central banks, international organisations, national statistics offices, development banks and regional statistics authorities who need an enterprise SDMX statistical data warehouse for storage of aggregated time series, data cataloguing and public dissemination.

::[[File:FR11 ScreenshotComputer.png|450px]]

__NOTOC__
 
==Change Logs==

{| class="wikitable sortable"
|-
| [[Change Log FR - V11|Fusion Registry Core 11]] || [[Change Log Fusion Data Browser|Fusion Data Browser]] || [[Change Log Fusion Security Manager|Fusion Security Manager]]
|-
| [[Change Log FR - V10|Fusion Registry Core 10]] || [[FusionXL Change Log| Fusion XL]] || [[Change Log Fusion Fame Mapper|Fusion Fame Mapper]]
|-
| [[Change Log Fusion Edge Server - V2| Fusion Edge Server 2]] || [[FXLData Change Log| FusionXLData]] || [[Change Log Fusion Transformer|Fusion Transformer]]
|-
|[[Change Log Fusion Edge Server -V3| Fusion Edge Server 3]] || [[Change Log Fusion Portal|Fusion Portal]] || [[Change Log Registry Command Line|Fusion Registry Command Line]]

|}

__TOC__

==Fusion Registry Components and Modules==
It consists of a number of modules centered on the Fusion Registry Core data engine, including a secure high-performance data dissemination server and a configurable data explorer and analytics tool for data consumers and analysts.

Fusion Registry Core has an integrated SDMX structural metadata registry that is 100% compatible with Fusion Metadata Registry.

[[File:FR11 Components.png|600px]]

====Fusion Registry Core====
SDMX statistical data engine and structural metadata registry. 
The main statistical engine for data collection, integration, processing, storage, query execution and time-series calculations. It provides an SDMX-compliant data and metadata REST API. 
The integrated SDMX structural metadata registry acts as the central repository, authoring and maintenance tool for all of the structures including Codelists, Concepts and Data Structure Definitions. 
[[Fusion Registry Core|More]]

====Fusion Edge Server====
High-performance data dissemination server. 
A light-weight SDMX data and metadata REST API server designed to be deployed in clusters in the DMZ or at the network edge for scalable, fault-tolerant public data dissemination use cases. 
[[EdgeServer|More]]

====Fusion Data Browser====
Web tool for data consumers to explore the Fusion Registry data catalogue, create charts and tables, and download data. 
Fusion Data Browser is a web application suitable for both internal and external data consumers to search and explore the catalogue of datasets, select series and cubes of interest, build visualisations using tables and charts, and retrieve selected data in a variety of formats including SDMX, Excel and CSV. 
It can connect to either a Fusion Registry Core engine, or a Fusion Edge Server cluster. 
[[Fusion Data Browser|More]]

====FusionXL====
Microsoft Excel addin for Fusion Registry. 
FusionXL is an Excel addin for browsing and modifying structural metadata, exploring the data catalogue, and downloading data directly into a spreadsheet.

====Fusion FAME Mapper====
Web tool for creating and maintaining Fusion Registry FAME to SDMX series mappings. 
Web tool for Fusion Registry Core for mapping series with a single dimension to richer multi-dimensional structures using SDMX structure mapping rules. 
The main use case is mapping FAME series which typically have a single 'series code' dimension to SDMX multi-dimensional structures including dimensions and attributes like: FREQ, INDICATOR, REF_AREA and UNIT_MULT. However, it will work with source data with a single dimension like 'Indicator'.

[[Fusion_Fame_Mapper|more]]

====Fusion Registry Command Line====
Windows and Linux command line interface. 
A command line interface to Fusion Registry's REST API providing utility scripts for loading structures, data and metadata, validating data, exporting content and deleting content.

Click here [https://wiki.sdmxcloud.org/Fusion_Registry_Command_Line to read the documentation].

==Fusion Registry Core Topics==
{| class="wikitable"
|-
| [[:Category:SDMX Structures|'''Structural Metadata''']]|| Detailed explanations of the main SDMX structural metadata artefacts
|-
| [[:Category:How To|'''How To''']] || Guidance on specific tasks using Fusion Registry
|-
| [[:Category:Behaviour|'''Behaviours''']] || Answers to 'What happens when...?' questions
|-
| [[:Category:Components and Modules|'''Components and Modules''']] || Components and modules reference
|-
| [[:Category:Functions|'''Functions Reference''']] || Features and functions reference
|-
| [[:Category:SdmxDataFormat|'''Data Formats''']] || A description of supported Data Formats
|-
| [[:Category:DynamicData|'''Dynamic Data''']] || Calculations, Transformations and Dynamic Attributes on queried data
|-
| [[:Category:RegistrySecurity|'''Security''']] || Authentication and Authorisation Features
|-
| [[:Category:WebService|'''Web Services''']] || Registry Web Services
|-
| [[Fusion_Registry_Command_Line|'''Command Line''']] || Structure, data, and metadata publishing from the command line
|-
| [[:Category:Glossary|'''Glossary of Terms''']] || Glossary of SDMX and Fusion Registry terminology
|}

==FusionXL==
FusionXL is an addin for Microsoft Excel which connects to a Fusion Registry server for both data and structure retrieval, and maintenance.
{| class="wikitable"
|-
| [[FusionXL_Install|'''Installation''']] || How to Install Fusion XL and connect to a Fusion Registry server
|-
| [[FXLData|'''Inline Data Retrieval''']] || Data Retrieval using the FXLData function
|-
| [[FusionXL_DataBrowser|'''Data Browser''']] || Graphical Data Browsing via the Excel Ribbon
|-
| [[FusionXL_DataBAuthor|'''Data Author''']] || Create datasets and load datasets for the purpose of data authoring, validation, conversion, and publishing
|-
| [[FusionXL_StructureExplorer|'''Structure Explorer''']] || Create, Retrieve and Modify Structures using the Excel Ribbon
|-
| [[FusionXL_DataExplorer|'''Data Explorer''']] || Browse datasets that have been made available to the linked Fusion Registry service.
|}

==FXLData==
Excel add in providing a collection of worksheet functions which retrieve data directly from a Fusion Registry data warehouse.

[[FXLData|FXLData function reference]] 
[[FXLData Wizard|FXLData Wizard - Excel ribbon bar tools to help with the task of creating and modifying FXLData function calls]] 
[[FXLData Refreshable Worksheet|FXLData Refreshable Worksheet - Fusion Registry refreshable Excel worksheets export option]]

==Fusion FAME Mapper==
Fusion Data Mapper provides a web user Interface to administer Structure Maps in the Fusion Registry, where the maps support a specific use case – mapping from Single Dimension Data Structures to multidimensional Data Structures. This use case is borne out of statistical applications which model time series datasets as a single unique identifier for the Series alongside a Time Dimension. By modelling these dataset as a DSD with one Dimension (and a Time Dimension), it is very easy to overlay SDMX metadata to describe the Time Series as a multidimensional Cube with additional series Attributes to carry richer metadata.

[[Fusion_Fame_Mapper|Fusion Fame Mapper]]

==Fusion Edge Server==
Fusion Edge Server is a solution for supporting high performance data and metadata dissemination, with high throughput. The Fusion Edge Server serves all data, related metadata, including reference metadata, from its highly optimised in memory store and as such has no reliance on a database. The Fusion Edge Server has been designed to support the use case of high availability of data through load balancing.

Fusion Edge Server provides access to its stored data and metadata through SDMX compliant web services, and includes support for non-SDMX data formats such as CSV and Excel. The Fusion Edge Server extends the SDMX web services by providing additional information such as data metrics, and data search.

Fusion Edge Server is compatible with Fusion Data Browser and Fusion XL, providing an off-the-shelf solution for both web based data exploration, and from within Microsoft Excel. Connectivity to statistical software such as R and Matlab can be achieved using third party SDMX Connector libraries such as the [https://github.com/amattioc/SDMX SDMX Connectors project].

{| class="wikitable"
|-
| [[EdgeServer|'''Product Overview''']] || Product and Features Overview
|-
| [[EdgeServerInstall|'''Installation''']] || Install and Configure Fusion Edge Server
|-
| [[Edge_Server_-_Publish_Content|'''Publishing Content''']] || How to publish data and metadata to the Fusion Edge Server
|-
| [[Fusion_Edge_Compiler|'''Compiler Reference''']] || Fusion Edge Server Compiler
|-
| [[:Category:FusionEdgeFunctions|'''Functions Reference''']] || Features and functions reference
|}

==Fusion Data Browser ==
Fusion Data Browser is a HTML and JavaScript client side web application which provides a user interface for querying, graphing, and exporting data from a Fusion Registry or Fusion Edge Server.

[https://demo.metadatatechnology.com/FusionDataBrowser/ Live Demo] running against data from the [https://demo.metadatatechnology.com/FusionRegistry Registry Demo Server].

{| class="wikitable"
|-
| [[Data_Browser_Install|'''Installation''']] || How to Install/Deploy the Data Browser
|-
| [[Data_Browser_Configuration|'''Configuration Reference''']] || Configuration of the Browser, Charts, and Series
|-
| [[:Category:DataBrowserFunctions|'''Functions Reference''']] || Features and functions reference
|}

==Fusion Transformer==
Fusion Transformer is a command-line application providing transformations between SDMX and GESMES data files and structure files.

Click here to access the [[Fusion_Transformer|Fusion Transformer article]].

Publication Table

2023-01-31T08:47:36Z

Mnelson: /* Calculated Observation Cells */

[[Category:Functions]]

@See [[Publication_Table_Web_Services|Publication Table Web Services]]

= Publication Tables =
'''Introduced to Fusion Registry Enterprise v11.0.0.17'''

== Overview ==
[[File:Publication Table Definition.png|thumb|Using SDMX concepts of Agency, Id, Name, Description]]

[[File:Publication Table.png|thumb|Table Layout and Content Design]]

[[File:MaterialisedTable Preview.png|thumb|Showing the Table Preview feature which materialises the table with the given variables]]

A Publication Table is a curated table of data, where the values in the table rows, columns and table body cells can be individually specified. This enables a single table to pull data from multiple sources and piece the data together in a way which is not possible when dealing with standard cubes of data, whose table model is typically based on a Pivot model, with columns and rows built around the Dimension values to which the column or row is assigned. A Publication Table breaks out of this restrictive model by allowing cells of data to reference observations which do not need to share any relationship, in terms of dimensionality or even data source, with adjacent cells,.

Data in a Publication Table may be sourced from a single [[Dataflow_-_Structural_Metadata_Management|Dataflow]] or it may be sourced from multiple Dataflows enabling cross comparison of data.

Publication Tables can be static (every cell is a defined Observation) or dynamic, where a cell includes [[#Variable Dimension Members|variables Dimensions]] in the [[#Observation Key|Observation Key]], which can be provided at [[#Metadata vs Materialised|table build time]].

[[#Time Variables|Time arithmetic and frequency conversion]] is supported, enabling a single table to pull data from different time periods and frequencies which are all derived from the base period on which the table is built.

Publication Tables support [[#Calculated Observation Cells|calculations]], where each calculation can take input from one or more Observations over one or more [[Dataflow_-_Structural_Metadata_Management|Dataflows]].

Variable values also play a big role in Publication Tables, and includes standard variable placeholders – as well as advanced [[#Conditional Variables|conditional variables]], whose list of variable values change based on the value of another Dimension for which the table is built.

== Metadata vs Materialised ==

A Publication Table is stored in Fusion Registry the same way any other piece of structural metadata is stored. In this way it has an owner (Agency), and identity (ID) and a version. The change history is tracked using the same services, and the table definition can be [[Publication_Table_Web_Services|queried]] using the same SDMX web services for Structural Metadata.

The Publication Table metadata contains information about which Dataflows are used by the Table, which additional variables are being imported, and a definition of the table structure, and which Observations to import by referencing them by their unique [[#Observation Key|Observation Key]].

A Publication Table definition can be '''materialised''' into a table of Data by from the [[Publication_Table_Web_Services|Publication Table Web Service]], which combines the information of the Table Identity, and Variable Values to rewrite the [[#Observation Key|Observation Key]] into actual observation values. If building a web User Interface, then response from this service can be a packet of JSON, which describes the layout and content of the final Table in a way which is quick and simple to render for the User to see. The process of materialising the table is known as ‘'''build time'''’.

Additional web services are provided to enable clients to ask questions about the table such as ‘what are the earliest and latest time periods for which there is data for this table’ or ‘what possible variable values are there for this table’. This enables user interfaces to provide a user with the option to change the variables ‘on the fly’ ensuing they are able to build a table which is known to have data.

= Publication Table Cell Content =
== Overview ==
[[File:PublicationTable Layout.png|thumb|Publication Table Layout]]
A Publication Table is made up of a '''Table Head Rows''' and '''Table Body Rows'''.

The Table Head is made up of 1 or more table rows. Each Table Head cell may span 1 or more columns and 1 or more rows. The Table Head content is made up of free text and / or variable placeholders whose values are resolved when the table is built. Table Head Cells can be left empty.

The Table Body is made up of 1 or more Table Rows. The Table Rows provide the Publication Table with its data, by referencing Observations from the Dataflow.

At table build time all variables are resolved, and the corresponding data are retrieved from the data stores which hold the data.

== Table Rows ==
=== Overview ===
A Table Row consist of a Table Row heading, this can be a mix of free text and [[#Table Variables|variable text]]. The Table Row Heading is always the first Table Row Cell, and can not span multiple columns or rows. A Table Row can not have more then one heading. The remaining Table Row Cells are the Observation Cells, which may contain free text and [[#Table Variables|variable text]] but will typically contain a reference to an Observation by an [[#Observation Key|Obsevation Keys]] or a [[#Calculated Observation Cells|Calculation]] which can combine multiple [[#Observation Key|Obsevation Keys]].

A Table Row which results in no Observation data will not be included in materialised table at build time, unless the Table Row is a grouping row (see [[#Row Levels|Row Levels]] for details on grouping rows).

=== Row Levels ===
[[File:PublicationTabel Levels.png|thumb|Showing Levels in the Table Rows]]
The Table Row Heading can be used to define a row level. The level indicates to the client responsible for rendering the table the row's position in a hierarchy. Row levels are depicted using one or more leading chevrons ‘>‘ before the text. The number of hyphens indicates the level, with one leading hypen indicating level 1, two indicating level 2, and so on. It is possible to skip levels if required, i.e. Row 1 is level 1 with Row 2 assigned to level 3, skipping out level 2 entirely.

An example of row headings using levels is as follows:

Countries
> Europe
>> France
>> Germany
> Asia
>> China

Levels can be an important way to '''group data'''. A Table Row which acts as a parent row in a hierarchy, can be used as grouping mechanism for the rows that follow. This enables Parent Rows to be included in the output table, even if the Table Row contains no Observation data. The default behaviour for a Table Row with no data is to exclude it from the built table.

=== Looping Rows ===

Instead of explicitly defining a row for each Dimension member, it is possible to define a row as a loop (to iterate over all, or a subset of Dimension members). For example, if a publication table contains a Dimension for countries with an ID of REF_AREA then a loop can be set up to loop through all REF_AREA values, or a subset of the values - for which there is data. If the values are backed by a Codelist which has a hierarchy, the hierarchy can be preserved in the final table. It is also possible to nest a loop within another loop, for example for each REF_AREA output each INDICATOR.

====Syntax====
The loop is defined by giving the table row a label which starts with a syntax of :=LOOP([args]), like in the following example
:=LOOP([dimension id], [preserve hierarchy], [loop order], [explicit values to include])

'''Argument Definition'''
''dimension id'' = Required. The ID of the Dimension to loop over. For example CURRENCY would loop over all the Currency values which have data in the given context. The row label is the Currency, for example USD.
''preserve hierarchy'' = Optional, default false. If true the hierarchy of the Codes in the underlying Codelist will be built into the row hierarchy
''loop order'' = Optional, default 1. 1=Natural Order (as they appear in the Codelist), 2=By Code ID, 3=By Code Label
''explicit values to include'' = Optional, default include all values. A var arg list of values to loop over.

'''Full Example'''

:=LOOP(REF_AREA, true, 3, EUR,FR,DE,UK,ES)

====Simple Loop====
The loop is defined by giving the table row a label which starts with a syntax of :=LOOP([args]), like in the following example
:=LOOP(REF_AREA)

This loop definition will loop through all the REF_AREA values, building a row for each value. The Observation keys for the row can refer to the loop variable using a $(DIM_ID) placeholder in the Observation Key, for example:

Employed Student
:=LOOP(REF_AREA) A:$(REF_AREA):EMP A:$(REF_AREA):STU

The output table would look like the following:

Employed Student
United Kingdom 20 12
France 22 11
Germany 25 15

====Nested Loop====
If a loop row is followed by a row (or rows) which is an extra level deep, the child row(s) will be included for each loop, for example

Employed Student
> :=LOOP(REF_AREA)
>> Total A:$(REF_AREA):'''T''':EMP A:$(REF_AREA):'''T''':STU
>>> Male A:$(REF_AREA):'''M''':EMP A:$(REF_AREA):'''M''':STU
>>> Female A:$(REF_AREA):'''F''':EMP A:$(REF_AREA):'''F''':STU

Because the '''Total''' row is a child row of the LOOP (denoted by the number of leading '>') and '''Male'''/'''Female''' are children of '''Total''' - for each REF_AREA there will be a Total, Male and Female, outputting a table like the following:

Employed Student
United Kingdom
Total
Male 5 1
Female 4 2
France
Total
Male 6 2
Female 7 3

It is also possible to define a subloop as a loop. The previous example can be rewritten as:

Employed Student
> :=LOOP(REF_AREA)
>> :=LOOP(SEX) A:$(REF_AREA):'''$(SEX)''':EMP A:$(REF_AREA):'''$(SEX)''':STU

There is no limit to the number of subloops that can be defined.

====Loop Hierarchical Values====
If the underlying Codelist has a hierarchy, a subsequent argument can be provided to ask for the hierarchy to be preserved

Countries
> :=LOOP(REF_AREA, true)

'''Note:''' A hierarchy will keep rows with no data if they are acting as a parent node for rows that do have data (e.g. there may not be data for Europe, but the row exists because there are child rows of France and Germany with data).

Nested loops can also preserve hierarchies for example:

:=LOOP(REF_AREA, true)
>:=LOOP(SEX, true)

With an example output (assuming no data exists for the hierarical parents (Continent, Europe)

Continent
Europe
France
Total (M+F)
Male
Female
Germany
Total (M+F)
Male
Female

====Order Loop Rows====
It is possible to order the labels by their natural order (that of the Codelist), by id or by label, using the numbers 1, 2, or 3 respectively. The following outputs the rows ordered by label.

Countries
> :=LOOP(REF_AREA, true, 3)

====Fixed Loop List====
It is also possible to define a subset of the values to loop over. For example, if REF_AREA has data for 100 countries, but only 3 are required in the table. In this scenario, the natural order is that of the provided list, not of the underlying Codelist.

Countries
> :=LOOP(REF_AREA, true, 3, UK, FR, DE)

====Loop over External Variable====
Instead of looping the values of the Dimension's Codelist, it is possible to loop another Codelist - the purpose for this is to support alternative labels (if the labels for the publication table need to be different from those of the Dimension members) - or a subset (as an alternative to passing the subset of values into the LOOP argument), or to define a hierarchy (if the Dimension's Codelist does not have a hierarchy, or an alternative is required).

In order to loop over an external Variable, the Publication Table must be linked to an external Codelist/Valuelist against an Alias. For example EXT=My Country Codelist with Hierarchy. The corresponding loop then loops the Codes in the external list, and stores these the corresponding Dimension ID variable, for example:

Countries
> '''REF_AREA''':=LOOP('''EXT''')

Noting that the loop is over the variable EXT which resolves to an external Codelist. The loop is assigned to the REF_AREA Dimension.

=== Anchor Rows ===
The default behaviour is for a row to be omitted if the row has no data and it has no child rows with data. However, if the desired result is to always output the row, then this can be achieved by using the [anchor] tag in the row header label.

Consider the following Publication Table definition:

Male Female
Employed A:M:EMP A:F:EMP
Student A:M:STU A:F:STU
Retired A:M:RET A:F:RET

If there are no data for Student, then the materialised table would look like this

Male Female
Employed 12 13
Retired 3 5

==== Anchor Single Row [anchor] ====
To anchor a single row, add the '''[anchor]''' tag at the start of the row

Male Female
Employed A:M:EMP A:F:EMP
[anchor]Student A:M:STU A:F:STU
Retired A:M:RET A:F:RET

Then the output table would preserve the 'Student' row

Male Female
Employed 12 13
Student
Retired 3 5

This rule can be coupled with [[Publication_Table#Formatting_Missing_Observations|formatting for missing observations]], then the table could be enhanced to look something like this

Male Female
Employed 12 13
Student - -
Retired 3 5

==== Anchor Child Rows [anchor+] ====
For a row with a hierarchy, an '''[anchor+]''' tag can be placed on a parent node to anchor the parent and all of its children, for example:

Continent
[anchor+]>Europe
>>France
>>>Total (M+F)
>>>>Male
>>>>Female
>>Germany
>>>Total (M+F)
>>>>Male
>>>>Female

The above rule would always output Europe and all of its descendants (France, Total, Male, Female, Germany).

==== Anchor Siblings and Child Rows [anchor*]====
The '''[anchor*]''' tag can be used as a way to anchor all following siblings and child nodes, this can be a useful mechanism to anchor all rows for the whole table by placing the anchor[*] on the first row of the table.

==== Anchor Loops ====
If a loop is preceeded by an '''[anchor]''' tag then the system will have the following behaviour:

* If an external Codelist variable is being looped, all codes will be ouput regardless of what data is stored in the system
* If a Dimension is being looped, all codes in the Codelist used by the Dimension will be output regardless of what data is stored in the system
* If a Dimension is being looped and it is not coded (i.e it is free text or numerical or other) then the [anchor] has no effect, as the looped values are based on what data exists

== Observation Key ==
=== Overview ===
The purpose of the Observation Key is to reference a Observation whose value will be resolved at table build time. The Observation Key can explicitly resolve to 1 observation value, if all parts of the Key are provided. However, it is possible to make parts of the key varaibles, allowing the values to be set at table build time.

An Observation Key is made up of 4 parts:

* Dataflow Reference
* Series Reference
* Time Period Reference
* Measure Reference

An example Observation Key with all of these parameters in place is:

DF_REG.M:UK:SUR:2008:BIRTHS

The key is broken down into the following parts:

{| class="wikitable"
|-
! Observation Key Part !! Name !! Description !! Required
|-
| DF_REG|| Dataflow Alias|| This identifies the Dataflow which has the data. The identification uses the alias, which is assiged to the Dataflow in the Publication Table definition. The Dataflow Alias is only required if the Publication Table references more then one Dataflow. If the Publication Table references more then one Dataflow, it can assign a Dataflow to a Default Alias, which means that it is required as part of the Observation Key. || No (if only 1 Dataflow or a default alias is set)
|-
| M:UK:SUR || Series Key || This identifies the Series of Data for the Dataflow, by providing a single value for each of the Dimension Members of the Dataflow, separated by a ‘:’ character. In this example the Dataflow has 3 Dimensions (excluding time), the value for the first is 'M', the second 'UK' and the last 'SUR'. As the Dataflow in this example has a Time Dimension this Series Key will uniquely identify a Series of Observations where each Observation refers to a unique Time Period || Yes (but can omit values)
|-
| 2008 || Time Period || This identifies the Time Period in the Series of Data identified by the previous section. The Combination of the Series and Time Period for a Time Series Dataflow will uniquely identify the Observation || No. Omit to make it variable for Time Series Data (defaults to last period). Not required if there is no Time Dimension for the DSD.
|-
| BIRTHS || Measure Dimension || If the Dataflow uses a Data Structure with multiple Measures, then the Measure ID must be included as the last part of the Observation Key. In this example the Dataflow has multiple measures (BIRTHS/DEATHS/MARRIAGES) so the Observation Key needs to include this value. || When there is only 1 measure it is not required, when there are multiple measures, it is required.
|}

Observation Keys are included on the materialised table in JSON format enabling the observation value to be tied back to the key. Observation Keys also provide a mechanism by which [[#Formatting_Cells|formatting rules]] can be applied to a subset of table cells.

=== Variable Dimension Members ===
It is possible to omit Dimension Values from the Observation Key, when this is the case that Dimension will become a variable – whose value can be set at table build time. For example, the second Dimension value of ‘UK’, from the previous example, can be omitted like this:

M::SUR:2008

The Publication Table can now be built for any Country by passing in the variable at table build time - i.e. UK, FR, DE. By passing in different variable values, the Observation Key will change, and as such it will reference a different Series in the Dataset.

When an Observation Key has a variable Dimension member, the ID of the Dimension can be referenced in text cells to create variable headings. For example, if the ID of the 2nd Dimension in the DSD is '''REF_AREA''', the Table Column could be given a title of $(REF_AREA) which would resolve to 'United Kingdom', 'France', or 'Germany' at build time depending on which varaible value is used to build the table. [[#Dimension Variables|#See Dimension Variables]].

=== Variable Time Period ===
For Time Series dataset, the Time Dimension can form part of the Observation Key, for example M:UK:SUR:'''2008'''. The 2008 part of the key refers to the year 2008.

It is possible to omit the value for Time Period in the Observation Key. Omitting Time Period will make the Time value a variable which can be provided at table build time, defaulting to the last period for which data exists if no value is supplied.

Extending the above example to omit the Time value results in an Observation Key whose Country and Time values can be provided to the web service at table build time.

M::SUR

'''Note:''' The above key is exactly the same as using the variable Time Period value 'P' in the Time Dimension, as follows:

M::SUR:'''P'''

Additional functions can be performed on the Time Dimension, such as moving forwards or backwards from the table base period, [[#Time Variables|#see Time Variables]] for more details.

=== Free Text Observation Keys ===
Whilst typically a cell in the body of the Publication Table would resolve to an Observation value, it is possible to use it for free text. To include free text cells, start the cell value with a leading apostrophe character, for example:

'Cell Text

'''Note''': An Observation Key can '''not''' contain a mix of free text as well as an Observation value

== Calculated Observation Cells ==
[[File:TableDefinition Calculations Variables.png|thumb|Publication Table with Calculated Cells and Variable Dimensions]]

An Observation Cell may contain calculated values, which can consist of a mix of simple algebra (+, -, *, /) and a reference to one or more Observations which may come from multiple Dataflows.

To define a Calculated Observation the Cell the following rules apply:

* The Observation Key must start with :=
* Any Observation Keys must appear within enclosing square brackets '[' and ']'
* An Observation key may embed calculations on a single Dimension by using parenthesis '(' and ')' in the key part
* Numbers in Observation Keys are treated as Code IDs, unless prefixed with a '#'
* The same rules for variable Dimensions and Time Period apply when resolving the Observation Keys used in the calculation

Examples are shown in the table below

{| class="wikitable"
|-
! Example Syntax !! Description
|-
| :=[A:UK:EMP]+10 || Add 10 to the observation value retrieved from the key A:UK:EMP
|-
| :=[A:(UK+FR+DE):EMP] || Add the series A:UK:EMP + A:FR:EMP + A:DE:EMP
|-
| :=[A:(UK+FR+12):EMP] || Add the series A:UK:EMP + A:FR:EMP + A:DE:12
|-
| :=[A:(UK+FR+#12):EMP] || Add the series A:UK:EMP + A:FR:EMP and add the Constant '12'
|-
| :=[A:UK:EMP]+[A:FR:EMP]+[A:DE:EMP] || Add the series A:UK:EMP + A:FR:EMP + A:DE:EMP
|-
| :=[A:(UK+FR):EMP]/[A:(UK+FR):UMP] || Adds the series A:UK:EMP + A:FR:EMP and divides by the result of A:UK:UMP + A:FR:UMP
|-
| :=((100/[A:UK:EMP:(M+F):P-1])*[A:UK:EMP:(M+F)])-100 || Demonstrates the ability to use parenthesis outside of the Observation Keys to control the order of mathematical operations
|-
| :=[POP.M:UK:(M+F)] / [LAB.M:UK:EMP] || Demonstrates the ability to calculate data from two Dataflows (defined by alias POP and LAB)
|-
| :=[M:UK:EMP:M(P)] + [A:UK:EMP:A(P)] || Demonstrates the ability to calculate data where the period is converted from the base period
|}

=== Calculated Cell Observation Key ===
Calculated Cells have implicit [[#Observation_Key|Observation Keys]] which are derived based on the inputs series in the calculation. The [[#Observation_Key|Observation Key]] can be used to assign [[#Formatting_Cells|Formatting Rules]] in the same way as a 'standard key' can be used. It is possible to assign an explicit key to the calculation, which will override the implicit key assigned. The following table shows how the key assignment works.

For example
{| class="wikitable"
|-
! Calculation !! Output Key || Explanation
|-
|:=[A:(UK+FR+DE):EMP] || A:#:EMP:2009 || The first and last dimension have a fixed value, the second dimension has a variable value and is replaced by '#'. The time period is added to the end of the key.
|-
|:=((100/[A:UK:EMP:(M+F):P-1])*[A:UK:EMP:(M+F)])-100 || A:UK:EMP:#:# || The first three dimensions have a fixed value, the last dimension has a variable value and is replaced by '#'. The time period is a variable as there are 2 time inputs to the calculation (P and P-1)
|-
|:=PCTOF([A:M:UK:P],[A:TOT:UK:P]) || A:#:UK:2009 || The first and last dimension have a fixed value, the second dimension has a variable value and is replaced by '#'. The time period is added to the end of the key.
|-
|:=[A:UK:EMP]+10 || A:UK:EMP || The input key is the same as the output key as none of the dimensions have a variable value. The time period is added to the end of the key.
|-
|A:UK10:EMP:=[A:UK:EMP]+10 || A:UK10:EMP:2009 || The output key is explicitly assigned. The time period is added to the end of the key.
|}

The output key conforms to the same Dimension breakdown as the underlying Data Structure. If a Observation Key is explicitly assigned, it must also honour the same Dimension breakdown. This allows formatting rules to be applied using the key component syntax, for example COUNTRY=UK10 would search for Observation Keys whose Country Dimension is reporting the value UK10.

== Percentage Of ==
The PCTOF (Percentage Of) function can be used to calculate 'x' as a percentage of 'y' following the syntax:

PCTOF(x,y)

This is reolved to the following mathmatical expression: '''(100/y) * x'''

x and y must either be a numerical value, and observation key, or an expression using a combination of the two. For example

:=PCTOF([A:M:UK:P],[A:TOT:UK:P])

The Observation Key follows the same standard rules with respect to varaibles and inline calculations - the following example shows an Observation Key where Male + Female values are added with the result being used in the Percentage Of function

:=PCTOF([A:M:UK],[A:('''M+F'''):UK])

A more complex example shows calculations around the PCTOF function:
:=(10 + PCTOF([A:M:UK:A(P)-1],[A:(M+F):UK:A(P)])) / 2

== Footnotes ==

[[File:Footnotes3.png|thumb|A table with footnotes]]
[[File:Footnotes2.png|thumb|Footnotes in the table builder]]

=== Overview ===
Footnotes can be added to any part of the publication table (table column/row header, or observation cells). The output table is generated with a footnote number, with the list of footnotes at the end of the table.

Footnotes need to be thought about in 2 parts: Part 1 are the list of footnotes which are defined at the Publication Table level. Part 2 are the references to the footnotes from within the table. This separation of allows footnotes to be reused within the same table, and when working with [[#Footnotes_with_Placeholders|placeholders]] footnote templates can be set up where the text changes based on the placeholder values defined in the table builder.

To add a footnote, add it to the Publication Table, for example: 
Footnote 1 
Footnote 2

Then in the table builder, add a reference to which footnote to use, $(1) references '''Footnote 1''', and $(2) references '''Footnote 2'''.

'''Note:''' The order of the list of footnotes in the table definition may not be reflected in the order they are output. As such a footnote referenced by $(1) may be output as the second footnote.

=== Footnotes with Variable ===
A footnote can reference variables, using the [[#Table_Variables|variable syntax]],

A footnote for $(REF_AREA.UK)

A footnote can even output an observation value, observation attribute or series attribute.

Measure Value
A footnote with obs value $(A:UK:EMP:2008:OBS_VALUE)

Observation Attribute
A footnote with obs attribute $(A:UK:EMP:2008:OBS_COMMENT)

Series Attribute
A footnote with obs attribute $(A:UK:EMP:P:SERIES_TITLE)

When combining variable footnotes with placeholder values and loops, it is possible to dynamically create references to observation attributes.

'''Note:''' the use of P in the final cell key which is a reference to the [[#Time_Variables|time period]] for which the table is being built.

=== Footnotes with Placeholders ===
It is possible to write a generic footnote for the publication table, which includes placeholders. When the footnote is referenced, the placeholder values are provided. The syntax for a placeholder is {n} where n is a positive integer, for example {1}.

Footnote Example

A footnote for $(REF_AREA.{1}) and $(REF_AREA.{2})

In the table builder, the footnote is referenced as usual, but with arguments for the placeholders:

Footnote reference in Table builder

My Table Row '''$(1,UK,FR)'''

The footnote reference in the above example is in bold, and consists of which footnote text to use (in this case the first one in the list, denoted by '1'), and 2 arguments are passed for the placeholders, placeholder 1 takes the value of UK and placeholder 2 takes the value of FR. During the materialise process the intermediate text is generated:

A footnote for $(REF_AREA.UK) and $(REF_AREA.FR)

Which then goes through the variable resolve process, in this case both variables, '''$(REF_AREA.UK) and $(REF_AREA.FR)''', match the [[#Coded_Variables|Coded Variable]] syntax and would resolve to:

Final Output
A footnote for United Kingdom and France

=== Footnotes in Observation Cell ===
Any number of footnotes can appear in an Observation cell, simply put the footnote(s) at the end of the [[#Observation_Key|Observation Key]].

Standard Footnote Reference 
DF_REG.M:UK:SUR:2008:BIRTHS $(1) $(2)

Footnote with Placeholder 
DF_REG.M:UK:SUR:2008:BIRTHS $(1,UK)

=== Footnotes in Loops ===
A row [[#Looping_Rows|loop]] contains a locally scoped variable, and as such it can be used when calling a footnote with a placeholder.

Example Footnotes
$(A:T:{1}:EMP:P:SERIES_TITLE)
$(A:T:{1}:EMP:P:TITLE_COMPL)
$(A:{1}:{2}:EMP:P:OBS_COMMENT)

Example Table Definition
Employed Student
> :=LOOP(REF_AREA) '''$(1,$(REF_AREA[id]))'''
>> Total '''$(2,$(REF_AREA[id]))''' A:$(REF_AREA):T:EMP A:$(REF_AREA):T:STU
>> :=LOOP(SEX) '''$(3,$(REF_AREA[id]),$(SEX[id]))''' A:$(REF_AREA):$(SEX):EMP A:$(REF_AREA):$(SEX):STU

Footnote Intermediate Resolution
Row 1 (REF_AREA=UK)
$(A:T:UK:EMP:P:SERIES_TITLE)

Row 2 (REF_AREA=UK)
$(A:T:UK:EMP:P:TITLE_COMPL)

Row 3 (REF_AREA=UK, SEX=M)
$(A:UK:M:EMP:P:OBS_COMMENT)

Final Footnote output
Row 1 (REF_AREA=UK)
This is a series for the United Kingdom

Row 2 (REF_AREA=UK)
This is a title compliment for the United Kingdom

Row 3 (REF_AREA=UK, SEX=M)
This is an observation comment for Males in the United Kingdom

= Formatting Cells =
== Formatting Numerical Observation Values ==
[[File:Obs formatting.png|thumb|Number Formatting Options]]
Numerical observation values can be formatted in the following way

# '''Number of Decimal Places''' or '''Significant Figures''' can be defined
# For Decimal places this can be set to a maximum rule or a fixed rule, i.e the value 1.3 to 2 decimal places is 1.3 using the maximum rule and 1.30 using the fixed rule
# '''Rounding mode''' can be specified
# '''Scaling Factor''' this is a multiplier applied to the observation value, the multiplier is x10^[scaling factor]. Example, a value of 1.1 with a scaling factor of 1 is 11, a scaling factor of 2 is 110, a scaling factor of 3 is 1100. If the Observation or related Series has a UNIT_MULT concept the difference between the unit multiplier and scaling factor will be taken. Example, an Observation with a value of 1.1 and a unit multiplier of 9 (Billions) is reported, the publication table has a scaling factor of 6 (millions) - the number 1.1 is scaled using 10^3 (calculated by 9-6), the 1.1 billion is converted to 1100 million.
# '''Scientific notation''' can be requested on output, i.e 123456 to 2 significant figures is 120000 in standard form or 1.2E4 in scientific form
# '''Decimal formatting''' for long numbers, i.e. 12345.8 there is not thousands separator by default, to include a thousand separator set the decimal formatting rule to either a single Language (i.e. French Formatting) or Dynamic formatting, which uses the Accept-Langauge locale/language sent be the web browser (this can be overridden using the locale=[loc] query parameter. The formatting rules can also influence which characters the numerical characters used.
# Rules can be applied to all series, or specific series

=== Match Series ===
When creating a formatting rule, it is possible to define which series the rule applies to. If left blank, the rule will apply to all observation values.

The syntax for defining specific series is shown in the following table:

{| class="wikitable"
|-
! Rule Type !! Syntax !! Description
|-
| Series Match || =A:UK:EMP || Match a specific series from the default Dataflow
|-
| Series Match Multiple Values || =A:UK+FR+DE:EMP || Multiple codes in 2nd Dimension (UK, FR and DE) any can match for the rule to be valid
|-
| Series Match by Dataflow Alias || =FLOW2.A:UK:EMP || Match a specific series for the Dataflow with alias FLOW2
|-
| Wildcard Series Match || =A::EMP || Match all series with the given dimension values (2nd Dimension is wildcarded)
|-
| Not Rule || !=A:UK:EMP || Match all series which do not match this series (wildcard series also supported)
|-
| Dimension Rule || CURRENCY=GBP || Match all series which have CURRENCY of GBP
|-
| Not Dimension Rule || !CURRENCY=GBP || Match all series do not have a CURRENCY of GBP
|-
| Multiple Rules || CURRENCY=GBP,FREQ=M || Comma is used to include multiple matches (also applies to series match)
|}

== Formatting Missing Observations ==

Observation Values can be missing for 2 reasons:
# There is no Observation in the database for the given Observation Key
# There is an Observation in the database but the measure value is either: NaN, null, or an empty string

In both cases it is possible to define an output value using formatting rules. For the first case, the observation does not exist, it is possible to define a default output value, for example ‘-‘. For the second case, where the observation does exist, it is possible to define a set of rules based on another Component of the Observation, for example the Observation output may be related to the OBS_STATUS Attribute.

When the formatted value is based on a Component, a set of rules must be defined which state ‘when component=x output y’, where x may be an fixed value or a pattern to match (using a regular expression). The rules for mapping component values are defined in a [https://fmrwiki.sdmxcloud.org/Representation_Map Representation Map], and the Publication Table simply provides the connection to the Representation Map to use. This enables reuse of the mapping rules across multiple Publication Tables. The Publication Table must define two properties in order to support Component Mapping rules:

# The Component to obtain the value from i.e OBS_STATUS
# A link to the [https://fmrwiki.sdmxcloud.org/Representation_Map Representation Map], which holds the mapping rules. Note, the Representation Map is expected to have 1 source mapping, this could be of any type (string, codelist, other)

An example Reprsentation Map for OBS_STATUS is shown below:

{| class="wikitable"
|-
! OBS_STATUS !! Is Reg Ex !! Missing Value !! Description
|-
| M || false || mv || When OBS_STATUS is M output mv
|-
| L || false || np || When OBS_STATUS is L output np
|-
| .* || true || - || When OBS_STATUS is anything else output -
|}

== Frequency Formatting ==
[[File:PublicationTable FreqMap.png|thumb|Representation Map with frequency mapping rules]]
[[File:PublicationTable Materialised FreqMap.png|thumb|Materialised table with formatted time headings ]]
Time Variables always resolve themselves to an [[SDMX_Time_Formats|SDMX Date Format]] when forming the [[#Observation Key|Observation Key]], this ensures the correct observation value is returned from the data store. When a time variable is used as a label, in a table heading for example, the same formatting will be applied by default but a custom format can be provided. Custom display formats for a Time Period enable a time period such as 2007-Q1 to be resolved to custom text string, for example '31 Mar 2007'.

Custom Time formats rules are defined by providing an association between an SDMX Frequency ID (#see [[SDMX_Time_Formats|SDMX Date Format]]) and the output format template. The output format template is a string that defines which components of time are output an in which order and style. For example to output a date as 'March 2001' the format template would look like this 'MMMM yyyy' to indicate a full month name (MMMM) and a 4 character year (yyyy). format rules make use of the [https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html Simple Date Format] which is used by Java as such all of the documentation on Simple Date Format applies.

Frequency Format Mapping is not defined as part of a Publication Table, instead the rules are defined in a [https://fmrwiki.sdmxcloud.org/Representation_Map Representation Map] and the Publication Table references the Representation Map as the source of its frequency conversion rules. This enables the same set of rules to be used for multiple Publication Tables. The Representation Map must map 1 source (Codelist or String) to 1 target (String), it can contain as many mapped frequencies as required. If the Representation Map contains a mapping for a value that is not a valid SDMX Frequency, then it will be ignored.

It is important to note that all time periods resolve to a single point in time, which is the end of the period, for example an Annual period of 2008 resolves to 2008-12-31:T23:59:59. The resolution of frequency to a point in time enables the frequency formatting rules to use all aspects of time, even when converting a low frequency, for example 2008 formatted using the pattern dd MMM yy would resolve to '31 Dec 2008'.

== Variables Summary ==

{| class="wikitable"
|-
! Variable Type !! Description !! Syntax !! Example !! Example Output
|-
| '''Dataflow Name''' || Displays the name of a Dataflow referenced by the Publication Table in the locale in which the Publication Table is built || $([dataflow alias]) || $(EDU) || World Bank, Education
|-
| '''Dataflow Description''' || Displays the description of a Dataflow referenced by the Publication Table in the locale in which the Publication Table is built || $([dataflow alias].desc) || $(EDU.desc) || World Bank, Education
|-
| '''Dataflow Id''' || Displays the ID of a Dataflow referenced by the Publication Table || $([dataflow alias].id) || $(EDU.id) || DF_EDU
|-
| '''Variable Dimension Value''' || Displays the name of the variable used when an Observation key has a wildcard member || $([dimension id]) || $(REF_AREA) || United Kingdom
|-
| '''Specific Component Value''' || Displays the name of the Component Value || $([component id].[component value]) || $(OBS_STATUS.C) || Confidential
|-
| '''Component Name''' || Displays the name of the DSD Component (this is the name of the Concept used by the Component) in the locale in which the Publication Table is built || $([component id].name) || $(OBS_STATUS) || Confidentiality Status
|-
| ''' Enumeration Value ''' || Displays the name of a Code in a Codelist or Value in a Valuelist || $([list alias].[list item ID] || $(SEX.M) || Male
|-
| '''Conditional Variable''' || Displays the variable value, which is conditional on the value of a variable Dimension || $([conditional variable id][variable id]) || $(BANKNOTES[NOTE1]) || £5
|-
| ''' Time ''' || Displays the base time period for which the table is built, or an offset from this time period / cast to a different frequency || see [[#Time Functions|Time Functions]] || $(P) || 2008
|}

= Table Variables =
== Overview ==
Variables are a very important part of Publication Tables. They can be used to make the content of the table dynamic, setting and changing the value at run time to change the table content. Variables can be used in Observation Keys, but the values of variables can also be used anywhere where there is text, for example the table heading, subheading, column or row headings, footnotes, and Observation Cells with free text.

The general syntax for a variable is

$(''variable id''.''sub part''[''detail''])

Where variable id is always required, the subpart and detail are optional. Subpart is used if the variable resolves to a Component and a coded value is required, the detail array part obtains the id, name, or description of the underlying SDMX type.

This section describes the various variables that exist, along with the syntax of how to use them.

== Dataflow Variables ==
Publication Tables define data cells across one or more Dataflows. Each ‘imported’ Dataflow is given an alias, which can be used to refer to the Dataflow in the [[#Observation Key|Observation Key]].

It is possible to refer to the Dataflow's Name, Description, and Id using the following variable syntax.

'''Default Dataflow''' 
Dataflow Id
$([id])

Dataflow Name
$() or $([name))

Dataflow Description
$([desc])

'''Dataflow with Alias=EXR''' 
Dataflow Id
$(EXR[id])

Dataflow Name
$(EXR) or $(EXR[name))

Dataflow Description
$(EXR[desc])

== Dimension Variables ==
When an [[#Observation Key|Observation Key]] has [[#Variable Dimension Members|variable Dimension Values]], the table will be built by replacing this variable with a value to complete the Observation Key. The variable value is either passed explicitly to the web service, or defaulted by the system if not provided. The Dimension Variable can be referenced in the table using the syntax $([dimension id]), for example:

Dimension Value Name
$(INDICATOR) resolves to 'Employment'

This would resolve to the '''name''' of the Indicator value used in the table, for example '''EMP''' may resolve to the label '''Employment'''. The label for the Dimension Value is obtained from the Codelist or Valuelist which the Dimension uses. If the Dimension does not use a list (i.e. it is free text or numerical) then the Dimension value is simply used in place of the variable placeholder.

To resolve to the '''id''' of the value for the Component, for example '''EMP''', use the following syntax

Dimension Value Id
$(INDICATOR[id]) resolves to 'EMP'

== Dependent Dimension Variable ==
It is possible that the value for one Dimension is dependent on the value of another Dimension, for example CURRENCY may be dependent on the COUNTRY and COUNTRY is a variable in the [[#Observation Key|Observation Key]] - it is possible to specify that the value to use for the CURRENCY Dimension can be derived using a lookup table. The lookup table in this instance is maintained as an [https://fmrwiki.sdmxcloud.org/Representation_Map SDMX Representation Map] which is used to map COUNTRY to CURRENCY. The Publication Table specifies that the two Dimensions are related and provides a reference to the Representation Map to use which has the rules.

When working with a dependent variable, both the dependent variable and the Dimension on which it depends should be omitted from the [[#Observation Key|Observation Key]] - for example consider a full key of:

A:UK:GBP:2007

Where UK is the COUNTRY and GBP is the Currency, to make the COUNTRY a variable Dimension, it should be omitted from the key, as follows:

A::GBP:2007

To then make the CURRENCY a variable it is also omitted , as follows:

A:::2007

The Publication Table is given the rule that CURRENCY depends on COUNTRY and should use the COUNTRY_2_CURRENCY [https://fmrwiki.sdmxcloud.org/Representation_Map Representation Map], which in this example looks like the following:

{| class="wikitable"
|-
! COUNTRY !! Is Reg Ex !! CURRENCY
|-
| UK || false || GBP
|-
| US || false || USD
|-
| .* || true || EUR
|}

When the Publication Table is materialised the [[Publication_Table_Web_Service|web service]] only requires the COUNTRY information, and it will complete the Observation key using the representation map lookup table.

== Component Values ==
Components of the underlying Data Structure can be explicitly referenced using the following syntax '''$([component id])''', for example:

Component Name
$(OBS_STATUS) or $(OBS_STATUS[name]) resolves to 'Observation Status'

Component Description
$(OBS_STATUS[desc]) resolves to 'Information on the quality of a value or an unusual or missing value.'

== Coded Variables ==
To reference a Code value, the syntax is $([component id / codelist alias].[item id]). The first variable part is the ID of a Component in the underlying Data Structure, or alternativly the alias of an external variable defined on the Publication Table. An external variable is a link to an SDMX Codelists or Valuelist. Example are:

Name of Code
$(CURRENCY.$) resolving to the text '''Dollar''

Description of Code
$(DWELLING.SM[desc])

== Conditional Variables ==
A Conditional Variable is one whose value is conditional on the value of a [[#Dimension Variables|Dimension Variable]].

An example use case is Banknotes, where the values for NOTE1, NOTE2, or NOTE3 is dependant on the Country Dimension. A Conditional Variable could be used to set a values for each variable (NOTE1, NOTE2, NOTE3) based on the value used for the current Country Dimension i.e. US NOTE1=$5, UK NOTE1=£5.

The syntax of referring to a Conditional Variable is $([conditional variable id][variable id]), for example
$(BANKNOTE[NOTE1])

== Reported Attribute for an Observation or Series ==
A variable can be a reference to a part of an Observation, for example the Unit Multiplier (UNIT_MULT). A Table Row header, for example, could include variable text such as 'Number of payment accounts (''thousands'')' where the word '''thousands''' is a variable because it depends on other variables which build the key.

To achieve this, use the syntax:

$(A:UK::2008:UNIT_MULT[name])

The observation key in this case is A:UK::2008, where the third Dimension is a variable as it has no value (in the main body of the table this dimension would probably also be a variable, which means the table can be built with different values, each value may change the output of this UNIT_MULT value). The last part of the key is the Attribute ID (UNIT_MULT) - this can be a reference to a series attribute, or an observation attribute. The [name] indicates that it should resolve to the name of the reported attribute value, alternative values are '''id''' for the reported value or '''desc''' for the description of the reported value.

If time is also a variable, then the key can change accordingly, using the variable syntax for time (P):

$(A:UK::P:UNIT_MULT[name])

== Time Variables ==
[[File:PublicationTable MovingTime.png|thumb|Publication Table Design, moving and converting the base time period]]

[[File:PublicationTable Materialised MovingTime.png|thumb|Moving and converting periods - showing the materialised table]]

When the data is time series (i.e. the [[Data_Structure_Definition_(DSD)_-_Structural_Metadata_Management|DSD]] has a Time Dimension), the [[#Observation_Key|Observation Key]] can include the time period, for example:

A:UK:EMP:'''2008'''

Where 2008 is the Observation Time Period to use. '''The format of the Time Period must always be the same format as the frequency for which the data are reported'''. For example if the data is annual, and an observation is reported for 2008, then the observation key must use 2008, not another frequency such as 2008-12-31. It is typical in a [[Data_Structure_Definition_(DSD)_-_Structural_Metadata_Management|DSD]] for the Frequency of the Series to be the first Dimension, in all the examples on this page this convention is applied, and as such the above key A:UK:EMP refers to Annual data, depicted by the initial dimension value 'A'.

If the key omits the time period, then the '''time period becomes a variable''' which can be set at table build time, if not set it defaults to the last period for which there is data.

A:UK:EMP

When a Publication Table has a variable time period, the resolved variable time value for this varaible is known as the '''base period'''. There can only be 1 base period for a Publication Table, however other Observation Keys (and corresponding headings) can perform time arithmetic from the base period for which the table is built.

The supported time functions include walking forwards and backwards in increments of the given frequency, converting to another frequency of data, and a combination of the two (convert and move, move and convert, move convert and move again). Ths supported functions are shown in the table below.

{| class="wikitable"
|-
! Syntax !! Description !! Example Observation Key !! Example Resolved Key
|-
| P || The base period (default if not provided) || A:UK:EMP:P || A:UK:EMP:2008 (base period=2008)
|-
| P-''n'' || Base Period minus 'n' Periods where n is a positive Integer || A:UK:EMP:P-1 || A:UK:EMP:2007 (base period=2008)
|-
| P+''n'' || Base Period plus 'n' Periods where n is a positive Integer || A:UK:EMP:P+1 || A:UK:EMP:2008-Q2 (base period=2008-Q1)
|-
| A(P) || Base Period converted to '''Annual''' || A:UK:EMP:A(P) || A:UK:EMP:2008 (base period=2008-Q2)
|-
| S(P) || Base Period converted to '''Bi-Annual''' (Semester) || S:UK:EMP:S(P) || S:UK:EMP:2008-S2 (base period=2008-Q3)
|-
| T(P) || Base Period converted to '''Tri-Annual''' || T:UK:EMP:T(P) || T:UK:EMP:2008-T3 (base period=2008)
|-
| Q(P) || Base Period converted to '''Quarterly''' || Q:UK:EMP:Q(P) || Q:UK:EMP:2008-Q4 (base period=2008)
|-
| M(P) || Base Period converted to '''Monthly''' || M:UK:EMP:M(P) || M:UK:EMP:2008-03 (base period=2008-Q1)
|-
| W(P) || Base Period converted to '''Weekly''' || W:UK:EMP:W(P) || W:UK:EMP:2008-04 (base period=2008-01-01)
|-
| D(P) || Base Period converted to '''Daily''' || D:UK:EMP:D(P) || D:UK:EMP:2008-12-31 (base period=2008)
|-
| H(P) || Base Period converted to '''Hourly''' || H:UK:EMP:D(P) || H:UK:EMP:2008-12-31T12 (base period=2008-02-02T12:00:03)
|-
| A(P-n) || Base Period minus (or plus) 'n' periods, converted to another frequency 'A' || A:UK:EMP:A(P-4) || A:UK:EMP:2007 (base period=2008-Q4)
|-
| A(P)-n || Base Period, converted to another frequency 'A' minus (or plus) n' periods || A:UK:EMP:A(P)-4 || A:UK:EMP:2004 (base period=2008-Q4)
|-
| A(P-n1)-n2 || Base Period, minus (or plus) 'n1' periods, converted to another frequency 'A' minus (or plus) 'n2' periods || A:UK:EMP:A(P-4)-4 || A:UK:EMP:2003 (base period=2008-Q4)
|-
| FA(P) || Subtracts 1 Period from the Converted Period IF the resolved Convert From end of period does not match the same point in time as the Convert To period.

'''Examples:'''
* Base Period =2008-Q4 Rule= FA(P) Output = 2008
* Base Period =2008-Q3 Rule= FA(P) Output = 2007 - the base period 2008-Q3 is not the same time as 2008 (end of period, so move back 1 year)
* Base Period =2008-Q1 Rule= FA(P) Output = 2007
* Base Period =2008-Q1 Rule= FA(P+3) Output = 2008
* Base Period =2008-Q1 Rule= FA(P+3)-1 Output = 2007

This works for ALL frequencies where the convert to frequency is lower then the base Period P, for example Quarter converted to Annual.

|| A:UK:EMP:FA(P) || A:UK:EMP:2007 (base period=2008-Q3)
|}

'''Compound expressions''' 
From Fusion Registry v11.5.1 and later it is possible to cast a period more the once in the same expression, for example

Q(FA(P-1))-1

If the input time period P=2019-Q4, then P-1 = 2019-Q3, the expression is rewritten as Q(FA(2019-Q3))-1. The Full Annual (FA) for 2019-Q3 is 2018, as the end of period 2019 does not match the end of period 2019-Q3, so the annual period moves back an additional period. The expression is rewritten as Q(2018)-1. 2018 to Quarterly moves to 2018-Q4 and then the last part of the expression steps back 1 period, the final period is 2018-Q3.

'''Notes on Time Period''' 
When converting from a lower frequency to a higher frequency (Weekly to Daily for example), the cast date always resolves to the end of the Period.

It is also important to note that as the [[#Observation Key|Observation Keys]] in these examples all have Frequency as the first Dimension, when converting to another frequency (Quarterly to Annual for example) the Observation Key changes the value for the Frequency Dimension, ensuring the correct series of data is retrieved.

'''Time Period variable in free text''' 
When refering to the Time Period as a variable in free text, for example in the table heading or a column heading, the variable is placed inside the variable syntax '''$()'''. For example $(P) would resolve to base period, and $(A(P)) would resolve to 'base period converted to annual frequency'. An example usage is shown below:

Table 1A showing data for $(P) to $(P+3)

Publication Table

2023-01-24T09:48:18Z

Mnelson: /* Calculated Cell Observation Key */

[[Category:Functions]]

@See [[Publication_Table_Web_Services|Publication Table Web Services]]

= Publication Tables =
'''Introduced to Fusion Registry Enterprise v11.0.0.17'''

== Overview ==
[[File:Publication Table Definition.png|thumb|Using SDMX concepts of Agency, Id, Name, Description]]

[[File:Publication Table.png|thumb|Table Layout and Content Design]]

[[File:MaterialisedTable Preview.png|thumb|Showing the Table Preview feature which materialises the table with the given variables]]

A Publication Table is a curated table of data, where the values in the table rows, columns and table body cells can be individually specified. This enables a single table to pull data from multiple sources and piece the data together in a way which is not possible when dealing with standard cubes of data, whose table model is typically based on a Pivot model, with columns and rows built around the Dimension values to which the column or row is assigned. A Publication Table breaks out of this restrictive model by allowing cells of data to reference observations which do not need to share any relationship, in terms of dimensionality or even data source, with adjacent cells,.

Data in a Publication Table may be sourced from a single [[Dataflow_-_Structural_Metadata_Management|Dataflow]] or it may be sourced from multiple Dataflows enabling cross comparison of data.

Publication Tables can be static (every cell is a defined Observation) or dynamic, where a cell includes [[#Variable Dimension Members|variables Dimensions]] in the [[#Observation Key|Observation Key]], which can be provided at [[#Metadata vs Materialised|table build time]].

[[#Time Variables|Time arithmetic and frequency conversion]] is supported, enabling a single table to pull data from different time periods and frequencies which are all derived from the base period on which the table is built.

Publication Tables support [[#Calculated Observation Cells|calculations]], where each calculation can take input from one or more Observations over one or more [[Dataflow_-_Structural_Metadata_Management|Dataflows]].

Variable values also play a big role in Publication Tables, and includes standard variable placeholders – as well as advanced [[#Conditional Variables|conditional variables]], whose list of variable values change based on the value of another Dimension for which the table is built.

== Metadata vs Materialised ==

A Publication Table is stored in Fusion Registry the same way any other piece of structural metadata is stored. In this way it has an owner (Agency), and identity (ID) and a version. The change history is tracked using the same services, and the table definition can be [[Publication_Table_Web_Services|queried]] using the same SDMX web services for Structural Metadata.

The Publication Table metadata contains information about which Dataflows are used by the Table, which additional variables are being imported, and a definition of the table structure, and which Observations to import by referencing them by their unique [[#Observation Key|Observation Key]].

A Publication Table definition can be '''materialised''' into a table of Data by from the [[Publication_Table_Web_Services|Publication Table Web Service]], which combines the information of the Table Identity, and Variable Values to rewrite the [[#Observation Key|Observation Key]] into actual observation values. If building a web User Interface, then response from this service can be a packet of JSON, which describes the layout and content of the final Table in a way which is quick and simple to render for the User to see. The process of materialising the table is known as ‘'''build time'''’.

Additional web services are provided to enable clients to ask questions about the table such as ‘what are the earliest and latest time periods for which there is data for this table’ or ‘what possible variable values are there for this table’. This enables user interfaces to provide a user with the option to change the variables ‘on the fly’ ensuing they are able to build a table which is known to have data.

= Publication Table Cell Content =
== Overview ==
[[File:PublicationTable Layout.png|thumb|Publication Table Layout]]
A Publication Table is made up of a '''Table Head Rows''' and '''Table Body Rows'''.

The Table Head is made up of 1 or more table rows. Each Table Head cell may span 1 or more columns and 1 or more rows. The Table Head content is made up of free text and / or variable placeholders whose values are resolved when the table is built. Table Head Cells can be left empty.

The Table Body is made up of 1 or more Table Rows. The Table Rows provide the Publication Table with its data, by referencing Observations from the Dataflow.

At table build time all variables are resolved, and the corresponding data are retrieved from the data stores which hold the data.

== Table Rows ==
=== Overview ===
A Table Row consist of a Table Row heading, this can be a mix of free text and [[#Table Variables|variable text]]. The Table Row Heading is always the first Table Row Cell, and can not span multiple columns or rows. A Table Row can not have more then one heading. The remaining Table Row Cells are the Observation Cells, which may contain free text and [[#Table Variables|variable text]] but will typically contain a reference to an Observation by an [[#Observation Key|Obsevation Keys]] or a [[#Calculated Observation Cells|Calculation]] which can combine multiple [[#Observation Key|Obsevation Keys]].

A Table Row which results in no Observation data will not be included in materialised table at build time, unless the Table Row is a grouping row (see [[#Row Levels|Row Levels]] for details on grouping rows).

=== Row Levels ===
[[File:PublicationTabel Levels.png|thumb|Showing Levels in the Table Rows]]
The Table Row Heading can be used to define a row level. The level indicates to the client responsible for rendering the table the row's position in a hierarchy. Row levels are depicted using one or more leading chevrons ‘>‘ before the text. The number of hyphens indicates the level, with one leading hypen indicating level 1, two indicating level 2, and so on. It is possible to skip levels if required, i.e. Row 1 is level 1 with Row 2 assigned to level 3, skipping out level 2 entirely.

An example of row headings using levels is as follows:

Countries
> Europe
>> France
>> Germany
> Asia
>> China

Levels can be an important way to '''group data'''. A Table Row which acts as a parent row in a hierarchy, can be used as grouping mechanism for the rows that follow. This enables Parent Rows to be included in the output table, even if the Table Row contains no Observation data. The default behaviour for a Table Row with no data is to exclude it from the built table.

=== Looping Rows ===

Instead of explicitly defining a row for each Dimension member, it is possible to define a row as a loop (to iterate over all, or a subset of Dimension members). For example, if a publication table contains a Dimension for countries with an ID of REF_AREA then a loop can be set up to loop through all REF_AREA values, or a subset of the values - for which there is data. If the values are backed by a Codelist which has a hierarchy, the hierarchy can be preserved in the final table. It is also possible to nest a loop within another loop, for example for each REF_AREA output each INDICATOR.

====Syntax====
The loop is defined by giving the table row a label which starts with a syntax of :=LOOP([args]), like in the following example
:=LOOP([dimension id], [preserve hierarchy], [loop order], [explicit values to include])

'''Argument Definition'''
''dimension id'' = Required. The ID of the Dimension to loop over. For example CURRENCY would loop over all the Currency values which have data in the given context. The row label is the Currency, for example USD.
''preserve hierarchy'' = Optional, default false. If true the hierarchy of the Codes in the underlying Codelist will be built into the row hierarchy
''loop order'' = Optional, default 1. 1=Natural Order (as they appear in the Codelist), 2=By Code ID, 3=By Code Label
''explicit values to include'' = Optional, default include all values. A var arg list of values to loop over.

'''Full Example'''

:=LOOP(REF_AREA, true, 3, EUR,FR,DE,UK,ES)

====Simple Loop====
The loop is defined by giving the table row a label which starts with a syntax of :=LOOP([args]), like in the following example
:=LOOP(REF_AREA)

This loop definition will loop through all the REF_AREA values, building a row for each value. The Observation keys for the row can refer to the loop variable using a $(DIM_ID) placeholder in the Observation Key, for example:

Employed Student
:=LOOP(REF_AREA) A:$(REF_AREA):EMP A:$(REF_AREA):STU

The output table would look like the following:

Employed Student
United Kingdom 20 12
France 22 11
Germany 25 15

====Nested Loop====
If a loop row is followed by a row (or rows) which is an extra level deep, the child row(s) will be included for each loop, for example

Employed Student
> :=LOOP(REF_AREA)
>> Total A:$(REF_AREA):'''T''':EMP A:$(REF_AREA):'''T''':STU
>>> Male A:$(REF_AREA):'''M''':EMP A:$(REF_AREA):'''M''':STU
>>> Female A:$(REF_AREA):'''F''':EMP A:$(REF_AREA):'''F''':STU

Because the '''Total''' row is a child row of the LOOP (denoted by the number of leading '>') and '''Male'''/'''Female''' are children of '''Total''' - for each REF_AREA there will be a Total, Male and Female, outputting a table like the following:

Employed Student
United Kingdom
Total
Male 5 1
Female 4 2
France
Total
Male 6 2
Female 7 3

It is also possible to define a subloop as a loop. The previous example can be rewritten as:

Employed Student
> :=LOOP(REF_AREA)
>> :=LOOP(SEX) A:$(REF_AREA):'''$(SEX)''':EMP A:$(REF_AREA):'''$(SEX)''':STU

There is no limit to the number of subloops that can be defined.

====Loop Hierarchical Values====
If the underlying Codelist has a hierarchy, a subsequent argument can be provided to ask for the hierarchy to be preserved

Countries
> :=LOOP(REF_AREA, true)

'''Note:''' A hierarchy will keep rows with no data if they are acting as a parent node for rows that do have data (e.g. there may not be data for Europe, but the row exists because there are child rows of France and Germany with data).

Nested loops can also preserve hierarchies for example:

:=LOOP(REF_AREA, true)
>:=LOOP(SEX, true)

With an example output (assuming no data exists for the hierarical parents (Continent, Europe)

Continent
Europe
France
Total (M+F)
Male
Female
Germany
Total (M+F)
Male
Female

====Order Loop Rows====
It is possible to order the labels by their natural order (that of the Codelist), by id or by label, using the numbers 1, 2, or 3 respectively. The following outputs the rows ordered by label.

Countries
> :=LOOP(REF_AREA, true, 3)

====Fixed Loop List====
It is also possible to define a subset of the values to loop over. For example, if REF_AREA has data for 100 countries, but only 3 are required in the table. In this scenario, the natural order is that of the provided list, not of the underlying Codelist.

Countries
> :=LOOP(REF_AREA, true, 3, UK, FR, DE)

====Loop over External Variable====
Instead of looping the values of the Dimension's Codelist, it is possible to loop another Codelist - the purpose for this is to support alternative labels (if the labels for the publication table need to be different from those of the Dimension members) - or a subset (as an alternative to passing the subset of values into the LOOP argument), or to define a hierarchy (if the Dimension's Codelist does not have a hierarchy, or an alternative is required).

In order to loop over an external Variable, the Publication Table must be linked to an external Codelist/Valuelist against an Alias. For example EXT=My Country Codelist with Hierarchy. The corresponding loop then loops the Codes in the external list, and stores these the corresponding Dimension ID variable, for example:

Countries
> '''REF_AREA''':=LOOP('''EXT''')

Noting that the loop is over the variable EXT which resolves to an external Codelist. The loop is assigned to the REF_AREA Dimension.

=== Anchor Rows ===
The default behaviour is for a row to be omitted if the row has no data and it has no child rows with data. However, if the desired result is to always output the row, then this can be achieved by using the [anchor] tag in the row header label.

Consider the following Publication Table definition:

Male Female
Employed A:M:EMP A:F:EMP
Student A:M:STU A:F:STU
Retired A:M:RET A:F:RET

If there are no data for Student, then the materialised table would look like this

Male Female
Employed 12 13
Retired 3 5

==== Anchor Single Row [anchor] ====
To anchor a single row, add the '''[anchor]''' tag at the start of the row

Male Female
Employed A:M:EMP A:F:EMP
[anchor]Student A:M:STU A:F:STU
Retired A:M:RET A:F:RET

Then the output table would preserve the 'Student' row

Male Female
Employed 12 13
Student
Retired 3 5

This rule can be coupled with [[Publication_Table#Formatting_Missing_Observations|formatting for missing observations]], then the table could be enhanced to look something like this

Male Female
Employed 12 13
Student - -
Retired 3 5

==== Anchor Child Rows [anchor+] ====
For a row with a hierarchy, an '''[anchor+]''' tag can be placed on a parent node to anchor the parent and all of its children, for example:

Continent
[anchor+]>Europe
>>France
>>>Total (M+F)
>>>>Male
>>>>Female
>>Germany
>>>Total (M+F)
>>>>Male
>>>>Female

The above rule would always output Europe and all of its descendants (France, Total, Male, Female, Germany).

==== Anchor Siblings and Child Rows [anchor*]====
The '''[anchor*]''' tag can be used as a way to anchor all following siblings and child nodes, this can be a useful mechanism to anchor all rows for the whole table by placing the anchor[*] on the first row of the table.

==== Anchor Loops ====
If a loop is preceeded by an '''[anchor]''' tag then the system will have the following behaviour:

* If an external Codelist variable is being looped, all codes will be ouput regardless of what data is stored in the system
* If a Dimension is being looped, all codes in the Codelist used by the Dimension will be output regardless of what data is stored in the system
* If a Dimension is being looped and it is not coded (i.e it is free text or numerical or other) then the [anchor] has no effect, as the looped values are based on what data exists

== Observation Key ==
=== Overview ===
The purpose of the Observation Key is to reference a Observation whose value will be resolved at table build time. The Observation Key can explicitly resolve to 1 observation value, if all parts of the Key are provided. However, it is possible to make parts of the key varaibles, allowing the values to be set at table build time.

An Observation Key is made up of 4 parts:

* Dataflow Reference
* Series Reference
* Time Period Reference
* Measure Reference

An example Observation Key with all of these parameters in place is:

DF_REG.M:UK:SUR:2008:BIRTHS

The key is broken down into the following parts:

{| class="wikitable"
|-
! Observation Key Part !! Name !! Description !! Required
|-
| DF_REG|| Dataflow Alias|| This identifies the Dataflow which has the data. The identification uses the alias, which is assiged to the Dataflow in the Publication Table definition. The Dataflow Alias is only required if the Publication Table references more then one Dataflow. If the Publication Table references more then one Dataflow, it can assign a Dataflow to a Default Alias, which means that it is required as part of the Observation Key. || No (if only 1 Dataflow or a default alias is set)
|-
| M:UK:SUR || Series Key || This identifies the Series of Data for the Dataflow, by providing a single value for each of the Dimension Members of the Dataflow, separated by a ‘:’ character. In this example the Dataflow has 3 Dimensions (excluding time), the value for the first is 'M', the second 'UK' and the last 'SUR'. As the Dataflow in this example has a Time Dimension this Series Key will uniquely identify a Series of Observations where each Observation refers to a unique Time Period || Yes (but can omit values)
|-
| 2008 || Time Period || This identifies the Time Period in the Series of Data identified by the previous section. The Combination of the Series and Time Period for a Time Series Dataflow will uniquely identify the Observation || No. Omit to make it variable for Time Series Data (defaults to last period). Not required if there is no Time Dimension for the DSD.
|-
| BIRTHS || Measure Dimension || If the Dataflow uses a Data Structure with multiple Measures, then the Measure ID must be included as the last part of the Observation Key. In this example the Dataflow has multiple measures (BIRTHS/DEATHS/MARRIAGES) so the Observation Key needs to include this value. || When there is only 1 measure it is not required, when there are multiple measures, it is required.
|}

Observation Keys are included on the materialised table in JSON format enabling the observation value to be tied back to the key. Observation Keys also provide a mechanism by which [[#Formatting_Cells|formatting rules]] can be applied to a subset of table cells.

=== Variable Dimension Members ===
It is possible to omit Dimension Values from the Observation Key, when this is the case that Dimension will become a variable – whose value can be set at table build time. For example, the second Dimension value of ‘UK’, from the previous example, can be omitted like this:

M::SUR:2008

The Publication Table can now be built for any Country by passing in the variable at table build time - i.e. UK, FR, DE. By passing in different variable values, the Observation Key will change, and as such it will reference a different Series in the Dataset.

When an Observation Key has a variable Dimension member, the ID of the Dimension can be referenced in text cells to create variable headings. For example, if the ID of the 2nd Dimension in the DSD is '''REF_AREA''', the Table Column could be given a title of $(REF_AREA) which would resolve to 'United Kingdom', 'France', or 'Germany' at build time depending on which varaible value is used to build the table. [[#Dimension Variables|#See Dimension Variables]].

=== Variable Time Period ===
For Time Series dataset, the Time Dimension can form part of the Observation Key, for example M:UK:SUR:'''2008'''. The 2008 part of the key refers to the year 2008.

It is possible to omit the value for Time Period in the Observation Key. Omitting Time Period will make the Time value a variable which can be provided at table build time, defaulting to the last period for which data exists if no value is supplied.

Extending the above example to omit the Time value results in an Observation Key whose Country and Time values can be provided to the web service at table build time.

M::SUR

'''Note:''' The above key is exactly the same as using the variable Time Period value 'P' in the Time Dimension, as follows:

M::SUR:'''P'''

Additional functions can be performed on the Time Dimension, such as moving forwards or backwards from the table base period, [[#Time Variables|#see Time Variables]] for more details.

=== Free Text Observation Keys ===
Whilst typically a cell in the body of the Publication Table would resolve to an Observation value, it is possible to use it for free text. To include free text cells, start the cell value with a leading apostrophe character, for example:

'Cell Text

'''Note''': An Observation Key can '''not''' contain a mix of free text as well as an Observation value

== Calculated Observation Cells ==
[[File:TableDefinition Calculations Variables.png|thumb|Publication Table with Calculated Cells and Variable Dimensions]]

An Observation Cell may contain calculated values, which can consist of a mix of simple algebra (+, -, *, /) and a reference to one or more Observations which may come from multiple Dataflows.

To define a Calculated Observation the Cell the following rules apply:

* The Observation Key must start with :=
* Any Observation Keys must appear within enclosing square brackets '[' and ']'
* An Observation key may embed calculations on a single Dimension by using parenthesis '(' and ')' in the key part
* The same rules for variable Dimensions and Time Period apply when resolving the Observation Keys used in the calculation

Examples are shown in the table below

{| class="wikitable"
|-
! Example Syntax !! Description
|-
| :=[A:UK:EMP]+10 || Add 10 to the observation value retrieved from the key A:UK:EMP
|-
| :=[A:(UK+FR+DE):EMP] || Add the series A:UK:EMP + A:FR:EMP + A:DE:EMP
|-
| :=[A:UK:EMP]+[A:FR:EMP]+[A:DE:EMP] || Add the series A:UK:EMP + A:FR:EMP + A:DE:EMP
|-
| :=[A:(UK+FR):EMP]/[A:(UK+FR):UMP] || Adds the series A:UK:EMP + A:FR:EMP and divides by the result of A:UK:UMP + A:FR:UMP
|-
| :=((100/[A:UK:EMP:(M+F):P-1])*[A:UK:EMP:(M+F)])-100 || Demonstrates the ability to use parenthesis outside of the Observation Keys to control the order of mathematical operations
|-
| :=[POP.M:UK:(M+F)] / [LAB.M:UK:EMP] || Demonstrates the ability to calculate data from two Dataflows (defined by alias POP and LAB)
|-
| :=[M:UK:EMP:M(P)] + [A:UK:EMP:A(P)] || Demonstrates the ability to calculate data where the period is converted from the base period
|}

=== Calculated Cell Observation Key ===
Calculated Cells have implicit [[#Observation_Key|Observation Keys]] which are derived based on the inputs series in the calculation. The [[#Observation_Key|Observation Key]] can be used to assign [[#Formatting_Cells|Formatting Rules]] in the same way as a 'standard key' can be used. It is possible to assign an explicit key to the calculation, which will override the implicit key assigned. The following table shows how the key assignment works.

For example
{| class="wikitable"
|-
! Calculation !! Output Key || Explanation
|-
|:=[A:(UK+FR+DE):EMP] || A:#:EMP:2009 || The first and last dimension have a fixed value, the second dimension has a variable value and is replaced by '#'. The time period is added to the end of the key.
|-
|:=((100/[A:UK:EMP:(M+F):P-1])*[A:UK:EMP:(M+F)])-100 || A:UK:EMP:#:# || The first three dimensions have a fixed value, the last dimension has a variable value and is replaced by '#'. The time period is a variable as there are 2 time inputs to the calculation (P and P-1)
|-
|:=PCTOF([A:M:UK:P],[A:TOT:UK:P]) || A:#:UK:2009 || The first and last dimension have a fixed value, the second dimension has a variable value and is replaced by '#'. The time period is added to the end of the key.
|-
|:=[A:UK:EMP]+10 || A:UK:EMP || The input key is the same as the output key as none of the dimensions have a variable value. The time period is added to the end of the key.
|-
|A:UK10:EMP:=[A:UK:EMP]+10 || A:UK10:EMP:2009 || The output key is explicitly assigned. The time period is added to the end of the key.
|}

The output key conforms to the same Dimension breakdown as the underlying Data Structure. If a Observation Key is explicitly assigned, it must also honour the same Dimension breakdown. This allows formatting rules to be applied using the key component syntax, for example COUNTRY=UK10 would search for Observation Keys whose Country Dimension is reporting the value UK10.

== Percentage Of ==
The PCTOF (Percentage Of) function can be used to calculate 'x' as a percentage of 'y' following the syntax:

PCTOF(x,y)

This is reolved to the following mathmatical expression: '''(100/y) * x'''

x and y must either be a numerical value, and observation key, or an expression using a combination of the two. For example

:=PCTOF([A:M:UK:P],[A:TOT:UK:P])

The Observation Key follows the same standard rules with respect to varaibles and inline calculations - the following example shows an Observation Key where Male + Female values are added with the result being used in the Percentage Of function

:=PCTOF([A:M:UK],[A:('''M+F'''):UK])

A more complex example shows calculations around the PCTOF function:
:=(10 + PCTOF([A:M:UK:A(P)-1],[A:(M+F):UK:A(P)])) / 2

== Footnotes ==

[[File:Footnotes3.png|thumb|A table with footnotes]]
[[File:Footnotes2.png|thumb|Footnotes in the table builder]]

=== Overview ===
Footnotes can be added to any part of the publication table (table column/row header, or observation cells). The output table is generated with a footnote number, with the list of footnotes at the end of the table.

Footnotes need to be thought about in 2 parts: Part 1 are the list of footnotes which are defined at the Publication Table level. Part 2 are the references to the footnotes from within the table. This separation of allows footnotes to be reused within the same table, and when working with [[#Footnotes_with_Placeholders|placeholders]] footnote templates can be set up where the text changes based on the placeholder values defined in the table builder.

To add a footnote, add it to the Publication Table, for example: 
Footnote 1 
Footnote 2

Then in the table builder, add a reference to which footnote to use, $(1) references '''Footnote 1''', and $(2) references '''Footnote 2'''.

'''Note:''' The order of the list of footnotes in the table definition may not be reflected in the order they are output. As such a footnote referenced by $(1) may be output as the second footnote.

=== Footnotes with Variable ===
A footnote can reference variables, using the [[#Table_Variables|variable syntax]],

A footnote for $(REF_AREA.UK)

A footnote can even output an observation value, observation attribute or series attribute.

Measure Value
A footnote with obs value $(A:UK:EMP:2008:OBS_VALUE)

Observation Attribute
A footnote with obs attribute $(A:UK:EMP:2008:OBS_COMMENT)

Series Attribute
A footnote with obs attribute $(A:UK:EMP:P:SERIES_TITLE)

When combining variable footnotes with placeholder values and loops, it is possible to dynamically create references to observation attributes.

'''Note:''' the use of P in the final cell key which is a reference to the [[#Time_Variables|time period]] for which the table is being built.

=== Footnotes with Placeholders ===
It is possible to write a generic footnote for the publication table, which includes placeholders. When the footnote is referenced, the placeholder values are provided. The syntax for a placeholder is {n} where n is a positive integer, for example {1}.

Footnote Example

A footnote for $(REF_AREA.{1}) and $(REF_AREA.{2})

In the table builder, the footnote is referenced as usual, but with arguments for the placeholders:

Footnote reference in Table builder

My Table Row '''$(1,UK,FR)'''

The footnote reference in the above example is in bold, and consists of which footnote text to use (in this case the first one in the list, denoted by '1'), and 2 arguments are passed for the placeholders, placeholder 1 takes the value of UK and placeholder 2 takes the value of FR. During the materialise process the intermediate text is generated:

A footnote for $(REF_AREA.UK) and $(REF_AREA.FR)

Which then goes through the variable resolve process, in this case both variables, '''$(REF_AREA.UK) and $(REF_AREA.FR)''', match the [[#Coded_Variables|Coded Variable]] syntax and would resolve to:

Final Output
A footnote for United Kingdom and France

=== Footnotes in Observation Cell ===
Any number of footnotes can appear in an Observation cell, simply put the footnote(s) at the end of the [[#Observation_Key|Observation Key]].

Standard Footnote Reference 
DF_REG.M:UK:SUR:2008:BIRTHS $(1) $(2)

Footnote with Placeholder 
DF_REG.M:UK:SUR:2008:BIRTHS $(1,UK)

=== Footnotes in Loops ===
A row [[#Looping_Rows|loop]] contains a locally scoped variable, and as such it can be used when calling a footnote with a placeholder.

Example Footnotes
$(A:T:{1}:EMP:P:SERIES_TITLE)
$(A:T:{1}:EMP:P:TITLE_COMPL)
$(A:{1}:{2}:EMP:P:OBS_COMMENT)

Example Table Definition
Employed Student
> :=LOOP(REF_AREA) '''$(1,$(REF_AREA[id]))'''
>> Total '''$(2,$(REF_AREA[id]))''' A:$(REF_AREA):T:EMP A:$(REF_AREA):T:STU
>> :=LOOP(SEX) '''$(3,$(REF_AREA[id]),$(SEX[id]))''' A:$(REF_AREA):$(SEX):EMP A:$(REF_AREA):$(SEX):STU

Footnote Intermediate Resolution
Row 1 (REF_AREA=UK)
$(A:T:UK:EMP:P:SERIES_TITLE)

Row 2 (REF_AREA=UK)
$(A:T:UK:EMP:P:TITLE_COMPL)

Row 3 (REF_AREA=UK, SEX=M)
$(A:UK:M:EMP:P:OBS_COMMENT)

Final Footnote output
Row 1 (REF_AREA=UK)
This is a series for the United Kingdom

Row 2 (REF_AREA=UK)
This is a title compliment for the United Kingdom

Row 3 (REF_AREA=UK, SEX=M)
This is an observation comment for Males in the United Kingdom

= Formatting Cells =
== Formatting Numerical Observation Values ==
[[File:Obs formatting.png|thumb|Number Formatting Options]]
Numerical observation values can be formatted in the following way

# '''Number of Decimal Places''' or '''Significant Figures''' can be defined
# For Decimal places this can be set to a maximum rule or a fixed rule, i.e the value 1.3 to 2 decimal places is 1.3 using the maximum rule and 1.30 using the fixed rule
# '''Rounding mode''' can be specified
# '''Scaling Factor''' this is a multiplier applied to the observation value, the multiplier is x10^[scaling factor]. Example, a value of 1.1 with a scaling factor of 1 is 11, a scaling factor of 2 is 110, a scaling factor of 3 is 1100. If the Observation or related Series has a UNIT_MULT concept the difference between the unit multiplier and scaling factor will be taken. Example, an Observation with a value of 1.1 and a unit multiplier of 9 (Billions) is reported, the publication table has a scaling factor of 6 (millions) - the number 1.1 is scaled using 10^3 (calculated by 9-6), the 1.1 billion is converted to 1100 million.
# '''Scientific notation''' can be requested on output, i.e 123456 to 2 significant figures is 120000 in standard form or 1.2E4 in scientific form
# '''Decimal formatting''' for long numbers, i.e. 12345.8 there is not thousands separator by default, to include a thousand separator set the decimal formatting rule to either a single Language (i.e. French Formatting) or Dynamic formatting, which uses the Accept-Langauge locale/language sent be the web browser (this can be overridden using the locale=[loc] query parameter. The formatting rules can also influence which characters the numerical characters used.
# Rules can be applied to all series, or specific series

=== Match Series ===
When creating a formatting rule, it is possible to define which series the rule applies to. If left blank, the rule will apply to all observation values.

The syntax for defining specific series is shown in the following table:

{| class="wikitable"
|-
! Rule Type !! Syntax !! Description
|-
| Series Match || =A:UK:EMP || Match a specific series from the default Dataflow
|-
| Series Match Multiple Values || =A:UK+FR+DE:EMP || Multiple codes in 2nd Dimension (UK, FR and DE) any can match for the rule to be valid
|-
| Series Match by Dataflow Alias || =FLOW2.A:UK:EMP || Match a specific series for the Dataflow with alias FLOW2
|-
| Wildcard Series Match || =A::EMP || Match all series with the given dimension values (2nd Dimension is wildcarded)
|-
| Not Rule || !=A:UK:EMP || Match all series which do not match this series (wildcard series also supported)
|-
| Dimension Rule || CURRENCY=GBP || Match all series which have CURRENCY of GBP
|-
| Not Dimension Rule || !CURRENCY=GBP || Match all series do not have a CURRENCY of GBP
|-
| Multiple Rules || CURRENCY=GBP,FREQ=M || Comma is used to include multiple matches (also applies to series match)
|}

== Formatting Missing Observations ==

Observation Values can be missing for 2 reasons:
# There is no Observation in the database for the given Observation Key
# There is an Observation in the database but the measure value is either: NaN, null, or an empty string

In both cases it is possible to define an output value using formatting rules. For the first case, the observation does not exist, it is possible to define a default output value, for example ‘-‘. For the second case, where the observation does exist, it is possible to define a set of rules based on another Component of the Observation, for example the Observation output may be related to the OBS_STATUS Attribute.

When the formatted value is based on a Component, a set of rules must be defined which state ‘when component=x output y’, where x may be an fixed value or a pattern to match (using a regular expression). The rules for mapping component values are defined in a [https://fmrwiki.sdmxcloud.org/Representation_Map Representation Map], and the Publication Table simply provides the connection to the Representation Map to use. This enables reuse of the mapping rules across multiple Publication Tables. The Publication Table must define two properties in order to support Component Mapping rules:

# The Component to obtain the value from i.e OBS_STATUS
# A link to the [https://fmrwiki.sdmxcloud.org/Representation_Map Representation Map], which holds the mapping rules. Note, the Representation Map is expected to have 1 source mapping, this could be of any type (string, codelist, other)

An example Reprsentation Map for OBS_STATUS is shown below:

{| class="wikitable"
|-
! OBS_STATUS !! Is Reg Ex !! Missing Value !! Description
|-
| M || false || mv || When OBS_STATUS is M output mv
|-
| L || false || np || When OBS_STATUS is L output np
|-
| .* || true || - || When OBS_STATUS is anything else output -
|}

== Frequency Formatting ==
[[File:PublicationTable FreqMap.png|thumb|Representation Map with frequency mapping rules]]
[[File:PublicationTable Materialised FreqMap.png|thumb|Materialised table with formatted time headings ]]
Time Variables always resolve themselves to an [[SDMX_Time_Formats|SDMX Date Format]] when forming the [[#Observation Key|Observation Key]], this ensures the correct observation value is returned from the data store. When a time variable is used as a label, in a table heading for example, the same formatting will be applied by default but a custom format can be provided. Custom display formats for a Time Period enable a time period such as 2007-Q1 to be resolved to custom text string, for example '31 Mar 2007'.

Custom Time formats rules are defined by providing an association between an SDMX Frequency ID (#see [[SDMX_Time_Formats|SDMX Date Format]]) and the output format template. The output format template is a string that defines which components of time are output an in which order and style. For example to output a date as 'March 2001' the format template would look like this 'MMMM yyyy' to indicate a full month name (MMMM) and a 4 character year (yyyy). format rules make use of the [https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html Simple Date Format] which is used by Java as such all of the documentation on Simple Date Format applies.

Frequency Format Mapping is not defined as part of a Publication Table, instead the rules are defined in a [https://fmrwiki.sdmxcloud.org/Representation_Map Representation Map] and the Publication Table references the Representation Map as the source of its frequency conversion rules. This enables the same set of rules to be used for multiple Publication Tables. The Representation Map must map 1 source (Codelist or String) to 1 target (String), it can contain as many mapped frequencies as required. If the Representation Map contains a mapping for a value that is not a valid SDMX Frequency, then it will be ignored.

It is important to note that all time periods resolve to a single point in time, which is the end of the period, for example an Annual period of 2008 resolves to 2008-12-31:T23:59:59. The resolution of frequency to a point in time enables the frequency formatting rules to use all aspects of time, even when converting a low frequency, for example 2008 formatted using the pattern dd MMM yy would resolve to '31 Dec 2008'.

== Variables Summary ==

{| class="wikitable"
|-
! Variable Type !! Description !! Syntax !! Example !! Example Output
|-
| '''Dataflow Name''' || Displays the name of a Dataflow referenced by the Publication Table in the locale in which the Publication Table is built || $([dataflow alias]) || $(EDU) || World Bank, Education
|-
| '''Dataflow Description''' || Displays the description of a Dataflow referenced by the Publication Table in the locale in which the Publication Table is built || $([dataflow alias].desc) || $(EDU.desc) || World Bank, Education
|-
| '''Dataflow Id''' || Displays the ID of a Dataflow referenced by the Publication Table || $([dataflow alias].id) || $(EDU.id) || DF_EDU
|-
| '''Variable Dimension Value''' || Displays the name of the variable used when an Observation key has a wildcard member || $([dimension id]) || $(REF_AREA) || United Kingdom
|-
| '''Specific Component Value''' || Displays the name of the Component Value || $([component id].[component value]) || $(OBS_STATUS.C) || Confidential
|-
| '''Component Name''' || Displays the name of the DSD Component (this is the name of the Concept used by the Component) in the locale in which the Publication Table is built || $([component id].name) || $(OBS_STATUS) || Confidentiality Status
|-
| ''' Enumeration Value ''' || Displays the name of a Code in a Codelist or Value in a Valuelist || $([list alias].[list item ID] || $(SEX.M) || Male
|-
| '''Conditional Variable''' || Displays the variable value, which is conditional on the value of a variable Dimension || $([conditional variable id][variable id]) || $(BANKNOTES[NOTE1]) || £5
|-
| ''' Time ''' || Displays the base time period for which the table is built, or an offset from this time period / cast to a different frequency || see [[#Time Functions|Time Functions]] || $(P) || 2008
|}

= Table Variables =
== Overview ==
Variables are a very important part of Publication Tables. They can be used to make the content of the table dynamic, setting and changing the value at run time to change the table content. Variables can be used in Observation Keys, but the values of variables can also be used anywhere where there is text, for example the table heading, subheading, column or row headings, footnotes, and Observation Cells with free text.

The general syntax for a variable is

$(''variable id''.''sub part''[''detail''])

Where variable id is always required, the subpart and detail are optional. Subpart is used if the variable resolves to a Component and a coded value is required, the detail array part obtains the id, name, or description of the underlying SDMX type.

This section describes the various variables that exist, along with the syntax of how to use them.

== Dataflow Variables ==
Publication Tables define data cells across one or more Dataflows. Each ‘imported’ Dataflow is given an alias, which can be used to refer to the Dataflow in the [[#Observation Key|Observation Key]].

It is possible to refer to the Dataflow's Name, Description, and Id using the following variable syntax.

'''Default Dataflow''' 
Dataflow Id
$([id])

Dataflow Name
$() or $([name))

Dataflow Description
$([desc])

'''Dataflow with Alias=EXR''' 
Dataflow Id
$(EXR[id])

Dataflow Name
$(EXR) or $(EXR[name))

Dataflow Description
$(EXR[desc])

== Dimension Variables ==
When an [[#Observation Key|Observation Key]] has [[#Variable Dimension Members|variable Dimension Values]], the table will be built by replacing this variable with a value to complete the Observation Key. The variable value is either passed explicitly to the web service, or defaulted by the system if not provided. The Dimension Variable can be referenced in the table using the syntax $([dimension id]), for example:

Dimension Value Name
$(INDICATOR) resolves to 'Employment'

This would resolve to the '''name''' of the Indicator value used in the table, for example '''EMP''' may resolve to the label '''Employment'''. The label for the Dimension Value is obtained from the Codelist or Valuelist which the Dimension uses. If the Dimension does not use a list (i.e. it is free text or numerical) then the Dimension value is simply used in place of the variable placeholder.

To resolve to the '''id''' of the value for the Component, for example '''EMP''', use the following syntax

Dimension Value Id
$(INDICATOR[id]) resolves to 'EMP'

== Dependent Dimension Variable ==
It is possible that the value for one Dimension is dependent on the value of another Dimension, for example CURRENCY may be dependent on the COUNTRY and COUNTRY is a variable in the [[#Observation Key|Observation Key]] - it is possible to specify that the value to use for the CURRENCY Dimension can be derived using a lookup table. The lookup table in this instance is maintained as an [https://fmrwiki.sdmxcloud.org/Representation_Map SDMX Representation Map] which is used to map COUNTRY to CURRENCY. The Publication Table specifies that the two Dimensions are related and provides a reference to the Representation Map to use which has the rules.

When working with a dependent variable, both the dependent variable and the Dimension on which it depends should be omitted from the [[#Observation Key|Observation Key]] - for example consider a full key of:

A:UK:GBP:2007

Where UK is the COUNTRY and GBP is the Currency, to make the COUNTRY a variable Dimension, it should be omitted from the key, as follows:

A::GBP:2007

To then make the CURRENCY a variable it is also omitted , as follows:

A:::2007

The Publication Table is given the rule that CURRENCY depends on COUNTRY and should use the COUNTRY_2_CURRENCY [https://fmrwiki.sdmxcloud.org/Representation_Map Representation Map], which in this example looks like the following:

{| class="wikitable"
|-
! COUNTRY !! Is Reg Ex !! CURRENCY
|-
| UK || false || GBP
|-
| US || false || USD
|-
| .* || true || EUR
|}

When the Publication Table is materialised the [[Publication_Table_Web_Service|web service]] only requires the COUNTRY information, and it will complete the Observation key using the representation map lookup table.

== Component Values ==
Components of the underlying Data Structure can be explicitly referenced using the following syntax '''$([component id])''', for example:

Component Name
$(OBS_STATUS) or $(OBS_STATUS[name]) resolves to 'Observation Status'

Component Description
$(OBS_STATUS[desc]) resolves to 'Information on the quality of a value or an unusual or missing value.'

== Coded Variables ==
To reference a Code value, the syntax is $([component id / codelist alias].[item id]). The first variable part is the ID of a Component in the underlying Data Structure, or alternativly the alias of an external variable defined on the Publication Table. An external variable is a link to an SDMX Codelists or Valuelist. Example are:

Name of Code
$(CURRENCY.$) resolving to the text '''Dollar''

Description of Code
$(DWELLING.SM[desc])

== Conditional Variables ==
A Conditional Variable is one whose value is conditional on the value of a [[#Dimension Variables|Dimension Variable]].

An example use case is Banknotes, where the values for NOTE1, NOTE2, or NOTE3 is dependant on the Country Dimension. A Conditional Variable could be used to set a values for each variable (NOTE1, NOTE2, NOTE3) based on the value used for the current Country Dimension i.e. US NOTE1=$5, UK NOTE1=£5.

The syntax of referring to a Conditional Variable is $([conditional variable id][variable id]), for example
$(BANKNOTE[NOTE1])

== Reported Attribute for an Observation or Series ==
A variable can be a reference to a part of an Observation, for example the Unit Multiplier (UNIT_MULT). A Table Row header, for example, could include variable text such as 'Number of payment accounts (''thousands'')' where the word '''thousands''' is a variable because it depends on other variables which build the key.

To achieve this, use the syntax:

$(A:UK::2008:UNIT_MULT[name])

The observation key in this case is A:UK::2008, where the third Dimension is a variable as it has no value (in the main body of the table this dimension would probably also be a variable, which means the table can be built with different values, each value may change the output of this UNIT_MULT value). The last part of the key is the Attribute ID (UNIT_MULT) - this can be a reference to a series attribute, or an observation attribute. The [name] indicates that it should resolve to the name of the reported attribute value, alternative values are '''id''' for the reported value or '''desc''' for the description of the reported value.

If time is also a variable, then the key can change accordingly, using the variable syntax for time (P):

$(A:UK::P:UNIT_MULT[name])

== Time Variables ==
[[File:PublicationTable MovingTime.png|thumb|Publication Table Design, moving and converting the base time period]]

[[File:PublicationTable Materialised MovingTime.png|thumb|Moving and converting periods - showing the materialised table]]

When the data is time series (i.e. the [[Data_Structure_Definition_(DSD)_-_Structural_Metadata_Management|DSD]] has a Time Dimension), the [[#Observation_Key|Observation Key]] can include the time period, for example:

A:UK:EMP:'''2008'''

Where 2008 is the Observation Time Period to use. '''The format of the Time Period must always be the same format as the frequency for which the data are reported'''. For example if the data is annual, and an observation is reported for 2008, then the observation key must use 2008, not another frequency such as 2008-12-31. It is typical in a [[Data_Structure_Definition_(DSD)_-_Structural_Metadata_Management|DSD]] for the Frequency of the Series to be the first Dimension, in all the examples on this page this convention is applied, and as such the above key A:UK:EMP refers to Annual data, depicted by the initial dimension value 'A'.

If the key omits the time period, then the '''time period becomes a variable''' which can be set at table build time, if not set it defaults to the last period for which there is data.

A:UK:EMP

When a Publication Table has a variable time period, the resolved variable time value for this varaible is known as the '''base period'''. There can only be 1 base period for a Publication Table, however other Observation Keys (and corresponding headings) can perform time arithmetic from the base period for which the table is built.

The supported time functions include walking forwards and backwards in increments of the given frequency, converting to another frequency of data, and a combination of the two (convert and move, move and convert, move convert and move again). Ths supported functions are shown in the table below.

{| class="wikitable"
|-
! Syntax !! Description !! Example Observation Key !! Example Resolved Key
|-
| P || The base period (default if not provided) || A:UK:EMP:P || A:UK:EMP:2008 (base period=2008)
|-
| P-''n'' || Base Period minus 'n' Periods where n is a positive Integer || A:UK:EMP:P-1 || A:UK:EMP:2007 (base period=2008)
|-
| P+''n'' || Base Period plus 'n' Periods where n is a positive Integer || A:UK:EMP:P+1 || A:UK:EMP:2008-Q2 (base period=2008-Q1)
|-
| A(P) || Base Period converted to '''Annual''' || A:UK:EMP:A(P) || A:UK:EMP:2008 (base period=2008-Q2)
|-
| S(P) || Base Period converted to '''Bi-Annual''' (Semester) || S:UK:EMP:S(P) || S:UK:EMP:2008-S2 (base period=2008-Q3)
|-
| T(P) || Base Period converted to '''Tri-Annual''' || T:UK:EMP:T(P) || T:UK:EMP:2008-T3 (base period=2008)
|-
| Q(P) || Base Period converted to '''Quarterly''' || Q:UK:EMP:Q(P) || Q:UK:EMP:2008-Q4 (base period=2008)
|-
| M(P) || Base Period converted to '''Monthly''' || M:UK:EMP:M(P) || M:UK:EMP:2008-03 (base period=2008-Q1)
|-
| W(P) || Base Period converted to '''Weekly''' || W:UK:EMP:W(P) || W:UK:EMP:2008-04 (base period=2008-01-01)
|-
| D(P) || Base Period converted to '''Daily''' || D:UK:EMP:D(P) || D:UK:EMP:2008-12-31 (base period=2008)
|-
| H(P) || Base Period converted to '''Hourly''' || H:UK:EMP:D(P) || H:UK:EMP:2008-12-31T12 (base period=2008-02-02T12:00:03)
|-
| A(P-n) || Base Period minus (or plus) 'n' periods, converted to another frequency 'A' || A:UK:EMP:A(P-4) || A:UK:EMP:2007 (base period=2008-Q4)
|-
| A(P)-n || Base Period, converted to another frequency 'A' minus (or plus) n' periods || A:UK:EMP:A(P)-4 || A:UK:EMP:2004 (base period=2008-Q4)
|-
| A(P-n1)-n2 || Base Period, minus (or plus) 'n1' periods, converted to another frequency 'A' minus (or plus) 'n2' periods || A:UK:EMP:A(P-4)-4 || A:UK:EMP:2003 (base period=2008-Q4)
|-
| FA(P) || Subtracts 1 Period from the Converted Period IF the resolved Convert From end of period does not match the same point in time as the Convert To period.

'''Examples:'''
* Base Period =2008-Q4 Rule= FA(P) Output = 2008
* Base Period =2008-Q3 Rule= FA(P) Output = 2007 - the base period 2008-Q3 is not the same time as 2008 (end of period, so move back 1 year)
* Base Period =2008-Q1 Rule= FA(P) Output = 2007
* Base Period =2008-Q1 Rule= FA(P+3) Output = 2008
* Base Period =2008-Q1 Rule= FA(P+3)-1 Output = 2007

This works for ALL frequencies where the convert to frequency is lower then the base Period P, for example Quarter converted to Annual.

|| A:UK:EMP:FA(P) || A:UK:EMP:2007 (base period=2008-Q3)
|}

'''Compound expressions''' 
From Fusion Registry v11.5.1 and later it is possible to cast a period more the once in the same expression, for example

Q(FA(P-1))-1

If the input time period P=2019-Q4, then P-1 = 2019-Q3, the expression is rewritten as Q(FA(2019-Q3))-1. The Full Annual (FA) for 2019-Q3 is 2018, as the end of period 2019 does not match the end of period 2019-Q3, so the annual period moves back an additional period. The expression is rewritten as Q(2018)-1. 2018 to Quarterly moves to 2018-Q4 and then the last part of the expression steps back 1 period, the final period is 2018-Q3.

'''Notes on Time Period''' 
When converting from a lower frequency to a higher frequency (Weekly to Daily for example), the cast date always resolves to the end of the Period.

It is also important to note that as the [[#Observation Key|Observation Keys]] in these examples all have Frequency as the first Dimension, when converting to another frequency (Quarterly to Annual for example) the Observation Key changes the value for the Frequency Dimension, ensuring the correct series of data is retrieved.

'''Time Period variable in free text''' 
When refering to the Time Period as a variable in free text, for example in the table heading or a column heading, the variable is placed inside the variable syntax '''$()'''. For example $(P) would resolve to base period, and $(A(P)) would resolve to 'base period converted to annual frequency'. An example usage is shown below:

Table 1A showing data for $(P) to $(P+3)

Publication Table

2023-01-24T09:46:12Z

Mnelson: /* Calculated Observation Cells */

[[Category:Functions]]

@See [[Publication_Table_Web_Services|Publication Table Web Services]]

= Publication Tables =
'''Introduced to Fusion Registry Enterprise v11.0.0.17'''

== Overview ==
[[File:Publication Table Definition.png|thumb|Using SDMX concepts of Agency, Id, Name, Description]]

[[File:Publication Table.png|thumb|Table Layout and Content Design]]

[[File:MaterialisedTable Preview.png|thumb|Showing the Table Preview feature which materialises the table with the given variables]]

A Publication Table is a curated table of data, where the values in the table rows, columns and table body cells can be individually specified. This enables a single table to pull data from multiple sources and piece the data together in a way which is not possible when dealing with standard cubes of data, whose table model is typically based on a Pivot model, with columns and rows built around the Dimension values to which the column or row is assigned. A Publication Table breaks out of this restrictive model by allowing cells of data to reference observations which do not need to share any relationship, in terms of dimensionality or even data source, with adjacent cells,.

Data in a Publication Table may be sourced from a single [[Dataflow_-_Structural_Metadata_Management|Dataflow]] or it may be sourced from multiple Dataflows enabling cross comparison of data.

Publication Tables can be static (every cell is a defined Observation) or dynamic, where a cell includes [[#Variable Dimension Members|variables Dimensions]] in the [[#Observation Key|Observation Key]], which can be provided at [[#Metadata vs Materialised|table build time]].

[[#Time Variables|Time arithmetic and frequency conversion]] is supported, enabling a single table to pull data from different time periods and frequencies which are all derived from the base period on which the table is built.

Publication Tables support [[#Calculated Observation Cells|calculations]], where each calculation can take input from one or more Observations over one or more [[Dataflow_-_Structural_Metadata_Management|Dataflows]].

Variable values also play a big role in Publication Tables, and includes standard variable placeholders – as well as advanced [[#Conditional Variables|conditional variables]], whose list of variable values change based on the value of another Dimension for which the table is built.

== Metadata vs Materialised ==

A Publication Table is stored in Fusion Registry the same way any other piece of structural metadata is stored. In this way it has an owner (Agency), and identity (ID) and a version. The change history is tracked using the same services, and the table definition can be [[Publication_Table_Web_Services|queried]] using the same SDMX web services for Structural Metadata.

The Publication Table metadata contains information about which Dataflows are used by the Table, which additional variables are being imported, and a definition of the table structure, and which Observations to import by referencing them by their unique [[#Observation Key|Observation Key]].

A Publication Table definition can be '''materialised''' into a table of Data by from the [[Publication_Table_Web_Services|Publication Table Web Service]], which combines the information of the Table Identity, and Variable Values to rewrite the [[#Observation Key|Observation Key]] into actual observation values. If building a web User Interface, then response from this service can be a packet of JSON, which describes the layout and content of the final Table in a way which is quick and simple to render for the User to see. The process of materialising the table is known as ‘'''build time'''’.

Additional web services are provided to enable clients to ask questions about the table such as ‘what are the earliest and latest time periods for which there is data for this table’ or ‘what possible variable values are there for this table’. This enables user interfaces to provide a user with the option to change the variables ‘on the fly’ ensuing they are able to build a table which is known to have data.

= Publication Table Cell Content =
== Overview ==
[[File:PublicationTable Layout.png|thumb|Publication Table Layout]]
A Publication Table is made up of a '''Table Head Rows''' and '''Table Body Rows'''.

The Table Head is made up of 1 or more table rows. Each Table Head cell may span 1 or more columns and 1 or more rows. The Table Head content is made up of free text and / or variable placeholders whose values are resolved when the table is built. Table Head Cells can be left empty.

The Table Body is made up of 1 or more Table Rows. The Table Rows provide the Publication Table with its data, by referencing Observations from the Dataflow.

At table build time all variables are resolved, and the corresponding data are retrieved from the data stores which hold the data.

== Table Rows ==
=== Overview ===
A Table Row consist of a Table Row heading, this can be a mix of free text and [[#Table Variables|variable text]]. The Table Row Heading is always the first Table Row Cell, and can not span multiple columns or rows. A Table Row can not have more then one heading. The remaining Table Row Cells are the Observation Cells, which may contain free text and [[#Table Variables|variable text]] but will typically contain a reference to an Observation by an [[#Observation Key|Obsevation Keys]] or a [[#Calculated Observation Cells|Calculation]] which can combine multiple [[#Observation Key|Obsevation Keys]].

A Table Row which results in no Observation data will not be included in materialised table at build time, unless the Table Row is a grouping row (see [[#Row Levels|Row Levels]] for details on grouping rows).

=== Row Levels ===
[[File:PublicationTabel Levels.png|thumb|Showing Levels in the Table Rows]]
The Table Row Heading can be used to define a row level. The level indicates to the client responsible for rendering the table the row's position in a hierarchy. Row levels are depicted using one or more leading chevrons ‘>‘ before the text. The number of hyphens indicates the level, with one leading hypen indicating level 1, two indicating level 2, and so on. It is possible to skip levels if required, i.e. Row 1 is level 1 with Row 2 assigned to level 3, skipping out level 2 entirely.

An example of row headings using levels is as follows:

Countries
> Europe
>> France
>> Germany
> Asia
>> China

Levels can be an important way to '''group data'''. A Table Row which acts as a parent row in a hierarchy, can be used as grouping mechanism for the rows that follow. This enables Parent Rows to be included in the output table, even if the Table Row contains no Observation data. The default behaviour for a Table Row with no data is to exclude it from the built table.

=== Looping Rows ===

Instead of explicitly defining a row for each Dimension member, it is possible to define a row as a loop (to iterate over all, or a subset of Dimension members). For example, if a publication table contains a Dimension for countries with an ID of REF_AREA then a loop can be set up to loop through all REF_AREA values, or a subset of the values - for which there is data. If the values are backed by a Codelist which has a hierarchy, the hierarchy can be preserved in the final table. It is also possible to nest a loop within another loop, for example for each REF_AREA output each INDICATOR.

====Syntax====
The loop is defined by giving the table row a label which starts with a syntax of :=LOOP([args]), like in the following example
:=LOOP([dimension id], [preserve hierarchy], [loop order], [explicit values to include])

'''Argument Definition'''
''dimension id'' = Required. The ID of the Dimension to loop over. For example CURRENCY would loop over all the Currency values which have data in the given context. The row label is the Currency, for example USD.
''preserve hierarchy'' = Optional, default false. If true the hierarchy of the Codes in the underlying Codelist will be built into the row hierarchy
''loop order'' = Optional, default 1. 1=Natural Order (as they appear in the Codelist), 2=By Code ID, 3=By Code Label
''explicit values to include'' = Optional, default include all values. A var arg list of values to loop over.

'''Full Example'''

:=LOOP(REF_AREA, true, 3, EUR,FR,DE,UK,ES)

====Simple Loop====
The loop is defined by giving the table row a label which starts with a syntax of :=LOOP([args]), like in the following example
:=LOOP(REF_AREA)

This loop definition will loop through all the REF_AREA values, building a row for each value. The Observation keys for the row can refer to the loop variable using a $(DIM_ID) placeholder in the Observation Key, for example:

Employed Student
:=LOOP(REF_AREA) A:$(REF_AREA):EMP A:$(REF_AREA):STU

The output table would look like the following:

Employed Student
United Kingdom 20 12
France 22 11
Germany 25 15

====Nested Loop====
If a loop row is followed by a row (or rows) which is an extra level deep, the child row(s) will be included for each loop, for example

Employed Student
> :=LOOP(REF_AREA)
>> Total A:$(REF_AREA):'''T''':EMP A:$(REF_AREA):'''T''':STU
>>> Male A:$(REF_AREA):'''M''':EMP A:$(REF_AREA):'''M''':STU
>>> Female A:$(REF_AREA):'''F''':EMP A:$(REF_AREA):'''F''':STU

Because the '''Total''' row is a child row of the LOOP (denoted by the number of leading '>') and '''Male'''/'''Female''' are children of '''Total''' - for each REF_AREA there will be a Total, Male and Female, outputting a table like the following:

Employed Student
United Kingdom
Total
Male 5 1
Female 4 2
France
Total
Male 6 2
Female 7 3

It is also possible to define a subloop as a loop. The previous example can be rewritten as:

Employed Student
> :=LOOP(REF_AREA)
>> :=LOOP(SEX) A:$(REF_AREA):'''$(SEX)''':EMP A:$(REF_AREA):'''$(SEX)''':STU

There is no limit to the number of subloops that can be defined.

====Loop Hierarchical Values====
If the underlying Codelist has a hierarchy, a subsequent argument can be provided to ask for the hierarchy to be preserved

Countries
> :=LOOP(REF_AREA, true)

'''Note:''' A hierarchy will keep rows with no data if they are acting as a parent node for rows that do have data (e.g. there may not be data for Europe, but the row exists because there are child rows of France and Germany with data).

Nested loops can also preserve hierarchies for example:

:=LOOP(REF_AREA, true)
>:=LOOP(SEX, true)

With an example output (assuming no data exists for the hierarical parents (Continent, Europe)

Continent
Europe
France
Total (M+F)
Male
Female
Germany
Total (M+F)
Male
Female

====Order Loop Rows====
It is possible to order the labels by their natural order (that of the Codelist), by id or by label, using the numbers 1, 2, or 3 respectively. The following outputs the rows ordered by label.

Countries
> :=LOOP(REF_AREA, true, 3)

====Fixed Loop List====
It is also possible to define a subset of the values to loop over. For example, if REF_AREA has data for 100 countries, but only 3 are required in the table. In this scenario, the natural order is that of the provided list, not of the underlying Codelist.

Countries
> :=LOOP(REF_AREA, true, 3, UK, FR, DE)

====Loop over External Variable====
Instead of looping the values of the Dimension's Codelist, it is possible to loop another Codelist - the purpose for this is to support alternative labels (if the labels for the publication table need to be different from those of the Dimension members) - or a subset (as an alternative to passing the subset of values into the LOOP argument), or to define a hierarchy (if the Dimension's Codelist does not have a hierarchy, or an alternative is required).

In order to loop over an external Variable, the Publication Table must be linked to an external Codelist/Valuelist against an Alias. For example EXT=My Country Codelist with Hierarchy. The corresponding loop then loops the Codes in the external list, and stores these the corresponding Dimension ID variable, for example:

Countries
> '''REF_AREA''':=LOOP('''EXT''')

Noting that the loop is over the variable EXT which resolves to an external Codelist. The loop is assigned to the REF_AREA Dimension.

=== Anchor Rows ===
The default behaviour is for a row to be omitted if the row has no data and it has no child rows with data. However, if the desired result is to always output the row, then this can be achieved by using the [anchor] tag in the row header label.

Consider the following Publication Table definition:

Male Female
Employed A:M:EMP A:F:EMP
Student A:M:STU A:F:STU
Retired A:M:RET A:F:RET

If there are no data for Student, then the materialised table would look like this

Male Female
Employed 12 13
Retired 3 5

==== Anchor Single Row [anchor] ====
To anchor a single row, add the '''[anchor]''' tag at the start of the row

Male Female
Employed A:M:EMP A:F:EMP
[anchor]Student A:M:STU A:F:STU
Retired A:M:RET A:F:RET

Then the output table would preserve the 'Student' row

Male Female
Employed 12 13
Student
Retired 3 5

This rule can be coupled with [[Publication_Table#Formatting_Missing_Observations|formatting for missing observations]], then the table could be enhanced to look something like this

Male Female
Employed 12 13
Student - -
Retired 3 5

==== Anchor Child Rows [anchor+] ====
For a row with a hierarchy, an '''[anchor+]''' tag can be placed on a parent node to anchor the parent and all of its children, for example:

Continent
[anchor+]>Europe
>>France
>>>Total (M+F)
>>>>Male
>>>>Female
>>Germany
>>>Total (M+F)
>>>>Male
>>>>Female

The above rule would always output Europe and all of its descendants (France, Total, Male, Female, Germany).

==== Anchor Siblings and Child Rows [anchor*]====
The '''[anchor*]''' tag can be used as a way to anchor all following siblings and child nodes, this can be a useful mechanism to anchor all rows for the whole table by placing the anchor[*] on the first row of the table.

==== Anchor Loops ====
If a loop is preceeded by an '''[anchor]''' tag then the system will have the following behaviour:

* If an external Codelist variable is being looped, all codes will be ouput regardless of what data is stored in the system
* If a Dimension is being looped, all codes in the Codelist used by the Dimension will be output regardless of what data is stored in the system
* If a Dimension is being looped and it is not coded (i.e it is free text or numerical or other) then the [anchor] has no effect, as the looped values are based on what data exists

== Observation Key ==
=== Overview ===
The purpose of the Observation Key is to reference a Observation whose value will be resolved at table build time. The Observation Key can explicitly resolve to 1 observation value, if all parts of the Key are provided. However, it is possible to make parts of the key varaibles, allowing the values to be set at table build time.

An Observation Key is made up of 4 parts:

* Dataflow Reference
* Series Reference
* Time Period Reference
* Measure Reference

An example Observation Key with all of these parameters in place is:

DF_REG.M:UK:SUR:2008:BIRTHS

The key is broken down into the following parts:

{| class="wikitable"
|-
! Observation Key Part !! Name !! Description !! Required
|-
| DF_REG|| Dataflow Alias|| This identifies the Dataflow which has the data. The identification uses the alias, which is assiged to the Dataflow in the Publication Table definition. The Dataflow Alias is only required if the Publication Table references more then one Dataflow. If the Publication Table references more then one Dataflow, it can assign a Dataflow to a Default Alias, which means that it is required as part of the Observation Key. || No (if only 1 Dataflow or a default alias is set)
|-
| M:UK:SUR || Series Key || This identifies the Series of Data for the Dataflow, by providing a single value for each of the Dimension Members of the Dataflow, separated by a ‘:’ character. In this example the Dataflow has 3 Dimensions (excluding time), the value for the first is 'M', the second 'UK' and the last 'SUR'. As the Dataflow in this example has a Time Dimension this Series Key will uniquely identify a Series of Observations where each Observation refers to a unique Time Period || Yes (but can omit values)
|-
| 2008 || Time Period || This identifies the Time Period in the Series of Data identified by the previous section. The Combination of the Series and Time Period for a Time Series Dataflow will uniquely identify the Observation || No. Omit to make it variable for Time Series Data (defaults to last period). Not required if there is no Time Dimension for the DSD.
|-
| BIRTHS || Measure Dimension || If the Dataflow uses a Data Structure with multiple Measures, then the Measure ID must be included as the last part of the Observation Key. In this example the Dataflow has multiple measures (BIRTHS/DEATHS/MARRIAGES) so the Observation Key needs to include this value. || When there is only 1 measure it is not required, when there are multiple measures, it is required.
|}

Observation Keys are included on the materialised table in JSON format enabling the observation value to be tied back to the key. Observation Keys also provide a mechanism by which [[#Formatting_Cells|formatting rules]] can be applied to a subset of table cells.

=== Variable Dimension Members ===
It is possible to omit Dimension Values from the Observation Key, when this is the case that Dimension will become a variable – whose value can be set at table build time. For example, the second Dimension value of ‘UK’, from the previous example, can be omitted like this:

M::SUR:2008

The Publication Table can now be built for any Country by passing in the variable at table build time - i.e. UK, FR, DE. By passing in different variable values, the Observation Key will change, and as such it will reference a different Series in the Dataset.

When an Observation Key has a variable Dimension member, the ID of the Dimension can be referenced in text cells to create variable headings. For example, if the ID of the 2nd Dimension in the DSD is '''REF_AREA''', the Table Column could be given a title of $(REF_AREA) which would resolve to 'United Kingdom', 'France', or 'Germany' at build time depending on which varaible value is used to build the table. [[#Dimension Variables|#See Dimension Variables]].

=== Variable Time Period ===
For Time Series dataset, the Time Dimension can form part of the Observation Key, for example M:UK:SUR:'''2008'''. The 2008 part of the key refers to the year 2008.

It is possible to omit the value for Time Period in the Observation Key. Omitting Time Period will make the Time value a variable which can be provided at table build time, defaulting to the last period for which data exists if no value is supplied.

Extending the above example to omit the Time value results in an Observation Key whose Country and Time values can be provided to the web service at table build time.

M::SUR

'''Note:''' The above key is exactly the same as using the variable Time Period value 'P' in the Time Dimension, as follows:

M::SUR:'''P'''

Additional functions can be performed on the Time Dimension, such as moving forwards or backwards from the table base period, [[#Time Variables|#see Time Variables]] for more details.

=== Free Text Observation Keys ===
Whilst typically a cell in the body of the Publication Table would resolve to an Observation value, it is possible to use it for free text. To include free text cells, start the cell value with a leading apostrophe character, for example:

'Cell Text

'''Note''': An Observation Key can '''not''' contain a mix of free text as well as an Observation value

== Calculated Observation Cells ==
[[File:TableDefinition Calculations Variables.png|thumb|Publication Table with Calculated Cells and Variable Dimensions]]

An Observation Cell may contain calculated values, which can consist of a mix of simple algebra (+, -, *, /) and a reference to one or more Observations which may come from multiple Dataflows.

To define a Calculated Observation the Cell the following rules apply:

* The Observation Key must start with :=
* Any Observation Keys must appear within enclosing square brackets '[' and ']'
* An Observation key may embed calculations on a single Dimension by using parenthesis '(' and ')' in the key part
* The same rules for variable Dimensions and Time Period apply when resolving the Observation Keys used in the calculation

Examples are shown in the table below

{| class="wikitable"
|-
! Example Syntax !! Description
|-
| :=[A:UK:EMP]+10 || Add 10 to the observation value retrieved from the key A:UK:EMP
|-
| :=[A:(UK+FR+DE):EMP] || Add the series A:UK:EMP + A:FR:EMP + A:DE:EMP
|-
| :=[A:UK:EMP]+[A:FR:EMP]+[A:DE:EMP] || Add the series A:UK:EMP + A:FR:EMP + A:DE:EMP
|-
| :=[A:(UK+FR):EMP]/[A:(UK+FR):UMP] || Adds the series A:UK:EMP + A:FR:EMP and divides by the result of A:UK:UMP + A:FR:UMP
|-
| :=((100/[A:UK:EMP:(M+F):P-1])*[A:UK:EMP:(M+F)])-100 || Demonstrates the ability to use parenthesis outside of the Observation Keys to control the order of mathematical operations
|-
| :=[POP.M:UK:(M+F)] / [LAB.M:UK:EMP] || Demonstrates the ability to calculate data from two Dataflows (defined by alias POP and LAB)
|-
| :=[M:UK:EMP:M(P)] + [A:UK:EMP:A(P)] || Demonstrates the ability to calculate data where the period is converted from the base period
|}

=== Calculated Cell Observation Key ===
Calculated Cells have implicit [[#Observation_Key|Observation Keys]] which are derived based on the inputs series in the calculation. The [[#Observation_Key|Observation Key]] can be used to assign [[#Formatting_Cells|Formatting Rules]] in the same way as a 'standard key' can be used. It is possible to assign an explicit key to the calculation, which will override the implicit key assigned. The following table shows how the key assignment works.

For example
{| class="wikitable"
|-
! Calculation !! Output Key || Explanation
|-
|:=[A:(UK+FR+DE):EMP] || A:#:EMP:2009 || The first and last dimension have a fixed value, the second dimension has a variable value and is replaced by '#'. The time period is added to the end of the key.
|-
|:=((100/[A:UK:EMP:(M+F):P-1])*[A:UK:EMP:(M+F)])-100 || A:UK:EMP:#:# || The first three dimensions have a fixed value, the last dimension has a variable value and is replaced by '#'. The time period is a variable as there are 2 time inputs to the calculation (P and P-1)
|-
|:=PCTOF([A:M:UK:P],[A:TOT:UK:P]) || A:#:UK:2009 || The first and last dimension have a fixed value, the second dimension has a variable value and is replaced by '#'. The time period is added to the end of the key.
|-
|:=[A:UK:EMP]+10 || A:UK:EMP || The input key is the same as the output key as none of the dimensions have a variable value. The time period is added to the end of the key.
|-
|A:UK10:EMP:=[A:UK:EMP]+10 || A:UK10:EMP:2009 || The output key is explicitly assigned. The time period is added to the end of the key.
|}

== Percentage Of ==
The PCTOF (Percentage Of) function can be used to calculate 'x' as a percentage of 'y' following the syntax:

PCTOF(x,y)

This is reolved to the following mathmatical expression: '''(100/y) * x'''

x and y must either be a numerical value, and observation key, or an expression using a combination of the two. For example

:=PCTOF([A:M:UK:P],[A:TOT:UK:P])

The Observation Key follows the same standard rules with respect to varaibles and inline calculations - the following example shows an Observation Key where Male + Female values are added with the result being used in the Percentage Of function

:=PCTOF([A:M:UK],[A:('''M+F'''):UK])

A more complex example shows calculations around the PCTOF function:
:=(10 + PCTOF([A:M:UK:A(P)-1],[A:(M+F):UK:A(P)])) / 2

== Footnotes ==

[[File:Footnotes3.png|thumb|A table with footnotes]]
[[File:Footnotes2.png|thumb|Footnotes in the table builder]]

=== Overview ===
Footnotes can be added to any part of the publication table (table column/row header, or observation cells). The output table is generated with a footnote number, with the list of footnotes at the end of the table.

Footnotes need to be thought about in 2 parts: Part 1 are the list of footnotes which are defined at the Publication Table level. Part 2 are the references to the footnotes from within the table. This separation of allows footnotes to be reused within the same table, and when working with [[#Footnotes_with_Placeholders|placeholders]] footnote templates can be set up where the text changes based on the placeholder values defined in the table builder.

To add a footnote, add it to the Publication Table, for example: 
Footnote 1 
Footnote 2

Then in the table builder, add a reference to which footnote to use, $(1) references '''Footnote 1''', and $(2) references '''Footnote 2'''.

'''Note:''' The order of the list of footnotes in the table definition may not be reflected in the order they are output. As such a footnote referenced by $(1) may be output as the second footnote.

=== Footnotes with Variable ===
A footnote can reference variables, using the [[#Table_Variables|variable syntax]],

A footnote for $(REF_AREA.UK)

A footnote can even output an observation value, observation attribute or series attribute.

Measure Value
A footnote with obs value $(A:UK:EMP:2008:OBS_VALUE)

Observation Attribute
A footnote with obs attribute $(A:UK:EMP:2008:OBS_COMMENT)

Series Attribute
A footnote with obs attribute $(A:UK:EMP:P:SERIES_TITLE)

When combining variable footnotes with placeholder values and loops, it is possible to dynamically create references to observation attributes.

'''Note:''' the use of P in the final cell key which is a reference to the [[#Time_Variables|time period]] for which the table is being built.

=== Footnotes with Placeholders ===
It is possible to write a generic footnote for the publication table, which includes placeholders. When the footnote is referenced, the placeholder values are provided. The syntax for a placeholder is {n} where n is a positive integer, for example {1}.

Footnote Example

A footnote for $(REF_AREA.{1}) and $(REF_AREA.{2})

In the table builder, the footnote is referenced as usual, but with arguments for the placeholders:

Footnote reference in Table builder

My Table Row '''$(1,UK,FR)'''

The footnote reference in the above example is in bold, and consists of which footnote text to use (in this case the first one in the list, denoted by '1'), and 2 arguments are passed for the placeholders, placeholder 1 takes the value of UK and placeholder 2 takes the value of FR. During the materialise process the intermediate text is generated:

A footnote for $(REF_AREA.UK) and $(REF_AREA.FR)

Which then goes through the variable resolve process, in this case both variables, '''$(REF_AREA.UK) and $(REF_AREA.FR)''', match the [[#Coded_Variables|Coded Variable]] syntax and would resolve to:

Final Output
A footnote for United Kingdom and France

=== Footnotes in Observation Cell ===
Any number of footnotes can appear in an Observation cell, simply put the footnote(s) at the end of the [[#Observation_Key|Observation Key]].

Standard Footnote Reference 
DF_REG.M:UK:SUR:2008:BIRTHS $(1) $(2)

Footnote with Placeholder 
DF_REG.M:UK:SUR:2008:BIRTHS $(1,UK)

=== Footnotes in Loops ===
A row [[#Looping_Rows|loop]] contains a locally scoped variable, and as such it can be used when calling a footnote with a placeholder.

Example Footnotes
$(A:T:{1}:EMP:P:SERIES_TITLE)
$(A:T:{1}:EMP:P:TITLE_COMPL)
$(A:{1}:{2}:EMP:P:OBS_COMMENT)

Example Table Definition
Employed Student
> :=LOOP(REF_AREA) '''$(1,$(REF_AREA[id]))'''
>> Total '''$(2,$(REF_AREA[id]))''' A:$(REF_AREA):T:EMP A:$(REF_AREA):T:STU
>> :=LOOP(SEX) '''$(3,$(REF_AREA[id]),$(SEX[id]))''' A:$(REF_AREA):$(SEX):EMP A:$(REF_AREA):$(SEX):STU

Footnote Intermediate Resolution
Row 1 (REF_AREA=UK)
$(A:T:UK:EMP:P:SERIES_TITLE)

Row 2 (REF_AREA=UK)
$(A:T:UK:EMP:P:TITLE_COMPL)

Row 3 (REF_AREA=UK, SEX=M)
$(A:UK:M:EMP:P:OBS_COMMENT)

Final Footnote output
Row 1 (REF_AREA=UK)
This is a series for the United Kingdom

Row 2 (REF_AREA=UK)
This is a title compliment for the United Kingdom

Row 3 (REF_AREA=UK, SEX=M)
This is an observation comment for Males in the United Kingdom

= Formatting Cells =
== Formatting Numerical Observation Values ==
[[File:Obs formatting.png|thumb|Number Formatting Options]]
Numerical observation values can be formatted in the following way

# '''Number of Decimal Places''' or '''Significant Figures''' can be defined
# For Decimal places this can be set to a maximum rule or a fixed rule, i.e the value 1.3 to 2 decimal places is 1.3 using the maximum rule and 1.30 using the fixed rule
# '''Rounding mode''' can be specified
# '''Scaling Factor''' this is a multiplier applied to the observation value, the multiplier is x10^[scaling factor]. Example, a value of 1.1 with a scaling factor of 1 is 11, a scaling factor of 2 is 110, a scaling factor of 3 is 1100. If the Observation or related Series has a UNIT_MULT concept the difference between the unit multiplier and scaling factor will be taken. Example, an Observation with a value of 1.1 and a unit multiplier of 9 (Billions) is reported, the publication table has a scaling factor of 6 (millions) - the number 1.1 is scaled using 10^3 (calculated by 9-6), the 1.1 billion is converted to 1100 million.
# '''Scientific notation''' can be requested on output, i.e 123456 to 2 significant figures is 120000 in standard form or 1.2E4 in scientific form
# '''Decimal formatting''' for long numbers, i.e. 12345.8 there is not thousands separator by default, to include a thousand separator set the decimal formatting rule to either a single Language (i.e. French Formatting) or Dynamic formatting, which uses the Accept-Langauge locale/language sent be the web browser (this can be overridden using the locale=[loc] query parameter. The formatting rules can also influence which characters the numerical characters used.
# Rules can be applied to all series, or specific series

=== Match Series ===
When creating a formatting rule, it is possible to define which series the rule applies to. If left blank, the rule will apply to all observation values.

The syntax for defining specific series is shown in the following table:

{| class="wikitable"
|-
! Rule Type !! Syntax !! Description
|-
| Series Match || =A:UK:EMP || Match a specific series from the default Dataflow
|-
| Series Match Multiple Values || =A:UK+FR+DE:EMP || Multiple codes in 2nd Dimension (UK, FR and DE) any can match for the rule to be valid
|-
| Series Match by Dataflow Alias || =FLOW2.A:UK:EMP || Match a specific series for the Dataflow with alias FLOW2
|-
| Wildcard Series Match || =A::EMP || Match all series with the given dimension values (2nd Dimension is wildcarded)
|-
| Not Rule || !=A:UK:EMP || Match all series which do not match this series (wildcard series also supported)
|-
| Dimension Rule || CURRENCY=GBP || Match all series which have CURRENCY of GBP
|-
| Not Dimension Rule || !CURRENCY=GBP || Match all series do not have a CURRENCY of GBP
|-
| Multiple Rules || CURRENCY=GBP,FREQ=M || Comma is used to include multiple matches (also applies to series match)
|}

== Formatting Missing Observations ==

Observation Values can be missing for 2 reasons:
# There is no Observation in the database for the given Observation Key
# There is an Observation in the database but the measure value is either: NaN, null, or an empty string

In both cases it is possible to define an output value using formatting rules. For the first case, the observation does not exist, it is possible to define a default output value, for example ‘-‘. For the second case, where the observation does exist, it is possible to define a set of rules based on another Component of the Observation, for example the Observation output may be related to the OBS_STATUS Attribute.

When the formatted value is based on a Component, a set of rules must be defined which state ‘when component=x output y’, where x may be an fixed value or a pattern to match (using a regular expression). The rules for mapping component values are defined in a [https://fmrwiki.sdmxcloud.org/Representation_Map Representation Map], and the Publication Table simply provides the connection to the Representation Map to use. This enables reuse of the mapping rules across multiple Publication Tables. The Publication Table must define two properties in order to support Component Mapping rules:

# The Component to obtain the value from i.e OBS_STATUS
# A link to the [https://fmrwiki.sdmxcloud.org/Representation_Map Representation Map], which holds the mapping rules. Note, the Representation Map is expected to have 1 source mapping, this could be of any type (string, codelist, other)

An example Reprsentation Map for OBS_STATUS is shown below:

{| class="wikitable"
|-
! OBS_STATUS !! Is Reg Ex !! Missing Value !! Description
|-
| M || false || mv || When OBS_STATUS is M output mv
|-
| L || false || np || When OBS_STATUS is L output np
|-
| .* || true || - || When OBS_STATUS is anything else output -
|}

== Frequency Formatting ==
[[File:PublicationTable FreqMap.png|thumb|Representation Map with frequency mapping rules]]
[[File:PublicationTable Materialised FreqMap.png|thumb|Materialised table with formatted time headings ]]
Time Variables always resolve themselves to an [[SDMX_Time_Formats|SDMX Date Format]] when forming the [[#Observation Key|Observation Key]], this ensures the correct observation value is returned from the data store. When a time variable is used as a label, in a table heading for example, the same formatting will be applied by default but a custom format can be provided. Custom display formats for a Time Period enable a time period such as 2007-Q1 to be resolved to custom text string, for example '31 Mar 2007'.

Custom Time formats rules are defined by providing an association between an SDMX Frequency ID (#see [[SDMX_Time_Formats|SDMX Date Format]]) and the output format template. The output format template is a string that defines which components of time are output an in which order and style. For example to output a date as 'March 2001' the format template would look like this 'MMMM yyyy' to indicate a full month name (MMMM) and a 4 character year (yyyy). format rules make use of the [https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html Simple Date Format] which is used by Java as such all of the documentation on Simple Date Format applies.

Frequency Format Mapping is not defined as part of a Publication Table, instead the rules are defined in a [https://fmrwiki.sdmxcloud.org/Representation_Map Representation Map] and the Publication Table references the Representation Map as the source of its frequency conversion rules. This enables the same set of rules to be used for multiple Publication Tables. The Representation Map must map 1 source (Codelist or String) to 1 target (String), it can contain as many mapped frequencies as required. If the Representation Map contains a mapping for a value that is not a valid SDMX Frequency, then it will be ignored.

It is important to note that all time periods resolve to a single point in time, which is the end of the period, for example an Annual period of 2008 resolves to 2008-12-31:T23:59:59. The resolution of frequency to a point in time enables the frequency formatting rules to use all aspects of time, even when converting a low frequency, for example 2008 formatted using the pattern dd MMM yy would resolve to '31 Dec 2008'.

== Variables Summary ==

{| class="wikitable"
|-
! Variable Type !! Description !! Syntax !! Example !! Example Output
|-
| '''Dataflow Name''' || Displays the name of a Dataflow referenced by the Publication Table in the locale in which the Publication Table is built || $([dataflow alias]) || $(EDU) || World Bank, Education
|-
| '''Dataflow Description''' || Displays the description of a Dataflow referenced by the Publication Table in the locale in which the Publication Table is built || $([dataflow alias].desc) || $(EDU.desc) || World Bank, Education
|-
| '''Dataflow Id''' || Displays the ID of a Dataflow referenced by the Publication Table || $([dataflow alias].id) || $(EDU.id) || DF_EDU
|-
| '''Variable Dimension Value''' || Displays the name of the variable used when an Observation key has a wildcard member || $([dimension id]) || $(REF_AREA) || United Kingdom
|-
| '''Specific Component Value''' || Displays the name of the Component Value || $([component id].[component value]) || $(OBS_STATUS.C) || Confidential
|-
| '''Component Name''' || Displays the name of the DSD Component (this is the name of the Concept used by the Component) in the locale in which the Publication Table is built || $([component id].name) || $(OBS_STATUS) || Confidentiality Status
|-
| ''' Enumeration Value ''' || Displays the name of a Code in a Codelist or Value in a Valuelist || $([list alias].[list item ID] || $(SEX.M) || Male
|-
| '''Conditional Variable''' || Displays the variable value, which is conditional on the value of a variable Dimension || $([conditional variable id][variable id]) || $(BANKNOTES[NOTE1]) || £5
|-
| ''' Time ''' || Displays the base time period for which the table is built, or an offset from this time period / cast to a different frequency || see [[#Time Functions|Time Functions]] || $(P) || 2008
|}

= Table Variables =
== Overview ==
Variables are a very important part of Publication Tables. They can be used to make the content of the table dynamic, setting and changing the value at run time to change the table content. Variables can be used in Observation Keys, but the values of variables can also be used anywhere where there is text, for example the table heading, subheading, column or row headings, footnotes, and Observation Cells with free text.

The general syntax for a variable is

$(''variable id''.''sub part''[''detail''])

Where variable id is always required, the subpart and detail are optional. Subpart is used if the variable resolves to a Component and a coded value is required, the detail array part obtains the id, name, or description of the underlying SDMX type.

This section describes the various variables that exist, along with the syntax of how to use them.

== Dataflow Variables ==
Publication Tables define data cells across one or more Dataflows. Each ‘imported’ Dataflow is given an alias, which can be used to refer to the Dataflow in the [[#Observation Key|Observation Key]].

It is possible to refer to the Dataflow's Name, Description, and Id using the following variable syntax.

'''Default Dataflow''' 
Dataflow Id
$([id])

Dataflow Name
$() or $([name))

Dataflow Description
$([desc])

'''Dataflow with Alias=EXR''' 
Dataflow Id
$(EXR[id])

Dataflow Name
$(EXR) or $(EXR[name))

Dataflow Description
$(EXR[desc])

== Dimension Variables ==
When an [[#Observation Key|Observation Key]] has [[#Variable Dimension Members|variable Dimension Values]], the table will be built by replacing this variable with a value to complete the Observation Key. The variable value is either passed explicitly to the web service, or defaulted by the system if not provided. The Dimension Variable can be referenced in the table using the syntax $([dimension id]), for example:

Dimension Value Name
$(INDICATOR) resolves to 'Employment'

This would resolve to the '''name''' of the Indicator value used in the table, for example '''EMP''' may resolve to the label '''Employment'''. The label for the Dimension Value is obtained from the Codelist or Valuelist which the Dimension uses. If the Dimension does not use a list (i.e. it is free text or numerical) then the Dimension value is simply used in place of the variable placeholder.

To resolve to the '''id''' of the value for the Component, for example '''EMP''', use the following syntax

Dimension Value Id
$(INDICATOR[id]) resolves to 'EMP'

== Dependent Dimension Variable ==
It is possible that the value for one Dimension is dependent on the value of another Dimension, for example CURRENCY may be dependent on the COUNTRY and COUNTRY is a variable in the [[#Observation Key|Observation Key]] - it is possible to specify that the value to use for the CURRENCY Dimension can be derived using a lookup table. The lookup table in this instance is maintained as an [https://fmrwiki.sdmxcloud.org/Representation_Map SDMX Representation Map] which is used to map COUNTRY to CURRENCY. The Publication Table specifies that the two Dimensions are related and provides a reference to the Representation Map to use which has the rules.

When working with a dependent variable, both the dependent variable and the Dimension on which it depends should be omitted from the [[#Observation Key|Observation Key]] - for example consider a full key of:

A:UK:GBP:2007

Where UK is the COUNTRY and GBP is the Currency, to make the COUNTRY a variable Dimension, it should be omitted from the key, as follows:

A::GBP:2007

To then make the CURRENCY a variable it is also omitted , as follows:

A:::2007

The Publication Table is given the rule that CURRENCY depends on COUNTRY and should use the COUNTRY_2_CURRENCY [https://fmrwiki.sdmxcloud.org/Representation_Map Representation Map], which in this example looks like the following:

{| class="wikitable"
|-
! COUNTRY !! Is Reg Ex !! CURRENCY
|-
| UK || false || GBP
|-
| US || false || USD
|-
| .* || true || EUR
|}

When the Publication Table is materialised the [[Publication_Table_Web_Service|web service]] only requires the COUNTRY information, and it will complete the Observation key using the representation map lookup table.

== Component Values ==
Components of the underlying Data Structure can be explicitly referenced using the following syntax '''$([component id])''', for example:

Component Name
$(OBS_STATUS) or $(OBS_STATUS[name]) resolves to 'Observation Status'

Component Description
$(OBS_STATUS[desc]) resolves to 'Information on the quality of a value or an unusual or missing value.'

== Coded Variables ==
To reference a Code value, the syntax is $([component id / codelist alias].[item id]). The first variable part is the ID of a Component in the underlying Data Structure, or alternativly the alias of an external variable defined on the Publication Table. An external variable is a link to an SDMX Codelists or Valuelist. Example are:

Name of Code
$(CURRENCY.$) resolving to the text '''Dollar''

Description of Code
$(DWELLING.SM[desc])

== Conditional Variables ==
A Conditional Variable is one whose value is conditional on the value of a [[#Dimension Variables|Dimension Variable]].

An example use case is Banknotes, where the values for NOTE1, NOTE2, or NOTE3 is dependant on the Country Dimension. A Conditional Variable could be used to set a values for each variable (NOTE1, NOTE2, NOTE3) based on the value used for the current Country Dimension i.e. US NOTE1=$5, UK NOTE1=£5.

The syntax of referring to a Conditional Variable is $([conditional variable id][variable id]), for example
$(BANKNOTE[NOTE1])

== Reported Attribute for an Observation or Series ==
A variable can be a reference to a part of an Observation, for example the Unit Multiplier (UNIT_MULT). A Table Row header, for example, could include variable text such as 'Number of payment accounts (''thousands'')' where the word '''thousands''' is a variable because it depends on other variables which build the key.

To achieve this, use the syntax:

$(A:UK::2008:UNIT_MULT[name])

The observation key in this case is A:UK::2008, where the third Dimension is a variable as it has no value (in the main body of the table this dimension would probably also be a variable, which means the table can be built with different values, each value may change the output of this UNIT_MULT value). The last part of the key is the Attribute ID (UNIT_MULT) - this can be a reference to a series attribute, or an observation attribute. The [name] indicates that it should resolve to the name of the reported attribute value, alternative values are '''id''' for the reported value or '''desc''' for the description of the reported value.

If time is also a variable, then the key can change accordingly, using the variable syntax for time (P):

$(A:UK::P:UNIT_MULT[name])

== Time Variables ==
[[File:PublicationTable MovingTime.png|thumb|Publication Table Design, moving and converting the base time period]]

[[File:PublicationTable Materialised MovingTime.png|thumb|Moving and converting periods - showing the materialised table]]

When the data is time series (i.e. the [[Data_Structure_Definition_(DSD)_-_Structural_Metadata_Management|DSD]] has a Time Dimension), the [[#Observation_Key|Observation Key]] can include the time period, for example:

A:UK:EMP:'''2008'''

Where 2008 is the Observation Time Period to use. '''The format of the Time Period must always be the same format as the frequency for which the data are reported'''. For example if the data is annual, and an observation is reported for 2008, then the observation key must use 2008, not another frequency such as 2008-12-31. It is typical in a [[Data_Structure_Definition_(DSD)_-_Structural_Metadata_Management|DSD]] for the Frequency of the Series to be the first Dimension, in all the examples on this page this convention is applied, and as such the above key A:UK:EMP refers to Annual data, depicted by the initial dimension value 'A'.

If the key omits the time period, then the '''time period becomes a variable''' which can be set at table build time, if not set it defaults to the last period for which there is data.

A:UK:EMP

When a Publication Table has a variable time period, the resolved variable time value for this varaible is known as the '''base period'''. There can only be 1 base period for a Publication Table, however other Observation Keys (and corresponding headings) can perform time arithmetic from the base period for which the table is built.

The supported time functions include walking forwards and backwards in increments of the given frequency, converting to another frequency of data, and a combination of the two (convert and move, move and convert, move convert and move again). Ths supported functions are shown in the table below.

{| class="wikitable"
|-
! Syntax !! Description !! Example Observation Key !! Example Resolved Key
|-
| P || The base period (default if not provided) || A:UK:EMP:P || A:UK:EMP:2008 (base period=2008)
|-
| P-''n'' || Base Period minus 'n' Periods where n is a positive Integer || A:UK:EMP:P-1 || A:UK:EMP:2007 (base period=2008)
|-
| P+''n'' || Base Period plus 'n' Periods where n is a positive Integer || A:UK:EMP:P+1 || A:UK:EMP:2008-Q2 (base period=2008-Q1)
|-
| A(P) || Base Period converted to '''Annual''' || A:UK:EMP:A(P) || A:UK:EMP:2008 (base period=2008-Q2)
|-
| S(P) || Base Period converted to '''Bi-Annual''' (Semester) || S:UK:EMP:S(P) || S:UK:EMP:2008-S2 (base period=2008-Q3)
|-
| T(P) || Base Period converted to '''Tri-Annual''' || T:UK:EMP:T(P) || T:UK:EMP:2008-T3 (base period=2008)
|-
| Q(P) || Base Period converted to '''Quarterly''' || Q:UK:EMP:Q(P) || Q:UK:EMP:2008-Q4 (base period=2008)
|-
| M(P) || Base Period converted to '''Monthly''' || M:UK:EMP:M(P) || M:UK:EMP:2008-03 (base period=2008-Q1)
|-
| W(P) || Base Period converted to '''Weekly''' || W:UK:EMP:W(P) || W:UK:EMP:2008-04 (base period=2008-01-01)
|-
| D(P) || Base Period converted to '''Daily''' || D:UK:EMP:D(P) || D:UK:EMP:2008-12-31 (base period=2008)
|-
| H(P) || Base Period converted to '''Hourly''' || H:UK:EMP:D(P) || H:UK:EMP:2008-12-31T12 (base period=2008-02-02T12:00:03)
|-
| A(P-n) || Base Period minus (or plus) 'n' periods, converted to another frequency 'A' || A:UK:EMP:A(P-4) || A:UK:EMP:2007 (base period=2008-Q4)
|-
| A(P)-n || Base Period, converted to another frequency 'A' minus (or plus) n' periods || A:UK:EMP:A(P)-4 || A:UK:EMP:2004 (base period=2008-Q4)
|-
| A(P-n1)-n2 || Base Period, minus (or plus) 'n1' periods, converted to another frequency 'A' minus (or plus) 'n2' periods || A:UK:EMP:A(P-4)-4 || A:UK:EMP:2003 (base period=2008-Q4)
|-
| FA(P) || Subtracts 1 Period from the Converted Period IF the resolved Convert From end of period does not match the same point in time as the Convert To period.

'''Examples:'''
* Base Period =2008-Q4 Rule= FA(P) Output = 2008
* Base Period =2008-Q3 Rule= FA(P) Output = 2007 - the base period 2008-Q3 is not the same time as 2008 (end of period, so move back 1 year)
* Base Period =2008-Q1 Rule= FA(P) Output = 2007
* Base Period =2008-Q1 Rule= FA(P+3) Output = 2008
* Base Period =2008-Q1 Rule= FA(P+3)-1 Output = 2007

This works for ALL frequencies where the convert to frequency is lower then the base Period P, for example Quarter converted to Annual.

|| A:UK:EMP:FA(P) || A:UK:EMP:2007 (base period=2008-Q3)
|}

'''Compound expressions''' 
From Fusion Registry v11.5.1 and later it is possible to cast a period more the once in the same expression, for example

Q(FA(P-1))-1

If the input time period P=2019-Q4, then P-1 = 2019-Q3, the expression is rewritten as Q(FA(2019-Q3))-1. The Full Annual (FA) for 2019-Q3 is 2018, as the end of period 2019 does not match the end of period 2019-Q3, so the annual period moves back an additional period. The expression is rewritten as Q(2018)-1. 2018 to Quarterly moves to 2018-Q4 and then the last part of the expression steps back 1 period, the final period is 2018-Q3.

'''Notes on Time Period''' 
When converting from a lower frequency to a higher frequency (Weekly to Daily for example), the cast date always resolves to the end of the Period.

It is also important to note that as the [[#Observation Key|Observation Keys]] in these examples all have Frequency as the first Dimension, when converting to another frequency (Quarterly to Annual for example) the Observation Key changes the value for the Frequency Dimension, ensuring the correct series of data is retrieved.

'''Time Period variable in free text''' 
When refering to the Time Period as a variable in free text, for example in the table heading or a column heading, the variable is placed inside the variable syntax '''$()'''. For example $(P) would resolve to base period, and $(A(P)) would resolve to 'base period converted to annual frequency'. An example usage is shown below:

Table 1A showing data for $(P) to $(P+3)

Publication Table

2023-01-24T09:24:58Z

Mnelson: /* Calculated Observation Cells */

[[Category:Functions]]

@See [[Publication_Table_Web_Services|Publication Table Web Services]]

= Publication Tables =
'''Introduced to Fusion Registry Enterprise v11.0.0.17'''

== Overview ==
[[File:Publication Table Definition.png|thumb|Using SDMX concepts of Agency, Id, Name, Description]]

[[File:Publication Table.png|thumb|Table Layout and Content Design]]

[[File:MaterialisedTable Preview.png|thumb|Showing the Table Preview feature which materialises the table with the given variables]]

A Publication Table is a curated table of data, where the values in the table rows, columns and table body cells can be individually specified. This enables a single table to pull data from multiple sources and piece the data together in a way which is not possible when dealing with standard cubes of data, whose table model is typically based on a Pivot model, with columns and rows built around the Dimension values to which the column or row is assigned. A Publication Table breaks out of this restrictive model by allowing cells of data to reference observations which do not need to share any relationship, in terms of dimensionality or even data source, with adjacent cells,.

Data in a Publication Table may be sourced from a single [[Dataflow_-_Structural_Metadata_Management|Dataflow]] or it may be sourced from multiple Dataflows enabling cross comparison of data.

Publication Tables can be static (every cell is a defined Observation) or dynamic, where a cell includes [[#Variable Dimension Members|variables Dimensions]] in the [[#Observation Key|Observation Key]], which can be provided at [[#Metadata vs Materialised|table build time]].

[[#Time Variables|Time arithmetic and frequency conversion]] is supported, enabling a single table to pull data from different time periods and frequencies which are all derived from the base period on which the table is built.

Publication Tables support [[#Calculated Observation Cells|calculations]], where each calculation can take input from one or more Observations over one or more [[Dataflow_-_Structural_Metadata_Management|Dataflows]].

Variable values also play a big role in Publication Tables, and includes standard variable placeholders – as well as advanced [[#Conditional Variables|conditional variables]], whose list of variable values change based on the value of another Dimension for which the table is built.

== Metadata vs Materialised ==

A Publication Table is stored in Fusion Registry the same way any other piece of structural metadata is stored. In this way it has an owner (Agency), and identity (ID) and a version. The change history is tracked using the same services, and the table definition can be [[Publication_Table_Web_Services|queried]] using the same SDMX web services for Structural Metadata.

The Publication Table metadata contains information about which Dataflows are used by the Table, which additional variables are being imported, and a definition of the table structure, and which Observations to import by referencing them by their unique [[#Observation Key|Observation Key]].

A Publication Table definition can be '''materialised''' into a table of Data by from the [[Publication_Table_Web_Services|Publication Table Web Service]], which combines the information of the Table Identity, and Variable Values to rewrite the [[#Observation Key|Observation Key]] into actual observation values. If building a web User Interface, then response from this service can be a packet of JSON, which describes the layout and content of the final Table in a way which is quick and simple to render for the User to see. The process of materialising the table is known as ‘'''build time'''’.

Additional web services are provided to enable clients to ask questions about the table such as ‘what are the earliest and latest time periods for which there is data for this table’ or ‘what possible variable values are there for this table’. This enables user interfaces to provide a user with the option to change the variables ‘on the fly’ ensuing they are able to build a table which is known to have data.

= Publication Table Cell Content =
== Overview ==
[[File:PublicationTable Layout.png|thumb|Publication Table Layout]]
A Publication Table is made up of a '''Table Head Rows''' and '''Table Body Rows'''.

The Table Head is made up of 1 or more table rows. Each Table Head cell may span 1 or more columns and 1 or more rows. The Table Head content is made up of free text and / or variable placeholders whose values are resolved when the table is built. Table Head Cells can be left empty.

The Table Body is made up of 1 or more Table Rows. The Table Rows provide the Publication Table with its data, by referencing Observations from the Dataflow.

At table build time all variables are resolved, and the corresponding data are retrieved from the data stores which hold the data.

== Table Rows ==
=== Overview ===
A Table Row consist of a Table Row heading, this can be a mix of free text and [[#Table Variables|variable text]]. The Table Row Heading is always the first Table Row Cell, and can not span multiple columns or rows. A Table Row can not have more then one heading. The remaining Table Row Cells are the Observation Cells, which may contain free text and [[#Table Variables|variable text]] but will typically contain a reference to an Observation by an [[#Observation Key|Obsevation Keys]] or a [[#Calculated Observation Cells|Calculation]] which can combine multiple [[#Observation Key|Obsevation Keys]].

A Table Row which results in no Observation data will not be included in materialised table at build time, unless the Table Row is a grouping row (see [[#Row Levels|Row Levels]] for details on grouping rows).

=== Row Levels ===
[[File:PublicationTabel Levels.png|thumb|Showing Levels in the Table Rows]]
The Table Row Heading can be used to define a row level. The level indicates to the client responsible for rendering the table the row's position in a hierarchy. Row levels are depicted using one or more leading chevrons ‘>‘ before the text. The number of hyphens indicates the level, with one leading hypen indicating level 1, two indicating level 2, and so on. It is possible to skip levels if required, i.e. Row 1 is level 1 with Row 2 assigned to level 3, skipping out level 2 entirely.

An example of row headings using levels is as follows:

Countries
> Europe
>> France
>> Germany
> Asia
>> China

Levels can be an important way to '''group data'''. A Table Row which acts as a parent row in a hierarchy, can be used as grouping mechanism for the rows that follow. This enables Parent Rows to be included in the output table, even if the Table Row contains no Observation data. The default behaviour for a Table Row with no data is to exclude it from the built table.

=== Looping Rows ===

Instead of explicitly defining a row for each Dimension member, it is possible to define a row as a loop (to iterate over all, or a subset of Dimension members). For example, if a publication table contains a Dimension for countries with an ID of REF_AREA then a loop can be set up to loop through all REF_AREA values, or a subset of the values - for which there is data. If the values are backed by a Codelist which has a hierarchy, the hierarchy can be preserved in the final table. It is also possible to nest a loop within another loop, for example for each REF_AREA output each INDICATOR.

====Syntax====
The loop is defined by giving the table row a label which starts with a syntax of :=LOOP([args]), like in the following example
:=LOOP([dimension id], [preserve hierarchy], [loop order], [explicit values to include])

'''Argument Definition'''
''dimension id'' = Required. The ID of the Dimension to loop over. For example CURRENCY would loop over all the Currency values which have data in the given context. The row label is the Currency, for example USD.
''preserve hierarchy'' = Optional, default false. If true the hierarchy of the Codes in the underlying Codelist will be built into the row hierarchy
''loop order'' = Optional, default 1. 1=Natural Order (as they appear in the Codelist), 2=By Code ID, 3=By Code Label
''explicit values to include'' = Optional, default include all values. A var arg list of values to loop over.

'''Full Example'''

:=LOOP(REF_AREA, true, 3, EUR,FR,DE,UK,ES)

====Simple Loop====
The loop is defined by giving the table row a label which starts with a syntax of :=LOOP([args]), like in the following example
:=LOOP(REF_AREA)

This loop definition will loop through all the REF_AREA values, building a row for each value. The Observation keys for the row can refer to the loop variable using a $(DIM_ID) placeholder in the Observation Key, for example:

Employed Student
:=LOOP(REF_AREA) A:$(REF_AREA):EMP A:$(REF_AREA):STU

The output table would look like the following:

Employed Student
United Kingdom 20 12
France 22 11
Germany 25 15

====Nested Loop====
If a loop row is followed by a row (or rows) which is an extra level deep, the child row(s) will be included for each loop, for example

Employed Student
> :=LOOP(REF_AREA)
>> Total A:$(REF_AREA):'''T''':EMP A:$(REF_AREA):'''T''':STU
>>> Male A:$(REF_AREA):'''M''':EMP A:$(REF_AREA):'''M''':STU
>>> Female A:$(REF_AREA):'''F''':EMP A:$(REF_AREA):'''F''':STU

Because the '''Total''' row is a child row of the LOOP (denoted by the number of leading '>') and '''Male'''/'''Female''' are children of '''Total''' - for each REF_AREA there will be a Total, Male and Female, outputting a table like the following:

Employed Student
United Kingdom
Total
Male 5 1
Female 4 2
France
Total
Male 6 2
Female 7 3

It is also possible to define a subloop as a loop. The previous example can be rewritten as:

Employed Student
> :=LOOP(REF_AREA)
>> :=LOOP(SEX) A:$(REF_AREA):'''$(SEX)''':EMP A:$(REF_AREA):'''$(SEX)''':STU

There is no limit to the number of subloops that can be defined.

====Loop Hierarchical Values====
If the underlying Codelist has a hierarchy, a subsequent argument can be provided to ask for the hierarchy to be preserved

Countries
> :=LOOP(REF_AREA, true)

'''Note:''' A hierarchy will keep rows with no data if they are acting as a parent node for rows that do have data (e.g. there may not be data for Europe, but the row exists because there are child rows of France and Germany with data).

Nested loops can also preserve hierarchies for example:

:=LOOP(REF_AREA, true)
>:=LOOP(SEX, true)

With an example output (assuming no data exists for the hierarical parents (Continent, Europe)

Continent
Europe
France
Total (M+F)
Male
Female
Germany
Total (M+F)
Male
Female

====Order Loop Rows====
It is possible to order the labels by their natural order (that of the Codelist), by id or by label, using the numbers 1, 2, or 3 respectively. The following outputs the rows ordered by label.

Countries
> :=LOOP(REF_AREA, true, 3)

====Fixed Loop List====
It is also possible to define a subset of the values to loop over. For example, if REF_AREA has data for 100 countries, but only 3 are required in the table. In this scenario, the natural order is that of the provided list, not of the underlying Codelist.

Countries
> :=LOOP(REF_AREA, true, 3, UK, FR, DE)

====Loop over External Variable====
Instead of looping the values of the Dimension's Codelist, it is possible to loop another Codelist - the purpose for this is to support alternative labels (if the labels for the publication table need to be different from those of the Dimension members) - or a subset (as an alternative to passing the subset of values into the LOOP argument), or to define a hierarchy (if the Dimension's Codelist does not have a hierarchy, or an alternative is required).

In order to loop over an external Variable, the Publication Table must be linked to an external Codelist/Valuelist against an Alias. For example EXT=My Country Codelist with Hierarchy. The corresponding loop then loops the Codes in the external list, and stores these the corresponding Dimension ID variable, for example:

Countries
> '''REF_AREA''':=LOOP('''EXT''')

Noting that the loop is over the variable EXT which resolves to an external Codelist. The loop is assigned to the REF_AREA Dimension.

=== Anchor Rows ===
The default behaviour is for a row to be omitted if the row has no data and it has no child rows with data. However, if the desired result is to always output the row, then this can be achieved by using the [anchor] tag in the row header label.

Consider the following Publication Table definition:

Male Female
Employed A:M:EMP A:F:EMP
Student A:M:STU A:F:STU
Retired A:M:RET A:F:RET

If there are no data for Student, then the materialised table would look like this

Male Female
Employed 12 13
Retired 3 5

==== Anchor Single Row [anchor] ====
To anchor a single row, add the '''[anchor]''' tag at the start of the row

Male Female
Employed A:M:EMP A:F:EMP
[anchor]Student A:M:STU A:F:STU
Retired A:M:RET A:F:RET

Then the output table would preserve the 'Student' row

Male Female
Employed 12 13
Student
Retired 3 5

This rule can be coupled with [[Publication_Table#Formatting_Missing_Observations|formatting for missing observations]], then the table could be enhanced to look something like this

Male Female
Employed 12 13
Student - -
Retired 3 5

==== Anchor Child Rows [anchor+] ====
For a row with a hierarchy, an '''[anchor+]''' tag can be placed on a parent node to anchor the parent and all of its children, for example:

Continent
[anchor+]>Europe
>>France
>>>Total (M+F)
>>>>Male
>>>>Female
>>Germany
>>>Total (M+F)
>>>>Male
>>>>Female

The above rule would always output Europe and all of its descendants (France, Total, Male, Female, Germany).

==== Anchor Siblings and Child Rows [anchor*]====
The '''[anchor*]''' tag can be used as a way to anchor all following siblings and child nodes, this can be a useful mechanism to anchor all rows for the whole table by placing the anchor[*] on the first row of the table.

==== Anchor Loops ====
If a loop is preceeded by an '''[anchor]''' tag then the system will have the following behaviour:

* If an external Codelist variable is being looped, all codes will be ouput regardless of what data is stored in the system
* If a Dimension is being looped, all codes in the Codelist used by the Dimension will be output regardless of what data is stored in the system
* If a Dimension is being looped and it is not coded (i.e it is free text or numerical or other) then the [anchor] has no effect, as the looped values are based on what data exists

== Observation Key ==
=== Overview ===
The purpose of the Observation Key is to reference a Observation whose value will be resolved at table build time. The Observation Key can explicitly resolve to 1 observation value, if all parts of the Key are provided. However, it is possible to make parts of the key varaibles, allowing the values to be set at table build time.

An Observation Key is made up of 4 parts:

* Dataflow Reference
* Series Reference
* Time Period Reference
* Measure Reference

An example Observation Key with all of these parameters in place is:

DF_REG.M:UK:SUR:2008:BIRTHS

The key is broken down into the following parts:

{| class="wikitable"
|-
! Observation Key Part !! Name !! Description !! Required
|-
| DF_REG|| Dataflow Alias|| This identifies the Dataflow which has the data. The identification uses the alias, which is assiged to the Dataflow in the Publication Table definition. The Dataflow Alias is only required if the Publication Table references more then one Dataflow. If the Publication Table references more then one Dataflow, it can assign a Dataflow to a Default Alias, which means that it is required as part of the Observation Key. || No (if only 1 Dataflow or a default alias is set)
|-
| M:UK:SUR || Series Key || This identifies the Series of Data for the Dataflow, by providing a single value for each of the Dimension Members of the Dataflow, separated by a ‘:’ character. In this example the Dataflow has 3 Dimensions (excluding time), the value for the first is 'M', the second 'UK' and the last 'SUR'. As the Dataflow in this example has a Time Dimension this Series Key will uniquely identify a Series of Observations where each Observation refers to a unique Time Period || Yes (but can omit values)
|-
| 2008 || Time Period || This identifies the Time Period in the Series of Data identified by the previous section. The Combination of the Series and Time Period for a Time Series Dataflow will uniquely identify the Observation || No. Omit to make it variable for Time Series Data (defaults to last period). Not required if there is no Time Dimension for the DSD.
|-
| BIRTHS || Measure Dimension || If the Dataflow uses a Data Structure with multiple Measures, then the Measure ID must be included as the last part of the Observation Key. In this example the Dataflow has multiple measures (BIRTHS/DEATHS/MARRIAGES) so the Observation Key needs to include this value. || When there is only 1 measure it is not required, when there are multiple measures, it is required.
|}

Observation Keys are included on the materialised table in JSON format enabling the observation value to be tied back to the key. Observation Keys also provide a mechanism by which [[#Formatting_Cells|formatting rules]] can be applied to a subset of table cells.

=== Variable Dimension Members ===
It is possible to omit Dimension Values from the Observation Key, when this is the case that Dimension will become a variable – whose value can be set at table build time. For example, the second Dimension value of ‘UK’, from the previous example, can be omitted like this:

M::SUR:2008

The Publication Table can now be built for any Country by passing in the variable at table build time - i.e. UK, FR, DE. By passing in different variable values, the Observation Key will change, and as such it will reference a different Series in the Dataset.

When an Observation Key has a variable Dimension member, the ID of the Dimension can be referenced in text cells to create variable headings. For example, if the ID of the 2nd Dimension in the DSD is '''REF_AREA''', the Table Column could be given a title of $(REF_AREA) which would resolve to 'United Kingdom', 'France', or 'Germany' at build time depending on which varaible value is used to build the table. [[#Dimension Variables|#See Dimension Variables]].

=== Variable Time Period ===
For Time Series dataset, the Time Dimension can form part of the Observation Key, for example M:UK:SUR:'''2008'''. The 2008 part of the key refers to the year 2008.

It is possible to omit the value for Time Period in the Observation Key. Omitting Time Period will make the Time value a variable which can be provided at table build time, defaulting to the last period for which data exists if no value is supplied.

Extending the above example to omit the Time value results in an Observation Key whose Country and Time values can be provided to the web service at table build time.

M::SUR

'''Note:''' The above key is exactly the same as using the variable Time Period value 'P' in the Time Dimension, as follows:

M::SUR:'''P'''

Additional functions can be performed on the Time Dimension, such as moving forwards or backwards from the table base period, [[#Time Variables|#see Time Variables]] for more details.

=== Free Text Observation Keys ===
Whilst typically a cell in the body of the Publication Table would resolve to an Observation value, it is possible to use it for free text. To include free text cells, start the cell value with a leading apostrophe character, for example:

'Cell Text

'''Note''': An Observation Key can '''not''' contain a mix of free text as well as an Observation value

== Calculated Observation Cells ==
[[File:TableDefinition Calculations Variables.png|thumb|Publication Table with Calculated Cells and Variable Dimensions]]

An Observation Cell may contain calculated values, which can consist of a mix of simple algebra (+, -, *, /) and a reference to one or more Observations which may come from multiple Dataflows.

To define a Calculated Observation the Cell the following rules apply:

* The Observation Key must start with :=
* Any Observation Keys must appear within enclosing square brackets '[' and ']'
* An Observation key may embed calculations on a single Dimension by using parenthesis '(' and ')' in the key part
* The same rules for variable Dimensions and Time Period apply when resolving the Observation Keys used in the calculation

Examples are shown in the table below

{| class="wikitable"
|-
! Example Syntax !! Description
|-
| :=[A:UK:EMP]+10 || Add 10 to the observation value retrieved from the key A:UK:EMP
|-
| :=[A:(UK+FR+DE):EMP] || Add the series A:UK:EMP + A:FR:EMP + A:DE:EMP
|-
| :=[A:UK:EMP]+[A:FR:EMP]+[A:DE:EMP] || Add the series A:UK:EMP + A:FR:EMP + A:DE:EMP
|-
| :=[A:(UK+FR):EMP]/[A:(UK+FR):UMP] || Adds the series A:UK:EMP + A:FR:EMP and divides by the result of A:UK:UMP + A:FR:UMP
|-
| :=((100/[A:UK:EMP:(M+F):P-1])*[A:UK:EMP:(M+F)])-100 || Demonstrates the ability to use parenthesis outside of the Observation Keys to control the order of mathematical operations
|-
| :=[POP.M:UK:(M+F)] / [LAB.M:UK:EMP] || Demonstrates the ability to calculate data from two Dataflows (defined by alias POP and LAB)
|-
| :=[M:UK:EMP:M(P)] + [A:UK:EMP:A(P)] || Demonstrates the ability to calculate data where the period is converted from the base period
|}

Calculated Cells have implicit [[#Observation_Key|Observation Keys]] which are derived based on the inputs series in the calculation. The [[#Observation_Key|Observation Key]] can be used to assign [[#Formatting_Cells|Formatting Rules]] in the same way as a 'standard key' can be used. It is possible to assign an explicit key to the calculation, which will override the implicit key assigned. The following table shows how the key assignment works.

For example
{| class="wikitable"
|-
! Calculation !! Output Key || Explanation
|-
|:=[A:(UK+FR+DE):EMP] || A:#:EMP || The first and last dimension have a fixed value, the second dimension has a variable value and is replaced by '#'
|-
|:=((100/[A:UK:EMP:(M+F):P-1])*[A:UK:EMP:(M+F)])-100 || A:UK:EMP:# || The first three dimensions have a fixed value, the last dimension has a variable value and is replaced by '#'
|-
|:=PCTOF([A:M:UK:P],[A:TOT:UK:P]) || A:#:UK || The first and last dimension have a fixed value, the second dimension has a variable value and is replaced by '#'
|-
|:=[A:UK:EMP]+10 || A:UK:EMP || The input key is the same as the output key as none of the dimensions have a variable value
|-
|A:UK10:EMP:=[A:UK:EMP]+10 || A:UK10:EMP || The output key is explicitly assigned
|}

== Percentage Of ==
The PCTOF (Percentage Of) function can be used to calculate 'x' as a percentage of 'y' following the syntax:

PCTOF(x,y)

This is reolved to the following mathmatical expression: '''(100/y) * x'''

x and y must either be a numerical value, and observation key, or an expression using a combination of the two. For example

:=PCTOF([A:M:UK:P],[A:TOT:UK:P])

The Observation Key follows the same standard rules with respect to varaibles and inline calculations - the following example shows an Observation Key where Male + Female values are added with the result being used in the Percentage Of function

:=PCTOF([A:M:UK],[A:('''M+F'''):UK])

A more complex example shows calculations around the PCTOF function:
:=(10 + PCTOF([A:M:UK:A(P)-1],[A:(M+F):UK:A(P)])) / 2

== Footnotes ==

[[File:Footnotes3.png|thumb|A table with footnotes]]
[[File:Footnotes2.png|thumb|Footnotes in the table builder]]

=== Overview ===
Footnotes can be added to any part of the publication table (table column/row header, or observation cells). The output table is generated with a footnote number, with the list of footnotes at the end of the table.

Footnotes need to be thought about in 2 parts: Part 1 are the list of footnotes which are defined at the Publication Table level. Part 2 are the references to the footnotes from within the table. This separation of allows footnotes to be reused within the same table, and when working with [[#Footnotes_with_Placeholders|placeholders]] footnote templates can be set up where the text changes based on the placeholder values defined in the table builder.

To add a footnote, add it to the Publication Table, for example: 
Footnote 1 
Footnote 2

Then in the table builder, add a reference to which footnote to use, $(1) references '''Footnote 1''', and $(2) references '''Footnote 2'''.

'''Note:''' The order of the list of footnotes in the table definition may not be reflected in the order they are output. As such a footnote referenced by $(1) may be output as the second footnote.

=== Footnotes with Variable ===
A footnote can reference variables, using the [[#Table_Variables|variable syntax]],

A footnote for $(REF_AREA.UK)

A footnote can even output an observation value, observation attribute or series attribute.

Measure Value
A footnote with obs value $(A:UK:EMP:2008:OBS_VALUE)

Observation Attribute
A footnote with obs attribute $(A:UK:EMP:2008:OBS_COMMENT)

Series Attribute
A footnote with obs attribute $(A:UK:EMP:P:SERIES_TITLE)

When combining variable footnotes with placeholder values and loops, it is possible to dynamically create references to observation attributes.

'''Note:''' the use of P in the final cell key which is a reference to the [[#Time_Variables|time period]] for which the table is being built.

=== Footnotes with Placeholders ===
It is possible to write a generic footnote for the publication table, which includes placeholders. When the footnote is referenced, the placeholder values are provided. The syntax for a placeholder is {n} where n is a positive integer, for example {1}.

Footnote Example

A footnote for $(REF_AREA.{1}) and $(REF_AREA.{2})

In the table builder, the footnote is referenced as usual, but with arguments for the placeholders:

Footnote reference in Table builder

My Table Row '''$(1,UK,FR)'''

The footnote reference in the above example is in bold, and consists of which footnote text to use (in this case the first one in the list, denoted by '1'), and 2 arguments are passed for the placeholders, placeholder 1 takes the value of UK and placeholder 2 takes the value of FR. During the materialise process the intermediate text is generated:

A footnote for $(REF_AREA.UK) and $(REF_AREA.FR)

Which then goes through the variable resolve process, in this case both variables, '''$(REF_AREA.UK) and $(REF_AREA.FR)''', match the [[#Coded_Variables|Coded Variable]] syntax and would resolve to:

Final Output
A footnote for United Kingdom and France

=== Footnotes in Observation Cell ===
Any number of footnotes can appear in an Observation cell, simply put the footnote(s) at the end of the [[#Observation_Key|Observation Key]].

Standard Footnote Reference 
DF_REG.M:UK:SUR:2008:BIRTHS $(1) $(2)

Footnote with Placeholder 
DF_REG.M:UK:SUR:2008:BIRTHS $(1,UK)

=== Footnotes in Loops ===
A row [[#Looping_Rows|loop]] contains a locally scoped variable, and as such it can be used when calling a footnote with a placeholder.

Example Footnotes
$(A:T:{1}:EMP:P:SERIES_TITLE)
$(A:T:{1}:EMP:P:TITLE_COMPL)
$(A:{1}:{2}:EMP:P:OBS_COMMENT)

Example Table Definition
Employed Student
> :=LOOP(REF_AREA) '''$(1,$(REF_AREA[id]))'''
>> Total '''$(2,$(REF_AREA[id]))''' A:$(REF_AREA):T:EMP A:$(REF_AREA):T:STU
>> :=LOOP(SEX) '''$(3,$(REF_AREA[id]),$(SEX[id]))''' A:$(REF_AREA):$(SEX):EMP A:$(REF_AREA):$(SEX):STU

Footnote Intermediate Resolution
Row 1 (REF_AREA=UK)
$(A:T:UK:EMP:P:SERIES_TITLE)

Row 2 (REF_AREA=UK)
$(A:T:UK:EMP:P:TITLE_COMPL)

Row 3 (REF_AREA=UK, SEX=M)
$(A:UK:M:EMP:P:OBS_COMMENT)

Final Footnote output
Row 1 (REF_AREA=UK)
This is a series for the United Kingdom

Row 2 (REF_AREA=UK)
This is a title compliment for the United Kingdom

Row 3 (REF_AREA=UK, SEX=M)
This is an observation comment for Males in the United Kingdom

= Formatting Cells =
== Formatting Numerical Observation Values ==
[[File:Obs formatting.png|thumb|Number Formatting Options]]
Numerical observation values can be formatted in the following way

# '''Number of Decimal Places''' or '''Significant Figures''' can be defined
# For Decimal places this can be set to a maximum rule or a fixed rule, i.e the value 1.3 to 2 decimal places is 1.3 using the maximum rule and 1.30 using the fixed rule
# '''Rounding mode''' can be specified
# '''Scaling Factor''' this is a multiplier applied to the observation value, the multiplier is x10^[scaling factor]. Example, a value of 1.1 with a scaling factor of 1 is 11, a scaling factor of 2 is 110, a scaling factor of 3 is 1100. If the Observation or related Series has a UNIT_MULT concept the difference between the unit multiplier and scaling factor will be taken. Example, an Observation with a value of 1.1 and a unit multiplier of 9 (Billions) is reported, the publication table has a scaling factor of 6 (millions) - the number 1.1 is scaled using 10^3 (calculated by 9-6), the 1.1 billion is converted to 1100 million.
# '''Scientific notation''' can be requested on output, i.e 123456 to 2 significant figures is 120000 in standard form or 1.2E4 in scientific form
# '''Decimal formatting''' for long numbers, i.e. 12345.8 there is not thousands separator by default, to include a thousand separator set the decimal formatting rule to either a single Language (i.e. French Formatting) or Dynamic formatting, which uses the Accept-Langauge locale/language sent be the web browser (this can be overridden using the locale=[loc] query parameter. The formatting rules can also influence which characters the numerical characters used.
# Rules can be applied to all series, or specific series

=== Match Series ===
When creating a formatting rule, it is possible to define which series the rule applies to. If left blank, the rule will apply to all observation values.

The syntax for defining specific series is shown in the following table:

{| class="wikitable"
|-
! Rule Type !! Syntax !! Description
|-
| Series Match || =A:UK:EMP || Match a specific series from the default Dataflow
|-
| Series Match Multiple Values || =A:UK+FR+DE:EMP || Multiple codes in 2nd Dimension (UK, FR and DE) any can match for the rule to be valid
|-
| Series Match by Dataflow Alias || =FLOW2.A:UK:EMP || Match a specific series for the Dataflow with alias FLOW2
|-
| Wildcard Series Match || =A::EMP || Match all series with the given dimension values (2nd Dimension is wildcarded)
|-
| Not Rule || !=A:UK:EMP || Match all series which do not match this series (wildcard series also supported)
|-
| Dimension Rule || CURRENCY=GBP || Match all series which have CURRENCY of GBP
|-
| Not Dimension Rule || !CURRENCY=GBP || Match all series do not have a CURRENCY of GBP
|-
| Multiple Rules || CURRENCY=GBP,FREQ=M || Comma is used to include multiple matches (also applies to series match)
|}

== Formatting Missing Observations ==

Observation Values can be missing for 2 reasons:
# There is no Observation in the database for the given Observation Key
# There is an Observation in the database but the measure value is either: NaN, null, or an empty string

In both cases it is possible to define an output value using formatting rules. For the first case, the observation does not exist, it is possible to define a default output value, for example ‘-‘. For the second case, where the observation does exist, it is possible to define a set of rules based on another Component of the Observation, for example the Observation output may be related to the OBS_STATUS Attribute.

When the formatted value is based on a Component, a set of rules must be defined which state ‘when component=x output y’, where x may be an fixed value or a pattern to match (using a regular expression). The rules for mapping component values are defined in a [https://fmrwiki.sdmxcloud.org/Representation_Map Representation Map], and the Publication Table simply provides the connection to the Representation Map to use. This enables reuse of the mapping rules across multiple Publication Tables. The Publication Table must define two properties in order to support Component Mapping rules:

# The Component to obtain the value from i.e OBS_STATUS
# A link to the [https://fmrwiki.sdmxcloud.org/Representation_Map Representation Map], which holds the mapping rules. Note, the Representation Map is expected to have 1 source mapping, this could be of any type (string, codelist, other)

An example Reprsentation Map for OBS_STATUS is shown below:

{| class="wikitable"
|-
! OBS_STATUS !! Is Reg Ex !! Missing Value !! Description
|-
| M || false || mv || When OBS_STATUS is M output mv
|-
| L || false || np || When OBS_STATUS is L output np
|-
| .* || true || - || When OBS_STATUS is anything else output -
|}

== Frequency Formatting ==
[[File:PublicationTable FreqMap.png|thumb|Representation Map with frequency mapping rules]]
[[File:PublicationTable Materialised FreqMap.png|thumb|Materialised table with formatted time headings ]]
Time Variables always resolve themselves to an [[SDMX_Time_Formats|SDMX Date Format]] when forming the [[#Observation Key|Observation Key]], this ensures the correct observation value is returned from the data store. When a time variable is used as a label, in a table heading for example, the same formatting will be applied by default but a custom format can be provided. Custom display formats for a Time Period enable a time period such as 2007-Q1 to be resolved to custom text string, for example '31 Mar 2007'.

Custom Time formats rules are defined by providing an association between an SDMX Frequency ID (#see [[SDMX_Time_Formats|SDMX Date Format]]) and the output format template. The output format template is a string that defines which components of time are output an in which order and style. For example to output a date as 'March 2001' the format template would look like this 'MMMM yyyy' to indicate a full month name (MMMM) and a 4 character year (yyyy). format rules make use of the [https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html Simple Date Format] which is used by Java as such all of the documentation on Simple Date Format applies.

Frequency Format Mapping is not defined as part of a Publication Table, instead the rules are defined in a [https://fmrwiki.sdmxcloud.org/Representation_Map Representation Map] and the Publication Table references the Representation Map as the source of its frequency conversion rules. This enables the same set of rules to be used for multiple Publication Tables. The Representation Map must map 1 source (Codelist or String) to 1 target (String), it can contain as many mapped frequencies as required. If the Representation Map contains a mapping for a value that is not a valid SDMX Frequency, then it will be ignored.

It is important to note that all time periods resolve to a single point in time, which is the end of the period, for example an Annual period of 2008 resolves to 2008-12-31:T23:59:59. The resolution of frequency to a point in time enables the frequency formatting rules to use all aspects of time, even when converting a low frequency, for example 2008 formatted using the pattern dd MMM yy would resolve to '31 Dec 2008'.

== Variables Summary ==

{| class="wikitable"
|-
! Variable Type !! Description !! Syntax !! Example !! Example Output
|-
| '''Dataflow Name''' || Displays the name of a Dataflow referenced by the Publication Table in the locale in which the Publication Table is built || $([dataflow alias]) || $(EDU) || World Bank, Education
|-
| '''Dataflow Description''' || Displays the description of a Dataflow referenced by the Publication Table in the locale in which the Publication Table is built || $([dataflow alias].desc) || $(EDU.desc) || World Bank, Education
|-
| '''Dataflow Id''' || Displays the ID of a Dataflow referenced by the Publication Table || $([dataflow alias].id) || $(EDU.id) || DF_EDU
|-
| '''Variable Dimension Value''' || Displays the name of the variable used when an Observation key has a wildcard member || $([dimension id]) || $(REF_AREA) || United Kingdom
|-
| '''Specific Component Value''' || Displays the name of the Component Value || $([component id].[component value]) || $(OBS_STATUS.C) || Confidential
|-
| '''Component Name''' || Displays the name of the DSD Component (this is the name of the Concept used by the Component) in the locale in which the Publication Table is built || $([component id].name) || $(OBS_STATUS) || Confidentiality Status
|-
| ''' Enumeration Value ''' || Displays the name of a Code in a Codelist or Value in a Valuelist || $([list alias].[list item ID] || $(SEX.M) || Male
|-
| '''Conditional Variable''' || Displays the variable value, which is conditional on the value of a variable Dimension || $([conditional variable id][variable id]) || $(BANKNOTES[NOTE1]) || £5
|-
| ''' Time ''' || Displays the base time period for which the table is built, or an offset from this time period / cast to a different frequency || see [[#Time Functions|Time Functions]] || $(P) || 2008
|}

= Table Variables =
== Overview ==
Variables are a very important part of Publication Tables. They can be used to make the content of the table dynamic, setting and changing the value at run time to change the table content. Variables can be used in Observation Keys, but the values of variables can also be used anywhere where there is text, for example the table heading, subheading, column or row headings, footnotes, and Observation Cells with free text.

The general syntax for a variable is

$(''variable id''.''sub part''[''detail''])

Where variable id is always required, the subpart and detail are optional. Subpart is used if the variable resolves to a Component and a coded value is required, the detail array part obtains the id, name, or description of the underlying SDMX type.

This section describes the various variables that exist, along with the syntax of how to use them.

== Dataflow Variables ==
Publication Tables define data cells across one or more Dataflows. Each ‘imported’ Dataflow is given an alias, which can be used to refer to the Dataflow in the [[#Observation Key|Observation Key]].

It is possible to refer to the Dataflow's Name, Description, and Id using the following variable syntax.

'''Default Dataflow''' 
Dataflow Id
$([id])

Dataflow Name
$() or $([name))

Dataflow Description
$([desc])

'''Dataflow with Alias=EXR''' 
Dataflow Id
$(EXR[id])

Dataflow Name
$(EXR) or $(EXR[name))

Dataflow Description
$(EXR[desc])

== Dimension Variables ==
When an [[#Observation Key|Observation Key]] has [[#Variable Dimension Members|variable Dimension Values]], the table will be built by replacing this variable with a value to complete the Observation Key. The variable value is either passed explicitly to the web service, or defaulted by the system if not provided. The Dimension Variable can be referenced in the table using the syntax $([dimension id]), for example:

Dimension Value Name
$(INDICATOR) resolves to 'Employment'

This would resolve to the '''name''' of the Indicator value used in the table, for example '''EMP''' may resolve to the label '''Employment'''. The label for the Dimension Value is obtained from the Codelist or Valuelist which the Dimension uses. If the Dimension does not use a list (i.e. it is free text or numerical) then the Dimension value is simply used in place of the variable placeholder.

To resolve to the '''id''' of the value for the Component, for example '''EMP''', use the following syntax

Dimension Value Id
$(INDICATOR[id]) resolves to 'EMP'

== Dependent Dimension Variable ==
It is possible that the value for one Dimension is dependent on the value of another Dimension, for example CURRENCY may be dependent on the COUNTRY and COUNTRY is a variable in the [[#Observation Key|Observation Key]] - it is possible to specify that the value to use for the CURRENCY Dimension can be derived using a lookup table. The lookup table in this instance is maintained as an [https://fmrwiki.sdmxcloud.org/Representation_Map SDMX Representation Map] which is used to map COUNTRY to CURRENCY. The Publication Table specifies that the two Dimensions are related and provides a reference to the Representation Map to use which has the rules.

When working with a dependent variable, both the dependent variable and the Dimension on which it depends should be omitted from the [[#Observation Key|Observation Key]] - for example consider a full key of:

A:UK:GBP:2007

Where UK is the COUNTRY and GBP is the Currency, to make the COUNTRY a variable Dimension, it should be omitted from the key, as follows:

A::GBP:2007

To then make the CURRENCY a variable it is also omitted , as follows:

A:::2007

The Publication Table is given the rule that CURRENCY depends on COUNTRY and should use the COUNTRY_2_CURRENCY [https://fmrwiki.sdmxcloud.org/Representation_Map Representation Map], which in this example looks like the following:

{| class="wikitable"
|-
! COUNTRY !! Is Reg Ex !! CURRENCY
|-
| UK || false || GBP
|-
| US || false || USD
|-
| .* || true || EUR
|}

When the Publication Table is materialised the [[Publication_Table_Web_Service|web service]] only requires the COUNTRY information, and it will complete the Observation key using the representation map lookup table.

== Component Values ==
Components of the underlying Data Structure can be explicitly referenced using the following syntax '''$([component id])''', for example:

Component Name
$(OBS_STATUS) or $(OBS_STATUS[name]) resolves to 'Observation Status'

Component Description
$(OBS_STATUS[desc]) resolves to 'Information on the quality of a value or an unusual or missing value.'

== Coded Variables ==
To reference a Code value, the syntax is $([component id / codelist alias].[item id]). The first variable part is the ID of a Component in the underlying Data Structure, or alternativly the alias of an external variable defined on the Publication Table. An external variable is a link to an SDMX Codelists or Valuelist. Example are:

Name of Code
$(CURRENCY.$) resolving to the text '''Dollar''

Description of Code
$(DWELLING.SM[desc])

== Conditional Variables ==
A Conditional Variable is one whose value is conditional on the value of a [[#Dimension Variables|Dimension Variable]].

An example use case is Banknotes, where the values for NOTE1, NOTE2, or NOTE3 is dependant on the Country Dimension. A Conditional Variable could be used to set a values for each variable (NOTE1, NOTE2, NOTE3) based on the value used for the current Country Dimension i.e. US NOTE1=$5, UK NOTE1=£5.

The syntax of referring to a Conditional Variable is $([conditional variable id][variable id]), for example
$(BANKNOTE[NOTE1])

== Reported Attribute for an Observation or Series ==
A variable can be a reference to a part of an Observation, for example the Unit Multiplier (UNIT_MULT). A Table Row header, for example, could include variable text such as 'Number of payment accounts (''thousands'')' where the word '''thousands''' is a variable because it depends on other variables which build the key.

To achieve this, use the syntax:

$(A:UK::2008:UNIT_MULT[name])

The observation key in this case is A:UK::2008, where the third Dimension is a variable as it has no value (in the main body of the table this dimension would probably also be a variable, which means the table can be built with different values, each value may change the output of this UNIT_MULT value). The last part of the key is the Attribute ID (UNIT_MULT) - this can be a reference to a series attribute, or an observation attribute. The [name] indicates that it should resolve to the name of the reported attribute value, alternative values are '''id''' for the reported value or '''desc''' for the description of the reported value.

If time is also a variable, then the key can change accordingly, using the variable syntax for time (P):

$(A:UK::P:UNIT_MULT[name])

== Time Variables ==
[[File:PublicationTable MovingTime.png|thumb|Publication Table Design, moving and converting the base time period]]

[[File:PublicationTable Materialised MovingTime.png|thumb|Moving and converting periods - showing the materialised table]]

When the data is time series (i.e. the [[Data_Structure_Definition_(DSD)_-_Structural_Metadata_Management|DSD]] has a Time Dimension), the [[#Observation_Key|Observation Key]] can include the time period, for example:

A:UK:EMP:'''2008'''

Where 2008 is the Observation Time Period to use. '''The format of the Time Period must always be the same format as the frequency for which the data are reported'''. For example if the data is annual, and an observation is reported for 2008, then the observation key must use 2008, not another frequency such as 2008-12-31. It is typical in a [[Data_Structure_Definition_(DSD)_-_Structural_Metadata_Management|DSD]] for the Frequency of the Series to be the first Dimension, in all the examples on this page this convention is applied, and as such the above key A:UK:EMP refers to Annual data, depicted by the initial dimension value 'A'.

If the key omits the time period, then the '''time period becomes a variable''' which can be set at table build time, if not set it defaults to the last period for which there is data.

A:UK:EMP

When a Publication Table has a variable time period, the resolved variable time value for this varaible is known as the '''base period'''. There can only be 1 base period for a Publication Table, however other Observation Keys (and corresponding headings) can perform time arithmetic from the base period for which the table is built.

The supported time functions include walking forwards and backwards in increments of the given frequency, converting to another frequency of data, and a combination of the two (convert and move, move and convert, move convert and move again). Ths supported functions are shown in the table below.

{| class="wikitable"
|-
! Syntax !! Description !! Example Observation Key !! Example Resolved Key
|-
| P || The base period (default if not provided) || A:UK:EMP:P || A:UK:EMP:2008 (base period=2008)
|-
| P-''n'' || Base Period minus 'n' Periods where n is a positive Integer || A:UK:EMP:P-1 || A:UK:EMP:2007 (base period=2008)
|-
| P+''n'' || Base Period plus 'n' Periods where n is a positive Integer || A:UK:EMP:P+1 || A:UK:EMP:2008-Q2 (base period=2008-Q1)
|-
| A(P) || Base Period converted to '''Annual''' || A:UK:EMP:A(P) || A:UK:EMP:2008 (base period=2008-Q2)
|-
| S(P) || Base Period converted to '''Bi-Annual''' (Semester) || S:UK:EMP:S(P) || S:UK:EMP:2008-S2 (base period=2008-Q3)
|-
| T(P) || Base Period converted to '''Tri-Annual''' || T:UK:EMP:T(P) || T:UK:EMP:2008-T3 (base period=2008)
|-
| Q(P) || Base Period converted to '''Quarterly''' || Q:UK:EMP:Q(P) || Q:UK:EMP:2008-Q4 (base period=2008)
|-
| M(P) || Base Period converted to '''Monthly''' || M:UK:EMP:M(P) || M:UK:EMP:2008-03 (base period=2008-Q1)
|-
| W(P) || Base Period converted to '''Weekly''' || W:UK:EMP:W(P) || W:UK:EMP:2008-04 (base period=2008-01-01)
|-
| D(P) || Base Period converted to '''Daily''' || D:UK:EMP:D(P) || D:UK:EMP:2008-12-31 (base period=2008)
|-
| H(P) || Base Period converted to '''Hourly''' || H:UK:EMP:D(P) || H:UK:EMP:2008-12-31T12 (base period=2008-02-02T12:00:03)
|-
| A(P-n) || Base Period minus (or plus) 'n' periods, converted to another frequency 'A' || A:UK:EMP:A(P-4) || A:UK:EMP:2007 (base period=2008-Q4)
|-
| A(P)-n || Base Period, converted to another frequency 'A' minus (or plus) n' periods || A:UK:EMP:A(P)-4 || A:UK:EMP:2004 (base period=2008-Q4)
|-
| A(P-n1)-n2 || Base Period, minus (or plus) 'n1' periods, converted to another frequency 'A' minus (or plus) 'n2' periods || A:UK:EMP:A(P-4)-4 || A:UK:EMP:2003 (base period=2008-Q4)
|-
| FA(P) || Subtracts 1 Period from the Converted Period IF the resolved Convert From end of period does not match the same point in time as the Convert To period.

'''Examples:'''
* Base Period =2008-Q4 Rule= FA(P) Output = 2008
* Base Period =2008-Q3 Rule= FA(P) Output = 2007 - the base period 2008-Q3 is not the same time as 2008 (end of period, so move back 1 year)
* Base Period =2008-Q1 Rule= FA(P) Output = 2007
* Base Period =2008-Q1 Rule= FA(P+3) Output = 2008
* Base Period =2008-Q1 Rule= FA(P+3)-1 Output = 2007

This works for ALL frequencies where the convert to frequency is lower then the base Period P, for example Quarter converted to Annual.

|| A:UK:EMP:FA(P) || A:UK:EMP:2007 (base period=2008-Q3)
|}

'''Compound expressions''' 
From Fusion Registry v11.5.1 and later it is possible to cast a period more the once in the same expression, for example

Q(FA(P-1))-1

If the input time period P=2019-Q4, then P-1 = 2019-Q3, the expression is rewritten as Q(FA(2019-Q3))-1. The Full Annual (FA) for 2019-Q3 is 2018, as the end of period 2019 does not match the end of period 2019-Q3, so the annual period moves back an additional period. The expression is rewritten as Q(2018)-1. 2018 to Quarterly moves to 2018-Q4 and then the last part of the expression steps back 1 period, the final period is 2018-Q3.

'''Notes on Time Period''' 
When converting from a lower frequency to a higher frequency (Weekly to Daily for example), the cast date always resolves to the end of the Period.

It is also important to note that as the [[#Observation Key|Observation Keys]] in these examples all have Frequency as the first Dimension, when converting to another frequency (Quarterly to Annual for example) the Observation Key changes the value for the Frequency Dimension, ensuring the correct series of data is retrieved.

'''Time Period variable in free text''' 
When refering to the Time Period as a variable in free text, for example in the table heading or a column heading, the variable is placed inside the variable syntax '''$()'''. For example $(P) would resolve to base period, and $(A(P)) would resolve to 'base period converted to annual frequency'. An example usage is shown below:

Table 1A showing data for $(P) to $(P+3)

Publication Table

2023-01-24T09:17:43Z

Mnelson: /* Overview */

[[Category:Functions]]

@See [[Publication_Table_Web_Services|Publication Table Web Services]]

= Publication Tables =
'''Introduced to Fusion Registry Enterprise v11.0.0.17'''

== Overview ==
[[File:Publication Table Definition.png|thumb|Using SDMX concepts of Agency, Id, Name, Description]]

[[File:Publication Table.png|thumb|Table Layout and Content Design]]

[[File:MaterialisedTable Preview.png|thumb|Showing the Table Preview feature which materialises the table with the given variables]]

A Publication Table is a curated table of data, where the values in the table rows, columns and table body cells can be individually specified. This enables a single table to pull data from multiple sources and piece the data together in a way which is not possible when dealing with standard cubes of data, whose table model is typically based on a Pivot model, with columns and rows built around the Dimension values to which the column or row is assigned. A Publication Table breaks out of this restrictive model by allowing cells of data to reference observations which do not need to share any relationship, in terms of dimensionality or even data source, with adjacent cells,.

Data in a Publication Table may be sourced from a single [[Dataflow_-_Structural_Metadata_Management|Dataflow]] or it may be sourced from multiple Dataflows enabling cross comparison of data.

Publication Tables can be static (every cell is a defined Observation) or dynamic, where a cell includes [[#Variable Dimension Members|variables Dimensions]] in the [[#Observation Key|Observation Key]], which can be provided at [[#Metadata vs Materialised|table build time]].

[[#Time Variables|Time arithmetic and frequency conversion]] is supported, enabling a single table to pull data from different time periods and frequencies which are all derived from the base period on which the table is built.

Publication Tables support [[#Calculated Observation Cells|calculations]], where each calculation can take input from one or more Observations over one or more [[Dataflow_-_Structural_Metadata_Management|Dataflows]].

Variable values also play a big role in Publication Tables, and includes standard variable placeholders – as well as advanced [[#Conditional Variables|conditional variables]], whose list of variable values change based on the value of another Dimension for which the table is built.

== Metadata vs Materialised ==

A Publication Table is stored in Fusion Registry the same way any other piece of structural metadata is stored. In this way it has an owner (Agency), and identity (ID) and a version. The change history is tracked using the same services, and the table definition can be [[Publication_Table_Web_Services|queried]] using the same SDMX web services for Structural Metadata.

The Publication Table metadata contains information about which Dataflows are used by the Table, which additional variables are being imported, and a definition of the table structure, and which Observations to import by referencing them by their unique [[#Observation Key|Observation Key]].

A Publication Table definition can be '''materialised''' into a table of Data by from the [[Publication_Table_Web_Services|Publication Table Web Service]], which combines the information of the Table Identity, and Variable Values to rewrite the [[#Observation Key|Observation Key]] into actual observation values. If building a web User Interface, then response from this service can be a packet of JSON, which describes the layout and content of the final Table in a way which is quick and simple to render for the User to see. The process of materialising the table is known as ‘'''build time'''’.

Additional web services are provided to enable clients to ask questions about the table such as ‘what are the earliest and latest time periods for which there is data for this table’ or ‘what possible variable values are there for this table’. This enables user interfaces to provide a user with the option to change the variables ‘on the fly’ ensuing they are able to build a table which is known to have data.

= Publication Table Cell Content =
== Overview ==
[[File:PublicationTable Layout.png|thumb|Publication Table Layout]]
A Publication Table is made up of a '''Table Head Rows''' and '''Table Body Rows'''.

The Table Head is made up of 1 or more table rows. Each Table Head cell may span 1 or more columns and 1 or more rows. The Table Head content is made up of free text and / or variable placeholders whose values are resolved when the table is built. Table Head Cells can be left empty.

The Table Body is made up of 1 or more Table Rows. The Table Rows provide the Publication Table with its data, by referencing Observations from the Dataflow.

At table build time all variables are resolved, and the corresponding data are retrieved from the data stores which hold the data.

== Table Rows ==
=== Overview ===
A Table Row consist of a Table Row heading, this can be a mix of free text and [[#Table Variables|variable text]]. The Table Row Heading is always the first Table Row Cell, and can not span multiple columns or rows. A Table Row can not have more then one heading. The remaining Table Row Cells are the Observation Cells, which may contain free text and [[#Table Variables|variable text]] but will typically contain a reference to an Observation by an [[#Observation Key|Obsevation Keys]] or a [[#Calculated Observation Cells|Calculation]] which can combine multiple [[#Observation Key|Obsevation Keys]].

A Table Row which results in no Observation data will not be included in materialised table at build time, unless the Table Row is a grouping row (see [[#Row Levels|Row Levels]] for details on grouping rows).

=== Row Levels ===
[[File:PublicationTabel Levels.png|thumb|Showing Levels in the Table Rows]]
The Table Row Heading can be used to define a row level. The level indicates to the client responsible for rendering the table the row's position in a hierarchy. Row levels are depicted using one or more leading chevrons ‘>‘ before the text. The number of hyphens indicates the level, with one leading hypen indicating level 1, two indicating level 2, and so on. It is possible to skip levels if required, i.e. Row 1 is level 1 with Row 2 assigned to level 3, skipping out level 2 entirely.

An example of row headings using levels is as follows:

Countries
> Europe
>> France
>> Germany
> Asia
>> China

Levels can be an important way to '''group data'''. A Table Row which acts as a parent row in a hierarchy, can be used as grouping mechanism for the rows that follow. This enables Parent Rows to be included in the output table, even if the Table Row contains no Observation data. The default behaviour for a Table Row with no data is to exclude it from the built table.

=== Looping Rows ===

Instead of explicitly defining a row for each Dimension member, it is possible to define a row as a loop (to iterate over all, or a subset of Dimension members). For example, if a publication table contains a Dimension for countries with an ID of REF_AREA then a loop can be set up to loop through all REF_AREA values, or a subset of the values - for which there is data. If the values are backed by a Codelist which has a hierarchy, the hierarchy can be preserved in the final table. It is also possible to nest a loop within another loop, for example for each REF_AREA output each INDICATOR.

====Syntax====
The loop is defined by giving the table row a label which starts with a syntax of :=LOOP([args]), like in the following example
:=LOOP([dimension id], [preserve hierarchy], [loop order], [explicit values to include])

'''Argument Definition'''
''dimension id'' = Required. The ID of the Dimension to loop over. For example CURRENCY would loop over all the Currency values which have data in the given context. The row label is the Currency, for example USD.
''preserve hierarchy'' = Optional, default false. If true the hierarchy of the Codes in the underlying Codelist will be built into the row hierarchy
''loop order'' = Optional, default 1. 1=Natural Order (as they appear in the Codelist), 2=By Code ID, 3=By Code Label
''explicit values to include'' = Optional, default include all values. A var arg list of values to loop over.

'''Full Example'''

:=LOOP(REF_AREA, true, 3, EUR,FR,DE,UK,ES)

====Simple Loop====
The loop is defined by giving the table row a label which starts with a syntax of :=LOOP([args]), like in the following example
:=LOOP(REF_AREA)

This loop definition will loop through all the REF_AREA values, building a row for each value. The Observation keys for the row can refer to the loop variable using a $(DIM_ID) placeholder in the Observation Key, for example:

Employed Student
:=LOOP(REF_AREA) A:$(REF_AREA):EMP A:$(REF_AREA):STU

The output table would look like the following:

Employed Student
United Kingdom 20 12
France 22 11
Germany 25 15

====Nested Loop====
If a loop row is followed by a row (or rows) which is an extra level deep, the child row(s) will be included for each loop, for example

Employed Student
> :=LOOP(REF_AREA)
>> Total A:$(REF_AREA):'''T''':EMP A:$(REF_AREA):'''T''':STU
>>> Male A:$(REF_AREA):'''M''':EMP A:$(REF_AREA):'''M''':STU
>>> Female A:$(REF_AREA):'''F''':EMP A:$(REF_AREA):'''F''':STU

Because the '''Total''' row is a child row of the LOOP (denoted by the number of leading '>') and '''Male'''/'''Female''' are children of '''Total''' - for each REF_AREA there will be a Total, Male and Female, outputting a table like the following:

Employed Student
United Kingdom
Total
Male 5 1
Female 4 2
France
Total
Male 6 2
Female 7 3

It is also possible to define a subloop as a loop. The previous example can be rewritten as:

Employed Student
> :=LOOP(REF_AREA)
>> :=LOOP(SEX) A:$(REF_AREA):'''$(SEX)''':EMP A:$(REF_AREA):'''$(SEX)''':STU

There is no limit to the number of subloops that can be defined.

====Loop Hierarchical Values====
If the underlying Codelist has a hierarchy, a subsequent argument can be provided to ask for the hierarchy to be preserved

Countries
> :=LOOP(REF_AREA, true)

'''Note:''' A hierarchy will keep rows with no data if they are acting as a parent node for rows that do have data (e.g. there may not be data for Europe, but the row exists because there are child rows of France and Germany with data).

Nested loops can also preserve hierarchies for example:

:=LOOP(REF_AREA, true)
>:=LOOP(SEX, true)

With an example output (assuming no data exists for the hierarical parents (Continent, Europe)

Continent
Europe
France
Total (M+F)
Male
Female
Germany
Total (M+F)
Male
Female

====Order Loop Rows====
It is possible to order the labels by their natural order (that of the Codelist), by id or by label, using the numbers 1, 2, or 3 respectively. The following outputs the rows ordered by label.

Countries
> :=LOOP(REF_AREA, true, 3)

====Fixed Loop List====
It is also possible to define a subset of the values to loop over. For example, if REF_AREA has data for 100 countries, but only 3 are required in the table. In this scenario, the natural order is that of the provided list, not of the underlying Codelist.

Countries
> :=LOOP(REF_AREA, true, 3, UK, FR, DE)

====Loop over External Variable====
Instead of looping the values of the Dimension's Codelist, it is possible to loop another Codelist - the purpose for this is to support alternative labels (if the labels for the publication table need to be different from those of the Dimension members) - or a subset (as an alternative to passing the subset of values into the LOOP argument), or to define a hierarchy (if the Dimension's Codelist does not have a hierarchy, or an alternative is required).

In order to loop over an external Variable, the Publication Table must be linked to an external Codelist/Valuelist against an Alias. For example EXT=My Country Codelist with Hierarchy. The corresponding loop then loops the Codes in the external list, and stores these the corresponding Dimension ID variable, for example:

Countries
> '''REF_AREA''':=LOOP('''EXT''')

Noting that the loop is over the variable EXT which resolves to an external Codelist. The loop is assigned to the REF_AREA Dimension.

=== Anchor Rows ===
The default behaviour is for a row to be omitted if the row has no data and it has no child rows with data. However, if the desired result is to always output the row, then this can be achieved by using the [anchor] tag in the row header label.

Consider the following Publication Table definition:

Male Female
Employed A:M:EMP A:F:EMP
Student A:M:STU A:F:STU
Retired A:M:RET A:F:RET

If there are no data for Student, then the materialised table would look like this

Male Female
Employed 12 13
Retired 3 5

==== Anchor Single Row [anchor] ====
To anchor a single row, add the '''[anchor]''' tag at the start of the row

Male Female
Employed A:M:EMP A:F:EMP
[anchor]Student A:M:STU A:F:STU
Retired A:M:RET A:F:RET

Then the output table would preserve the 'Student' row

Male Female
Employed 12 13
Student
Retired 3 5

This rule can be coupled with [[Publication_Table#Formatting_Missing_Observations|formatting for missing observations]], then the table could be enhanced to look something like this

Male Female
Employed 12 13
Student - -
Retired 3 5

==== Anchor Child Rows [anchor+] ====
For a row with a hierarchy, an '''[anchor+]''' tag can be placed on a parent node to anchor the parent and all of its children, for example:

Continent
[anchor+]>Europe
>>France
>>>Total (M+F)
>>>>Male
>>>>Female
>>Germany
>>>Total (M+F)
>>>>Male
>>>>Female

The above rule would always output Europe and all of its descendants (France, Total, Male, Female, Germany).

==== Anchor Siblings and Child Rows [anchor*]====
The '''[anchor*]''' tag can be used as a way to anchor all following siblings and child nodes, this can be a useful mechanism to anchor all rows for the whole table by placing the anchor[*] on the first row of the table.

==== Anchor Loops ====
If a loop is preceeded by an '''[anchor]''' tag then the system will have the following behaviour:

* If an external Codelist variable is being looped, all codes will be ouput regardless of what data is stored in the system
* If a Dimension is being looped, all codes in the Codelist used by the Dimension will be output regardless of what data is stored in the system
* If a Dimension is being looped and it is not coded (i.e it is free text or numerical or other) then the [anchor] has no effect, as the looped values are based on what data exists

== Observation Key ==
=== Overview ===
The purpose of the Observation Key is to reference a Observation whose value will be resolved at table build time. The Observation Key can explicitly resolve to 1 observation value, if all parts of the Key are provided. However, it is possible to make parts of the key varaibles, allowing the values to be set at table build time.

An Observation Key is made up of 4 parts:

* Dataflow Reference
* Series Reference
* Time Period Reference
* Measure Reference

An example Observation Key with all of these parameters in place is:

DF_REG.M:UK:SUR:2008:BIRTHS

The key is broken down into the following parts:

{| class="wikitable"
|-
! Observation Key Part !! Name !! Description !! Required
|-
| DF_REG|| Dataflow Alias|| This identifies the Dataflow which has the data. The identification uses the alias, which is assiged to the Dataflow in the Publication Table definition. The Dataflow Alias is only required if the Publication Table references more then one Dataflow. If the Publication Table references more then one Dataflow, it can assign a Dataflow to a Default Alias, which means that it is required as part of the Observation Key. || No (if only 1 Dataflow or a default alias is set)
|-
| M:UK:SUR || Series Key || This identifies the Series of Data for the Dataflow, by providing a single value for each of the Dimension Members of the Dataflow, separated by a ‘:’ character. In this example the Dataflow has 3 Dimensions (excluding time), the value for the first is 'M', the second 'UK' and the last 'SUR'. As the Dataflow in this example has a Time Dimension this Series Key will uniquely identify a Series of Observations where each Observation refers to a unique Time Period || Yes (but can omit values)
|-
| 2008 || Time Period || This identifies the Time Period in the Series of Data identified by the previous section. The Combination of the Series and Time Period for a Time Series Dataflow will uniquely identify the Observation || No. Omit to make it variable for Time Series Data (defaults to last period). Not required if there is no Time Dimension for the DSD.
|-
| BIRTHS || Measure Dimension || If the Dataflow uses a Data Structure with multiple Measures, then the Measure ID must be included as the last part of the Observation Key. In this example the Dataflow has multiple measures (BIRTHS/DEATHS/MARRIAGES) so the Observation Key needs to include this value. || When there is only 1 measure it is not required, when there are multiple measures, it is required.
|}

Observation Keys are included on the materialised table in JSON format enabling the observation value to be tied back to the key. Observation Keys also provide a mechanism by which [[#Formatting_Cells|formatting rules]] can be applied to a subset of table cells.

=== Variable Dimension Members ===
It is possible to omit Dimension Values from the Observation Key, when this is the case that Dimension will become a variable – whose value can be set at table build time. For example, the second Dimension value of ‘UK’, from the previous example, can be omitted like this:

M::SUR:2008

The Publication Table can now be built for any Country by passing in the variable at table build time - i.e. UK, FR, DE. By passing in different variable values, the Observation Key will change, and as such it will reference a different Series in the Dataset.

When an Observation Key has a variable Dimension member, the ID of the Dimension can be referenced in text cells to create variable headings. For example, if the ID of the 2nd Dimension in the DSD is '''REF_AREA''', the Table Column could be given a title of $(REF_AREA) which would resolve to 'United Kingdom', 'France', or 'Germany' at build time depending on which varaible value is used to build the table. [[#Dimension Variables|#See Dimension Variables]].

=== Variable Time Period ===
For Time Series dataset, the Time Dimension can form part of the Observation Key, for example M:UK:SUR:'''2008'''. The 2008 part of the key refers to the year 2008.

It is possible to omit the value for Time Period in the Observation Key. Omitting Time Period will make the Time value a variable which can be provided at table build time, defaulting to the last period for which data exists if no value is supplied.

Extending the above example to omit the Time value results in an Observation Key whose Country and Time values can be provided to the web service at table build time.

M::SUR

'''Note:''' The above key is exactly the same as using the variable Time Period value 'P' in the Time Dimension, as follows:

M::SUR:'''P'''

Additional functions can be performed on the Time Dimension, such as moving forwards or backwards from the table base period, [[#Time Variables|#see Time Variables]] for more details.

=== Free Text Observation Keys ===
Whilst typically a cell in the body of the Publication Table would resolve to an Observation value, it is possible to use it for free text. To include free text cells, start the cell value with a leading apostrophe character, for example:

'Cell Text

'''Note''': An Observation Key can '''not''' contain a mix of free text as well as an Observation value

== Calculated Observation Cells ==
[[File:TableDefinition Calculations Variables.png|thumb|Publication Table with Calucated Cells and Variable Dimensions]]

An Observation Cell may contain calculated values, which can consist of a mix of simple algebra (+, -, *, /) and a reference to one or more Observations which may come from multiple Dataflows.

To define a Calculated Observation the Cell the following rules apply:

* The Observation Key must start with :=
* Any Observation Keys must appear within enclosing sequare brackets '[' and ']'
* An Observation key may embed calculations on a single Dimension by using parenthesis '(' and ')' in the key part
* The same rules for variable Dimensions and Time Period apply when resolving the Obseration Keys used in the calculation

Examples are shown in the table below

{| class="wikitable"
|-
! Example Syntax !! Description
|-
| :=[A:UK:EMP]+10 || Add 10 to the observation value retrieved from the key A:UK:EMP
|-
| :=[A:(UK+FR+DE):EMP] || Add the series A:UK:EMP + A:FR:EMP + A:DE:EMP
|-
| :=[A:UK:EMP]+[A:FR:EMP]+[A:DE:EMP] || Add the series A:UK:EMP + A:FR:EMP + A:DE:EMP
|-
| :=[A:(UK+FR):EMP]/[A:(UK+FR):UMP] || Adds the series A:UK:EMP + A:FR:EMP and divides by the result of A:UK:UMP + A:FR:UMP
|-
| :=((100/[A:UK:EMP:(M+F):P-1])*[A:UK:EMP:(M+F)])-100 || Demonstrates the ability to use parenthesis outside of the Observation Keys to control the order of mathematical operations
|-
| :=[POP.M:UK:(M+F)] / [LAB.M:UK:EMP] || Demonstrates the ability to calculate data from two Dataflows (defined by alias POP and LAB)
|-
| :=[M:UK:EMP:M(P)] + [A:UK:EMP:A(P)] || Demonstrates the ability to calculate data where the period is converted from the base period
|}

== Percentage Of ==
The PCTOF (Percentage Of) function can be used to calculate 'x' as a percentage of 'y' following the syntax:

PCTOF(x,y)

This is reolved to the following mathmatical expression: '''(100/y) * x'''

x and y must either be a numerical value, and observation key, or an expression using a combination of the two. For example

:=PCTOF([A:M:UK:P],[A:TOT:UK:P])

The Observation Key follows the same standard rules with respect to varaibles and inline calculations - the following example shows an Observation Key where Male + Female values are added with the result being used in the Percentage Of function

:=PCTOF([A:M:UK],[A:('''M+F'''):UK])

A more complex example shows calculations around the PCTOF function:
:=(10 + PCTOF([A:M:UK:A(P)-1],[A:(M+F):UK:A(P)])) / 2

== Footnotes ==

[[File:Footnotes3.png|thumb|A table with footnotes]]
[[File:Footnotes2.png|thumb|Footnotes in the table builder]]

=== Overview ===
Footnotes can be added to any part of the publication table (table column/row header, or observation cells). The output table is generated with a footnote number, with the list of footnotes at the end of the table.

Footnotes need to be thought about in 2 parts: Part 1 are the list of footnotes which are defined at the Publication Table level. Part 2 are the references to the footnotes from within the table. This separation of allows footnotes to be reused within the same table, and when working with [[#Footnotes_with_Placeholders|placeholders]] footnote templates can be set up where the text changes based on the placeholder values defined in the table builder.

To add a footnote, add it to the Publication Table, for example: 
Footnote 1 
Footnote 2

Then in the table builder, add a reference to which footnote to use, $(1) references '''Footnote 1''', and $(2) references '''Footnote 2'''.

'''Note:''' The order of the list of footnotes in the table definition may not be reflected in the order they are output. As such a footnote referenced by $(1) may be output as the second footnote.

=== Footnotes with Variable ===
A footnote can reference variables, using the [[#Table_Variables|variable syntax]],

A footnote for $(REF_AREA.UK)

A footnote can even output an observation value, observation attribute or series attribute.

Measure Value
A footnote with obs value $(A:UK:EMP:2008:OBS_VALUE)

Observation Attribute
A footnote with obs attribute $(A:UK:EMP:2008:OBS_COMMENT)

Series Attribute
A footnote with obs attribute $(A:UK:EMP:P:SERIES_TITLE)

When combining variable footnotes with placeholder values and loops, it is possible to dynamically create references to observation attributes.

'''Note:''' the use of P in the final cell key which is a reference to the [[#Time_Variables|time period]] for which the table is being built.

=== Footnotes with Placeholders ===
It is possible to write a generic footnote for the publication table, which includes placeholders. When the footnote is referenced, the placeholder values are provided. The syntax for a placeholder is {n} where n is a positive integer, for example {1}.

Footnote Example

A footnote for $(REF_AREA.{1}) and $(REF_AREA.{2})

In the table builder, the footnote is referenced as usual, but with arguments for the placeholders:

Footnote reference in Table builder

My Table Row '''$(1,UK,FR)'''

The footnote reference in the above example is in bold, and consists of which footnote text to use (in this case the first one in the list, denoted by '1'), and 2 arguments are passed for the placeholders, placeholder 1 takes the value of UK and placeholder 2 takes the value of FR. During the materialise process the intermediate text is generated:

A footnote for $(REF_AREA.UK) and $(REF_AREA.FR)

Which then goes through the variable resolve process, in this case both variables, '''$(REF_AREA.UK) and $(REF_AREA.FR)''', match the [[#Coded_Variables|Coded Variable]] syntax and would resolve to:

Final Output
A footnote for United Kingdom and France

=== Footnotes in Observation Cell ===
Any number of footnotes can appear in an Observation cell, simply put the footnote(s) at the end of the [[#Observation_Key|Observation Key]].

Standard Footnote Reference 
DF_REG.M:UK:SUR:2008:BIRTHS $(1) $(2)

Footnote with Placeholder 
DF_REG.M:UK:SUR:2008:BIRTHS $(1,UK)

=== Footnotes in Loops ===
A row [[#Looping_Rows|loop]] contains a locally scoped variable, and as such it can be used when calling a footnote with a placeholder.

Example Footnotes
$(A:T:{1}:EMP:P:SERIES_TITLE)
$(A:T:{1}:EMP:P:TITLE_COMPL)
$(A:{1}:{2}:EMP:P:OBS_COMMENT)

Example Table Definition
Employed Student
> :=LOOP(REF_AREA) '''$(1,$(REF_AREA[id]))'''
>> Total '''$(2,$(REF_AREA[id]))''' A:$(REF_AREA):T:EMP A:$(REF_AREA):T:STU
>> :=LOOP(SEX) '''$(3,$(REF_AREA[id]),$(SEX[id]))''' A:$(REF_AREA):$(SEX):EMP A:$(REF_AREA):$(SEX):STU

Footnote Intermediate Resolution
Row 1 (REF_AREA=UK)
$(A:T:UK:EMP:P:SERIES_TITLE)

Row 2 (REF_AREA=UK)
$(A:T:UK:EMP:P:TITLE_COMPL)

Row 3 (REF_AREA=UK, SEX=M)
$(A:UK:M:EMP:P:OBS_COMMENT)

Final Footnote output
Row 1 (REF_AREA=UK)
This is a series for the United Kingdom

Row 2 (REF_AREA=UK)
This is a title compliment for the United Kingdom

Row 3 (REF_AREA=UK, SEX=M)
This is an observation comment for Males in the United Kingdom

= Formatting Cells =
== Formatting Numerical Observation Values ==
[[File:Obs formatting.png|thumb|Number Formatting Options]]
Numerical observation values can be formatted in the following way

# '''Number of Decimal Places''' or '''Significant Figures''' can be defined
# For Decimal places this can be set to a maximum rule or a fixed rule, i.e the value 1.3 to 2 decimal places is 1.3 using the maximum rule and 1.30 using the fixed rule
# '''Rounding mode''' can be specified
# '''Scaling Factor''' this is a multiplier applied to the observation value, the multiplier is x10^[scaling factor]. Example, a value of 1.1 with a scaling factor of 1 is 11, a scaling factor of 2 is 110, a scaling factor of 3 is 1100. If the Observation or related Series has a UNIT_MULT concept the difference between the unit multiplier and scaling factor will be taken. Example, an Observation with a value of 1.1 and a unit multiplier of 9 (Billions) is reported, the publication table has a scaling factor of 6 (millions) - the number 1.1 is scaled using 10^3 (calculated by 9-6), the 1.1 billion is converted to 1100 million.
# '''Scientific notation''' can be requested on output, i.e 123456 to 2 significant figures is 120000 in standard form or 1.2E4 in scientific form
# '''Decimal formatting''' for long numbers, i.e. 12345.8 there is not thousands separator by default, to include a thousand separator set the decimal formatting rule to either a single Language (i.e. French Formatting) or Dynamic formatting, which uses the Accept-Langauge locale/language sent be the web browser (this can be overridden using the locale=[loc] query parameter. The formatting rules can also influence which characters the numerical characters used.
# Rules can be applied to all series, or specific series

=== Match Series ===
When creating a formatting rule, it is possible to define which series the rule applies to. If left blank, the rule will apply to all observation values.

The syntax for defining specific series is shown in the following table:

{| class="wikitable"
|-
! Rule Type !! Syntax !! Description
|-
| Series Match || =A:UK:EMP || Match a specific series from the default Dataflow
|-
| Series Match Multiple Values || =A:UK+FR+DE:EMP || Multiple codes in 2nd Dimension (UK, FR and DE) any can match for the rule to be valid
|-
| Series Match by Dataflow Alias || =FLOW2.A:UK:EMP || Match a specific series for the Dataflow with alias FLOW2
|-
| Wildcard Series Match || =A::EMP || Match all series with the given dimension values (2nd Dimension is wildcarded)
|-
| Not Rule || !=A:UK:EMP || Match all series which do not match this series (wildcard series also supported)
|-
| Dimension Rule || CURRENCY=GBP || Match all series which have CURRENCY of GBP
|-
| Not Dimension Rule || !CURRENCY=GBP || Match all series do not have a CURRENCY of GBP
|-
| Multiple Rules || CURRENCY=GBP,FREQ=M || Comma is used to include multiple matches (also applies to series match)
|}

== Formatting Missing Observations ==

Observation Values can be missing for 2 reasons:
# There is no Observation in the database for the given Observation Key
# There is an Observation in the database but the measure value is either: NaN, null, or an empty string

In both cases it is possible to define an output value using formatting rules. For the first case, the observation does not exist, it is possible to define a default output value, for example ‘-‘. For the second case, where the observation does exist, it is possible to define a set of rules based on another Component of the Observation, for example the Observation output may be related to the OBS_STATUS Attribute.

When the formatted value is based on a Component, a set of rules must be defined which state ‘when component=x output y’, where x may be an fixed value or a pattern to match (using a regular expression). The rules for mapping component values are defined in a [https://fmrwiki.sdmxcloud.org/Representation_Map Representation Map], and the Publication Table simply provides the connection to the Representation Map to use. This enables reuse of the mapping rules across multiple Publication Tables. The Publication Table must define two properties in order to support Component Mapping rules:

# The Component to obtain the value from i.e OBS_STATUS
# A link to the [https://fmrwiki.sdmxcloud.org/Representation_Map Representation Map], which holds the mapping rules. Note, the Representation Map is expected to have 1 source mapping, this could be of any type (string, codelist, other)

An example Reprsentation Map for OBS_STATUS is shown below:

{| class="wikitable"
|-
! OBS_STATUS !! Is Reg Ex !! Missing Value !! Description
|-
| M || false || mv || When OBS_STATUS is M output mv
|-
| L || false || np || When OBS_STATUS is L output np
|-
| .* || true || - || When OBS_STATUS is anything else output -
|}

== Frequency Formatting ==
[[File:PublicationTable FreqMap.png|thumb|Representation Map with frequency mapping rules]]
[[File:PublicationTable Materialised FreqMap.png|thumb|Materialised table with formatted time headings ]]
Time Variables always resolve themselves to an [[SDMX_Time_Formats|SDMX Date Format]] when forming the [[#Observation Key|Observation Key]], this ensures the correct observation value is returned from the data store. When a time variable is used as a label, in a table heading for example, the same formatting will be applied by default but a custom format can be provided. Custom display formats for a Time Period enable a time period such as 2007-Q1 to be resolved to custom text string, for example '31 Mar 2007'.

Custom Time formats rules are defined by providing an association between an SDMX Frequency ID (#see [[SDMX_Time_Formats|SDMX Date Format]]) and the output format template. The output format template is a string that defines which components of time are output an in which order and style. For example to output a date as 'March 2001' the format template would look like this 'MMMM yyyy' to indicate a full month name (MMMM) and a 4 character year (yyyy). format rules make use of the [https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html Simple Date Format] which is used by Java as such all of the documentation on Simple Date Format applies.

Frequency Format Mapping is not defined as part of a Publication Table, instead the rules are defined in a [https://fmrwiki.sdmxcloud.org/Representation_Map Representation Map] and the Publication Table references the Representation Map as the source of its frequency conversion rules. This enables the same set of rules to be used for multiple Publication Tables. The Representation Map must map 1 source (Codelist or String) to 1 target (String), it can contain as many mapped frequencies as required. If the Representation Map contains a mapping for a value that is not a valid SDMX Frequency, then it will be ignored.

It is important to note that all time periods resolve to a single point in time, which is the end of the period, for example an Annual period of 2008 resolves to 2008-12-31:T23:59:59. The resolution of frequency to a point in time enables the frequency formatting rules to use all aspects of time, even when converting a low frequency, for example 2008 formatted using the pattern dd MMM yy would resolve to '31 Dec 2008'.

== Variables Summary ==

{| class="wikitable"
|-
! Variable Type !! Description !! Syntax !! Example !! Example Output
|-
| '''Dataflow Name''' || Displays the name of a Dataflow referenced by the Publication Table in the locale in which the Publication Table is built || $([dataflow alias]) || $(EDU) || World Bank, Education
|-
| '''Dataflow Description''' || Displays the description of a Dataflow referenced by the Publication Table in the locale in which the Publication Table is built || $([dataflow alias].desc) || $(EDU.desc) || World Bank, Education
|-
| '''Dataflow Id''' || Displays the ID of a Dataflow referenced by the Publication Table || $([dataflow alias].id) || $(EDU.id) || DF_EDU
|-
| '''Variable Dimension Value''' || Displays the name of the variable used when an Observation key has a wildcard member || $([dimension id]) || $(REF_AREA) || United Kingdom
|-
| '''Specific Component Value''' || Displays the name of the Component Value || $([component id].[component value]) || $(OBS_STATUS.C) || Confidential
|-
| '''Component Name''' || Displays the name of the DSD Component (this is the name of the Concept used by the Component) in the locale in which the Publication Table is built || $([component id].name) || $(OBS_STATUS) || Confidentiality Status
|-
| ''' Enumeration Value ''' || Displays the name of a Code in a Codelist or Value in a Valuelist || $([list alias].[list item ID] || $(SEX.M) || Male
|-
| '''Conditional Variable''' || Displays the variable value, which is conditional on the value of a variable Dimension || $([conditional variable id][variable id]) || $(BANKNOTES[NOTE1]) || £5
|-
| ''' Time ''' || Displays the base time period for which the table is built, or an offset from this time period / cast to a different frequency || see [[#Time Functions|Time Functions]] || $(P) || 2008
|}

= Table Variables =
== Overview ==
Variables are a very important part of Publication Tables. They can be used to make the content of the table dynamic, setting and changing the value at run time to change the table content. Variables can be used in Observation Keys, but the values of variables can also be used anywhere where there is text, for example the table heading, subheading, column or row headings, footnotes, and Observation Cells with free text.

The general syntax for a variable is

$(''variable id''.''sub part''[''detail''])

Where variable id is always required, the subpart and detail are optional. Subpart is used if the variable resolves to a Component and a coded value is required, the detail array part obtains the id, name, or description of the underlying SDMX type.

This section describes the various variables that exist, along with the syntax of how to use them.

== Dataflow Variables ==
Publication Tables define data cells across one or more Dataflows. Each ‘imported’ Dataflow is given an alias, which can be used to refer to the Dataflow in the [[#Observation Key|Observation Key]].

It is possible to refer to the Dataflow's Name, Description, and Id using the following variable syntax.

'''Default Dataflow''' 
Dataflow Id
$([id])

Dataflow Name
$() or $([name))

Dataflow Description
$([desc])

'''Dataflow with Alias=EXR''' 
Dataflow Id
$(EXR[id])

Dataflow Name
$(EXR) or $(EXR[name))

Dataflow Description
$(EXR[desc])

== Dimension Variables ==
When an [[#Observation Key|Observation Key]] has [[#Variable Dimension Members|variable Dimension Values]], the table will be built by replacing this variable with a value to complete the Observation Key. The variable value is either passed explicitly to the web service, or defaulted by the system if not provided. The Dimension Variable can be referenced in the table using the syntax $([dimension id]), for example:

Dimension Value Name
$(INDICATOR) resolves to 'Employment'

This would resolve to the '''name''' of the Indicator value used in the table, for example '''EMP''' may resolve to the label '''Employment'''. The label for the Dimension Value is obtained from the Codelist or Valuelist which the Dimension uses. If the Dimension does not use a list (i.e. it is free text or numerical) then the Dimension value is simply used in place of the variable placeholder.

To resolve to the '''id''' of the value for the Component, for example '''EMP''', use the following syntax

Dimension Value Id
$(INDICATOR[id]) resolves to 'EMP'

== Dependent Dimension Variable ==
It is possible that the value for one Dimension is dependent on the value of another Dimension, for example CURRENCY may be dependent on the COUNTRY and COUNTRY is a variable in the [[#Observation Key|Observation Key]] - it is possible to specify that the value to use for the CURRENCY Dimension can be derived using a lookup table. The lookup table in this instance is maintained as an [https://fmrwiki.sdmxcloud.org/Representation_Map SDMX Representation Map] which is used to map COUNTRY to CURRENCY. The Publication Table specifies that the two Dimensions are related and provides a reference to the Representation Map to use which has the rules.

When working with a dependent variable, both the dependent variable and the Dimension on which it depends should be omitted from the [[#Observation Key|Observation Key]] - for example consider a full key of:

A:UK:GBP:2007

Where UK is the COUNTRY and GBP is the Currency, to make the COUNTRY a variable Dimension, it should be omitted from the key, as follows:

A::GBP:2007

To then make the CURRENCY a variable it is also omitted , as follows:

A:::2007

The Publication Table is given the rule that CURRENCY depends on COUNTRY and should use the COUNTRY_2_CURRENCY [https://fmrwiki.sdmxcloud.org/Representation_Map Representation Map], which in this example looks like the following:

{| class="wikitable"
|-
! COUNTRY !! Is Reg Ex !! CURRENCY
|-
| UK || false || GBP
|-
| US || false || USD
|-
| .* || true || EUR
|}

When the Publication Table is materialised the [[Publication_Table_Web_Service|web service]] only requires the COUNTRY information, and it will complete the Observation key using the representation map lookup table.

== Component Values ==
Components of the underlying Data Structure can be explicitly referenced using the following syntax '''$([component id])''', for example:

Component Name
$(OBS_STATUS) or $(OBS_STATUS[name]) resolves to 'Observation Status'

Component Description
$(OBS_STATUS[desc]) resolves to 'Information on the quality of a value or an unusual or missing value.'

== Coded Variables ==
To reference a Code value, the syntax is $([component id / codelist alias].[item id]). The first variable part is the ID of a Component in the underlying Data Structure, or alternativly the alias of an external variable defined on the Publication Table. An external variable is a link to an SDMX Codelists or Valuelist. Example are:

Name of Code
$(CURRENCY.$) resolving to the text '''Dollar''

Description of Code
$(DWELLING.SM[desc])

== Conditional Variables ==
A Conditional Variable is one whose value is conditional on the value of a [[#Dimension Variables|Dimension Variable]].

An example use case is Banknotes, where the values for NOTE1, NOTE2, or NOTE3 is dependant on the Country Dimension. A Conditional Variable could be used to set a values for each variable (NOTE1, NOTE2, NOTE3) based on the value used for the current Country Dimension i.e. US NOTE1=$5, UK NOTE1=£5.

The syntax of referring to a Conditional Variable is $([conditional variable id][variable id]), for example
$(BANKNOTE[NOTE1])

== Reported Attribute for an Observation or Series ==
A variable can be a reference to a part of an Observation, for example the Unit Multiplier (UNIT_MULT). A Table Row header, for example, could include variable text such as 'Number of payment accounts (''thousands'')' where the word '''thousands''' is a variable because it depends on other variables which build the key.

To achieve this, use the syntax:

$(A:UK::2008:UNIT_MULT[name])

The observation key in this case is A:UK::2008, where the third Dimension is a variable as it has no value (in the main body of the table this dimension would probably also be a variable, which means the table can be built with different values, each value may change the output of this UNIT_MULT value). The last part of the key is the Attribute ID (UNIT_MULT) - this can be a reference to a series attribute, or an observation attribute. The [name] indicates that it should resolve to the name of the reported attribute value, alternative values are '''id''' for the reported value or '''desc''' for the description of the reported value.

If time is also a variable, then the key can change accordingly, using the variable syntax for time (P):

$(A:UK::P:UNIT_MULT[name])

== Time Variables ==
[[File:PublicationTable MovingTime.png|thumb|Publication Table Design, moving and converting the base time period]]

[[File:PublicationTable Materialised MovingTime.png|thumb|Moving and converting periods - showing the materialised table]]

When the data is time series (i.e. the [[Data_Structure_Definition_(DSD)_-_Structural_Metadata_Management|DSD]] has a Time Dimension), the [[#Observation_Key|Observation Key]] can include the time period, for example:

A:UK:EMP:'''2008'''

Where 2008 is the Observation Time Period to use. '''The format of the Time Period must always be the same format as the frequency for which the data are reported'''. For example if the data is annual, and an observation is reported for 2008, then the observation key must use 2008, not another frequency such as 2008-12-31. It is typical in a [[Data_Structure_Definition_(DSD)_-_Structural_Metadata_Management|DSD]] for the Frequency of the Series to be the first Dimension, in all the examples on this page this convention is applied, and as such the above key A:UK:EMP refers to Annual data, depicted by the initial dimension value 'A'.

If the key omits the time period, then the '''time period becomes a variable''' which can be set at table build time, if not set it defaults to the last period for which there is data.

A:UK:EMP

When a Publication Table has a variable time period, the resolved variable time value for this varaible is known as the '''base period'''. There can only be 1 base period for a Publication Table, however other Observation Keys (and corresponding headings) can perform time arithmetic from the base period for which the table is built.

The supported time functions include walking forwards and backwards in increments of the given frequency, converting to another frequency of data, and a combination of the two (convert and move, move and convert, move convert and move again). Ths supported functions are shown in the table below.

{| class="wikitable"
|-
! Syntax !! Description !! Example Observation Key !! Example Resolved Key
|-
| P || The base period (default if not provided) || A:UK:EMP:P || A:UK:EMP:2008 (base period=2008)
|-
| P-''n'' || Base Period minus 'n' Periods where n is a positive Integer || A:UK:EMP:P-1 || A:UK:EMP:2007 (base period=2008)
|-
| P+''n'' || Base Period plus 'n' Periods where n is a positive Integer || A:UK:EMP:P+1 || A:UK:EMP:2008-Q2 (base period=2008-Q1)
|-
| A(P) || Base Period converted to '''Annual''' || A:UK:EMP:A(P) || A:UK:EMP:2008 (base period=2008-Q2)
|-
| S(P) || Base Period converted to '''Bi-Annual''' (Semester) || S:UK:EMP:S(P) || S:UK:EMP:2008-S2 (base period=2008-Q3)
|-
| T(P) || Base Period converted to '''Tri-Annual''' || T:UK:EMP:T(P) || T:UK:EMP:2008-T3 (base period=2008)
|-
| Q(P) || Base Period converted to '''Quarterly''' || Q:UK:EMP:Q(P) || Q:UK:EMP:2008-Q4 (base period=2008)
|-
| M(P) || Base Period converted to '''Monthly''' || M:UK:EMP:M(P) || M:UK:EMP:2008-03 (base period=2008-Q1)
|-
| W(P) || Base Period converted to '''Weekly''' || W:UK:EMP:W(P) || W:UK:EMP:2008-04 (base period=2008-01-01)
|-
| D(P) || Base Period converted to '''Daily''' || D:UK:EMP:D(P) || D:UK:EMP:2008-12-31 (base period=2008)
|-
| H(P) || Base Period converted to '''Hourly''' || H:UK:EMP:D(P) || H:UK:EMP:2008-12-31T12 (base period=2008-02-02T12:00:03)
|-
| A(P-n) || Base Period minus (or plus) 'n' periods, converted to another frequency 'A' || A:UK:EMP:A(P-4) || A:UK:EMP:2007 (base period=2008-Q4)
|-
| A(P)-n || Base Period, converted to another frequency 'A' minus (or plus) n' periods || A:UK:EMP:A(P)-4 || A:UK:EMP:2004 (base period=2008-Q4)
|-
| A(P-n1)-n2 || Base Period, minus (or plus) 'n1' periods, converted to another frequency 'A' minus (or plus) 'n2' periods || A:UK:EMP:A(P-4)-4 || A:UK:EMP:2003 (base period=2008-Q4)
|-
| FA(P) || Subtracts 1 Period from the Converted Period IF the resolved Convert From end of period does not match the same point in time as the Convert To period.

'''Examples:'''
* Base Period =2008-Q4 Rule= FA(P) Output = 2008
* Base Period =2008-Q3 Rule= FA(P) Output = 2007 - the base period 2008-Q3 is not the same time as 2008 (end of period, so move back 1 year)
* Base Period =2008-Q1 Rule= FA(P) Output = 2007
* Base Period =2008-Q1 Rule= FA(P+3) Output = 2008
* Base Period =2008-Q1 Rule= FA(P+3)-1 Output = 2007

This works for ALL frequencies where the convert to frequency is lower then the base Period P, for example Quarter converted to Annual.

|| A:UK:EMP:FA(P) || A:UK:EMP:2007 (base period=2008-Q3)
|}

'''Compound expressions''' 
From Fusion Registry v11.5.1 and later it is possible to cast a period more the once in the same expression, for example

Q(FA(P-1))-1

If the input time period P=2019-Q4, then P-1 = 2019-Q3, the expression is rewritten as Q(FA(2019-Q3))-1. The Full Annual (FA) for 2019-Q3 is 2018, as the end of period 2019 does not match the end of period 2019-Q3, so the annual period moves back an additional period. The expression is rewritten as Q(2018)-1. 2018 to Quarterly moves to 2018-Q4 and then the last part of the expression steps back 1 period, the final period is 2018-Q3.

'''Notes on Time Period''' 
When converting from a lower frequency to a higher frequency (Weekly to Daily for example), the cast date always resolves to the end of the Period.

It is also important to note that as the [[#Observation Key|Observation Keys]] in these examples all have Frequency as the first Dimension, when converting to another frequency (Quarterly to Annual for example) the Observation Key changes the value for the Frequency Dimension, ensuring the correct series of data is retrieved.

'''Time Period variable in free text''' 
When refering to the Time Period as a variable in free text, for example in the table heading or a column heading, the variable is placed inside the variable syntax '''$()'''. For example $(P) would resolve to base period, and $(A(P)) would resolve to 'base period converted to annual frequency'. An example usage is shown below:

Table 1A showing data for $(P) to $(P+3)