Data Formats

From Fusion Registry Wiki
Revision as of 00:27, 23 March 2020 by Mnelson (talk | contribs) (CSV Formats)
Jump to navigation Jump to search

Overview

Fusion Registry accepts and outputs datasets in a number of formats. When consuming data, the Fusion Registry will analyse the dataset to try to determine what data format it has received, so that it is able to direct it to the right reader. All datasets are read by the Fusion Registry in excatly the same way, so any data processing performed on a Dataset is the same, regardless of the input Data Format.

When querying for data from the Registry web service, or performing a Data Transformation via the Web service, the output data format is described using the HTTP Accept Header which describes the required data format.


HTTP Accept Headers

SDMX Formats

SDMX Fomats are supported as described by the web services specification


CSV Formats

There are a number of CSV 'flavous' supported by Fusion Software, including the SDMX-CSV format which is an official SDMX data format.

The following VND Headers are supported SDMX-CSV application/vnd.sdmx.data+csv Fusion-CSV application/vnd.csv Fusion-CSV-TS application/vnd.csv-ts

Each VND Header can then take the additional arguments of:

Argument Description Supported Values Default
version the version of the format 1.0.0 1.0.0
timeFormat values are converted to the most granular ISO 8601 representation
taking into account the highest frequency of the data in the message
original or normalized original
labels output both code/concept ids and the respective labels
in the specified language
both or id id
  • version - the version of the format
  • normalised time - true/false to indicate wether to normalise time
  • labels - id|both whether to output only code Ids or to include the code labels

Note: The Labels parameter can be used in conjuntion with the HTTP Accept-Language Header to indicate which language to resolve the labels in. If the labels are not available in the requested language, another language will be selected, defaulting to English.

Examples

application/vnd.sdmx.data+csv application/vnd.sdmx.data+csv;version=1.0.0; application/vnd.sdmx.data+csv;version=1.0.0;timeFormat=normalized application/vnd.sdmx.data+csv;version=1.0.0;timeFormat=normalized;labels=both

application/vnd.csv application/vnd.csv;version=1.0.0; application/vnd.csv;version=1.0.0;timeFormat=normalized application/vnd.csv;version=1.0.0;timeFormat=normalized;labels=both

application/vnd.csv-ts application/vnd.csv-ts;version=1.0.0; application/vnd.csv-ts;version=1.0.0;timeFormat=normalized application/vnd.csv-ts;version=1.0.0;timeFormat=normalized;labels=both

Data Reporting Template

There are two types of output when converting data to a Data Reporting Template format. The first is where the Data reporting template is constructed in the usual way, with the Universe of Data being derived from the Dataflow, and related Content Constraints. The second is where the Universe of Data is derived from the dataset being written into the Excel workbook. The default output is to base the Report Template Universe on the constraints, to change this behaviour, use the +partial indicator in the VND Header.

Accept Header Description
application/vnd.reporttemplate Excel Report Template pre-populated with the data from a dataset.


The dataset should contain the Provision Agreement reference, to enable the Fusion Registry to determine the Data Provider
The excel file will be the same as a Report Template generated via the Reporting Template Web Service, but it will be pre-populated with observation and attribute values from the dataset.

application/vnd.reporttemplate.ACY:BANKING(1.0) This is an extension of application/vnd.reporttemplate, it tells the Fusion Registry which Reporting Template to use. Only required if there is more then one Reporting Template for the Dataflow(s) being written.
application/vnd.reporttemplate;DATA_PROVIDER=ONS This is an extension of application/vnd.reporttemplate, it tells the Fusion Registry who the Data Provider is.

The Data Provider's Agency defaults to SDMX. If this is not true, use the syntax DATA_PROVIDER=ACY_ID.ONS

Can be used in conjunction with other VND arguments such as the Report Template identifer.

application/vnd.reporttemplate+partial This outputs the dataset conforming to the layout of the Report Template, but includes only the worksheets, and observation cells for which there is data in the dataset. There is no main worksheet.

A Data Provider reference is not necessary, however information about which Report Template to use can be provided using the .ACY:TEMPLATE_ID(1.0) syntax.