Fusion Transformer

Overview

The Fusion Transformer is a command line application providing transformations between SDMX, and GESMES/TS data files and structure files.

The following structure file formats are supported:

• SDMX Structure Document 1.0

• SDMX Structure Document 2.0

• SDMX Structure Document 2.1

• SDMX Registry Interface Document 2.0

• GESMES/TS

The following data file formats are supported:

• Generic 1.0

• Generic 2.0

• Generic 2.1

• GenericTimeSeries (input only)

• Compact 1.0

• Compact 2.0

• StructureSpecific 2.1

• StructureSpecificTimeSeries 2.1 (input only)

• Message Group 1.0 (containing Compact or Generic Datasets only)

• Message Group 2.0 (containing Compact or Generic Datasets only)

• GESMES/TS

Structure Transformation

The Structure Transformer can be run by executing the command:

java -cp FusionTransformer.jar org.bis.fusion.dataparser.StructureParseMain

For convenience there is a structureTransform.bat file provided that Windows user can use to launch the main class.

Example usage:

structureTransform.bat -o ediStructreOut21.edi -s StructureOut21.xml -v edi

For UNIX users there is an equivalent file: structureTransform.sh.

The following additional arguments are available:

Argument	Mandatory	Description	Allowed Arguments
-edi_lenient	False	Puts the Transformer into EDI Lenient mode.	-No Arguments-
-o <arg>	False	Output file.
-pretty_print	False	Outputs the SDMX-ML structures in a more readable format.	-No Arguments-
-s <arg>	False	URI of structure file to transform. If this option is not specified or the argument is - then input is taken from Standard Input.	If this option is not specified or the argument is - then output is sent to Standard Output. ||
-ug	False	If present will ‘upgrade group’ attributes in a SDMX v2.0 DSD to become a Dimension Group Attribute in v2.1. The Group will still be present in the DSD so the v2.1 Schema is backwards compatible with v2.0 (allowing data to be submitted as either a Group or Series level attribute).	-No Arguments-
-v <arg>	False	The output version.	edi / edi-lenient/ 1.0 / 2.0 / 2.1

Data Transformation

The Data Transformer can be run by executing the command:

java -cp FusionTransformer.jar org.bis.fusion.dataparser.DataParserMain

For convenience there is a dataTransform.bat file provided that Windows users can use to launch the main class.

Example usage:

dataTransform.bat -d TestData/inputData.ges -s TestData/inputDSD.ges -o output.xml -f compact

For UNIX users there is an equivalent file: dataTransform.sh.

The following additional arguments are available:

Argument	Mandatory	Description	Allowed Arguments
-d <arg>	False	URI of data file to transform. If this option is not specified or the argument is - then input is taken from Standard Input.
-f <arg>	True	Output data format.	compact / generic /edi
-o <arg>	False	Output file. If this option is not specified or the argument is - then output is sent to Standard Output.
-s <arg>	True	URI of structure file with DSD.	1.0 / 2.0 / 2.1
-v <arg>	False	Output SDMX Version if appropriate.	-No Arguments-
-tf	False	Prevents the Time Format attribute from being output in SDMX-EDI.	-No Arguments-
-edi_lenient	False	Puts the Transformer into EDI Lenient mode.	-No Arguments-
-pretty_print	False	Outputs the SDMX-ML structures in a clearer fashion.	-No Arguments-
-unify_monthly	False	Outputs SDMX-ML monthly data in the format YYYY-Mmm.	-No Arguments-
-character_mapping	False	Enables character mapping mode.	-No Arguments-
-edi_tf_attr	False	Creates the series attribute “Time Format” from an EDI input file.	-No Arguments-

File Formats and Character Encoding

The Fusion Transformer expects that all files supplied to it are encoded using the charset UTF-8, except for EDI files which are expected to be encoded using the charset ISO-8995-1. If you supply a file that is not explicitly encoded to the appropriate charset, the Fusion Transformer will make the assumption that it is encoded in the expected charset and attempt to decipher it using that charset.

All of the files generated by the Fusion Transformer will be encoded using the charset UTF-8, except for EDI files which will be encoded using the charset ISO-8995-1.

If you experience the Fusion Transformer behaving strangely with certain characters, please check the encoding of your input files using an appropriate software tool.

Character Mapping

When the Fusion Transformer works on a data file, it is possible to map individual characters to alternative characters or a String. This may be useful when trying to get a simplified output (for example converting the character ‘ to the character ' ) or for trying to avoid XML characters being encoded (for example replacing the character “ with the character ' would prevent the quotes character being output as " )

To enable character mapping the following conditions must be met:

• The data transform operation must be supplied with the argument -character_mapping

• A file called “character_mapping.properties” must exist in the directory where Fusion Transformer is running from.

• This file must conform to the standard Java property file format. Each entry in the file has a key and a value separated by the equals sign. Each key may only be a single character long, whereas the value may be any length (including zero length to exclude certain characters from the output).

For performance reasons, Character Mapping only affects attribute fields. This is because otherwise the semantic meaning of the file would be lost.

An example of a legal character mapping file:

# Example of a Fusion Transformer Character Mapping File 

# The effect of this file is that 

# ; characters will be removed from the output 

# the " character will be replaced with the ' character # the @ character will be replaced by the phrase "at symbol"

 ;= 

"=' 

@=at symbol 

# EOF

Time Format Series Attribute Creation

By default, reading an EDI data file does not convert the TIME_FORMAT into an attribute in the output data file, however an option exists in Fusion Transformer to perform this. To enable this behaviour the flag “edi_tf_attr” must be passed to the Fusion Transformer and the Data Structure must have a series attribute with the id of “TIME_FORMAT”.

If both of these conditions are met, then the EDI values are converted in the following manner:

Time Format	EDI Value	ISO Format
Yearly	602 / 702	P1Y
Half Yearly	604 / 704	P6M
Quarterly	608 / 708	P3M
Monthly	610 / 710	P1M
Weekly	616 / 716	P1W
Daily	101 / 102	P1D
Minutely	201 / 203	PT1M