Mapping behaviour using item validity

Overview

Fusion Registry's Structure Mapping function allows datasets to be transformed from one structure and classification scheme to another. Mapping of classifications is performed using Codelist Maps which define how Codes in one Codelist map to those in another.

When defining Codelist Maps, multiple mapping rules can be specified for each Code with a Validity Period for each. The Validity Period sets when in real time the rule will be active. Mapping rules with Validity Periods are executed if the time at which the mapping is performed falls within the time range specified. Conversely, rules without Validity Periods are always active and are executed irrespective of when the mapping is actually performed.

Use Cases

The main use case for setting Validity Periods on Codelist Maps is to forward schedule changes to the behaviour of a map with respect to how classifications are transformed. By way of an example, consider a map that translates a country code dimension into a trading bloc dimension. If it is known that a particular country will move to a different trading bloc on a particular date, two separate rules can be created: one describing the country-bloc mapping before the move, the other describing the mapping after. This is illustrated in the example below:

A DSD with INDICATOR, REF_AREA and FREQ dimensions is transformed into a similar DSD with the addition of a TRADE_BLOC dimension. When defining the mapping rules, most dimensions can be implicitly mapped (an implicit mapping is when the dimension value for each series is simply copied from the source dataset to the target dataset without any transformation). The mapping rules for the TRADE_BLOC dimension however are described using a Codelist Map working from the REF_AREA dimension.

Codelist Map rules 1 to 4 have no Validity Period defined so are always valid. Rule 5 maps reference area XXX to trading bloc BLOC_1, only until 31 December 2020. Transformations performed after that date will use Rule 6 which is defined with a Validity Period of 01 January 2021 to 31 December 2099.

The example below shows how this works in practice:

The Source Series with Key A_IND:YYY:A is discarded and does not appear in the transformed dataset because no mapping rule is defined for REF_AREA YYY in the Codelist Map.

Discontinuous Validity Periods

The Validity Periods on a Codelist Mapping Rule do not have to be continuous i.e. it is permissible for there to be periods where no rule is valid. In the example below, no rules are valid for the period covering 2017 to 2018. Transformations performed during that period would result in series with AAA in the relevant dimension being discarded.

Source Code	Target Code	Valid From	Valid To
AAA	BLOC_1	01-Jan-2016	31-Dec-2016
AAA	BLOC_2	1-Jan-2019	31-Dec-2022

Ambiguous Validity Period Rules

Fusion Registry ignores Codelist Mapping Rules with Validity Periods that overlap because they are ambiguous. The example below illustrates an ambiguous rule. In this case, source series with AAA in the relevant dimension are discarded because the Fusion Registry transformation is unable to determine which rule should be applied.

Source Code	Target Code	Valid From	Valid To
AAA	BLOC_1	01-Jan-2019	31-Dec-2022
AAA	BLOC_2	1-Jan-2020	31-Dec-2023

Note that rule with ambiguous Validitory Periods are always ignored, even if only one rule is valid at the time the transformation is performed as would be the case on 01-Jan-2023 in the above example.

While the Fusion Registry user interface prevents the interactive creation of rules with ambiguous Validity Periods, it is still possible to load Structure Sets containing them. This is because the Item Validity information, which is not part of the official SDMX standard, is encoded as SDMX Annotations on the Structure Set. A Structure Set with Annotations on a Codelist Map that describe ambiguous Codelist Map validity periods remains valid SDMX.