Change Log FR - V11

Changelog for Fusion Registry Version 11

Version 11.13.0 (Released: 2023/02/08)

Based on FMR 11.14.1

Change:
RFS-22: Creating "flat CSV" no longer includes unecessary Sibling Group entries at the start of the file

Improvements:
RFS-181: Support for SDMX-ML v3.0 Data
RFS-39: Update vulnerable third-party dependencies

Bug Fixes:
RFS-163: Data Web Service contains duplicate entry for "Excel"
RFS-164: Data Web Service contains unstyled text label
RFS-76: "lastUpdated" parameter not working correctly on Data Query for external databases

Version 11.12.3 (Released: 2023/11/24)

Based on FMR 11.13.0

Bug Fix:
FR11-217: Concurrent modification exception in cache

Version 11.12.2 (Released: 2023/11/22)

Based on FMR 11.13.0

Change:
FR11-216: REST query which results in no data now returns a 404 with an SDMX error message, was previously an empty dataset

Bug Fix:
FR11-215: Publication Tables: 'includePeriod' REST query parameter returns no data when multiple Dataflows are used and 1 has no data

Version 11.12.1 (Released: 2023/11/21)

Based on FMR 11.13.0

Improvement:
FR11-211: Improvements to the "Test Mapping" feature

Bug Fix:
FR11-214: Registry can consume all connections to an External Data Store

Version 11.12.0 (Released: 2023/11/20)

Based on FMR 11.13.0

New Features:
FR11-210: Publication Table: Calculation expression to support default value when no Observation found
FR11-213: Publication Tables: Calculation expression to support default value if no reported value found

Bug Fix:
FR11-212: Publication Table: Formatting issue using partial keys - returns 400 exception

Version 11.11.0 (Released: 2023/11/03)

New Features:
FR11-206: Create aggregate series using linked Hierarchy
FR11-207: Data Query detail parameter; added 'raw' as a new value to support export of data with no calcualted values via hierarchical aggregations

Bug Fixes:
FR11-208: In a Generated Reporting Template permit format modification on Observation Cells
FR11-209: Publication Table: Issue applying formatting rules across multiple Dataflows

Version 11.10.1 (Released: 2023/09/15)

Bug Fix:
FR11-205: saveas=[filename].zip from v1 SDMX web service results in a corrupt zip

Version 11.10.0 (Released: 2023/09/11)

Based on FMR 11.10.1

New Features:
FR11-195: Publication Tables: Allow dependant variables to be overridden from the API
FR11-197: Default response formats in v2 web service are now configurable
FR11-203: Kafa: new topic to publish structure transaction events to
FR11-204: Kafa: new topic to publish reference metadata events to

Change:
FR11-202: Structure Format response is no longer auto detected if not explicitly set (takes value from default setting)

Improvement:
FR11-198: Querying for data by observation attributes

Bug Fixes:
FR11-199: saveas=zip is not working on v2 API
FR11-200: Publication table: Formatting cells ignores UNIT_MULT when it is reported as a dataset level attribute
FR11-201: Publication table: Conditional Formatting being applied when condition references a Component which is not present in DSD

Version 11.9.6 (Released: 2023/08/25)

Changes:
FR11-163: Validator: "Missing Attributes" renamed to "Missing Components" and reports on missing Measures
FR11-164: Add "Output Required" UI control to Structure Mapping
FR110188: Adding a observation formatting rule treats comma separeted component ids as an AND operator (was OR)

Improvements:
FR11-146: Update Apache POI to version 5
FR11-147: Update Commons-fileupload third-party dependency
FR11-148: Update Spring dependencies
FR11-149: Update Jettison third-party dependency
FR11-150: Web Service to refresh caches
FR11-158: Merging a Codelist should merge Locale information
FR11-160: Update Spring and Jackson Dependencies
FR11-161: Implment caching on publication tables
FR11-167: Improve Mapping functionality so that it can perform Validation
FR11-171: Portal File system sweep - allow files to be placed in root of sweep directory
FR11-173: Update Kryo to version 5.5.0
FR11-174: Publication Tables - not formatting numbers with scientific notation
FR11-175: Components with Time based Mappings cannot be edited in the Structure Map Wizard
FR11-176: Support Custom Plugins
FR11-190: Add SDMX-JSON v2 to the UI
FR11-191: Restore Fusion Security as an authentication mechanism
FR11-192: FMR#481: Duplicated group IDs for schema queries in SDMX-JSON
FR11-193: Mapped Datasets Configuration - new option "Output Required"

Bug Fixes:
FR11-151: Item Validity not returning most recent entry prior to today's date
FR11-152: Subscript out of range error with FusionXL when carriage return is in the name of a structure
FR11-153: No transactions returned on SDMX Transactions page when using date slider
FR11-155: Publicaiton Table Editor: Dependent Variables, remove variable button does not work
FR11-156: Publication Table availabililty algorithm modified to better calculate default values when using multiple dataflows
FR11-157: FusionXL resolve carriage returns in concept names
FR11-159: Cross references not updated correctly when structure already exists
FR11-162: Update Vulnerable Dependencies for FMR 11.7.0
FR11-165: Validating Generic Data can result in IllegalElement error
FR11-166: FMR#418 - Conversion and mapping services fail to read SDMX CSV 2.0 format files with BOM
FR11-168: CSV-TS data format outputting names as labels does not output labels in last Dimension/Attribute column
FR11-169: RegEx Capture Groups for Representation Map only support 0-9 (\10 and above does not work)

Version 11.9.5 (Released: 2023/03/08)

Change:
FR11-140: Modification of interpration of SDMX-CSV V2 row https://github.com/bis-med-it/fmr/issues/349

Improvement:
FR11-145: Fusion Excel Export Data API now takes an additional 'attachment' parameter

Bug Fixes:
FR11-137: Removal of dependencies: gson, protobuf, closure, args4j
FR11-138: Unable to load Structure File when Mapping refers to DSD with ValueList coded dimension https://github.com/bis-med-it/fmr/issues/359
FR11-141: Caching in reference metadata requests https://github.com/bis-med-it/fmr/issues/356
FR11-142: Parsing EDI fails to detect malformed period https://github.com/bis-med-it/fmr/issues/362
FR11-143: EDI REL segment must be followed by ARR segment https://github.com/bis-med-it/fmr/issues/322
FR11-144: Publication Table - Calculations which contain missing values should not output a value

Version 11.9.4 (Released: 2023/02/17)

Bug Fixes
FR11-135: Publication Table - calculated values could come back as enginnering string when standard form is requested
FR11-136: Regression mapped data store was not mapping rows with null attribute values

Version 11.9.3 (Released: 2023/02/16)

Bug Fixes:
FR11-129: Temporary "MV" files not being cleared up after data transformation https://github.com/bis-med-it/fmr/issues/346

Improvements:
FR11-131: Improve Performance of large Codelists which have Item Validity https://github.com/bis-med-it/fmr/issues/353
FR11-133: Improve linking to external datasource by auto detecting foreign key

Version 11.9.2 (Released: 2023/02/13)

Change:
FR11-132: Supported 'latest' in v2 query to prevent errors in connected tools

Version 11.9.1 (Released: 2023/02/10)

Bug Fix:
FR11-128: Security rule on OBS Attribute blocks observation when value is null

Version 11.9.0 (Released: 2023/02/03)

Improvements:
FR11-117: Provide option to skip Validation phase in Transform Web Service https://github.com/bis-med-it/fmr/issues/237
FR11-120: Publication Table - apply number formatting to calculated cells
FR11-121: Publication Table - Apply number scaling before calculations

Change:
FR11-118: Use the public fastUtil library https://github.com/bis-med-it/fmr/issues/338

Bug Fixes:
FR11-116: Timezone issue with REST API resulting in no data
FR11-119: Codelists with Annotations not being correctly parsed in SDMX-JSON https://github.com/bis-med-it/fmr/issues/345
FR11-123: EDI should prevent Quadruplet attributes in Footnote Section https://github.com/bis-med-it/fmr/issues/347
FR11-124: EDI file missing mandatory Period Information validates successfully https://github.com/bis-med-it/fmr/issues/348
FR11-125: Improve validation of EDI DTM segment https://github.com/bis-med-it/fmr/issues/351
FR11-126: Publication Table - decimal places (fixed) truncated trailing zero places when locale was also set
FR11-127: Duplicate elements in SDMX CSV result in unexpected output https://github.com/bis-med-it/fmr/issues/342

Version 11.8.2 (Released: 2023/01/17)

Improvements:
FR11-108: Prevent contradictory excel mappings (generted from Fame Metadata Manager) worksheets being uploaded to Registry
FR11-109: Flag contradictory excel mappings (generted from Fame Metadata Manager) in generated worksheet
FR11-134: Improve Metrics Report with IncUnmappedReport option

Changes:
FR11-111: Change ID Generation of Representation Map from FusionFameMapper xlsx files.
FR11-115: Ensure Implicit Mappings in Structure Maps validate codelists for valid intersection

Bug Fixes:
FR11-107: Parent codes should be included in reference partial
FR11-110: Complete Cube web service feature outputs empty measure values (not 0 values)
FR11-112: Data export formats excel-ts and excel-series can miss observations in output series if series contains observations with no measure values and no attribute values
FR11-113: Fix 'too much recursion' error with reverse engineer data structure with large number of Dimensions
FR11-114: Validate submission of Structure Map's Implicit mappings to ensure Coded mappings have a valid intersection from source to target

Version 11.8.1 (Released: 2023/01/11)

Bug Fixes:
FR11-101: Unable to import Codelists with multiple levels of inheritence in same submission as structures that use inherited codes
FR11-102: Bulk actions - detail=raw is ignored
FR11-103: REST query: detail=raw is not applied to referenced codelists
FR11-104: IncValid output may contain empty series when it should not be generated
FR11-105: Annotations are ignored when importing 2.1 XML Hierarchical Codelists
FR11-106: format=csv-series null pointer

Version 11.8.0 (Released: 2022/12/21)

Important Note for MySQL users: due to changes in the distribution of the MySQL connector (change FR11-94 below), please read this article.

New Feature:
FR11-98: Publication Table - support a scaling function applied to numerical observations

Improvement:
FR11-93: Improve Performance of Viewing a Hierarchy https://wiki.sdmxcloud.org/Publication_Table#Formatting_Numerical_Observation_Values

Changes:
FR11-91: Validation and Transformation should preserve original format unless output format specified
https://wiki.sdmxcloud.org/Data_Validation_Web_Service#HTTP_Headers see Accept Header
https://wiki.sdmxcloud.org/Data_Transformation_Web_Service#HTTP_Headers see Accept Header
FR11-94: Remove MySQL Connector https://wiki.sdmxcloud.org/Install_MySQL#Installing_MySQL_5.7

Bug Fixes:
FR11-89: Support dataset attribute consolidation (detection of duplicates)
FR11-90: Observation deletions are no longer written in EDI format
FR11-92: NPE when publishing EDI data from Command Line
FR11-95: Publication Table - UI changes table definition (looses column) when clicking preview table
FR11-96: Publication Table - Preview table error when table heading Rowspan > 1
FR11-97: Publication Table - bug fix on formatting obs which removed values from output table
FR11-99: Publication Table - Scientific Notation shows NaN in UI and is not applied to observation values
FR11-100: Refreshable workbook - issue security challenge when connecting to private server
FMR11-207: Transforming Data with "Delete" action can leave erroneous output in EDI format
FMR11-217: Publication Tables: Numerical Observation, "Add Rule" button does not work
FMR11-224: Failure to report unmapped data when using TIME_PERIOD mapping

Version 11.7.2 (Released: 2022/11/22)

Change:
FR11-84 - In SDMX CSV, SERIES_KEY and OBS_KEY will output with the quote marks around the whole key

Bug Fixes:
FR11-80 - Validation on Group ID should ignore case
FR11-82 - Size of column in Portal Table not large enough to contain Data Source ID
FR11-85 - When loading via Registry UI, Data file with Duplicate Series results in MV Store Error
FR11-86 - Viewing a Representation Map has issue with "valid from" on Mappings

Version 11.7.1 (Released: 2022/11/15)

New Feature:
FR11-83 - Support Categorise Publication Table via the UI

Version 11.7.0 (Released: 2022/11/11)

New Feature:
FR11-78 - Publication Table Support Numerical Observation formatting

Bug Fixes:
FR11-79 - Fix an issue when "edi.datagroup.identifier" specified with value of asterisk
FR11-80 - Validation on Group ID should ignore case
FR11-81 - Data Portal - ESTAT Connector fix

Version 11.6.0 (Released: 2022/11/31)

New Features:
FR11-76 - Audit Structure Queries
FR11-77 - Audit Data Queries

Improvements:
FR11-75 - Support for extended UTF-8 characters https://github.com/MEDIT-Org/fmr/issues/297

Change:
FR11-65 - Update FontAwesome to use Version 6

Bug Fixes:
FR11-71 - Corrected issue with Dynamic Attributes
FR11-72 - Version Drop-Down not Displayed when Selected Item on Page Load
FR11-74 - Security Group Web Service does not output UTF-8 characters
FR11-65 - Update Web Services "structure" page so the plus symbol is used for "latest"

Version 11.5.2 (Released: 2022/10/14)

New Feature:
FMR11-194 - Publication Table - support casting dates more then once in same expression

Improvement:
FMR11-202 - Update dependencies: Hibernate, Spring-Kafka, Jettison and Jersey https://github.com/MEDIT-Org/fmr/issues/294

Bug Fixes:
FR11-66 - "Metadata Provider Scheme" not listed on Web Service -> Structures page
FR11-67 - "Metadata Provider Wizard" does not allow Providers to start with a numeric yet the SDMX Standard allows this
FR11-68 - "Metadata Provider Scheme" not listed on Cross References page
FR11-69 - Attaching Reference Metadata to Items in an item scheme fails to add link (cache issue)
FMR11-189 - Behaviour of web service resolving multiple Codelist inheritance with "details=raw" https://github.com/MEDIT-Org/fmr/issues/257
FMR11-196 - Unable to read Excel Spreadsheet where all Frequencies are Formulas
FMR11-197 - Request for Codelist in SDMX-JSON, "details=raw", references="children" doesn't return children as raw https://github.com/MEDIT-Org/fmr/issues/257
FMR11-198 - Request for Codelist in SDMX-JSON, "details=raw", references="descendants" doesn't return descendants https://github.com/MEDIT-Org/fmr/issues/257
FMR11-199 - "Metadata Provider Scheme" not listed on Web Service -> Structures page
FMR11-200 - "Metadata Provider Wizard" does not allow Providers to start with a numeric yet the SDMX Standard allows this
FMR11-201 - "Metadata Provider Scheme" not listed on Cross References page
FMR11-203 - Attaching Reference Metadata to Items in an item scheme fails to add link (cache issue)
FMR11-205 - Version Number not Displayed when Selected Item on Page Load

Version 11.5.1.1 (Released: 2022/10/10)

Change:
FR11-73 - Modify SQL to ensure GMT Timezone is used

Version 11.5.1 (Released: 2022/10/05)

New Features:
FR11-60 - Publication Table - support casting dates more then once in same expression
FR11-61 - orderMeasure web service parameter to sort observations by date asc/desc

Version 11.5.0 (Released: 2022/09/28)

New Features:
FMR11-187/FR11-53 - Support recovery of structure transactions from backup on installation https://github.com/MEDIT-Org/fmr/issues/277
FR11-52 - Support variable number of max arguments to a SQL IN statement
FR11-58 - Support import of datafile containing mulitple datasets

Improvements:
FMR11-183 - Update dependencies https://github.com/MEDIT-Org/fmr/issues/277
FR11-48 - Update third-party dependencies: apache-jstl, commons-dcp2, joda-time, spring, tika-core
FR11-45 - Add Version Number to Provision Agreement Selector in Data Source Manager Modal

Bug Fixes:
FMR11-181/FR11-46 - SDMX CSV v 2.0 Data Format does not report its format https://github.com/MEDIT-Org/fmr/issues/110
FMR11-184/FR11-49 - Support '*' for structure type in REST API https://github.com/MEDIT-Org/fmr/issues/273
FMR11-185/FR11-51 - Failure on Compare Revision shows Stack Trace https://github.com/MEDIT-Org/fmr/issues/278
FMR11-186/FR11-55 - SDMX-JSON Writer not outputting DSD as stub https://github.com/MEDIT-Org/fmr/issues/274
FR11-50 - FameMapper code is creating Excel files in temporary storage which are not deleted
FR11-54 - EDI data not fully consolidated when publishing data via web service

Version 11.4.3 (Released: 2022/09/13)

New Features:
FMR11-174 - Publication Tables: Formatting of Missing Values support ObsValue=NaN https://github.com/MEDIT-Org/fmr/issues/253
FMR11-175 - Support Reading of Reference Metadata (with restrictions) in version 2.1 SDMX https://github.com/MEDIT-Org/fmr/issues/250

Improvement:
FMR11-182 - Modify ID generation strategy for importing Reference Metadata at v2.1

Bug Fixes:
FR11-39 - OBS_PRE_BREAK validation corrected
FR11-43 - SDMX DSD output may include invalid attribute "isMultiLingual"
FR11-44 - SQL Server does not delete temporary tables
FMR11-169 - SDMX CSV v1 Data Reader does not allow header fields (row 1) to be double quoted
FMR11-170 - Bulk Export - spelling mistake on Raw help text
FMR11-171 - Order of revisions not as expected
FMR11-176 - OBS_PRE_BREAK validation corrected
FMR11-178 - Quadruplet in EDI vs the other formats in the context of data consolidation or integration https://github.com/MEDIT-Org/fmr/issues/118
FMR11-179 - SDMX DSD output may include invalid attribute "isMultiLingual" https://github.com/MEDIT-Org/fmr/issues/255

Version 11.4.2 (Released: 2022/08/19)

Bug Fixes:
FR11-30 - Fusion Security Manager will not accept AD group names longer than 30 characters
FR11-34 - Updated third-party dependencies

Version 11.4.1 (Released: 2022/08/10)

Bug Fixes:
FR11-21 - Issue loading data into SQL Server in Registry managed table FR11-26 - Bulk Export - spelling mistake on Raw help text
FR11-27 - Order of revisions not as expected
FR11-28 - JNDI Registry Managed Database - unable to load data
FR11-25 - SDMX CSV v1 Data Reader does not allow header fields (row 1) to be double quoted

Version 11.4.0 (Released: 2022/07/22)

New Feature:
FR11-18 - Support Reference Metadata https://wiki.sdmxcloud.org/Reference_Metadata

Improvement:
FMR11-146 - Dependency updates
FMR11-159 - Add "Raw" option to Download structures and remove "Include internal settings"
FMR11-160 - Allow a Concept Representation to be a ValueList

Bug Fixes:
FR11-14 - Issue loading Data Constraint with a Component element under Cube Region element
FR11-17 - Attribute variable does not work in table heading
FR11-19 - Increase max size of security group ID from 30 to 100 characters
FMR11-162 - SERIES_KEY column should be reserved for series keys
FMR11-161 - Issue loading Data Constraint with a Component element under Cube Region element
FMR11-163 - Attribute variable does not work in table heading
FMR11-166 - Duplicate attribute is not detected anymore in FMR
FMR11-167 - Publication Table - Array Variables fail to save
FMR11-168 - Error encountered whist attempting to deserialize structure

Version 11.3.0 (Released: 2022/07/06)

New Feature:
FR11-10/FMR11-153 - Environment Synchronisation - compare the structural metadata content with that from another Registry, push or pull changes

Improvement:
FMR11-158 - Publication Table: Footnotes support

Bug Fixes:
FMR11-152 - Comma not being processed correctly in Web Service query for ItemBeans
FMR11-154 - Unexpected structure validation error from FMR
FMR11-155 - Structure validation errors are not detected anymore in FMR
FMR11-156 - v2 structure REST API does not support queries for multiple items using the comma syntax
FMR11-157 - Typo in header of Representation Maps page

Version 11.2.0 (Released: 2022/06/27)

New Feature:
[FMR11-150] Specify the validation and transformation behaviour when duplicate observations are found - https://wiki.sdmxcloud.org/Data_Transformation_Web_Service#HTTP_Headers

Bug Fixes:
[FR11-3] Schema Query in structure format does not work
[FR11-4] Fusion Cache ignoring parameters like normalise freq on V1 data queries
[FR11-5] Data Point Security - allow rule against a series attribute value that does not exist in the dataset restricts all data
[FR11-8] Comma not being processed correctly in Web Service query for ItemBeans
[FMR11-141] Re-opening of Alternate Half year Issue
[FMR11-143] TIME_FORMAT not being generated correctly
[FMR11-144] Problem with Date data type when reverse engineering a DSD from a CSV dataset
[FMR11-145] Problem with Description data type when reverse engineering a DSD from a CSV dataset
[FMR11-147] DSD Wizard step 3 - not allowing Enumerated Representation
[FMR11-148] Schema Query in Structure Format does not work

Improvement:
[FMR11-140] Revert xsi:type=SiblingType, change the schema to output the Group as Sibling not SiblingType -

Version 11.1.0 (Released: 2022/06/01)

New Feature:
Publication Tables
[FR-5130] Publication Table - Support Dynamic Rows
[FR-5131] Publication Table - Format rules on cells with no data
[FR-5132] Publication Table - Rules to derive a Dimension Value from another Dimenison's value
[FR-5133] Publication Table - Option to output rows with no data
[FR-5134] Publication Table - Ouput Materialised Table in Excel

Improvement:
[FR-5129] - Support for MySQL 8 and above

Version 11.0.0 (Released: 2022/05/22)

Changes and New Features:

• Change to the SDMX 3.0 information model as Fusion Registry’s internal metadata object model
• SDMX 3.0 REST API support
• Backward compatibility for SDMX 2.1 and 2.0 structures and formats
• Backward compatibility with SDMX 2.1 REST API (REST API v1 specification)
• Improved data and metadata access control
• Granular control over admin rights
• Fusion Portal – scheduled and on demand data and structure loads
• Refreshable Excel data format
• JNDI database connection option
• Publication Tables - publish curated but dynamic user interactive web data tables that show precisely the statistics required even if they come from multiple different data sets

Breaking Changes from Fusion Registry 10:

• Change to the operating database schema - upgrading from Fusion Registry 10 to Fusion Registry 11 requires a fresh install and migration of the metadata, data and settings
• Environment sync function temporarily removed
• Fusion Security support deprecated - Active Directory and LDAP are the supported authentication services in Fusion Registry 11