Asynchronous Data Validation and Transformation

2024-02-16T15:19:53Z

Plazarou: /* Data Load Response */

[[Category:WebService]]

= Overview =
The Asynchronous Data Load web service consumes a submitted file for Validation. On receipt of the file, the service returns a UID (token), which can be used to track the validation process, and perform further actions such as transformation to another format, or publish to a database once validation has completed.

The Fusion Registry stores the data on the instance it was sent to, so in a load balanced system, the same server must be accessed, if there is no activity on the file for 15 minutes, the registry will automatically remove the file from its cache.

= Asynchronous Data Load =
{| class="wikitable"
|-
|style="background-color:#eaecf0"|Entry Point|| /ws/public/data/load
|-
|style="background-color:#eaecf0"|Access|| Public (default). Configurable to Private
|-
|style="background-color:#eaecf0"|Http Method|| POST
|-
|style="background-color:#eaecf0"|Accepts|| CSV, XLSX, SDMX-ML, SDMX-EDI (any format for which there is a Data Reader)
|-
|style="background-color:#eaecf0"|Compression || Zip files supported, if loading from URL gzip responses supported
|-
|style="background-color:#eaecf0"|Content-Type|| 1. multipart/form-data (if attaching file) – the attached file must be in field name of uploadFile
2. application/text or application/xml (if submitting data in the body of the POST)
|-
|style="background-color:#eaecf0"|Response Format|| JSON
|-
|style="background-color:#eaecf0"|Response Statuses|| 200 - Data file recieved
400 - Trasformation could not be performed (either an unreadable dataset, or unresolvable reference to a required structure)
401 - Unauthorized (if access has been restricted)
500 - Server Error
|}

== Data Load Response ==
The response to a data load is a token, which can be used in subsequent calls to track the data load and validation process and, once validation is complete, the token can be used to perform actions such as a publish, obtain validation report, export in a different format, or export with mapping.

{
"Success" : true,
"uid" : "unique token"
}

= Request Load Status =
{| class="wikitable"
|-
|style="background-color:#eaecf0"|Entry Point|| /ws/public/data/loadStatus
|-
|style="background-color:#eaecf0"|Access|| Public (default). Configurable to Private
|-
|style="background-color:#eaecf0"|Http Method|| GET
|-
|style="background-color:#eaecf0"|Response Format|| JSON
|}

== Query Parameters ==
{| class="wikitable"
|-
|style="background-color:#eaecf0"|uid|| Unique identifier for the loaded dataset, returned from the data load operation

|}

== Load Status Response ==
The response is a validation report, to either indicate [[/Data_Validation_Web_Service#Valid_Dataset|validation success]] or [[Data_Validation_Web_Service#Dataset_with_Errors|validation with errors]].

The report '''Status''' indicates how far into the validation process the server has reached. The following table shows the various stages:

{| class="wikitable"
|-
! Status !! Description
|-
| Initialising || Initial status
|-
| Analysing || The dataset is being analysed for series and obs count, and which dsd's it references
|-
| Validating || The dataset is being validated
|-
| Complete || The dataset validation process has finished, there may/man not be errors
|-
| Consolidating || The dataset is being consolidated, duplicate series and observations are being merged into one final dataset
|-
| IncorrectDSD || The dataset references a DSD that can not be used to validate the data
|-
| InvalidRef || The dataset references a DSD/Dataflow/Provision that does not exist in the Registry
|-
| MissingDSD|| The dataset does not not reference any structure so the system can not read the dataset
|-
| Error|| The dataset can not be read at all
|}

= Data export/transform =
{| class="wikitable"
|-
|style="background-color:#eaecf0"|Entry Point|| /ws/public/data/download
|-
|style="background-color:#eaecf0"|Access|| Public (default). Configurable to Private
|-
|style="background-color:#eaecf0"|Http Method|| GET
|-
|style="background-color:#eaecf0"|Response Format|| Determined by [[Data_Formats#HTTP_Accept_Headers|Accept Header]]
|}

== Query Parameters ==
{| class="wikitable"
|-
! Query Parmeter !! Format !! Description
|-
|style="background-color:#eaecf0"|uid
|| string
|| Unique identifier for the loaded dataset, returned from the data load operation
|-
|style="background-color:#eaecf0"|datasetIndex
|| integer
|| If multiple datasets are in the data file, identifies which one to export, zero indexed
|-
|style="background-color:#eaecf0"|map
|| string
|| URN of Dataflow or DataStucture to map to, there must be a Structure Map which describes the mapping from the source to target
|-
|style="background-color:#eaecf0"|zip
|| boolean (default is false)
|| Zips the respone
|-
|style="background-color:#eaecf0"|includeMetrics
|| boolean (default is false)
|| Include Metrics in the response see [[Data_Transformation_Web_Service|Data Transformation]]
|-
|style="background-color:#eaecf0"|unmapped
|| boolean (default is false)
|| If the map parameter is supplied, and some series or observation can not be mapped, the unmapped data will be included in either the zip file (if zip is true) or in a multipart-form boundry (if zip is false)
|}

= Revalidate =
A Re-validation service is provided if there is more information that can be provided to the dataset(s) loaded against the token. The reason to revalidate is if the dataset is to be attached to a different Dataflow or Provision Agreement. When this link changes, it may impact the validation results due to the application of different Constraints or Validation Schemes. The underlying datasaet will be updated to refer to this different Dataflow/.Provision Agreement, which will be reflected in the exported dataset if the data format supports this information (for example SDMX-ML contains the linked structure in the Header section of the dataset).

Revalidation will be faster than the initial load and validate process because there is not as much work to do. The dataset is already consolidated, and has already had parts of it validated that will not change even if linked to a different Dataflow. The revalidation service only revalidates parts of the dataset that will change due to a different structure link, for example it would revalidate against constraints if these are different, and validation schemes (mathematical validation rules) if there are different rules based on changing the linked structure for the dataset.

Revalidation is an asynchronous action, whose progress can be tracked using the same token and tracking web services provided.

{| class="wikitable"
|-
|style="background-color:#eaecf0"|Entry Point|| /ws/public/data/revalidate
|-
|style="background-color:#eaecf0"|Access|| Public (default). Configurable to Private
|-
|style="background-color:#eaecf0"|Http Method|| POST
|-
|style="background-color:#eaecf0"|Response Format|| JSON
|-
|style="background-color:#eaecf0"|Accepts|| JSON
|-
|style="background-color:#eaecf0"|Content-Type|| application/json
|-
|style="background-color:#eaecf0"|Response Format|| JSON
|-
|style="background-color:#eaecf0"|Response Statuses|| 200 - request received and being processed
400 - request could not be performed (possibly due to bad syntax of JSON POST)
401 - Unauthorized (if access has been restricted)
500 - Server Error
|}

== Post Body ==
The POST request must contain the token of the dataset to revalidate, this is the same token that was provided by the server on data load. The SRef is an array of URNs, one for each Dataset that was present in the data file loaded to the server. Each URN refers to which structure to use to validate the dataset against. The URN can either be to a Data Structure Definition, Dataflow, or Provision Agreement.
{
UID : "datasetdetailsuid",
SRef : ["urn1", "urn2", urn3"]
}

== Revalidate Response ==
The response is a JSON message with a success message. The validation progress should be tracked using the token that was used in the re-validation request against the [[Asynchronous_Data_Validation_and_Transformation#Request_Load_Status|Load Status]] web service.

{
"Success" : true,
}

= Data Publish =
{| class="wikitable"
|-
|style="background-color:#eaecf0"|Entry Point|| /ws/public/data/publish
|-
|style="background-color:#eaecf0"|Access|| Public (default). Configurable to Private
|-
|style="background-color:#eaecf0"|Http Method|| POST
|-
|style="background-color:#eaecf0"|Accepts|| JSON
|-
|style="background-color:#eaecf0"|Response Format|| JSON
|-
|style="background-color:#eaecf0"|Response Statuses|| 200 - Publish request accepted
400 - Bad request
401 - Unauthorized (if access has been restricted)
500 - Server Error
|}

Publishes the dataset loaded against the uid the expected json is as follows
{
UID : "datasetdetailsuid",
Action : "Append|Replace|Delete",
DeleteAction : "DEFAULT|OBSERVATIONS|SERIES",
Dataset: int
}

The If the action is Delete, then there are three different options for Delete Action. The DEFAULT option is to use the SDMX Delete rules, OBSERVATION will delete all the observations in the loaded dataset from the database, SERIES will delete all the series in the dataset from the database (including all the observations that belong to the series).

Change Log FR - V11 and subsequent releases

2024-02-12T09:37:55Z

Plazarou: /* Changelog for Fusion Registry Version 11 */

[[Category:Fusion Registry Change Logs]]
[[Category:ChangeLogFR]]
__NOTOC__
=Changelog for Fusion Registry Version 11=

==Version 11.13.0 (Released: 2023/02/08)==
Based on FMR 11.14.1 

'''Change:''' 
RFS-22: Creating "flat CSV" no longer includes unecessary Sibling Group entries at the start of the file 

'''Improvements:''' 
RFS-181: Support for SDMX-ML v3.0 Data 
RFS-39: Update vulnerable third-party dependencies 

'''Bug Fixes:''' 
RFS-163: Data Web Service contains duplicate entry for "Excel" 
RFS-164: Data Web Service contains unstyled text label 
RFS-76: "lastUpdated" parameter not working correctly on Data Query for external databases 

==Version 11.12.3 (Released: 2023/11/24)==
Based on FMR 11.13.0 

'''Bug Fix:''' 
FR11-217: Concurrent modification exception in cache

==Version 11.12.2 (Released: 2023/11/22)==
Based on FMR 11.13.0 

'''Change:''' 
FR11-216: REST query which results in no data now returns a 404 with an SDMX error message, was previously an empty dataset 

'''Bug Fix:''' 
FR11-215: Publication Tables: 'includePeriod' REST query parameter returns no data when multiple Dataflows are used and 1 has no data 

==Version 11.12.1 (Released: 2023/11/21)==
Based on FMR 11.13.0 

'''Improvement:''' 
FR11-211: Improvements to the "Test Mapping" feature 

'''Bug Fix:''' 
FR11-214: Registry can consume all connections to an External Data Store 

==Version 11.12.0 (Released: 2023/11/20)==
Based on FMR 11.13.0 

'''New Features:''' 
FR11-210: Publication Table: Calculation expression to support default value when no Observation found 
FR11-213: Publication Tables: Calculation expression to support default value if no reported value found 

'''Bug Fix:''' 
FR11-212: Publication Table: Formatting issue using partial keys - returns 400 exception 

==Version 11.11.0 (Released: 2023/11/03)==
'''New Features:''' 
FR11-206: Create aggregate series using linked Hierarchy 
FR11-207: Data Query detail parameter; added 'raw' as a new value to support export of data with no calcualted values via hierarchical aggregations 

'''Bug Fixes:''' 
FR11-208: In a Generated Reporting Template permit format modification on Observation Cells 
FR11-209: Publication Table: Issue applying formatting rules across multiple Dataflows 

==Version 11.10.1 (Released: 2023/09/15)==
'''Bug Fix:''' 
FR11-205: saveas=[filename].zip from v1 SDMX web service results in a corrupt zip 

==Version 11.10.0 (Released: 2023/09/11)==
Based on FMR 11.10.1

'''New Features:''' 
FR11-195: Publication Tables: Allow dependant variables to be overridden from the API 
FR11-197: Default response formats in v2 web service are now configurable 
FR11-203: Kafa: new topic to publish structure transaction events to 
FR11-204: Kafa: new topic to publish reference metadata events to 

'''Change:''' 
FR11-202: Structure Format response is no longer auto detected if not explicitly set (takes value from default setting) 

'''Improvement:''' 
FR11-198: Querying for data by observation attributes 

'''Bug Fixes:''' 
FR11-199: saveas=zip is not working on v2 API 
FR11-200: Publication table: Formatting cells ignores UNIT_MULT when it is reported as a dataset level attribute 
FR11-201: Publication table: Conditional Formatting being applied when condition references a Component which is not present in DSD 

==Version 11.9.6 (Released: 2023/08/25)==
'''Changes:''' 
FR11-163: Validator: "Missing Attributes" renamed to "Missing Components" and reports on missing Measures 
FR11-164: Add "Output Required" UI control to Structure Mapping 
FR110188: Adding a observation formatting rule treats comma separeted component ids as an AND operator (was OR) 

'''Improvements:''' 
FR11-146: Update Apache POI to version 5 
FR11-147: Update Commons-fileupload third-party dependency 
FR11-148: Update Spring dependencies 
FR11-149: Update Jettison third-party dependency 
FR11-150: Web Service to refresh caches 
FR11-158: Merging a Codelist should merge Locale information 
FR11-160: Update Spring and Jackson Dependencies 
FR11-161: Implment caching on publication tables 
FR11-167: Improve Mapping functionality so that it can perform Validation 
FR11-171: Portal File system sweep - allow files to be placed in root of sweep directory 
FR11-173: Update Kryo to version 5.5.0 
FR11-174: Publication Tables - not formatting numbers with scientific notation 
FR11-175: Components with Time based Mappings cannot be edited in the Structure Map Wizard 
FR11-176: Support Custom Plugins 
FR11-190: Add SDMX-JSON v2 to the UI 
FR11-191: Restore Fusion Security as an authentication mechanism 
FR11-192: FMR#481: Duplicated group IDs for schema queries in SDMX-JSON 
FR11-193: Mapped Datasets Configuration - new option "Output Required" 

'''Bug Fixes:''' 
FR11-151: Item Validity not returning most recent entry prior to today's date 
FR11-152: Subscript out of range error with FusionXL when carriage return is in the name of a structure 
FR11-153: No transactions returned on SDMX Transactions page when using date slider 
FR11-155: Publicaiton Table Editor: Dependent Variables, remove variable button does not work 
FR11-156: Publication Table availabililty algorithm modified to better calculate default values when using multiple dataflows 
FR11-157: FusionXL resolve carriage returns in concept names 
FR11-159: Cross references not updated correctly when structure already exists 
FR11-162: Update Vulnerable Dependencies for FMR 11.7.0 
FR11-165: Validating Generic Data can result in IllegalElement error 
FR11-166: FMR#418 - Conversion and mapping services fail to read SDMX CSV 2.0 format files with BOM 
FR11-168: CSV-TS data format outputting names as labels does not output labels in last Dimension/Attribute column 
FR11-169: RegEx Capture Groups for Representation Map only support 0-9 (\10 and above does not work) 

==Version 11.9.5 (Released: 2023/03/08)==
'''Change:''' 
FR11-140: Modification of interpration of SDMX-CSV V2 row https://github.com/bis-med-it/fmr/issues/349 

'''Improvement:''' 
FR11-145: Fusion Excel Export Data API now takes an additional 'attachment' parameter 

'''Bug Fixes:''' 
FR11-137: Removal of dependencies: gson, protobuf, closure, args4j 
FR11-138: Unable to load Structure File when Mapping refers to DSD with ValueList coded dimension https://github.com/bis-med-it/fmr/issues/359 
FR11-141: Caching in reference metadata requests https://github.com/bis-med-it/fmr/issues/356 
FR11-142: Parsing EDI fails to detect malformed period https://github.com/bis-med-it/fmr/issues/362 
FR11-143: EDI REL segment must be followed by ARR segment https://github.com/bis-med-it/fmr/issues/322 
FR11-144: Publication Table - Calculations which contain missing values should not output a value 

==Version 11.9.4 (Released: 2023/02/17)==
'''Bug Fixes''' 
FR11-135: Publication Table - calculated values could come back as enginnering string when standard form is requested 
FR11-136: Regression mapped data store was not mapping rows with null attribute values 

==Version 11.9.3 (Released: 2023/02/16)==
'''Bug Fixes:''' 
FR11-129: Temporary "MV" files not being cleared up after data transformation https://github.com/bis-med-it/fmr/issues/346 

'''Improvements:''' 
FR11-131: Improve Performance of large Codelists which have Item Validity https://github.com/bis-med-it/fmr/issues/353 
FR11-133: Improve linking to external datasource by auto detecting foreign key 

==Version 11.9.2 (Released: 2023/02/13)==
'''Change:''' 
FR11-132: Supported 'latest' in v2 query to prevent errors in connected tools 

==Version 11.9.1 (Released: 2023/02/10)==
'''Bug Fix:''' 
FR11-128: Security rule on OBS Attribute blocks observation when value is null 

==Version 11.9.0 (Released: 2023/02/03)==
'''Improvements:''' 
FR11-117: Provide option to skip Validation phase in Transform Web Service https://github.com/bis-med-it/fmr/issues/237 
FR11-120: Publication Table - apply number formatting to calculated cells 
FR11-121: Publication Table - Apply number scaling before calculations 

'''Change:''' 
FR11-118: Use the public fastUtil library https://github.com/bis-med-it/fmr/issues/338 

'''Bug Fixes:''' 
FR11-116: Timezone issue with REST API resulting in no data 
FR11-119: Codelists with Annotations not being correctly parsed in SDMX-JSON https://github.com/bis-med-it/fmr/issues/345 
FR11-123: EDI should prevent Quadruplet attributes in Footnote Section https://github.com/bis-med-it/fmr/issues/347 
FR11-124: EDI file missing mandatory Period Information validates successfully https://github.com/bis-med-it/fmr/issues/348 
FR11-125: Improve validation of EDI DTM segment https://github.com/bis-med-it/fmr/issues/351 
FR11-126: Publication Table - decimal places (fixed) truncated trailing zero places when locale was also set 
FR11-127: Duplicate elements in SDMX CSV result in unexpected output https://github.com/bis-med-it/fmr/issues/342 

==Version 11.8.2 (Released: 2023/01/17)==
'''Improvements:''' 
FR11-108: Prevent contradictory excel mappings (generted from Fame Metadata Manager) worksheets being uploaded to Registry 
FR11-109: Flag contradictory excel mappings (generted from Fame Metadata Manager) in generated worksheet 
FR11-134: Improve Metrics Report with IncUnmappedReport option 

'''Changes:''' 
FR11-111: Change ID Generation of Representation Map from FusionFameMapper xlsx files. 
FR11-115: Ensure Implicit Mappings in Structure Maps validate codelists for valid intersection 

'''Bug Fixes:''' 
FR11-107: Parent codes should be included in reference partial 
FR11-110: Complete Cube web service feature outputs empty measure values (not 0 values) 
FR11-112: Data export formats excel-ts and excel-series can miss observations in output series if series contains observations with no measure values and no attribute values 
FR11-113: Fix 'too much recursion' error with reverse engineer data structure with large number of Dimensions 
FR11-114: Validate submission of Structure Map's Implicit mappings to ensure Coded mappings have a valid intersection from source to target 

==Version 11.8.1 (Released: 2023/01/11)==
'''Bug Fixes:''' 
FR11-101: Unable to import Codelists with multiple levels of inheritence in same submission as structures that use inherited codes 
FR11-102: Bulk actions - detail=raw is ignored 
FR11-103: REST query: detail=raw is not applied to referenced codelists 
FR11-104: IncValid output may contain empty series when it should not be generated 
FR11-105: Annotations are ignored when importing 2.1 XML Hierarchical Codelists 
FR11-106: format=csv-series null pointer 

==Version 11.8.0 (Released: 2022/12/21)==
'''Important Note for MySQL users:''' due to changes in the distribution of the MySQL connector (change FR11-94 below), please read [[Install_MySQL#Installing_MySQL_5.7|this article]].

'''New Feature:''' 
FR11-98: Publication Table - support a scaling function applied to numerical observations

'''Improvement:''' 
FR11-93: Improve Performance of Viewing a Hierarchy https://wiki.sdmxcloud.org/Publication_Table#Formatting_Numerical_Observation_Values

'''Changes:''' 
FR11-91: Validation and Transformation should preserve original format unless output format specified 
https://wiki.sdmxcloud.org/Data_Validation_Web_Service#HTTP_Headers see Accept Header 
https://wiki.sdmxcloud.org/Data_Transformation_Web_Service#HTTP_Headers see Accept Header 
FR11-94: Remove MySQL Connector https://wiki.sdmxcloud.org/Install_MySQL#Installing_MySQL_5.7

'''Bug Fixes:''' 
FR11-89: Support dataset attribute consolidation (detection of duplicates) 
FR11-90: Observation deletions are no longer written in EDI format 
FR11-92: NPE when publishing EDI data from Command Line 
FR11-95: Publication Table - UI changes table definition (looses column) when clicking preview table 
FR11-96: Publication Table - Preview table error when table heading Rowspan > 1 
FR11-97: Publication Table - bug fix on formatting obs which removed values from output table 
FR11-99: Publication Table - Scientific Notation shows NaN in UI and is not applied to observation values 
FR11-100: Refreshable workbook - issue security challenge when connecting to private server 
FMR11-207: Transforming Data with "Delete" action can leave erroneous output in EDI format 
FMR11-217: Publication Tables: Numerical Observation, "Add Rule" button does not work 
FMR11-224: Failure to report unmapped data when using TIME_PERIOD mapping 

==Version 11.7.2 (Released: 2022/11/22)==
'''Change:''' 
FR11-84 - In SDMX CSV, SERIES_KEY and OBS_KEY will output with the quote marks around the whole key 

'''Bug Fixes:''' 
FR11-80 - Validation on Group ID should ignore case 
FR11-82 - Size of column in Portal Table not large enough to contain Data Source ID 
FR11-85 - When loading via Registry UI, Data file with Duplicate Series results in MV Store Error 
FR11-86 - Viewing a Representation Map has issue with "valid from" on Mappings 

==Version 11.7.1 (Released: 2022/11/15)==
'''New Feature:''' 
FR11-83 - Support Categorise Publication Table via the UI 

==Version 11.7.0 (Released: 2022/11/11)==
'''New Feature:''' 
FR11-78 - Publication Table Support Numerical Observation formatting 

'''Bug Fixes:''' 
FR11-79 - Fix an issue when "edi.datagroup.identifier" specified with value of asterisk 
FR11-80 - Validation on Group ID should ignore case 
FR11-81 - Data Portal - ESTAT Connector fix 

==Version 11.6.0 (Released: 2022/11/31)==
'''New Features:''' 
FR11-76 - [[Audit_Web_Services#Search_Structure_Queries|Audit Structure Queries]] 
FR11-77 - [[Audit_Web_Services#Search_Data_Queries|Audit Data Queries]] 

'''Improvements:''' 
FR11-75 - Support for extended UTF-8 characters https://github.com/MEDIT-Org/fmr/issues/297 

'''Change:''' 
FR11-65 - Update FontAwesome to use Version 6 

'''Bug Fixes:''' 
FR11-71 - Corrected issue with Dynamic Attributes 
FR11-72 - Version Drop-Down not Displayed when Selected Item on Page Load 
FR11-74 - Security Group Web Service does not output UTF-8 characters 
FR11-65 - Update Web Services "structure" page so the plus symbol is used for "latest"

==Version 11.5.2 (Released: 2022/10/14)==
'''New Feature:''' 
FMR11-194 - Publication Table - support casting dates more then once in same expression 

'''Improvement:''' 
FMR11-202 - Update dependencies: Hibernate, Spring-Kafka, Jettison and Jersey https://github.com/MEDIT-Org/fmr/issues/294 

'''Bug Fixes:''' 
FR11-66 - "Metadata Provider Scheme" not listed on Web Service -> Structures page 
FR11-67 - "Metadata Provider Wizard" does not allow Providers to start with a numeric yet the SDMX Standard allows this 
FR11-68 - "Metadata Provider Scheme" not listed on Cross References page 
FR11-69 - Attaching Reference Metadata to Items in an item scheme fails to add link (cache issue) 
FMR11-189 - Behaviour of web service resolving multiple Codelist inheritance with "details=raw" https://github.com/MEDIT-Org/fmr/issues/257 
FMR11-196 - Unable to read Excel Spreadsheet where all Frequencies are Formulas 
FMR11-197 - Request for Codelist in SDMX-JSON, "details=raw", references="children" doesn't return children as raw https://github.com/MEDIT-Org/fmr/issues/257 
FMR11-198 - Request for Codelist in SDMX-JSON, "details=raw", references="descendants" doesn't return descendants https://github.com/MEDIT-Org/fmr/issues/257 
FMR11-199 - "Metadata Provider Scheme" not listed on Web Service -> Structures page 
FMR11-200 - "Metadata Provider Wizard" does not allow Providers to start with a numeric yet the SDMX Standard allows this 
FMR11-201 - "Metadata Provider Scheme" not listed on Cross References page 
FMR11-203 - Attaching Reference Metadata to Items in an item scheme fails to add link (cache issue) 
FMR11-205 - Version Number not Displayed when Selected Item on Page Load 

==Version 11.5.1.1 (Released: 2022/10/10)==
'''Change:''' 
FR11-73 - Modify SQL to ensure GMT Timezone is used 

==Version 11.5.1 (Released: 2022/10/05)==
'''New Features:''' 
FR11-60 - Publication Table - support casting dates more then once in same expression 
FR11-61 - orderMeasure web service parameter to sort observations by date asc/desc 

==Version 11.5.0 (Released: 2022/09/28)==
'''New Features:''' 
FMR11-187/FR11-53 - Support recovery of structure transactions from backup on installation https://github.com/MEDIT-Org/fmr/issues/277 
FR11-52 - Support variable number of max arguments to a SQL IN statement 
FR11-58 - Support import of datafile containing mulitple datasets 

'''Improvements:''' 
FMR11-183 - Update dependencies https://github.com/MEDIT-Org/fmr/issues/277 
FR11-48 - Update third-party dependencies: apache-jstl, commons-dcp2, joda-time, spring, tika-core 
FR11-45 - Add Version Number to Provision Agreement Selector in Data Source Manager Modal 

'''Bug Fixes:''' 
FMR11-181/FR11-46 - SDMX CSV v 2.0 Data Format does not report its format https://github.com/MEDIT-Org/fmr/issues/110 
FMR11-184/FR11-49 - Support '*' for structure type in REST API https://github.com/MEDIT-Org/fmr/issues/273 
FMR11-185/FR11-51 - Failure on Compare Revision shows Stack Trace https://github.com/MEDIT-Org/fmr/issues/278 
FMR11-186/FR11-55 - SDMX-JSON Writer not outputting DSD as stub https://github.com/MEDIT-Org/fmr/issues/274 
FR11-50 - FameMapper code is creating Excel files in temporary storage which are not deleted 
FR11-54 - EDI data not fully consolidated when publishing data via web service 

==Version 11.4.3 (Released: 2022/09/13)==
'''New Features:''' 
FMR11-174 - Publication Tables: Formatting of Missing Values support ObsValue=NaN https://github.com/MEDIT-Org/fmr/issues/253 
FMR11-175 - Support Reading of Reference Metadata (with restrictions) in version 2.1 SDMX https://github.com/MEDIT-Org/fmr/issues/250 

'''Improvement:''' 
FMR11-182 - Modify ID generation strategy for importing Reference Metadata at v2.1 

'''Bug Fixes:''' 
FR11-39 - OBS_PRE_BREAK validation corrected 
FR11-43 - SDMX DSD output may include invalid attribute "isMultiLingual" 
FR11-44 - SQL Server does not delete temporary tables 
FMR11-169 - SDMX CSV v1 Data Reader does not allow header fields (row 1) to be double quoted 
FMR11-170 - Bulk Export - spelling mistake on Raw help text 
FMR11-171 - Order of revisions not as expected 
FMR11-176 - OBS_PRE_BREAK validation corrected 
FMR11-178 - Quadruplet in EDI vs the other formats in the context of data consolidation or integration https://github.com/MEDIT-Org/fmr/issues/118 
FMR11-179 - SDMX DSD output may include invalid attribute "isMultiLingual" https://github.com/MEDIT-Org/fmr/issues/255 

==Version 11.4.2 (Released: 2022/08/19)==
'''Bug Fixes:''' 
FR11-30 - Fusion Security Manager will not accept AD group names longer than 30 characters 
FR11-34 - Updated third-party dependencies 

==Version 11.4.1 (Released: 2022/08/10)==
'''Bug Fixes:''' 
FR11-21 - Issue loading data into SQL Server in Registry managed table
FR11-26 - Bulk Export - spelling mistake on Raw help text 
FR11-27 - Order of revisions not as expected 
FR11-28 - JNDI Registry Managed Database - unable to load data 
FR11-25 - SDMX CSV v1 Data Reader does not allow header fields (row 1) to be double quoted 

==Version 11.4.0 (Released: 2022/07/22)==
'''New Feature:''' 
FR11-18 - Support Reference Metadata https://wiki.sdmxcloud.org/Reference_Metadata 

'''Improvement:''' 
FMR11-146 - Dependency updates 
FMR11-159 - Add "Raw" option to Download structures and remove "Include internal settings" 
FMR11-160 - Allow a Concept Representation to be a ValueList 

'''Bug Fixes:''' 
FR11-14 - Issue loading Data Constraint with a Component element under Cube Region element 
FR11-17 - Attribute variable does not work in table heading 
FR11-19 - Increase max size of security group ID from 30 to 100 characters 
FMR11-162 - SERIES_KEY column should be reserved for series keys 
FMR11-161 - Issue loading Data Constraint with a Component element under Cube Region element 
FMR11-163 - Attribute variable does not work in table heading 
FMR11-166 - Duplicate attribute is not detected anymore in FMR 
FMR11-167 - Publication Table - Array Variables fail to save 
FMR11-168 - Error encountered whist attempting to deserialize structure 

==Version 11.3.0 (Released: 2022/07/06)==
'''New Feature:''' 
FR11-10/FMR11-153 - [[Synchronise_structural_metadata_between_environments|Environment Synchronisation - compare the structural metadata content with that from another Registry, push or pull changes]]

'''Improvement:''' 
FMR11-158 - [[Publication_Table#Footnotes|Publication Table: Footnotes support]]

'''Bug Fixes:''' 
FMR11-152 - Comma not being processed correctly in Web Service query for ItemBeans 
FMR11-154 - Unexpected structure validation error from FMR 
FMR11-155 - Structure validation errors are not detected anymore in FMR 
FMR11-156 - v2 structure REST API does not support queries for multiple items using the comma syntax 
FMR11-157 - Typo in header of Representation Maps page 

==Version 11.2.0 (Released: 2022/06/27)==
'''New Feature:''' 
[FMR11-150] Specify the validation and transformation behaviour when duplicate observations are found - [[Data_Transformation_Web_Service#HTTP_Headers|https://wiki.sdmxcloud.org/Data_Transformation_Web_Service#HTTP_Headers]] 

'''Bug Fixes:''' 
[FR11-3] Schema Query in structure format does not work 
[FR11-4] Fusion Cache ignoring parameters like normalise freq on V1 data queries 
[FR11-5] Data Point Security - allow rule against a series attribute value that does not exist in the dataset restricts all data 
[FR11-8] Comma not being processed correctly in Web Service query for ItemBeans 
[FMR11-141] Re-opening of Alternate Half year Issue 
[FMR11-143] TIME_FORMAT not being generated correctly 
[FMR11-144] Problem with Date data type when reverse engineering a DSD from a CSV dataset 
[FMR11-145] Problem with Description data type when reverse engineering a DSD from a CSV dataset 
[FMR11-147] DSD Wizard step 3 - not allowing Enumerated Representation 
[FMR11-148] Schema Query in Structure Format does not work 

'''Improvement: '''
[FMR11-140] Revert xsi:type=SiblingType, change the schema to output the Group as Sibling not SiblingType -

==Version 11.1.0 (Released: 2022/06/01)==
'''New Feature:''' 
[[Publication_Table|Publication Tables]] 
[FR-5130] [https://wiki.sdmxcloud.org/Publication_Table#Looping_Rows Publication Table - Support Dynamic Rows] 
[FR-5131] [https://wiki.sdmxcloud.org/Publication_Table#Formatting_Missing_Observations Publication Table - Format rules on cells with no data ] 
[FR-5132] [https://wiki.sdmxcloud.org/Publication_Table#Dependent_Dimension_Variable Publication Table - Rules to derive a Dimension Value from another Dimenison's value] 
[FR-5133] [https://wiki.sdmxcloud.org/Publication_Table#Anchor_Rows Publication Table - Option to output rows with no data ] 
[FR-5134] [https://wiki.sdmxcloud.org/Publication_Table_Web_Services#Materialise_Publication_Table Publication Table - Ouput Materialised Table in Excel ] 

'''Improvement:''' 
[FR-5129] - Support for MySQL 8 and above 

==Version 11.0.0 (Released: 2022/05/22)==
'''Changes and New Features:''' 
* Change to the SDMX 3.0 information model as Fusion Registry’s internal metadata object model
* SDMX 3.0 REST API support
* Backward compatibility for SDMX 2.1 and 2.0 structures and formats
* Backward compatibility with SDMX 2.1 REST API (REST API v1 specification)
* Improved data and metadata access control
* Granular control over admin rights
* Fusion Portal – scheduled and on demand data and structure loads
* Refreshable Excel data format
* JNDI database connection option
* Publication Tables - publish curated but dynamic user interactive web data tables that show precisely the statistics required even if they come from multiple different data sets

'''Breaking Changes from Fusion Registry 10:''' 
* Change to the operating database schema - upgrading from Fusion Registry 10 to Fusion Registry 11 requires a fresh install and migration of the metadata, data and settings
* Environment sync function temporarily removed
* Fusion Security support deprecated - Active Directory and LDAP are the supported authentication services in Fusion Registry 11

Change Log FR - V11 and subsequent releases

2024-02-12T09:34:45Z

Plazarou: /* Version 11.13.0 (Released: 2023/02/08) */

[[Category:Fusion Registry Change Logs]]
[[Category:ChangeLogFR]]
__NOTOC__
=Changelog for Fusion Registry Version 11=

==Version 11.13.0 (Released: 2023/02/08)==
Based on FMR 11.14.1 

'''Change:''' 
RFS-22 - Creating "flat CSV" no longer includes unecessary Sibling Group entries at the start of the file 

'''Improvements:''' 
RFS-181 - Support for SDMX-ML v3.0 Data 
RFS-39 - Update vulnerable third-party dependencies 

'''Bug Fixes:''' 
RFS-163 - Data Web Service contains duplicate entry for "Excel" 
RFS-164 - Data Web Service contains unstyled text label 
RFS-76 - "lastUpdated" parameter not working correctly on Data Query for external databases 

==Version 11.12.3 (Released: 2023/11/24)==
Based on FMR 11.13.0 

'''Bug Fix:''' 
FR11-217: Concurrent modification exception in cache

==Version 11.12.2 (Released: 2023/11/22)==
Based on FMR 11.13.0 

'''Change:''' 
FR11-216: REST query which results in no data now returns a 404 with an SDMX error message, was previously an empty dataset 

'''Bug Fix:''' 
FR11-215: Publication Tables: 'includePeriod' REST query parameter returns no data when multiple Dataflows are used and 1 has no data 

==Version 11.12.1 (Released: 2023/11/21)==
Based on FMR 11.13.0 

'''Improvement:''' 
FR11-211: Improvements to the "Test Mapping" feature 

'''Bug Fix:''' 
FR11-214: Registry can consume all connections to an External Data Store 

==Version 11.12.0 (Released: 2023/11/20)==
Based on FMR 11.13.0 

'''New Features:''' 
FR11-210: Publication Table: Calculation expression to support default value when no Observation found 
FR11-213: Publication Tables: Calculation expression to support default value if no reported value found 

'''Bug Fix:''' 
FR11-212: Publication Table: Formatting issue using partial keys - returns 400 exception 

==Version 11.11.0 (Released: 2023/11/03)==
'''New Features:''' 
FR11-206: Create aggregate series using linked Hierarchy 
FR11-207: Data Query detail parameter; added 'raw' as a new value to support export of data with no calcualted values via hierarchical aggregations 

'''Bug Fixes:''' 
FR11-208: In a Generated Reporting Template permit format modification on Observation Cells 
FR11-209: Publication Table: Issue applying formatting rules across multiple Dataflows 

==Version 11.10.1 (Released: 2023/09/15)==
'''Bug Fix:''' 
FR11-205: saveas=[filename].zip from v1 SDMX web service results in a corrupt zip 

==Version 11.10.0 (Released: 2023/09/11)==
Based on FMR 11.10.1

'''New Features:''' 
FR11-195: Publication Tables: Allow dependant variables to be overridden from the API 
FR11-197: Default response formats in v2 web service are now configurable 
FR11-203: Kafa: new topic to publish structure transaction events to 
FR11-204: Kafa: new topic to publish reference metadata events to 

'''Change:''' 
FR11-202: Structure Format response is no longer auto detected if not explicitly set (takes value from default setting) 

'''Improvement:''' 
FR11-198: Querying for data by observation attributes 

'''Bug Fixes:''' 
FR11-199: saveas=zip is not working on v2 API 
FR11-200: Publication table: Formatting cells ignores UNIT_MULT when it is reported as a dataset level attribute 
FR11-201: Publication table: Conditional Formatting being applied when condition references a Component which is not present in DSD 

==Version 11.9.6 (Released: 2023/08/25)==
'''Changes:''' 
FR11-163: Validator: "Missing Attributes" renamed to "Missing Components" and reports on missing Measures 
FR11-164: Add "Output Required" UI control to Structure Mapping 
FR110188: Adding a observation formatting rule treats comma separeted component ids as an AND operator (was OR) 

'''Improvements:''' 
FR11-146: Update Apache POI to version 5 
FR11-147: Update Commons-fileupload third-party dependency 
FR11-148: Update Spring dependencies 
FR11-149: Update Jettison third-party dependency 
FR11-150: Web Service to refresh caches 
FR11-158: Merging a Codelist should merge Locale information 
FR11-160: Update Spring and Jackson Dependencies 
FR11-161: Implment caching on publication tables 
FR11-167: Improve Mapping functionality so that it can perform Validation 
FR11-171: Portal File system sweep - allow files to be placed in root of sweep directory 
FR11-173: Update Kryo to version 5.5.0 
FR11-174: Publication Tables - not formatting numbers with scientific notation 
FR11-175: Components with Time based Mappings cannot be edited in the Structure Map Wizard 
FR11-176: Support Custom Plugins 
FR11-190: Add SDMX-JSON v2 to the UI 
FR11-191: Restore Fusion Security as an authentication mechanism 
FR11-192: FMR#481: Duplicated group IDs for schema queries in SDMX-JSON 
FR11-193: Mapped Datasets Configuration - new option "Output Required" 

'''Bug Fixes:''' 
FR11-151: Item Validity not returning most recent entry prior to today's date 
FR11-152: Subscript out of range error with FusionXL when carriage return is in the name of a structure 
FR11-153: No transactions returned on SDMX Transactions page when using date slider 
FR11-155: Publicaiton Table Editor: Dependent Variables, remove variable button does not work 
FR11-156: Publication Table availabililty algorithm modified to better calculate default values when using multiple dataflows 
FR11-157: FusionXL resolve carriage returns in concept names 
FR11-159: Cross references not updated correctly when structure already exists 
FR11-162: Update Vulnerable Dependencies for FMR 11.7.0 
FR11-165: Validating Generic Data can result in IllegalElement error 
FR11-166: FMR#418 - Conversion and mapping services fail to read SDMX CSV 2.0 format files with BOM 
FR11-168: CSV-TS data format outputting names as labels does not output labels in last Dimension/Attribute column 
FR11-169: RegEx Capture Groups for Representation Map only support 0-9 (\10 and above does not work) 

==Version 11.9.5 (Released: 2023/03/08)==
'''Change:''' 
FR11-140: Modification of interpration of SDMX-CSV V2 row https://github.com/bis-med-it/fmr/issues/349 

'''Improvement:''' 
FR11-145: Fusion Excel Export Data API now takes an additional 'attachment' parameter 

'''Bug Fixes:''' 
FR11-137: Removal of dependencies: gson, protobuf, closure, args4j 
FR11-138: Unable to load Structure File when Mapping refers to DSD with ValueList coded dimension https://github.com/bis-med-it/fmr/issues/359 
FR11-141: Caching in reference metadata requests https://github.com/bis-med-it/fmr/issues/356 
FR11-142: Parsing EDI fails to detect malformed period https://github.com/bis-med-it/fmr/issues/362 
FR11-143: EDI REL segment must be followed by ARR segment https://github.com/bis-med-it/fmr/issues/322 
FR11-144: Publication Table - Calculations which contain missing values should not output a value 

==Version 11.9.4 (Released: 2023/02/17)==
'''Bug Fixes''' 
FR11-135: Publication Table - calculated values could come back as enginnering string when standard form is requested 
FR11-136: Regression mapped data store was not mapping rows with null attribute values 

==Version 11.9.3 (Released: 2023/02/16)==
'''Bug Fixes:''' 
FR11-129: Temporary "MV" files not being cleared up after data transformation https://github.com/bis-med-it/fmr/issues/346 

'''Improvements:''' 
FR11-131: Improve Performance of large Codelists which have Item Validity https://github.com/bis-med-it/fmr/issues/353 
FR11-133: Improve linking to external datasource by auto detecting foreign key 

==Version 11.9.2 (Released: 2023/02/13)==
'''Change:''' 
FR11-132: Supported 'latest' in v2 query to prevent errors in connected tools 

==Version 11.9.1 (Released: 2023/02/10)==
'''Bug Fix:''' 
FR11-128: Security rule on OBS Attribute blocks observation when value is null 

==Version 11.9.0 (Released: 2023/02/03)==
'''Improvements:''' 
FR11-117: Provide option to skip Validation phase in Transform Web Service https://github.com/bis-med-it/fmr/issues/237 
FR11-120: Publication Table - apply number formatting to calculated cells 
FR11-121: Publication Table - Apply number scaling before calculations 

'''Change:''' 
FR11-118: Use the public fastUtil library https://github.com/bis-med-it/fmr/issues/338 

'''Bug Fixes:''' 
FR11-116: Timezone issue with REST API resulting in no data 
FR11-119: Codelists with Annotations not being correctly parsed in SDMX-JSON https://github.com/bis-med-it/fmr/issues/345 
FR11-123: EDI should prevent Quadruplet attributes in Footnote Section https://github.com/bis-med-it/fmr/issues/347 
FR11-124: EDI file missing mandatory Period Information validates successfully https://github.com/bis-med-it/fmr/issues/348 
FR11-125: Improve validation of EDI DTM segment https://github.com/bis-med-it/fmr/issues/351 
FR11-126: Publication Table - decimal places (fixed) truncated trailing zero places when locale was also set 
FR11-127: Duplicate elements in SDMX CSV result in unexpected output https://github.com/bis-med-it/fmr/issues/342 

==Version 11.8.2 (Released: 2023/01/17)==
'''Improvements:''' 
FR11-108: Prevent contradictory excel mappings (generted from Fame Metadata Manager) worksheets being uploaded to Registry 
FR11-109: Flag contradictory excel mappings (generted from Fame Metadata Manager) in generated worksheet 
FR11-134: Improve Metrics Report with IncUnmappedReport option 

'''Changes:''' 
FR11-111: Change ID Generation of Representation Map from FusionFameMapper xlsx files. 
FR11-115: Ensure Implicit Mappings in Structure Maps validate codelists for valid intersection 

'''Bug Fixes:''' 
FR11-107: Parent codes should be included in reference partial 
FR11-110: Complete Cube web service feature outputs empty measure values (not 0 values) 
FR11-112: Data export formats excel-ts and excel-series can miss observations in output series if series contains observations with no measure values and no attribute values 
FR11-113: Fix 'too much recursion' error with reverse engineer data structure with large number of Dimensions 
FR11-114: Validate submission of Structure Map's Implicit mappings to ensure Coded mappings have a valid intersection from source to target 

==Version 11.8.1 (Released: 2023/01/11)==
'''Bug Fixes:''' 
FR11-101: Unable to import Codelists with multiple levels of inheritence in same submission as structures that use inherited codes 
FR11-102: Bulk actions - detail=raw is ignored 
FR11-103: REST query: detail=raw is not applied to referenced codelists 
FR11-104: IncValid output may contain empty series when it should not be generated 
FR11-105: Annotations are ignored when importing 2.1 XML Hierarchical Codelists 
FR11-106: format=csv-series null pointer 

==Version 11.8.0 (Released: 2022/12/21)==
'''Important Note for MySQL users:''' due to changes in the distribution of the MySQL connector (change FR11-94 below), please read [[Install_MySQL#Installing_MySQL_5.7|this article]].

'''New Feature:''' 
FR11-98: Publication Table - support a scaling function applied to numerical observations

'''Improvement:''' 
FR11-93: Improve Performance of Viewing a Hierarchy https://wiki.sdmxcloud.org/Publication_Table#Formatting_Numerical_Observation_Values

'''Changes:''' 
FR11-91: Validation and Transformation should preserve original format unless output format specified 
https://wiki.sdmxcloud.org/Data_Validation_Web_Service#HTTP_Headers see Accept Header 
https://wiki.sdmxcloud.org/Data_Transformation_Web_Service#HTTP_Headers see Accept Header 
FR11-94: Remove MySQL Connector https://wiki.sdmxcloud.org/Install_MySQL#Installing_MySQL_5.7

'''Bug Fixes:''' 
FR11-89: Support dataset attribute consolidation (detection of duplicates) 
FR11-90: Observation deletions are no longer written in EDI format 
FR11-92: NPE when publishing EDI data from Command Line 
FR11-95: Publication Table - UI changes table definition (looses column) when clicking preview table 
FR11-96: Publication Table - Preview table error when table heading Rowspan > 1 
FR11-97: Publication Table - bug fix on formatting obs which removed values from output table 
FR11-99: Publication Table - Scientific Notation shows NaN in UI and is not applied to observation values 
FR11-100: Refreshable workbook - issue security challenge when connecting to private server 
FMR11-207: Transforming Data with "Delete" action can leave erroneous output in EDI format 
FMR11-217: Publication Tables: Numerical Observation, "Add Rule" button does not work 
FMR11-224: Failure to report unmapped data when using TIME_PERIOD mapping 

==Version 11.7.2 (Released: 2022/11/22)==
'''Change:''' 
FR11-84 - In SDMX CSV, SERIES_KEY and OBS_KEY will output with the quote marks around the whole key 

'''Bug Fixes:''' 
FR11-80 - Validation on Group ID should ignore case 
FR11-82 - Size of column in Portal Table not large enough to contain Data Source ID 
FR11-85 - When loading via Registry UI, Data file with Duplicate Series results in MV Store Error 
FR11-86 - Viewing a Representation Map has issue with "valid from" on Mappings 

==Version 11.7.1 (Released: 2022/11/15)==
'''New Feature:''' 
FR11-83 - Support Categorise Publication Table via the UI 

==Version 11.7.0 (Released: 2022/11/11)==
'''New Feature:''' 
FR11-78 - Publication Table Support Numerical Observation formatting 

'''Bug Fixes:''' 
FR11-79 - Fix an issue when "edi.datagroup.identifier" specified with value of asterisk 
FR11-80 - Validation on Group ID should ignore case 
FR11-81 - Data Portal - ESTAT Connector fix 

==Version 11.6.0 (Released: 2022/11/31)==
'''New Features:''' 
FR11-76 - [[Audit_Web_Services#Search_Structure_Queries|Audit Structure Queries]] 
FR11-77 - [[Audit_Web_Services#Search_Data_Queries|Audit Data Queries]] 

'''Improvements:''' 
FR11-75 - Support for extended UTF-8 characters https://github.com/MEDIT-Org/fmr/issues/297 

'''Change:''' 
FR11-65 - Update FontAwesome to use Version 6 

'''Bug Fixes:''' 
FR11-71 - Corrected issue with Dynamic Attributes 
FR11-72 - Version Drop-Down not Displayed when Selected Item on Page Load 
FR11-74 - Security Group Web Service does not output UTF-8 characters 
FR11-65 - Update Web Services "structure" page so the plus symbol is used for "latest"

==Version 11.5.2 (Released: 2022/10/14)==
'''New Feature:''' 
FMR11-194 - Publication Table - support casting dates more then once in same expression 

'''Improvement:''' 
FMR11-202 - Update dependencies: Hibernate, Spring-Kafka, Jettison and Jersey https://github.com/MEDIT-Org/fmr/issues/294 

'''Bug Fixes:''' 
FR11-66 - "Metadata Provider Scheme" not listed on Web Service -> Structures page 
FR11-67 - "Metadata Provider Wizard" does not allow Providers to start with a numeric yet the SDMX Standard allows this 
FR11-68 - "Metadata Provider Scheme" not listed on Cross References page 
FR11-69 - Attaching Reference Metadata to Items in an item scheme fails to add link (cache issue) 
FMR11-189 - Behaviour of web service resolving multiple Codelist inheritance with "details=raw" https://github.com/MEDIT-Org/fmr/issues/257 
FMR11-196 - Unable to read Excel Spreadsheet where all Frequencies are Formulas 
FMR11-197 - Request for Codelist in SDMX-JSON, "details=raw", references="children" doesn't return children as raw https://github.com/MEDIT-Org/fmr/issues/257 
FMR11-198 - Request for Codelist in SDMX-JSON, "details=raw", references="descendants" doesn't return descendants https://github.com/MEDIT-Org/fmr/issues/257 
FMR11-199 - "Metadata Provider Scheme" not listed on Web Service -> Structures page 
FMR11-200 - "Metadata Provider Wizard" does not allow Providers to start with a numeric yet the SDMX Standard allows this 
FMR11-201 - "Metadata Provider Scheme" not listed on Cross References page 
FMR11-203 - Attaching Reference Metadata to Items in an item scheme fails to add link (cache issue) 
FMR11-205 - Version Number not Displayed when Selected Item on Page Load 

==Version 11.5.1.1 (Released: 2022/10/10)==
'''Change:''' 
FR11-73 - Modify SQL to ensure GMT Timezone is used 

==Version 11.5.1 (Released: 2022/10/05)==
'''New Features:''' 
FR11-60 - Publication Table - support casting dates more then once in same expression 
FR11-61 - orderMeasure web service parameter to sort observations by date asc/desc 

==Version 11.5.0 (Released: 2022/09/28)==
'''New Features:''' 
FMR11-187/FR11-53 - Support recovery of structure transactions from backup on installation https://github.com/MEDIT-Org/fmr/issues/277 
FR11-52 - Support variable number of max arguments to a SQL IN statement 
FR11-58 - Support import of datafile containing mulitple datasets 

'''Improvements:''' 
FMR11-183 - Update dependencies https://github.com/MEDIT-Org/fmr/issues/277 
FR11-48 - Update third-party dependencies: apache-jstl, commons-dcp2, joda-time, spring, tika-core 
FR11-45 - Add Version Number to Provision Agreement Selector in Data Source Manager Modal 

'''Bug Fixes:''' 
FMR11-181/FR11-46 - SDMX CSV v 2.0 Data Format does not report its format https://github.com/MEDIT-Org/fmr/issues/110 
FMR11-184/FR11-49 - Support '*' for structure type in REST API https://github.com/MEDIT-Org/fmr/issues/273 
FMR11-185/FR11-51 - Failure on Compare Revision shows Stack Trace https://github.com/MEDIT-Org/fmr/issues/278 
FMR11-186/FR11-55 - SDMX-JSON Writer not outputting DSD as stub https://github.com/MEDIT-Org/fmr/issues/274 
FR11-50 - FameMapper code is creating Excel files in temporary storage which are not deleted 
FR11-54 - EDI data not fully consolidated when publishing data via web service 

==Version 11.4.3 (Released: 2022/09/13)==
'''New Features:''' 
FMR11-174 - Publication Tables: Formatting of Missing Values support ObsValue=NaN https://github.com/MEDIT-Org/fmr/issues/253 
FMR11-175 - Support Reading of Reference Metadata (with restrictions) in version 2.1 SDMX https://github.com/MEDIT-Org/fmr/issues/250 

'''Improvement:''' 
FMR11-182 - Modify ID generation strategy for importing Reference Metadata at v2.1 

'''Bug Fixes:''' 
FR11-39 - OBS_PRE_BREAK validation corrected 
FR11-43 - SDMX DSD output may include invalid attribute "isMultiLingual" 
FR11-44 - SQL Server does not delete temporary tables 
FMR11-169 - SDMX CSV v1 Data Reader does not allow header fields (row 1) to be double quoted 
FMR11-170 - Bulk Export - spelling mistake on Raw help text 
FMR11-171 - Order of revisions not as expected 
FMR11-176 - OBS_PRE_BREAK validation corrected 
FMR11-178 - Quadruplet in EDI vs the other formats in the context of data consolidation or integration https://github.com/MEDIT-Org/fmr/issues/118 
FMR11-179 - SDMX DSD output may include invalid attribute "isMultiLingual" https://github.com/MEDIT-Org/fmr/issues/255 

==Version 11.4.2 (Released: 2022/08/19)==
'''Bug Fixes:''' 
FR11-30 - Fusion Security Manager will not accept AD group names longer than 30 characters 
FR11-34 - Updated third-party dependencies 

==Version 11.4.1 (Released: 2022/08/10)==
'''Bug Fixes:''' 
FR11-21 - Issue loading data into SQL Server in Registry managed table
FR11-26 - Bulk Export - spelling mistake on Raw help text 
FR11-27 - Order of revisions not as expected 
FR11-28 - JNDI Registry Managed Database - unable to load data 
FR11-25 - SDMX CSV v1 Data Reader does not allow header fields (row 1) to be double quoted 

==Version 11.4.0 (Released: 2022/07/22)==
'''New Feature:''' 
FR11-18 - Support Reference Metadata https://wiki.sdmxcloud.org/Reference_Metadata 

'''Improvement:''' 
FMR11-146 - Dependency updates 
FMR11-159 - Add "Raw" option to Download structures and remove "Include internal settings" 
FMR11-160 - Allow a Concept Representation to be a ValueList 

'''Bug Fixes:''' 
FR11-14 - Issue loading Data Constraint with a Component element under Cube Region element 
FR11-17 - Attribute variable does not work in table heading 
FR11-19 - Increase max size of security group ID from 30 to 100 characters 
FMR11-162 - SERIES_KEY column should be reserved for series keys 
FMR11-161 - Issue loading Data Constraint with a Component element under Cube Region element 
FMR11-163 - Attribute variable does not work in table heading 
FMR11-166 - Duplicate attribute is not detected anymore in FMR 
FMR11-167 - Publication Table - Array Variables fail to save 
FMR11-168 - Error encountered whist attempting to deserialize structure 

==Version 11.3.0 (Released: 2022/07/06)==
'''New Feature:''' 
FR11-10/FMR11-153 - [[Synchronise_structural_metadata_between_environments|Environment Synchronisation - compare the structural metadata content with that from another Registry, push or pull changes]]

'''Improvement:''' 
FMR11-158 - [[Publication_Table#Footnotes|Publication Table: Footnotes support]]

'''Bug Fixes:''' 
FMR11-152 - Comma not being processed correctly in Web Service query for ItemBeans 
FMR11-154 - Unexpected structure validation error from FMR 
FMR11-155 - Structure validation errors are not detected anymore in FMR 
FMR11-156 - v2 structure REST API does not support queries for multiple items using the comma syntax 
FMR11-157 - Typo in header of Representation Maps page 

==Version 11.2.0 (Released: 2022/06/27)==
'''New Feature:''' 
[FMR11-150] Specify the validation and transformation behaviour when duplicate observations are found - [[Data_Transformation_Web_Service#HTTP_Headers|https://wiki.sdmxcloud.org/Data_Transformation_Web_Service#HTTP_Headers]] 

'''Bug Fixes:''' 
[FR11-3] Schema Query in structure format does not work 
[FR11-4] Fusion Cache ignoring parameters like normalise freq on V1 data queries 
[FR11-5] Data Point Security - allow rule against a series attribute value that does not exist in the dataset restricts all data 
[FR11-8] Comma not being processed correctly in Web Service query for ItemBeans 
[FMR11-141] Re-opening of Alternate Half year Issue 
[FMR11-143] TIME_FORMAT not being generated correctly 
[FMR11-144] Problem with Date data type when reverse engineering a DSD from a CSV dataset 
[FMR11-145] Problem with Description data type when reverse engineering a DSD from a CSV dataset 
[FMR11-147] DSD Wizard step 3 - not allowing Enumerated Representation 
[FMR11-148] Schema Query in Structure Format does not work 

'''Improvement: '''
[FMR11-140] Revert xsi:type=SiblingType, change the schema to output the Group as Sibling not SiblingType -

==Version 11.1.0 (Released: 2022/06/01)==
'''New Feature:''' 
[[Publication_Table|Publication Tables]] 
[FR-5130] [https://wiki.sdmxcloud.org/Publication_Table#Looping_Rows Publication Table - Support Dynamic Rows] 
[FR-5131] [https://wiki.sdmxcloud.org/Publication_Table#Formatting_Missing_Observations Publication Table - Format rules on cells with no data ] 
[FR-5132] [https://wiki.sdmxcloud.org/Publication_Table#Dependent_Dimension_Variable Publication Table - Rules to derive a Dimension Value from another Dimenison's value] 
[FR-5133] [https://wiki.sdmxcloud.org/Publication_Table#Anchor_Rows Publication Table - Option to output rows with no data ] 
[FR-5134] [https://wiki.sdmxcloud.org/Publication_Table_Web_Services#Materialise_Publication_Table Publication Table - Ouput Materialised Table in Excel ] 

'''Improvement:''' 
[FR-5129] - Support for MySQL 8 and above 

==Version 11.0.0 (Released: 2022/05/22)==
'''Changes and New Features:''' 
* Change to the SDMX 3.0 information model as Fusion Registry’s internal metadata object model
* SDMX 3.0 REST API support
* Backward compatibility for SDMX 2.1 and 2.0 structures and formats
* Backward compatibility with SDMX 2.1 REST API (REST API v1 specification)
* Improved data and metadata access control
* Granular control over admin rights
* Fusion Portal – scheduled and on demand data and structure loads
* Refreshable Excel data format
* JNDI database connection option
* Publication Tables - publish curated but dynamic user interactive web data tables that show precisely the statistics required even if they come from multiple different data sets

'''Breaking Changes from Fusion Registry 10:''' 
* Change to the operating database schema - upgrading from Fusion Registry 10 to Fusion Registry 11 requires a fresh install and migration of the metadata, data and settings
* Environment sync function temporarily removed
* Fusion Security support deprecated - Active Directory and LDAP are the supported authentication services in Fusion Registry 11

Change Log FR - V11 and subsequent releases

2024-02-12T09:31:14Z

Plazarou: /* Version 11.13.0 (Released: 2023/02/12) */

[[Category:Fusion Registry Change Logs]]
[[Category:ChangeLogFR]]
__NOTOC__
=Changelog for Fusion Registry Version 11=

==Version 11.13.0 (Released: 2023/02/08)==
Based on FMR 11.14.1 

'''Change:''' 
RFS-22 - Creating "flat CSV" with unecessary Sibling Group entries at the start of the file 

'''Improvements:''' 
RFS-181 - Support for SDMX-ML v3.0 Data 
RFS-39 - Fix software vulnerabilities 

'''Bug Fixes:''' 
RFS-163 - Data Web Service contains duplicate entry for "Excel" 
RFS-164 - Data Web Service contains unstyled text label 
RFS-76 - "lastUpdated" parameter not working correctly on Data Query for external databases 

==Version 11.12.3 (Released: 2023/11/24)==
Based on FMR 11.13.0 

'''Bug Fix:''' 
FR11-217: Concurrent modification exception in cache

==Version 11.12.2 (Released: 2023/11/22)==
Based on FMR 11.13.0 

'''Change:''' 
FR11-216: REST query which results in no data now returns a 404 with an SDMX error message, was previously an empty dataset 

'''Bug Fix:''' 
FR11-215: Publication Tables: 'includePeriod' REST query parameter returns no data when multiple Dataflows are used and 1 has no data 

==Version 11.12.1 (Released: 2023/11/21)==
Based on FMR 11.13.0 

'''Improvement:''' 
FR11-211: Improvements to the "Test Mapping" feature 

'''Bug Fix:''' 
FR11-214: Registry can consume all connections to an External Data Store 

==Version 11.12.0 (Released: 2023/11/20)==
Based on FMR 11.13.0 

'''New Features:''' 
FR11-210: Publication Table: Calculation expression to support default value when no Observation found 
FR11-213: Publication Tables: Calculation expression to support default value if no reported value found 

'''Bug Fix:''' 
FR11-212: Publication Table: Formatting issue using partial keys - returns 400 exception 

==Version 11.11.0 (Released: 2023/11/03)==
'''New Features:''' 
FR11-206: Create aggregate series using linked Hierarchy 
FR11-207: Data Query detail parameter; added 'raw' as a new value to support export of data with no calcualted values via hierarchical aggregations 

'''Bug Fixes:''' 
FR11-208: In a Generated Reporting Template permit format modification on Observation Cells 
FR11-209: Publication Table: Issue applying formatting rules across multiple Dataflows 

==Version 11.10.1 (Released: 2023/09/15)==
'''Bug Fix:''' 
FR11-205: saveas=[filename].zip from v1 SDMX web service results in a corrupt zip 

==Version 11.10.0 (Released: 2023/09/11)==
Based on FMR 11.10.1

'''New Features:''' 
FR11-195: Publication Tables: Allow dependant variables to be overridden from the API 
FR11-197: Default response formats in v2 web service are now configurable 
FR11-203: Kafa: new topic to publish structure transaction events to 
FR11-204: Kafa: new topic to publish reference metadata events to 

'''Change:''' 
FR11-202: Structure Format response is no longer auto detected if not explicitly set (takes value from default setting) 

'''Improvement:''' 
FR11-198: Querying for data by observation attributes 

'''Bug Fixes:''' 
FR11-199: saveas=zip is not working on v2 API 
FR11-200: Publication table: Formatting cells ignores UNIT_MULT when it is reported as a dataset level attribute 
FR11-201: Publication table: Conditional Formatting being applied when condition references a Component which is not present in DSD 

==Version 11.9.6 (Released: 2023/08/25)==
'''Changes:''' 
FR11-163: Validator: "Missing Attributes" renamed to "Missing Components" and reports on missing Measures 
FR11-164: Add "Output Required" UI control to Structure Mapping 
FR110188: Adding a observation formatting rule treats comma separeted component ids as an AND operator (was OR) 

'''Improvements:''' 
FR11-146: Update Apache POI to version 5 
FR11-147: Update Commons-fileupload third-party dependency 
FR11-148: Update Spring dependencies 
FR11-149: Update Jettison third-party dependency 
FR11-150: Web Service to refresh caches 
FR11-158: Merging a Codelist should merge Locale information 
FR11-160: Update Spring and Jackson Dependencies 
FR11-161: Implment caching on publication tables 
FR11-167: Improve Mapping functionality so that it can perform Validation 
FR11-171: Portal File system sweep - allow files to be placed in root of sweep directory 
FR11-173: Update Kryo to version 5.5.0 
FR11-174: Publication Tables - not formatting numbers with scientific notation 
FR11-175: Components with Time based Mappings cannot be edited in the Structure Map Wizard 
FR11-176: Support Custom Plugins 
FR11-190: Add SDMX-JSON v2 to the UI 
FR11-191: Restore Fusion Security as an authentication mechanism 
FR11-192: FMR#481: Duplicated group IDs for schema queries in SDMX-JSON 
FR11-193: Mapped Datasets Configuration - new option "Output Required" 

'''Bug Fixes:''' 
FR11-151: Item Validity not returning most recent entry prior to today's date 
FR11-152: Subscript out of range error with FusionXL when carriage return is in the name of a structure 
FR11-153: No transactions returned on SDMX Transactions page when using date slider 
FR11-155: Publicaiton Table Editor: Dependent Variables, remove variable button does not work 
FR11-156: Publication Table availabililty algorithm modified to better calculate default values when using multiple dataflows 
FR11-157: FusionXL resolve carriage returns in concept names 
FR11-159: Cross references not updated correctly when structure already exists 
FR11-162: Update Vulnerable Dependencies for FMR 11.7.0 
FR11-165: Validating Generic Data can result in IllegalElement error 
FR11-166: FMR#418 - Conversion and mapping services fail to read SDMX CSV 2.0 format files with BOM 
FR11-168: CSV-TS data format outputting names as labels does not output labels in last Dimension/Attribute column 
FR11-169: RegEx Capture Groups for Representation Map only support 0-9 (\10 and above does not work) 

==Version 11.9.5 (Released: 2023/03/08)==
'''Change:''' 
FR11-140: Modification of interpration of SDMX-CSV V2 row https://github.com/bis-med-it/fmr/issues/349 

'''Improvement:''' 
FR11-145: Fusion Excel Export Data API now takes an additional 'attachment' parameter 

'''Bug Fixes:''' 
FR11-137: Removal of dependencies: gson, protobuf, closure, args4j 
FR11-138: Unable to load Structure File when Mapping refers to DSD with ValueList coded dimension https://github.com/bis-med-it/fmr/issues/359 
FR11-141: Caching in reference metadata requests https://github.com/bis-med-it/fmr/issues/356 
FR11-142: Parsing EDI fails to detect malformed period https://github.com/bis-med-it/fmr/issues/362 
FR11-143: EDI REL segment must be followed by ARR segment https://github.com/bis-med-it/fmr/issues/322 
FR11-144: Publication Table - Calculations which contain missing values should not output a value 

==Version 11.9.4 (Released: 2023/02/17)==
'''Bug Fixes''' 
FR11-135: Publication Table - calculated values could come back as enginnering string when standard form is requested 
FR11-136: Regression mapped data store was not mapping rows with null attribute values 

==Version 11.9.3 (Released: 2023/02/16)==
'''Bug Fixes:''' 
FR11-129: Temporary "MV" files not being cleared up after data transformation https://github.com/bis-med-it/fmr/issues/346 

'''Improvements:''' 
FR11-131: Improve Performance of large Codelists which have Item Validity https://github.com/bis-med-it/fmr/issues/353 
FR11-133: Improve linking to external datasource by auto detecting foreign key 

==Version 11.9.2 (Released: 2023/02/13)==
'''Change:''' 
FR11-132: Supported 'latest' in v2 query to prevent errors in connected tools 

==Version 11.9.1 (Released: 2023/02/10)==
'''Bug Fix:''' 
FR11-128: Security rule on OBS Attribute blocks observation when value is null 

==Version 11.9.0 (Released: 2023/02/03)==
'''Improvements:''' 
FR11-117: Provide option to skip Validation phase in Transform Web Service https://github.com/bis-med-it/fmr/issues/237 
FR11-120: Publication Table - apply number formatting to calculated cells 
FR11-121: Publication Table - Apply number scaling before calculations 

'''Change:''' 
FR11-118: Use the public fastUtil library https://github.com/bis-med-it/fmr/issues/338 

'''Bug Fixes:''' 
FR11-116: Timezone issue with REST API resulting in no data 
FR11-119: Codelists with Annotations not being correctly parsed in SDMX-JSON https://github.com/bis-med-it/fmr/issues/345 
FR11-123: EDI should prevent Quadruplet attributes in Footnote Section https://github.com/bis-med-it/fmr/issues/347 
FR11-124: EDI file missing mandatory Period Information validates successfully https://github.com/bis-med-it/fmr/issues/348 
FR11-125: Improve validation of EDI DTM segment https://github.com/bis-med-it/fmr/issues/351 
FR11-126: Publication Table - decimal places (fixed) truncated trailing zero places when locale was also set 
FR11-127: Duplicate elements in SDMX CSV result in unexpected output https://github.com/bis-med-it/fmr/issues/342 

==Version 11.8.2 (Released: 2023/01/17)==
'''Improvements:''' 
FR11-108: Prevent contradictory excel mappings (generted from Fame Metadata Manager) worksheets being uploaded to Registry 
FR11-109: Flag contradictory excel mappings (generted from Fame Metadata Manager) in generated worksheet 
FR11-134: Improve Metrics Report with IncUnmappedReport option 

'''Changes:''' 
FR11-111: Change ID Generation of Representation Map from FusionFameMapper xlsx files. 
FR11-115: Ensure Implicit Mappings in Structure Maps validate codelists for valid intersection 

'''Bug Fixes:''' 
FR11-107: Parent codes should be included in reference partial 
FR11-110: Complete Cube web service feature outputs empty measure values (not 0 values) 
FR11-112: Data export formats excel-ts and excel-series can miss observations in output series if series contains observations with no measure values and no attribute values 
FR11-113: Fix 'too much recursion' error with reverse engineer data structure with large number of Dimensions 
FR11-114: Validate submission of Structure Map's Implicit mappings to ensure Coded mappings have a valid intersection from source to target 

==Version 11.8.1 (Released: 2023/01/11)==
'''Bug Fixes:''' 
FR11-101: Unable to import Codelists with multiple levels of inheritence in same submission as structures that use inherited codes 
FR11-102: Bulk actions - detail=raw is ignored 
FR11-103: REST query: detail=raw is not applied to referenced codelists 
FR11-104: IncValid output may contain empty series when it should not be generated 
FR11-105: Annotations are ignored when importing 2.1 XML Hierarchical Codelists 
FR11-106: format=csv-series null pointer 

==Version 11.8.0 (Released: 2022/12/21)==
'''Important Note for MySQL users:''' due to changes in the distribution of the MySQL connector (change FR11-94 below), please read [[Install_MySQL#Installing_MySQL_5.7|this article]].

'''New Feature:''' 
FR11-98: Publication Table - support a scaling function applied to numerical observations

'''Improvement:''' 
FR11-93: Improve Performance of Viewing a Hierarchy https://wiki.sdmxcloud.org/Publication_Table#Formatting_Numerical_Observation_Values

'''Changes:''' 
FR11-91: Validation and Transformation should preserve original format unless output format specified 
https://wiki.sdmxcloud.org/Data_Validation_Web_Service#HTTP_Headers see Accept Header 
https://wiki.sdmxcloud.org/Data_Transformation_Web_Service#HTTP_Headers see Accept Header 
FR11-94: Remove MySQL Connector https://wiki.sdmxcloud.org/Install_MySQL#Installing_MySQL_5.7

'''Bug Fixes:''' 
FR11-89: Support dataset attribute consolidation (detection of duplicates) 
FR11-90: Observation deletions are no longer written in EDI format 
FR11-92: NPE when publishing EDI data from Command Line 
FR11-95: Publication Table - UI changes table definition (looses column) when clicking preview table 
FR11-96: Publication Table - Preview table error when table heading Rowspan > 1 
FR11-97: Publication Table - bug fix on formatting obs which removed values from output table 
FR11-99: Publication Table - Scientific Notation shows NaN in UI and is not applied to observation values 
FR11-100: Refreshable workbook - issue security challenge when connecting to private server 
FMR11-207: Transforming Data with "Delete" action can leave erroneous output in EDI format 
FMR11-217: Publication Tables: Numerical Observation, "Add Rule" button does not work 
FMR11-224: Failure to report unmapped data when using TIME_PERIOD mapping 

==Version 11.7.2 (Released: 2022/11/22)==
'''Change:''' 
FR11-84 - In SDMX CSV, SERIES_KEY and OBS_KEY will output with the quote marks around the whole key 

'''Bug Fixes:''' 
FR11-80 - Validation on Group ID should ignore case 
FR11-82 - Size of column in Portal Table not large enough to contain Data Source ID 
FR11-85 - When loading via Registry UI, Data file with Duplicate Series results in MV Store Error 
FR11-86 - Viewing a Representation Map has issue with "valid from" on Mappings 

==Version 11.7.1 (Released: 2022/11/15)==
'''New Feature:''' 
FR11-83 - Support Categorise Publication Table via the UI 

==Version 11.7.0 (Released: 2022/11/11)==
'''New Feature:''' 
FR11-78 - Publication Table Support Numerical Observation formatting 

'''Bug Fixes:''' 
FR11-79 - Fix an issue when "edi.datagroup.identifier" specified with value of asterisk 
FR11-80 - Validation on Group ID should ignore case 
FR11-81 - Data Portal - ESTAT Connector fix 

==Version 11.6.0 (Released: 2022/11/31)==
'''New Features:''' 
FR11-76 - [[Audit_Web_Services#Search_Structure_Queries|Audit Structure Queries]] 
FR11-77 - [[Audit_Web_Services#Search_Data_Queries|Audit Data Queries]] 

'''Improvements:''' 
FR11-75 - Support for extended UTF-8 characters https://github.com/MEDIT-Org/fmr/issues/297 

'''Change:''' 
FR11-65 - Update FontAwesome to use Version 6 

'''Bug Fixes:''' 
FR11-71 - Corrected issue with Dynamic Attributes 
FR11-72 - Version Drop-Down not Displayed when Selected Item on Page Load 
FR11-74 - Security Group Web Service does not output UTF-8 characters 
FR11-65 - Update Web Services "structure" page so the plus symbol is used for "latest"

==Version 11.5.2 (Released: 2022/10/14)==
'''New Feature:''' 
FMR11-194 - Publication Table - support casting dates more then once in same expression 

'''Improvement:''' 
FMR11-202 - Update dependencies: Hibernate, Spring-Kafka, Jettison and Jersey https://github.com/MEDIT-Org/fmr/issues/294 

'''Bug Fixes:''' 
FR11-66 - "Metadata Provider Scheme" not listed on Web Service -> Structures page 
FR11-67 - "Metadata Provider Wizard" does not allow Providers to start with a numeric yet the SDMX Standard allows this 
FR11-68 - "Metadata Provider Scheme" not listed on Cross References page 
FR11-69 - Attaching Reference Metadata to Items in an item scheme fails to add link (cache issue) 
FMR11-189 - Behaviour of web service resolving multiple Codelist inheritance with "details=raw" https://github.com/MEDIT-Org/fmr/issues/257 
FMR11-196 - Unable to read Excel Spreadsheet where all Frequencies are Formulas 
FMR11-197 - Request for Codelist in SDMX-JSON, "details=raw", references="children" doesn't return children as raw https://github.com/MEDIT-Org/fmr/issues/257 
FMR11-198 - Request for Codelist in SDMX-JSON, "details=raw", references="descendants" doesn't return descendants https://github.com/MEDIT-Org/fmr/issues/257 
FMR11-199 - "Metadata Provider Scheme" not listed on Web Service -> Structures page 
FMR11-200 - "Metadata Provider Wizard" does not allow Providers to start with a numeric yet the SDMX Standard allows this 
FMR11-201 - "Metadata Provider Scheme" not listed on Cross References page 
FMR11-203 - Attaching Reference Metadata to Items in an item scheme fails to add link (cache issue) 
FMR11-205 - Version Number not Displayed when Selected Item on Page Load 

==Version 11.5.1.1 (Released: 2022/10/10)==
'''Change:''' 
FR11-73 - Modify SQL to ensure GMT Timezone is used 

==Version 11.5.1 (Released: 2022/10/05)==
'''New Features:''' 
FR11-60 - Publication Table - support casting dates more then once in same expression 
FR11-61 - orderMeasure web service parameter to sort observations by date asc/desc 

==Version 11.5.0 (Released: 2022/09/28)==
'''New Features:''' 
FMR11-187/FR11-53 - Support recovery of structure transactions from backup on installation https://github.com/MEDIT-Org/fmr/issues/277 
FR11-52 - Support variable number of max arguments to a SQL IN statement 
FR11-58 - Support import of datafile containing mulitple datasets 

'''Improvements:''' 
FMR11-183 - Update dependencies https://github.com/MEDIT-Org/fmr/issues/277 
FR11-48 - Update third-party dependencies: apache-jstl, commons-dcp2, joda-time, spring, tika-core 
FR11-45 - Add Version Number to Provision Agreement Selector in Data Source Manager Modal 

'''Bug Fixes:''' 
FMR11-181/FR11-46 - SDMX CSV v 2.0 Data Format does not report its format https://github.com/MEDIT-Org/fmr/issues/110 
FMR11-184/FR11-49 - Support '*' for structure type in REST API https://github.com/MEDIT-Org/fmr/issues/273 
FMR11-185/FR11-51 - Failure on Compare Revision shows Stack Trace https://github.com/MEDIT-Org/fmr/issues/278 
FMR11-186/FR11-55 - SDMX-JSON Writer not outputting DSD as stub https://github.com/MEDIT-Org/fmr/issues/274 
FR11-50 - FameMapper code is creating Excel files in temporary storage which are not deleted 
FR11-54 - EDI data not fully consolidated when publishing data via web service 

==Version 11.4.3 (Released: 2022/09/13)==
'''New Features:''' 
FMR11-174 - Publication Tables: Formatting of Missing Values support ObsValue=NaN https://github.com/MEDIT-Org/fmr/issues/253 
FMR11-175 - Support Reading of Reference Metadata (with restrictions) in version 2.1 SDMX https://github.com/MEDIT-Org/fmr/issues/250 

'''Improvement:''' 
FMR11-182 - Modify ID generation strategy for importing Reference Metadata at v2.1 

'''Bug Fixes:''' 
FR11-39 - OBS_PRE_BREAK validation corrected 
FR11-43 - SDMX DSD output may include invalid attribute "isMultiLingual" 
FR11-44 - SQL Server does not delete temporary tables 
FMR11-169 - SDMX CSV v1 Data Reader does not allow header fields (row 1) to be double quoted 
FMR11-170 - Bulk Export - spelling mistake on Raw help text 
FMR11-171 - Order of revisions not as expected 
FMR11-176 - OBS_PRE_BREAK validation corrected 
FMR11-178 - Quadruplet in EDI vs the other formats in the context of data consolidation or integration https://github.com/MEDIT-Org/fmr/issues/118 
FMR11-179 - SDMX DSD output may include invalid attribute "isMultiLingual" https://github.com/MEDIT-Org/fmr/issues/255 

==Version 11.4.2 (Released: 2022/08/19)==
'''Bug Fixes:''' 
FR11-30 - Fusion Security Manager will not accept AD group names longer than 30 characters 
FR11-34 - Updated third-party dependencies 

==Version 11.4.1 (Released: 2022/08/10)==
'''Bug Fixes:''' 
FR11-21 - Issue loading data into SQL Server in Registry managed table
FR11-26 - Bulk Export - spelling mistake on Raw help text 
FR11-27 - Order of revisions not as expected 
FR11-28 - JNDI Registry Managed Database - unable to load data 
FR11-25 - SDMX CSV v1 Data Reader does not allow header fields (row 1) to be double quoted 

==Version 11.4.0 (Released: 2022/07/22)==
'''New Feature:''' 
FR11-18 - Support Reference Metadata https://wiki.sdmxcloud.org/Reference_Metadata 

'''Improvement:''' 
FMR11-146 - Dependency updates 
FMR11-159 - Add "Raw" option to Download structures and remove "Include internal settings" 
FMR11-160 - Allow a Concept Representation to be a ValueList 

'''Bug Fixes:''' 
FR11-14 - Issue loading Data Constraint with a Component element under Cube Region element 
FR11-17 - Attribute variable does not work in table heading 
FR11-19 - Increase max size of security group ID from 30 to 100 characters 
FMR11-162 - SERIES_KEY column should be reserved for series keys 
FMR11-161 - Issue loading Data Constraint with a Component element under Cube Region element 
FMR11-163 - Attribute variable does not work in table heading 
FMR11-166 - Duplicate attribute is not detected anymore in FMR 
FMR11-167 - Publication Table - Array Variables fail to save 
FMR11-168 - Error encountered whist attempting to deserialize structure 

==Version 11.3.0 (Released: 2022/07/06)==
'''New Feature:''' 
FR11-10/FMR11-153 - [[Synchronise_structural_metadata_between_environments|Environment Synchronisation - compare the structural metadata content with that from another Registry, push or pull changes]]

'''Improvement:''' 
FMR11-158 - [[Publication_Table#Footnotes|Publication Table: Footnotes support]]

'''Bug Fixes:''' 
FMR11-152 - Comma not being processed correctly in Web Service query for ItemBeans 
FMR11-154 - Unexpected structure validation error from FMR 
FMR11-155 - Structure validation errors are not detected anymore in FMR 
FMR11-156 - v2 structure REST API does not support queries for multiple items using the comma syntax 
FMR11-157 - Typo in header of Representation Maps page 

==Version 11.2.0 (Released: 2022/06/27)==
'''New Feature:''' 
[FMR11-150] Specify the validation and transformation behaviour when duplicate observations are found - [[Data_Transformation_Web_Service#HTTP_Headers|https://wiki.sdmxcloud.org/Data_Transformation_Web_Service#HTTP_Headers]] 

'''Bug Fixes:''' 
[FR11-3] Schema Query in structure format does not work 
[FR11-4] Fusion Cache ignoring parameters like normalise freq on V1 data queries 
[FR11-5] Data Point Security - allow rule against a series attribute value that does not exist in the dataset restricts all data 
[FR11-8] Comma not being processed correctly in Web Service query for ItemBeans 
[FMR11-141] Re-opening of Alternate Half year Issue 
[FMR11-143] TIME_FORMAT not being generated correctly 
[FMR11-144] Problem with Date data type when reverse engineering a DSD from a CSV dataset 
[FMR11-145] Problem with Description data type when reverse engineering a DSD from a CSV dataset 
[FMR11-147] DSD Wizard step 3 - not allowing Enumerated Representation 
[FMR11-148] Schema Query in Structure Format does not work 

'''Improvement: '''
[FMR11-140] Revert xsi:type=SiblingType, change the schema to output the Group as Sibling not SiblingType -

==Version 11.1.0 (Released: 2022/06/01)==
'''New Feature:''' 
[[Publication_Table|Publication Tables]] 
[FR-5130] [https://wiki.sdmxcloud.org/Publication_Table#Looping_Rows Publication Table - Support Dynamic Rows] 
[FR-5131] [https://wiki.sdmxcloud.org/Publication_Table#Formatting_Missing_Observations Publication Table - Format rules on cells with no data ] 
[FR-5132] [https://wiki.sdmxcloud.org/Publication_Table#Dependent_Dimension_Variable Publication Table - Rules to derive a Dimension Value from another Dimenison's value] 
[FR-5133] [https://wiki.sdmxcloud.org/Publication_Table#Anchor_Rows Publication Table - Option to output rows with no data ] 
[FR-5134] [https://wiki.sdmxcloud.org/Publication_Table_Web_Services#Materialise_Publication_Table Publication Table - Ouput Materialised Table in Excel ] 

'''Improvement:''' 
[FR-5129] - Support for MySQL 8 and above 

==Version 11.0.0 (Released: 2022/05/22)==
'''Changes and New Features:''' 
* Change to the SDMX 3.0 information model as Fusion Registry’s internal metadata object model
* SDMX 3.0 REST API support
* Backward compatibility for SDMX 2.1 and 2.0 structures and formats
* Backward compatibility with SDMX 2.1 REST API (REST API v1 specification)
* Improved data and metadata access control
* Granular control over admin rights
* Fusion Portal – scheduled and on demand data and structure loads
* Refreshable Excel data format
* JNDI database connection option
* Publication Tables - publish curated but dynamic user interactive web data tables that show precisely the statistics required even if they come from multiple different data sets

'''Breaking Changes from Fusion Registry 10:''' 
* Change to the operating database schema - upgrading from Fusion Registry 10 to Fusion Registry 11 requires a fresh install and migration of the metadata, data and settings
* Environment sync function temporarily removed
* Fusion Security support deprecated - Active Directory and LDAP are the supported authentication services in Fusion Registry 11

Change Log FR - V11 and subsequent releases

2024-02-12T09:12:49Z

Plazarou:

[[Category:Fusion Registry Change Logs]]
[[Category:ChangeLogFR]]
__NOTOC__
=Changelog for Fusion Registry Version 11=

==Version 11.13.0 (Released: 2023/02/12)==
Based on FMR 11.14.1 

==Version 11.12.3 (Released: 2023/11/24)==
Based on FMR 11.13.0 

'''Bug Fix:''' 
FR11-217: Concurrent modification exception in cache

==Version 11.12.2 (Released: 2023/11/22)==
Based on FMR 11.13.0 

'''Change:''' 
FR11-216: REST query which results in no data now returns a 404 with an SDMX error message, was previously an empty dataset 

'''Bug Fix:''' 
FR11-215: Publication Tables: 'includePeriod' REST query parameter returns no data when multiple Dataflows are used and 1 has no data 

==Version 11.12.1 (Released: 2023/11/21)==
Based on FMR 11.13.0 

'''Improvement:''' 
FR11-211: Improvements to the "Test Mapping" feature 

'''Bug Fix:''' 
FR11-214: Registry can consume all connections to an External Data Store 

==Version 11.12.0 (Released: 2023/11/20)==
Based on FMR 11.13.0 

'''New Features:''' 
FR11-210: Publication Table: Calculation expression to support default value when no Observation found 
FR11-213: Publication Tables: Calculation expression to support default value if no reported value found 

'''Bug Fix:''' 
FR11-212: Publication Table: Formatting issue using partial keys - returns 400 exception 

==Version 11.11.0 (Released: 2023/11/03)==
'''New Features:''' 
FR11-206: Create aggregate series using linked Hierarchy 
FR11-207: Data Query detail parameter; added 'raw' as a new value to support export of data with no calcualted values via hierarchical aggregations 

'''Bug Fixes:''' 
FR11-208: In a Generated Reporting Template permit format modification on Observation Cells 
FR11-209: Publication Table: Issue applying formatting rules across multiple Dataflows 

==Version 11.10.1 (Released: 2023/09/15)==
'''Bug Fix:''' 
FR11-205: saveas=[filename].zip from v1 SDMX web service results in a corrupt zip 

==Version 11.10.0 (Released: 2023/09/11)==
Based on FMR 11.10.1

'''New Features:''' 
FR11-195: Publication Tables: Allow dependant variables to be overridden from the API 
FR11-197: Default response formats in v2 web service are now configurable 
FR11-203: Kafa: new topic to publish structure transaction events to 
FR11-204: Kafa: new topic to publish reference metadata events to 

'''Change:''' 
FR11-202: Structure Format response is no longer auto detected if not explicitly set (takes value from default setting) 

'''Improvement:''' 
FR11-198: Querying for data by observation attributes 

'''Bug Fixes:''' 
FR11-199: saveas=zip is not working on v2 API 
FR11-200: Publication table: Formatting cells ignores UNIT_MULT when it is reported as a dataset level attribute 
FR11-201: Publication table: Conditional Formatting being applied when condition references a Component which is not present in DSD 

==Version 11.9.6 (Released: 2023/08/25)==
'''Changes:''' 
FR11-163: Validator: "Missing Attributes" renamed to "Missing Components" and reports on missing Measures 
FR11-164: Add "Output Required" UI control to Structure Mapping 
FR110188: Adding a observation formatting rule treats comma separeted component ids as an AND operator (was OR) 

'''Improvements:''' 
FR11-146: Update Apache POI to version 5 
FR11-147: Update Commons-fileupload third-party dependency 
FR11-148: Update Spring dependencies 
FR11-149: Update Jettison third-party dependency 
FR11-150: Web Service to refresh caches 
FR11-158: Merging a Codelist should merge Locale information 
FR11-160: Update Spring and Jackson Dependencies 
FR11-161: Implment caching on publication tables 
FR11-167: Improve Mapping functionality so that it can perform Validation 
FR11-171: Portal File system sweep - allow files to be placed in root of sweep directory 
FR11-173: Update Kryo to version 5.5.0 
FR11-174: Publication Tables - not formatting numbers with scientific notation 
FR11-175: Components with Time based Mappings cannot be edited in the Structure Map Wizard 
FR11-176: Support Custom Plugins 
FR11-190: Add SDMX-JSON v2 to the UI 
FR11-191: Restore Fusion Security as an authentication mechanism 
FR11-192: FMR#481: Duplicated group IDs for schema queries in SDMX-JSON 
FR11-193: Mapped Datasets Configuration - new option "Output Required" 

'''Bug Fixes:''' 
FR11-151: Item Validity not returning most recent entry prior to today's date 
FR11-152: Subscript out of range error with FusionXL when carriage return is in the name of a structure 
FR11-153: No transactions returned on SDMX Transactions page when using date slider 
FR11-155: Publicaiton Table Editor: Dependent Variables, remove variable button does not work 
FR11-156: Publication Table availabililty algorithm modified to better calculate default values when using multiple dataflows 
FR11-157: FusionXL resolve carriage returns in concept names 
FR11-159: Cross references not updated correctly when structure already exists 
FR11-162: Update Vulnerable Dependencies for FMR 11.7.0 
FR11-165: Validating Generic Data can result in IllegalElement error 
FR11-166: FMR#418 - Conversion and mapping services fail to read SDMX CSV 2.0 format files with BOM 
FR11-168: CSV-TS data format outputting names as labels does not output labels in last Dimension/Attribute column 
FR11-169: RegEx Capture Groups for Representation Map only support 0-9 (\10 and above does not work) 

==Version 11.9.5 (Released: 2023/03/08)==
'''Change:''' 
FR11-140: Modification of interpration of SDMX-CSV V2 row https://github.com/bis-med-it/fmr/issues/349 

'''Improvement:''' 
FR11-145: Fusion Excel Export Data API now takes an additional 'attachment' parameter 

'''Bug Fixes:''' 
FR11-137: Removal of dependencies: gson, protobuf, closure, args4j 
FR11-138: Unable to load Structure File when Mapping refers to DSD with ValueList coded dimension https://github.com/bis-med-it/fmr/issues/359 
FR11-141: Caching in reference metadata requests https://github.com/bis-med-it/fmr/issues/356 
FR11-142: Parsing EDI fails to detect malformed period https://github.com/bis-med-it/fmr/issues/362 
FR11-143: EDI REL segment must be followed by ARR segment https://github.com/bis-med-it/fmr/issues/322 
FR11-144: Publication Table - Calculations which contain missing values should not output a value 

==Version 11.9.4 (Released: 2023/02/17)==
'''Bug Fixes''' 
FR11-135: Publication Table - calculated values could come back as enginnering string when standard form is requested 
FR11-136: Regression mapped data store was not mapping rows with null attribute values 

==Version 11.9.3 (Released: 2023/02/16)==
'''Bug Fixes:''' 
FR11-129: Temporary "MV" files not being cleared up after data transformation https://github.com/bis-med-it/fmr/issues/346 

'''Improvements:''' 
FR11-131: Improve Performance of large Codelists which have Item Validity https://github.com/bis-med-it/fmr/issues/353 
FR11-133: Improve linking to external datasource by auto detecting foreign key 

==Version 11.9.2 (Released: 2023/02/13)==
'''Change:''' 
FR11-132: Supported 'latest' in v2 query to prevent errors in connected tools 

==Version 11.9.1 (Released: 2023/02/10)==
'''Bug Fix:''' 
FR11-128: Security rule on OBS Attribute blocks observation when value is null 

==Version 11.9.0 (Released: 2023/02/03)==
'''Improvements:''' 
FR11-117: Provide option to skip Validation phase in Transform Web Service https://github.com/bis-med-it/fmr/issues/237 
FR11-120: Publication Table - apply number formatting to calculated cells 
FR11-121: Publication Table - Apply number scaling before calculations 

'''Change:''' 
FR11-118: Use the public fastUtil library https://github.com/bis-med-it/fmr/issues/338 

'''Bug Fixes:''' 
FR11-116: Timezone issue with REST API resulting in no data 
FR11-119: Codelists with Annotations not being correctly parsed in SDMX-JSON https://github.com/bis-med-it/fmr/issues/345 
FR11-123: EDI should prevent Quadruplet attributes in Footnote Section https://github.com/bis-med-it/fmr/issues/347 
FR11-124: EDI file missing mandatory Period Information validates successfully https://github.com/bis-med-it/fmr/issues/348 
FR11-125: Improve validation of EDI DTM segment https://github.com/bis-med-it/fmr/issues/351 
FR11-126: Publication Table - decimal places (fixed) truncated trailing zero places when locale was also set 
FR11-127: Duplicate elements in SDMX CSV result in unexpected output https://github.com/bis-med-it/fmr/issues/342 

==Version 11.8.2 (Released: 2023/01/17)==
'''Improvements:''' 
FR11-108: Prevent contradictory excel mappings (generted from Fame Metadata Manager) worksheets being uploaded to Registry 
FR11-109: Flag contradictory excel mappings (generted from Fame Metadata Manager) in generated worksheet 
FR11-134: Improve Metrics Report with IncUnmappedReport option 

'''Changes:''' 
FR11-111: Change ID Generation of Representation Map from FusionFameMapper xlsx files. 
FR11-115: Ensure Implicit Mappings in Structure Maps validate codelists for valid intersection 

'''Bug Fixes:''' 
FR11-107: Parent codes should be included in reference partial 
FR11-110: Complete Cube web service feature outputs empty measure values (not 0 values) 
FR11-112: Data export formats excel-ts and excel-series can miss observations in output series if series contains observations with no measure values and no attribute values 
FR11-113: Fix 'too much recursion' error with reverse engineer data structure with large number of Dimensions 
FR11-114: Validate submission of Structure Map's Implicit mappings to ensure Coded mappings have a valid intersection from source to target 

==Version 11.8.1 (Released: 2023/01/11)==
'''Bug Fixes:''' 
FR11-101: Unable to import Codelists with multiple levels of inheritence in same submission as structures that use inherited codes 
FR11-102: Bulk actions - detail=raw is ignored 
FR11-103: REST query: detail=raw is not applied to referenced codelists 
FR11-104: IncValid output may contain empty series when it should not be generated 
FR11-105: Annotations are ignored when importing 2.1 XML Hierarchical Codelists 
FR11-106: format=csv-series null pointer 

==Version 11.8.0 (Released: 2022/12/21)==
'''Important Note for MySQL users:''' due to changes in the distribution of the MySQL connector (change FR11-94 below), please read [[Install_MySQL#Installing_MySQL_5.7|this article]].

'''New Feature:''' 
FR11-98: Publication Table - support a scaling function applied to numerical observations

'''Improvement:''' 
FR11-93: Improve Performance of Viewing a Hierarchy https://wiki.sdmxcloud.org/Publication_Table#Formatting_Numerical_Observation_Values

'''Changes:''' 
FR11-91: Validation and Transformation should preserve original format unless output format specified 
https://wiki.sdmxcloud.org/Data_Validation_Web_Service#HTTP_Headers see Accept Header 
https://wiki.sdmxcloud.org/Data_Transformation_Web_Service#HTTP_Headers see Accept Header 
FR11-94: Remove MySQL Connector https://wiki.sdmxcloud.org/Install_MySQL#Installing_MySQL_5.7

'''Bug Fixes:''' 
FR11-89: Support dataset attribute consolidation (detection of duplicates) 
FR11-90: Observation deletions are no longer written in EDI format 
FR11-92: NPE when publishing EDI data from Command Line 
FR11-95: Publication Table - UI changes table definition (looses column) when clicking preview table 
FR11-96: Publication Table - Preview table error when table heading Rowspan > 1 
FR11-97: Publication Table - bug fix on formatting obs which removed values from output table 
FR11-99: Publication Table - Scientific Notation shows NaN in UI and is not applied to observation values 
FR11-100: Refreshable workbook - issue security challenge when connecting to private server 
FMR11-207: Transforming Data with "Delete" action can leave erroneous output in EDI format 
FMR11-217: Publication Tables: Numerical Observation, "Add Rule" button does not work 
FMR11-224: Failure to report unmapped data when using TIME_PERIOD mapping 

==Version 11.7.2 (Released: 2022/11/22)==
'''Change:''' 
FR11-84 - In SDMX CSV, SERIES_KEY and OBS_KEY will output with the quote marks around the whole key 

'''Bug Fixes:''' 
FR11-80 - Validation on Group ID should ignore case 
FR11-82 - Size of column in Portal Table not large enough to contain Data Source ID 
FR11-85 - When loading via Registry UI, Data file with Duplicate Series results in MV Store Error 
FR11-86 - Viewing a Representation Map has issue with "valid from" on Mappings 

==Version 11.7.1 (Released: 2022/11/15)==
'''New Feature:''' 
FR11-83 - Support Categorise Publication Table via the UI 

==Version 11.7.0 (Released: 2022/11/11)==
'''New Feature:''' 
FR11-78 - Publication Table Support Numerical Observation formatting 

'''Bug Fixes:''' 
FR11-79 - Fix an issue when "edi.datagroup.identifier" specified with value of asterisk 
FR11-80 - Validation on Group ID should ignore case 
FR11-81 - Data Portal - ESTAT Connector fix 

==Version 11.6.0 (Released: 2022/11/31)==
'''New Features:''' 
FR11-76 - [[Audit_Web_Services#Search_Structure_Queries|Audit Structure Queries]] 
FR11-77 - [[Audit_Web_Services#Search_Data_Queries|Audit Data Queries]] 

'''Improvements:''' 
FR11-75 - Support for extended UTF-8 characters https://github.com/MEDIT-Org/fmr/issues/297 

'''Change:''' 
FR11-65 - Update FontAwesome to use Version 6 

'''Bug Fixes:''' 
FR11-71 - Corrected issue with Dynamic Attributes 
FR11-72 - Version Drop-Down not Displayed when Selected Item on Page Load 
FR11-74 - Security Group Web Service does not output UTF-8 characters 
FR11-65 - Update Web Services "structure" page so the plus symbol is used for "latest"

==Version 11.5.2 (Released: 2022/10/14)==
'''New Feature:''' 
FMR11-194 - Publication Table - support casting dates more then once in same expression 

'''Improvement:''' 
FMR11-202 - Update dependencies: Hibernate, Spring-Kafka, Jettison and Jersey https://github.com/MEDIT-Org/fmr/issues/294 

'''Bug Fixes:''' 
FR11-66 - "Metadata Provider Scheme" not listed on Web Service -> Structures page 
FR11-67 - "Metadata Provider Wizard" does not allow Providers to start with a numeric yet the SDMX Standard allows this 
FR11-68 - "Metadata Provider Scheme" not listed on Cross References page 
FR11-69 - Attaching Reference Metadata to Items in an item scheme fails to add link (cache issue) 
FMR11-189 - Behaviour of web service resolving multiple Codelist inheritance with "details=raw" https://github.com/MEDIT-Org/fmr/issues/257 
FMR11-196 - Unable to read Excel Spreadsheet where all Frequencies are Formulas 
FMR11-197 - Request for Codelist in SDMX-JSON, "details=raw", references="children" doesn't return children as raw https://github.com/MEDIT-Org/fmr/issues/257 
FMR11-198 - Request for Codelist in SDMX-JSON, "details=raw", references="descendants" doesn't return descendants https://github.com/MEDIT-Org/fmr/issues/257 
FMR11-199 - "Metadata Provider Scheme" not listed on Web Service -> Structures page 
FMR11-200 - "Metadata Provider Wizard" does not allow Providers to start with a numeric yet the SDMX Standard allows this 
FMR11-201 - "Metadata Provider Scheme" not listed on Cross References page 
FMR11-203 - Attaching Reference Metadata to Items in an item scheme fails to add link (cache issue) 
FMR11-205 - Version Number not Displayed when Selected Item on Page Load 

==Version 11.5.1.1 (Released: 2022/10/10)==
'''Change:''' 
FR11-73 - Modify SQL to ensure GMT Timezone is used 

==Version 11.5.1 (Released: 2022/10/05)==
'''New Features:''' 
FR11-60 - Publication Table - support casting dates more then once in same expression 
FR11-61 - orderMeasure web service parameter to sort observations by date asc/desc 

==Version 11.5.0 (Released: 2022/09/28)==
'''New Features:''' 
FMR11-187/FR11-53 - Support recovery of structure transactions from backup on installation https://github.com/MEDIT-Org/fmr/issues/277 
FR11-52 - Support variable number of max arguments to a SQL IN statement 
FR11-58 - Support import of datafile containing mulitple datasets 

'''Improvements:''' 
FMR11-183 - Update dependencies https://github.com/MEDIT-Org/fmr/issues/277 
FR11-48 - Update third-party dependencies: apache-jstl, commons-dcp2, joda-time, spring, tika-core 
FR11-45 - Add Version Number to Provision Agreement Selector in Data Source Manager Modal 

'''Bug Fixes:''' 
FMR11-181/FR11-46 - SDMX CSV v 2.0 Data Format does not report its format https://github.com/MEDIT-Org/fmr/issues/110 
FMR11-184/FR11-49 - Support '*' for structure type in REST API https://github.com/MEDIT-Org/fmr/issues/273 
FMR11-185/FR11-51 - Failure on Compare Revision shows Stack Trace https://github.com/MEDIT-Org/fmr/issues/278 
FMR11-186/FR11-55 - SDMX-JSON Writer not outputting DSD as stub https://github.com/MEDIT-Org/fmr/issues/274 
FR11-50 - FameMapper code is creating Excel files in temporary storage which are not deleted 
FR11-54 - EDI data not fully consolidated when publishing data via web service 

==Version 11.4.3 (Released: 2022/09/13)==
'''New Features:''' 
FMR11-174 - Publication Tables: Formatting of Missing Values support ObsValue=NaN https://github.com/MEDIT-Org/fmr/issues/253 
FMR11-175 - Support Reading of Reference Metadata (with restrictions) in version 2.1 SDMX https://github.com/MEDIT-Org/fmr/issues/250 

'''Improvement:''' 
FMR11-182 - Modify ID generation strategy for importing Reference Metadata at v2.1 

'''Bug Fixes:''' 
FR11-39 - OBS_PRE_BREAK validation corrected 
FR11-43 - SDMX DSD output may include invalid attribute "isMultiLingual" 
FR11-44 - SQL Server does not delete temporary tables 
FMR11-169 - SDMX CSV v1 Data Reader does not allow header fields (row 1) to be double quoted 
FMR11-170 - Bulk Export - spelling mistake on Raw help text 
FMR11-171 - Order of revisions not as expected 
FMR11-176 - OBS_PRE_BREAK validation corrected 
FMR11-178 - Quadruplet in EDI vs the other formats in the context of data consolidation or integration https://github.com/MEDIT-Org/fmr/issues/118 
FMR11-179 - SDMX DSD output may include invalid attribute "isMultiLingual" https://github.com/MEDIT-Org/fmr/issues/255 

==Version 11.4.2 (Released: 2022/08/19)==
'''Bug Fixes:''' 
FR11-30 - Fusion Security Manager will not accept AD group names longer than 30 characters 
FR11-34 - Updated third-party dependencies 

==Version 11.4.1 (Released: 2022/08/10)==
'''Bug Fixes:''' 
FR11-21 - Issue loading data into SQL Server in Registry managed table
FR11-26 - Bulk Export - spelling mistake on Raw help text 
FR11-27 - Order of revisions not as expected 
FR11-28 - JNDI Registry Managed Database - unable to load data 
FR11-25 - SDMX CSV v1 Data Reader does not allow header fields (row 1) to be double quoted 

==Version 11.4.0 (Released: 2022/07/22)==
'''New Feature:''' 
FR11-18 - Support Reference Metadata https://wiki.sdmxcloud.org/Reference_Metadata 

'''Improvement:''' 
FMR11-146 - Dependency updates 
FMR11-159 - Add "Raw" option to Download structures and remove "Include internal settings" 
FMR11-160 - Allow a Concept Representation to be a ValueList 

'''Bug Fixes:''' 
FR11-14 - Issue loading Data Constraint with a Component element under Cube Region element 
FR11-17 - Attribute variable does not work in table heading 
FR11-19 - Increase max size of security group ID from 30 to 100 characters 
FMR11-162 - SERIES_KEY column should be reserved for series keys 
FMR11-161 - Issue loading Data Constraint with a Component element under Cube Region element 
FMR11-163 - Attribute variable does not work in table heading 
FMR11-166 - Duplicate attribute is not detected anymore in FMR 
FMR11-167 - Publication Table - Array Variables fail to save 
FMR11-168 - Error encountered whist attempting to deserialize structure 

==Version 11.3.0 (Released: 2022/07/06)==
'''New Feature:''' 
FR11-10/FMR11-153 - [[Synchronise_structural_metadata_between_environments|Environment Synchronisation - compare the structural metadata content with that from another Registry, push or pull changes]]

'''Improvement:''' 
FMR11-158 - [[Publication_Table#Footnotes|Publication Table: Footnotes support]]

'''Bug Fixes:''' 
FMR11-152 - Comma not being processed correctly in Web Service query for ItemBeans 
FMR11-154 - Unexpected structure validation error from FMR 
FMR11-155 - Structure validation errors are not detected anymore in FMR 
FMR11-156 - v2 structure REST API does not support queries for multiple items using the comma syntax 
FMR11-157 - Typo in header of Representation Maps page 

==Version 11.2.0 (Released: 2022/06/27)==
'''New Feature:''' 
[FMR11-150] Specify the validation and transformation behaviour when duplicate observations are found - [[Data_Transformation_Web_Service#HTTP_Headers|https://wiki.sdmxcloud.org/Data_Transformation_Web_Service#HTTP_Headers]] 

'''Bug Fixes:''' 
[FR11-3] Schema Query in structure format does not work 
[FR11-4] Fusion Cache ignoring parameters like normalise freq on V1 data queries 
[FR11-5] Data Point Security - allow rule against a series attribute value that does not exist in the dataset restricts all data 
[FR11-8] Comma not being processed correctly in Web Service query for ItemBeans 
[FMR11-141] Re-opening of Alternate Half year Issue 
[FMR11-143] TIME_FORMAT not being generated correctly 
[FMR11-144] Problem with Date data type when reverse engineering a DSD from a CSV dataset 
[FMR11-145] Problem with Description data type when reverse engineering a DSD from a CSV dataset 
[FMR11-147] DSD Wizard step 3 - not allowing Enumerated Representation 
[FMR11-148] Schema Query in Structure Format does not work 

'''Improvement: '''
[FMR11-140] Revert xsi:type=SiblingType, change the schema to output the Group as Sibling not SiblingType -

==Version 11.1.0 (Released: 2022/06/01)==
'''New Feature:''' 
[[Publication_Table|Publication Tables]] 
[FR-5130] [https://wiki.sdmxcloud.org/Publication_Table#Looping_Rows Publication Table - Support Dynamic Rows] 
[FR-5131] [https://wiki.sdmxcloud.org/Publication_Table#Formatting_Missing_Observations Publication Table - Format rules on cells with no data ] 
[FR-5132] [https://wiki.sdmxcloud.org/Publication_Table#Dependent_Dimension_Variable Publication Table - Rules to derive a Dimension Value from another Dimenison's value] 
[FR-5133] [https://wiki.sdmxcloud.org/Publication_Table#Anchor_Rows Publication Table - Option to output rows with no data ] 
[FR-5134] [https://wiki.sdmxcloud.org/Publication_Table_Web_Services#Materialise_Publication_Table Publication Table - Ouput Materialised Table in Excel ] 

'''Improvement:''' 
[FR-5129] - Support for MySQL 8 and above 

==Version 11.0.0 (Released: 2022/05/22)==
'''Changes and New Features:''' 
* Change to the SDMX 3.0 information model as Fusion Registry’s internal metadata object model
* SDMX 3.0 REST API support
* Backward compatibility for SDMX 2.1 and 2.0 structures and formats
* Backward compatibility with SDMX 2.1 REST API (REST API v1 specification)
* Improved data and metadata access control
* Granular control over admin rights
* Fusion Portal – scheduled and on demand data and structure loads
* Refreshable Excel data format
* JNDI database connection option
* Publication Tables - publish curated but dynamic user interactive web data tables that show precisely the statistics required even if they come from multiple different data sets

'''Breaking Changes from Fusion Registry 10:''' 
* Change to the operating database schema - upgrading from Fusion Registry 10 to Fusion Registry 11 requires a fresh install and migration of the metadata, data and settings
* Environment sync function temporarily removed
* Fusion Security support deprecated - Active Directory and LDAP are the supported authentication services in Fusion Registry 11

Logging

2024-02-02T09:29:19Z

Plazarou:

The Fusion Registry logs information regarding the requests made to it and the work that it is performing. These logs can be long and complicated, but this document aims to explain the logging process.

Logging information is stored in the filesystem. The files on your system are temporary, and without intervention, they could be overwritten by more recent log data. This measure is in place to avoid log files occupying your entire disk space.

=File-based logging=

During operation the Fusion Registry writes information to its logs. The level of logging is controlled by a system setting in the Fusion Registry which directly relates to the logging tool that the Fusion Registry uses, which is called Logback. The files generated are also affected by the Java Web-Application Server you are running (for example Apache Tomcat). For the rest of this section, only the scenario when running on Tomcat will be detailed.

Since log files will naturally grow larger over time, if allowed to grow indefinitely, their size would become a problem consuming all the available disk space. This is addressed using rolling file appenders, which automatically “roll” or archive the current log file, and resume logging in a new file when certain predefined conditions occur.

==Tomcat files==

In Apache Tomcat, various log files are generated during the execution of a Java web application. These log files provide valuable information about the application's behavior, errors, and performance. Here's a summary of some common log files in Tomcat:

catalina.log:
Purpose: General logging for the Tomcat server.
Contents: Includes information about server startup, shutdown, and any errors that occur during the server's operation.

catalina.<date>.log:
Purpose: Daily rotating log file for Catalina.
Contents: Includes information specific to the Catalina component of Tomcat. Useful for troubleshooting Catalina-related issues.

localhost.log:
Purpose: Logs information related to the default "localhost" context.
Contents: Records information about the deployment and undeployment of web applications under the default context.

localhost.<date>.log:
Purpose: Daily rotating log file for the default "localhost" context.
Contents: Contains information specific to the default web application context. Useful for troubleshooting application-specific issues.

localhost_access_log.txt:
Purpose: Records access information, such as requests made to the Tomcat server.
Contents: Contains details like client IP address, request date and time, HTTP status, and bytes sent.

localhost_access_log.<date>.txt:
Purpose: Daily rotating log file for localhost_access_log
Contents: The daily rotation of the localhost_access_log

manager.log:
Purpose: Logs information about the Tomcat Manager application.
Contents: Records deployment and undeployment activities performed through the Tomcat Manager.

host-manager.log:
Purpose: Logs information about the Host Manager application.
Contents: Records deployment and undeployment activities performed through the Host Manager.

FusionRegistry.log:
Purpose: Logs Specific information about the Fusion Registry
Contents: This is the most useful log file when debugging an issue with the Fusion Registry.

==Logback==

The logging is controlled by a file called "logback.xml" which is located in the WEB-INF\classes directory under the installed FusionRegistry web application. For example this could be: C:\tomcats\apache-tomcat-9.0.58\webapps\FusionRegistry\WEB-INF\classes

Configuration can be performed by modifying this file. There are different levels of logging:

'''TRACE''': This is for extremely detailed and potentially high volume logs that you don’t typically want enabled even during normal development. 
'''DEBUG''': Helpful in tracking the flow through the system and isolating issues, especially during the development and QA phases. 
'''INFO''': Used for system lifecycle events (system start, stop) and significant boundary events (e.g., database calls, remote API calls). Typical business exceptions can go here. 
'''WARN''': An unexpected technical or business event happened, customers may be affected, but probably no immediate human intervention is required. 
'''ERROR''': The system is in distress, customers are probably being affected (or will soon be) and the fix probably requires human intervention. 
'''OFF''': This level is intended to turn off logging. 

'''It is not recommended to switch to TRACE or DEBUG level unless specifically requested to by the development team, since the amount of logging produced can be excessive and can cause performance issues.'''

[[File:LoggingLevels.png||]]

===Logback Configuration===
The logging is controlled by a file called "logback.xml" which is located in the WEB-INF\classes directory under the installed FusionRegistry web application. The file consists of multiple options that can be configured by the user. Some of the more useful parts of this file are:

Line 2: <configuration scan="true" scanPeriod="30 seconds">

This configuration tells Logback to automatically check for changes in the configuration file every 30 seconds, and if any changes are detected, Logback will reload the configuration.

Line 8: <appender name="fileout" class="ch.qos.logback.core.rolling.RollingFileAppender">

Start of file appender, responsible for writing logs to the file

Line 9: <file>${catalina.base:-./temp}/logs/FusionRegistry.log</file>
This line tells Logback the directory and filename to store Registry information into. The default is "FusionRegistry.log"

Line 10: <encoder> <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} %p %t %c - %m%n</pattern> </encoder>

Encoder is responsible for formatting log messages. Current configuration tells logback to format logs like the following:
2024-01-30 13:34:41.123 INFO main com.example.MyClass - This is a log message

Line 13:
<rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
<maxIndex>4</maxIndex>
<FileNamePattern>${catalina.base:-./temp}/logs/FusionRegistry.log.%i
</FileNamePattern>
</rollingPolicy>

Rolling policy is responsible for creating multiple log files with archived logs.
Current configuration tells logback to create up to 4 files named FusionRegistry.log.1 FusionRegistry.log.2 and so on.
When a new log file is created (due to a rollover event), the oldest file (if the maximum of 4 is reached) will be deleted, and the other files will be renamed to increment their index.

Line 18:
<triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
<MaxFileSize>10024KB</MaxFileSize>
</triggeringPolicy>

Triggering policy is responsible for defining when the file will rollover. In current configuration it is around 10MBs. Log file rollover and removal of old logs is necessary to prevent logs consuming all the available disk space.

The next appender tells logback to create additional file FusionRegistryWarnings.log and store there log messages with levels WARN and ERROR, all other configurations align with the first appender
Line 49:
<logger name="com.metadatatechnology" level="INFO" additivity="false">
<appender-ref ref="fileout" />
<appender-ref ref="STDOUT" />
<appender-ref ref="fusionaudit" />
<appender-ref ref="warnout" />
</logger>

Logger provides an option to select what is the minimal level of logging and which appenders to use for specified package and all its subpackages. In this case it tells to set minimal level of logging to INFO, and use all 4 appenders specified in file for "com.metadatatechnology".

Line 63: <root level="INFO">
Similar options for all remaining packages in the project, and third party libraries.

==Log Detail==

Sample log entry:
2024-01-30 13:34:41.123 INFO main com.example.MyClass - This is a log message

Every log entry consists of the following parts:
# Timestamp when error occurred (eg. 2024-01-30 13:34:41.123)
# Log level (eg. INFO)
# Thread on which log was produced (eg. main)
# Class that produced the log (eg. com.example.MyClass)
# Content of the log (eg. This is a log message)

There are multiple types of logs stored in FusionRegistry.log file. Some are more helpful for the end user than others.

===System Information Logs===
Those log entries inform about the state of the system, services being started or system settings being set. Examples:

2024-01-25 13:49:24.187 INFO RMI TCP Connection(3)-127.0.0.1 io.sdmx.utils.spring.ApplicationStarterImpl - Starting class io.sdmx.core.dao.base.PersistableMaintainableDao

2024-01-25 13:49:24.137 INFO RMI TCP Connection(3)-127.0.0.1 io.sdmx.core.fmr.manager.settings.RegistryPropertiesFileManagerImpl - Set up Email Server Settings

2024-01-25 13:49:24.115 INFO RMI TCP Connection(3)-127.0.0.1 io.sdmx.fusion.audit.manager.log.LogManagerImpl - Max Age for Transactional Log Events set to -1 hours

===Loading Structural Metadata===
When loading structural metadata into the registry, many log entries are created. They have an INFO level and may contain the url from which the metadata is loaded, identifiers of loaded metadata as well as process description. Example:

2024-01-29 10:54:55.998 INFO http-nio-8080-exec-3 io.sdmx.utils.http.broker.RestMessageBroker - Call URL: https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2/structure/structure/all/all/all/?format=sdmx-3.0
2024-01-29 10:54:57.624 INFO http-nio-8080-exec-3 io.sdmx.utils.http.broker.RestMessageBroker - Call URL Complete: https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2/structure/structure/all/all/all/?format=sdmx-3.0
2024-01-29 10:54:57.754 INFO http-nio-8080-exec-3 io.sdmx.core.sdmx.manager.structure.StructureReaderManagerImpl - Parse structures
2024-01-29 10:54:57.755 INFO http-nio-8080-exec-3 io.sdmx.format.ml.factory.structure.SdmxMLStructureReaderFactory - Read SdmxBeans using io.sdmx.format.ml.engine.structure.reader.v3.StaxStructureReaderEngineV3
2024-01-29 10:54:57.925 INFO http-nio-8080-exec-3 io.sdmx.core.sdmx.manager.structure.StructureReaderManagerImpl - Build Beans Complete
2024-01-29 10:54:57.926 INFO http-nio-8080-exec-3 io.sdmx.core.sdmx.manager.structure.StructureReaderManagerImpl - Post-Process Beans
2024-01-29 10:54:57.927 INFO http-nio-8080-exec-3 io.sdmx.core.sdmx.manager.structure.StructureReaderManagerImpl - Parse Structures Complete
2024-01-29 10:54:57.948 INFO http-nio-8080-exec-3 io.sdmx.core.structure.manager.persist.StructurePreProcessManagerImpl - Pre-process Beans Event
2024-01-29 10:54:57.949 INFO http-nio-8080-exec-3 io.sdmx.core.fmr.engine.structure.RegistryStructureValidationEngine - Validate structures for insertion
2024-01-29 10:54:58.021 INFO http-nio-8080-exec-3 io.sdmx.core.fmr.engine.structure.RegistryStructureValidationEngine - Validate Structure for Insertion:urn:sdmx:org.sdmx.infomodel.base.DataProviderScheme=EAPRO:DATA_PROVIDERS(1.0)
...
2024-01-29 10:54:58.051 INFO http-nio-8080-exec-3 com.metadatatechnology.fusion.registry.manager.persistence.PersistedStructureManagerEnterprise - structures valid, save to database
2024-01-29 10:54:58.051 INFO http-nio-8080-exec-3 io.sdmx.core.fmr.manager.persistence.AbstractPersistenceManager - Process Cross References
2024-01-29 10:54:58.060 INFO http-nio-8080-exec-3 io.sdmx.core.structure.manager.persist.StructurePreProcessManagerImpl - Pre-process Beans Event
2024-01-29 10:54:58.065 INFO http-nio-8080-exec-3 com.metadatatechnology.fusion.registry.manager.persistence.PersistedStructureManagerEnterprise - Save Successful - Publish Message
2024-01-29 10:55:02.287 INFO http-nio-8080-exec-6 io.sdmx.core.dao.base.PersistableMaintainableDao - Save Structure 'urn:sdmx:org.sdmx.infomodel.base.DataProviderScheme=EAPRO:DATA_PROVIDERS(1.0)'
2024-01-29 10:55:02.332 INFO http-nio-8080-exec-6 io.sdmx.core.dao.base.PersistableMaintainableDao - Save Structure 'urn:sdmx:org.sdmx.infomodel.codelist.Codelist=ISO:COUNTRY_ALPHA3(1.0)'

===Saving Time-Series Data===
When inserting/modifying/replacing data using url or file there are many log entries created. They have an INFO level and contain step by step process description. Example:

2024-01-29 11:03:26.258 INFO Thread-9 com.metadatatechnology.fusion.sdmx.jdbc.dao.publish.generic.GenericDataLoadDao - Start Loading Dataset '-2662089662959301465' valid from '2024-01-29 11:03:26.0' valid to '2999-12-31 01:00:00.0'
2024-01-29 11:03:26.265 INFO Thread-9 com.metadatatechnology.fusion.core.data.engine.writer.timeseries.PercentageCompleteWriterEngine - 1.0011855% complete
2024-01-29 11:03:26.265 INFO Thread-33 com.metadatatechnology.fusion.registry.manager.data.publish.impl.ImportProcessorEngine - Percentage Complete: 1%
2024-01-29 11:03:26.270 INFO Thread-9 com.metadatatechnology.fusion.core.data.engine.writer.timeseries.PercentageCompleteWriterEngine - 2.002371% complete
2024-01-29 11:03:26.270 INFO Thread-35 com.metadatatechnology.fusion.registry.manager.data.publish.impl.ImportProcessorEngine - Percentage Complete: 2%
2024-01-29 11:03:26.273 INFO Thread-9 com.metadatatechnology.fusion.core.data.engine.writer.timeseries.PercentageCompleteWriterEngine - 3.0035567% complete
...
com.metadatatechnology.fusion.core.data.engine.writer.timeseries.PercentageCompleteWriterEngine - 99.11737% complete
2024-01-29 11:03:26.386 INFO Thread-229 com.metadatatechnology.fusion.registry.manager.data.publish.impl.ImportProcessorEngine - Percentage Complete: 99%
2024-01-29 11:03:26.387 INFO Thread-231 com.metadatatechnology.fusion.registry.manager.data.publish.impl.ImportProcessorEngine - Percentage Complete: 100%
2024-01-29 11:03:26.402 INFO Thread-9 com.metadatatechnology.fusion.sdmx.jdbc.engine.TransactionalDataWriterEngine - Commit data
2024-01-29 11:03:27.264 INFO Thread-9 com.metadatatechnology.fusion.sdmx.jdbc.dao.publish.mysql.MySqlDBulkLoadDao - Perform Commit
2024-01-29 11:03:27.287 INFO Thread-9 com.metadatatechnology.fusion.sdmx.jdbc.dao.publish.mysql.MySqlDBulkLoadDao - Final Commit
2024-01-29 11:03:27.289 INFO Thread-9 com.metadatatechnology.fusion.sdmx.jdbc.dao.publish.mysql.MySqlDBulkLoadDao - Bulk Load Complete

===Creating a Data Store===
When creating data storage from an admin panel in the system there are many log entries created. They have an INFO level, and contain information about database configuration provided by the user. Example:

2024-01-29 11:00:05.245 INFO http-nio-8080-exec-5 com.metadatatechnology.fusion.registry.manager.data.publish.impl.DataSourceManagerImpl - Create datasource: root@localhost:3306/fmr_data
2024-01-29 11:00:05.247 INFO http-nio-8080-exec-5 io.sdmx.utils.hibernate.model.PersistantStoragePlatform - Constructed URL: jdbc:mysql://localhost:3306/fmr_data
2024-01-29 11:00:05.328 INFO http-nio-8080-exec-5 com.metadatatechnology.fusion.registry.manager.data.publish.impl.DataSourceManagerImpl - Cache DataSource: root@localhost:3306/fmr_data
2024-01-29 11:00:05.330 INFO http-nio-8080-exec-5 io.sdmx.utils.hibernate.model.PersistantStoragePlatform - Constructed URL: jdbc:mysql://localhost:3306/fmr_data
2024-01-29 11:00:05.355 INFO http-nio-8080-exec-5 com.metadatatechnology.fusion.sdmx.jdbc.dao.MatrixPluginDao - SQL In Argument Limit: 1000

=== Requesting Structural Metadata===
When requesting for metadata in the system there are log entries created. They have an INFO level, and contain information about what resource was requested, and if the result was returned from the database or from the cache.

Examples:
2024-01-26 13:35:27.894 INFO http-nio-8080-exec-10 com.metadatatechnology.fusion.registry.filter.FusionRegistryHTTPDataCacheHeaderFilter - Request issued for: http://localhost:8080/FusionRegistry/sdmx/v2/structure/provisionagreement/*/*/*
2024-01-26 13:35:27.939 INFO http-nio-8080-exec-10 com.metadatatechnology.fusion.rest.manager.cache.SDMXRESTGetRequestCacheManager - Cache miss structure/provisionagreement/*/*/*?format=fusion-json&includeAllAnnotations=true&includeMetadata=true
2024-01-26 13:35:27.980 INFO http-nio-8080-exec-10 com.metadatatechnology.fusion.registry.filter.FusionRegistryHTTPDataCacheHeaderFilter - Request completed

2024-01-29 08:31:34.651 INFO http-nio-8080-exec-7 com.metadatatechnology.fusion.registry.filter.FusionRegistryHTTPDataCacheHeaderFilter - Request issued for: http://localhost:8080/FusionRegistry/sdmx/v2/structure/datastructure/*/*/*
2024-01-29 08:31:34.674 INFO http-nio-8080-exec-7 com.metadatatechnology.fusion.rest.manager.cache.SDMXRESTGetRequestCacheManager - Cache miss structure/datastructure/*/*/*?detail=allstubs&format=fusion-json&includeAllAnnotations=true&includeMetadata=true
2024-01-29 08:31:34.710 INFO http-nio-8080-exec-7 com.metadatatechnology.fusion.registry.filter.FusionRegistryHTTPDataCacheHeaderFilter - Request completed

com.metadatatechnology.fusion.registry.filter.FusionRegistryHTTPDataCacheHeaderFilter - Request issued for: http://localhost:8080/FusionRegistry/sdmx/v2/structure/dataproviderscheme/WB/DATA_PROVIDERS/1.0/WB
2024-01-29 10:56:24.812 INFO http-nio-8080-exec-3 com.metadatatechnology.fusion.rest.manager.cache.SDMXRESTGetRequestCacheManager - Cache miss structure/dataproviderscheme/WB/DATA_PROVIDERS/1.0/WB?format=fusion-json&includeAllAnnotations=true&includeMetadata=true
2024-01-29 10:56:24.843 INFO http-nio-8080-exec-3 com.metadatatechnology.fusion.registry.filter.FusionRegistryHTTPDataCacheHeaderFilter - Request completed

===Requesting Time-Series Data===
When requesting for time series in the system there are log entries created. They have an INFO level, and contain information about what resource was requested. If the request is against an external database the query may well include SQL statements showing the database queries performed.

2024-01-29 13:43:42.911 INFO http-nio-8080-exec-7 com.metadatatechnology.fusion.registry.filter.FusionRegistryHTTPDataCacheHeaderFilter - Request issued for: http://localhost:8080/FusionRegistry/sdmx/v2/data/dataflow/BIS/WS_SPP/1.0/
2024-01-29 13:43:42.933 INFO http-nio-8080-exec-7 com.metadatatechnology.fusion.rest.manager.cache.SDMXRESTGetRequestCacheManager - Cache miss data/dataflow/BIS/WS_SPP/1.0?c[FREQ]=Q&c[REF_AREA]=5R&c[UNIT_MEASURE]=771&c[VALUE]=N&format=sdmx-compact-2.1&saveAs=BIS property prices: selected series
2024-01-29 13:43:42.934 INFO http-nio-8080-exec-7 com.metadatatechnology.fusion.core.data.manager.index.IndexRetrievalManagerImpl - Data Query By Provision Agreement:urn:sdmx:org.sdmx.infomodel.registry.ProvisionAgreement=BIS:WS_SPP_BIS(1.0)
2024-01-29 13:43:42.950 INFO http-nio-8080-exec-7 com.metadatatechnology.fusion.sdmx.jdbc.dao.MatrixPluginDao - Execute Query against 'fmr_data' : SELECT ID, FR_GROUP_ID, FR_GROUP_ID, K.FREQ, K.REF_AREA, K.VALUE, K.UNIT_MEASURE FROM `BIS_WS_SPP_1_0_KEY` K WHERE FR_GROUP_ID != ? AND ((FR_PROVIDER_ACY = ? AND FR_PROVIDER_ID = ? )) AND K.FREQ IN (?,? ) AND K.REF_AREA IN (?,? ) AND K.UNIT_MEASURE IN (?,? ) AND K.VALUE IN (?,? ) with parameters, ^,BIS,BIS,^,Q,^,5R,^,771,^,N
2024-01-29 13:43:42.954 INFO http-nio-8080-exec-7 com.metadatatechnology.fusion.sdmx.jdbc.dao.MatrixPluginDao - Completed Query against 'fmr_data', Processed '0' Table Rows for query: SELECT ID, FR_GROUP_ID, FR_GROUP_ID, K.FREQ, K.REF_AREA, K.VALUE, K.UNIT_MEASURE FROM `BIS_WS_SPP_1_0_KEY` K WHERE FR_GROUP_ID != ? AND ((FR_PROVIDER_ACY = ? AND FR_PROVIDER_ID = ? )) AND K.FREQ IN (?,? ) AND K.REF_AREA IN (?,? ) AND K.UNIT_MEASURE IN (?,? ) AND K.VALUE IN (?,? ) with parameters, ^,BIS,BIS,^,Q,^,5R,^,771,^,N
2024-01-29 13:43:42.957 INFO http-nio-8080-exec-7 com.metadatatechnology.fusion.sdmx.jdbc.dao.MatrixPluginDao - Execute Query against 'fmr_data' : SELECT K.FREQ, K.REF_AREA, K.VALUE, K.UNIT_MEASURE, K.BREAKS, K.COVERAGE, K.TITLE_TS, K.FR_UPDATED_DATE, K.ID, O.OBS_VALUE, O.OBS_STATUS, O.OBS_CONF, O.OBS_PRE_BREAK, O.FR_DATASET_ID, O.FR_IS_DELETED, O.FR_UPDATED_DATE_OBS, O.FR_VALID_FROM, O.TIME_PERIOD, O.FR_OBS_DATE FROM `BIS_WS_SPP_1_0_OBS` O RIGHT JOIN (SELECT K.FREQ, K.REF_AREA, K.VALUE, K.UNIT_MEASURE, K.BREAKS, K.COVERAGE, K.TITLE_TS, K.FR_UPDATED_DATE, K.ID FROM `BIS_WS_SPP_1_0_KEY` K WHERE ((FR_PROVIDER_ACY = ? AND FR_PROVIDER_ID = ? )) AND K.FREQ = ? AND K.REF_AREA = ? AND K.UNIT_MEASURE = ? AND K.VALUE = ? AND K.FR_UPDATED_DATE IS NOT NULL AND K.FR_GROUP_ID = ? ) K ON O.FR_KEY_ID = K.ID WHERE (O.FR_VALID_TO = ? OR O.FR_VALID_TO IS NULL) AND O.FR_IS_DELETED=0 ORDER BY K.ID ASC,O.FR_OBS_DATE ASC with parameters, BIS,BIS,Q,5R,771,N,^,2999-12-30T23:00:00
2024-01-29 13:43:42.962 INFO http-nio-8080-exec-7 com.metadatatechnology.fusion.sdmx.jdbc.dao.MatrixPluginDao - Completed Query against 'fmr_data', Processed '0' Table Rows for query: SELECT K.FREQ, K.REF_AREA, K.VALUE, K.UNIT_MEASURE, K.BREAKS, K.COVERAGE, K.TITLE_TS, K.FR_UPDATED_DATE, K.ID, O.OBS_VALUE, O.OBS_STATUS, O.OBS_CONF, O.OBS_PRE_BREAK, O.FR_DATASET_ID, O.FR_IS_DELETED, O.FR_UPDATED_DATE_OBS, O.FR_VALID_FROM, O.TIME_PERIOD, O.FR_OBS_DATE FROM `BIS_WS_SPP_1_0_OBS` O RIGHT JOIN (SELECT K.FREQ, K.REF_AREA, K.VALUE, K.UNIT_MEASURE, K.BREAKS, K.COVERAGE, K.TITLE_TS, K.FR_UPDATED_DATE, K.ID FROM `BIS_WS_SPP_1_0_KEY` K WHERE ((FR_PROVIDER_ACY = ? AND FR_PROVIDER_ID = ? )) AND K.FREQ = ? AND K.REF_AREA = ? AND K.UNIT_MEASURE = ? AND K.VALUE = ? AND K.FR_UPDATED_DATE IS NOT NULL AND K.FR_GROUP_ID = ? ) K ON O.FR_KEY_ID = K.ID WHERE (O.FR_VALID_TO = ? OR O.FR_VALID_TO IS NULL) AND O.FR_IS_DELETED=0 ORDER BY K.ID ASC,O.FR_OBS_DATE ASC with parameters, BIS,BIS,Q,5R,771,N,^,2999-12-30T23:00:00
2024-01-29 13:43:43.032 ERROR http-nio-8080-exec-7 io.sdmx.format.ml.engine.error.ErrorWriterEngineV21 - No results for query
2024-01-29 13:43:43.033 INFO http-nio-8080-exec-7 com.metadatatechnology.fusion.registry.filter.FusionRegistryHTTPDataCacheHeaderFilter - Request completed

===Error/Warn Information Logs===
If any issues were encountered, please attach logs to your report. Error log entities usually contain stack-trace which may be helpful to the development team to locate the issue in the code. Examples:

2024-01-29 11:05:12.475 ERROR Thread-457 io.sdmx.api.exception.SdmxException - Can not delete mandatory attribute UNIT_MULT
io.sdmx.api.exception.SdmxException: Can not delete mandatory attribute UNIT_MULT
at com.metadatatechnology.fusion.sdmx.jdbc.engine.ImportPreProcessorEngine.validateAttributes(ImportPreProcessorEngine.java:298)
at com.metadatatechnology.fusion.sdmx.jdbc.engine.ImportPreProcessorEngine.validate(ImportPreProcessorEngine.java:253)
at com.metadatatechnology.fusion.sdmx.jdbc.engine.ImportPreProcessorEngine.run(ImportPreProcessorEngine.java:99)
at com.metadatatechnology.fusion.sdmx.jdbc.engine.ImportPreProcessorEngine.lambda$new$0(ImportPreProcessorEngine.java:75)
at io.sdmx.fusion.audit.manager.event.AuditEventManager.lambda$startNewThread$7(AuditEventManager.java:308)
...

2024-01-29 13:47:38.233 ERROR http-nio-8080-exec-5 io.sdmx.api.exception.SdmxException - unauthorised
io.sdmx.api.exception.SdmxUnauthorisedException: unauthorised
at io.sdmx.api.security.ISecurityAdminAuthorisationManager.authorise(ISecurityAdminAuthorisationManager.java:33)
at io.sdmx.api.security.ISecurityAdminAuthorisationManager.authorise(ISecurityAdminAuthorisationManager.java:45)
at com.metadatatechnology.fusion.security.manager.settings.AbstractFusionSecurityRulesManager.checkAuth(AbstractFusionSecurityRulesManager.java:138)
at com.metadatatechnology.fusion.security.manager.settings.AbstractFusionSecurityRulesManager.saveRule(AbstractFusionSecurityRulesManager.java:87)
at com.metadatatechnology.fusion.security.service.GenericFusionSecurityRuleServiceActivator.saveRule(GenericFusionSecurityRuleServiceActivator.java:39)
...

2024-01-18T16:29:38Z

Plazarou: /* Response Formats */

[[Category:Fusion Edge Server]]
[[Category:FESInstall]]
This page explains details the Audit capabilities of Fusion Edge Server.

== Audit File==

All Audit information from Fusion Edge Server is written to Audit files which are located in the "Audit" sub-folder of your [[Fusion_Edge_Server_Directory|Edge Server Directory]]. Audit files are named according to the format:
EdgeServerAudit_<launch time>_<log index>.json
For example:
EdgeServerAudit_1704067199000_1.json

The "launch time" value represents the time when the Edhe Server was started. However, this value is expressed as a lengthy numerical figure, indicating the time in milliseconds since 1970.

Audit information will be written to this file until either the Edge Server is terminated or the file reaches the file limit of 10Mb in size. If the file limit is reached, a new file is created with the next incremental Log Index value. Audit files that are actively being written to may be "locked" by your Operating System until either the Edge Server is terminated or a new audit file is started.

Each Audit file contains JSON, but this is not formatted for ease of reaability. Until the file is "completed", either by termination of the Edge Server, or by a new log file being started, a final closing brace character "]" is required to make the file contents become valid JSON. So you may find that if you wish to look at the currently being written Audit file you have to:
* Create a copy of the Audit file
* Open this copy in the editor of your choice (for example Notepad++)
* Add a final "]" character to the end of the file
* Format the JSON for readability/

== Disabling Audit ==
Auditing can be disabled by modifying the Edge Server properties file and adding the entry:
audit.disabled=true

See the [[Fusion_Edge_Server_Properties|properties page]] for more information.

== Contents of the Audit File ==
It is necessary to have a basic understanding of the JSON format to understand how to parse the Audit file. The file contains a JSON array of items. Each item in the array is a JSON Object. Each JSON Object is identified by "curly braces" { and }. Some JSON objects contain other JSON objects (e.g. "properties" in the example below). Each item in the highest level JSON array, represents a unique audit event in the Edge Server. The following shows an example of one such event:

{
"uid": "e3de1d84-2413-4b7a-ae1d-754ad38d3a9f",
"process_id": "REST_API",
"thread": "http-nio-8084-exec-8",
"event_type": "GET",
"username": "guest",
"process_start": 1699023147303,
"process_end": 1699023147342,
"duration": 39,
"status": 200,
"vmid": "a58a00a880f5f938:4d097009:18b95a67fb4:-8000",
"machine_id": "DESKTOP-DSTGA0Q/192.168.1.14",
"software_version": "4.7.2.0.0.0",
"properties": {
"QueryParameters": {
"c[FREQ]": "A",
"c[REF_AREA]": "BE"
},
"HttpHeaders": {
"host": "localhost:8084",
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/119.0",
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
"accept-language": "en-GB,en;q=0.5",
"connection": "keep-alive"
},
"IP": "127.0.0.1",
"Path": "/sdmx/v2",
"PathInfo": "/data/dataflow/BIS/BIS_CBPOL/1.0",
"Locale": "en_GB",
"HttpStatus": 200
}
}

The above JSON can be described in the following manner, where the items in brackets indicate the field that is being referred to.

A GET event (''"event_type"'') was performed on on Friday, 3 November 2023 14:52:27.303 GMT (''"process_start"'' of 1699023147303 ) by user "guest" (''"username"''). This process completed succesfully (''"HttpStatus"'' of 200) and was a request against the Dataflow BIS:BIS_CBPOL(1.0) (''"PathInfo"''). The query was further constrained by the Frequency of "A" and Reference Area of "BE" (''"QueryParameters"'')

There is more information contained in this one JSON object, but that allows us to get a quick overview of what the request was for.

=== Child events ===
Some audit events have a parent UID (element: "parent"). This shows that this audit event was created from the UID referenced in the "parent" value.

=== Elements ===
The following describes each of the elements that can be found in a JSON Object:

{| class="wikitable"
|-
| '''uid''' || The Unique Identifier (UID) of the event
|-
| '''process_id''' || The type of process requested. Such values are "APPLICATION_START" (the edge server start event), "SDMX_GET" (??), "REST_API" (a query for structures or data)
|-
| '''thread''' || The internal thread namethat this ran on within the Edge Server
|-
| '''event_type''' || The HTTP Event Type that was performed
|-
| '''username''' || The identity of the user that performed this request
|-
| '''process_start''' || The start time (in milliseconds since 1970) that this request was performed
|-
| '''process_end''' || The end time (in milliseconds since 1970) that this request was performed
|-
| '''duration''' || The total time (in milliseconds) to complete the request
|-
| '''status''' || The returned HTTP status
|-
| '''vmid''' || The VMID of the system running the request
|-
| '''machine_id''' || The identity of the machine that was running the Edge Server that performed this request
|-
| '''software_version''' || The version of the software of Fusion Edge Server
|-
| '''properties''' || A JSON Object containing parameters submitted in the request
|-
| '''QueryParameters''' || A JSON Object containing keys and values that were in the request (e.g. Key: "FREQ" and value "A")
|-
| '''HttpHeaders''' || A JSON Object containing elements that were present in the header request
|-
| '''IP''' || The IP of the originating request
|-
| '''Path''' || Part of the request's URL and can be used to identify various Web Services. E.g. it could be: "/sdmx/v2" showing a version 2 SDMX request, or "/ws/public/sdmxapi/rest" showing a request to the version 1 API
|-
| '''PathInfo''' || The extra information added to the "path". This identifies individual structures or datasets being requested.
|-
| '''Locale''' || The locale used in the request
|-
| '''HttpStatus''' || The HTTP Status of the request
|-
|}

== Specific Events in the Audit File ==
When specific requests are made to the Edge Server, the Audit file will contain specific elements. This section details what to expect in the Audit file when particular events occur.

=== Startup Information ===

On succesfull start up of Fusion Edge Server, 9 events will be written to the Audit Log. These can be identified by the Process Ids of "LOAD_PROPERTIES", "APPLICATION_START" and "ENVIRONMENT".

The first event has a Process Id of "LOAD_PROPERTIES". This states that the Edge Server properties file has successfully loaded.

Next (although will likely appear last in the list of 9) is a Process Id of "APPLICATION_START" which contains all of the properties of the system. This spawns 5 child processes, all the with Process Ids of "APPLICATION_START". Each of these children has a property object with a single class within it. These classes are: SpringBeansContainer; EdgeServerAuditPersistenceManager; EdgeServerLedgerReaderManager; AuditEventManager; SDMXCacheManager

The process "EdgeServerLedgerReaderManager" will also launch a child process which has a Process Id of "ENVIRONMENT" and an "event_type" of "UPDATE". This in turn launches a child process with Process Id of "ENVIRONMENT" and an "event_type" of "LIVE".

Since child processes are written before their parents, the logical tree structure of the audit events is as follows. The number at the start of the line shows the likely order it will be written to the Audit file:

1. process_id: LOAD_PROPERTIES
9. process_id: APPLICATION_START - with all the system props
2. process_id: APPLICATION_START - Class: SpringBeansContainer
3. process_id: APPLICATION_START - Class: EdgeServerAuditPersistenceManager
6. process_id: APPLICATION_START - Class: EdgeServerLedgerReaderManager
5. process_id: ENVIRONMENT event_type: UPDATE
4. process_id: ENVIRONMENT event_type: LIVE
7. process_id: APPLICATION_START - Class: AuditEventManager
8. process_id: APPLICATION_START - Class: SDMXCacheManager

=== Data Request ===

When a request for Data is performed, a single audit event is created with the following elements:

* process_id: REST_API
* event_type: GET
* Path: /ws/public/sdmxapi/rest or /sdmx/v2
* PathInfo: the dataflow requested such as /data/dataflow/BIS/BIS_CBPOL/1.0

The status element will likely have a value of "200", indicating the request was handled correctly. However to determine if the request returned data or not, this needs to be checked by the element "HttpStatus" which is located within the "properties" element. If this value is "404" then no data was returned for the request.

The following, truncated, audit event shows a successful request (status and HttpStatus of 200):

{
"uid": "7dd76de6-895c-4f53-bb35-18519b85580e",
"process_id": "REST_API",
"thread": "http-nio-8084-exec-6",
"event_type": "GET",
"username": "guest",
"process_start": 1705568446233,
"process_end": 1705568450320,
"duration": 4087,
"status": 200,
"vmid": "27bece96ce048b52:9c501c3:18d194af243:-8000",
"machine_id": "DESKTOP-DSTGA0Q/192.168.1.16",
"software_version": "4.8.2",
"properties": {
"QueryParameters": {
"c[FREQ]": "A"
},
"HttpHeaders": {
"host": "localhost:8084",
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0",
...
},
"IP": "127.0.0.1",
"Path": "/sdmx/v2",
"PathInfo": "/data/dataflow/BIS/BIS_CBPOL/1.0",
"Locale": "en_GB",
"HttpStatus": 200
}
}

The following (truncated) event shows a request which returned no data. Note that the differences are the UID, properties (the requested parameters) and the HttpStatus of 404:
{
"uid": "0e55c832-8844-46a5-9fa5-6aa6875e66b6",
"process_id": "REST_API",
...
"status": 200,
...
"properties": {
"QueryParameters": {
"c[FREQ]": "ZZZ"
},
"HttpHeaders": {
...
},
"IP": "127.0.0.1",
"Path": "/sdmx/v2",
"PathInfo": "/data/dataflow/BIS/BIS_CBPOL/1.0",
"Locale": "en_GB",
"HttpStatus": 404
}
}

If the request contain a ResponseFormat request (e.g. ?format=sdmx-json ) then a child audit event will also be written which contains the "ResponseFormat" in a more explicit format. For example: "sdmx-json v2.0.0". This child event may look like this and is identifiable from the "parent" element which has the UID of the event that spawned it:

{
"uid": "84f98074-0176-48a2-95ba-95d3b1465aad",
"parent": "7609d4fb-1583-4440-8a3e-4ec76c324455",
"process_id": "SDMX_GET",
"thread": "http-nio-8084-exec-7",
"event_type": "structure",
"username": "guest",
"process_start": 1705527795701,
"process_end": 1705527795798,
"duration": 97,
"status": 200,
"properties": {
"Cache": "miss",
"ResponseFormat": "sdmx-json v2.0.0"
}
}

=== Structure Request ===

When a request for Structural Metadata is performed, a single audit event is created with the following elements:
* process_id: REST_API
* event_type: GET
* Path: /ws/public/sdmxapi/rest or /sdmx/v2
* PathInfo: the structure requested such as /structure/agencyscheme/all/all

This audit event is extremely similar to the data request, and likewise if no information was returned, the "httpstatus" will be 404, and if the request contain a ResponseFormat request (e.g. ?format=sdmx-3.0 ) then a child audit event will also be written which contains the "ResponseFormat" in a more explicit format. For example: "SDMX_V3_STRUCTURE_DOCUMENT"

=== Availability Request from Fusion Data Browser ===

If the Fusion Data Browser is being used against the Fusion Edge Server, availability requests may be audited. These can be identified by the following elements:
* process_id: REST_API
* event_type: GET
* Path: /sdmx/v2
* PathInfo: /availability followed by the dataflow being queried. For example: /availability/dataflow/BIS/BIS_CBPOL/1.0

The field: "properties" -> "QueryParameters" -> "mode" will also contain the value "available"

=== Search Request from Fusion Data Browser ===

If the Fusion Data Browser is being used against the Fusion Edge Server, search requests may be audited. These can be identified by the following elements:
* process_id: REST_API
* event_type: GET
* Path: /ws/public
* PathInfo: /datasearch

The field: "properties" -> "QueryParameters" -> "query" will contain the search value. Since this requested was created by the Data Browser, the Http Header, "referer" field will state the location of the Data Browser.

Since the search field of Fusion Data Browser performs searches as the user types, you may see multiple similiar or sub-searches when a user makes a single search. For example if the user wants to search for "CREDIT", you may see audit entries for searches for "C", "CR", "CRE", "CRED", "CREDI" and "CREDIT" as the user types.

== Distinguishing Between Different Request Mechanisms ==
Requests to the Edge server can come from a number of different mechanisms. For example a request from a Web Browser, a command-line tool such as curl, or the Fusion Data Browser.

The "user-agent" field within the "HttpHeaders" object can help here. Some tools, such as curl, supply an obvious distinct "user-agent". The following snippet shows a request that has come from curl

"properties": {
"HttpHeaders": {
"host": "MyTestEdgeServer:8084",
"user-agent": "curl/8.4.0",
"accept": "*/*"
},

Other values you may encounter depend upoin the user's configuration. The following is the user-agent value for Firefox v121.0 running on Windows 10:
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0"

And the following is the user-agent for Microsoft Edge, running on Windows 10:
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 Edg/120.0.0.0",

For the Fusion Data Browser, this identifies itself in a slightly different manner. There is a referer field under the objects: "properties" -> "HttpHeaders", which shows that the request originated from the Fusion Data Browser. The value in the "referer" field will contain the home location of the Fusion Data Browser:

"properties": {
"QueryParameters": {
"locale": "en",
"saveAs": "BIS_CBPOL",
"format": "csv",
"labels": "both"
},
"HttpHeaders": {
..
"referer": "http://localhost:8084/FusionDataBrowser/",
}
}

== Response Formats ==
When a request format is specified on a data request, a child audit event will be created. Note this will be output before the parent audit event. The child event contains a value under "properties" -> "ResponseFormat" to state what the output was.

The following table lists the response format for each of the specified format values:

{| class="wikitable"
|-
! Format Value !! Output String
|-
| sdmx-compact-2.1 || "Structure Specific (Compact) 2.1"
|-
| sdmx-generic-2.1 || "Generic 2.1"
|-
| sdmx-compact-2.0 || "Structure Specific (Compact) 2.0"
|-
| sdmx-generic-2.0 || "Generic 2.0"
|-
| sdmx-csv-2.0.0 || "csv"
|-
| sdmx-json || "SDMX-JSON"
|-
| sdmx-edi || "SDMX-EDI"
|-
| excel || "Excel (XLSX)"
|-
| ?? || "fusion-excel-table"
|-
| ?? || "fusion-excel-series"
|-
| ?? || "fusion-excel-ts"
|-
| ?? || "Excel Refreshable"|-
|}

Fusion Edge Server Audit

2024-01-18T16:29:14Z

Plazarou: /* Response Formats */

[[Category:Fusion Edge Server]]
[[Category:FESInstall]]
This page explains details the Audit capabilities of Fusion Edge Server.

== Audit File==

All Audit information from Fusion Edge Server is written to Audit files which are located in the "Audit" sub-folder of your [[Fusion_Edge_Server_Directory|Edge Server Directory]]. Audit files are named according to the format:
EdgeServerAudit_<launch time>_<log index>.json
For example:
EdgeServerAudit_1704067199000_1.json

The "launch time" value represents the time when the Edhe Server was started. However, this value is expressed as a lengthy numerical figure, indicating the time in milliseconds since 1970.

Audit information will be written to this file until either the Edge Server is terminated or the file reaches the file limit of 10Mb in size. If the file limit is reached, a new file is created with the next incremental Log Index value. Audit files that are actively being written to may be "locked" by your Operating System until either the Edge Server is terminated or a new audit file is started.

Each Audit file contains JSON, but this is not formatted for ease of reaability. Until the file is "completed", either by termination of the Edge Server, or by a new log file being started, a final closing brace character "]" is required to make the file contents become valid JSON. So you may find that if you wish to look at the currently being written Audit file you have to:
* Create a copy of the Audit file
* Open this copy in the editor of your choice (for example Notepad++)
* Add a final "]" character to the end of the file
* Format the JSON for readability/

== Disabling Audit ==
Auditing can be disabled by modifying the Edge Server properties file and adding the entry:
audit.disabled=true

See the [[Fusion_Edge_Server_Properties|properties page]] for more information.

== Contents of the Audit File ==
It is necessary to have a basic understanding of the JSON format to understand how to parse the Audit file. The file contains a JSON array of items. Each item in the array is a JSON Object. Each JSON Object is identified by "curly braces" { and }. Some JSON objects contain other JSON objects (e.g. "properties" in the example below). Each item in the highest level JSON array, represents a unique audit event in the Edge Server. The following shows an example of one such event:

{
"uid": "e3de1d84-2413-4b7a-ae1d-754ad38d3a9f",
"process_id": "REST_API",
"thread": "http-nio-8084-exec-8",
"event_type": "GET",
"username": "guest",
"process_start": 1699023147303,
"process_end": 1699023147342,
"duration": 39,
"status": 200,
"vmid": "a58a00a880f5f938:4d097009:18b95a67fb4:-8000",
"machine_id": "DESKTOP-DSTGA0Q/192.168.1.14",
"software_version": "4.7.2.0.0.0",
"properties": {
"QueryParameters": {
"c[FREQ]": "A",
"c[REF_AREA]": "BE"
},
"HttpHeaders": {
"host": "localhost:8084",
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/119.0",
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
"accept-language": "en-GB,en;q=0.5",
"connection": "keep-alive"
},
"IP": "127.0.0.1",
"Path": "/sdmx/v2",
"PathInfo": "/data/dataflow/BIS/BIS_CBPOL/1.0",
"Locale": "en_GB",
"HttpStatus": 200
}
}

The above JSON can be described in the following manner, where the items in brackets indicate the field that is being referred to.

A GET event (''"event_type"'') was performed on on Friday, 3 November 2023 14:52:27.303 GMT (''"process_start"'' of 1699023147303 ) by user "guest" (''"username"''). This process completed succesfully (''"HttpStatus"'' of 200) and was a request against the Dataflow BIS:BIS_CBPOL(1.0) (''"PathInfo"''). The query was further constrained by the Frequency of "A" and Reference Area of "BE" (''"QueryParameters"'')

There is more information contained in this one JSON object, but that allows us to get a quick overview of what the request was for.

=== Child events ===
Some audit events have a parent UID (element: "parent"). This shows that this audit event was created from the UID referenced in the "parent" value.

=== Elements ===
The following describes each of the elements that can be found in a JSON Object:

{| class="wikitable"
|-
| '''uid''' || The Unique Identifier (UID) of the event
|-
| '''process_id''' || The type of process requested. Such values are "APPLICATION_START" (the edge server start event), "SDMX_GET" (??), "REST_API" (a query for structures or data)
|-
| '''thread''' || The internal thread namethat this ran on within the Edge Server
|-
| '''event_type''' || The HTTP Event Type that was performed
|-
| '''username''' || The identity of the user that performed this request
|-
| '''process_start''' || The start time (in milliseconds since 1970) that this request was performed
|-
| '''process_end''' || The end time (in milliseconds since 1970) that this request was performed
|-
| '''duration''' || The total time (in milliseconds) to complete the request
|-
| '''status''' || The returned HTTP status
|-
| '''vmid''' || The VMID of the system running the request
|-
| '''machine_id''' || The identity of the machine that was running the Edge Server that performed this request
|-
| '''software_version''' || The version of the software of Fusion Edge Server
|-
| '''properties''' || A JSON Object containing parameters submitted in the request
|-
| '''QueryParameters''' || A JSON Object containing keys and values that were in the request (e.g. Key: "FREQ" and value "A")
|-
| '''HttpHeaders''' || A JSON Object containing elements that were present in the header request
|-
| '''IP''' || The IP of the originating request
|-
| '''Path''' || Part of the request's URL and can be used to identify various Web Services. E.g. it could be: "/sdmx/v2" showing a version 2 SDMX request, or "/ws/public/sdmxapi/rest" showing a request to the version 1 API
|-
| '''PathInfo''' || The extra information added to the "path". This identifies individual structures or datasets being requested.
|-
| '''Locale''' || The locale used in the request
|-
| '''HttpStatus''' || The HTTP Status of the request
|-
|}

== Specific Events in the Audit File ==
When specific requests are made to the Edge Server, the Audit file will contain specific elements. This section details what to expect in the Audit file when particular events occur.

=== Startup Information ===

On succesfull start up of Fusion Edge Server, 9 events will be written to the Audit Log. These can be identified by the Process Ids of "LOAD_PROPERTIES", "APPLICATION_START" and "ENVIRONMENT".

The first event has a Process Id of "LOAD_PROPERTIES". This states that the Edge Server properties file has successfully loaded.

Next (although will likely appear last in the list of 9) is a Process Id of "APPLICATION_START" which contains all of the properties of the system. This spawns 5 child processes, all the with Process Ids of "APPLICATION_START". Each of these children has a property object with a single class within it. These classes are: SpringBeansContainer; EdgeServerAuditPersistenceManager; EdgeServerLedgerReaderManager; AuditEventManager; SDMXCacheManager

The process "EdgeServerLedgerReaderManager" will also launch a child process which has a Process Id of "ENVIRONMENT" and an "event_type" of "UPDATE". This in turn launches a child process with Process Id of "ENVIRONMENT" and an "event_type" of "LIVE".

Since child processes are written before their parents, the logical tree structure of the audit events is as follows. The number at the start of the line shows the likely order it will be written to the Audit file:

1. process_id: LOAD_PROPERTIES
9. process_id: APPLICATION_START - with all the system props
2. process_id: APPLICATION_START - Class: SpringBeansContainer
3. process_id: APPLICATION_START - Class: EdgeServerAuditPersistenceManager
6. process_id: APPLICATION_START - Class: EdgeServerLedgerReaderManager
5. process_id: ENVIRONMENT event_type: UPDATE
4. process_id: ENVIRONMENT event_type: LIVE
7. process_id: APPLICATION_START - Class: AuditEventManager
8. process_id: APPLICATION_START - Class: SDMXCacheManager

=== Data Request ===

When a request for Data is performed, a single audit event is created with the following elements:

* process_id: REST_API
* event_type: GET
* Path: /ws/public/sdmxapi/rest or /sdmx/v2
* PathInfo: the dataflow requested such as /data/dataflow/BIS/BIS_CBPOL/1.0

The status element will likely have a value of "200", indicating the request was handled correctly. However to determine if the request returned data or not, this needs to be checked by the element "HttpStatus" which is located within the "properties" element. If this value is "404" then no data was returned for the request.

The following, truncated, audit event shows a successful request (status and HttpStatus of 200):

{
"uid": "7dd76de6-895c-4f53-bb35-18519b85580e",
"process_id": "REST_API",
"thread": "http-nio-8084-exec-6",
"event_type": "GET",
"username": "guest",
"process_start": 1705568446233,
"process_end": 1705568450320,
"duration": 4087,
"status": 200,
"vmid": "27bece96ce048b52:9c501c3:18d194af243:-8000",
"machine_id": "DESKTOP-DSTGA0Q/192.168.1.16",
"software_version": "4.8.2",
"properties": {
"QueryParameters": {
"c[FREQ]": "A"
},
"HttpHeaders": {
"host": "localhost:8084",
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0",
...
},
"IP": "127.0.0.1",
"Path": "/sdmx/v2",
"PathInfo": "/data/dataflow/BIS/BIS_CBPOL/1.0",
"Locale": "en_GB",
"HttpStatus": 200
}
}

The following (truncated) event shows a request which returned no data. Note that the differences are the UID, properties (the requested parameters) and the HttpStatus of 404:
{
"uid": "0e55c832-8844-46a5-9fa5-6aa6875e66b6",
"process_id": "REST_API",
...
"status": 200,
...
"properties": {
"QueryParameters": {
"c[FREQ]": "ZZZ"
},
"HttpHeaders": {
...
},
"IP": "127.0.0.1",
"Path": "/sdmx/v2",
"PathInfo": "/data/dataflow/BIS/BIS_CBPOL/1.0",
"Locale": "en_GB",
"HttpStatus": 404
}
}

If the request contain a ResponseFormat request (e.g. ?format=sdmx-json ) then a child audit event will also be written which contains the "ResponseFormat" in a more explicit format. For example: "sdmx-json v2.0.0". This child event may look like this and is identifiable from the "parent" element which has the UID of the event that spawned it:

{
"uid": "84f98074-0176-48a2-95ba-95d3b1465aad",
"parent": "7609d4fb-1583-4440-8a3e-4ec76c324455",
"process_id": "SDMX_GET",
"thread": "http-nio-8084-exec-7",
"event_type": "structure",
"username": "guest",
"process_start": 1705527795701,
"process_end": 1705527795798,
"duration": 97,
"status": 200,
"properties": {
"Cache": "miss",
"ResponseFormat": "sdmx-json v2.0.0"
}
}

=== Structure Request ===

When a request for Structural Metadata is performed, a single audit event is created with the following elements:
* process_id: REST_API
* event_type: GET
* Path: /ws/public/sdmxapi/rest or /sdmx/v2
* PathInfo: the structure requested such as /structure/agencyscheme/all/all

This audit event is extremely similar to the data request, and likewise if no information was returned, the "httpstatus" will be 404, and if the request contain a ResponseFormat request (e.g. ?format=sdmx-3.0 ) then a child audit event will also be written which contains the "ResponseFormat" in a more explicit format. For example: "SDMX_V3_STRUCTURE_DOCUMENT"

=== Availability Request from Fusion Data Browser ===

If the Fusion Data Browser is being used against the Fusion Edge Server, availability requests may be audited. These can be identified by the following elements:
* process_id: REST_API
* event_type: GET
* Path: /sdmx/v2
* PathInfo: /availability followed by the dataflow being queried. For example: /availability/dataflow/BIS/BIS_CBPOL/1.0

The field: "properties" -> "QueryParameters" -> "mode" will also contain the value "available"

=== Search Request from Fusion Data Browser ===

If the Fusion Data Browser is being used against the Fusion Edge Server, search requests may be audited. These can be identified by the following elements:
* process_id: REST_API
* event_type: GET
* Path: /ws/public
* PathInfo: /datasearch

The field: "properties" -> "QueryParameters" -> "query" will contain the search value. Since this requested was created by the Data Browser, the Http Header, "referer" field will state the location of the Data Browser.

Since the search field of Fusion Data Browser performs searches as the user types, you may see multiple similiar or sub-searches when a user makes a single search. For example if the user wants to search for "CREDIT", you may see audit entries for searches for "C", "CR", "CRE", "CRED", "CREDI" and "CREDIT" as the user types.

== Distinguishing Between Different Request Mechanisms ==
Requests to the Edge server can come from a number of different mechanisms. For example a request from a Web Browser, a command-line tool such as curl, or the Fusion Data Browser.

The "user-agent" field within the "HttpHeaders" object can help here. Some tools, such as curl, supply an obvious distinct "user-agent". The following snippet shows a request that has come from curl

"properties": {
"HttpHeaders": {
"host": "MyTestEdgeServer:8084",
"user-agent": "curl/8.4.0",
"accept": "*/*"
},

Other values you may encounter depend upoin the user's configuration. The following is the user-agent value for Firefox v121.0 running on Windows 10:
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0"

And the following is the user-agent for Microsoft Edge, running on Windows 10:
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 Edg/120.0.0.0",

For the Fusion Data Browser, this identifies itself in a slightly different manner. There is a referer field under the objects: "properties" -> "HttpHeaders", which shows that the request originated from the Fusion Data Browser. The value in the "referer" field will contain the home location of the Fusion Data Browser:

"properties": {
"QueryParameters": {
"locale": "en",
"saveAs": "BIS_CBPOL",
"format": "csv",
"labels": "both"
},
"HttpHeaders": {
..
"referer": "http://localhost:8084/FusionDataBrowser/",
}
}

== Response Formats ==
When a request format is specified on a data request, a child audit event will be created. Note this will be output before the parent audit event. The child event contains a value under "properties" -> "ResponseFormat" to state what the output was.

The following table lists the response format for each of the specified format values:

{| class="wikitable"
|-
! Format Value !! Output String
|-
| sdmx-compact-2.1 || "Structure Specific (Compact) 2.1"
|-
| sdmx-generic-2.1 || "Generic 2.1"
|-
| sdmx-compact-2.0 || "Structure Specific (Compact) 2.0"
|-
| sdmx-generic-2.0 || "Generic 2.0"
|-
| sdmx-csv-2.0.0 || "csv"
|-
| sdmx-json || "SDMX-JSON"
|-
| sdmx-edi || "SDMX-EDI"
|-
| excel || "Excel (XLSX)"
|-
| ?? || "fusion-excel-table" |-
| ?? || "fusion-excel-series" |-
| ?? || "fusion-excel-ts" |-
| ?? || "Excel Refreshable" |-
|}