Data Publication Web Service

Overview

Importing data is an asynchronous request to the Fusion Registry, which recieves a token in the response which can be used to track the progress of import. This page describes both the submission web service, and the progress report web service.

Entry Point	/ws/public/data/publish
Access	Secure Admin or Data Provider
Http Method	POST
Accepts	CSV, XLSX, SDMX-ML, SDMX-EDI (any format for which there is a Data Reader)
Compression	Zip files supported, if loading from URL gzip responses supported
Content-Type	1. multipart/form-data (if attaching file) – the attached file must be in field name of uploadFile 2. application/text or application/xml (if submitting data in the body of the POST)
Response Format	Determined by Accept Header - default SDMX 2.1 Structure Specific
Response Statuses	200 - Submisson Accepted 400 - Submission could not be performed (either an unreadable dataset, or unresolvable reference to a required structure) 401 - Unauthorized (if access has been restricted) 500 - Server Error

HTTP Headers

The Accept Header is used to define the output format, to transform the data to. The supported accept header values are

In addition, the following optional header parameters can be used to provide further details on the incoming dataset. If these details are not provided, the Fusion Registry will interrogate the dataset header to get the information. If the dataset is a non-SDMX format, or does not contain the required information in the header, then an error response will be returned.

HTTP Header	Purpose	Allowed Values
ProvisionURN	This can be used to tell the Fusion Registry which Provision Agreement to use for Data Import. This will override the Dataset Header information if it exists.	A valid Provision Agreement URN
DatasetAction	Specifies the action to perform when submitting the dataset. If this value is specified it will override any action specified in the dataset.	Append, Replace or Delete

Server Response

The server will respond with a JSON object containing a token to track the progress of the import.

{
  "Id" : "unique-token-as-a-UUID"
} 

Error Response

If the server rejects the submission request, it will reply with an error reponse in JSON format:

{
 "Error": [
  "Error while attempting to import data",
  "Data Publishing is not supported for Provision Agreement 'urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=WB:WDI_ECON_POL_DEBT(1.0)'.  Provision Agreement must be configured to using a Registry Managed or Fusion Data Store.  Provision Agreement has been configured to use data from the following source: 'Registration (FILE)'"
]}

Check Import Progress

The progress of the import is checked by posting the Id response from the data submission to poll the following service

Entry Point	/ws/public/data/status/find
Access	Secure Admin or Data Provider
Http Method	GET
Reponst Format	application/json
Response Statuses	200 - Import Check Success 401 - Unauthorized (if access has been restricted) 500 - Server Error

HTTP Query Parameters

Request Parameter	Required	Purpose
id	Yes	This is the token that was passed back from the Fusion registry server on the publication request

Progress Check Response

The response status, any errors, import action, and general details about the Dataset.

{ 
  "Status": "SUCCESS", 
  "Username": "user_id", 
  "Start": 1464781420000, 
  "Completion": 1464781422000, 
  "Errors": [],    
  "Keys": 960, 
  "Obs": 11220, 
  "Action": "Information", 
  "DataflowUrn": "urn…", 
  "ProvisionUrn": "urn…", 
  "DatasetId": "2401cfbb-1783-4ee2-ad4f-d900332e0034" 
} 

Status returned can be one of the following:

• PENDING – the data is undergoing initial validation
• QUEUED – the data has been validated and is awaiting database imoport
• PROCESSING – the data is being loaded into the database
• PROCESSED – the data has been imported but not yet indexed
• INDEXING – the fusion registry is reindexing the data
• SUCCESS – the data import has completed with no errors
• ERROR – the import resulted in error

If the status is PROCESSING then the response Object will also contain PercentageComplete as a number, for example:

{ 
  "Status": "PROCESSING", 
  "PercentageComplete": 72,    ... 
} 

The DatasetId is the unique identifier for the dataset, and will match the 'id' query parameter.

NOTE: Start and End Dates is the measured in number of milliseconds since January 1st 1970.

Publish Error

If the data publication resulted in error the Errors feild is populated with the error message(s).  The data publish is set to fail on first error, so the array of errors are only relevent if the error message contains a number of levels of detail.  For a more comprehensive error report, use the Data Validation web service, or the Asynchronous equivalent.

{
  Status: "ERROR",
  Username: "root",
  Start: 1589832922000,
  Completion: 1589832922000,
  Errors: [ "C=Constraint,P=SERIES,DS=0,K=A:BN_KLT_DINV_CD:AUS,Comp=REF_AREA:AUS,M=Disallowed Dimension Value: REF_AREA=AUS"],
  Action: "Information",
  DataflowUrn: "urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=WB:WDI_ECON_POL_DEBT(1.0)",
  ProvisionUrn: "urn:sdmx:org.sdmx.infomodel.registry.ProvisionAgreement=WB:WB_ECON_DP1(1.0)",
  DatasetId: "1af3cb4f-979c-4956-b595-a107f6926ff7"
}