Vmurrell at 08:40, 6 October 2025

2025-10-06T08:40:48Z

Vmurrell at 13:06, 8 August 2023

2023-08-08T13:06:46Z

Vmurrell: Created page with "Category:WebService Category:How_To V11 = Overview = The Data Process Service is a web service hosted by Fusion Registry which accepts a dataset (multiple formats supp..."

2023-08-08T13:03:17Z

Created page with "Category:WebService Category:How_To V11 = Overview = The Data Process Service is a web service hosted by Fusion Registry which accepts a dataset (multiple formats supp..."

New page

[[Category:WebService]]
[[Category:How_To V11]]
= Overview =
The Data Process Service is a web service hosted by Fusion Registry which accepts a dataset (multiple formats supported) and a process instruction, which tells the Fusion Registry how to process the data, for example validate, map, transform, export.

[[File:FR Process Example.jpg|thumb]]

= Workflow =

The Data Process Service has the following workflow

# '''Load Data''' - Data is submitted using HTTP Post. The server provides a token to be able to perform further actions on the data
# '''Run Process''' - The process to execute is sent to the server, along with the token of which dataset to apply the process to. The server provides a token to track the process
# '''Track Progress''' - A GET request to track the progress of a running process, using the processes' unique token as supplied by the server
# '''Export Data''' - If the final stage of a Process is to store the data in a temporary store, then it can be exported in any data format supported by Fusion Registry

Any data loaded is stored for 2 minutes, if there is no activity on the data 2 minutes after it is loaded, it will be evicted from the system.

== Load Data ==
=== Purpose ===
To load a dataset into the system, it is stored once loaded to await a process request. Loaded data may have more then one process request executed against it. After 2 minutes of inactivity, the loaded data is removed from the system.

=== Service Definition ===
{| class="wikitable"
|-
|style="background-color:#eaecf0"|Entry Point|| /ws/secure/dataprocess/load
|-
|style="background-color:#eaecf0"|Access|| Restricted
|-
|style="background-color:#eaecf0"|Http Method|| POST
|-
|style="background-color:#eaecf0"|Accepts|| Any Supported Data Format
|-
|style="background-color:#eaecf0"|Compression || Zip files supported
|-
|style="background-color:#eaecf0"|Content-Type|| 1. multipart/form-data (if attaching file) – the attached file must be in field name of uploadFile
2. application/text or application/xml (if submitting data in the body of the POST)
|-
|style="background-color:#eaecf0"|Response Format|| JSON
|-
|style="background-color:#eaecf0"|Response Statuses|| 200 - Data file received
400 - If not dataset provided
401 - Unauthorized (if access has been restricted)
500 - Server Error
|}

'''Response'''
{
"token" : "uid123"
}

== Run Process ==
=== Purpose ===
Runs a process against the dataset, the purpose of running a process can include transformation, mapping, data manipulation

=== Service Definition ===
{| class="wikitable"
|-
|style="background-color:#eaecf0"|Entry Point|| /ws/secure/dataprocess/run/{token}
|-
|style="background-color:#eaecf0"|Access|| Restricted
|-
|style="background-color:#eaecf0"|Http Method|| POST
|-
|style="background-color:#eaecf0"|Accepts || Process Instruction (JSON format)
|-
|style="background-color:#eaecf0"|Content-Type|| 1. multipart/form-data (if attaching file) – the attached file must be in field name of uploadFile
2. application/json (if submitting data in the body of the POST)
|-
|style="background-color:#eaecf0"|Response Format|| JSON
|-
|style="background-color:#eaecf0"|Response Statuses|| 200 - Data file received
400 - If the token does not match a known dataset
401 - Unauthorized (if access has been restricted)
500 - Server Error
|}

'''Response'''
{
"token" : "uid123"
}

=== Process Instruction ===
The process instruction chains together one or more Process Steps. Each Process Steps has an Identification (StepId) links to a [[DRAFT_Data_Process_Service#Supported_Processes|supported process]] (VALIDATE). The supported Properties of a Process Step are specific to the Process being run, a typical example for processes which output a dataset is the '''Output''' property which defines which Process Step to pass the output to.

{
"Structure" : "urn" //Provision, Dataflow, or DSD URN used to read the data, only required if the dataset does not contain this, or if overriding the Dataset
"ProcessSteps" : [
{
"ProcessId" : "VALIDATE" //Id of a registered process
"StepId" : "STEP1" //Unique Id for the step
"Metrics" : true //true to capture metrics for the process which are output in the track progress report
"Properties" : {} //optional map of properties specific to the process being run
}
]
}

== Track Process ==
=== Purpose ===
To track the execution status of a process request.

=== Service Definition ===
{| class="wikitable"
|-
|style="background-color:#eaecf0"|Entry Point|| /ws/secure/dataprocess/status/{token}
|-
|style="background-color:#eaecf0"|Access|| Restricted
|-
|style="background-color:#eaecf0"|Http Method|| GET
|-
|style="background-color:#eaecf0"|Response Format|| JSON
|-
|style="background-color:#eaecf0"|Response Statuses|| 200 - Data file received
404 - If the token does not match a known process
401 - Unauthorized (if access has been restricted)
500 - Server Error
|}

'''Response Example'''
{
"ProcessToken" : "abcd",
"StartTime" : 123456, //unix time milliseconds since 1970
"EndTime" : null, //unix time milliseconds since 1970
"Status" : 1 //0=running,1=success,2=error
"Steps" :
{
"Step1" :
{
"StartTime" : 123456, //unix time milliseconds since 1970
"EndTime" : null, //unix time milliseconds since 1970
"Series" : 12,
"Rows" : 132
}
}
}

== Export Data ==
=== Purpose ===
Any data that is sent to the a process with Id STORE_TMP can be exported in any format that the Fusion Registry supports. This enables datasets to be exported in different formats from which they were loaded, and as this service can be called multiple times, the dataset can be exported in more then one format.

=== Service Definition ===
{| class="wikitable"
|-
|style="background-color:#eaecf0"|Entry Point|| /ws/secure/dataprocess/download/{token}/{stepId}
|-
|style="background-color:#eaecf0"|Access|| Restricted
|-
|style="background-color:#eaecf0"|Http Method|| GET
|-
|style="background-color:#eaecf0"|Response Format|| JSON
|-
|style="background-color:#eaecf0"|Response Statuses|| 200 - Data file received
404 - If the token does not match a known process
401 - Unauthorized (if access has been restricted)
500 - Server Error
|}

{| class="wikitable"
|+ Request Parameters
|-
! Parameter !! Required !! Description
|-
| saveAs || False || If provided, the response will include a content-disposition HttpHeader with the value attachment; filename = {param value}
|-
| format || False || This can be used to define the export data format, as opposed to using the HTTP Accept Header. @see formats reference for valid values for each format.
|}

= Supported Processes =
== Mapping ==
'''Process ID: ''' STRUCTURE_MAP

=== Purpose ===
Uses a Structure Map to map data conforming to one Data Structure Definition to another Data Structure Definition. Mapped output is sent to the Output step, any unmapped data is sent to the UnmappedStep (if configured to do so)

=== Parameters ===
{| class="wikitable"
|-
! Parameter Id !! Required !! Value
|-
| Output || true || String. The Process Step ID to pass the mapped data to
|-
| Map || true || URN of the Structure Map to use
|-
| UnmappedStep || false || The Process Step ID to pass the unmapped data to (if no mapping found)
|}

== Dynamic Component ==
'''Process ID: ''' DYNAMIC_COMPONENT

=== Purpose ===
Enriches a dataset by adding one or more components whose values can be fixed, or contain content taken from other reported values for the same row of data. For example the creation of a Series Title based on the reported Indicator, Reference Area, and Frequency

=== Parameters ===
{| class="wikitable"
|-
! Parameter Id !! Required !! Value
|-
| Output || true || String. The Process Step ID to pass the mapped data to
|-
| DynamicComponents || true || A JSON Map of Component Id to value to use
 
'''Variable Placeholder''' The value can contain variable using the syntax $[COMPONENT_ID.{postfix}] where postfix is either '''name''' or '''desc''' which will take the component's Name or Description respectively
|}

'''Example'''
{
"Output" : "Step3",
"DynamicComponents" :
{
"SERIES_TITLE" : "$[INDICATOR.name] for $[REF_AREA.name]"
"OBS_STATUS" : "A"
}
}

== Temporary Store (for download) ==
'''Process ID: ''' STORE_TMP

=== Purpose ===
This process step is a terminal step, which stores the data in a location which will be available for download via the download service. The store is preserved for 10 minutes before being removed from the system.

=== Parameters ===
''Not Applicable''

@@ Line 1: / Line 1: @@
-[[Category:WebService]]
+[[Category:RetiredFunctionality]]
 = Overview =
 The Data Process Service is a web service hosted by Fusion Registry which accepts a dataset (multiple formats supported) and a process instruction, which tells the Fusion Registry how to process the data, for example validate, map, transform, export.

DRAFT Data Process Service - Revision history

Vmurrell at 08:40, 6 October 2025

Vmurrell at 13:06, 8 August 2023

Vmurrell: Created page with "Category:WebService Category:How_To V11 = Overview = The Data Process Service is a web service hosted by Fusion Registry which accepts a dataset (multiple formats supp..."