Difference between revisions of "Fusion Registry Command Line"

From Fusion Registry Wiki
Jump to navigation Jump to search
(Created page with "= Overview = The Fusion Registry command line client (FusionRegistryCL) runs on a Windows or UNIX command line, to support loading content to the Fusion Registry via the Fusi...")
 
(→‎Data Validation)
 
(12 intermediate revisions by one other user not shown)
Line 9: Line 9:
  
 
== Structures ==
 
== Structures ==
−
To Publish structures call the publishStructures.bat (or sh) with the following commands:
+
To Publish structures call the '''publishStructures.bat''' (or sh) with the following commands:
  
 
{| class="wikitable"
 
{| class="wikitable"
Line 42: Line 42:
 
|-
 
|-
 
| u || false  || Username of a Fusion Registry user with permissions to load the structure file.<br/>
 
| u || false  || Username of a Fusion Registry user with permissions to load the structure file.<br/>
−
Not required if using Certificates for Authentication.|| -u myuser
+
Not required if using Certificates for Authentication.  
 +
|| -u myuser
 
|-
 
|-
 
| p || false || Password of a Fusion Registry user with permissions to load the structure file. <br/>Not required if using Certificates for Authentication. || -p mypass
 
| p || false || Password of a Fusion Registry user with permissions to load the structure file. <br/>Not required if using Certificates for Authentication. || -p mypass
Line 54: Line 55:
 
| certType|| false || The Certificate Type to use. Defaults to PKCS12 if not specified || -certType "PKCS12"
 
| certType|| false || The Certificate Type to use. Defaults to PKCS12 if not specified || -certType "PKCS12"
 
|}
 
|}
 +
 +
 +
== Data ==
 +
To Publish data call the '''publishData.bat''' (or sh) with the following commands:
 +
 +
{| class="wikitable"
 +
|-
 +
! Argument !! Required !! Purpose !! Example
 +
|-
 +
| uri || true || The location of the Data to load. <br/>This can resolve to a web URL or local file system. <br/>The format of the data file can be any supported Fusion Registry format. <br/>The file may also be in zip format. <br/>If this URI resolves to a folder on the file system, all the files in the folder will be submitted for import via a single zip file. <br/>It is possible to load multiple files/folder’s by supplying multiple Uris each separated by a semicolon
 +
||
 +
'''File''' <br/>
 +
-uri "file:///C:/Sample/Data.xml" <br/>
 +
-uri "file:///C:/Sample/Data.zip" <br/><br/>
 +
'''Folder''' <br/>
 +
-uri "file:///C:/SampleDir" <br/><br/>
 +
'''URL''' <br/>
 +
-uri "http://myserver/rest/data" <br/>
 +
-uri "http://myserver/file/data.xml" <br/><br/>
 +
'''Multiple''' <br/>
 +
-uri "http://myserver/rest/data; http://myserver/file/data.xml; file:///C:/SampleDir"
 +
|-
 +
| ref|| false || Optionally a Provision Agreement reference can be supplied, so the server can identify the Dataflow and Data Provider.
 +
This is not required if the data file contains this information.<br/>
 +
The format is Agency Id, Provision Id, Provision Version.
 +
|| -ref "ACY,PROV_ID,1.0"
 +
|-
 +
| u || false  || Username of a Fusion Registry user with permissions to load the structure file.<br/>
 +
Not required if using Certificates for Authentication.
 +
|| -u myuser
 +
|-
 +
| p || false || Password of a Fusion Registry user with permissions to load the structure file. <br/>Not required if using Certificates for Authentication. || -p mypass
 +
|-
 +
| o || false || An output file to record which structures were added, modified or removed. || -o "C:/temp/Result.csv"
 +
|-
 +
| cert || false || File of Certificate to be used to authenticate with server (if username and password not used). <br/>Not required if using Username/Password for Authentication. || -cert "C:/Cert/mycert.p12"
 +
|-
 +
| certPwd || false || The Certificate Password. || -certPwd "password"
 +
|-
 +
| certType|| false || The Certificate Type to use. Defaults to PKCS12 if not specified || -certType "PKCS12"
 +
|}
 +
 +
== Reference Metadata ==
 +
'''Note''' Discontinued at version 2.0.0
 +
 +
To Publish metadata call the '''publishMetadata.bat''' (or sh) with the following commands:
 +
 +
{| class="wikitable"
 +
|-
 +
! Argument !! Required !! Purpose !! Example
 +
|-
 +
| uri || true || TThe location of the Metadata to load. <br/>This can resolve to a web URL or local file system. <br/>The format of the file can be any supported Fusion Registry format. <br/>The file may also be in zip format.
 +
||
 +
'''File''' <br/>
 +
-uri "file:///C:/Sample/Metadata.xml" <br/>
 +
-uri "file:///C:/Sample/Metadata.zip" <br/><br/>
 +
'''URL''' <br/>
 +
-uri "http://myserver/rest/metadata" <br/>
 +
-uri "http://myserver/file/metadata.xml"
 +
|-
 +
| a || false || The action.<br/> If the action is DELETE and a metadata file is provided, then the submission will delete the metadata.
 +
If the action is DELETE and a metadata file is not provided, then an id argument is expected.
 +
||  -a "DELETE"
 +
|-
 +
| id || false || If the action is DELETE and a URI is not provided, then this argument can be used to identify the metadata set to delete by its id.
 +
||  -id "MY_METADATASET"
 +
|-
 +
| u || false  || Username of a Fusion Registry user with permissions to load the structure file.<br/>
 +
Not required if using Certificates for Authentication.
 +
|| -u myuser
 +
|-
 +
| p || false || Password of a Fusion Registry user with permissions to load the structure file. <br/>Not required if using Certificates for Authentication. || -p mypass
 +
|-
 +
| o || false || An output file to record which structures were added, modified or removed. || -o "C:/temp/Result.csv"
 +
|-
 +
| cert || false || File of Certificate to be used to authenticate with server (if username and password not used). <br/>Not required if using Username/Password for Authentication. || -cert "C:/Cert/mycert.p12"
 +
|-
 +
| certPwd || false || The Certificate Password. || -certPwd "password"
 +
|-
 +
| certType|| false || The Certificate Type to use. Defaults to PKCS12 if not specified || -certType "PKCS12"
 +
|}
 +
 +
= Exporting Content =
 +
== Data ==
 +
A full data export can be run against a Dataflow, in a specified format. The export is written to a file, whose name is specified as an argument to the export function.
 +
 +
To export data call the '''exportData.bat''' (or sh) with the following commands:
 +
 +
{| class="wikitable"
 +
|-
 +
! Argument !! Required !! Purpose !! Example
 +
|-
 +
| ex || true || The dataflow to export all the data for, the format is Agency Id, Dataflow Id, Dataflow Version || -ex "ACY,DF_ID,1.0"
 +
|-
 +
| exfmt || true || The Export format. Allowed values vary by format, please refer to the [https://wiki.sdmxcloud.org/Category:SdmxDataFormat Formats] page to select a data format and check the parameters for that format || -exfmt "[[SDMX-ML_Generic_Data|sdmx-generic-2.1]]" <br/>  -exfmt "[[SDMX-CSV|sdmx-csv&labels=both&serieskey=exclude]]"
 +
|-
 +
| exf || true ||
 +
Export file, the file will be created if it does not exist, or deleted and recreated if it does exist.
 +
 +
To export as a compressed zip file, ensure the file name ends in .zip
 +
 +
If the file postfix is not provided, it will be determined by the server based on the export format
 +
 +
The file name may include placeholders for Dataflow Agency Id (%agency%), Dataflow Id (%id%), and Dataflow version (%version%).
 +
 +
The placeholders will be replaced with the values from the dataflow
 +
||
 +
'''XML''' <br/>
 +
-exf "genricdata.xml" <br/><br/>
 +
'''ZIP''' <br/>
 +
-exf "genericdata.zip" <br/><br/>
 +
'''Placeholders''' <br/>
 +
-exf "%agency%_%id%_%version%" <br/><br/>
 +
'''No Postfix''' <br/>
 +
-exf "myexport"
 +
|-
 +
| s|| false || Timeout time in seconds, default is 120 || -s 600 
 +
|-
 +
| pp || false || Pretty Print.  Only required for SDMX-ML data formats || -pp true
 +
|-
 +
| params || false || Query Parameters.  Additional REST query parameters to use in the request || -params "param1=value1&param2=value2"
 +
|-
 +
|}
 +
 +
= Deleting Content =
 +
== Data ==
 +
{| class="wikitable"
 +
|-
 +
! Argument !! Required !! Purpose !! Example
 +
|-
 +
| df || df or pa is required || The dataflow to delete all the data for, the format is Agency Id, Dataflow Id, Dataflow Version || -df "ACY,DF_ID,1.0"
 +
|-
 +
| pa || df or pa is required ||  The provision agreement to delete all the data for, the format is Agency Id, Dataflow Id, Dataflow Version || -pa "ACY,PA_ID,1.0"
 +
|-
 +
| ds || true ||  The Id of the Data Source to delete the data from. (see Data Source Manager of Fusion Registry) ||
 +
'''Fusion Store''' <br/>
 +
-ds FUSION <br/><br/>
 +
'''Custom Data Store'''<br/>
 +
-ds MY_SQL_STORE
 +
|-
 +
| u || false  || Username of a Fusion Registry user with permissions to load the structure file.<br/>
 +
Not required if using Certificates for Authentication.
 +
|| -u myuser
 +
|-
 +
| p || false || Password of a Fusion Registry user with permissions to load the structure file. <br/>Not required if using Certificates for Authentication. || -p mypass
 +
|-
 +
| o || false || An output file to record which structures were added, modified or removed. || -o "C:/temp/Result.csv"
 +
|-
 +
| cert || false || File of Certificate to be used to authenticate with server (if username and password not used). <br/>Not required if using Username/Password for Authentication. || -cert "C:/Cert/mycert.p12"
 +
|-
 +
| certPwd || false || The Certificate Password. || -certPwd "password"
 +
|-
 +
| certType|| false || The Certificate Type to use. Defaults to PKCS12 if not specified || -certType "PKCS12"
 +
|}
 +
 +
= Data Validation =
 +
To validate data call the '''validateData.bat''' (or sh) with the following commands:
 +
 +
The output is the same as [[Data_Validation_Web_Service#Response|calling the web service directly]].
 +
 +
{| class="wikitable"
 +
|-
 +
! Argument !! Required !! Purpose !! Example
 +
|-
 +
| dsd || false || The Data Sructure to validate the dataset against. <br/><br/>'''Note:''' dsd, df, and pa are mutually exclusive || -dsd "ACY,DSD_ID,1.0"
 +
|-
 +
| df || false || The Dataflow to validate the dataset against. <br/><br/>'''Note:''' dsd, df, and pa are mutually exclusive || -df "ACY,DF_ID,1.0"
 +
|-
 +
| pa || false || The Provision Agreement to validate the dataset against. <br/><br/>'''Note:''' dsd, df, and pa are mutually exclusive || -pa "ACY,PA_ID,1.0"
 +
|-
 +
| uri || true || The data file to validate ||
 +
'''File'''<br/>
 +
-uri "file:///C:/Sample/Data.xml"<br/>
 +
-uri "file:///C:/Sample/Data.zip"<br/><br/>
 +
'''URL'''<br/>
 +
-uri "http://myserver/rest/data"<br/>
 +
-uri "http://myserver/file/data.xml"
 +
|-
 +
| f || true || Denotes the format and delimiter. Only required if data format is CSV. See data-format in Fusion Registry web services guide (validation section) || csv;delimiter=[delimiter] <br/> Where [delimiter] is either: 
 +
* comma
 +
* tab
 +
* semicolon
 +
'''Comma Separated'''<br/>
 +
-f csv:delimiter=comm
 +
|-
 +
| o || false|| The output file, if not present then the validation report will be written to the System.out ||
 +
'''Windows'''<br/>
 +
-o c:/mydir/myFile.json
 +
'''Unix'''<br/>
 +
-o /mydir/myFile.json
 +
|-
 +
| u || false  || Username of a Fusion Registry user with permissions to load the structure file.<br/>
 +
Only required if Fusion Registry is enforcing user login and if not using Certificate for Authentication
 +
|| -u myuser
 +
|-
 +
| p || false || Password of a Fusion Registry user with permissions to load the structure file. <br/>Only required if Fusion Registry is enforcing user login and if not using Certificate for Authentication. || -p mypass
 +
|-
 +
| cert || false || File of Certificate to be used to authenticate with server (if username and password not used). <br/>Only required if Fusion Registry is enforcing user login and not required if using Username/Password for Authentication. || -cert "C:/Cert/mycert.p12"
 +
|-
 +
| certPwd || false || The Certificate Password. || -certPwd "password"
 +
|-
 +
| certType|| false || The Certificate Type to use. Defaults to PKCS12 if not specified || -certType "PKCS12"
 +
|}
 +
 +
= Chane Log =
 +
See [[Change_Log_Registry_Command_Line|Command Line Change Log]]

Latest revision as of 01:56, 6 October 2023

Overview

The Fusion Registry command line client (FusionRegistryCL) runs on a Windows or UNIX command line, to support loading content to the Fusion Registry via the Fusion Registry secure web services.

FusionRegistryCL is distributed with a single properties file fusionregistrycl.properties. This file needs to be modified to point to the server URL of the Fusion Registry to communicate with.

Batch files (Windows) and sh files (UNIX) are provided to support loading (and deleting) structures, data, and reference metadata.

Publishing Content

Structures

To Publish structures call the publishStructures.bat (or sh) with the following commands:

Argument Required Purpose Example
uri true The location of the Structures to load.
This can resolve to a web URL or local file system.
The format of the structure file can be any supported Fusion Registry format.
The file may also be in zip format.

File
-uri "file:///C:/Sample/Structures.xml"
-uri "file:///C:/Sample/Structures.zip"
URL
-uri "http://myserver/rest/conceptscheme"
-uri "http://myserver/file/structures.xml"

ds false The Data Source argument will create a link between all submitted Provision Agreements and a Data Source in the Fusion Registry. See Data Source Manager of Fusion Registry.

If not provided the Provision Agreements will default to using the REST or File Data Registration mechanism for data.

Fusion Store

-ds FUSION
Custom Data Store
-ds MY_SQL_STORE

a false Action – allowable actions are:
  • Append
  • Replace
  • FullReplace
  • Delete

The publish action, defaults to Replace.

Delete
-a DELETE
REPLACE
-a REPLACE
FullReplace
-a FullReplace

u false Username of a Fusion Registry user with permissions to load the structure file.

Not required if using Certificates for Authentication.

-u myuser
p false Password of a Fusion Registry user with permissions to load the structure file.
Not required if using Certificates for Authentication.		 -p mypass
o false An output file to record which structures were added, modified or removed. -o "C:/temp/Result.csv"
cert false File of Certificate to be used to authenticate with server (if username and password not used).
Not required if using Username/Password for Authentication.		 -cert "C:/Cert/mycert.p12"
certPwd false The Certificate Password. -certPwd "password"
certType false The Certificate Type to use. Defaults to PKCS12 if not specified -certType "PKCS12"


Data

To Publish data call the publishData.bat (or sh) with the following commands:

Argument Required Purpose Example
uri true The location of the Data to load.
This can resolve to a web URL or local file system.
The format of the data file can be any supported Fusion Registry format.
The file may also be in zip format.
If this URI resolves to a folder on the file system, all the files in the folder will be submitted for import via a single zip file.
It is possible to load multiple files/folder’s by supplying multiple Uris each separated by a semicolon

File
-uri "file:///C:/Sample/Data.xml"
-uri "file:///C:/Sample/Data.zip"

Folder
-uri "file:///C:/SampleDir"

URL
-uri "http://myserver/rest/data"
-uri "http://myserver/file/data.xml"

Multiple
-uri "http://myserver/rest/data; http://myserver/file/data.xml; file:///C:/SampleDir"

ref false Optionally a Provision Agreement reference can be supplied, so the server can identify the Dataflow and Data Provider.

This is not required if the data file contains this information.
The format is Agency Id, Provision Id, Provision Version.

-ref "ACY,PROV_ID,1.0"
u false Username of a Fusion Registry user with permissions to load the structure file.

Not required if using Certificates for Authentication.

-u myuser
p false Password of a Fusion Registry user with permissions to load the structure file.
Not required if using Certificates for Authentication.		 -p mypass
o false An output file to record which structures were added, modified or removed. -o "C:/temp/Result.csv"
cert false File of Certificate to be used to authenticate with server (if username and password not used).
Not required if using Username/Password for Authentication.		 -cert "C:/Cert/mycert.p12"
certPwd false The Certificate Password. -certPwd "password"
certType false The Certificate Type to use. Defaults to PKCS12 if not specified -certType "PKCS12"

Reference Metadata

Note Discontinued at version 2.0.0

To Publish metadata call the publishMetadata.bat (or sh) with the following commands:

Argument Required Purpose Example
uri true TThe location of the Metadata to load.
This can resolve to a web URL or local file system.
The format of the file can be any supported Fusion Registry format.
The file may also be in zip format.

File
-uri "file:///C:/Sample/Metadata.xml"
-uri "file:///C:/Sample/Metadata.zip"

URL
-uri "http://myserver/rest/metadata"
-uri "http://myserver/file/metadata.xml"

a false The action.
If the action is DELETE and a metadata file is provided, then the submission will delete the metadata.

If the action is DELETE and a metadata file is not provided, then an id argument is expected.

-a "DELETE"
id false If the action is DELETE and a URI is not provided, then this argument can be used to identify the metadata set to delete by its id. -id "MY_METADATASET"
u false Username of a Fusion Registry user with permissions to load the structure file.

Not required if using Certificates for Authentication.

-u myuser
p false Password of a Fusion Registry user with permissions to load the structure file.
Not required if using Certificates for Authentication.		 -p mypass
o false An output file to record which structures were added, modified or removed. -o "C:/temp/Result.csv"
cert false File of Certificate to be used to authenticate with server (if username and password not used).
Not required if using Username/Password for Authentication.		 -cert "C:/Cert/mycert.p12"
certPwd false The Certificate Password. -certPwd "password"
certType false The Certificate Type to use. Defaults to PKCS12 if not specified -certType "PKCS12"

Exporting Content

Data

A full data export can be run against a Dataflow, in a specified format. The export is written to a file, whose name is specified as an argument to the export function.

To export data call the exportData.bat (or sh) with the following commands:

Argument Required Purpose Example
ex true The dataflow to export all the data for, the format is Agency Id, Dataflow Id, Dataflow Version -ex "ACY,DF_ID,1.0"
exfmt true The Export format. Allowed values vary by format, please refer to the Formats page to select a data format and check the parameters for that format -exfmt "sdmx-generic-2.1"
-exfmt "sdmx-csv&labels=both&serieskey=exclude"
exf true

Export file, the file will be created if it does not exist, or deleted and recreated if it does exist.

To export as a compressed zip file, ensure the file name ends in .zip

If the file postfix is not provided, it will be determined by the server based on the export format

The file name may include placeholders for Dataflow Agency Id (%agency%), Dataflow Id (%id%), and Dataflow version (%version%).

The placeholders will be replaced with the values from the dataflow

XML
-exf "genricdata.xml"

ZIP
-exf "genericdata.zip"

Placeholders
-exf "%agency%_%id%_%version%"

No Postfix
-exf "myexport"

s false Timeout time in seconds, default is 120 -s 600
pp false Pretty Print. Only required for SDMX-ML data formats -pp true
params false Query Parameters. Additional REST query parameters to use in the request -params "param1=value1&param2=value2"

Deleting Content

Data

Argument Required Purpose Example
df df or pa is required The dataflow to delete all the data for, the format is Agency Id, Dataflow Id, Dataflow Version -df "ACY,DF_ID,1.0"
pa df or pa is required The provision agreement to delete all the data for, the format is Agency Id, Dataflow Id, Dataflow Version -pa "ACY,PA_ID,1.0"
ds true The Id of the Data Source to delete the data from. (see Data Source Manager of Fusion Registry)

Fusion Store
-ds FUSION

Custom Data Store
-ds MY_SQL_STORE

u false Username of a Fusion Registry user with permissions to load the structure file.

Not required if using Certificates for Authentication.

-u myuser
p false Password of a Fusion Registry user with permissions to load the structure file.
Not required if using Certificates for Authentication.		 -p mypass
o false An output file to record which structures were added, modified or removed. -o "C:/temp/Result.csv"
cert false File of Certificate to be used to authenticate with server (if username and password not used).
Not required if using Username/Password for Authentication.		 -cert "C:/Cert/mycert.p12"
certPwd false The Certificate Password. -certPwd "password"
certType false The Certificate Type to use. Defaults to PKCS12 if not specified -certType "PKCS12"

Data Validation

To validate data call the validateData.bat (or sh) with the following commands:

The output is the same as calling the web service directly.

Argument Required Purpose Example
dsd false The Data Sructure to validate the dataset against.

Note: dsd, df, and pa are mutually exclusive		 -dsd "ACY,DSD_ID,1.0"
df false The Dataflow to validate the dataset against.

Note: dsd, df, and pa are mutually exclusive		 -df "ACY,DF_ID,1.0"
pa false The Provision Agreement to validate the dataset against.

Note: dsd, df, and pa are mutually exclusive		 -pa "ACY,PA_ID,1.0"
uri true The data file to validate

File
-uri "file:///C:/Sample/Data.xml"
-uri "file:///C:/Sample/Data.zip"

URL
-uri "http://myserver/rest/data"
-uri "http://myserver/file/data.xml"

f true Denotes the format and delimiter. Only required if data format is CSV. See data-format in Fusion Registry web services guide (validation section) csv;delimiter=[delimiter]
Where [delimiter] is either:
  • comma
  • tab
  • semicolon

Comma Separated
-f csv:delimiter=comm

o false The output file, if not present then the validation report will be written to the System.out

Windows
-o c:/mydir/myFile.json Unix
-o /mydir/myFile.json

u false Username of a Fusion Registry user with permissions to load the structure file.

Only required if Fusion Registry is enforcing user login and if not using Certificate for Authentication

-u myuser
p false Password of a Fusion Registry user with permissions to load the structure file.
Only required if Fusion Registry is enforcing user login and if not using Certificate for Authentication.		 -p mypass
cert false File of Certificate to be used to authenticate with server (if username and password not used).
Only required if Fusion Registry is enforcing user login and not required if using Username/Password for Authentication.		 -cert "C:/Cert/mycert.p12"
certPwd false The Certificate Password. -certPwd "password"
certType false The Certificate Type to use. Defaults to PKCS12 if not specified -certType "PKCS12"

Chane Log

See Command Line Change Log

Retrieved from "https://wiki.sdmxcloud.org/index.php?title=Fusion_Registry_Command_Line&oldid=6434"