Pull data from third party REST API

Overview

If an organisation is hosting a SDMX REST API, then it is possible to pull structural metadata and data from this service into your local Fusion Registry instance via a number of web service calls. There are two ways this can be achieved. The simplest way is to use the Fusion Registry data portal feature (in Beta). Another approach is to use the Fusion Registry load structures/data pages and insert the appropriate REST URL. The benefit of the data portal is that the Fusion Registry does all of the work for you, it handles errors and merging of structures, it will even split the data query into multiple parts if the end service does not support large queries. This how to guide discusses both approaches.

Example REST API

For the purpose of this 'how to' the web service of the European Central Bank will be used, the API can be found by Googling 'ECB SDMX REST API' which at the time of writing links to this page:

The important thing to look for in a REST API is the entry point. The URL is the constructed by taking a base URL and postfixing the query parameters to the end, for data queries this starts with the word 'data' and for structure queries it starts with the structure type being queried, for example 'codelist'. On the ECB REST API page, example queries are provided which look like this:

http://ec.europa.eu/eurostat/SDMX/diss-web/rest/data/nama_10_gdp/.CLV10_MEUR.B1GQ.BE/?startperiod=2005&endPeriod=2011

The important part of the above URL is everything before the path /data/, this is the entry point to their web service, which is as follows:

http://ec.europa.eu/eurostat/SDMX/diss-web/rest/

We can use this URL to construct queries for both data and structures, for example

http://ec.europa.eu/eurostat/SDMX/diss-web/rest/codelist = query for all codelists

Pull Data using Data Portal Feature

The Fusion Registry has a Beta feature called the Data Portal. To access this feature, log in as an Admin user and navigate to the Fusion Data Portal which is available at the following URL:

https://your.server/FusionRegistry/admin/portal.html

Click the 'Add Service' button and complete the form, which must include the REST entry point, service name, and if approprite the username and password for the external service. A username and password should only be provided if the external service requires authentication and supports HTTP Basic Authentication, if this is not the case then leave this blank.

The portal can be set up to synchronise all datasets, specific datasets, or structures only. If the external datasource has a lot of datasets, then a subset should be selected unless all datasets are required. The more datasets that are synchronised, the more memory the Fusion Registry will require to maintain the indexes. The following video demonstrates the memory required to store datasets in memory.