Pull data from third party REST API - Revision history

Vmurrell at 06:23, 11 September 2023

2023-09-11T06:23:36Z

Mnelson: /* Pull Data using Data Portal Feature */

2020-11-23T21:34:46Z

‎Pull Data using Data Portal Feature

Mnelson: /* Pull Data using Data Portal Feature */

2020-11-23T20:15:37Z

‎Pull Data using Data Portal Feature

Mnelson: /* Pull Data using Data Portal Feature */

2020-11-23T20:11:59Z

‎Pull Data using Data Portal Feature

Mnelson: /* Pull Data using Data Portal Feature */

2020-11-23T20:10:30Z

‎Pull Data using Data Portal Feature

Mnelson at 20:10, 23 November 2020

2020-11-23T20:10:14Z

Mnelson: Created page with "Category:How_To ==Overview== If an organisation is hosting a SDMX REST API, then it is possible to pull structural metadata and data from this service into your local Fusi..."

2020-11-23T20:06:02Z

Created page with "Category:How_To ==Overview== If an organisation is hosting a SDMX REST API, then it is possible to pull structural metadata and data from this service into your local Fusi..."

New page

[[Category:How_To]]
==Overview==
If an organisation is hosting a SDMX REST API, then it is possible to pull structural metadata and data from this service into your local Fusion Registry instance via a number of web service calls. There are two ways this can be achieved. The simplest way is to use the Fusion Registry data portal feature (in Beta). Another approach is to use the Fusion Registry load structures/data pages and insert the appropriate REST URL. The benefit of the data portal is that the Fusion Registry does all of the work for you, it handles errors and merging of structures, it will even split the data query into multiple parts if the end service does not support large queries. This how to guide discusses both approaches.

For the purpose of this 'how to' the web service of the European Central Bank will be used, the API can be found by Googling 'ECB SDMX REST API' which at the time of writing links to this [https://ec.europa.eu/eurostat/web/sdmx-web-services/rest-sdmx-2.1 page]:

The important thing to look for in a REST API is the entry point. The URL is the constructed by taking a base URL and postfixing the query parameters to the end, for data queries this starts with the word 'data' and for structure queries it starts with the structure type being queried, for example 'codelist'. On the ECB REST API page, example queries are provided which look like this:

[http://ec.europa.eu/eurostat/SDMX/diss-web/rest/data/nama_10_gdp/.CLV10_MEUR.B1GQ.BE/?startperiod=2005&endPeriod=2011 '''http://ec.europa.eu/eurostat/SDMX/diss-web/rest/'''data/nama_10_gdp/.CLV10_MEUR.B1GQ.BE/?startperiod=2005&endPeriod=2011]

The important part of the above URL is everything before the path '''/data/''', this is the entry point to their web service, which is as follows:

'''http://ec.europa.eu/eurostat/SDMX/diss-web/rest/'''

We can use this URL to construct queries for both data and structures, for example

[http://ec.europa.eu/eurostat/SDMX/diss-web/rest/codelist http://ec.europa.eu/eurostat/SDMX/diss-web/rest/'''codelist'''] = query for all codelists

== Use Case

@@ Line 25: / Line 25: @@
 Click the 'Add Service' button and complete the form, which must include the REST entry point, service name, and if approprite the username and password for the external service.  A username and password should only be provided if the external service requires authentication and supports HTTP Basic Authentication, if this is not the case then leave this blank.
-The portal can be set up to synchronise all datasets, specific datasets, or structures only.  If the external datasource has a lot of datasets, then a subset should be selected unless all datasets are required.  The more datasets that are synchronised, the more memory the Fusion Registry will require to maintain the indexes.  The following [https://www.youtube.com/watch?v=4nLZNKTN1BE video] demonstrates the memory required to store datasets in memory.
+The portal can be set up to synchronise all datasets, specific datasets, or structures only.  If specific datasets is selected, the choice of which datasets is performed after saving the configuration.
 [[File:Fusion portal.png|External service form]]

← Older revision		Revision as of 20:15, 23 November 2020
Line 24:		Line 24:

	Click the 'Add Service' button and complete the form, which must include the REST entry point, service name, and if approprite the username and password for the external service. A username and password should only be provided if the external service requires authentication and supports HTTP Basic Authentication, if this is not the case then leave this blank.		Click the 'Add Service' button and complete the form, which must include the REST entry point, service name, and if approprite the username and password for the external service. A username and password should only be provided if the external service requires authentication and supports HTTP Basic Authentication, if this is not the case then leave this blank.
		+
		+	The portal can be set up to synchronise all datasets, specific datasets, or structures only. If the external datasource has a lot of datasets, then a subset should be selected unless all datasets are required. The more datasets that are synchronised, the more memory the Fusion Registry will require to maintain the indexes. The following [https://www.youtube.com/watch?v=4nLZNKTN1BE video] demonstrates the memory required to store datasets in memory.

	[[File:Fusion portal.png\|External service form]]		[[File:Fusion portal.png\|External service form]]

@@ Line 23: / Line 23: @@
   https://your.server/FusionRegistry/admin/portal.html
-Click the 'Add Service' button and complete the form
+Click the 'Add Service' button and complete the form, which must include the REST entry point, service name, and if approprite the username and password for the external service.  A username and password should only be provided if the external service requires authentication and supports HTTP Basic Authentication, if this is not the case then leave this blank.
 [[File:Fusion portal.png|External service form]]

← Older revision		Revision as of 20:10, 23 November 2020
Line 25:		Line 25:
	Click the 'Add Service' button and complete the form		Click the 'Add Service' button and complete the form

−	[[File:Fusion portal.png~~\|thumb~~\|External service form]]	+	[[File:Fusion portal.png\|External service form]]

@@ Line 1: / Line 1: @@
 [[Category:How_To]]
 ==Overview==
 If an organisation is hosting a SDMX REST API, then it is possible to pull structural metadata and data from this service into your local Fusion Registry instance via a number of web service calls.  There are two ways this can be achieved. The simplest way is to use the Fusion Registry data portal feature (in Beta).  Another approach is to use the Fusion Registry load structures/data pages and insert the appropriate REST URL.  The benefit of the data portal is that the Fusion Registry does all of the work for you, it handles errors and merging of structures, it will even split the data query into multiple parts if the end service does not support large queries.  This how to guide discusses both approaches.

@@ Line 3: / Line 3: @@
 If an organisation is hosting a SDMX REST API, then it is possible to pull structural metadata and data from this service into your local Fusion Registry instance via a number of web service calls.  There are two ways this can be achieved. The simplest way is to use the Fusion Registry data portal feature (in Beta).  Another approach is to use the Fusion Registry load structures/data pages and insert the appropriate REST URL.  The benefit of the data portal is that the Fusion Registry does all of the work for you, it handles errors and merging of structures, it will even split the data query into multiple parts if the end service does not support large queries.  This how to guide discusses both approaches.
 For the purpose of this 'how to' the web service of the European Central Bank will be used, the API can be found by Googling 'ECB SDMX REST API' which at the time of writing links to this [https://ec.europa.eu/eurostat/web/sdmx-web-services/rest-sdmx-2.1 page]:
@@ Line 17: / Line 18: @@
   [http://ec.europa.eu/eurostat/SDMX/diss-web/rest/codelist http://ec.europa.eu/eurostat/SDMX/diss-web/rest/'''codelist'''] = query for all codelists
-== Use Case
+== Pull Data using Data Portal Feature ==