ETLServerInstall

From Metadata Technology Wiki
Jump to navigation Jump to search

Overview

The Fusion ETL server is supplied as a WAR file. As such it should be deployed to a Java web application sever such as Apache Tomcat.

The Fusion ETL server has a seperate web user interface, which can be deployed to the same application server.

The Fusion ETL server is designed to run as a single instance and does not support load balancing. The Fusion ETL server is designed to pull data from a local database, supporting Oracle, SQL server, MySQL and Maria DB. Fusion ETL publishes the extracted data to a Fusion Registry instance via the exposed publication web services of the Fusion Registry based on pre-defined schedules and views. Fusion ETL stores all configuration, auditing, and logging information in the file system and as such the only database requirement is a readonly connection to a database to extract the data for publication.

Quick Start with Apache Tomcat

Install Tomcat and Java

Install Apache Tomcat, and Java

Warning.PNG Some releases of the Oracle Java Runtime Environment (JRE) prior to 1.8.0_151 require a change to their default security policy configuration to support the default 256-bit AES encryption used by Fusion Registry for sensitive data.

If the ETL Server reports an 'Encryption Failed - Illegal key size or default parameters' error, then alter the JRE security policy to allow encryption keys of unlimited length. Java encryption key - setting unlimited length

Deploy ETL Server and Launch Web Server

The Fusion ETL Server consists of a single .war file called FusionETL.war.

  1. Copy FusionETL.war into the directory: <TOMCAT_HOME>/webapps
  2. Start the Tomcat Server

During the Tomcat start-up process, the war file will be unpacked into a directory of the same name, for example:

<TOMCAT_HOME>/webapps/FusionETL

On successful launch the FusionETL will have created the FusionETL home directory on your file system.

The Fusion ETL Server product information service can be queried from the following URL:

http://[server]:[port]/FusionETL/ws/fusion/info/product

The values for server and port must be replaced with the IP address and port number that the web application server is running on. For example, if the web browser is running on the same machine as the web application server and the Apache Tomcat has not had its default port settings modified, then the following address can be used:

http://localhost:8080/FusionETL

The Fusion Edge Server is now deployed and running in a web container, the URL will result in an empty (white page). The User Interface is deployed separately.

Deploy ETL User Interface

The Fusion ETL User Interface is provided as a zip file which contains a combination of HTML, JavaScript, CSS and a configuration file.

In <TOMCAT_HOME>/webapps create a new folder of any name, for example ETL and unzip the User Interface files into this folder. Open the file <TOMCAT_HOME>/webapps/ETL/properties/siteconfig.json, edit the ServicesAPI to match the product info service of your FusionETL server.

http(s)://[server]:[port]/FusionETL/ws/fusion/info/product

The User Interface will be accessible in a web browser from the URL:

http(s)://[server]:[port]/ETL

Where ETL is the name of the folder created under <TOMCAT_HOME>/webapps and 8080 is the port Tomcat is using (which defaults to 8080).

The default login is:
username: root
password: password

For more advanced security, set up an connection to Active Directory or OpenLDAP

Clicking on the Audit Logs navigation item, it should show that the Application Start event of the ETL Server resulted in no errors.

Appstart.png

Post Install Configuration

Connect to Database

A Database is required by the ETL server as the source of datasets to extract, before loading to the Fusion Registry server. The Database is not used to store any information from the ETL server.

The database is configured through the User Interface.

Note: Due to licensing restrictions the ETL Server does not provide a JDBC driver for Oracale. In order to connect to these database platforms the driver has to be downloaded separately. Search for OracleServer JDBC driver, download the jar file and add to the web application. The jar file can be placed under [tomcat]/webapps/FusionETLServer/WEB-INF/lib

Connect to Fusion Registry

The Connection to the Fusion Registry is established when a Fusion Registry generated node license file is placed in the registries folder in the ETL Server Directory. The ETL server does not scan the directories for changes, so after updating the contents of this directory with the license file, either restart the Tomcat server, or click the Server Status menu to reload the service.

Configure Publications

Once the ETL server is connected to a local database and a Fusion Registry instance, it is time to set up a publication process. This is achieved in the following steps:

  1. Set up a Dataflow to Database Mapping for each Dataflow to report data for
  2. Set up one or more Schedules to automate the ETL process
  3. Optionally define one or more Views to pull subcubes of data from the database

The ETL Server is then able to run each Schedule at the pre-defined time, to extract the data from the linked database as defined by the Dataflow Mapping. The Extract process may use one or more optional filters on one or more Dimenisons based on any linked Views. The Schedule determines if the Extract process is pulling a delta (only the data that has been modified since the last successful publication) or a full release (no time filters applied).

The high level information model is shown in the image below.

ETLProcess.png