the power of etl with datamigrataor - taylor...

The Power of ETL with DataMigrator of 44 6/13/2016

Copyright © 2016 Information Builders

The Power of ETL with DataMigrator

Clif Kranish

Information Builders

June 2016

Author: Clif Kranish

Company: Information Builders

Presentation Title: The Power of ETL with DataMigrator

Presentation Abstract: DataMigrator is Information Builders ETL (Extract-Transform-Load) tool. This point-and-click data integration session is for those interested in building data warehouses, data marts, operational data stores, and for those who want to extract data for application loading and business intelligence. You'll see how easy it is to build complex data flows (to extract, transform, and load data targets) and process flows (to control the data flow) with the DataMigrator graphical user interface.

Business Need: Since WebFOCUS can directly report from virtually any data source, why would anyone want to copy data from an operational system to a Data Warehouse for analysis? The answer is that a Data Warehouse is not just a copy of the data in operational systems. It‘s much more than that: it’s been integrated, cleansed and organized into subject areas. While operational systems are designed for transaction processing, a data warehouse is architected specifically to facilitate analysis.

DataMigrator simplifies the process of loading a Data Warehouse, applying business rules as the data is loaded.



Step 0 - Setup ............................................................................................................................ 3

Connect to MS SQL Server and create a database ................................................................ 3

Connect to the DataMigrator server ........................................................................................ 4

Create an application directory ............................................................................................... 4

Configure an adapter to connect to MS SQL Server ............................................................... 4

Create sample data for the lab ............................................................................................... 5

Direct report output to log ....................................................................................................... 6

Disconnect ............................................................................................................................. 7

Step 1 - Create a synonym for a data target ............................................................................... 8

Step 2 - Review the data sources..............................................................................................12

Step 3 – Create a data flow .......................................................................................................15

Add source tables ..................................................................................................................15

Source Transformations ........................................................................................................19

Select columns and aggregation............................................................................................22

Add a filter to select rows.......................................................................................................26

Select the target table and map from source to target ...........................................................28

Save and run the data flow ....................................................................................................32

View the target table ..............................................................................................................35

Step 4 – Control processing with a process flow .......................................................................36

Create a stored procedure .....................................................................................................36

Create a process flow ............................................................................................................36

Step 5 – Reports .......................................................................................................................42

Impact Analysis .....................................................................................................................42

Data Lineage .........................................................................................................................42

Flow Reports .........................................................................................................................43



Step 0 - Setup Note: If you are at a live presentation of this lab at Summit, then the setup procedures have been done for you. Skip to Step 1.

These setup instructions are provided to create the database, tables, and metadata to do this lab on your own PC with the following software installed:

DataMigrator Release 7.7.07

Microsoft SQL Server 2014 or later. Any version including the free Express edition.

Connect to MS SQL Server and create a database

You can skip this step if you know the name of your MS SQL Server and have a database that you can create tables in.

1. Click Start ► All Apps ► Microsoft SQL Server 2014 ► SQL Server Management Studio

2. In the Connect to Server dialog box note the name of your server.

Click on Connect.

3. Right-click Databases and select New Database.

4. For Database Name, enter a name for your MS SQL Server database, such as main.

5. Click OK.

6. Close MS SQL Server Management Studio.



Connect to the DataMigrator server

7. Click Start ► All Apps ► iWay Software ► iWay 77 DataMigrator Server ► Start Security ON.

8. If a Windows Security Alert dialog box appears for edapgwy, click Unblock.

9. Click Start ► All Apps ► iWay Software ► iWay 77 DataMigrator Server ► Data Management Console.

10. The DMC shows a list of all servers available. Right-click LOOPBACK and select Connect.

11. If the DMC Sign in: LOOPACK dialog appears enter your User ID and Password, check the checkbox labeled Save Credentials and click Sign In.

Create an application directory

An application directory is a location on the server where we can store metadata and DataMigrator flows. For this lab, we’ll create an application directory called dmlab.

12. Right-click Application Directories and select New ► Application Directory.

13. In the Create New Application dialog box for Application Name, enter dmlab. Add directory to APPATH should be checked. Change Position in APPPATH to First. Click OK.

Configure an adapter to connect to MS SQL Server

The DataMigrator server must be configured to access an MS SQL Server database. These instructions are for a server with Windows security.

14. Expand Adapters ►Available ► SQL ► MS SQL Server 2012:2014 then right-click and select Configure.

15. In the LOOPBACK: Add MS SQL Server…dialog box, enter the following parameters: Server: The name of your server or localhost Security: Trusted Default Database: The name of a database, such as main, where you can create tables.



16. Click Configure and Click OK.

17. In the LOOPBACK: Configuring Adapters or Create Synonym dialog box, click Close.

Create sample data for the lab

This lab uses sample tables that you must create.

18. Under Application Directories right click on dmlab and select New ► Tutorials.

19. The Create Tutorial Framework dialog box appears. From the Tutorial drop-down list, select DataMigrator – General. Click Create.



A confirmation dialog box opens. Click OK.

20. A processing dialog box appears. When it closes, a series of messages appear in the console area at the bottom of the screen. The last message should read: (ICM18666) All sample data successfully created.

Display Column Names

Ensure that actual column names are displayed in the Data Management Console.

21. In the Browser window expand the ibisamp folder. Scroll down to the synonym dmcntr and double click to open it.

22. Right click on the header bar labeled Name and select Column Name Display Strategy ►Name.

23. Click the x to close the tab.

Direct report output to log

24. Redirect output from a procedure to the DataMigrator log. Expand Workspace and Special Services and Listeners. Right-click Scheduler and select Properties.



25. The Scheduler Configuration window opens. Select the check box for Merge formatted output into log and click Save and Restart scheduler.

26. Click x to close the Loopback: Special Services and Listeners page.

Disconnect

27. Right-click LOOPBACK and select Disconnect.



Step 1 - Create a synonym for a data target A DataMigrator server, like WebFOCUS, uses a synonym to describe a data source or target. In this lab, the sources and targets are MS SQL Server tables.

In this section, we will use the Data Management Console to connect to a server and create a synonym for an MS SQL Server table that will be target for the DataMigrator flow that we will create.

1. Start the Data Management Console by clicking on the desktop icon or, if it is not present, click Start ► All Apps ► iWay Software ► iWay 77 DataMigrator Server ► Data Management Console.

2. When you start the Data Management Console, it automatically starts the DataMigrator server, but it takes a moment. When the message Local server has been started appears, you can continue.

3. The DMC shows a list of all servers available. Right-click LOOPBACK and select Connect. Note that if you see a choice of Disconnect, then you are already connected.

4. Right-click on the folder dmlab, select New ► Synonym (Create or Update).

5. Under the Configured and MS SQL Server … folder, select CON01 and click OK.



6. In the LOOPBACK: Select Synonym candidates dialog box, the value for Select database should remain Default database. Click Next.



7. In the LOOPBACK: Create Synonym… Step 2 of 2 dialog box, for Select application the directory dmlab is pre-selected. Scroll down to find the table name dmrpts, and select the check box on that line. Then click Next.

8. In the Create Synonym… Status dialog box, the status should be Created successfully. Click Close.



9. In the browser, double-click the folder dmlab to expand it, and then double-click dmrpts to open the synonym.

Note that the table has three keys, PROD_NUM, STORE_CODE and YRMTH. When we load this table, we need to ensure that we have one row for each unique combination of these three columns.

10. Close the window by clicking the x in the tab.



Step 2 - Review the data sources DataMigrator incorporates data profiling capabilities to help you understand your data.

1. The browser window on the left side of the screen shows the Application Directories on the server as folders.

Double-click the ibisamp folder to expand it (or click the plus icon ). This folder contains different types of files stored on the server. Those used by DataMigrator include:

- Stored Procedures

- Flows

- Synonyms



2. Scroll down to the synonyms and select dmord. Right-click and select Data Profiling Statistics.

Here we can see that there are 104 orders in the table and 600 line items. We can also see the order dates range from 2004 to 2006.

Note: You can drill down on any of the column names to see the values for that column.



Close the data profiling window from the Home tab on the ribbon by selecting

Window ► Close. You can also close a window by clicking .

3. We can also sample the source table. Right-click on the table name dmord and select Sample Data.

Close the sample data window.



Step 3 – Create a data flow In this section, we will create a data flow that extracts data from two source tables, aggregates the data, and loads the dmrpts table.

1. Create a new flow. Scroll up and right-click on the dmlab folder, select New ► Flow.

Add source tables

2. On the ribbon, on the Home tab, click the Synonyms button to filter the list of objects so that only synonyms are shown.



3. Scroll down to the ibisamp folder and find the synonym dminv. Click and drag it onto the workspace to the left of the SQL object.

4. Add a second source table. Note: This is an alternate way to add a source table that lets you view the synonyms in one application directory at a time and to sort the synonym names by name, type, or description. Right-click in the workspace to the left of the SQL object and select Add source.



5. Select the second table. In the Look in pane select ibisamp. Scroll down to the dmord table and select it, then click Select.

6. The dmord table is added to the workspace. Note that a Join object is automatically added.



7. Double-click on the Join object to open the Join Editor.

The two selected tables are automatically joined on PROD_NUM because columns with the same name exist in both tables, and it’s a key in one of the tables. However you could remove the join condition or add a new one. Note that by clicking on the diagram under Join Type, you can choose the join type by clicking the left or right part of the diagram. Leave Inner Join selected for this lab. Click OK to close the Join Editor.



Source Transformations

One of the keys of the target table dmrpts is the YRMTH column that contains the Year and Month. We want to aggregate (GROUP BY) the combination of year and month, so we need to create a virtual (DEFINE) column with that value. As you will see, the orders table dmord contains the order date from which we can extract the year and month and create a new virtual column with year and month.

8. Right-click on dmord and select Source Transformations.

9. The Source Transforms dialog box opens. Note that the table contains several virtual columns, those with an f over the column symbol. These columns exist only in the metadata (synonym) and their values are generated as the rows are read, if they are used in a data flow. This synonym has virtual columns that extract the year, quarter, and month from the order date. We can use these columns to add an additional transformation just for use in this data flow that contains the combined year and month.



10. Click the Insert Transformations button.

11. The transformation calculator opens. Enter: Name: YM Format: I6 (the letter I, not the digit 1) Expression: Enter 100 * Double-click on ORDYEAR Enter + Double-click on ORDMONTH

Then click OK to close the Transformation Calculator.



12. The transformation is added to the Source Transformations. Click Test Transforms to see the calculated value.

13. The Test Transformations window opens.

Close the Test Transformations window, then click OK to close Source Transformations.



Select columns and aggregation

Now that we have identified the source tables, we’re going to select the columns that we want to extract from those tables, and identify the columns on which we will aggregate the data.

We will also create a filter to restrict the rows that are extracted.

14. Right-click the SQL object and select Column Selection... Under dminv click PROD_NUM, then under dmord Ctrl+click STORE_CODE and scroll down to Ctrl+Click YM.

Click the right arrow button to copy the columns to Selected Columns.

15. The three columns (Product number, store code, year and month) are the key columns of the target table, so we want to do a GROUP BY on them so that we get one row for each unique combination of them. With the three columns still all highlighted (blue), click the Aggregate drop-



down list and select Group By.

16. Double-click the following columns to select them.

QUANTITY DMORD (T2)

LINEPRICE DMORD (T2)

17. The last column is a calculated column. Click the Insert Columns button.



18. The Line Cost of Goods is the Cost times the Quantity. Since these columns are in different tables, we couldn’t use an Extract Transform, so we will calculate it as part of the SQL Select statement. Enter the following values into the SQL Calculator: Alias LINECOGS Expression: Double-click COST under dminv (T1) Click * Expand DMORD under dmord (T2) Expand DMPROD Double-click on QUANTITY

Press OK to close the window.

19. Note that three columns are red. That’s a warning that if we do a group by on any column, we should do a group by or aggregate on all of them. These columns are all numeric and we will sum them. LINECOGS should be selected, Ctrl+Click on QUANTITY and LINEPRICE to select them too. Select Sum from the drop-down list in the Aggregate column.



20. Test the SQL. Click the SQL button .

You can see the calculation performed for Line Cost of Goods (LINECOGS). Close the Test SQL window, click OK to close the Column Selection window.



Add a filter to select rows

21. For this lab, we only want to include data from 2005 and later.

Right-click on the SQL object and select Expand DMORD under dmord (T2) Double-click on ORDYEAR to add to Relational Expression On the Relation drop-down list select >= For Value type 2005

Click OK to close.



22. View and test the SQL. Now that we have completed the SQL generation, we can take a look at the SELECT statement.

Right-click on the SQL object and select .



23. Click the SQL button. This tests the query and displays a sample answer set.

Close the Test SQL window, click OK to close the Select Statement window.

Select the target table and map from source to target

Now that we’ve defined how the data is extracted from the source, we will specify how it’s loaded into the target. We will select the target table and create transformations to map from the selected columns to the target table.

24. Expand the dmlab folder and drag the table dmrpts to the right of the SQL object in the data flow.



25. Right-click the dmrpts object and select Properties. For the Prior to load Option, select Truncate table. For If the record exists, select Update the existing record. Leave the other options as is.

Click to close the Properties page.

26. Right-click on the dmrpts object and select Target Transformations.

At the top of the window, near Target Columns - dmlab/dmrpts, click on the Automap button

to map the identically named columns from source to target.



Notice that the mapped columns under target columns have a green checkmark to indicate that they are mapped.

27. Columns with different names can be mapped manually. On the left side, click YRMTH. On the right side, click YM.

Click the equal sign .

28. Now all the columns are mapped except for PROFIT. It requires the LINECOGS column that was calculated in the SQL SELECT statement. This is the last place where we can do transformations. Double-click on PROFIT to add it to the list of transformed columns. The calculator opens. Double-click on LINEPRICE. Click the minus sign. Double-click LINECOGS.



Click OK to close the Transformations Calculator.

29. Review the transformations.



30. Click the Test Transforms button to test the transformations.

Close the Test Transformations window, click OK to close the Transformations window.

Save and run the data flow

31. On the Ribbon on the Flow tab, click down arrow on the Run Flow button and select Submit.



DataMigrator will ask Do you want to save changes… Click OK. In the Save Procedure As dialog box, for Save in leave the dmlab folder selected, and for Name enter lab1.

Click Save. When the flow completes, you will see a message in the console log.



32. View the log. On the ribbon, click . Then scroll down to view the log. You can see the number of records processed and the number or rows written to the target table.



Close the report window.

View the target table

33. Expand the dmlab folder, select dmrpts, then right-click and select Sample data.

To close all open windows, on the Home tab select Window and Close All.



Step 4 – Control processing with a process flow A DataMigrator process flow lets us control the processing of data flows, for example to schedule a flow, to run multiple flows in sequence, or to send an email message on completion or failure of a flow. It also lets us customize processing by adding a stored procedure.

Create a stored procedure

In this step, we will create a stored procedure to summarize the data in the target table and to add the totals to the log.

1. On the Ribbon on the Home tab click the button so that flows and synonyms are displayed.

2. Right-click on the dmlab folder and select New ► Procedure.

3. In the text window that opens, type the following: SQL

SELECT SUM(LINECOGS), SUM(LINEPRICE), SUM(PROFIT)

FROM DMRPTS;

END

On the ribbon, click the Run button . A report is displayed with the totals.

Close the report to return to the stored procedure.

4. On the Quick Access Toolbar, click the Save button . Leave or select the Save in directory as dmlab. For Name, enter labrpt and click Save.

Create a process flow

5. Right-click on the dmlab folder and select New ► Flow.



6. Click the Process Flow tab to switch to the process flow view.

7. From the dmlab folder, click lab1 and. without releasing the mouse button, drag the object to the right of the Start icon, then release the mouse button. Then do the same for labrpt to the right of lab1.

8. Right-click Start and, without releasing the mouse button, drag to dmlab/lab1 and release the mouse button. Then right-click on dmlab/lab1 and drag to dmlab/labrpt and release the mouse button.



The green arrow from dmlab/lab1 to labrpt indicates that if the data flow completes successfully, the stored procedure is run. You can view the condition or change it by double-clicking on the arrow.

Click OK to close the dialog box.



9. Next we’re going to add an email object that sends an email if data flow does not complete successfully.

On the ribbon, click the Email button and drag it into the process flow under labrpt.

10. Right-click on lab1 and drag it to the email object and release the mouse button.

The red arrow indicates the default failure condition. If the data flow fails, then the email action is done.

11. To set the email properties, right-click on the email object and select Properties.



Here you can enter the email address to receive the message (or a procedure that returns a list), the subject, and the message, or alternately a file to be sent. Note: To actually send an email message, the server must be configured to connect to a mail server that is not available for this lab. Close the property page by clicking on the small x on its upper right corner.

12. We could also schedule the flow to run at a specific time or interval. Right-click on the Start object and select Properties.

13. From the Schedule Status drop-down list, select Active. From the Schedule Type drop-down list, select Recurring to see the recurring scheduler options.

14. Since we’re going to run this flow directly from the DMC, we can delete the schedule. On the Schedule Status drop-down list, select No Schedule.



Close the property page by clicking on the small x in its upper right corner.

15. Now submit the process flow. On the Flow tab, click the Run Flow button and on

the drop-down list select . The DMC asks Do you want to save changes… and you should click OK. In the Save Procedure As dialog box, leave Save in as dmab, and for name enter labpf. Then click Save.

16. Wait for the message to appear in the console log that the request is complete.

17. On the ribbon, click the View Last Log button .

The totals for the three columns appear in the log. Note the blue line with Job ID. It’s a hotlink to the detail log for the data flow. To review the detail log, click the link.

18. On the Home tab, select Windows ► Close all.



Step 5 – Reports DataMigrator has additional reporting capabilities to assist the ETL developer in managing their data flows and the development process. In this section we will look at a few of them.

Impact Analysis

For a selected table, the Impact Analysis report shows you where the table is used, and every flow and stored procedure.

1. In the dmlab folder, right-click on dmrpts and select Impact Analysis.

Data Lineage

The Data Lineage report shows how a column in a target table was derived.

2. In the dmlab folder, right-click dmrpts and select Open.



3. The dmrpts Synonym opens. Right-click PROFIT and select Data Lineage.

4. The Data Lineage Report opens.

Here you can see for the lab1 flow for which we created the steps to calculate PROFIT. Line 1 shows that it is calculated as LINEPRICE – LINECOGS. Lines 2 and 3 show where LINEPRICE originates. Line 4 shows how LINECOGS is calculated.

Flow Reports

The DataMigrator Flow Reports are useful for documenting your work. The report for a flow report is a text view of a flow.

5. Right-click on lab1 and select Flow Report.

the power of etl with datamigrataor - taylor...

Documents