ETL

There’s a New Sheriff in Town:Oracle OR…

Not Just another Pretty Face

Presented by: Bonnie O’Neil

Introduction

ETL (Extract, Transformation and Loading) is having a paradigm shift. Previously, ETL was done with tools outside the database. Ab Initio, Data Stage and Informatica were kings of the ETL world. However, there were major disadvantages to this paradigm. Third party ETL tools required a separate box because they had no controls on the consumption of CPU resources.

Third party ETL tools were not integrated into the database. This meant that ETL developers became experts in running the tool, but often they had no knowledge on how the tool related to the database. These tools often did not keep up with the new releases of Oracle and would not take advantage of Oracle’s new features. In order to create fast and efficient ETL processes, you needed to know both.

Introduction

Oracle 9i is changing the ETL paradigm.

Oracle 9i supports such important features as:

• External tables

• MERGE statements (INSERT/UPDATE)

• Conditional multi-table inserts

• Pipelined Table functions

• Native Compilation of PL/SQL programs

Introduction

With External Tables, you can now read flat files in parallel into a virtual table. This virtual table can be accessed just as any other table.

FlatFiles

VirtualTable

With the MERGE statement, you can replace a complex “update the row if it exists otherwise insert it” logic with one easy to understand statement. With conditional multi-table inserts, I can read a row from an external table and insert it into many tables based upon conditions. I can replace most ETL processes with a relatively simple INSERT statement.

Pipelined table functions allow Oracle to pass a set of rows to a function and send the results out after each row is finished. With Native Compilation of PL/SQL programs, I can now take a slow PL/SQL program and compile it into C code and then link it to the database.

External Tables

The first component of the new Oracle ETL package is External Tables. External Tables are flat files that Oracle can read and treat as regular tables.

• No indexes can be on External tables.

• External Tables are read only ( No DML allowed )

In the past, you had to load the data to temporary tables utilizing SQLLOADER, do some transformations, and then load the data into tables. Now, you can just read the data straight from flat files. This eliminates much overhead with rollback segments, writing out data to the temporary table and Redo Log activity.

If you have enough memory, you can cache the table in memory. Additionally, you can specify a degree of parallelism for accessing the external flat files.

External Tables

Steps to using external tables.

• Create a directory for accessing the flat files. This is required by the access driver.

CREATE DIRECTORY extdir AS ‘/u01/app/oracle/extfiles’;

• Grant access to the directory

GRANT read,write ON DIRECTORY extdir TO scott;

This Directory object is used in the External Table clause:

• Using the LOCATION clause for input files

• To specify the location for output files.

External Tables

External tables are based upon a SQLLOADER format. In the future, a format of EXPORT / IMPORT will also be supported.

KEYWORD:

CREATE TABLE ….

ORGANIZATION EXTERNAL

TYPE:

type oracle_loader

create table tag_ext( log_type varchar2(60), user_id varchar2(128), trans_id varchar2(64), trans_name varchar2(128), dttm timestamp(4))organization external ( type oracle_loader default directory extdir access parameters …

DEFAULT DIRECTORY specifies the directory to use for input/output files if no location is specified.

External Tables

Access Parameters - Contains two sections

• Record Format

Contains information about the record such as the format of the records, names of output files, and what rules are used to exclude data from being loaded.

…access parameters ( records delimited by newline load when log_type=‘T1’ badfile extdir:’t1.bad’ nodiscardfile logfile extdir:’t1.log’ skip 1 fields terminated by '|' …

Use SKIP statement if your flat file contains a header row.

Output files can contain %p (process number) or %a (agent number) as part of the filename to create unique output files.

External Tables

• Field Format

Describes what characters are used to separate fields, what character is optionally used to enclose fields, and the data format of the fields in the datafile.

…fields terminated by '|' missing field values are null ( log_type char(60), user_id char(128), trans_id char(64), trans_name char(128), trx_class char(10), dttm date(24) mask “dd-mm-yyyy hh24:mi:ssxff“ ))

External Tables

Use the LOCATION clause to specify the filenames of the input files.

REJECT LIMIT specifies the maximum number of rejected records that are allowed. This number applies to each parallel slave used to query the data.

PARALLEL specifies the number of access drivers that are started to process the datafiles.

… location(‘t1.dat’,‘t2.dat’)) reject limit unlimited parallel 2/

External Tables

Data Dictionary Views:

• DBA_EXTERNAL_TABLES

• DBA_EXTERNAL_LOCATIONS

Some DDL statements are allowed on external tables:

REJECT LIMIT, PARALLEL, DEFAULT LOCATION, ACCESS PARAMETERS, LOCATION, ADD/MODIFY/DROP COLUMN, RENAME TO

Performance Issues:

Fixed width is faster than delimited fields

Not writing log, bad and discard files

No conditions clauses

Merge Statement

Before Oracle 9i, a common load, which entailed updating a row if it existed and if it does not insert the row, required two statements or writing PL/SQL code. This data load now can be done in a single statement.

This statement is sometimes called an UPSERT – a cross between an update and an insert statement.

• The merge fires any Insert or Update Triggers.

• Cannot update the same row of the target table multiple times in the same MERGE statement.

ORA-30926: unable to get a stable set of rows in the source tables

Merge Statement

MERGE INTO tag t

USING tag_ext x

ON (t.user_id=x.user_id AND t.trans_id=x.trans_id)

WHEN NOT MATCHED THEN

INSERT(log_type,user_id,trans_id,trans_name,dttm)

VALUES(x.log_type,x.user_id,x.trans_id,x.trans_name,x.dttm)

WHEN MATCHED THEN

UPDATE SET

log_type=x.log_type,

trans_name=x.trans_name,

dttm=x.dttm

Update cannot update columnsUsed in the ON condition

Source can be a table orthe results of a query

Merge Statement

Common Errors:

• Running merge statement and get

ORA-00904: <column_name>: invalid identifier

CAUSE: The reason for this is because you specified the column name in ON clause and the UPDATE clause. Columns used in the ON clause for the join cannot be updated.

Multi-table Inserts

In Oracle 9i, a single insert statement can place data values into multiple tables, both unconditionally and conditionally. This is more efficient than having to parse and execute several insert statements. The format is an extension of the INSERT … SELECT statement.

• Unconditional Multi-table Insert

The ALL keyword is required.

INSERT ALL INTO emp VALUES(empno,ename,title,salary) INTO commision VALUES(empno,comm) SELECT empno,ename,title,salary,salary*.10 comm FROM employees_external;

Multi-Table Inserts

In addition, Oracle 9i allows a conditional clause to be included is a multi table insert.

A conditional insert will insert into a table if the WHEN condition is true. You can insert based on the FIRST WHEN clause that evaluates to true or ALL WHEN clauses that evaluate to true.

INSERT FIRSTWHEN (title=‘Oracle DBA’) THEN INTO high_paid_employees VALUES(empno,ename,title,salary)WHEN (title=‘SQL Server DBA’) THEN INTO low_paid_employees VALUES(empno,ename,title,salary)SELECT * FROM employees_external;

FIRST keyword means that each row will be evaluated until the row is evaluated as true with a WHEN clause. After that the row is not evaluated against the other WHEN conditions.

Multi-Table Inserts

Use the ALL keyword to specify that the INSERT should occur for all WHEN clauses that evaluate to true.

INSERT ALLWHEN (title=‘Oracle DBA’) THEN INTO bonus_due VALUES(empno,ename,title,salary)WHEN (salary > 100000) THEN INTO high_paid_employees VALUES(empno,ename,title,salary)ELSE INTO low_paid_employees VALUES(empno,ename,title,salary)SELECT * FROM employees_external;

Table Functions - Pipelined

Table functions produce sets of rows as output. Pipelined table functions return the data iteratively, instead of in a batch, thus eliminating the need for intermediate staging. Table functions use the TABLE keyword.

• Table functions return not a single row but a set or collection of rows.

• The result set can be a nested table or varray.

• Table functions can be queried like any table in the FROM clause of a query.

• Table functions can accept a collection type as input or a REF cursor.

• Table functions can be parallelized.

• Table functions can return all the rows at once or PIPELINE the results as they are produced. (one row at a time).

Table Functions - Pipelined

KEYWORD: PIPELINED

PIPELINED functions use less memory because the object cache doesn’t have to materialize the entire result set.

PIPELINED functions can accept a REF Cursor as an input parameter.

CREATE FUNCTION managerlist( cur cursor_emp_pkg.emp_cur )RETURN emp_type_tablePIPELINED IS …

The keyword pipe row returns the row immediately rather than waiting for all rows to be processed. Table functions can perform the complex transformations in an efficient manner.

Table Functions - Pipelined

-- Create a Type to define the result type collectionCREATE OR REPLACE TYPE emp_type AS OBJECT(empno number(4),ename varchar2(10),job varchar2(9),Sal number(7,2))/-- Create a collection used as the return type Create type emp_type_table as table of emp_type/-- Create a REF CURSOR as a package variableCREATE OR REPLACE PACKAGE cursor_emp_pkg AS type emp_rec is record ( empno number(4), ename varchar2(10), job varchar2(9), sal number(7,2)); type strong_emp_cur is ref cursor return emp_rec; End;/

Table Functions - Pipelined-- Create FunctionCREATE FUNCTION DBA_LIST(cur cursor_emp_pkg.strong_emp_cur) RETURN emp_type_table PIPELINED IS out_rec emp_type := emp_type(NULL,NULL,NULL,NULL); in_rec cur%ROWTYPE;BEGIN LOOP FETCH cur INTO in_rec; EXIT WHEN cur%NOTFOUND; IF in_rec.job = 'DBA' THEN

out_rec.empno := in_rec.empno;out_rec.ename := in_rec.ename;out_rec.job := in_rec.job;out_rec.sal := in_rec.sal;PIPE ROW(out_rec);

out_rec.sal := in_rec.sal *.10; PIPE ROW(out_rec); END IF; END LOOP; CLOSE cur; RETURN;END;/

Table Functions - Pipelined

SELECT empno,ename,job,sal from emp where job=‘DBA’;

EMPNO ENAME JOB SAL---------- ---------- --------- ---------- 7788 SCOTT DBA 3000 7902 FORD DBA 3000

SELECT *FROM TABLE(DBA_LIST(CURSOR(SELECT empno,ename,job,sal FROM emp)));

EMPNO ENAME JOB SAL---------- ---------- --------- ---------- 7788 SCOTT DBA 3000 7788 SCOTT DBA 300 7902 FORD DBA 3000 7902 FORD DBA 300

Table Functions - Pipelined

OBSERVATIONS:

Explain plan output for Pipelined Table Functions

Rows Execution Plan------- --------------------------------------------------- 0 SELECT STATEMENT GOAL: CHOOSE 4 VIEW 4 COLLECTION ITERATOR (PICKLER FETCH) OF 'DBA_LIST‘

Although the number of records returned is twice as many, theSQL*Net roundtrips to/from client remained the same.

Many more recursive calls for the pipelined function.

Native Compilation

Before Oracle 9i, PL/SQL programs were compiled to P-code and interpreted at runtime.

Interpreted languages are much slower than compiled languages. This interpreted feature allows PL/SQL to be portable.

Oracle 9i offers an option to turn the PL/SQL code into C code automatically. You write the same PL/SQL code and Oracle will covert it, compile it and execute it when the PL/SQL code is called.

SQL statements do not run much faster. NO CODE CHANGES IN THE PL/SQL CODE!

Native Compilation

To utilize Native Compilation, you must set several INIT.ORA parameters.

plsql_compiler_flags=‘NATIVE’• Default is INTERPRETED (can be changed at session level)

plsql_native_make_utility=‘/usr/ccs/bin/make’plsql_native_make_file_name= ‘/u01/app/oracle/product/9.2.0/plsql/spnc_makefile.mk’

plsql_native_library_dir=‘/u01/app/oracle/lib’• Location where shared libraries are created

plsql_native_c_compiler=‘/opt/SUNWspro/bin/cc’plsql_native_linker=‘/usr/sbin/link’

Native Compilation

CREATE FUNCTION CALC_BONUS(…….

Produces a C file called CALC_BONUS__SCOTT__1.c and the shared library file CALC_BONUS__SCOTT__1.so in the directory defined by the parameter plsql_native_library_dir.

You can check to see how code was compiled by viewing the PARAM_NAME and PARMA_VALUE from the dictionary view DBA_STORED_SETTINGS

SELECT param_name,param_value FROM user_stored_settingsWHERE object_name=‘CALC_BONUS’;

PARAM_NAME PARAM_VALUE------------------------------ --------------------plsql_compiler_flags NATIVE,NON_DEBUG

Native Compilation

Check metalink for latest patches and documents regarding NATIVE compilation.

There are known bugs (unexpected features) with compiling on a client versus the server.

• Error compiling from client.PLS-00923: Native compilation falied: make:spdtexmk:?

Also, issues with 32 bit versus 64 bit O/S’s. On Sun 2.8, I ran into a problem that had to be fixed by setting an environment variable and bouncing the listener.

Conclusion

The ETL paradigm shift is under way. We will start seeing more ETL be done inside the database as opposed to outside the database. With the ETL processing done at the database level, Oracle 9i can take advantage of resource allocations using resource groups. These changes will eventually result in a simpler and more efficient ETL process at a lower cost of ownership.

Oracle 9i offers vast improvements in ETL functionality.• Economics• Efficiency• Performance

Thank You

Suzanne RiddellPresidentApex Solutions, Inc.303 216 9491...office303 809 4914...cellsuzanne@apexsolutions.comwww.apexsolutions.com

"The Business Intelligence Source"