Oracle Utility: SQL*LoaderOracle Utility: SQL*Loader

SQL*Loader

Input Data Files

Loader Control File

Data Base

Log FileDiscard

Files

BadFiles

Tables

Indexes

Record

SQL*LoaderField Processing

Bad

SQL*LoaderWhen Clause

Discard

SQL*LoaderDML Inserting

Data Base

KO

OK

sqlldr...Valid Keywords:

Userid -- ORACLE username/passwordcontrol -- Control file namelog -- Log file namebad -- Bad file namedata -- Data file namediscard -- Discard file namediscardmax -- Number of discards to allow (Default all)skip -- Number of logical records to skip (Default 0)load -- Number of logical records to load (Default all)errors -- Number of errors to allow (Default 50)rows -- Number of rows in conventional path bind array or between direct path data saves (Default: Conventional path 64, Direct path all)silent -- Suppress messages during run (header,feedback,errors,discards,partitions)direct -- use direct path (Default FALSE)parfile -- parameter file: name of file that contains parameter specifications

-- Loads EMP records from first 23 characters-- Creates and loads PROJ records for each PROJNO listed-- for each employee

LOAD DATAINFILE ’ulcase5.dat’BADFILE ’ulcase5.bad’DISCARDFILE ’ulcase5.dsc’REPLACEINTO TABLE emp (empno POSITION( 1: 4) INTEGER EXTERNAL, ename POSITION( 6:15) CHAR, deptno POSITION(17:18) CHAR, mgr POSITION(20:23) INTEGER EXTERNAL)

INTO TABLE proj-- PROJ has two columns, both not null: EMPNO and PROJNO WHEN projno != ’ ’ (empno POSITION( 1: 4) INTEGER EXTERNAL, projno POSITION(25:27) INTEGER EXTERNAL) -- 1st proj

INTO TABLE proj WHEN projno != ’ ’ (empno POSITION( 1: 4) INTEGER EXTERNAL, projno POSITION(29:31) INTEGER EXTERNAL) -- 2nd proj

INTO TABLE proj WHEN projno != ’ ’ (empno POSITION( 1: 4) INTEGER EXTERNAL, projno POSITION(33:35) INTEGER EXTERNAL) -- 3rd proj

External Tables Concepts

The Oracle9i external tables feature is a complement to existing SQL*Loaderfunctionality. It allows you to access data in external sources as if it were in a table in the database.

External tables are read-only.

No data manipulation language (DML) operations or index creation is allowed on an external table.

Therefore, SQL*Loader may be the better choice in data loading situations that require additional indexing of the staging table.

To use the external tables feature, you must have some knowledge of the file format and record format of the datafiles on your platform.

You must also know enough about SQL to be able to create an external table and execute DML statements that access the external table.

CREATE TABLE empxt (empno NUMBER(4), ename VARCHAR2(10), job VARCHAR2(9), mgr NUMBER(4) ) ORGANIZATION EXTERNAL ( TYPE ORACLE_LOADER DEFAULT DIRECTORY dat_dir ACCESS PARAMETERS ( records delimited by newline badfile dat_dir:'empxt%a_%p.bad' logfile dat_dir:'empxt%a_%p.log' fields terminated by ',' missing field values are null ( empno, ename, job, mgr ) ) LOCATION ('empxt1.dat') )REJECT LIMIT UNLIMITED;

Data WarehousingData Warehousing

DataWarehouse

Discoverer

Development Tool

Forms& Reports

PL/SQL

Pro*C

S.Q.L.

fasi del processo di integrazione...fasi del processo di integrazione...

Integrazione di dati:

DataWarehouse

Integrazione di piattaforme e

applicazioni:

Intranet

Integrazione di dati:

DataWarehouse

OLAPsu Web

Data Warehouse DefinitionData Warehouse Definition

Un DWH raccoglie i dati da diverse sorgenti trasformandoli in un formato Un DWH raccoglie i dati da diverse sorgenti trasformandoli in un formato

consistente ed omogeneo.consistente ed omogeneo.

Ogni Punto nel “Ogni Punto nel “CuboCubo” contiene “Fact Data” ” contiene “Fact Data” per una particolare combinazione di n per una particolare combinazione di n “Dimension Data”. Nel caso a 3 dimensioni “Dimension Data”. Nel caso a 3 dimensioni riportato a lato in particolare troviamo dati riportato a lato in particolare troviamo dati statistici organizzati per Product, Market, Time.statistici organizzati per Product, Market, Time.

data entrydata entry

data entrydata entry

O.L.T.P.O.L.T.P.

interrogazioni e reportsinterrogazioni e reports

Sistema transazionaleSistema transazionale

Data Data WarehouseWarehouse

reports integrati edreports integrati edanalisi on-lineanalisi on-line

Ambiente di reporting eAmbiente di reporting eDecision Support SystemDecision Support System

O.L.A.P.O.L.A.P.

Strumenti per l’analisi on-line dei Strumenti per l’analisi on-line dei dati con caratteristiche di efficienza, dati con caratteristiche di efficienza, facilità d’uso ed elaborazione graficafacilità d’uso ed elaborazione grafica

EExtraction, xtraction, TTransformationransformationLLoading oading

Data Data MartMart

Data Data WarehouseWarehouse

EstrazioneEstrazione

Database Database operazionaleoperazionale

TrasportoTrasporto

Meta DataMeta Data

ReportingReportingtoolstools

TrasformazioneTrasformazione

Sistema transazionaleSistema transazionale Decision Support SystemDecision Support System

Architettura tipica di un DWHArchitettura tipica di un DWH

Analisti

Quanto si prevede divendere quest’annoin base al trend attuale?

Utenti evoluti

Chi ha contribuitoall’aumentodelle vendite?

IT SviluppaUtenti Consultano

Quanto abbiamovendutol’anno scorso?

Cosa èsuccesso?

Perchè è successo?

Cosa accadrebbe se?

TacticalTactical StrategicStrategicOperationalOperationalReportingReportingStandardStandard

Analisi e Analisi e Domande ad hocDomande ad hoc

Analisi Analisi AvanzateAvanzate

OracleReports

OracleDiscoverer

OracleExpress

Differenze di un data warehouse rispetto ad un database Differenze di un data warehouse rispetto ad un database operazionale:operazionale:

• differenti requisiti di hardware, storage e tuningdifferenti requisiti di hardware, storage e tuning• differente modellazione dei datidifferente modellazione dei dati• diversi strumenti di interrogazione e reportingdiversi strumenti di interrogazione e reporting

Data MartData Mart

• Schema composto da una o più tabelle dei Schema composto da una o più tabelle dei FATTIFATTI contenenti l’elemento quantitativo e diverse contenenti l’elemento quantitativo e diverse tabelle delle tabelle delle DIMESIONIDIMESIONI rappresentanti un tipo di rappresentanti un tipo di gerarchia.gerarchia.

• Schema a stella (Schema a stella (star schemastar schema))

• Più stelle collegate prendono il nome di Più stelle collegate prendono il nome di costellazionicostellazioni

Star queriesStar queries

Il processo di join di Oracle per N tabelle è Il processo di join di Oracle per N tabelle è quello di creare dei risultati intermedi quello di creare dei risultati intermedi considerando in join due tabelle per volta considerando in join due tabelle per volta (pair-wise).(pair-wise).

Con l’ottimizzazione delle star query Con l’ottimizzazione delle star query Oracle si riserva di mettere in join la tabella Oracle si riserva di mettere in join la tabella dei fatti per ultimo essendo quella di dei fatti per ultimo essendo quella di maggiori dimensioni.maggiori dimensioni.

Schema a StellaSchema a Stella ( (Star SchemaStar Schema))

Oracle9i

Star Schema ExampleStar Schema Example

Data Warehousing con ORACLEData Warehousing con ORACLE

• Viste materializzate / SnapshotsViste materializzate / Snapshots

• Tabelle partizionateTabelle partizionate

• Tabelle Organizzate ad IndiceTabelle Organizzate ad Indice

• ClusterCluster

• Star queryStar query

• Indici Function BasedIndici Function Based

• Read only tablespacesRead only tablespaces

DataBaseDataBase TablespaceTablespace DataFileDataFile Oracle BlockOracle Block

O.S. BlockO.S. BlockOwnerOwner

SchemaSchema SegmentSegment ExtentExtent

TableTable

IndexIndex

ClusterCluster

SnapshotSnapshot

Oracle Architectural Components (Logical)Oracle Architectural Components (Logical)

PartitionPartition

CREATE TABLECREATE TABLE sales sales ( invoice_no ( invoice_no NUMBER NOT NULL,NUMBER NOT NULL, sale_year NUMBER NOT sale_year NUMBER NOT NULL,NULL, sale_month NUMBER NOT sale_month NUMBER NOT NULL,NULL, sale_day NUMBER NOT sale_day NUMBER NOT NULL )NULL )PARTITION BY RANGE PARTITION BY RANGE (sale_year, (sale_year, sale_month, sale_month, sale_day) sale_day)(( PARTITIONPARTITION sales_q1 sales_q1 VALUES LESS THANVALUES LESS THAN (1998, 04, (1998, 04, 01)01) TABLESPACE tsa STORAGE (……….), TABLESPACE tsa STORAGE (……….), PARTITIONPARTITION sales_q2 sales_q2 VALUES LESS THANVALUES LESS THAN (1998, 07, (1998, 07, 01)01) TABLESPACE tsb STORAGE (……….), TABLESPACE tsb STORAGE (……….), PARTITIONPARTITION sales_q3 sales_q3 VALUES LESS THANVALUES LESS THAN (1998, 10, (1998, 10, 01)01) TABLESPACE tsc STORAGE (……….), TABLESPACE tsc STORAGE (……….), PARTITIONPARTITION sales_q4 sales_q4 VALUES LESS THANVALUES LESS THAN (1999, 01, (1999, 01, 01)01) TABLESPACE tsd STORAGE (……….) TABLESPACE tsd STORAGE (……….)););

Maintaining Partitions

This section describes how to perform the following specific

partition maintenance operations:

1. Moving Partitions

2. Adding Partitions

3. Dropping Partitions

4. Coalescing Partitions

5. Modifying Partition Default Attributes

6. Truncating Partitions

7. Splitting Partitions

8. Merging Partitions

9. Exchanging Table Partitions

Regular table accessRegular table access

ROWIDROWID

Index-Organized TablesIndex-Organized TablesIndex-Organized TablesIndex-Organized TablesIOT accessIOT access

Non-key columnsNon-key columnsKey columnKey columnRow headerRow header

Index-Organized TablesIndex-Organized TablesCompared with Regular TablesCompared with Regular Tables

Index-Organized TablesIndex-Organized TablesCompared with Regular TablesCompared with Regular Tables

Faster key-based access to table dataFaster key-based access to table data Reduced storage requirementsReduced storage requirements Main restrictions:Main restrictions:

Must have a primary keyMust have a primary key

Cannot use unique constraintsCannot use unique constraints

Cannot be clusteredCannot be clustered

Faster key-based access to table dataFaster key-based access to table data Reduced storage requirementsReduced storage requirements Main restrictions:Main restrictions:

Must have a primary keyMust have a primary key

Cannot use unique constraintsCannot use unique constraints

Cannot be clusteredCannot be clustered

Creating Index-Organized Creating Index-Organized TablesTables

Creating Index-Organized Creating Index-Organized TablesTables

SQL> create table sales 2 (office_cd number(3) 3 ,qtr_end date 4 ,revenue number(10,2) 5 ,constraint sales_pk 6 PRIMARY KEY (office_cd,qtr_end) 7 ) 8 ORGANIZATION INDEX 9 tablespace indx 10 storage (…);

SQL> create table sales 2 (office_cd number(3) 3 ,qtr_end date 4 ,revenue number(10,2) 5 ,constraint sales_pk 6 PRIMARY KEY (office_cd,qtr_end) 7 ) 8 ORGANIZATION INDEX 9 tablespace indx 10 storage (…);

ClusterClusterClusterCluster

Use clusters to store one or more tables that:1) Are primarily queried 2) Not predominantly inserted into or updated 3) Which the queries often join data of multiple tables in the cluster.

CREATE CLUSTER emp_dept (deptno NUMBER(3))PCTFREE 5TABLESPACE usersSTORAGE (INITIAL n NEXT m MINEXTENTS 1 MAXEXTENTS 121 PCTINCREASE 0 );

CREATE TABLE dept (deptno NUMBER(3) PRIMARY KEY, . . )CLUSTER emp_dept (deptno);

CREATE TABLE emp (empno NUMBER(5) PRIMARY KEY, ename VARCHAR2(15) NOT NULL, . . . deptno NUMBER(3) REFERENCES dept)CLUSTER emp_dept (deptno);

CREATE INDEX emp_dept_index ON CLUSTER emp_deptTABLESPACE usersSTORAGE (INITIAL n NEXT m MINEXTENTS 1 MAXEXTENTS 121 PCTINCREASE 0) PCTFREE 5;

Index Function BasedIndex Function BasedIndex Function BasedIndex Function BasedFeatures of Function-Based Indexes

Function-based indexes allow you to:

1) More powerful

2) Precompute the value of a computationally intensive function and store it in the index

3) Increase the number of situations where the optimizer can perform a range scan instead of a full table scan

You must have the following initialization parameters defined tocreate a function-based index:

*) QUERY_REWRITE_ENABLED set to TRUE*) COMPATIBLE set to 8.1.0.0.0 or a greater value

Index Function BasedIndex Function BasedIndex Function BasedIndex Function BasedExample: Function-Based Index for Case Insensitive Searches

The following statement creates function-based index idx on table emp based on an uppercase evaluation of the ename column:

CREATE INDEX idx ON emp (UPPER(ename));

Now the SELECT statement uses the function-based index on UPPER(ename) To retrieve all employees with names that start with JOH:

SELECT * FROM emp WHERE UPPER(ename) LIKE 'JOH%';

Example: Precomputing Arithmetic Expressions with a Function-Based Index This statement creates a function-based index on an expression:

CREATE INDEX idx ON t (a + b * (c - 1), a, b);

SELECT statements can use either an index range scan (in the followingSELECT statement the expression is a prefix of the index) or index full scan (preferable when the index specifies a high degree of parallelism).

SELECT a FROM t WHERE a + b * (c - 1) < 100;

Discoverer Architecture

User editionUser edition Administration edition

Administration edition

End User Layer

Database (OLTP, Data Warehouse, Data Mart)

BusinessArea

Database Complexity is Hidden From Users

Viewer editionViewer edition

Data Base DWHData Base DWH

Data Warehouse ToolsData Warehouse Tools

Low-end OLAP ToolsLow-end OLAP Tools per Simple Queries e Reports ad hoc per Simple Queries e Reports ad hoc Adeguati per rispondere a domande della tipologia Adeguati per rispondere a domande della tipologia “ “che cosa?”che cosa?”

Powerful Multi-Dimensional Analysis OLAP ToolsPowerful Multi-Dimensional Analysis OLAP Tools Supportano “drill down” in “detail data” per rispondere a Supportano “drill down” in “detail data” per rispondere a

domande della tipologia domande della tipologia “perche?”“perche?” e e “come?”“come?”

OLAP ToolsOLAP Tools Slicing del “Cubo”Slicing del “Cubo” PivotPivot

Ruotare l’approccio visuale di un punto del cubo stesso al Ruotare l’approccio visuale di un punto del cubo stesso al fine di ottenere una nuova prospettivafine di ottenere una nuova prospettiva

Drill-DownDrill-Down

Data Reconciliation Steps Data Reconciliation Steps

Data Reconciliation ProcessData Reconciliation Process

CaptureCapture Static - initial loadStatic - initial load Incremental - ongoing updateIncremental - ongoing update

Scrub or data cleansingScrub or data cleansing Pattern recognition and other artificial intelligence Pattern recognition and other artificial intelligence

techniquestechniques

TransformTransform Convert the data format from the source to the target Convert the data format from the source to the target

systemsystem Record-Level FunctionsRecord-Level Functions

SelectionSelection

JoiningJoining

Aggregation (for data marts)Aggregation (for data marts) Field-Level FunctionsField-Level Functions

Single-field transformationSingle-field transformation

Multi-field transformationnMulti-field transformationn