optimizing data warehouse loads via parallel pro-c and parallel/direct-mode sql
DESCRIPTION
Optimizing Data Warehouse Loads via Parallel Pro-C and Parallel/Direct-Mode SQL. Bert Scalzo, Ph.D. [email protected]. About the Author. Oracle DBA from 4 through 10g Worked for Oracle Education Worked for Oracle Consulting Holds several Oracle Masters - PowerPoint PPT PresentationTRANSCRIPT
Optimizing Data WarehouseOptimizing Data WarehouseLoads via Parallel Pro-C and Loads via Parallel Pro-C and
Parallel/Direct-Mode SQLParallel/Direct-Mode SQL
Bert Scalzo, Ph.D.Bert Scalzo, Ph.D.
[email protected]@Quest.com
About the AuthorAbout the Author
• Oracle DBA from 4 through 10g
• Worked for Oracle Education
• Worked for Oracle Consulting
• Holds several Oracle Masters
• BS, MS and PhD in Computer Science
• MBA and insurance industry designations
• Articles in
• Oracle Magazine
• Oracle Informant
• PC Week (now E-Magazine)
About Quest SoftwareAbout Quest Software
Star Schema DesignStar Schema Design
Dimensions: smaller, de-normalized tables containing business descriptive columns that users use to query
Facts: very large tables with primary keys formed from the concatenation of related dimension table foreign key columns, and also possessing numerically additive, non-key columns used for calculations during user queries
“Star schema” approach to dimensional data modeling was pioneered by Ralph Kimball
Dimensions
Facts
108th -1010th
103rd -105th
The Loading ChallengeThe Loading Challenge
How much data would a data loader load, if a data loader could load data?
Dimensions: often reloaded in their entirety, since they only have have tens to hundreds of thousands of rows
Facts: must be cumulatively loaded, since they generally have hundreds of millions to billions of rows – with daily data loading requirements of 10-20 million rows or more
Hardware Won’t CompensateHardware Won’t Compensate
Often people have unrealistic expectation that using expensive hardware is only way to obtain optimal application performance
•CPU•SMP•MPP
•Disk IO•15,000 RPM•RAID (EMC)
•OS•UNIX•64-bit
•Oracle•OPS / PQO•64-bit
Hardware Tuning ExampleHardware Tuning Example
Problem: Data load runtime 4+ hours•8 400-MHz 64-bit CPU’s•4 Gigabytes UNIX RAM•2 Gigabytes EMC Cache•RAID 5 (slower on writes)
Attempt #1: Bought more hardware•16 400-MHz 64-bit CPU’s•8 Gigabytes UNIX RAM•4 Gigabytes EMC Cache•RAID 1 (faster on writes)•Runtime still 4+ hours !!!
Application Tuning ExampleApplication Tuning Example
Attempt #2: Redesigned application•Convert PL/SQL to Pro-C•Run 16 streams in parallel•Better utilize UNIX capabilities•Run time = 20 minutes !!!
Attempt #3: Tuned the SQL code•Tune individual SQL statements•Use Dynamic SQL method # 2
•Prepare SQL outside loop•Execute Prep SQL in loop
•Run time = 15 minutes !!!
Lesson LearnedLesson Learned
Hardware:•Cost approximately $1,000,000•System downtime for upgrades•Zero runtime improvement•Loss of credibility with customer
Redesign:• 4 hours DBA $150/hour = $600•20 hours Developer $100/hour = $2000•Total cost = $2600 or 385 times less
Golden Rule #1: Application redesign much cheaper than hardware!!!
Program Design ParamountProgram Design Paramount
In reality, the loading program’s design is the key factor for the fastest possible data loads into any large-scale data warehouse
Data loading programs must be designed to utilize SMP/MPP architectures, otherwise CPU usage may not exceed 1 / # of CPU’s
Golden Rule #2: minimize inter-process waits and maximize total concurrent CPU usage
Example Loading ProblemExample Loading Problem
Hardware:• HP-9000, V2200, 16 CPU, 8 GB RAM• EMC 3700 RAID-1 with 4 GB cache
Database:• Oracle 8.1.5.0 (32 bit)• Tables partitioned by month• Indexes partitioned by month
Nightly Load:• 6000 files with 20 million detail rows• Summarize details across 3 aggregates
Original, Bad DesignOriginal, Bad Design
catBigFile#1
sortBigFile#2
PL/SQL
DetailTable
AggregateTables
SQLLoader
StagingTable
LookupTables
6000Files
Original’s Physical ProblemsOriginal’s Physical Problems
IO Intensive:• 5 IO’s from source to destination• Wasted IO’s to copy files twice
Large Waits:• Each step dependent on predecessor• No overlapping of any load operations
Single Threaded:• No aspect of processing is parallel• Overall CPU usage less than 7%
Original’s Logical ProblemsOriginal’s Logical Problems
Brute Force:• Simple for programmers to visualize• Does not leverage UNIX’s strengths
Record Oriented:• Simple for programmers to code (cursors)• Does not leverage SQL’s strengths (sets)
Stupid Aggregation:• Process record #1, create aggregate record• Process record #2, update aggregate record• Repeat last step for each record being input
Original Version’s TimingsOriginal Version’s Timings
ProcessStep
StartTime
Duration(minutes)
Cat T-000 30
Sort T-030 45
SQL Loader T-075 15
PL/SQL T-090 180
T-270 270
CPU Utilization
Glance Plus Display HP-UX 11.0 16 CPU’s
Parallel Design OptionsParallel Design Options
Parallel/Direct SQL Loader:• Use Parallel, Direct option for speed• Cannot do data lookups and data scrubbing
without complex pre-insert/update triggers
Multi-Threaded Pro-C:• Hard to monitor via UNIX commands• Difficult to program and hard to debug
“Divide & Conquer”:• Leverages UNIX’s key strengths• Easy to monitor via UNIX commands • Simple UNIX shell scripting exercise• Simple Pro-C programming exercise
What Are Threads?What Are Threads?
Multithreaded applications have multiple threads, where each thread:
• is a "lightweight" sub-processes• executes within the main process• shares code and data segments (i.e. address space)• has its own program counters, registers and stack
Global and static variables are common to all threads and require a mutual exclusivity mechanism to manage access to from multiple threads within the application.
Non-Mutex ArchitectureNon-Mutex Architecture
Non-Mutex CodeNon-Mutex Code
main() { sql_context ctx1,ctx2; /* declare runtime contexts */ EXEC SQL ENABLE THREADS; EXEC SQL CONTEXT ALLOCATE :ctx1; EXEC SQL CONTEXT ALLOCATE :ctx2; .../* spawn thread, execute function1 (in the thread) passing ctx1 */ thread_create(..., function1, ctx1); /* spawn thread, execute function2 (in the thread) passing ctx2 */ thread_create(..., function2, ctx2); ... EXEC SQL CONTEXT FREE :ctx1; EXEC SQL CONTEXT FREE :ctx2; ...}
void function1(sql_context ctx){ EXEC SQL CONTEXT USE :ctx;/* execute executable SQL statements on runtime context ctx1!!! */ ...} void function2(sql_context ctx) { EXEC SQL CONTEXT USE :ctx;/* execute executable SQL statements on runtime context ctx2!!! */ ...}
““Divide & Conquer” DesignDivide & Conquer” Design
DetailTable
AggregateTables
LookupTables
6000Files
sortPro-
C
sortPro-
C
sortPro-
C
ParalelInsert
Step #1: Form StreamsStep #1: Form Streams
degree=16file_name= ras.dltx.postrnfile_count=`ll ${file_name}.* | wc -l`if [ $file_count ]then if [ -f file_list* ] then rm -f file_list* fi ls ${file_name}.* > file_list split_count=`expr \( $file_count + file_count % $degree \) / $degree` split -$split_count file_list file_list_ ### Step #2’s code goes here ###fi
Unix shell script to form N streams (i.e. groups) of M/N data sets from M input files
Example for Step #1Example for Step #1
files:ras.dltx.postrn.1ras.dltx.postrn.2ras.dltx.postrn.3ras.dltx.postrn.4ras.dltx.postrn.5ras.dltx.postrn.6ras.dltx.postrn.7ras.dltx.postrn.8ras.dltx.postrn.9ras.dltx.postrn.10ras.dltx.postrn.11ras.dltx.postrn.12
file_list_aa: ras.dltx.postrn.1 ras.dltx.postrn.2 ras.dltx.postrn.3 ras.dltx.postrn.4
file_list_ab: ras.dltx.postrn.5 ras.dltx.postrn.6 ras.dltx.postrn.7 ras.dltx.postrn.8
…
Data Set 1
Data Set 2
Step #2: Process StreamsStep #2: Process Streams
for file in `ls file_list_*` do (( cat $file | while read line do if [ -s $line ] then cat $line | pro_c_program fi done )&)& done
waitwait
Unix shell script to create N concurrent background processes, each handling one of the streams’ data sets
Example for Step #2Example for Step #2
file_list_aa: ras.dltx.postrn.1 ras.dltx.postrn.2 ras.dltx.postrn.3 ras.dltx.postrn.4
• for each file• skip if empty• grep file• sort file• run Pro-C• inserts data
file_list_ab: ras.dltx.postrn.5 ras.dltx.postrn.6 ras.dltx.postrn.7 ras.dltx.postrn.8
• for each file• skip if empty• grep file• sort file• run Pro-C• inserts data
All running concurrently, with no wait states
Step #3: Calc AggregationsStep #3: Calc Aggregations
sqlplus $user_id/$user_pw@sid @daily_aggregate
sqlplus $user_id/$user_pw@$sid @weekly_aggregate
sqlplus $user_id/$user_pw@$sid @daily_aggregate
alter session enable parallel dml;insert /*+ parallel (aggregate_table, 16) append */ into aggregate_table (period_id, location_id, product_id, vendor_id, … )select /*+ parallel (detail_table,16 full(detail_table) ) */ period_id, location_id, product_id, vendor_id, …, sum(nvl(column_x,0))from detail_table where period_id between $BEG_ID and $END_ID group by period_id, location_id, product_id, vendor_id;
Pro-C ProgramPro-C Program
Algorithm:• Read records from Standard IO until EOF• Perform record lookups and data scrubbing• Insert processed record into detail table• If record already exists, update instead• Commit every 1000 inserts or updates
Special Techniques:• Dynamic SQL Method #2 (15% improvement)• Prepare SQL outside record processing loop• Execute SQL inside record processing loop
Dynamic SQLDynamic SQL
““Divide & Conquer” Version’s TimingsDivide & Conquer” Version’s Timings
ProcessStep
StartTime
Duration(minutes)
Stream #1 T-000 15
Stream #… T-000 15
Stream #16 T-000 15
Aggregate T-015 10
T-25 25
CPU Utilization
Glance Plus Display HP-UX 11.0 16 CPU’s
ResultsResults
Old Run Time = 270 Minutes
New Run Time = 25 Minutes
RT Improvement = 1080 %
Customer’s Reaction:
• took team to Ranger’s baseball game• gave team a pizza party & 1/2 day off• gave entire team plaques of appreciation
Other Possible ImprovementsOther Possible Improvements
Shell Script:• Split files based upon size to better balance load
Pro-C:• Use Pro-C host arrays for inserts/updates• Read lookup tables into process memory (PGA)
Fact Tables:• Partition tables and indexes by both time and
parallel separation criteria (e.g. time zone)