traditional data warehousing, & business analytics market segmentation
DESCRIPTION
IBM DB2 Data Analytics Accelerator Technology Exploration - User Group Road Show - Detroit Columbus Cincinnati - April 08 09 10. 1. Trans. Latency. Number of Users. Trans. Volume. Trans. Type. Availability. Less Important. Less Important. Few. Few. Small. Complex. C Level - PowerPoint PPT PresentationTRANSCRIPT
© 2014 IBM Corporation 1
IBM DB2 Data Analytics Accelerator Technology Exploration - User Group Road Show - Detroit Columbus Cincinnati - April 08 09 10
© 2014 IBM Corporation
Traditional Data Warehousing, & Business Analytics Market Segmentation
Customers(e.g. external, Web)
Customer Service & Support(e.g. call centers, sales personnel)
CompanyManagement
Analysts(e.g. Mktg, Research)
C LevelMgt
User CommunityTrans.
Volume
Small
Very Large
Trans.Type
Complex
Simple
AvailabilityLess
Important
Critical
Numberof Users
Few
Many
FewTraditional Distributed
Market
Trans.Latency
LessImportant
Critical
© 2014 IBM Corporation3
Typical Current State of Affairs for Analytics: Execution by Department
Marketing
Sales
Finance
Operations
Customer CareIT
Executive
Management
Finance
Research & Dev.
© 2014 IBM Corporation
Multiply that Across each DepartmentDevelopment
Quality Assurance
Production
Disaster Recovery
Development
Quality Assurance
Production
Disaster Recovery
Development
Quality Assurance
Production
Disaster Recovery
Development
Quality Assurance
Production
Disaster Recovery
R&DMarketing
Sales Finance
© 2014 IBM Corporation
What this Looks Like in the Bigger Picture
Report
MIS SystemBudgeting
Campaign managementFinancial AnalysisSelling Platforms
Customer Profit AnalysisCRM
Scoring Rules
AnalyticalForesight
Optimized Business ProcessesOptimized Business ProcessesCustomer SupportCustomer Support
Claims ProcessingClaims Processing
UnderwritingUnderwriting
Fraud ManagementFraud Management
Sales EffectivenessSales Effectiveness
MarketingMarketing
Staging Area
Transformation Server
x/p serverData Mover
Staging Area
ODS (RDBMS)
x/p/z server
CleanseTransform
Warehouse
Enterprise Data Warehouse
(RDBMS)
x/p/z server
Batch Process
Continuous feed
Multi-Dimensional
Analysis
x/p server
Departmantal Data Marts
x/p/z server
Hourly/dailyBatch
Process
Analyze
AnalyticsServer
x/p server
Bulk
Data Mining Segmentation
PredictionStatistical Analysis
OLTPOperational
Systems
Online Queries & ReportingBA Tooling
© 2014 IBM Corporation
World’s Best Companies Run DB2 for z/OS and System zUndisputed leader in total system availability, scalability, security and reliability
Availability Zero downtime - maintenance and upgrades without service disruption. Hyperswap for storage availability and GDPS for unmatched disaster recovery
Security Highest protection of business critical data and applications on Evaluation Assurance Level 5 (Common Criteria Security Certification)
Scalability Unmatched Near-linear scalability through hardware-based System zCoupling Facility for DB2 data sharing
Performance CPU reductions out-of-the-box
Integration Management integration, multi-platform integration and stack integration
Efficiency Reduced infrastructure complexity through consolidation, automation and virtualization. Savings on software and environmental costs, reduce labor, energy and developmental costs
© 2014 IBM Corporation
OLTP vs. Analytics – Examples
OLTP - “Transactional”Transactional Analytics:
(Operational BA)Deep Analytics
Withdrawal from a bank account using an ATM
Approve request to increase credit line based on credit history and customer profile
Regular reporting to central bank – sum of transactions by account
Buying a book at Amazon.com Propose additional books based on similar purchases by other customers
Which books were best-sellers in Europe over the last 2 months?
Check-In for a flight at the airport
Offer an upgrade based on frequent flyer history of all passengers and available seats
Marketing campaign to sell more tickets in off-peak times
Hand-over manufactured printers to an oversea-carrier
Optimize shipping by selecting cheapest and most reliable carrier on demand
Trend of printers sold in emerging countries versus established markets.
© 2014 IBM Corporation88
Creating the Hybrid Data Server – PureData System for Analytics (Netezza) and System z
Transaction Processing Systems (OLTP)
Deep Analytics
DB2 z/OS: Recognized leader in
transactional workloads with security, availabilityand recoverability
PureData System for Analytics
Recognized leader in cost-effective high speed deep
analytics
Data Mart Data Mart Data Mart
Data Mart Consolidation
Best in Deep Analytics
Proven appliance leader in high speed analytic systems
Best in Consolidation
Unprecedented mixed workload flexibility and virtualization providing the most options for cost effective consolidation
Best in OLTP and Transactional Analytics
Industry recognized leader in mission critical transaction systems
Together:Destroying the myth that transactional and decision support workloads have to be on separate platforms
Transactional Analytics
© 2014 IBM Corporation99
IBM DB2 Analytics Accelerator for z/OSBlending zEnterprise and Netezza technologies
FastComplex queries run up
to 2000x faster while retaining single record
lookup speed
Cost SavingEliminate costly query tuning while offloading
complex query processing
ApplianceNo applications to
change, just plug it in, load the data, and gain
the value
A high performance analytics
accelerator appliance for IBM
zEnterprise, delivering dramatically
faster complex business analysis
transparently to all users.
© 2014 IBM Corporation101010
IBM DB2 Analytics Accelerator for z/OS What is it? What is the value?
What is it? The IBM DB2 Analytics Accelerator is a dedicated workload optimized, appliance add-on to a DB2 for z/OS environment that services long-running, complex queries.
IDAADeep Analytics
z Enterprise z196, z114, EC12, BC12OLTP and Transactional Analytics
What is the value?
Breakthrough technology enabling new opportunities
Extreme performance for complex analytics (aka Train of Thought Analysis)
Improve price/performance for analytic workloads
Minimize the need to create data marts for performance
Transparent to DB2 applications and users
Highly secure environment for sensitive data analysis
DB2 for z/OS
Versions Supported
IDAA V2 - DB2 for z/OS V9 & V10
IDAA V3 - DB2 for z/OS V10 & V11
IDAA V4 (GA 11/29) - DB2 for z/OS V10 & V11
© 2014 IBM Corporation
DB2 Analytics Accelerator V3 Features
High Performance Storage Saver• Significantly reduces the cost for storage resources• Option to store data only once in the accelerator
Incremental Update • Data changes are propagated to the accelerator as they happen• Uses change data capture technology• Extends the accelerator use to reporting on operational data
New optimization• Tables or partitions refresh much faster and less resources intensive• Optimized unloading data from DB2
High Capacity• Capacity has been extended to 1.28 PB for a single Accelerator
New functions• More queries eligible for acceleration
PureData System for Analytics N200111
Better decisions through lower latency of data
Dramatic improvement in scale and growth opportunities
Lowering the cost of analytic computing
Lowering the cost of historical data
© 2014 IBM Corporation
More Query Acceleration
E n a b l i n g n e w u s e c a s e s
Static SQL
DB2 Version 11 Support added
Multi-row fetch from local applications
EBCDIC & Unicode in same DB2 system & accelerator
Incremental Update• Greatly improved scalability• Better performance
Improved performance for large result sets
HPSS• Archive to multiple accelerators• Better access control for
archived partitions
Extend WLM support to local applications
Richer system scope monitoring
Report prospective CPU cost & Elapsed time savings
Separation of duties for accelerator system administration operations
EnhancedCapabilities
Automatic workload balancing over multiple accelerators
New RTS ‘last-changed-at’ timestamp
Automated NZKit installation
HPSS
• Built-in restore
• Protection for image copies of archived partitions
Profile controlled special registers
Improved continuous operations for Incremental Update
ImprovedTransparency
DB2 Analytics Accelerator V4 FeaturesNEW
© 2014 IBM Corporation1313
Deep DB2 Integration within zEnterprise
DataManager
BufferManager
IRLMLog
Manager
IBMDB2
AnalyticsAccelerator
Applications DBA Tools, z/OS Console, ...
. .
.
Operational Interfaces(e.g. DB2 Commands)
Application Interfaces(standard SQL dialects)
z/OS on System z
PureData System for Analytics
DB2 for z/OS
Superior availabilityreliability, security,
Workload management
Superior performance on analytic queries
© 2014 IBM Corporation1414
Large Insurance Company Adding value by Accelerating the Delivery of Business Reporting
Customer Quote: “we had this up and running in days with queries that ran over 1000 times faster”
Initial Load Performance400 GB Loaded in 29 Minutes570 Million Rows (Actual: Loaded 800 GB to 1.3 TB per hour)
Extreme Query Acceleration - 1908x faster2 Hours 39 minutes to 5 Seconds
CPU Utilization Reduction35% to ~0%
IBM DB2 Analytics Accelerator (PureData System for Analytics 1000-12)
Production ready - 1 person, 2 days
Table Acceleration Setup in 2 Hours - DB2 “Add Accelerator” - Choose a Table for “Acceleration” - Load the Table (DB2 Loads Data to the Accelerator) - Knowledge Transfer - Query Comparisons
With Accelerated Time to Value
Times Faster
QueryTotal Rows Reviewed
Total Qualifying
Rows
Total Rows
Returned Hours Sec(s) Hours Sec(s)
Query 1 591,941,065 2,813,571 853,320 2:39 9,540 0.0 5 1,908
Query 2 591,941,065 2,813,571 585,780 2:16 8,220 0.0 5 1,644
Query 3 813,343,052 8,260,214 274 1:16 4,560 0.0 6 760
Query 4 283,105,125 2,813,571 601,197 1:08 4,080 0.0 5 816
Query 5 591,941,089 3,422,765 508 0:57 4,080 0.0 70 58
Query 6 813,343,052 4,290,648 165 0:53 3,180 0.0 6 530
Query 7 591,941,065 361,521 58,236 0:51 3,120 0.0 4 780
Query 8 813,343,052 3,425,292 724 0:44 2,640 0.0 2 1,320Query 9 813,343,052 4,130,107 137 0:42 2,520 0.1 193 13
DB2 Only DB2 with
IDAA
Load times may vary based other workload running on z/OS
© 2014 IBM Corporation151515
IBM DB2 Analytics Accelerator Product Components
10Gb
Primary
Backup
CLIENT
Data Studio Foundation
DB2 Analytics Accelerator
Admin Plug-in
zEnterprise
Data Warehouse applicationDB2 for z/OS enabled for IBM
DB2 Analytics Accelerator
IBM DB2 Analytics Acelerator
PureData System for Analytics
Technology
Users/Applications
Network
OSA-Express3/4/5
10 GbE
© 2014 IBM Corporation161616
Query routing analysis Values for CURRENT QUERY ACCELERATION
DB2 for z/OS and
IBM DB2 Analytics Accelerator OLTP-like query
OLTP-like query
Light ODS-query
Light ODS-query
Heavy BI Query
Heavy BI Query
Light BI Query
Light BI Query
DB2 Native ProcessingDB2 Native Processing
Use
r con
trol a
nd
DB
2 h
eu
ristic Optimized processing
for BI Workload
Value Description
NONE No query is routed to the accelerator
ENABLE A query is routed to the accelerator if it satisfies the acceleration criteria including the cost and heuristics criteria. Otherwise it is executed in DB2.
If there is an accelerator failure while running the query, or the accelerator returns an error, DB2 will return a negative SQL Code to the application
ENABLE WITH FAILBACK
A query is routed to the accelerator if it satisfies the acceleration criteria including the cost and heuristics criteria. Otherwise it is executed in DB2.
Under certain conditions the query will run on DB2 after it fails in the accelerator. In particular any negative SQL code will cause failback to DB2 during PREPARE or first OPEN. No failback is possible after a successful OPEN of a query
ELIGIBLE A query is routed to the accelerator if it satisfies the acceleration criteria irrespective of the cost and heuristics criteria. Otherwise it is executed in DB2
ALL A query is routed to the accelerator, if it cannot execute the query fails and a negative return code is passed back to the application
Single and unique system for mixed query workloads
Dynamic decision for most efficient execution platform
New special register QUERY ACCELERATION
New heuristic in DB2 optimizer
© 2014 IBM Corporation1717
Routing Criteria
The entire query can be accelerated, i.e. the unit of acceleration is a whole query
– Individual query blocks are no longer units of acceleration– The whole query will either run in DB2 or in the accelerator
The associated cursor is not defined as a scrollable or a rowset cursor
The query is defined as read-only
The query is dynamic (V4 will support static)
The query is a SELECT statement.
The private protocol is not in effect.
Routing to Accelerator is considered more efficient for performance than to execute the query in DB2 mainline
– The decision is based on some heuristic rules
A query can be routed to Accelerator if is one of these types:
© 2014 IBM Corporation18
Heuristic Routing Criteria – not just based on “elapsed time” … DB2 Optimizer uses a set of rules to determine whether a given query is better off being
executed in DB2 core engine or routed to the accelerator, such as:– In general, typical OLTP access path patterns are not routed to the accelerator
e.g. Equal unique access, One fetch access
– If none of these: WHERE, GROUP BY, ORDER BY, aggregate functions is specified (i.e. all rows are to be returned), the query is not routed
– Threshold specified by the DB2 Profile (1) mechanism:
• If all the tables referred in the query are “small”, the query is not routed
ACCEL_TABLE_THRESHOLD determines total table cardinality for a query
The default value is 1,000,000
• If a “large” result set is expected, the query is not routed
ACCEL_RESULTSIZE_THRESHOLD (number of rows) determines what is a “large” result set.
The default value is -1, which means that this check is ignored
• If estimated total cost for a query is treated as “short running”, the query is not routed
ACCEL_TOTALCOST_THRESHOLD determines estimated total cost for a query
The default value is 5,000 (value is in milliseconds = 5 seconds)
• Recommendation: Use default values. Change only after rigorous testing!
(1) http://publib.boulder.ibm.com/infocenter/dzichelp/v2r2/index.jsp?topic=/com.ibm.db2z10.doc.perf/src/tpc/db2z_profiles.htm
© 2014 IBM Corporation1919
Query Off-load Applicability Accelerator is based on Netezza which supports rich set of SQL and data types
– BI tools such as Cognos has run on Netezza for years and will run on Accelerator as well
Due to very large number of query types and SQL functions not all of them could be processed in V3. Key restrictions include:
– No static SQL (Lifted in V4)– Not all DB2 functions,
No Mathematical functions such as SIN, COS, TAN.
No advanced string functions such as HEX, POSITION, LOCATE, LEFT, OVERLAY
No advanced OLAP functions such as RANK, ROLLUP, CUBE– No User Defined Functions– No correlated table expressions or recursive correlated table expressions– No correlated subquery in the SELECT list – Not UTF-16 and MIXED/DBCS EBCDIC– No multiple encoding schemes in the same statement (fixed in V4)– Not all DB2 special registers: CURRENT PATH, SERVER, SQLID, SCHEMA,
APPLICATION ENCODING SCHEME– Not all DB2 data types: LOBs, ROWID, XML, DECFLOAT, BINARY
None of these restrictions is a design problem, IBM plans to lift them in future releases based on customer feedback and needs
© 2014 IBM Corporation202020
Query Execution Process Flow
DB2 for z/OS
Optimizer
IDA
A D
RD
A R
equestor
DB2 Analytics Accelerator
Application
Application
Interface
Queries executed with DB2 Analytics Accelerator
Queries executed without DB2 Analytics Accelerator
Query execution run-time for queries that cannot be or should
not be off-loaded to IDAA
SPU
CPU FPGA
Memory
SPU
CPU FPGA
Memory
SPU
CPU FPGA
Memory
SPU
CPU FPGA
Memory
SM
P H
ost
Queries executed with value of “ALL” may receive a SQL Error Code if the query cannot run on the accelerator
Heartbeat (DB2 Analytics Accelerator availability and performance indicators)
© 2014 IBM Corporation2121
Accelerator Administrative Stored Procedures ACCEL_ADD_ACCELERATOR ACCEL_TEST_CONNECTION ACCEL_REMOVE_ACCELERATOR ACCEL_UPDATE_CREDENTIALS ACCEL_ADD_TABLES ACCEL_ALTER_TABLES ACCEL_REMOVE_TABLES ACCEL_GET_TABLES_INFO ACCEL_GET_TABLES_DETAILS ACCEL_LOAD_TABLES ACCEL_SET_TABLES_ACCELERATION ACCEL_SET_TABLES_REPLICATION ACCEL_CONTROL_ACCELERATOR ACCEL_UPDATE_SOFTWARE ACCEL_ARCHIVE_TABLES ACCEL_GET_QUERIES ACCEL_GET_QUERY_DETAILS ACCEL_GET_QUERY_EXPLAIN
* For a list of the description of each stored procedure refer to the IDAA Stored Procedure Reference Manual
© 2014 IBM Corporation2222
The Key to the Speed
FPGA Core CPU Core
Uncompress
FROM
Project
SELECT
Restrict,Visibility
WHERE
Complex ∑Joins, Aggs, etc.
GROUP BY
select DISTRICT, PRODUCTGRP, sum(NRX)from MTHLY_RX_TERR_DATAwhere MONTH = '20091201'and MARKET = 509123and SPECIALTY = 'GASTRO'
select DISTRICT, PRODUCTGRP, sum(NRX)from MTHLY_RX_TERR_DATAwhere MONTH = '20091201'and MARKET = 509123and SPECIALTY = 'GASTRO'
Slice of tableMTHLY_RX_TERR_DATA(compressed)
Slice of tableMTHLY_RX_TERR_DATA(compressed)
where MONTH = '20091201'and MARKET = 509123and SPECIALTY = 'GASTRO'
where MONTH = '20091201'and MARKET = 509123and SPECIALTY = 'GASTRO'
sum(NRX)sum(NRX)
select DISTRICT, PRODUCTGRP, sum(NRX)
select DISTRICT, PRODUCTGRP, sum(NRX)
© 2014 IBM Corporation2323
IBM DB2 Analytics Accelerator
Table Definition and Deployment
IDAA StudioIDAA AdministrativeStored Procedures
PureData Catalog
IBM Data Studio Client DB2 for z/OS IDAA
The tables need to be defined and deployed to IDAA before data is loaded and queries sent to it for processing Definition: identifying tables for which queries need to be accelerated Deployment: making tables known to DB2, i.e. storing table metadata in the DB2 and PureData catalog
IDAA Studio guides you through the process of defining and deploying tables, as wall as invoking other administrative tasks
IDAA Stored Procedures implement and execute various administrative operations such as table deployment, load and update, and serve as the primary administrative interface to IDAA from the outside world including IDAA Studio
DB2 Catalog
© 2014 IBM Corporation242424
Synchronization Options with IBM DB2 Analytics Accelerator
Synchronization options Use cases, characteristics and requirements
Full Table Refresh
The entire content of a database table is refreshed for accelerator processing
Existing ETL process replaces entire table
Multiple sources or complex transformations
Smaller, un-partitioned tables
Reporting based on consistent snapshot
Table Partition Refresh
For a partitioned database table, selected partitions can be refreshed for accelerator processing
Optimization for partitioned warehouse tables, typically appending changes “at the end”
More efficient than full table refresh for larger tables
Reporting based on consistent snapshot
Changed partitions only through real-time statistics (starting with IDAA V3 for DB2 V10)
Incremental Update (starting w/IDAA V3 for DB2 V10)
Log-based capturing of changes and propagation to IBM DB2 Analytics Accelerator with low latency (typically few minutes)
Scattered updates after “bulk” load
Reporting on continuously updated data (e.g., an ODS), considering most recent changes
More efficient for smaller updates than full table refresh
© 2014 IBM Corporation
DB2 for z/OS Accelerator
Accelerator Data Load
Accelerator Studio
Ac
ce
lera
tor A
dm
inis
trativ
e
Sto
red
Pro
ce
du
res
.
.
.
.
.
.
.
.
.
Table A
Part 1
Part 2
Part m
Table C
Table B
Table D
Part 1
Part 2
Part 3
Unload USS Pipe
Unload
Unload
USS Pipe
USS Pipe
CPU FPGA
Memory
CPU FPGA
Memory
CPU FPGA
Memory
CPU FPGA
Memory
Co
ord
inato
r
• 1 TB / h – can vary, depending on CPU resources, table partitioning, …• Update on table partition level, concurrent queries allowed during load• Unload in DB2 internal format, single translation by accelerator
© 2014 IBM Corporation2626
Incremental Update – IDAA V3 for DB2 V10 Changes in data warehouse tables typically
driven by replication or an other update process
– Corrections after a bulk-ETL-load of a data warehouse table
– Continuously changing data (e.g. trickle-feed updates from a transactional system to the warehouse or ODS)
Reporting and analysis based on most recent data
May be combined with a full/partition table refresh DB2 z/OS Query OptimizerDB2 z/OS Query Optimizer
ChangesDB2 for z/OS databaseDB2 for z/OS database
DB2 native processingDB2 native processing
AcceleratorprocessingAcceleratorprocessing
Operational Analytics, Reports, OLAP, …Operational Analytics, Reports, OLAP, …
ContinuousQuery
Processing
ContinuousQuery
Processing
Incremental Update
Rep
licat
ion
Rep
licat
ion
Ap
plic
atio
nA
pp
licat
ion Incremental update
can be configured per table
Initial full table load is required
© 2014 IBM Corporation2727
DB2 Analytics Accelerator (Host Node)DB2 Analytics Accelerator (Host Node)
Incremental Update Details
DB2 for z/OSDB2 for z/OS
insert
delete
update
CaptureAgent
CaptureAgent
AcceleratorDatabase
AcceleratorDatabase
Apply Agenton NPS host
(Receives log events)
Apply Agenton NPS host
(Receives log events)
Accelerator SERVERAccelerator SERVER
(private network)
Cataloginformation
Cataloginformation
Accelerator Stored Procedures
ACCEL_SET_TABLES_REPLICATIONACCEL_CONTROL_ACCELERATOR
...
Accelerator Stored Procedures
ACCEL_SET_TABLES_REPLICATIONACCEL_CONTROL_ACCELERATOR
...
JCLJCL
Automation code(creates data sources,
subscriptions, etc.)
Automation code(creates data sources,
subscriptions, etc.)
ControllerController
AcceleratorStudio
AcceleratorStudio
IFI LogReads
© 2014 IBM Corporation2828
User InterfaceIncremental update UI elements only visible if it has been enabled on the DB2 subsystem via IBM DB2 Analytics Accelerator configuration console
Start / stop replication process (per subsystem-accelerator pair)
Enable / disable replication (per table)
Trace collection
Information on replication latency and events
© 2014 IBM Corporation2929
Save Over 95% of Host Disk Space for Historical Data
1Q1Q
2Q2Q
3Q3Q
4Q4Q
1Q1Q
2Q2Q
3Q3Q
4Q4Q
1Q1Q
2Q2Q
3Q3Q
4Q4Q
1Q1Q
2Q2Q
3Q3Q
4Q4Q
1Q1Q
2Q2Q
3Q3Q
4Q4Q
1Q1Q
2Q2Q
3Q3Q
4Q4Q
1Q1Q
2Q2Q
3Q3Q
4Q4Q
YearYear Year -7Year -7Year -2Year -2 Year -3Year -3 Year -4Year -4 Year -5Year -5Year -1Year -1
Historical DataHistorical Data
Current DataCurrent Data
One Quarter = 3.57% of 7 years of data
One Month = 1.19% of 7 years of data
One month = 2.78% of 3 years of data
One Quarter = 3.57% of 7 years of data
One Month = 1.19% of 7 years of data
One month = 2.78% of 3 years of data
© 2014 IBM Corporation3030
Accelerator
High Performance Storage Saver Reducing the cost of high speed storage
Time-partitioned tables where:– only the recent partitions are used in a transactional context (frequent
data changes, short running queries) – the entire table is used for analytics (data intensive, complex queries).
High Performance Storage Saver’s “Archive” Process:– Data is loaded into Accelerator if not already loaded– Automatically takes Image Copy of Each Partition to be Archived– Automatically Remove data from DB2 archived tablespace partitions– DBA starts archived partitions as read-only
Part #1
Query from Application
Or
No longer present on DB2 Storage
Part #1
Part #2
Part #3
Part #4
Part #5
Part #6
Part #7
DB2
Active Archive
© 2014 IBM Corporation3131
High Performance Storage SaverReducing the cost of high speed storage
DB2Table A
AcceleratorTable A
Tables can be resident on:1. DB2 Only2. DB2 and Accelerator3. Archive to Accelerator
Applications
Managed by zPARMs
Controlled by Special Registers:
CURRENT QUERY ACCELERATION
CURRENT GET_ACCEL_ARCHIVE
DB2Table A
SQL
Store historic data on the Accelerator only
When data no longer requires updating, reclaim
the DB2 storage
Best for OLTPBest for OLTP
High Speed High Speed Indexed queriesIndexed queries
Active OnlyActive Only Archive OnlyArchive Only Active & ArchiveActive & Archive Mixed WorkloadMixed Workload
Mixed WorkloadMixed Workload
AcceleratorTable AActive & Archive
DB2 Table A
Active
© 2014 IBM Corporation32
Key Elements of the Implementation Approach
• Older partitions are moved to IDAA and their data no longer exists in DB2• DB2 is still solely responsible for the recovery and maintains all backups (copy images)
• The most recent partitions exist in both DB2 and IDAA
• They are synchronized by existing means• Partition refresh or incremental update (replication-based propagation of changes)
• Having the most recent partition in both, DB2 and IDAA, provides IDAA-driven performance acceleration for analytical queries that access most recent partitions only
• The data move process is encapsulated in a stored procedure• The stored procedure can be invoked directly or via IDAA Studio
• The SQL statements do not change• The fact that some partitions have been moved to IDAA is transparent
• By default, queries access only the data from the most recent partitions• The queries can be executed in DB2 or IDAA based on the standard routing criteria
• If all the data need to be accessed, one of the following mechanisms is used:• Setting a zparm which activates the 'all data' scope for the DB2 subsystem/data sharing group.
This way, none of the applications need to be changed (but this setting has global impact).
• Setting a special register “CURRENT GET_ACCEL_ARCHIVE”, which allows switching between the 'all data' scope and the 'most recent data' scope at any time. This way the application can use both scopes within the same execution at choose scope at SQL statement level.
© 2014 IBM Corporation3333
Work Load Management - Usage scenarios
• Workload Isolation: Ensure that the workload of one DB2 subsystem doesn’t monopolize the resources of a shared accelerator. A development subsystem, attached to the same accelerator as a production subsystem, should not be able to drain all accelerator resources.
• Query Prioritization:More important queries should be executed before and faster than less important queries that are sent from the same DB2 subsystem against the accelerator.
© 2013 IBM Corporation
IBM DB2 Analytics Accelerator
Instrumentation
34
© 2014 IBM Corporation35
IBM DB2 Analytics Accelerator Studio
© 2014 IBM Corporation3636
Moving Partitions with HPSS via IDAA Studio
© 2014 IBM Corporation37
DB2 Command – DISPLAY ACCEL DETAIL
© 2014 IBM Corporation
Product Feature Description Benefit
OM/PE Reporting Utilization of Accelerator (under/over) ROI/Performance
Real time and short term monitoring
Detailed perf reports (batch and online) for Accelerator and apps routed to the Accelerator – takes advantage of Extended Insight
ROI/Performance
Assessment Determine if workload can be accelerated ROI
Query Monitor Cost comparison Measure perf of query with & w/out Accelerator ROI/Performance
Capture workload Capture SQL workload (or queries) for designated time – can be used as input to OQWT for tuning
ROI/Performance
Separate Queries Isolate non-accelerated queries from workload ROI/Performance
OQWT Best fit object Determine best set of objects for query acceleration (both Static and dynamic SQL) ROI/Performance
Best fit queries Which queries are eligible, non-eligible or can be rewritten ROI/Performance
What-if analysis Expert recommendations with estimated cost savings (no actual execution) ROI/Performance
Loader Load non DB2 data Ability to load non-DB2 data directly to Accelerator Hybrid envROI for System z
P.i.T. load Load to a specific point-in-time to fit business needs ROI/reporting
Operational data Ability to load to Accelerator w/out impact to applications Avail/Perf/ROI
Admin/OC AcceleratorAdmin/Mgt
Complete mgt of Accelerator and accelerated objects via familiar tool w/ ISPF Usability
Refresh data? Ability to use RUNSTATS to help determine if refresh of Accelerator data is recommended
ROI/Performance
Chg Mgt Automated reload of data to Accelerator after chg Avail/Usability
© 2014 IBM Corporation3939
Product Documentation Links Information Center for IBM DB2 Analytics Accelerator for z/OS V3.1
– http://publib.boulder.ibm.com/infocenter/dzichelp/v2r2/topic/com.ibm.datatools.aqt.doc/welcome/topics/idaa_start.html
Quick Start Guide– http://publibfp.dhe.ibm.com/epubs/pdf/h1269820.pdf
Installation Guide– http://publibfp.dhe.ibm.com/epubs/pdf/h1269830.pdf
Stored Procedure Reference– http://publibfp.dhe.ibm.com/epubs/pdf/h1269840.pdf
Analytics Accelerator Studio Users Guide– http://publibfp.dhe.ibm.com/epubs/pdf/h1269850.pdf
Getting Started Guide– http://publibfp.dhe.ibm.com/epubs/pdf/h1269860.pdf
Program Directory– http://publibfp.dhe.ibm.com/epubs/pdf/i1950060.pdf
Redbook – Optimizing DB2 Queries with IBM DB2 Analytics Accelerator for z/OS– http://www.redbooks.ibm.com/redpieces/abstracts/sg248005.html?Open
Redbook – Hybrid Analytics Solution using IBM DB2 Analytics Accelerator for z/OS V3.1– http://www.redbooks.ibm.com/redpieces/abstracts/sg248151.html?Open
© 2014 IBM Corporation404040 04/22/23
40