australian computer society – sybase iq
TRANSCRIPT
Australian Computer Society – Sybase IQ & NSW Health Case StudyLee Marshall, Sybase Australia & Peter Brandt, NSW Health30 April 2008
Sybase Confidential
Gartner
" Of all the Vendors, Sybase made the most impressive move in the 2007 Magic Quadrant. They moved a very long way to the right although slightly lower. Sybase IQ is a columnar DBMS that is very, very fast for analytics. Sybase has been turning in very impressive revenue growth with Sybase IQ after they have positioned themselves correctly in their markets around analytics. Just last quarter and in all of 2007, Sybase has recorded their best revenue and profit growth in their history".
-- Donald Feinberg, 02/072008, presentation to a packed audience on "2007 Gartner MQ (for DW DBMS)" at the Gartner BI conference at Amsterdam
Sybase Confidential
Agenda
• Intro to Sybase• Challenges of BI • How does Sybase IQ address BI Challenges?• NSW Case Study
Sybase Confidential
Introduction to Sybase
Sybase Confidential
Why Sybase
Creating an Information Edge — Today Open, cross-platform software solutions that
securely deliver information anytime, anywhere
Strongest portfolio of information management and enterprise mobility solutions in the market
The world’s most critical data in commerce, communications, finance, government and healthcare runs on Sybase
~4,000 employees in 60 countries Profitable and growing, $1 billion revenue
mark crossed in 2007 2008 - Most profitable 1st quarter in our history
Sybase is the largest global enterprise software company exclusively focused on managing and mobilizing information, from the data center to the point of action.
Sybase Confidential
Challenges of BI
Sybase Confidential
Growing Challenges for BI & the DW
MORE DATAMORE DATA - GB to TB
More end users- 100’s to 1000’s
More complex queries- ad hoc reports
Higher systems costs
Slower performance
Missed reporting windows
Impact to financial performance
Delayed decisions or responses
Missed deadlines and penalties
Today: IT Pain: Business Pain:
The technology supporting BI today must provide SPEED, SCALABILITY, FLEXIBILITY and AVAILABILITY at a low operating cost.
Sybase Confidential
BI & DW Challenge Questions
How do you manage the data explosion?
How do you provide the query response times users now expect?
How do you provide analytics across large data sets without compromising performance
How do you achieve pervasive, operational BI?
Sybase Confidential
We are Pushing the Limits!
What’s the likelihood that… the amount of data will decrease? user demands will slow down? competitive pressures will lighten up? compliance is a passing fad?
Business Intelligence infrastructures need to adapt to keep pace
Sybase Confidential
How Does Sybase IQ Address BI Challenges?
Sybase Confidential
General purpose engines are not
optimized engines
Sybase IQ removes the restrictions and limits required by traditional DBMSs to maintain performance and prevent system failure as you add more data, more users, more complexity.
No limits to amount of data available for analytics No limits to amount of concurrent users & queries No limits on launching new applications No limits to the complexity of queries No limits to access windows
The right tool for the right job
Sybase Confidential
Introduction to Sybase IQ
Designed from the start for analytics — not transactions, unlike traditional RDBMS
Column-based structure and patented indexing put Sybase IQ in
Gartner Group’s Magic Quadrant as a “Visionary.”1
High speed data loads for analytics
Exceptional query performance with ad hoc query performance up to
100x faster than a row based RDBMS
Kills RDBMS on performance, appliances on TCO
A highly optimised analytics server, designed specifically to deliver ultra-high-speed business intelligence and reporting on standard hardware and operating systems.
1 Magic Quadrant for Data Warehouse Database Management Systems 2007. Gartner Group, October, 2007.
IQ Review
Sybase Confidential
Sybase IQ’s Secret Ingredient:Column-Based Architecture
Conventional Database
c5
c4
c3
c2
c1 …c
9c8
c7
c6
r1
r2
r3
r4
r5
SYBASE IQ
c5
c4
c3
c2
c1 …c
9c8
c7
c6
r1
r2
r3
r4
r5
Data is stored vertically – Each column is stored separately
The data is the index Retrieve only columns used in the query – Reduce
System I/O dramatically Allocate a thread for each column individually –
Process the query in parallel
Data is stored horizontally Querying without indexes and views is extremely
I/O intensive Building indexes and views is a huge time and
resource drain Database footprint must be dramatically expanded
to make the environment efficient for querying
Sybase Confidential
Column-Based ArchitectureDramatically Reduced I/O = Faster Query Results
Q: How many MALES are NOT INSURED in NSW?
GenderMMFMM-
800 Bytes/Row
StateVIC
NSWQLDNSWVIC
Row-based RDBMSInsured
YYNYN
800 Bytes x 10M 16K Page = 500,000 I/Os
Process large amounts of unused data Often requires full table scan
Gender Insured State
= 2+ +1101
1101
10MBits
10M Bits x 3 column/8 Bits16K Page = 234 I/OsColumn-based Sybase IQ
0101
10MROWS
MMFM
YNYN
NSWNSWVICNSW
Sybase Confidential
Can start with one server and add CPUs and memory as needed
Can add servers and CPUs with little or no loss in scalability
Can add terabytes of disk to the SAN and Sybase IQ will manage it efficiently
IQ
CPU CPU
Mem CPU
CPU Mem
IQ
CPU CPU
CPU Mem
CPU Mem
IQ
CPU CPU
Mem Mem
Mem Mem
IQ
CPU CPU
CPU CPU
Mem Mem
IQ
CPU Mem
CPU Mem
CPU Mem
Fiber Channel
With Sybase IQ, you can support multiple users per CPU
Scales like a Grid
Mutiplexing for Scalability
Sybase Confidential
Conventional DBMS
LOAD
Base table: 0.2 - 0.5 TB
Indexes: 0.05 - 0.3 TB
Aggr/Summ: 0 - 0.1 TB
SummariesAggregates
1 – 2 TB
Indexes0.5 – 3 TB
Base table“RAW data”no indexes0.9 – 1.1 TB
2.4-6TB
Same INPUT data:“Conventional DW”is 3x-6x larger than
Sybase IQ DW
0.25 - 0.9TB
INPUT DATA:1 TBSource: flat files,ETL, replication,
ODS
Lowest TCO from data compressionDramatically reduced storage and maintenance
LOAD
Sybase Confidential
VLDW CUSTOMERS Proven Data Compression Savings
SYBASE IQ DATA COMPRESSION EXAMPLES
Raw Data Loaded
Sybase IQ Compressed
Estimated Competitor’s Data Explosion
VLDW Ref. Architecture (InfoSizing – June 2007)
1PB 260 TB 2 – 3 PB
Telefonica 70TB 15TB 140 TB to 210TB
comScore Networks 40 TB 16 TB 80 TB to 120 TB
Health Insurance Review Agency 27 TB 12 TB 54 TB to 81 TB
Samsung Card 15 TB 7 TB 30 TB to 45 TB
Nielsen Media Research 12 TB 12 TB 24 TB to 36 TB
Large Credit Card Company 80 TB 39 TB 200 TB to 300 TB
Sybase IQ Data Compression: saves money – saves time
Sybase Confidential
Sybase IQ: BenchmarkWorld’s Largest Data Warehouse
Achievement: 1 Petabyte Why it mattersShowed an 85% data compression ratio by storing a Petabyte of raw transactional data in less than 260 Terabytes of actual disk space
The bigger your data warehouse gets, the more money you save on hardware with Sybase IQ- 250 TB of storage + 1 Server (Sybase IQ)- 1500 TB of storage + 3 Servers (OLTP)
Demonstrated a substantial reduction in the number of disk drives needed for storage, translating directly into at least 90% reduction in CO2 emissions over the lifetime of the Reference Architecture
• More than 10x the savings annually on electricity bill
• 10x reduction in annual CO2 emissions• Shrink storage foot print by a factor of 6x• Contribution to your company’s “green”
initiativesFor several benchmarks, half of the Transactional data was replaced with “EDM” (Emails, Documents and Multimedia) data, creating a data warehouse populated with 572 Terabytes of raw “EDMT” data
Sybase IQ is the only analytics server with a cost effective solution to manage large volumes of unstructured data at extremely low costs for near on-line archiving or for applications that can query both structured and unstructured data
Source: Sun Data Warehouse Reference Architecture for Structured and Unstructured Data, InfoSizing, August 20, 2007
Sybase Confidential
Scale, speed, compression
"The world's largest data warehouse was not built on Teradata or Oracle but on Sybase IQ, a columnar database with very impressive data compression. 1 PB was stored in 250 TB". -- Donald Feinberg, 07/02/2008, presentation to a packed audience on "Architectures for Modern Data Warehouses", Gartner BI conference, Amsterdam
Sybase Confidential
NSW Health Case Study
Sybase Confidential
Who are NSW Health?
The New South Wales Department of Health (NSW Health) provides a comprehensive range of health and health-related services to the 7 million residents of the state including:
Public hospitals Ambulance service Mental health services Drug and alcohol services Community health centers A range of ancillary services
These services are provided in a wide range of settings from primary care outposts in remote regional areas to metropolitan-based tertiary teaching hospitals.
Sybase Confidential
What is the HIE?
The HIE is a data warehouse solution built on Sybase ASE with a Sybase IQ server for reporting and analytics. It is used and maintained by the Demand and Performance Evaluation Branch (DPEB) and captures a wide range of critical information including:
Daily hospital events Admissions Discharges Supplies Ambulance Services Back-office Functions
Information on wait times at various locations, cancer incidences across the State over long timeframes, patient demographic analysis and emergency department performance are just a few examples of the information extracted from the data warehouse.
Given the nature of the information, and NSW Health’s mission statement, any and all information must be timely, accurate and representative.
Sybase Confidential
HIE Under Stress
Sybase Engagement Summary
Initial IQ health check – identified that new hardware was required and that the IQ server required some performance and tuning. This led to an IQ upgrade engagement.
IQ Upgrade – new hardware was purchased and new version of IQ was obtained.
Environment Discovery – only a small number of users accessing IQ due to data currency and performance problems. HIE ASE server under sever stress due to ad-hoc query usage, users frustrated with HIE performance.
Architecture Discovery – existing data movement architecture was examined in order to perform IQ migration, found to be lacking in several areas.
New Architecture Proposed – several alternatives were presented to NSW Health that would eliminate the complicated and time consuming approach currently in use.
Direct Load Architecture Implemented – simplified a very complicated load process and provided far greater flexibility, error management and reporting and progress monitoring.
Dataset Greatly Expanded – used Sybase PowerDesigner to reverse engineer expanded ASE schema and implemented in IQ.
Server Tuned and Loading Optimised – queries returned far quicker even when compared to previous IQ schema, load reduced to 8 – 10 hours. Meant users could now use IQ instead of ASE.
Sybase Confidential
Old Data Refresh Architecture
ASE12.5.0.2
Data Feeds
Pre-Process
008 Server (Tru64)
Ageing Files New Files
Updates Inserts
HIE
Backup Server
SAN
5
6
7
8
hdc005 Server (ASE Staging and IQ)
ASE 12.5.0.2
Backup Server
IQ 12.4.3
1 4
2
3
Sybase Confidential
New Data Refresh Architecture
ASE12.5.0.2
Data Feeds
Pre-Process
008 Server (Tru64)
Ageing Files New Files
Updates Inserts
HIE
New HP-UX Itanium Server
IQ 12.6
Sybase Confidential
Results
Highlights of What Sybase Delivered at the Department
Reduced severe stress on HIE – removing reporting load reduced stress on HIE and delivered reports to users in record times (in some cases 400+ times faster)
Delivered extremely fast reports to an expanded user community and on an expanded dataset – new IQ server able to support all reporting users with both historical and current datasets
Massive reduction in data load times – from 40 hours down to 8 – 12 hours
Far greater data currency - data loads can occur overnight allowing reporting against current data rather than week (or more) old data
70% Data Compression – required far less disk space than expected
Greatly reduced administration effort and load failures - simplified a very complicated load process and provided far greater flexibility, error management and reporting and progress monitoring
2007 Computerworld Honours Laureate – industry recognised innovative solution
Sybase Confidential
Success at NSW Health
• “Congratulations. Its a terrific result which will make reporting much more efficient” – Peter Brandt (Associate Director, NSW Health)
• “Just had a play for the first time. Bloody blistering fast” - James Prior (HIE Manager, Health Technologies)
Sybase Confidential
Summary
Sybase Confidential
IQ Summary
Sybase IQ is designed specifically to deliver ultra-high-speed business intelligence & reporting on standard hardware and operating systems
"We have been watching and waiting for database implementers to catch up with Sybase IQ and its unique table and index structure, which deliver impressive performance for complex queries against large data warehouses. Judging from sales over the past couple of years, it seems that the market finally 'gets it'.“
Carl Olofson, Research Vice President Information Management and Data Integration Software Research IDC, 2007
It delivers for today’s operational BI needs …high speed data loads for analyticsexceptional query performancead hoc query performance up to 100x faster than a row
based RDBMS dramatically lower TCO by reducing performance overhead,
storage costs & maintenance requirements
Sybase Confidential
Project Summary
“We have a great relationship with Sybase. Our consultant really listened to what we needed, brought fresh ideas to the discussion, and solicited our feedback on all improvements. Throughout this process, we always knew the Professional Services team had our best interests at the forefront.”
“This project is definitely a success. We’ve been able to improve our user’s experience with queries as well as improve our operational efficiency. This helps us meet our stated goals and preserve our IT resources.”
Peter Brandt, Associate Director of Data Collections and Reporting, New South Wales Health
Sybase Confidential
Analyst Summary
"Sybase IQ is a columnar DBMS that is very, very fast for analytics. Sybase has been turning in very impressive revenue growth with Sybase IQ after they have positioned themselves correctly in their markets around analytics. Just last quarter and in all of 2007, Sybase has recorded their best revenue and profit growth in their history".
-- Donald Feinberg, 02/072008, presentation to a packed audience on "2007 Gartner MQ (for DW DBMS)" at the Gartner BI conference at Amsterdam
Sybase Confidential
Customer Summary
Strong, established, fast-growing customer base More than 2000 customer projects at more than 1200 sites
worldwide
Sybase Confidential