TPC BenchmarksCharles LevineMicrosoftclevine@microsoft.com

Western Institute of Computer ScienceStanford, CAAugust 6, 1999

OutlineIntroductionHistory of TPCTPC-A/B LegacyTPC-CTPC-H/RTPC Futures

Benchmarks: What and WhyWhat is a benchmark?Domain specificNo single metric possibleThe more general the benchmark, the less useful it is for anything in particular.A benchmark is a distillation of the essential attributes of a workloadDesirable attributesRelevant meaningful within the target domainUnderstandableGood metric(s) linear, orthogonal, monotonicScaleable applicable to a broad spectrum of hardware/architectureCoverage does not oversimplify the typical environmentAcceptance Vendors and Users embrace it

Benefits and LiabilitiesGood benchmarksDefine the playing fieldAccelerate progressEngineers do a great job once objective is measureable and repeatableSet the performance agendaMeasure release-to-release progressSet goals (e.g., 100,000 tpmC, < 10 $/tpmC) Something managers can understand (!)Benchmark abuse BenchmarketingBenchmark wars more $ on ads than development

Benchmarks have a LifetimeGood benchmarks drive industry and technology forward.At some point, all reasonable advances have been made.Benchmarks can become counter productive by encouraging artificial optimizations.So, even good benchmarks become obsolete over time.

OutlineIntroductionHistory of TPCTPC-A LegacyTPC-CTPC-H/RTPC Futures

What is the TPC?TPC = Transaction Processing Performance CouncilFounded in Aug/88 by Omri Serlin and 8 vendors.Membership of 40-45 for last several years Everybody whos anybody in software & hardwareDe facto industry standards body for OLTP performanceAdministered by: Shanley Public Relationsph: (408) 295-8894 650 N. Winchester Blvd, Suite 1fax: (408) 271-6648 San Jose, CA 95128email: shanley@tpc.orgMost TPC specs, info, results are on the web page: http: www.tpc.org

Two Seminal Events Leading to TPCAnon, et al, A Measure of Transaction Processing Power, Datamation, April fools day, 1985.Anon = Jim Gray (Dr. E. A. Anon)Sort: 1M 100 byte recordsMini-batch: copy 1000 recordsDebitCredit: simple ATM style transactionTandem TopGun BenchmarkDebitCredit212 tps on NonStop SQL in 1987 (!)Audited by Tom Sawyer of Codd and Date (A first)Full Disclosure of all aspects of tests (A first)Started the ET1/TP1 Benchmark wars of 87-89

TPC Milestones1989: TPC-A ~ industry standard for Debit Credit1990: TPC-B ~ database only version of TPC-A1992: TPC-C ~ more representative, balanced OLTP1994: TPC requires all results must be audited1995: TPC-D ~ complex decision support (query)1995: TPC-A/B declared obsolete by TPCNon-starters: TPC-E ~ Enterprise for the mainframersTPC-S ~ Server component of TPC-CBoth failed during final approval in 19961999: TPC-D replaced by TPC-H and TPC-R

TPC vs. SPECSPEC (System Performance Evaluation Cooperative)SPECMarksSPEC ships codeUnix centricCPU centricTPC ships specificationsEcumenicalDatabase/System/TP centricPrice/PerformanceThe TPC and SPEC happily coexistThere is plenty of room for both

OutlineIntroductionHistory of TPCTPC-A/B LegacyTPC-CTPC-H/RTPC Futures

TPC-A LegacyFirst results in 1990: 38.2 tpsA, 29.2K$/tpsA (HP)Last results in 1994: 3700 tpsA, 4.8 K$/tpsA (DEC)WOW! 100x on performance and 6x on price in five years!!!TPC cut its teeth on TPC-A/B; became functioning, representative bodyLearned a lot of lessons:If benchmark is not meaningful, it doesnt matter how many numbers or how easy to run (TPC-B).How to resolve ambiguities in specHow to police complianceRules of engagement

TPC-A Established OLTP Playing FieldTPC-A criticized for being irrelevant, unrepresentative, misleadingBut, truth is that TPC-A drove performance, drove price/performance, and forced everyone to clean up their products to be competitive.Trend forced industry toward one price/performance, regardless of size.Became means to achieve legitimacy in OLTP for some.

OutlineIntroductionHistory of TPCTPC-A/B LegacyTPC-CTPC-DTPC Futures

TPC-C OverviewModerately complex OLTPThe result of 2+ years of development by the TPCApplication models a wholesale supplier managing orders.Order-entry provides a conceptual model for the benchmark; underlying components are typical of any OLTP system.Workload consists of five transaction types.Users and database scale linearly with throughput.Spec defines full-screen end-user interface.Metrics are new-order txn rate (tpmC) and price/performance ($/tpmC)Specification was approved July 23, 1992.

TPC-Cs Five TransactionsOLTP transactions:New-order: enter a new order from a customerPayment: update customer balance to reflect a paymentDelivery: deliver orders (done as a batch transaction)Order-status: retrieve status of customers most recent orderStock-level: monitor warehouse inventoryTransactions operate against a database of nine tables.Transactions do update, insert, delete, and abort; primary and secondary key access.Response time requirement: 90% of each type of transaction must have a response time 5 seconds, except stock-level which is 20 seconds.

TPC-C Database SchemaItem100K (fixed)

TPC-C Workflow21Select txn from menu:1. New-Order 45%2. Payment 43%3. Order-Status4%4. Delivery 4%5. Stock-Level 4%Input screenOutput screenMeasure menu Response TimeMeasure txn Response TimeKeying timeThink time3Go back to 1Cycle Time Decomposition(typical values, in seconds, for weighted average txn)

Menu = 0.3

Keying = 9.6

Txn RT = 2.1

Think = 11.4

Average cycle time = 23.4

Data SkewNURand - Non Uniform RandomNURand(A,x,y) = (((random(0,A) | random(x,y)) + C) % (y-x+1)) + xCustomer Last Name: NURand(255, 0, 999)Customer ID: NURand(1023, 1, 3000)Item ID: NURand(8191, 1, 100000)bitwise OR of two random valuesskews distribution toward values with more bits on 75% chance that a given bit is one (1 - * )skewed data pattern repeats with period of smaller random number

NURand Distribution

ACID TestsTPC-C requires transactions be ACID.Tests included to demonstrate ACID properties met.AtomicityVerify that all changes within a transaction commit or abort.ConsistencyIsolationANSI Repeatable reads for all but Stock-Level transactions.Committed reads for Stock-Level.DurabilityMust demonstrate recovery fromLoss of powerLoss of memoryLoss of media (e.g., disk crash)

TransparencyTPC-C requires that all data partitioning be fully transparent to the application code. (See TPC-C Clause 1.6)Both horizontal and vertical partitioning is allowedAll partitioning must be hidden from the applicationMost DBMSs do this today for single-node horizontal partitioning. Much harder: multiple-node transparency.For example, in a two-node cluster:1-100Warehouses:Node A select * from warehouse where W_ID = 150Node B select * from warehouse where W_ID = 77Any DML operation must beable to operate against the entire database, regardless of physical location.100-200

Transparency (cont.)How does transparency affect TPC-C?Payment txn: 15% of Customer table records are non-local to the home warehouse.New-order txn: 1% of Stock table records are non-local to the home warehouse.In a distributed cluster, the cross warehouse traffic causes cross node traffic and either 2 phase commit, distributed lock management, or both.For example, with distributed txns:Number of nodes% Network Txns1025.537.3n 10.9

TPC-C Rules of Thumb 1.2 tpmC per User/terminal (maximum)10 terminals per warehouse (fixed)65-70 MB/tpmC priced disk capacity (minimum) ~ 0.5 physical IOs/sec/tpmC (typical)100-700 KB main memory/tpmC (how much $ do you have?)So use rules of thumb to size 5000 tpmC system:How many terminals?How many warehouses?How much memory?How much disk capacity?How many spindles?

Typical TPC-C Configuration (Conceptual)Response Timemeasured hereDatabaseServer...ClientC/SLANTerm.LANPresentation ServicesDatabase FunctionsEmulated User LoadHardwareRTE, e.g.:Performix,LoadRunner,or proprietary

SoftwareTPC-C application +Txn Monitor and/ordatabase RPC librarye.g., Tuxedo, ODBCTPC-C application (stored procedures) + Database enginee.g., SQL ServerDriver System

Competitive TPC-C Configuration 19965677 tpmC; $135/tpmC; 5-yr COO= 770.2 K$2 GB memory, 91 4-GB disks (381 GB total)4xPent 166 MHz5000 users

Competitive TPC-C Configuration Today40,013 tpmC; $18.86/tpmC; 5-yr COO= 754.7 K$4 GB memory, 252 9-GB disks & 225 4-GB disks (5.1 TB total)8xPentium III Xeon 550MHz32,400 users

The Complete Guide to TPC-CIn the spirit of The Compleat Works of Wllm Shkspr (Abridged)The Complete Guide to TPC-C:First, do several years of prep work. Next,Install OSInstall and configure databaseBuild TPC-C databaseInstall and configure TPC-C applicationInstall and configure RTERun benchmarkAnalyze resultsPublish

Typical elapsed time: 2 6 monthsThe Challenge: Do it all in the next 30 minutes!

TPC-C Demo ConfigurationResponse Timemeasured hereDB Server...C/SLANPresentation ServicesDatabase FunctionsEmulated User LoadDriver SystemClientCOM+RemoteTerminalEmulator(RTE)COMPONENTODBC APPUI APPODBCSQL ServerWeb Server

TPC-C Current Results - 1996Best Performance is 30,390 tpmC @ $305/tpmC (Digital)Best Price/Perf. is 6,185 tpmC @ $111/tpmC (Compaq)CompaqDigitalHPIBMSun$100/tpmC not yet. Soon!

TPC-C Current ResultsBest Performance is 115,395 tpmC @ $105/tpmC (Sun)Best Price/Perf. is 20,195 tpmC @ $15/tpmC (Compaq)$10/tpmC not yet. Soon!

Chart1

27.9823.178127.5327.2424.83

28.3426.8725.7131.677624.96

26.3156.6725.2105.6322.19

49.9387.7188.3723.73

23.03128.9120.44

21.7169.9619.93

22.5122.4420.87

15.1122.16

15.5123.18

14.62

18.84

18.88

37.07

49.88

79.43

18.86

18.86

139.49

Sun

IBM

Compaq

Sequent

HP

Unisys

Compaq

HP

IBM

Sequent

Sun

Unisys

tpcc_results

CompanySystemSpec. RevisiontpmC$/tpmCTotal Sys. CostCurrencyDatabase SoftwareOperating SystemTP MonitorServer CPU Type# Server CPU'sCluster# Front EndsDate SubmittedAvailability Date

AcerAcerAltos 19000Pro4 (c/s)3.311,07227.25301,663US $Microsoft SQL Server Enterprise Edition 6.5Microsoft Windows NT Enterprise Edition 4.0Tuxedo 6.3 CFSIntel PentiumPro 200 MHz4N52/16/982/16/98

AcerAcerAltos 21000 c/s3.423,23616.66387,010US $Microsoft SQL Server Enterprise Edition 7.0Microsoft Windows NT Enterprise Edition 4.0BEA Tuxedo 6.4 CFSIntel Pentium III Xeon 500MHz4N35/17/995/17/99

CompaqProLiant 3000 6/450-512 1P3.46,29027.98176,042US $Microsoft SQL Server Enterprise Edition 7.0Microsoft Windows NT 4.0Compaq Non-Stop Tuxedo CSIntel Pentium II 450 MHz1N111/23/9812/26/98

CompaqProLiant 3000-6/500-1P3.47,08528.34200,829US $Microsoft SQL Server Enterprise Edition 7.0Microsoft Windows NT Enterprise Edition 4.0Compaq Non-Stop Tuxedo CSIntel Pentium III Xeon 500MHz1N25/7/995/6/99

CompaqProLiant 3000-6/450-5123.410,17926.31267,824US $Microsoft SQL Server Enterprise Edition 7.0Microsoft Windows NT Enterprise Edition 4.0NonStop Tuxedo CS for NT ServerIntel Pentium II 450 MHz2N211/3/9812/26/98

CompaqAlphaServer DS203.411,61649.93580,011US $Sybase Adaptive Server Enterprise 11.9.3Digital UNIX V4.0EApplication Optimizer WNT ServerDigital Alphachip 21264 500MHz2N42/1/997/1/99

CompaqProLiant 3000-6/500-1S3.412,03423.03277,220US $Microsoft SQL Server Enterprise Edition 7.0Microsoft Windows NT Enterprise Edition 4.0Compaq NonStop Tuxedo 1.3Intel Pentium III 500MHz2N23/23/993/22/99

CompaqProLiant 5500-6/400 c/s3.317,71621.71384,727US $Microsoft SQL Server Enterprise Edition 7.0Microsoft Windows NT 4.0TuxedoIntel Pentium II Xeon 400 MHz4N39/11/9812/26/98

CompaqProLiant 7000 6/400-1M-4P3.419,72522.5443,951US $Microsoft SQL Server Enterprise Edition 7.0Microsoft Windows NT Enterprise Edition 4.0Compaq Non-Stop Tuxedo CSIntel Pentium II Xeon 400 MHz4N412/11/986/10/99

CompaqProLiant 5500-6/5003.420,19615.11305,181US $Microsoft SQL Server Enterprise Edition 7.0Microsoft Windows NT Enterprise Edition 4.0Compaq Non-Stop Tuxedo CSIntel Pentium II Xeon 500MHz4N35/7/995/6/99

CompaqProLiant 5500R-6/5003.420,19615.51313,351US $Microsoft SQL Server Enterprise Edition 7.0Microsoft Windows NT Enterprise Edition 4.0Compaq Non-Stop Tuxedo CSIntel Pentium III Xeon 500MHz4N35/7/995/6/99

CompaqProLiant 5500-500-2M3.422,05714.62322,537US $Microsoft SQL Server Enterprise Edition 7.0Microsoft Windows NT Enterprise Edition 4.0Microsoft COM+Intel Pentium III Xeon 500MHz4N37/12/998/15/99

CompaqProLiant 7000-6/450-2S3.422,47918.84423,542US $Microsoft SQL Server Enterprise Edition 7.0Microsoft Windows NT Enterprise Edition 4.0NonStop Tuxedo 1.3Intel Pentium II Xeon 450MHz4N42/10/993/1/99

CompaqProLiant 7000-6/5003.425,06518.88473,346US $Microsoft SQL Server Enterprise Edition 7.0Microsoft Windows NT Enterprise Edition 4.0Compaq Non-Stop Tuxedo CSIntel Pentium III Xeon 500MHz4N56/8/998/16/99

CompaqAlphaServer ES403.425,90337.07960,104US $Sybase Adaptive Server Enterprise 11.9.3Compaq Tru64 UNIX V4.OFApplication Optimizer WNT ServerDigital Alphachip 21264 500MHz4N57/9/9911/24/99

CompaqProLiant 6500-PDC/010003.433,93649.881,692,980US $Oracle 8 Parallel Server v 8.05Microsoft Windows NT 4.0Digital TP Web Server-DB V1.1Intel Pentium II Xeon 400 MHz16Y412/23/9812/22/98

CompaqCompaq AlphaServer GS140 6/5753.437,54079.432,981,728US $Sybase SQL Server 11.9.3Digital UNIX V4.0EApplication Optimizer WNT ServerAlphachip 21264 575 MHz8N910/7/981/10/99

CompaqProLiant 8500-550-2M3.440,00118.86754,744US $Microsoft SQL Server Enterprise Edition 7.0Microsoft Windows NT Enterprise Edition 4.0Microsoft COM+Intel Pentium III Xeon 550 MHz8N57/12/9912/31/99

CompaqProLiant 8000-550-2M3.440,01318.86754,744US $Microsoft SQL Server Enterprise Edition 7.0Microsoft Windows NT Enterprise Edition 4.0Microsoft COM+Intel Pentium III Xeon 550 MHz8N57/12/9912/31/99

CompaqAlphaServer 8400 8Node 96CPU Cluster (c/s)3.3102,542139.4914,303,550US $Oracle Oracle8 Enterprise Edition 8.0Digital Digital UNIX 4.0DDigital TP Web Server-DB V1.1Digital DECchip 21164 612 MHz96Y245/5/989/15/98

DellPowerEdge 63503.423,18816.29377,795US $Microsoft SQL Server Enterprise Edition 7.0Microsoft Windows NT Enterprise Edition 4.0BEA Tuxedo 6.4Intel Pentium III Xeon 500MHz4N35/28/999/1/99

DGAViiON AV 87003.423,07127.96644,899US $Microsoft SQL Server Enterprise Edition 7.0Microsoft Windows NT Enterprise Edition 4.0BEA Tuxedo 6.4 CFSIntel Pentium II 450 MHz4N41/22/993/25/99

Fujitsu/ICLGRANPOWER 5000 Model 6803.317,05736.24618,090US $Fujitsu/ICL SymfoWARE Server 1.1Microsoft Windows NT Enterprise Edition 4.0BEA Tuxedo 6.4 CFSIntel Pentium II Xeon 400 MHz4N89/22/983/1/99

Fujitsu/ICLGRANPOWER 5000 Model 6803.425,44128.57726,817US $Fujitsu/ICL SymfoWARE Server 1.1Microsoft Windows NT Enterprise Edition 4.0BEA Tuxedo 6.4 CFSIntel Pentium III Xeon 500MHz4N67/19/991/16/00

Fujitsu/ICLGP7000F Model 600 c/s3.433,11483.742,772,999US $Fujitsu/ICL SymfoWARE Server for VLM 2.0Sun Solaris 2.6BEA Tuxedo 6.4 CFSSPARC64 GP 300MHz8N66/24/9912/23/99

HPNetServer LH4r c/s3.419,05023.1440,047US $Microsoft SQL Server Enterprise Edition 7.0Microsoft Windows NT Enterprise Edition 4.0BEA TuxedoIntel Xeon 450 MHz4N31/15/996/15/99

HPNetServer LXr 80003.423,14326.87621,784US $Microsoft SQL Server Enterprise Edition 7.0Microsoft Windows NT 4.0Tuxedo 6.4 CFSIntel Xeon 450 MHz4N31/5/992/1/99

HP9000 Model N4000 Enterprise Server3.449,30856.672,794,055US $Sybase Adaptive Server Enterprise 11.9.3HP HP-UX 11.00BEA Tuxedo 6.4PA-RISC 8500 440MHz8N144/13/997/31/99

HP9000 Model V2500 Enterprise Server3.492,83387.718,142,782US $Oracle8i 8.1.5.1.1HP HP-UX 11.00BEA Tuxedo 6.4PA-RISC 8500 440MHz32N243/2/996/30/99

IBMRISC System/6000 Enterprise Server H70 c/s3.417,134781,343,526US $Oracle v8.1.5IBM AIX 4.3.2IBM TXSeries 4.2 for AIXIBM RS64-II 340 Mhz4N85/28/9911/19/99

IBMNetfinity 7000 M103.422,46025.7577,117US $Microsoft SQL Server Enterprise Edition 7.0Microsoft Windows NT Enterprise Edition 4.0IBM TXSeries 4.2 for NTIntel Pentium II Xeon 450MHz4N41/29/991/19/99

IBMNetfinity 7000 M103.423,64025.2595,772US $Microsoft SQL Server Enterprise Edition 7.0Microsoft Windows NT Enterprise Edition 4.0TX Series 4.2Intel Pentium II Xeon 500MHz4N43/25/996/30/99

IBMRS/6000 Enterprise Server S703.334,14088.373,016,996US $Oracle Oracle8 Enterprise Edition 8.0IBM AIX 4.3.1IBM TXSeries 4.2 for AIXIBM PowerPC RS604-2 262 MHz12N158/11/981/21/99

IBMAS/400e Server Model S40 22083.343,170128.915,564,844US $IBM DB2 for AS/400 V4R3IBM OS/400 V4R3CICS for AS/400 V4R3IBM PowerPC AS A50 262 MHz12N979/1/989/11/98

IBMAS/400e Model 740 Feature 2070 c/s3.443,41969.963,037,802US $IBM DB2 for AS/400 V4R3IBM OS/400 V4R3BEA Tuxedo 6.4IBM PowerPC AS 262MHz12N66/7/996/1/99

IBMRS 6000 Enterprise Server S703.4110,434122.4413,521,883US $Oracle Oracle8 V8.0.5IBM AIX 4.3.2IBM TXSeries 4.2 for AIXIBM RS64-II 262 MHz60Y606/30/996/30/99

NECExpress5800 HX4500 (c/s)3.318,32333.55614,908US $Microsoft SQL Server Enterprise Edition 7.0Microsoft Windows NT Enterprise Edition 4.0Tuxedo 6.3 CFSIntel Pentium II Xeon 400 MHz4N58/5/9812/29/98

NECExpress5800 HV86003.450,20894.054,723,164US $Oracle8i Ent. Edition 8.1.6.0Microsoft Windows NT Enterprise Edition 4.0BEA Tuxedo 6.4 CFSIntel Pentium III Xeon 500MHz32Y86/29/9911/30/99

SequentNUMACenter 20003.448,793127.536,222,683US $Oracle Oracle8 Enterprise Edition 8.0.4Sequent DYNIX/ptx 4.4BEA Tuxedo 6.4 CFSIntel Xeon 405 MHz32N410/13/983/15/99

SequentNUMACenter 20003.493,901131.6712,363,684US $Oracle Oracle8 Enterprise Edition 8.0.4DYNIX/ptx 4.4BEA Tuxedo 6.3Intel Xeon 405 MHz64N812/18/986/15/99

SNIPrimergy 870 with 7 Primergy 4603.318,52936.9683,682US $Microsoft SQL Server Enterprise Edition 7.0Microsoft Windows NT 4.0Open UTMIntel Pentium II Xeon 400 MHz4N79/28/9812/29/98

SNIPrimergy 870-403.422,34925.84577,459US $Microsoft SQL Server Enterprise Edition 7.0Microsoft Windows NT Enterprise Edition 4.0BEA Tuxedo 6.4 CFSIntel Pentium II 450 MHz4N412/23/982/28/99

SNIPrimergy 870-403.423,57023.41551,727US $Microsoft SQL Server Enterprise Edition 7.0Microsoft Windows NT Enterprise Edition 4.0BEA Tuxedo 6.4 CFSIntel Pentium III Xeon 500MHz4N43/17/994/26/99

SunEnterprise 450 Server3.417,67527.24481,381US $Sybase Adaptive Server Enterprise 11.5Sun Solaris 2.6BEA Tuxedo CFS 6.3Ultra SPARC II 400MHz4N42/2/997/31/99

SunEnterprise 6500 Server3.453,050764,031,901US $Sybase Adaptive Server Enterprise 11.9.3Sun SolarisBEA Tuxedo 6.3Sun UltraSPARC 336 Mhz24N1610/27/981/31/99

SunStarfire Enterprise 100003.4115,396105.6312,189,298US $Oracle 8i v.8.1.5.1Sun Solaris 7BEA Tuxedo 6.3Ultra SPARC II 400MHz64N323/24/998/22/99

UnisysAquanta QR/2 Server3.418,34324.83455,520US $Microsoft SQL Server Enterprise Edition 7.0Microsoft Windows NT Enterprise Edition 4.0Tuxedo 6.3 CFSIntel Pentium II Xeon 400 MHz4N311/9/9812/29/98

UnisysAquanta QS/2 Server3.418,70824.96466,883US $Microsoft SQL Server Enterprise Edition 7.0Microsoft Windows NT Enterprise Edition 4.0Tuxedo 6.3 CFSIntel Pentium II Xeon 400 MHz4N311/11/9812/29/98

UnisysAquanta QR/2V Server3.419,11822.19424,297US $Microsoft SQL Server Enterprise Edition 7.0Microsoft Windows NT Enterprise Edition 4.0Tuxedo 6.3 CFSIntel Pentium II Xeon 400 MHz4N312/4/9812/29/98

UnisysAquanta QR/2 Server3.421,35523.73506,676US $Microsoft SQL Server Enterprise Edition 7.0Microsoft Windows NT Enterprise Edition 4.0Tuxedo 6.3 CFSIntel Xeon 450 MHz4N31/5/991/5/99

UnisysAquanta ES2043R Server3.423,19020.44474,066US $Microsoft SQL Server Enterprise Edition 7.0Microsoft Windows NT Enterprise Edition 4.0BEA Tuxedo CFS 6.3Intel Pentium III Xeon 500MHz4N35/11/995/11/99

UnisysAquanta ES2043 Server3.423,19019.93462,101US $Microsoft SQL Server Enterprise Edition 7.0Microsoft Windows NT Enterprise Edition 4.0BEA Tuxedo CFS 6.3Intel Pentium III Xeon 500MHz4N35/7/995/7/99

UnisysAquanta ES2045 Server3.423,85220.87497,688US $Microsoft SQL Server Enterprise Edition 7.0Microsoft Windows NT Enterprise Edition 4.0BEA Tuxedo CFS 6.3Intel Pentium III Xeon 500MHz4N34/1/993/31/99

UnisysAquanta ES50453.424,32922.16539,131US $Microsoft SQL Server Enterprise Edition 7.0Microsoft Windows NT Enterprise Edition 4.0BEA Tuxedo CFS 6.3Intel Pentium III Xeon 500MHz4N33/17/993/17/99

UnisysAquanta ES2085R Server3.437,75723.18875,180US $Microsoft SQL Server Enterprise Edition 7.0Microsoft Windows NT Enterprise Edition 4.0BEA Tuxedo CFS 6.3Intel Pentium III Xeon 550 MHz8N46/22/999/30/99

tpcc_results

Sun

IBM

Compaq

Sequent

HP

Unisys

Compaq

HP

IBM

Sequent

Sun

Unisys

TPC-C SummaryBalanced, representative OLTP mixFive transaction typesDatabase intensive; substantial IO and cache loadScaleable workloadComplex data: data attributes, size, skewRequires Transparency and ACIDFull screen presentation servicesDe facto standard for OLTP performance

Preview of TPC-C rev 4.0Rev 4.0 is major revision. Previous results will not be comparable; dropped from result list after six months.Make txns heavier, so fewer users compared to rev 3.Add referential integrity.Adjust R/W mix to have more read, less write.Reduce response time limits (e.g., 2 sec 90th %-tile vs 5 sec)TVRand Time Varying Random causes workload activity to vary across database

OutlineIntroductionHistory of TPCTPC-A/B LegacyTPC-CTPC-H/RTPC Futures

TPC-H/R OverviewComplex Decision Support workloadOriginally released as TPC-Dthe result of 5 years of development by the TPCBenchmark models ad hoc queries (TPC-H) or reporting (TPC-R)extract database with concurrent updatesmulti-user environmentWorkload consists of 22 queries and 2 update streamsSQL as written in specDatabase is quantized into fixed sizes (e.g., 1, 10, 30, GB)Metrics are Composite Queries-per-Hour (QphH or QphR), and Price/Performance ($/QphH or $/QphR)TPC-D specification was approved April 5, 1995 TPC-H/R specifications were approved April, 1999

TPC-H/R SchemaCustomerSF*150KLineItemSF*6000KOrderSF*1500KSupplierSF*10KNation25Region5PartSuppSF*800KPartSF*200KLegend: Arrows point in the direction of one-to-many relationships. The value below each table name is its cardinality. SF is the Scale Factor.

TPC-H/R Database Scaling and LoadDatabase size is determined from fixed Scale Factors (SF):1, 10, 30, 100, 300, 1000, 3000, 10000 (note that 3 is missing, not a typo)These correspond to the nominal database size in GB. (i.e., SF 10 is approx. 10 GB, not including indexes and temp tables.)Indices and temporary tables can significantly increase the total disk capacity. (3-5x is typical)Database is generated by DBGENDBGEN is a C program which is part of the TPC-H/R specsUse of DBGEN is strongly recommended.TPC-H/R database contents must be exact.Database Load time must be reportedIncludes time to create indexes and update statistics.Not included in primary metrics.

How are TPC-H and TPC-R Different?PartitioningTPC-H: only on primary keys, foreign keys, and date columns; only using simple key breaksTPC-R: unrestricted for horizontal partitioningVertical partitioning is not allowedIndexesTPC-H: only on primary keys, foreign keys, and date columns; cannot span multiple tablesTPC-R: unrestrictedAuxiliary StructuresWhat? materialized views, summary tables, join indexesTPC-H: not allowedTPC-R: allowed

TPC-H/R Query Set22 queries written in SQL92 to implement business questions.Queries are pseudo ad hoc:Substitution parameters are replaced with constants by QGENQGEN replaces substitution parameters with random valuesNo host variablesNo static SQLQueries cannot be modified -- SQL as writtenThere are some minor exceptions.All variants must be approved in advance by the TPC

TPC-H/R Update StreamsUpdate 0.1% of data per query streamAbout as long as a medium sized TPC-H/R queryImplementation of updates is left to sponsor, except:ACID properties must be maintainedUpdate Function 1 (RF1)Insert new rows into ORDER and LINEITEM tables equal to 0.1% of table sizeUpdate Function 2 (RF2)Delete rows from ORDER and LINEITEM tables equal to 0.1% of table size

TPC-H/R ExecutionDatabase BuildTimed and reported, but not a primary metric

Power TestQueries submitted in a single stream (i.e., no concurrency)Sequence: Proceed directly to Power TestProceed directly to Throughput Test

TPC-H/R Execution (cont.)Throughput TestMultiple concurrent query streamsNumber of Streams (S) is determined by Scale Factor (SF)e.g.: SF=1 S=2; SF=100 S=5; SF=1000 S=7Single update stream Sequence:

TPC-H/R Secondary MetricsPower MetricGeometric queries per hour times SFThroughput MetricLinear queries per hour times SF

TPC-R/H Primary MetricsComposite Query-Per-Hour Rating (QphH or QphR)The Power and Throughput metrics are combined to get the composite queries per hour.Reported metrics are:Composite: QphH@SizePrice/Performance: $/QphH@SizeAvailability DateComparability:Results within a size category (SF) are comparable.Comparisons among different size databases are strongly discouraged.

TPC-H/R ResultsNo TPC-R results yet.One TPC-H result:Sun Enterprise 4500 (Informix), 1280 QphH@100GB, 816 $/QphH@100GB, available 11/15/99Too early to know how TPC-H and TPC-R will fareIn general, hardware vendors seem to be more interested in TPC-H

OutlineIntroductionHistory of TPCTPC-A/BTPC-CTPC-H/RTPC Futures

Next TPC Benchmark: TPC-WTPC-W (Web) is a transactional web benchmark.TPC-W models a controlled Internet Commerce environment that simulates the activities of a business oriented web server. The application portrayed by the benchmark is a Retail Store on the Internet with a customer browse-and-order scenario.TPC-W measures how fast an E-commerce system completes various E-commerce-type transactions

TPC-W CharacteristicsTPC-W features: The simultaneous execution of multiple transaction types that span a breadth of complexity. On-line transaction execution modes. Databases consisting of many tables with a wide variety of sizes, attributes, and relationship. Multiple on-line browser sessions. Secure browser interaction for confidential data. On-line secure payment authorization to an external server. Consistent web object update. Transaction integrity (ACID properties). Contention on data access and update. 24x7 operations requirement. Three year total cost of ownership pricing model.

TPC-W MetricsThere are three workloads in the benchmark, representing different customer environments. Primarily shopping (WIPS). Representing typical browsing, searching and ordering activities of on-line shopping. Browsing (WIPSB). Representing browsing activities with dynamic web page generation and searching activities. Web-based Ordering (WIPSO). Representing intranet and business to business secure web activities. Primary metrics are: WIPS rate (WIPS), price/performance ($/WIPS), and the availability date of the priced configuration.

TPC-W Public ReviewTPC-W specification is currently available for public review on TPC web site.Approved standard likely in Q1/2000

Reference MaterialJim Gray, The Benchmark Handbook for Database and Transaction Processing Systems, Morgan Kaufmann, San Mateo, CA, 1991.Raj Jain, The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling, John Wiley & Sons, New York, 1991.William Highleyman, Performance Analysis of Transaction Processing Systems, Prentice Hall, Englewood Cliffs, NJ, 1988.TPC Web site: www.tpc.orgIDEAS web site: www.ideasinternational.com

The End

Background Material on TPC-A/B

TPC-A OverviewTransaction is simple bank account debit/creditDatabase scales with throughputTransaction submitted from terminalRead 100 bytes including Aid, Tid, Bid, Delta from terminal (see Clause 1.3) BEGIN TRANSACTION Update Account where Account_ID = Aid: Read Account_Balance from Account Set Account_Balance = Account_Balance + Delta Write Account_Balance to Account Write to History: Aid, Tid, Bid, Delta, Time_stamp Update Teller where Teller_ID = Tid: Set Teller_Balance = Teller_Balance + Delta Write Teller_Balance to Teller Update Branch where Branch_ID = Bid: Set Branch_Balance = Branch_Balance + Delta Write Branch_Balance to Branch COMMIT TRANSACTION Write 200 bytes including Aid, Tid, Bid, Delta, Account_Balance to terminal TPC-A Transaction

TPC-A Database Schema10 Terminals per Branch row10 second cycle time per terminal1 transaction/second per Branch row

TPC-A TransactionWorkload is vertically aligned with BranchMakes scaling easyBut not very realistic15% of accounts non-localProduces cross database activityWhats good about TPC-A?Easy to understandEasy to measured Stresses high transaction rate, lots of physical IOWhats bad about TPC-A?Too simplistic! Lends itself to unrealistic optimizations

TPC-A Design RationaleBranch & Tellerin cache, hotspot on branchAccounttoo big to cache requires disk accessHistorysequential inserthotspot at end90-day capacity ensures reasonable ratio of disk to cpu

RTE SUTRTE - Remote Terminal EmulatorEmulates real user behaviorSubmits txns to SUT, measures RTTransaction rate includes think timeMany, many users (10 x tpsA)SUT - System Under TestAll components except for terminalModel of system:

TPC-A MetrictpsA = transactions per second, average rate over 15+ minute interval, at which 90% of txns get

TPC-A PricePrice5 year Cost of Ownership: hardware, software, maintenanceDoes not include development, comm lines, operators, power, cooling, etc.Strict pricing model one of TPCs big contributionsList pricesSystem must be orderable & commercially availableCommitted ship date

Differences between TPC-A and TPC-BTPC-B is database only portion of TPC-ANo terminalsNo think timesTPC-B reduces history capacity to 30 daysLess disk in priced configurationTPC-B was easier to configure and run, BUTEven though TPC-B was more popular with vendors, it did not have much credibility with customers.

TPC LoopholesPricingPackage pricingPrice does not include cost of five star wizards needed to get optimal performance, so performance is not what a customer could get.Client/ServerOffload presentation services to cheap clients, but report performance of serverBenchmark specialsDiscrete transactionsCustom transaction monitorsHand coded presentation services