3 September 2014 VLDB14, Hangzhou, CHINA

OLTP-Bench Framework: An Extensible Testbed for Benchmarking Relational Databases

Djellel Eddine Difallah, Andy Pavlo, Carlo Curino, Philippe Cudré-Mauroux

How many researchers wasted time writing benchmarking infrastructure to run their

experiments?

Measuring Reproducibility in Computer Systems Research – Collberg et al. (20% success)

Experimental Research ProcessWe continuously re-invent the wheel.

DevelopmentCreate the prototype

Search for Benchmarks (ad-hoc workloads)

Search for Datasets (synthetic data)

Building a test toolkit

MeasureMonitor different resources

AnalysisPlot and make sense of the results

OLTP-Bench is an open-source “batteries-included” DBMS

benchmarking testbed tailored for OLTP/Web workloads

OLTP-Bench16 Workloads

Tested on 8 DBMSs

Rich metricsFine-grained Rate controlWorkload MixtureExtensible

OLTP-Bench Architecture● Tight and dynamic control on parallel load generation● Statistics gathering● SQL dialect handling

Parallel Connections vs Performance

Current Workloads and Benchmarks

Class Benchmark Application Domain

Transactional

AuctionMarkCH-benCHmarkSEATSTATPTPC-CVoterSmallBank

On-line AuctionsMixture of OLTP and OLAPOn-line Airline TicketingCaller Location AppOrder ProcessingTalent Show VotingBanking System

Web-Oriented

EpinionsLinkBenchTwitterWikipedia

Social NetworkingSocial NetworkingSocial NetworkingOn-line Encyclopedia

Feature Testing

ResourceStresserYCSBJPABSIBench

Isolated Resource StresserScalable Key-value StoreObject-Relational MappingTransactional Isolation

JDBC Compliant DBMSsMS SQL ServerOracleMySQLPostgresDB2NuoDBSQLiteApache DerbyMonetDB

Features showcase

Fine-Grained Rate ControlMySQL running Wikipedia workload at increasing throughput. Demonstrating tight control of transactional throughput imposed on the system (and saturation)

Simulating Evolving Skew

MySQL running Twitter with an evolving skew (alternating zipfian and uniform). Different skew imposes stress on different resources.

Multi-tenancyComparing 3 relational databases under multi-tenancy resource contention. (No DBMS is perfect)

Changing Mixture over timeMySQL running YCSB with a varying transaction mixture

(highlighting various resource bottlenecks)

Parallel Connections vs Performance

Future Work“We explicitly avoid to define benchmark rules, as they are (often) restrictive, arbitrary, biased, and not-future-proof”

- Djellel, Andy, Carlo, Philippe

The project received an NSF Grant!!

Possible directions:● Better tooling (e.g. real-time visualization)● Synchronized distributed clients● Support for NoSQL● A public repository for experiments and results

https://github.com/oltpbenchmark/

Q&A

External contributors and acknowledgements: Evan Jones, Barzan Mozfari, Dimitri Vorona, Ben Reilly, Yu Su, Adam Seering, Simon Krenger, Tommy Reilly, Mark Callaghan, Keving Grittner, David L. Day, Nik Lanham, Awajeet Arya, Flavio Sousa, Stein Petter Tronstad, Lance FAng, Rodrigo Felix de Almeida, Woonhak kang, Florian Funke, Ahmad Hassan, Ivo Jimenez,Mickaël Hoerdt, Dimokritos Stamatakis, Benjamin Reilly, Mark Callaghan, Zheng Da, Xi Chen, Jean de Lavarene, Ning Zhang, Erik Paulson, Alec Jindal, ADD YOUR NAME HERE