improving nosql database benchmarking - uni … · steffen friedrich | improving nosql database...
TRANSCRIPT
![Page 1: Improving NoSQL Database Benchmarking - uni … · Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3. Relational DBMS Benchmarking ~ 50 years database theory](https://reader030.vdocuments.net/reader030/viewer/2022021712/5b95156709d3f2d7438bdd78/html5/thumbnails/1.jpg)
ImprovingNoSQL Database Benchmarking
Lessons Learned
Steffen Friedrich
University of HamburgDepartment of Informatics
Databases and Information Systems
March 23, 2017 @ UIOP 2017, DKRZ Hamburg
![Page 2: Improving NoSQL Database Benchmarking - uni … · Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3. Relational DBMS Benchmarking ~ 50 years database theory](https://reader030.vdocuments.net/reader030/viewer/2022021712/5b95156709d3f2d7438bdd78/html5/thumbnails/2.jpg)
Part 1
RDBMS => TPC
VS
NoSQL => YCSB
![Page 3: Improving NoSQL Database Benchmarking - uni … · Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3. Relational DBMS Benchmarking ~ 50 years database theory](https://reader030.vdocuments.net/reader030/viewer/2022021712/5b95156709d3f2d7438bdd78/html5/thumbnails/3.jpg)
Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3
Relational DBMS Benchmarking
~ 50 years database theory
T PC * 1988
StandardBenchmarks
=> high degree of standardization greatly simplified the development of
RDBMS
Transaction Processing Performance Council
![Page 4: Improving NoSQL Database Benchmarking - uni … · Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3. Relational DBMS Benchmarking ~ 50 years database theory](https://reader030.vdocuments.net/reader030/viewer/2022021712/5b95156709d3f2d7438bdd78/html5/thumbnails/4.jpg)
Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 4
T PC Benchmarks: Multiple domain specific benchmarks:
• The TPC-C OLT P benchmark• workload consists of five transaction types simulating
activities of a wholesale supplier
• requires ACID transactions.
• only requirements specification => vendors may implement and run TPC-C => TPC consortium approve result reports
• Metrics: transactions per minute (tpmC), price / tpmC
• TPC-DI, TPC-DS, TPC-E, TPC-H, …
• Obsolete: TPC-A, TPC-B, TPC-W, ...
![Page 5: Improving NoSQL Database Benchmarking - uni … · Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3. Relational DBMS Benchmarking ~ 50 years database theory](https://reader030.vdocuments.net/reader030/viewer/2022021712/5b95156709d3f2d7438bdd78/html5/thumbnails/5.jpg)
Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 5
NoSQL Databases
> “NoSQL” term coined in 2009
> Interpretation: ,,Not Only SQL”
> Development driven by large web companies
> Main motivation: Scalability
?
Large user-generated data /
Request load
![Page 6: Improving NoSQL Database Benchmarking - uni … · Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3. Relational DBMS Benchmarking ~ 50 years database theory](https://reader030.vdocuments.net/reader030/viewer/2022021712/5b95156709d3f2d7438bdd78/html5/thumbnails/6.jpg)
Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 6
RDBMS VS NoSQL DB
Scale-Up(vertical scaling)
More RAM
More CPU
More HDD
Scale-Out(horizontal scaling)
Commodity hardware
Connected by network
Scalability
SpecializedDB hardware
![Page 7: Improving NoSQL Database Benchmarking - uni … · Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3. Relational DBMS Benchmarking ~ 50 years database theory](https://reader030.vdocuments.net/reader030/viewer/2022021712/5b95156709d3f2d7438bdd78/html5/thumbnails/7.jpg)
Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 7
RDBMS VS NoSQL DB
Relational data model
Schema free => implicit schemaExplicit schema
Different data models:› Key-Value,› Document,› Wide-Cloumn,› Graph
SQL query language Many query languages / APIs
No transactionsACID-Transactions& eventual consistency… …
normalization denormalization
![Page 8: Improving NoSQL Database Benchmarking - uni … · Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3. Relational DBMS Benchmarking ~ 50 years database theory](https://reader030.vdocuments.net/reader030/viewer/2022021712/5b95156709d3f2d7438bdd78/html5/thumbnails/8.jpg)
Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 8
RDBMS VS NoSQL DB
One Size
Fits All
Polyglot
PersistenceVS
Specialized Databases
for special requirements
![Page 9: Improving NoSQL Database Benchmarking - uni … · Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3. Relational DBMS Benchmarking ~ 50 years database theory](https://reader030.vdocuments.net/reader030/viewer/2022021712/5b95156709d3f2d7438bdd78/html5/thumbnails/9.jpg)
Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 9
More About NoSQL Databases?
Felix Gessert, Wolfram Wingerath, Steffen Friedrich & Norbert Ritter: "NoSQL database systems: a survey and decision guidance",Computer Science - Research and Development 1-13 (2016)
NoSQL Toolbox
NoSQL Decision tree
![Page 10: Improving NoSQL Database Benchmarking - uni … · Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3. Relational DBMS Benchmarking ~ 50 years database theory](https://reader030.vdocuments.net/reader030/viewer/2022021712/5b95156709d3f2d7438bdd78/html5/thumbnails/10.jpg)
Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 10
NoSQL Performance Evaluation ?
Cooper et al.: Benchmarking Cloud Serving Systems with YCSB, SoCC‘10, ACM, 2010
Heterogeneous NoSQL landscape
Yahoo Cloud Serving Benchmark !
https://github.com/brianfrankcooper/YCSB/wiki
De facto standardbenchmarking framework
=> User perspective on web app. performance=> Not only throughput => response times / latencies
![Page 11: Improving NoSQL Database Benchmarking - uni … · Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3. Relational DBMS Benchmarking ~ 50 years database theory](https://reader030.vdocuments.net/reader030/viewer/2022021712/5b95156709d3f2d7438bdd78/html5/thumbnails/11.jpg)
Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 11
Limited to the functionality all NoSQL systems have in common
› Key-Value interface of CRUD-operations
› No domain driven workloads
=> configurable mix of operations
Databases only have toimpliment the simple Database Interface Layer
› No Transactions› No Joins› No Complex Queries
![Page 12: Improving NoSQL Database Benchmarking - uni … · Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3. Relational DBMS Benchmarking ~ 50 years database theory](https://reader030.vdocuments.net/reader030/viewer/2022021712/5b95156709d3f2d7438bdd78/html5/thumbnails/12.jpg)
Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 12
The Coordinated Omission Problem“a conspiracy we’re all a part of”
Gil Tene, CTO @ Azul Systems:
How NOT to Measure Latency, QCon, 2013 - 2016infoq.com/presentations/latency-response-time
Part 1I
Steffen Friedrich, Wolfram Wingerath & Norbert Ritter:"Coordinated Omission in NoSQL Database Benchmarking",BTW 2017, Workshopband, p. 215-225, 2017
Quote
![Page 13: Improving NoSQL Database Benchmarking - uni … · Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3. Relational DBMS Benchmarking ~ 50 years database theory](https://reader030.vdocuments.net/reader030/viewer/2022021712/5b95156709d3f2d7438bdd78/html5/thumbnails/13.jpg)
Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 13
YCSBs load generation
_targetOpsTickNanos = (long) (1 000 000 000 / target)long overallStartTime = System.nanoTime();
while (_opsdone < _opcount) {long startTime = System.nanoTime();Status status = _db.read( table, key, fields, result );long endTime = System.nanoTime();
_measurements.measure("READ", (int)( (endTime - startTime) / 1000));
_opsdone++;
long deadline = overallStartTime + _opsdone * _targetOpsTickNanos;long now = System.nanoTime();while((now = System.nanoTime()) < deadline) {LockSupport.parkNanos( deadline – now );
}}
![Page 14: Improving NoSQL Database Benchmarking - uni … · Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3. Relational DBMS Benchmarking ~ 50 years database theory](https://reader030.vdocuments.net/reader030/viewer/2022021712/5b95156709d3f2d7438bdd78/html5/thumbnails/14.jpg)
Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 14
YCSBs load generation
_targetOpsTickNanos = (long) (1 000 000 000 / target)long overallStartTime = System.nanoTime();
while (_opsdone < _opcount) {long startTime = System.nanoTime();Status status = _db.read( table, key, fields, result );long endTime = System.nanoTime();
_measurements.measure("READ", (int)( (endTime - startTime) / 1000));
_opsdone++;
long deadline = overallStartTime + _opsdone * _targetOpsTickNanos;long now = System.nanoTime();while((now = System.nanoTime()) < deadline) {LockSupport.parkNanos( deadline – now );
}}
![Page 15: Improving NoSQL Database Benchmarking - uni … · Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3. Relational DBMS Benchmarking ~ 50 years database theory](https://reader030.vdocuments.net/reader030/viewer/2022021712/5b95156709d3f2d7438bdd78/html5/thumbnails/15.jpg)
Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 15
YCSBs load generation
_targetOpsTickNanos = (long) (1 000 000 000 / targetThroughput)long overallStartTime = System.nanoTime();
while (_opsdone < _opcount) {long startTime = System.nanoTime();Status status = _db.read( table, key, fields, result );long endTime = System.nanoTime();
_measurements.measure("READ", (int)( (endTime - startTime) / 1000));
_opsdone++;
long deadline = overallStartTime + _opsdone * _targetOpsTickNanos;long now = System.nanoTime();while((now = System.nanoTime()) < deadline) {LockSupport.parkNanos( deadline – now );
}}
![Page 16: Improving NoSQL Database Benchmarking - uni … · Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3. Relational DBMS Benchmarking ~ 50 years database theory](https://reader030.vdocuments.net/reader030/viewer/2022021712/5b95156709d3f2d7438bdd78/html5/thumbnails/16.jpg)
Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 16
YCSBs load generation
_targetOpsTickNanos = (long) (1 000 000 000 / targetThroughput)long overallStartTime = System.nanoTime();
while (_opsdone < _opcount) {long startTime = System.nanoTime();Status status = _db.read( table, key, fields, result );long endTime = System.nanoTime();
_measurements.measure("READ", (int)( (endTime - startTime) / 1000));
_opsdone++;
long deadline = overallStartTime + _opsdone * _targetOpsTickNanos;long now = System.nanoTime();while((now = System.nanoTime()) < deadline) {LockSupport.parkNanos( deadline – now );
}}
What if
latency >> _targetOpsTickNanos ?
=> now >> deadline ?
![Page 17: Improving NoSQL Database Benchmarking - uni … · Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3. Relational DBMS Benchmarking ~ 50 years database theory](https://reader030.vdocuments.net/reader030/viewer/2022021712/5b95156709d3f2d7438bdd78/html5/thumbnails/17.jpg)
The Coordinated Omission Problem
17Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned
Example
![Page 18: Improving NoSQL Database Benchmarking - uni … · Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3. Relational DBMS Benchmarking ~ 50 years database theory](https://reader030.vdocuments.net/reader030/viewer/2022021712/5b95156709d3f2d7438bdd78/html5/thumbnails/18.jpg)
The Coordinated Omission Problem
18Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned
Example
![Page 19: Improving NoSQL Database Benchmarking - uni … · Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3. Relational DBMS Benchmarking ~ 50 years database theory](https://reader030.vdocuments.net/reader030/viewer/2022021712/5b95156709d3f2d7438bdd78/html5/thumbnails/19.jpg)
The Coordinated Omission Problem
19Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned
Example
Database is able to influence the request rate !
=> coordinated omission of relevant measurement points
![Page 20: Improving NoSQL Database Benchmarking - uni … · Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3. Relational DBMS Benchmarking ~ 50 years database theory](https://reader030.vdocuments.net/reader030/viewer/2022021712/5b95156709d3f2d7438bdd78/html5/thumbnails/20.jpg)
The Coordinated Omission Problem
20Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned
Example
![Page 21: Improving NoSQL Database Benchmarking - uni … · Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3. Relational DBMS Benchmarking ~ 50 years database theory](https://reader030.vdocuments.net/reader030/viewer/2022021712/5b95156709d3f2d7438bdd78/html5/thumbnails/21.jpg)
The Coordinated Omission Problem
21Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned
Example
The Results:
![Page 22: Improving NoSQL Database Benchmarking - uni … · Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3. Relational DBMS Benchmarking ~ 50 years database theory](https://reader030.vdocuments.net/reader030/viewer/2022021712/5b95156709d3f2d7438bdd78/html5/thumbnails/22.jpg)
The Coordinated Omission Problem
22Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned
Example
The Results:
Do not just look at average latencies (+ StdDeviation), because latencies are not normally distributed!
![Page 23: Improving NoSQL Database Benchmarking - uni … · Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3. Relational DBMS Benchmarking ~ 50 years database theory](https://reader030.vdocuments.net/reader030/viewer/2022021712/5b95156709d3f2d7438bdd78/html5/thumbnails/23.jpg)
Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 23
Coordinated Omission Correctionsince YCSB Version 0.2.0 RC 1, June 2015
while (_opsdone < _opcount) {long startTime = System.nanoTime();Status status = _db.read( table, key, fields, result );long endTime = System.nanoTime();
_measurements.measure("INTENDED_READ", (int)( (endTime - _deadline) / 1000));
_opsdone++;
_deadline = overallStartTime + _opsdone * _targetOpsTickNanos;long now = System.nanoTime();while((now = System.nanoTime()) < deadline) {LockSupport.parkNanos( deadline – now );
}}
…
startTime = _deadline(computed after previous request)
=> intended measurement interval
![Page 24: Improving NoSQL Database Benchmarking - uni … · Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3. Relational DBMS Benchmarking ~ 50 years database theory](https://reader030.vdocuments.net/reader030/viewer/2022021712/5b95156709d3f2d7438bdd78/html5/thumbnails/24.jpg)
Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 24
Coordinated Omission Correctionsince YCSB Version 0.2.0 RC 1, June 2015
while (_opsdone < _opcount) {long startTime = System.nanoTime();Status status = _db.read( table, key, fields, result );long endTime = System.nanoTime();
_measurements.measure("INTENDED_READ", (int)( (endTime - _deadline) / 1000));
_opsdone++;
_deadline = overallStartTime + _opsdone * _targetOpsTickNanos;long now = System.nanoTime();while((now = System.nanoTime()) < deadline) {LockSupport.parkNanos( deadline – now );
}}
…
=> intended measurement interval
=> but still influenceable request rate !
startTime = _deadline(computed after previous request)
![Page 25: Improving NoSQL Database Benchmarking - uni … · Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3. Relational DBMS Benchmarking ~ 50 years database theory](https://reader030.vdocuments.net/reader030/viewer/2022021712/5b95156709d3f2d7438bdd78/html5/thumbnails/25.jpg)
Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 25
Closed VS. Open System Model
Closed System Model
Open System Model
for load generation
Schröder et al. Open Versus Closed: A Cautionary Tale, 2006
![Page 26: Improving NoSQL Database Benchmarking - uni … · Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3. Relational DBMS Benchmarking ~ 50 years database theory](https://reader030.vdocuments.net/reader030/viewer/2022021712/5b95156709d3f2d7438bdd78/html5/thumbnails/26.jpg)
Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 27
Scalable NoSQL-Benchmarkingnosqlmark.informatik.uni-hamburg.de
![Page 27: Improving NoSQL Database Benchmarking - uni … · Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3. Relational DBMS Benchmarking ~ 50 years database theory](https://reader030.vdocuments.net/reader030/viewer/2022021712/5b95156709d3f2d7438bdd78/html5/thumbnails/27.jpg)
Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 28
> built to implement our consistency measurement approach
> Scales YCSB compatible workloads to multiple benchmarking
nodes => Automatically aggregates results
> Compatible to the YCSB database interface layer
> Closed and Open System Model
Scalable NoSQL-Benchmarkingnosqlmark.informatik.uni-hamburg.de
![Page 28: Improving NoSQL Database Benchmarking - uni … · Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3. Relational DBMS Benchmarking ~ 50 years database theory](https://reader030.vdocuments.net/reader030/viewer/2022021712/5b95156709d3f2d7438bdd78/html5/thumbnails/28.jpg)
Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 29
Coordinated Omission Avoidancein NoSQLMark
implicit val ec = context.system.dispatchers.lookup("blocking-io-dispatcher")
case DoOperation => { val operation = workload.nextOperationval startTime = System.nanoTime
val future = Future {sendRequest(operation)
}future.onComplete {case Success(status) => { val endTime = System.nanoTimemeasurementActor ! Measure(operation.name, (endTime - startTime) / 1000)
}case Failure(ex) => {log.error(ex, "Error occured during operation {}", operation.name)
}}
Asynchronous loadgeneration !
…
![Page 29: Improving NoSQL Database Benchmarking - uni … · Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3. Relational DBMS Benchmarking ~ 50 years database theory](https://reader030.vdocuments.net/reader030/viewer/2022021712/5b95156709d3f2d7438bdd78/html5/thumbnails/29.jpg)
Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 31
Coordinated Omission Validation with
Originally developed to validate consitencymeasurement approaches
Lesson we have learned:Validate your tools!
Wingerath, Friedrich, Gesser, Ritter: Who Watches the Watchmen?On the Lack of Validation in NoSQL Benchmarking, BTW 2015
Single-node inconsistent key-value Store
github.com/steffenfriedrich/SickStore
![Page 30: Improving NoSQL Database Benchmarking - uni … · Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3. Relational DBMS Benchmarking ~ 50 years database theory](https://reader030.vdocuments.net/reader030/viewer/2022021712/5b95156709d3f2d7438bdd78/html5/thumbnails/30.jpg)
Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 32
Coordinated Omission Validation with
New Feature: Simulation of maximum throughput and database hiccups
1. Compute theoretical waiting time Ti of request i in the databse system
2. Calling client thread has to sleep for Ti
Single-node inconsistent key-value Store
![Page 31: Improving NoSQL Database Benchmarking - uni … · Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3. Relational DBMS Benchmarking ~ 50 years database theory](https://reader030.vdocuments.net/reader030/viewer/2022021712/5b95156709d3f2d7438bdd78/html5/thumbnails/31.jpg)
Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 33
Experimental Validation: SickStoreBenchmark: 90 000 ops, target = 1000 ops/sec,
SickStore: 1 second hiccup, max throughput = 1250 ops/sec, 80% ofmaxthroughput
YCSB NoSQLMark YCSB Intended
AVG.: 2 ms 29 ms 180 ms
![Page 32: Improving NoSQL Database Benchmarking - uni … · Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3. Relational DBMS Benchmarking ~ 50 years database theory](https://reader030.vdocuments.net/reader030/viewer/2022021712/5b95156709d3f2d7438bdd78/html5/thumbnails/32.jpg)
Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 34
Experimental Validation: SickStoreBenchmark: 90 000 ops, target = 1000 ops/sec,
SickStore: 1 second hiccup, max throughput = 1250 ops/sec,
YCSB NoSQLMark YCSB Intended
AVG.: 6 ms 29 ms 49ms
80% ofmaxthroughput
![Page 33: Improving NoSQL Database Benchmarking - uni … · Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3. Relational DBMS Benchmarking ~ 50 years database theory](https://reader030.vdocuments.net/reader030/viewer/2022021712/5b95156709d3f2d7438bdd78/html5/thumbnails/33.jpg)
Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 35
Experimental Validation: SickStoreBenchmark: 90 000 ops, target = 1000 ops/sec,
SickStore: 1 second hiccup, max throughput = 1250 ops/sec,
YCSB NoSQLMark YCSB Intended
AVG.: 19 ms 29 ms 44ms
YCSB NoSQLMark YCSB Intended
AVG.: 49 ms 29 ms 54ms
![Page 34: Improving NoSQL Database Benchmarking - uni … · Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3. Relational DBMS Benchmarking ~ 50 years database theory](https://reader030.vdocuments.net/reader030/viewer/2022021712/5b95156709d3f2d7438bdd78/html5/thumbnails/34.jpg)
Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 36
Experimental Validation: SickStoreDifferrent max throughputs
~ 70% load~ 10% load
~ 95% load~ 90% load
![Page 35: Improving NoSQL Database Benchmarking - uni … · Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3. Relational DBMS Benchmarking ~ 50 years database theory](https://reader030.vdocuments.net/reader030/viewer/2022021712/5b95156709d3f2d7438bdd78/html5/thumbnails/35.jpg)
Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 37
Elasticity Benchmark with Cassandra
• One Cassandra node loaded with 10 million records
• After 5 min add a second node => it starts serving after ~ 5 min=> roughly the time it takes latency to stabilize
• Run each experiment for max 15 min on a fresh cluster
Kuhlenkamp et al.: Benchmarking Scalability and Elasticity of Distributed Database Systems, VLDB, 2014
YCSB without intendedmeasurement interval
![Page 36: Improving NoSQL Database Benchmarking - uni … · Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3. Relational DBMS Benchmarking ~ 50 years database theory](https://reader030.vdocuments.net/reader030/viewer/2022021712/5b95156709d3f2d7438bdd78/html5/thumbnails/36.jpg)
Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 38
Elasticity Benchmark with CassandraTarget throughput = 10 000 ops /sec
![Page 37: Improving NoSQL Database Benchmarking - uni … · Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3. Relational DBMS Benchmarking ~ 50 years database theory](https://reader030.vdocuments.net/reader030/viewer/2022021712/5b95156709d3f2d7438bdd78/html5/thumbnails/37.jpg)
Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 39
Elasticity Benchmark with CassandraYCSB: 48 threads
![Page 38: Improving NoSQL Database Benchmarking - uni … · Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3. Relational DBMS Benchmarking ~ 50 years database theory](https://reader030.vdocuments.net/reader030/viewer/2022021712/5b95156709d3f2d7438bdd78/html5/thumbnails/38.jpg)
Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 40
Benchmarking is hard and your
latency values are probably lying to you !
> The coordinated omission problem can’t be corrected !
=> Wisely implement / choose your load generators system model !
> Do not just look at average latencies(+StdDeviation), because latencies are not normally distributed!
> Validate your tools!
![Page 39: Improving NoSQL Database Benchmarking - uni … · Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 3. Relational DBMS Benchmarking ~ 50 years database theory](https://reader030.vdocuments.net/reader030/viewer/2022021712/5b95156709d3f2d7438bdd78/html5/thumbnails/39.jpg)
Steffen Friedrich | Improving NoSQL Database Benchmarking: Lessons Learned 41
Further Improvement in NoSQLMark
More realistic distributions for request rate
• User requests => Poisson process => exponential inter-request/arrival times
• Many authors consider Perato or hyper-exponential distributed inter-arrival times
James F. Brady & Neil J. Gunther: How to Emulate Web Traffic Using Standard Load Testing Tools, CoRR, 2016
perfdynamics.blogspot.de/2010/05/load-testing-think-time-distributions.htmlNeil J. Gunther: Load Testing Think Time Distributions, blogpost, 2010
AlreadyImplementedIn