It’s Not You...Ending a 15 year relationship with RRD

OpenNMS User Conference EuropeApril 11, 2014

Eric Evanseevans@opennms.org

@jericevans

Five Stages of Grief

1. Denial2. Anger3. Bargaining4. Depression5. Acceptance

RRDTool

● Round robin database● First released 1999● Time-series storage● File-based● Constant-size● Automatic, amortized aggregation

Graph All The Things

Consider

● 2 5+ IOPs per update (read-modify-write)!● 1 RRD per data source (storeByGroup=false)● 100,000s of data sources, 1,000s IOPS● 1,000,000s of data sources, 10,000s IOPS● 15,000 RPM SAS drive, ~175-200 IOPS

Also

● Not everything is a graph● Inflexible● Incremental backups impractical● ...

Bottleneck!?

● We need to be collecting even more!● We need to be collecting more frequently!● The Internet of Things is upon us!!

But can’t we … ?

● Serialize RRD writes?● Cache?● Distribute the RRDs?● … ?

How about:● Distributed, decoupled architecture● High throughput● Horizontally scalable● Pluggable, extensible graphing● Facilitate new forms of analytics● More?

Starting Over Can Be Fun!

Observation #1

We collect and write a great deal; We read (graph) relatively little.

We are read-optimized.

Observation #2

Grouping samples that are accessed together is an easy optimization.

Project: NewtsGoals:● Stand-alone time-series data store● High-throughput● Horizontally scalable● Grouped metric storage/retrieval● Late-aggregating

Cassandra

● Apache top-level project● Distributed database● Tunable consistency● NoSQL MoSQL

Partitioning

A

B

C

Key: Apple

...

Placement

A

B

C

Key: Apple

...

Replication

A

B

C

Key: Apple

...

Cap Theorem

Consistency

Availability

Partition tolerance

Consistency

A

?

?

W=1

Consistency

A

B

C

R=3

Consistency

A

B

?

W=2

Consistency

?

B

C

R=2

R+W > N

Properties

● Symmetrical● Linearly scalable● Redundant● Highly available

SSTables

Writes

Commitlog

Memtable

SSTable

DiskMemory

Properties

● Optimized for write throughput● Sorted on disk● Perfect for time series!

Gist

● Samples stored as-is.● Samples can be retrieved as-is.● Measurements are aggregations calculated

from samples (at time of query).

Samples vs. Measurementssam·ple/ˈsampəl/noun

1. a small part or quantity intended to show what the whole is like."investigations involved analyzing samples of handwriting"synonyms: representative, illustrative, selected, specimen, test, trial, typical

meas·ure·ment/ˈmeZHərmənt/noun

1. the action of measuring something."accurate measurement is essential"synonyms: quantification, computation, calculation, mensuration

Sample{ ‘timestamp’: 1395278097, ‘resource’ : ‘localhost.eth0’, ‘name’ : ‘ifInOctets’, ‘type’ : ‘COUNTER’, ‘value’ : 457283782231}

NewtsCREATE TABLE newts.samples ( resource text, collected_at timestamp, metric_name text, metric_type text, value blob, attributes map<text, text>, PRIMARY KEY(resource, collected_at, metric_name));

Behind the scenes...

KSAT (1970-02-10 12:42:00,dewPoint,value): 0xc01a0000

(1970-02-10 12:42:00,maxTemp,value): 0x40280000

...

Ascending Order

Newts

resource | collected_at | metric_name | value

---------+---------------------+--------------+-----------

KSAT | 1970-02-10 12:42:00 | dewPoint | 0xc01a0000

KSAT | 1970-02-10 12:42:00 | maxTemp | 0x40280000

KSAT | 1970-02-10 12:42:00 | maxWindGust | 0x7ff80000

KSAT | 1970-02-10 12:42:00 | maxWindSpeed | 0x40180000

KSAT | 1970-02-10 12:42:00 | meanTemp | 0xbfe00000

POSTPOST /samples HTTP/1.1Host: example.comContent-Type: application/json

[ { ‘timestamp’: 1395278097, ‘resource’ : ‘localhost.eth0’, ‘name’ : ‘ifInOctets’, ‘type’ : ‘COUNTER’, ‘value’ : 457283782231 }, { ... },]

GET samplesGET /samples/localhost.eth0?start=900000000 HTTP/1.1

[ [ {“name”: “ifInOctets”, “timestamp”: 900000000, “type”: “COUNTER”, “value”: 12345678900}, {“name”: “ifOutOctets”, “timestamp”: 900000000, “type”: “COUNTER”, “value”: 87654321000} ], [ {“name”: “ifInOctets”, “timestamp”: 900000300, “type”: “COUNTER”, “value”: 23456789000}, {“name”: “ifOutOctets”, “timestamp”: 900000300, “type”: “COUNTER”, “value”: 98765432100} ]]

GET measurementsGET /measurements/octets/localhost.eth0?resolution=20m HTTP/1.1

[ [ {“name”: “ifInOctets”, “timestamp”: 900000000, “value”: 102400.00}, {“name”: “ifOutOctets”, “timestamp”: 900000000, “value”: 409600.00} ], [ {“name”: “ifInOctets”, “timestamp”: 900001200, “value”: 102400.00}, {“name”: “ifOutOctets”, “timestamp”: 900001200, “value”: 409600.00} ]]

http://github.com/OpenNMS/newts