www.dataloop.io | @dataloopio | info@dataloop.io

Monitoring for Online Services

Disclaimer

• Not an Erlang developer!

• May defer questions Tomasz!

• Based on a true story

What is Dataloop?

PerformanceUp / Down Alerts

Dev Env Enterprise Stuff

Architecture

First Year

First Year

Measure

Putting out the fire

rollup workermetric worker

Problems

• NodeJS metrics workers not scaling!

• Memory management was an issue!

• Needed big caches to reduce database load!

• GC cycles too long!

• 8 x single processes on an 8 core server

Languages

• Decided on Erlang!

• Memory management!

• Fault tolerance!

• Good libraries for Rabbit and Riak!

• Live code tracing

Metric worker re-write

• Approximately 6 weeks from no Erlang experience to working version!

• No more crashes!

• Reduced servers needed from 16 to 8

New Features

Dalmatiner DB

• Open Source Time-Series DB!

• Written in Erlang!

• Based on Riak-Core and uses ZFS!

• Optimised for write throughput!

• Needed for developer analytics features!

• https://dalmatiner.io/

Modifications

• Floating point support!

• Interfaces with C via NIF!

• Lots of fixes for our shape of data

New metrics worker

• Worked with Erlang solutions!

• Cross trained team (Dave and Tomasz)!

• Removed the Redis!

• Reduced servers needed from 8 to 2

new metrics worker

New Things

• Lager!

• Pooler!

• Dialyzer!

• Quick Check!

• Rebar3

• Recon!

• Dave

Data Migration

• Used existing NodeJS code with Node_Erlastic!

• Uses ports interface!

• Saved a lot of development time vs writing from scratch!

• Migrated one organisation at a time over several weeks!

• Ran Riak and Dalmatiner in parallel and then switched

Today

Happy Ending

Next

• Convert the other workers to Erlang!

• Add metric dimensions to Dalmatiner DB!

• Make RabbitMQ more robust!

• Hire more Erlang developers!

Q&A

www.dataloop.io!

@dataloopio!

info@dataloop.io