![Page 2: Monitoring for Online Services - Meetupfiles.meetup.com/1512314/erlang-solutions-meetup.pdf · • Used existing NodeJS code with Node_Erlastic! • Uses ports interface! • Saved](https://reader034.vdocuments.net/reader034/viewer/2022042911/5f448feb549cff39562da2d8/html5/thumbnails/2.jpg)
Disclaimer
• Not an Erlang developer!
• May defer questions Tomasz!
• Based on a true story
![Page 3: Monitoring for Online Services - Meetupfiles.meetup.com/1512314/erlang-solutions-meetup.pdf · • Used existing NodeJS code with Node_Erlastic! • Uses ports interface! • Saved](https://reader034.vdocuments.net/reader034/viewer/2022042911/5f448feb549cff39562da2d8/html5/thumbnails/3.jpg)
What is Dataloop?
PerformanceUp / Down Alerts
Dev Env Enterprise Stuff
![Page 4: Monitoring for Online Services - Meetupfiles.meetup.com/1512314/erlang-solutions-meetup.pdf · • Used existing NodeJS code with Node_Erlastic! • Uses ports interface! • Saved](https://reader034.vdocuments.net/reader034/viewer/2022042911/5f448feb549cff39562da2d8/html5/thumbnails/4.jpg)
Architecture
![Page 5: Monitoring for Online Services - Meetupfiles.meetup.com/1512314/erlang-solutions-meetup.pdf · • Used existing NodeJS code with Node_Erlastic! • Uses ports interface! • Saved](https://reader034.vdocuments.net/reader034/viewer/2022042911/5f448feb549cff39562da2d8/html5/thumbnails/5.jpg)
First Year
![Page 6: Monitoring for Online Services - Meetupfiles.meetup.com/1512314/erlang-solutions-meetup.pdf · • Used existing NodeJS code with Node_Erlastic! • Uses ports interface! • Saved](https://reader034.vdocuments.net/reader034/viewer/2022042911/5f448feb549cff39562da2d8/html5/thumbnails/6.jpg)
First Year
![Page 7: Monitoring for Online Services - Meetupfiles.meetup.com/1512314/erlang-solutions-meetup.pdf · • Used existing NodeJS code with Node_Erlastic! • Uses ports interface! • Saved](https://reader034.vdocuments.net/reader034/viewer/2022042911/5f448feb549cff39562da2d8/html5/thumbnails/7.jpg)
Measure
![Page 8: Monitoring for Online Services - Meetupfiles.meetup.com/1512314/erlang-solutions-meetup.pdf · • Used existing NodeJS code with Node_Erlastic! • Uses ports interface! • Saved](https://reader034.vdocuments.net/reader034/viewer/2022042911/5f448feb549cff39562da2d8/html5/thumbnails/8.jpg)
Putting out the fire
rollup workermetric worker
![Page 9: Monitoring for Online Services - Meetupfiles.meetup.com/1512314/erlang-solutions-meetup.pdf · • Used existing NodeJS code with Node_Erlastic! • Uses ports interface! • Saved](https://reader034.vdocuments.net/reader034/viewer/2022042911/5f448feb549cff39562da2d8/html5/thumbnails/9.jpg)
Problems
• NodeJS metrics workers not scaling!
• Memory management was an issue!
• Needed big caches to reduce database load!
• GC cycles too long!
• 8 x single processes on an 8 core server
![Page 10: Monitoring for Online Services - Meetupfiles.meetup.com/1512314/erlang-solutions-meetup.pdf · • Used existing NodeJS code with Node_Erlastic! • Uses ports interface! • Saved](https://reader034.vdocuments.net/reader034/viewer/2022042911/5f448feb549cff39562da2d8/html5/thumbnails/10.jpg)
Languages
• Decided on Erlang!
• Memory management!
• Fault tolerance!
• Good libraries for Rabbit and Riak!
• Live code tracing
![Page 11: Monitoring for Online Services - Meetupfiles.meetup.com/1512314/erlang-solutions-meetup.pdf · • Used existing NodeJS code with Node_Erlastic! • Uses ports interface! • Saved](https://reader034.vdocuments.net/reader034/viewer/2022042911/5f448feb549cff39562da2d8/html5/thumbnails/11.jpg)
Metric worker re-write
• Approximately 6 weeks from no Erlang experience to working version!
• No more crashes!
• Reduced servers needed from 16 to 8
![Page 12: Monitoring for Online Services - Meetupfiles.meetup.com/1512314/erlang-solutions-meetup.pdf · • Used existing NodeJS code with Node_Erlastic! • Uses ports interface! • Saved](https://reader034.vdocuments.net/reader034/viewer/2022042911/5f448feb549cff39562da2d8/html5/thumbnails/12.jpg)
New Features
![Page 13: Monitoring for Online Services - Meetupfiles.meetup.com/1512314/erlang-solutions-meetup.pdf · • Used existing NodeJS code with Node_Erlastic! • Uses ports interface! • Saved](https://reader034.vdocuments.net/reader034/viewer/2022042911/5f448feb549cff39562da2d8/html5/thumbnails/13.jpg)
Dalmatiner DB
• Open Source Time-Series DB!
• Written in Erlang!
• Based on Riak-Core and uses ZFS!
• Optimised for write throughput!
• Needed for developer analytics features!
• https://dalmatiner.io/
![Page 14: Monitoring for Online Services - Meetupfiles.meetup.com/1512314/erlang-solutions-meetup.pdf · • Used existing NodeJS code with Node_Erlastic! • Uses ports interface! • Saved](https://reader034.vdocuments.net/reader034/viewer/2022042911/5f448feb549cff39562da2d8/html5/thumbnails/14.jpg)
Modifications
• Floating point support!
• Interfaces with C via NIF!
• Lots of fixes for our shape of data
![Page 15: Monitoring for Online Services - Meetupfiles.meetup.com/1512314/erlang-solutions-meetup.pdf · • Used existing NodeJS code with Node_Erlastic! • Uses ports interface! • Saved](https://reader034.vdocuments.net/reader034/viewer/2022042911/5f448feb549cff39562da2d8/html5/thumbnails/15.jpg)
New metrics worker
• Worked with Erlang solutions!
• Cross trained team (Dave and Tomasz)!
• Removed the Redis!
• Reduced servers needed from 8 to 2
new metrics worker
![Page 16: Monitoring for Online Services - Meetupfiles.meetup.com/1512314/erlang-solutions-meetup.pdf · • Used existing NodeJS code with Node_Erlastic! • Uses ports interface! • Saved](https://reader034.vdocuments.net/reader034/viewer/2022042911/5f448feb549cff39562da2d8/html5/thumbnails/16.jpg)
New Things
• Lager!
• Pooler!
• Dialyzer!
• Quick Check!
• Rebar3
• Recon!
• Dave
![Page 17: Monitoring for Online Services - Meetupfiles.meetup.com/1512314/erlang-solutions-meetup.pdf · • Used existing NodeJS code with Node_Erlastic! • Uses ports interface! • Saved](https://reader034.vdocuments.net/reader034/viewer/2022042911/5f448feb549cff39562da2d8/html5/thumbnails/17.jpg)
Data Migration
• Used existing NodeJS code with Node_Erlastic!
• Uses ports interface!
• Saved a lot of development time vs writing from scratch!
• Migrated one organisation at a time over several weeks!
• Ran Riak and Dalmatiner in parallel and then switched
![Page 18: Monitoring for Online Services - Meetupfiles.meetup.com/1512314/erlang-solutions-meetup.pdf · • Used existing NodeJS code with Node_Erlastic! • Uses ports interface! • Saved](https://reader034.vdocuments.net/reader034/viewer/2022042911/5f448feb549cff39562da2d8/html5/thumbnails/18.jpg)
Today
![Page 19: Monitoring for Online Services - Meetupfiles.meetup.com/1512314/erlang-solutions-meetup.pdf · • Used existing NodeJS code with Node_Erlastic! • Uses ports interface! • Saved](https://reader034.vdocuments.net/reader034/viewer/2022042911/5f448feb549cff39562da2d8/html5/thumbnails/19.jpg)
Happy Ending
![Page 20: Monitoring for Online Services - Meetupfiles.meetup.com/1512314/erlang-solutions-meetup.pdf · • Used existing NodeJS code with Node_Erlastic! • Uses ports interface! • Saved](https://reader034.vdocuments.net/reader034/viewer/2022042911/5f448feb549cff39562da2d8/html5/thumbnails/20.jpg)
Next
• Convert the other workers to Erlang!
• Add metric dimensions to Dalmatiner DB!
• Make RabbitMQ more robust!
• Hire more Erlang developers!
![Page 21: Monitoring for Online Services - Meetupfiles.meetup.com/1512314/erlang-solutions-meetup.pdf · • Used existing NodeJS code with Node_Erlastic! • Uses ports interface! • Saved](https://reader034.vdocuments.net/reader034/viewer/2022042911/5f448feb549cff39562da2d8/html5/thumbnails/21.jpg)
Q&A