Rethink Monitoring

What Just Happened? Leveraging Puppet & Boundary to Manage Dynamic Environments

Molly Stamos, Director of Products

Boundary: Highly Distributed Platform delivering SaaS

•  Boundary is a SaaS application that monitors application performance by analyzing application communication over the network"

•  Platform runs on Ubuntu Linux with a mix of off the shelf and home grown components "

•  Continuous deployment shop - code rolled into production as its ready "

"

Boundary: The Team

•  “devops” shop – but two primary Ops guys managing about 200 nodes and overseeing all automation "

"

"

•  Everyone participates in operations – everyone is on call rotation (and gets the joy of 2am wake up calls)"

•  Environment is hosted – Provider racks & stacks – Joe & Scott do everything else"

Scott Smith Joe Williams

An Example (but first a quick look at Boundary)

Configuration Change in Production

•  Kobayashi is our historical data store "•  9 node cluster "•  Riak under the covers"•  Provides historical feed of data for Boundary dashboards (beyond streaming system)"

•  Configuration change needed to alleviate feared future performance problems"

•  In Riak, must perform a “rolling upgrade”"•  Push config change to each node one at a time, restart services, wait for cluster to reach steady state"

•  Unsure how cluster behavior would change "

Using Boundary to Assess Cluster Health

"

Unexpected Benefit: Evenly Distributed Writes Across the Cluster

BEFORE" AFTER"

Application Latency (App RTT) Before the Change

•  Node Data06 is 2x slower than the rest of the nodes"

Application Latency (App RTT) After the Change

•  App RTT are tightly grouped around 4-5 ms"

Boundary & Puppet Using the Puppet Module

Download and Install the Boundary Module

•  Fairly Straightforward:"•  sudo puppet module search boundary •  sudo puppet module install puppetlabs-boundary

Make sure you get this one: puppetlabs-boundary (NOT the bprobe one) !

•  **Some configuration required**"•  Boundary::params class – set your API Key and your OrgID "

•  Located in /etc/puppetlabs/puppet/boundary/manifests/params.pp"

•  Boundary.yaml file – set your API Key and OrgID "•  Located in /etc/puppetlabs/puppet/boundary/"

Where to get your API Key & OrgID in Boundary

•  Go to Organization -> Settings"

•  API Key & OrgID"•  API Key is like your password – keep it secret"

Using Puppet to Deploy the Boundary Meter

Get Boundary Now

•  Boundary: http://boundary.com/signup/"•  Literally takes less than 2 minutes to deploy"

•  When was the last time a monitoring tool took less than 2 minutes to set up?"

•  Free to use up to 2GB per day "

•  How do I get more information? "•  Website: www.boundary.com"

•  Email: service@boundary.com"

Rethink Monitoring