chef actions: delightful near real-time activity tracking!

Chef ActionsActions: Delightful near real-time activity tracking!

Allen Goodman (@goodmanio), Software Engineer, ChefJames Casey (@jamesc_000), Engineering Lead, Chef

A first look at Chef Actions

Chef Actions answers questions about what is happening on your Chef Server

• What changed on your Chef Server ?• Clients, Cookbooks, Data Bags, Environments, Nodes, Roles

• Who changed it ?

• What did they do ?• Create, Update, Delete

• When did they do it ?

Principles

• Provide a read-only view of what happened• Road to audit

• Allow to react to events as they happen

• Also, enable after the fact investigation• “What happened just before nodes started failing runs?”• “When did our systems gets patched for Heartbleed?”

Architecture

Chef Analytics’ Web Application

Event Processing

Message Format from Chef{ "entity_name": "2.8.1", "entity_type": "version", "parent_name": "apache2", "parent_type": "cookbook”, "task": "update", "organization_name": ”my_corp", "recorded_at": "20014-04-05T18:26:13Z", "remote_hostname": ”33.33.33.10", "remote_request_id": "D5974D8A-E961-4845-8D4F-5BB92DAD1BD5", "service_hostname": ”private-chef-fe.example.com”, "request_id": "hY2UqBZTuqAr65M8t4FmYAD5RUjbnhpBhubxBZqVZJZgNX7x", "requestor_name": ”bob", "requestor_type": ”user”, "user_agent": "Chef Client/11.12.0"}

Event Processing Architecture

Classification

• Static and Dynamic Analysis• Conformance

• Are conformant cookbooks less susceptible to failure?• Coverage

• Is coverage correlated with success? • Style

• Does my preferred style require more maintenance? • Syntactic and Semantic Mistakes

• Is this resource broken?

• Supervised Learning• Predicting Imminent and Long-term Problems

• Why does my infrastructure break?

Notifications

IRC

notifications: irc: - "chat.freenode.net#chef" template: - "%{resource} %{action}d!” skip_join: true use_notice: true

E-mail

notifications: email: - a@getchef.com - james@getchef.com on_update: never

Campfire, HipChat, Slack, et al.

Webhooks

• POST a full copy of a message to an external service

• Optionally contains copy of object• E.g. full node object or role

• Ships with a stub service for you start from

• Usage:• Monitoring – notified on create/delete• CMDB – monitor software/OS version

changes for compliance

Webhooks

hooks: endpoint: - https://my-service.example.com/ingest with_object: true

Creating Metricsmetrics: - id: james - "What’s up with James’ weird hours?” - actions: - create - delete - update - actors: - "james" - resources: - cookbook - statistics: - frequency - mean - window - hourly

Persistence

Query Parameters

/actions?tasks=create, delete, update&types=environment&name=prod&before=2014-04-17&since=2014-04-10

Response{ "id": "e1e2c057-b18f-46db-8c84-7d87efc07260", "tags": [ "Knife” ], "recorded_at": "2014-04-13T04:38:03.000Z", "remote_hostname": "76.115.172.22", "remote_request_id": "d57845ce-3b48-4070-9ed4-4b4333d112cb", "request_id": "g3IAA2QAEGVyY2hlZkAxMjcuMC4wLjEDAAKEjwAAAAkAAAAA", "service_hostname": "action-rspreprod-i-c63b3e97.opscode.us", "task": "delete", "url": "/activities/e1e2c057-b18f-46db-8c84-7d87efc07260", "user_agent": "Chef Knife/11.12.2 (ruby-1.9.3-p194; ohai-7.0.2; x86_64-darwin11.4.2; +http://opscode.com)", "requestable": { "name": "bob", "type": "user”, "url": "/users/bob”, "properties": {"email": "james+1@getchef.com", "gravatar_id": "55bf19f1b9fa6ac9ab755488e29a5779”, "avatar_url": “https://gravatar.com/avatar/55bf19f1b9fa6ac9ab755488e29a5779.png?default=mm” }, "entity": { "name": "prod", "parent": null, "type": "environment”, "url": "/organizations/chef/environments/prod”, "manage_url": https://manage-chefconf-demo.opscode.com/organizations/chef/environments/prod } }

Ruby Exampleclient = Actions::Client.new(ACTIONS_KEY)

actions = client.actions.where action: %i(create delete update), actor: 'james' resource: :cookbook, window: { recorded_at: :hourly }

{ frequency: actions.frequency, mean: actions.mean }

R Exampleinstall.packages('chef-actions')

actions = chef.actions.where(action %in% ["create", "delete", "update"]) .where(actor %in% ["james"]) .where(resource %in% ["cookbook"]) .window(recorded_at, "hourly")

summary(actions)

Integration points

• RabbitMQ• Consume raw messages in real-time out of Chef Server

• Notifications• Send formatted messages to external services in real-time

• Webhooks• Send whole message with annotations to your external service in real-time

• REST API• Query-only view of historical data

Deployment

Deployment topologies

• Supports same HA architecture as Enterprise Chef• Backend

• PostgreSQL• Frontend

• Analytics pipeline, Query API, Ingest service

• Recommended to deploy on separate hardware than Enterprise Chef

Omnibus

• New add-on chef-analytics• Delivered as a single omnibus package• Hosted on separate domain

• E.g. analytics.getchef.com• Only interactions with Private Chef

• RabbitMQ configuration details• Manage root URL for generation of links

Futures

What’s Next?

• Permissions – ACL changes, group join/leave• Object Diffs• Live Feed• Aggregation (all actions from a chef-client run, berkshelf, knife run)• Knife instrumentation • Cloud plugins

Q1 Q2 Q3 Q4Account Improvements Large Customer Features Service Provider Features HA Restructuring

Chef Actions and Run History Compliance Reporting Compliance Content

Windows Container Improvements Red Hat

The right premium features

• Make existing differentiation more accessible, address concerns of our largest customers• Extend our analytics capability to provide best of breed compliance reporting• Ensure we retain a lead on Windows support• Embrace containers as a first class part of our ecosystem

Event Processing Architecture

Chef Actions

• We know there is valuable data locked up inside your Chef Server• Chef Actions exposes that data to you via a variety of APIs • We have built a flexible architecture that will allow us to expose more and more data

• Available today – Limited availability• No release schedule – continuous updates• We’ll rapidly open to more and more customers

• Defined roadmap for 2014 for compliance and audit