Wrapper FormulasPut Your Formulas in a Formula So You Can Orchestrate Your Orchestration!

About me

• Principal Software Engineer, Virtustream• Previously:

– System Administrator– Site Reliability Engineer– Automation Engineer

• 4+ years experience with Saltstack

Virtustream– Virtustream is a member of the EMC Federation– Providing IaaS, Object Storage, and Managed Services– Mission Critical Enterprise workloads in the Cloud

Platform Engineering

• Deploy and Manage:– Mesos Platform– Openstack Platform

Salt at Virtustream

• 15,000+ managed nodes• 25+ sites• Multi-Exabyte Object Storage• VMware• OpenStack• Apache Mesos

Quick Review

Beginnings

• Deployed SaltStack primarily for remote execution on our object storage platform

• Built up some states and pillar data• Kept the state and pillar trees in a GitHub

repository

Beginnings

• Worked reasonably well for a single project• Difficult to test• Difficult to deploy• Not multi-project friendly:

– Poor code reuse– Divergence across projects– Lacks consistent patterns

Individual Formulas (what they solve)• Promotes code reuse• Cookiecutter projects make it easy to get everyone using the same patterns.• Debian packaging solves the distribution problem• State and Execution modules bundled• TeamCity runs our tests, builds, and deployments• Version controlled• Peer reviewed• Tests executed on pull request and added to comments• Test individual states• Pillar data stored with project• We’re up to 140+ formulas built this way!

Individual Formulas (results)

• Our project’s /srv/salt directory is far less populated

• /srv/formulas directory is heavily populated

Questions

Individual Formulas (what they don’t solve)

• Poor support for formula interdependency• Reusable pillar data is not exposed• Does not allow testing of pillar data

Wrapper Formulas

• Captures a single platform or product

• Ties individual formulas together

Wrapper Formulas

• Orchestration and/or states from each formula are included in one orchestration state

• This orchestration state orders the installation and executes blocking health checks between steps as necessary

Wrapper Formulas

• Includes orchestration and states from each formula

Wrapper Formulas

• Extends orchestration to enforce order

Wrapper Formulas

• Formulas bring their own orchestration as needed.

• Contains all non-environment pillar data for the platform

• Allows testing of the entire platform while still allowing individual formula testing

Installs to known location

• Orchestration and state data installs into /srv/salt/ext/• Pillar data installs into /srv/pillar/ext/• This is done so that we know that the state and pillar

data came from a wrapper formula

Turn up a whole stack with one command!

• Now when we need to turn up a new datacenter or testing environment it’s as easy as:– salt-run state.orchestrate

ext.tw.orchestration.bootstrap

Mesos Platform

• Our flagship wrapper formula• Runs on bare metal in production and staging• Runs on Vagrant and Openstack VMs for

development• Chad Heuschober’s talk after this for more

details

Next steps

• Full platform testing via TeamCity• Migration of other projects to this model• Event driven orchestration?

Questions