the blackbox project: safely store secrets in git/mercurial (originally for puppet)
DESCRIPTION
A presentation given at PuppetCamp NYC 2014 about why Puppet users should stop storing secrets in Git/Hg and encrypt them instead. TLDR: It enables collaboration.TRANSCRIPT
The BlackBox projectSafely storing secrets and credentials in
Git/Mg (mostly for use by Puppet)
Tom Limoncelli, SRE, StackExchange.comBlog: EverythingSysadmin.com
My new book! the-cloud-book.com
125+ Q&A CommunitiesServerFault.comStackOverflow.com
(We <3 Puppet!)
StackExchange.com
What are secrets?
Anything you don’t want exposed externally.
● SSL Certificates (the private bits)● Passwords● API keys
Puppet manages secrets
If you store secrets in git, you’re gonna have a bad time.
● Laptops get stolen.● Workstations have guest accounts● Git server “Circle of Trust” includes:
○ Everyone with admin access to workstations.■ Your desktop support people?
○ Everyone with admin access to your git server:■ Server team, storage team, backup team
○ Everyone you collaborate with that wants read-only access to Puppet manifests.
You have 3 bad options:
1. Deny git access. (Hurts collaboration)2. Permit git access. (Hurts security)3. Email individual files. (Hurts… just hurts)
Option 4: Encrypt secret parts
● If a file contains secrets, encrypt before checking into Git.
● Need to edit a secret?○ Decrypt - Edit - Encrypt
What about Puppet master?
● After “git pull”, decrypt all files.○ Automate this as part of CI.
● Files are unencrypted “at rest”.● This does not decrease security:
○ No worse than what we were doing before.○ If you can break into root or puppet on the master,
you’ve already won.
Easy, right?
Decrypt:
Encrypt:
Easy, right?
Decrypt:
Encrypt:
● ...and don’t make any typos when entering the command● ...and don't accidentally check in the unencrypted version
Security is 1% technology plus 99% following the procedures correctly.
Any process with more than 1 step probably won't be followed consistently most of the time.
Related reading: "Why Johnny Can't Encrypt: A Usability Evaluation of PGP 5.0”, Alma Whitten", Usenix Security 1999
Therefore…. we automate
Introducing: Blackbox
Scripts for keeping Puppet secrets in git/hg.
User commands:
Decrypt for editing:
Encrypt when done:
First time a file is encrypted:
Enroll a file into the system:
Commands that act on all GPG files:
Decrypt all files: (for use on puppet master)
Re-encrypt all files: (after new users added)
Everyone has their own key
This doesn’t use “symmetric encryption” where there is one passphrase to decrypt/encrypt all files.
We maintain a keyring of:● Each person that should have access.● A key for the Puppet master.
Indoctrinate a new user:
1. New user does this:
●●
●
(Currently a doc, not a script. Patches gladly accepted.)
Indoctrinate a new user:
2. Existing admin does this:
Demo: Edit a file
Demo: Edit a file
Demo: Edit a file
Demo: Edit a file
Demo: Edit a file
Demo: Edit a file
Demo: Edit a file
Code is open source as of 7/2014
● Entirely written in bash.● MIT License.● Download it now:
○ https://github.com/StackExchange/blackbox
In the project’s first 9 months:
StackExchange/ServerFault has eliminated plaintext secrets in our Puppet git repo.● 7 SREs+Devs sharing the repo securely.● 50+ files now stored encrypted.
○ Mostly SSL certs and SSH private keys.● 40+ individual passwords/API keys:
○ Everything from SNMP communities, SaaS API keys, and many many passwords.
Future plans
❏ Open source scripts.❏ More usability enhancements.❏ Better setup documentation.
Join the open source projecthttp://github.com/StackExchange/blackbox
Q&A
URLs from this talk:https://github.com/StackExchange/blackbox
EverythingSysadmin.com
Shameless plugPre-order now! Save 35%
Ships in September.
informit.com/TPOSADiscount code TPOSA35
Read “rough cuts” today:safaribooksonline.com
Q&A
URLs from this talk:https://github.com/StackExchange/blackbox
EverythingSysadmin.comthe-cloud-book.com
informit.com/TPOSA (code TPOSA35)
● Easier transition. No Puppet code changes for big files like SSL certs.
● Faster. Zero run-time performance impact on master.
● eyaml didn’t exist when we started.
Why didn’t we use eyaml?