the 5 stages of secrets management grief, and how to prevail
TRANSCRIPT
The 5 Stages of Secrets Management Grief(And How to Prevail)
Josh BregmanConjur
Josh has 20 years experience successfully architecting, evangelizing, and delivering innovative identity management and security products to customers. Prior to joining Conjur , Josh spent a decade as a solutions and pre-sales leader in the Oracle ecosystem. A developer at heart, early in his career Josh worked as a software engineer at IBM, GTE Labs, and Netegrity. He has 3 U.S. patents and received a B.A. in Math from the University of Rochester in 1995.
Thanks Dave!
Denial - We don’t have a problemAnger - Why is this my problem?Bargaining - A series of trade-offs?Depression - This isn’t fixed?Acceptance - We have a problem
Denial - We don’t have a problemAnger - Why is this my problem?Bargaining - A series of trade-offs?Depression - This isn’t fixed?Acceptance - We have a problem
You’re at Puppet Camp, so your infrastructure is coded, and your code is in source control.
class { 'wordpress': db_user => 'wordpress', db_password => 'hvyH(S%t(\"0\"16', db_host => 'db.example.com', create_db => false, create_db_user => false,}
If you put your secrets in source control, then anyone who has access to the repo can access all the secrets.
3/10/16
“Searching GitHub for AWS and Azure credentials reveals that many people are making the same mistake as Ashley Madison, Uber and D-Link.”
“Ashley Madison’s leaked code included hard-coded AWS tokens, database credentials, certificate private keys and other credentials.
“Uber had a database containing personal information about drivers compromised in 2014, after storing the key in a publicly available repo”
“...and D-Link recently published its private code signing keys in the open source code for a firmware update.”
“Your cloud credentials are likely to end up subsidizing Bitcoin miners, who scan GitHub for keys and use them to run up hundreds or thousands of dollars of bills.”
(Sound of everyone making sure that repo is private)
Should everyone at your company who has read access to the repo have access to the database?
NO
If you put your secrets in source control, then anyone who has access to the repo can access all the secrets.
Denial - We don’t have a problemAnger - Why is it my problem?Bargaining - A Series of Trade-offs?Depression - This isn’t fixed?Acceptance - We have a problem
Typical conversation BEFORE something bad happens…
20
Question to Information Security:I’m concerned that storing secrets in source control isn’t safe. Is there a recommended approach that I should be following?
21
Answer from Information Security: We’re super busy right now protecting the company from APT, passing our ISO 27001 audit, and assessing our compliance NIST CSF…
22
Question to Information Security:Well, this initiative is super important to the business. Is there anything that you can recommend?
23
Answer from Information Security: We’ll we have an existing system that we use to manage privileged accounts. You just open a ServiceNow ticket and…
24
That’s OK….we’ll just figure it out
25
Typical conversation AFTER something bad happens…
26
“I’m going to automate those *@! out of a job.” – Anonymous DevOps“Those *@! are running with scissors” – Anonymous InfoSec
27
Few organizations practice blame free post-mortems, if they are on the front page of the Wall Street Journal. This is a huge cultural change for Information Security.
28
SecDevOps or DevSecOps or RuggedDevOps are all terms for the inclusion of information security in the DevOps workflow
29
Automated testing that includes security tests like code scanning, application security testing, automated patching of vulnerabilities are all pretty easy…
30
…because they can be added without the direct cooperation of information security teams.
31
Question: How many people here have information security professionals as part of their DevOps teams?
32
Question: How many people here ACTIVELY seek out the information security professionals in their organization?
33
NIST CyberSecurity Framework - The Framework Implementation Tiers (“Tiers”) provide context on how an organization viewscybersecurity risk and the processes
34
NIST CSF Tiers:Tier 1 -> PartialTier 2 -> Risk InformedTier 3 -> RepeatableTier 4 -> Adaptive
35
Go Find your Security Engineering Team. This is the team that owns and operates security solutions. Tell them you can help them with automation.
36
Denial - We don’t have a problemAnger - It’s their fault!Bargaining - A Series of Trade-offs?Depression - This isn’t fixed?Acceptance - We have a problem
Two main camps on secrets management with Puppet: Masterful and Masterless
Masterful: I’m OK if all of my secrets are on the master; my master is a hardened command bunker.Masterless: Secrets are ‘need to know’ and my master doesn’t.
The “Masterful” approaches can be accomplished with little commitment from information security.
If you don’t engage them, and something goes wrong, it’s all on you.
Masterful - E-Yaml
If you use E-YAML, secrets are encrypted in source control, and in the catalog.
If you use E-YAML, you have to figure out how to secure the keys, and rotate the keys, and work with encrypted files...
Another “Masterful” approach that some customers use is to deploy separate Production and Non-Production Puppet Masters
This keeps production information limited to only those that need access to production. By design, this pattern makes Continuous Delivery hard.
If you use DSL extensions or Hiera backend, then secrets are in the catalog, but not in source control.
Retrieves a Secret from Conjur programmatically
https://github.com/dgrstl/puppet_conjurdemo/blob/master/lib/puppet/functions/conjur_secret.rb
See it in action
https://github.com/dgrstl/puppet_conjurdemo/blob/master/tests/notify.pp
Masterless approach - a.k.a - Node Side Secrets - secrets aren’t in source control and they are not on the master
Scenario 1 - Write a configuration file on the node that has a list of secrets
Puppet::conjur_demo {‘/opt/foo.conf’:secrets =>
[‘/production/db/user’,’/production/db/password’],
}
Scenario 2 - Control the value of an attribute via Conjur
Puppet::conjurdemo_secert_value { ‘foo’:secret_key_name => ‘bar’,resource => Wordpress[‘server1’],field => ‘db_password’,
}
datacat_collector { "$title Conjur secret": template_body => template('puppet_conjurdemo/conjur_simple_secret.erb'), target_resource => $resource, target_field => $field, notify => $resource, }
<%% require 'conjur/cli' require 'yaml' Conjur::Config.load Conjur::Config.apply api = Conjur::Authn.connect-%><% @secrets.each do |secret_key| %><%% secret = api.variable "<%=secret_key%>" %><%=secret_key%> = <%%=secret.value-%><%end%>
Denial - We don’t have a problemAnger - It’s their fault!Bargaining - A Series of Trade-offs?Depression - This isn’t fixed?Acceptance - We have a problem
In both the masterful and masterless models, the secrets wind up on the nodes.
This represents a whole new threat surface - a way that your secret information can be compromised
Applications and services, not just infrastructure also need access to credentials. And applications are stored in source control.
Summon is an open-source project that allows for the retrieval of secrets safely without checking the secrets into source control
Summon works well with 12 factor apps - those that expect to get their configuration from the environment - e.g. Java Application
secrets.yaml
MYSECRET: !var secret/pathMYSECRET2: !var secret/path2
USAGE: summon [global options] command [command options] [arguments...]Ex: summon -f /opt/secrets.yaml printenv
Supports simple provider interface
variable = sys.argv[1] value = keyring.get_password( os.environ.get('SUMMON_KEYRING_SERVICE', 'summon'), variable )
https://github.com/jbregman/puppet-summon/tree/master/jbregman-summon/tests
Test 1 - Simple secrets.yamlTest 1a - Parameterized secrets.yamlTest 2a - Creating Config Test 3 - facter
Denial - We don’t have a problemAnger - It’s their fault!Bargaining - A Series of Trade-offs?Depression - This isn’t fixed?Acceptance - We have a problem
This is an evolution from the Conjur Puppet integration that I presented at Puppet Conf in 2015
Integration with Puppet is an important but emerging area. These modules and repos are works in progress.
Customers are very interested in additional capabilities such as rotation, versioning, secure service lifecycle
Resources:https://github.com/dgrstl/puppet_conjurdemohttps://puppetlabs.com/blog/using-node-side-secrets-with-puppethttps://conjurinc.github.io/summon/
Call to action:• Have a discussion “Are we a
masterless or masterful shop?”• Make friends with information
security
QUESTIONS?
