Nice and Secure: Good OpSec Hygiene with Puppet!

Peter SouterProfessional Services Engineer | Puppet

@petersouter

@petersouter 2

Who am I?

@petersouter

Professional Services Engineer

5 years using Puppet

2 years @ Puppet Inc

Help customers deploy Puppet Enterprise

Teach Puppet classes

petems IRC/Slack/GitHub

@petersouter

Warning: I speak quicklyAnd I have a different accent...

3

@petersouter

My feelings on Q&Ahttp://bit.ly/why_no_talk_qa

● Tweet me @petersouter● Come up after this talk ● Meet me in the hallway

4

When will this QA be over so I can

leave?

We’ve got lots to cover - No Q&A!

@petersouter

So, why are we here?(This room specifically, listening to this talk...)

5

@petersouter 6

Every time someone uses this picture, Pete Cheslock gets his wings!

https://twitter.com/petecheslock/status/595617204273618944

@petersouter

Show of hands in the roomLet’s take the temperature of security here

7

@petersouter

Why is Puppet good for security?

Infrastructure as code RBAC Auditing Enforcement

8

@petersouter

Don’t let Puppet be the attack vector!aka. How do we make sure we’re not pushing the problem elsewhere?

9

@petersouter

What is OPSEC?Before we talk about something we should define it

10

@petersouter

“Operations Security, or OPSEC, is the process by which we protect unclassified information that can be used against us. OPSEC challenges us to look at ourselves through the eyes of an adversary (individuals, groups, countries, organizations). Essentially, anyone who can harm people, resources, or mission is an adversary.”

11

Department of Defense Education Activityhttp://www.dodea.edu/offices/safety/opsec.cfm

@petersouter 12

● Keeping your code clear of sensitive information

● Approaches to secrets management with the Puppet toolchain

● Making sure security is part of your workflow, rather than an afterthought

What are we going to cover?

https://flic.kr/p/7LcF2W

@petersouter

Let’s start with secrets...

13

We’ve all got them...

@petersouter 14

What are secrets in IT?

RadioactiveConsequences are dire from a leak

ExamplesPasswords, API Keys, SSH Keys, SSL Certs...

SmallA few kb at most

RequiredThe infrastructure won't work without them!

https://flic.kr/p/dHrwpb

@petersouter

Easiest to hardest● Avoid exposing secrets in logs

● Remove data from code and into the data layer (hiera)

● Encryption

15

How do we avoid exposing secrets in Puppet?

https://flic.kr/p/aCJZrf

@petersouter

Don’t expose secrets in logsKeep your secrets hidden

16

@petersouter

show_diffThe first place for leaks

17

@petersouter 18

root@homebox:~# puppet agent --show_diff

Notice: Compiled catalog for homebox.home in environment production in 0.10 seconds

Notice: /Stage[main]/Main/File[/etc/sensitive]/content:

--- /etc/sensitive 2016-08-14 23:01:37.036863915 +0100

+++ /tmp/puppet-file20160814-24654-ak1ywd 2016-08-14 23:01:56.852882307 +0100

@@ -1 +1 @@

-Not Secret

\ No newline at end of file

+SECRET-CONTENT

\ No newline at end of file

Notice: /Stage[main]/Main/File[/etc/sensitive]/content: content changed

'{md5}2ab96390c7dbe3439de74d0c9b0b1767' to '{md5}44c7be48226ebad5dca8216674cad62b'

Notice: Applied catalog in 0.20 seconds

How it looks...

@petersouter

Anywhere reports go:

● syslog● interactive terminal output● PE Console● ENC● report processors

19

Where does the information from show_diff go?

@petersouter 20

file { ‘/etc/secrets.txt’:

ensure => 'file',

owner => 'root',

mode => '0600',

content => 'hunter2',

show_diff => false,

}

Setting show_diff to false at the resource level

@petersouter 21

An example from a Supported Module: mysql

file { "${::root_home}/.my.cnf":

content => template('mysql/my.cnf.pass.erb'),

owner => 'root',

mode => '0600',

}

# show_diff was added with puppet 3.0

if versioncmp($::puppetversion, '3.0') >= 0 {

File["${::root_home}/.my.cnf"] { show_diff => false }

}

https://github.com/puppetlabs/puppetlabs-mysql/blob/d58a100fa67bc99b4388d4ea3921b11647d483d7/manifests/server/root_password.pp#L39

@petersouter

Setting show_diff to false at resource scopeshow_diff = false

22

root@homebox:~# puppet apply secret.pp

Notice: Compiled catalog for homebox.home in environment production in 0.10 seconds

Notice: /Stage[main]/Main/File[/etc/sensitive]/content: content changed '{md5}d3b07384d113edec49eaa6238ad5ff00' to

'{md5}44c7be48226ebad5dca8216674cad62b'

Notice: Applied catalog in 0.19 seconds

@petersouter

There’s a balance...Hiding diffs reduces visibility of change...

23

@petersouter

inifile module to hide changesAllows you to only hide the sensitive fields

24

@petersouter 25

ini_file now has show_diff from the 1.5.0 release

@petersouter

Setting show_diff on individual sensitive fieldsshow_diff = false

26

ini_setting { 'ACME App Timezone':

section => 'TimeDate',

setting => 'TimeZone',

value => $acme_app_time_zone,

}

ini_setting { 'ACME App Password:

section => 'Settings',

setting => 'Password',

value => $acme_app_password,

show_diff => false,

}

@petersouter

Sensitive typeNew for the Puppet 4.6+ release

27

@petersouter 28

file { '/etc/sensitive':

ensure => 'present',

owner => 'root',

group => 'root',

content => Sensitive('hunter2'),

}

root@homebox:~# puppet apply secret.ppNotice: /Stage[main]/Main/File[/etc/sensitive]/ensure: changed [redacted] to [redacted]Notice: Applied catalog in 0.18 seconds

Ability to redact strings with the new Sensitive Type

@petersouter 29

$secret = Sensitive(‘Unwrapped’)

$unwrapped = $secret.unwrap |$sensitive| { $sensitive }

notice("Unwrapped: ${unwrapped}")

$secret.unwrap |$sensitive| { notice("Lambda: ${sensitive}") }

Unwrapping the secrets

https://www.devco.net/archives/2016/09/05/puppet-4-sensitive-data-types.php

@petersouter 30

You can use a dedicated redacted resource

Still on < 4.6?

30

@petersouter 31https://github.com/openstack/puppet-barbican/blob/2e2b10ae58fdc9ad27d88d3195260ef02af853ad/lib/puppet/type/barbican_config.rb

newproperty(:value, :array_matching => :all) do

desc 'The value of the setting to be defined.'

munge do |value|

value = value.to_s.strip

value.capitalize! if value =~ /^(true|false)$/i

value

end

newvalues(/^[\S ]*$/)

def is_to_s( currentvalue )

if resource.secret?

return '[old secret redacted]'

else

return currentvalue

end

end

def should_to_s( newvalue )

if resource.secret?

return '[new secret redacted]'

else

return newvalue

end

end

@petersouter 3232

Encrypt secrets on a node by node basis

Try binford2k-node_encrypt

32

@petersouter 33https://github.com/binford2k/binford2k-node_encrypt

● Master encrypts secrets for each node using their own certificate● Secret can only be decrypted with the node's private key● Uses built-in Puppet CA, so the base case is zero-config

node_encrypt::file {'/etc/company_app/credentials':

ensure => file,

owner => 'root',

content => 'hunter2', # transparently encrypted

}

How does it work?

@petersouter

node_encrypt

34

$ puppet agent -t

Info: Using configured environment 'production'

Info: Retrieving pluginfacts

Info: Retrieving plugin

Info: Loading facts

Info: Caching catalog for master.puppetlabs.vm

Info: Applying configuration version '1450109738'

Notice: /Stage[main]/Main/Node[default]/Node_encrypt::File[/tmp/foo]/Node_encrypted_file[/tmp/foo]/ensure: created

Notice: Applied catalog in 9.33 seconds

$ echo blah > /tmp/foo

$ puppet agent -t

Info: Using configured environment 'production'

Info: Retrieving pluginfacts

Info: Retrieving plugin

Info: Loading facts

Info: Caching catalog for master.puppetlabs.vm

Info: Applying configuration version '1450109821'

Notice: /Stage[main]/Main/Node[default]/Node_encrypt::File[/tmp/foo]/Node_encrypted_file[/tmp/foo]/content: content changed '<<encrypted>>' to

'<<encrypted>>'

Notice: Applied catalog in 7.61 seconds

@petersouter

So data is no longer exposed in logsBut the data is still visible in the code!

35

@petersouter

Remove data from codeEspecially organisation specific data

36

@petersouter

Bad!Don’t do this...

37

@petersouter 38

class example_company_app {

if $::fqdn == 'prod.example.com' {

class {'company_app':

ensure => 'present',

password => 'hunter2',

ssl_enable => true,

}

} else {

class {'company_app':

ensure => 'present',

password => 'example123',

ssl_enable => false,

}

}

}

@petersouter

Good!Do this...

39

@petersouter 40

class profile::example_company_app {

$app_password = hiera('profile::example_company_app::password')

$app_ssl_enable_password = hiera('profile::example_company_app::password')

class {'company_app':

ensure => 'present',

password => $app_password,

ssl_enable => $ssl_enable,

}

}

@petersouter

Roles and profiles help a lotAbstracting implementation specifics away

41

@petersouter

Allows you to set organisational defaults in your roles and profiles

42

● Keep organisational specific data in hiera● Move organisational specific setup into role and profile wrappers

Advantage: Not only more secure, cleaner code that’s more reusable!

@petersouter

Default parameters are important!Abstracting implementation specifics away

43

@petersouter

Storytime: OpenstackThe perils of bad defaults...

44

@petersouter 45https://archive.fosdem.org/2015/schedule/event/public_puppet/

The open source OpenStack project infrastructure

Fully public Puppet

@petersouter

The more abstracted your control-repo, the less chance of leaking or compromising of secrets...

46

@petersouter

Another gotcha: hierarchy lookupsAka. Why trusted facts are good!

47

@petersouter

Take a look at this hiera config...

48

# hiera.yaml

---

:hierarchy:

- "node/%{fqdn}"

- "common"

:backends:

- yaml

:datadir: '/etc/puppet/environments/%{environment}/hieradata'

@petersouter

Facts are spoofable!

49

[root@testbox]# facter fqdn

pe-201620-master.puppetdebug.vlan

[root@testbox]# FACTER_fqdn=evil.example.com facter fqdn

evil.example.com

Facts are spoofable

@petersouter

Trusted facts got your back!

50

Locked in from the certificate request

@petersouter

Trusted facts are stamped on Node creation

51

Trusted facts are stamped on Node creation

@petersouter

We have a bunch of OIDs for this also...

52https://docs.puppet.com/puppet/latest/reference/ssl_attributes_extensions.html

@petersouter

New, better hiera hierarchy...

53

# hiera.yaml

---

:hierarchy:

- "node/%{trusted.certname}"

- "common"

:backends:

- yaml

:datadir: '/etc/puppet/environments/%{environment}/hieradata'

@petersouter

Theoretically, you should be able to release most of the code you write publically

without any sort of security issues

54

@petersouter 55

This is actually a tenet of 12 Factor Apps...

Apps sometimes store config as constants in the code. This is a violation of twelve-factor, which requires strict separation of config from code. Config varies substantially across deploys, code does not.

A litmus test for whether an app has all config correctly factored out of the code is whether the codebase could be made open source at any moment, without compromising any credentials.

Note that this definition of “config” does not include internal application config, such as config/routes.rb in Rails, or how code modules are connected in Spring. This type of config does not vary between deploys, and so is best done in the code.

http://12factor.net/config

@petersouter

Example: GDSGovernment Digital Service, UK

56

@petersouter 57

Meeting the Digital Service Standard

To meet point 8 (understand security and privacy issues) you must:

● Make all new source code open and reusable

● Publish code under an appropriate licence

● Explain your reasoning for any code you haven’t made open

You’ll have to explain how you did this at your service assessments.

https://www.gov.uk/service-manual/technology/making-source-code-open-and-reusable

@petersouter 58

Meeting the Digital Service StandardWhen GOV.UK was first set up we were unable to publish our Puppet repository because our code and secrets were tied together. This goes against patterns like the 12-factor app which “requires strict separation of config from code”

This wasn’t true for our Puppet repository, but we gradually moved our credentials into a separate repository (rotating them as we did so).

“A litmus test for whether an app has all config correctly factored out of the code is whether the codebase could be made open source at any moment, without compromising any credentials.”

@petersouter 59

$ strings modules/**/*.pp | tr ' '

'\n' | sort -n | uniq | view -

Check code for unique strings that look secret-y

Note: Requires zsh for the strings function!

@petersouter

It’s not just Puppet code!Git commits can sustain sensitive data!

60

@petersouter 61

$ git commit -a -m "Changed the

password to password1"

@petersouter

$ while read line; do echo $line;

git --no-pager log -p -S $line; done

< puppet_search

62

Manually searching through git commits for sensitive information...

@petersouter

Opening GOV.UK’s Puppet Repository https://gdstechnology.blog.gov.uk/2016/01/19/opening-gov-uks-puppet-repository/

Git Repo https://github.com/alphagov/govuk-puppet

Want to know more?

63

@petersouter

Your data is now separated. Hooray!But it’s still plaintext in Hiera. Boo! :-(

64

@petersouter

EncryptionTold you we’d come back to it!

65

@petersouter

Bad!Don’t do this!

66

@petersouter 67

@petersouter

Good!Do this!

68

@petersouter 69

Preso title goes here. To update, go to File > Page Setup > Header/Footer, paste title, Apply All

@petersouter 70https://github.com/TomPoulton/hiera-eyaml

Hiera eyaml

@petersouter

hiera-eyaml is probably the best method for internal data encryption with PuppetIt’s widely used, and has a number of plugins

71

@petersouter

eyaml plugins

72

● https://github.com/sihil/hiera-eyaml-gpg● https://github.com/tehmaze/hiera-eyaml-secretbox● https://github.com/acidprime/hiera-eyaml-pkcs11● https://github.com/adenot/hiera-eyaml-kms● https://github.com/gtmtechltd/hiera-eyaml-twofac

@petersouter

The idea is that Puppet will natively support encrypted data in the future

Follow this ticket for the roadmap view: PUP-1974

73

@petersouter

● TranscryptGit-CryptBlackbox

● Turtles All The Way Down: Storing Secrets in the Cloud and in the Data Center behind Closed Doors

VCS based encryption

74http://danielsomerfield.github.io/turtles

https://www.youtube.com/watch?v=OUSvv2maMYI

@petersouter

Dedicated secret devicesGoing deeper...

75

@petersouter

Why use a secret server?

76

● Dynamic secrets● ACL (Access control policies)● Leasing and renewal● Revocation● Encryption● Auditing● Supportability

@petersouter 77

Conjur, Vault, Keywhiz, Amazon KMS, Confidant

@petersouter 78

Hiera will plug into any secret service app with a little bit of Ruby glue

Hiera is just key/value lookup A hiera backend to basically any secret server setup is possible

@petersouter 79

$planet = conjur_variable('planet')

file { '/etc/hello.txt':

content => "Hello ${planet}!\n"

}

conjurize_file { '/etc/hello.txt':

variable_map => {

planet => ‘!var puppetdemo/planet’

}

}

https://www.conjur.net/puppet-secret-serverhttps://forge.puppet.com/conjur/conjur

Conjur and Puppet

@petersouter 80https://github.com/jsok/hiera-vault

hiera-vault

@petersouter

If you want to know more about Puppet + Vault, Seth Vargo from Hashicorp is presenting tomorrow

81

@petersouter

Behind Closed Doors - Managing Passwords in a Dangerous World by Noah Kantrowitz

● Really great in-depth presentation

● https://coderanger.net/talks/secrets/

● https://www.youtube.com/watch?v=TVEfYO-5-RA

● Great breakdown of secret management, advantages and disadvantages of approaches and tooling

Want to know more about secrets?

82

@petersouter

Cleaning up the current codebase

83

How to find secrets currently exposed

@petersouter

Manual Grepping

84

$ git grep -i -e

"(api\\|key\\|username\\|user\\|pw\\|password\\|pass\\|email\\|mail

)" -- `git ls-files | grep -v .html` | cat

@petersouter

--------------------------------------------------------------------

gittyleaks' Bot Detective at work ...

--------------------------------------------------------------------

file: site/profiles/templates/rhn/RHN-ORG-TRUSTED-SSL-CERT.erb

what: Key

value: (2048

match:

Public-Key: (2048 bit)

num_of_revisions: 59

Gittyleaks

85https://github.com/kootenpv/gittyleaks

@petersouter

Scumblr

86https://github.com/Netflix/Scumblr

@petersouter

Unfortunately, there’s no silver bullet to detect leaked credentials or proper usage of encryption...

87

@petersouter

A lot of it is about process, gating and reviews

88

@petersouter 89

Sometimes the job is too big for a repository, and it’s better to migrate to a cleaner repo

@petersouter

Ensuring it stays clean

90

After cleanup, ensuring it stays clean

@petersouter 91

At a minimum, you want to make sure that what you’re doing with Puppet isn’t

making things worse!

@petersouter 92

It’s largely a people and process problem

@petersouter

Making sure security is part of your workflow, rather than an afterthought“Shift security left”

93

@petersouter 94

“To keep up with the pace of Continuous Delivery, security must “shift left,” earlier into design and coding and into the automated test cycles, instead of waiting until the system is designed and built and then trying to fit some security checks just before release. In DevOps, security must fit into the way that engineers think and work: more iterative and incremental, and automated in ways that are efficient, repeatable, and easy to use.”

- DevOpsSec: Delivering Secure Software Through Continuous Delivery, Jim Bird

Shifting left!

@petersouter 95

How do we check things aren’t getting worse?

● Game days and internal evil attempt teams● Continuous security integration (Gittyleaks/code-review)● Dedicated security stories for sprints

○ Evil users or (mis)use cases● Embedded security team members● Dedicated audits on sensitive apps/stacks from external firms

@petersouter 96

Game Day example: Agent spoofingLet's say someone gets access to an agent.

What’s the worst they can do?

@petersouter

As soon as security becomes a blocker, you’ve lost!Security has to be automated where possible, otherwise we’re back to the throw-over-the-wall problems of pre-DevOps!

97

@petersouter

SummaryWhat have we learnt?

98

@petersouter

Remove sensitive data from your logsUse the new sensitive type or write custom providers

99

@petersouter

Use the roles and profiles pattern for original defaultsSeparate secrets, reduce the surface area for problematic setups and always read the docs before using Puppet modules

100

@petersouter

Use Trusted Facts in your hiera hierarchyMake sure hiera lookups can’t spoofed

101

@petersouter

Encrypt the separated dataHiera-eyaml or a dedicated secret server

102

@petersouter

Ensure your code stays cleanPeople, processes and automated testing

103

@petersouter

Move security leftMake it a part of your process, rather than an afterthought

104

@petersouter