puppetconf 2016: successful puppet implementation in large organizations – james sweeny, puppet
TRANSCRIPT
Successful Puppet Implementation in Large Organizations James Sweeny Puppet, Inc.
You will hear me blather about:
DevOps and Puppet
Large vs. Small/Medium Orgs
Deployment Strategies
Common Roadblocks and Pitfalls
Who am I?
This talk is not about…
● Scaling Puppet
● High Availability
● Monitoring
● Security
Section ## (if desired) and/or Subtitle
Leading platform. Datacenter standard.
Experience Founded in 2005
Scale More than 10 million nodes managed
Ecosystem Deep partnerships with datacenter titans
Customers 1,000+ enterprise customers
Community 4,200+ community-contributed modules
Users 30,000+ organizations using Puppet
Backing
DevOps - CAMS
● Culture
● Automation
● Measurement
● Sharing
How most companies see it
● Culture
● Automation ● Measurement
● Sharing
How they should see it
● Culture ● Automation
● Measurement
● Sharing
Puppet is (just) a tool.
Puppet doesn’t…
● Do everything
● Fix broken culture
● Shuffle the org chart
● Train your staff
● Get you out of an abusive MSP relationship
But it can… ● Help you to understand your own
systems
● Provide a focal point and a Lingua Franca
● Save time and open new opportunities
● Act as a force multiplier for velocity, consistency, and reliability
Puppet code is state!
Infrastructure as Code ● Puppet code describes the entire system
(or everything you care about)
● The code is the system
● Changing the code is the same as changing the server
● Designing, provisioning, changing all rolled into one
What is a “Large” Organization?
Large Organizations…
● Complex org chart
● Separation of concerns
● Many teams
● Many distinct business functions
● Headcount, not node count
Naïve Deployment Example
Server
Application
In house software, provides business value
Server
Middleware Apache, IIS, Oracle, etc.
Server
Management Netbackup, SEP, Nagios agents, etc.
Server
OS Windows, Linux, Solaris, etc.
Server in a small company
OS
Management
Middleware
Application
Developer(s) Operations/IT
Developer(s) Operations/IT
DevOps!
OS
Management
Middleware
Application
Developer(s)
Operations/IT
Problem in Large Organizations
● Many “development” teams
● “Operations/IT” is highly fractured
● Many overlay teams
App teams
Release team
Middleware
Windows
Security
Linux
… But it’s usually worse ● Multiple variants of each team
● Windows/Linux/Solaris/AIX…
● Secure vs. Non-secure
● Different team for each Management product
● Overlays for compliance, security, governance…
● Changing ownership depending on lifecycle stages
Crazy Org Charts ● CIO
● SVP Engineering
— Middleware
— Windows Engineering
— Unix Engineering
— Backup Team
— Monitoring Team
● SVP Application Engineering
— Hundreds of app teams
— Release Engineering
● SVP Security and Compliance
● …
Journey, not a sprint
So what *does* work? (Reduce the problem domain)
Start small, shallow, and wide Option #1
Shallow and Wide
OS
Management
Middleware
Application
OS
Management
Middleware
Application
OS
Management
Middleware
Application
Single Team
Unmanaged
Shallow and Wide
● Start with one existing team (usually the one with root/Administrator)
● Do minimal work
● Get comfortable with Puppet and version control
● Don’t boil the ocean
Shallow and Wide (evolution)
● Add new teams as responsibilities expand
● Share code repository
● Central module repository
● Slowly work up the stack
Platform as a Service Option #2
Platform as a Service
OS
Management
Middleware
Application
OS
Management
Middleware
Application
OS
Management
Middleware
Application
Single Team Unmanaged (Legacy)
Platform as a Service
● Puppet is hidden from customers (App/Dev teams)
● Limited, but complete stacks provided
● Likely need to build/use an additional interface
● Exceptions to normal policies
Platform as a Service (evolution)
● Accept Puppet code contributions from other teams
● Add additional Middleware and OS platforms
● Decommission/migrate legacy servers
Which is right for us?
Shallow and Wide
● Most successful real-world
● Most flexible early on
● Easiest way to learn your org
● Gives time to evolve
● Business value not as easily apparent
PaaS
● Single team can own and slowly grow
● Very high upfront cost
● Quick wins for new applications
● Side-by-side with legacy for longer (or forever)
● No benefit to legacy fleet
Other Options
Hybrid Approaches
● Grant limited Hiera exposure
● Mix of Options 1 & 2
● Puppet as a Service
● Multiple agents
The “Puppet Team”
“Puppet Team”
● Over time, role becomes advisory
● Own the initial workflow, Puppet infrastructure, supporting services
● Become Puppet SMEs
Who is the “Puppet Team”
● Technical experts on your existing systems
● Can cut red tape
● Have root/Admin already
● Eager to learn, excited to solve problems
● Dedicated to big picture success
Likely Roadblocks
“That’s not how we do it” The internal “standard”
Battling “That’s not how we do it”
● Prefer Industry standards to esoteric ones
● Software owners must articulate reasoning
● Make your case to “SMEs”
● Use political pressure as a last resort
Plan, Build, Run Probably the reason everything is awful
Plan, Build, Run
● Diffusion of responsibility
● Conflicting sources of truth
● Destroys feedback loops
● Massively slows change
● Puppet makes it obsolete
Politics
Politics
● Have well connected people committed to Puppet
● Need strong management
● Understand other groups needs
● Be willing to prove real value before winning other groups
● “Puppet won’t steal your job”
Final Advice
More tips
● Learn/know your own systems
● Always question internal dogma
● Limit scope, standardize, leave things out
● Think “shift left”
● Don’t overbuild!
Must See Presentations
● Charlie Sharpsteen on performance tuning, 1:30PM Terrace Salon
● Martin Jackson on collaboration culture, 11:15AM Friday, Grand Hall
● Russ Mull and Zack Smith on Puppet HA, 2:30PM Friday, California room
● Multi-tenant station in the Exhibit Hall