devops (start walking in the same direction) by ops
TRANSCRIPT
DEVOPSSTART WALKING IN THE SAME DIRECTION
WHO I AM?
Demis Rizzotto
System Engineer,DevOps Linux Engineer @Exxoss
Currently consultant @Lampiris
THE PROBLEM OF DOWNWARD SPIRAL
DIAMETRICALLY OPPOSED GOALS AND INCENTIVES
DEV▸ Respond to the rapidly
changing competitive landscape
OPS▸ Provide stable, reliable, and
secure service to the customer
THE CORE, CHRONIC CONFLICT WHEN ORGANIZATIONAL MEASUREMENTS AND INCENTIVES ACROSS DIFFERENT SILOS PREVENT THE
ACHIEVEMENT OF GLOBAL, ORGANIZATIONAL GOALS
FRUSTRATION
▸Unnecessary process steps
▸Unnecessary rework
▸Unnecessary features
▸Waiting for someone else to act on an action or task
▸Waiting for an approval
▸Waiting for an environment
▸Creating a ticket for a manual task
▸Reporting to management by manually updating spreadsheets
▸Manual status reporting
▸…
RESULTS
▸Fragile application are prone to failure
▸Ever increasing backlog and technical debt
▸Urgent security rework and remediation
▸More urgent projects into the queue
▸Big deployments very difficult to diagnose when fail
DEVOPS
More features quicker Stability
DEVOPS
DEV OPS
FACILITATE AND AUTOMATE DEPLOYMENT
HELP DEV TEAM BECOME AUTONOMUS
FACILITATE DIAGNOSTIC
GET DEVS INTERESTED IN « PROD STUFF »
GIVE ACCESS TO LOGS
GIVE ACCESS TO MONITORING
OPTIMIZE LOGS
DEFINE METRICS
QUALITY TESTS
MONITOR YOUR APPS
DEV IN DEVOPS
DEV IN DEVOPS
IMPROVEMENTS
▸ The ability to accurately track dependencies
▸ Services that are resilient and degrade gracefully
▸ The ability to archive data to manage the size of the production data set
▸ The ability to easily search and understand log messages across services
▸ The ability to trace requests from users through multiple services
▸ Simple, centralized runtime configuration using feature flags, …
DEV IN DEVOPS
▸ Make Dev responsible for their own services
▸ Integrate Dev into the incident management processes
▸ Have Dev cross-train Ops
▸ Embed Ops knowledge and capabilities into Dev
▸ Design for IT Operations
EMBED IT OPERATIONS INTO DEVELOPMENT
OPS IN DEVOPS
OPS IN DEVOPS
IMPROVEMENTS
▸ Self service
▸ Infrastructure as code
▸ Standardize infrastructure
▸ Make all infrastructure data visible
▸ Make application info data visible
▸ Modify the incident resolution process and blameless post-mortems
▸ Monitor the health of the deployment pipelines
OPS IN DEVOPS
▸ Single “repository of truth” containing both the code and environments
▸ Create the one-step Dev, Test and Production environment build process
▸ Extend the deployment pipeline processes into production
EXTEND DEVELOPMENT INTO IT OPERATIONS
WE ARE ALL DEVOPS
WE ARE ALL DEVOPS
▸ Stop think . Do
▸ Share responsability
▸ It’s Never Someone Else’s Problem
▸ Dedicated Designated Ops in every team
▸ Different type of devops, but devops is possible for all type and size of companies
▸ Iterate!
▸ Identify bottleneck and remediate
▸ Define small target and share results
DEVOPS IS FOR ALL
DEVOPS IS FOR ALL
DEVOPS IS FOR ALL
▸ AUTONOMUS TEAMS
▸ Trust but verify
▸ Local autonomy, but global consistency
▸ SELF SERVICE
▸ How ? API, web portal, infra as code , AWS accounts, …
▸ AUTOMATE
▸ If you are tired to do it manually. Automate it.
▸ REDUCE NON-VALUE ADDED
▸ If isn't your business it's maybe better don't do it your self
KEY OF SUCCESS : REDUCE FRUSTRATION
INFRASTRUCTURE
ON DEMAND ( IAAS )
EASIER TO REBUILD THAN TO REPAIR
INFRASTUCTURE
INFRASTRUCTURE
IMMUTABLE INFRASTRUCTURE▸ Automate the setup and deployment for every part and
every layer of your infrastructure.
▸ Never change any part of your system once it is deployed. If you need to change it, deploy a new system.
AVOID HISTORY
INFRASTRUCTURE
INFRASTRUCTURE AS CODE
Versionning and code review
Easier track changes
Bootstrap your infrastructure from scratch
Automate
CI/CD
TESTING
TESTING
CONTINUOUS TESTING
▸ Automated tests
▸ code quality
▸ Unit tests
▸ Stress tests
▸ Don’t forget to test infrastructure (load banacer, recovery etc.)
▸ Extend you build pipeline
SECURITY
SECURITY
INFORMATION SECURITY AS EVERYONE’S JOB, EVERY DAY
▸ Updates
▸ Use secrets management
▸ Integrate security into our CI/CD pipeline
▸ Check code libraries and their recommended configurations
▸ Dependency scanning
▸ Static analysis
▸ Dynamic analysis (ex: OWASP Zed Attack Proxy + Selenium)
SECURITY
SECURITY METRICS MONITORING
▸ Successful and unsuccessful user logins
▸ User password resets
▸ User email address resets
▸ Database syntax error
▸ User credit card changes
DEPLOYMENTS
DEPLOYMENTS
DEPLOYMENTS ARE ROUTINE AND PREDICTABLE
No stress . It’s a routine.
On business day when everyone is already in the office and without our customers even noticing
DEPLOYMENTS
HOW▸ Work in small batches and delivering quickly and
frequently
▸ Blue green deployment
▸ Feature toggle
▸ Canary testing
▸ Dark launch
▸ Automated deployments
MONITORING
ALERTING
▸ Mesure service health not only server health (Health pattern)
▸ Reducing Alert Fatigue
▸ Cut alerts that aren’t actionable
▸ Adjust thresholds
▸ Consolidate related alerts
▸ Give alerts relevant names & descriptions
▸ Make sure the right people are getting alerts
▸ Multiple notification rules and choose right channel
▸ Use incident management tools(Pagerduty, Victoryops,..)
MONITORING
IF IT MOVES, TRACK IT
ANOMALY DETECTIONTRENDS
APM
End user monitoring
Application Performance Management
DELIVERY PIPELINE METRICS - TEAM LEVEL DASHBOARD
LOGS
LOGS
▸ Set your log severity right
▸ Remember you will not be the only one reading these logs
▸ Track your communication with other systems
▸ Log exceptions correctly
▸ Use an ID to track your events
▸ Normalize yours logs
▸ Separate and Centralize your Log Data
LOGS
DOCUMENT, MEASURE, COMMUNICATE AND COLLABORATE
DOCUMENT
WRITING EFFECTIVE DOCUMENTATION
▸ Treat documentation like a requirement
▸ Put the Information in the Most Appropriate Place
▸ *Keep your documentation close to the code
▸ Keep documentation just simple enough, but not too simple
▸ Avoid creating large volumes of documentation of questionable detail which become obsolete shortly after they are written
MEASURE
MEASURE PROGRESS
▸ KPIs
▸ Deployment frequency
▸ Deployment speed
▸ Failure rate
▸ Time from code to prod
▸ Share goals
▸ Increase the visibility of work
COMMUNICATE
▸ Find better ways to communicante
▸ Promote day to day direct communication
▸ Coffee machine meetings
▸ Chat (Slack, HipChat, Team,…)
▸ Use video (Slack, Skype, Hangout,..)
▸ Quality over quantity
▸ Post Mortem meetings
▸ Bootcamps,…
IN DEVOPS CULTURE, COMMUNICATION AND COLLABORATION ARE KEY
COLLABORATE
TIPS FOR BETTER COLLABORATION
▸ Change the seating and rotate people through development
▸ Improve the environment for collaboration (Breakout areas, whiteboards, nice meeting rooms, a decent coffee machine)
▸ Cross domain workshops for analyze your process
▸ Block time for process improvement work (lean)
▸ Run lunch and learn / brown bag sessions
▸ Put a persistent chat room in place
