continuous delivery and continuous agile by andy singleton - agile maine day 2016
TRANSCRIPT
Continuous Deliveryand Continuous Agile
From Andy Singleton, http://continuousagile.com
www.assembla.com
The secret weapon that Silicon Valley is using todisrupt and destroy competitors
• Retailer X deploys changes to their monolithic online ordering app once every six weeks. Ops holds for three weeks to make sure the complete system is stable.
• Amazon has thousands of services and more than 1000 service teams. They release something about once every 11.6 seconds. In the time that Retailer X takes to try one new release, Amazon has made 100,000 changes.
• Amazon hosting competitor: “It’s an emergency”.• 6 Apex Predators in the top 10 public companies worth
$2.4 trillion: Google, Amazon, Facebook, Microsoft, GE, Apple
Survey on Continuous Delivery
46% think their competitors
have adopted Continuous
Delivery
Problems solved by Continuous
• Stressful releases
• Speed to market• Competitiveness• Quality
• Scaling – building large systems
Not Scrum
• Continuous improvement versus CadenceUnblock your teams
• Web services versus monolithsSmaller, faster testing, faster releases
• Technical practices, not team practicesHUGE productivity gains from automation
• Tech leads versus self-managing teamsOne leader, ready to build services
• MAXOS versus SAFeScale different
改善1. リリースを頻繁にする2. 改善する
/ 月割りのリリース
Kaizen
1. Release more frequently2. Improve
Continuous Agile Agenda
1) DevOps - Continuous delivery of software – Build/Test/Deploy
2) Continuous, metric-driven product management
3) MAXOS enterprise management
4) Making the Move
Continuous Deliveryor “DevOps”
Scrum SprintValidate and
Sort DeliverablesSelect a
Sprint Plan
Completed = observed
velocity
Expand stories and estimate
tasks
CollectIdeas
Close Sprint
Into next sprint
Work on tasks
Scrumban Iteration
Validate and Sort Deliverables
Pull DeliverablesWhen Ready
CollectIdeas
StabilizeRelease
Into next release
Feature Freeze
Triage
Idealized ScrumBan Release
Copyright 2012 Perforce Software, Inc. - Assembla, Inc.
Kanban / ContinuousValidate and
Sort DeliverablesPull Deliverables
When ReadyCollectIdeas
Continuous Release
Stay within WIP limit
Release Features Continuously
Program Launch
Releases
Plan Program Test Doc Deploy
SkipAutomate& Blend Lag Automate
Pull
CI & CD
End up with
Waterfall to Continuous
Measure
Code Contribution PatternsManage code if possible. People are hard to manage and can’t be automated. They want to contribute.• Centralized continuous delivery
– No branches, finds and fixes problems as early as possible• Distributed continuous delivery
– Release every change with its own branch and test• Temporary branches
– Combines benefits of centralized and distributed• MAXOS
– Use centralized continuous integration to manage a massively scalable IT system
Centralized CI/CDContributor Commits – “as early as possible” to find problems
ContinuousIntegration tests
Fail - alarm
Release CandidateTest System Release
QA Testing
Pass
Continuous delivery at Edmunds.comwith PERFORCE
The big question: How to test?
• We release software in batches so that we can test it.
• In continuous delivery, we might get as little at 10 minutes to test a release candidate
Test Layering
Monitor your released software: Errors, Usage volume, usage patterns, user feedback
QA System with Human test consultants
Code review: Both a manual test, and a place to ask for test scripts.
Continuous integration: Run automated tests before using human review time
Unit tests in the development environment
Switch new features and architectureStart h
ere to
add layers
Start here to
release a change
More frequent releases can increase quality
9 (sparse) Layers at Edmunds
Feature Switch and Unveil
HiddenProgrammer sees a change locally. Change is tested in the main version but not seen.
Test Story Owner and testers see the change on test systems.
BetaInsiders see it and use it. Story Owner can show it to selected users for feedback or A/B testing.
UNVEIL! The big event. Communicate with all users. Measure reaction.
One code version
No special test builds
No long-running branches
Role: Developer
• Developers have more power and responsibility.• Developers have more responsibility for testing.• Developers (not QA or PM) decide when to release.
This is a strong finding.• Incentives are correct. Developer might have to
come back from Friday night beers to fix a problem. This provides a motivation to make good decisions and automate testing.
• Features can be released but hidden. Product Managers and Marketers will unveil when they are ready. Unblock!
Programmers approve releases, not QA
Role: IT VendorOld Agreement• Fixed Price• Fixed deliverables• Large probability of late delivery
New Agreement• Fixed price• Fixed Time• Variable deliverables
– Small guaranteed deliverable– Weekly meetings to agree on improvements
Advantages• No late delivery• Deliverable is better
than original specification
• Start sooner and finish sooner
Role: Product Manager/Owner
• Batch -> Continuous
• Requirements -> User Experience
• Strategy -> Measurement– Usage measurements are so important, so
underutilized– Double your productivity
Batch Agile Continuous
Explains the resurgence of vertical integration(eg Apple, Tesla). Requires CI
Making one good change is not a team sport
Product Owner -> Story Owner
What you realize when you begin to be very data-driven is a very large percentage of our ideas are bad.
Microsoft Product CTO Adam Pisoni
Measure everything• Measurements are power. They are impossible to ignore. They
win arguments.
• Double your development capacity for zero extra cost! Users ignore 50% of all new features. If you can find and ignore these features, you double your capacity.
• Measure everything possible about usage. Easy with an online service.
• Collect measurements from installed products and optionally transmit them.
• Use your marketing department. They measure.
Make Product, not Quality
ContinuousRelease
InventoryRelease
Improve Quality
Improve Product Usage
Matrix of Services(MAXOS)
Breaking the scale barrier
The Services MegatrendDesktop Web App Cloud
Services
App
DB
Service
Service
Service
Scale it like Google
• 15,000 developers, 5,000 projects, one current version of the code (2013). They can go from an idea to a release in 48 hours
• Vast Internet system divided into thousands of "services"• Most programming done by teams of 3-4• Centralized process with single version of the test system – run
100 million test cases daily• Before developers release a change, they test it with the most
recent version of all the other services. If a test script finds conflicts, it tells developers who to contact to resolve them
Matrix of Services - MAXOS
PrioritizedBacklog
CurrentWork
Each team releases
when ready
Hundredsof releases
per day
Service team Productionservice
Service team Productionservice
Service team Productionservice
Feedback on speed, errors, usage, and requests
Test as one system
Integrationtest env
Integrationtest env
Integrationtest env
Coordinate without big meetingsContinuous Integration between latest dev version of each service
• Continuous integration helps teams coordinate.
• See dependencies between “producers” and “consumers”
• Errors and conflicts show related team contact info
• Meetings and changes negotiated between two teams, not many
PrioritizedBacklog
CurrentWork
Service team
Service team
Service team
Integrationtest env
Integrationtest env
Integrationtest env
Machines can replace layers of management
Teams are largely self-managing
PrioritizedBacklog
CurrentWork
Service team Integrationtest env
Up to 50% of workfrom backlog
At least 50% of work is self-plannedProblems get fixed quickly
Productionservice
Productionservice
Productionservice
Feedback: quality, reliability, speed, user support
Productionservice
ProductionServer
Sense, respond, self manageminimize planning
Hubspot – Great at Mid-scale• Transformed a monolithic app to 200 services over one year• 3-person programming teams. Each of 20 teams is
responsible for about 10 services• Dev teams responsible for design, programming, testing,
release, monitoring, and responding to production problems. No full-time QA. Shared PM and UX. 4 Ops guys for 2000 servers.
• Lot’s of tooling and dashboards to help teams deploy, manage, and monitor their services
• Feedback from customer support also grouped by team
Team Building
PrioritizedBacklog
CurrentWork
Each team releases
when ready
Service team Productionservice
Service team Productionservice
Add capacity fast by building aroundSingle-function programmer/tech leads
Integrationtest env
Integrationtest env
Teams are not permanently multifunctional
Ways to Scale
Scrum + SAFe• Add more hierarchy• Complex multifunction
teams• Hold big meetings and
teleconferences• Block everyone into
one cadence• Coordinate big releases
Top Tech CompaniesAutomate management,
as well as testing and deployment.
Dev-lead teamsCommunicate peer to
peerUnblock teams to move as
fast a possibleRelease more frequently
SAFe versus MAXOS
How to move to CD and Services
• Measure• Release more frequently• Assign blame and other organizational and
authority hacks• Automate• Organize the fast layer• Reverse Conway’s Law• Manage products, not programs
Measure
People respond directly to measurements. They are very difficult to ignore. They are also MORE IMPORTANT than internal process discussions.• Usage and customer value• Errors• Fix times• Release frequency
Release more frequently
1. Release more frequently2. Improve
Authority Hacks
Training, persuasion, browbeating, and “culture change” will not work. They don’t change the game. You first change the game. Then, people figure out how to play the game and win.• Need automated tests? Mandatory code review• Need higher quality and shorter test cycles? Force
programmers to push the “release button”, not QA• Stop your boss from launching big waterfall
projects? Give him monthly budgeting, not annual.
Code Review gets you tests
AutomateIf the machine works, people will use it
Core ITannual budget
Reliability & security mission
WebAPI Layer
Mobile SaaS / Cloud
Marketing
Fast (Awesome Mode) ITmonthly budget
Mission to respond to opportunities
Service team Productionservice
Service team Productionservice
Productionservice
Integrationtest env
Integrationtest env
Integrationtest env
Fast IT
Core IT (stable service)
Reverse Conway’s Law
Boss
Dept A Dept B
PartA
PartB
Service A1
Service A2
Service B1
Service B2
Team A1
Team A2 Team B2
Team B1
Manage Products, not Programs
• Strong product managers deliver value to users as directly as possible
• The Matrix of Service teams respond to product managers. The front end teams provide the product. They act as “consumers” to pull what they need from back-end teams
• Service teams support multiple products, dissolving the program hierarchy
Product Matrix
Boss
Dept A Dept B
PartA
PartB
Service A1
Service A2
Service B1
Service B2
Team A1
Team A2 Team B2
Team B1
People first, or Technology first?
• Your comments