scaling continuous delivery to minimize risks (delivery of things 2016)
TRANSCRIPT
Scaling Continuous Delivery to minimize risks
Why Continuous Delivery?
For Speed
Speed is a lever to mitigate risk
Typical cycle of software delivery
HypothesisGeneration Development Experimentation
Validation
How do you scale it?
Foundational elements to scale it
ORGANIZATION
TOOLS
PROCESS
CULTURE
CONTINUOUS DELIVERY
Organization
Spotify Org Model
Takeaways
● Tribes : Optimize team composition for autonomy
● Alliances : Organize team groupings for alignment
● Guilds : Foster channels for knowledge sharing to break silos
● Tribes mapped to metrics : Have explicit accountability by
assigning team metrics
Story of our Aha! moment
Real story anyone?
Series of aha!, aha!, aha!, aha! ...
There were 4 phases
The startup phase
● 1 location
● 50+ engineers
● Teams mapped to features
Teams = Features
The expansion phase
● 1 engineering hub in Stockholm
● 2 small engineering offices in NYC and Gothenburg
● 100+ engineers
● Teams still mapped to features but shared ownership across
geographies
The autonomy phase
● Global company with 2 engineering hubs in NYC and Stockholm
● Multiple smaller engineering offices
● 250+ engineers
● Geo local missions and full local autonomy
Teams Tribes = Missions
The alignment phase
● Global company with multiple engineering offices
● 500+ engineers
● 10+ autonomous missions
● Many missions working towards same or similar metric
Teams Tribes Alliances = Related Missions
One more thing ...
Spotify Org model Universal Org Model
Tools
Make pipelines fast and low-touch
Enable ease of deployment
● Think Containers to enable easy deployments
● Standardize production and test setup to enable easy build
progression
Make tests visible
GitPR
Pre-mergeTests Pass Post-merge
Tests Log to TRS
View TestResults
Optimize build times
● Minimize recompiles of the whole dependency tree
● Pre-compile common dependencies and store as binaries
● Think hermetic builds
Maximize Automation, Monitoring and Alerting
Make experimentation part of the delivery flow
Feature Flags rollouts AB Tests
Process
Standardize Release Management
Feature Complete
Beta Testers & Apple / Incremental Rollout to
Android
Master Branch
Release Branch
Release Blocking Bug Fix(es)
2 week client release cycle
Enable Speedy Reviews
Review flow
Software DesignReview
SoftwareDevelopment
CodeReview
Pre-releaseReview
SoftwareDesign
SoftwareRelease
Speedy Reviews at Spotify
● Code merges require +1 during code review but aren’t blocked● Changes are tracked for auditing purposes● Software review is front-loaded to avoid delivery blockers.
Encourages thinking through completely about architectural setup, complexity and dependencies
Have a company operating rhythm
Culture
Quality is owned by everyone but managed by QA
Agile is not a process but a mindset
Be explicit about beliefs and expectations
