agilelessons scanagile-final 2013
TRANSCRIPT
Pieces of the Big Agile Puzzle -problems and solutions
11.11.2013
ANTTI VIRTANEN+358 44 507 0050 [email protected]
Agenda
1. The context and background1. Ammunition for ad hominem arguments2. The projects we will talk about3. Lean and agile4. What is a big agile project (our context)5. Some relevant human limitations6. Things they didn‟t teach you at the university
2. Fundamental principles to follow1. The foundation you need2. Some unsatisfactory solutions
3. Actual problems and applying the principles4. QA and throwing stones
Who? What experience?
Short background
About meProgrammer and a jack of all trades for some time.
Hobbyist -> Professional developer -> Researcher -> Developer -> Architect
About Solita250 employees. Enough to make serious things happen.
Growing, succeeding, profitable, constantly improving.
Expert at making complex tailored business software.
Serious capacity also at BI/DW, integration, consulting and UX.
The projects (in TOP-3 for Solita)
ERP (2007-2010) Kirre (2010-2013)
Multiple teams Yes. Also multi-site. Yes. (multi-site)
Long build times Hours :-( Semi-solved.
Quality pressures Yes. Delayed much. Extreme. Delayed a a bit.
DB tables (oper.) Approx. 800 Approx. 300
Codebase LOC About 1 million About 300k
Parallel work? No. Yes,after a fashion.
Happy developers? No.. :-( Yes.. mostly
Complex domain? No, but unfamiliar. Yes. And unfamiliar.
More about the ERP project
Solita‟s “death march” projectUnrealistic timetable.
Arrogance on our part.
A Pyrrhic victoryWe possess a certain amount of sisu.
There were no good books or guides to follow.
My reason for being here today.
Picture: Sisukas Nainen @ youtube
Kirre, the second big one..
Kirre keeps track of all land property ownership in Finland.Problems would make Big Money very angry. Failure in this project would have been fatal to Solita as a company.
Keys to successSisu and stubborness again. Defeat does not exist in this dojo. Less arrogance. Hired professional coaches.Books had been written. Studied a lot.Not brute-forcing. Reshaped “process” totally.
This feels like a real victory.There were issues, but this is something I can be proud of.
Agile and lean.
“Lean architecture and Agile feature development are not about working harder. They are about working „smarter‟ in the academic or traditional computer science senses of the word „smart.‟”
Coplien, Lean Architecture
Big project
What is different?Why can‟t humans cope with it?What new skills are required
What is a big project? How is it different?
Bigger system poses technical challengesLonger build times, slower test runs, complex deploymentsToo many details.Some documentation is required.
People issuesMultiple teams -> management and communication challenge.Treating people as resources does not work.Training new people is seriously expensive. Motivation will falter after three years with no productiondeployment.
Human limitations
Some well-understood limitsContext-switching is not free.Eight hours per day available for work. At most.Communication is unreliable, unpredictable and time-consuming. There are limits to handling complexity
Social limitations and issuesThere is no ”team” of 25 people.Team formation is tricky
We are quite irrationalRational choice is an illusionWe are geared towards short-term profits and instant rewards.
Engineering is not enough
Programming
UI design
Testing
Marketing Systems thinking
Psychology Mathematics
Philosophy Drama and acting
We value things on the left.. in a big project things on the right even more!
Fundamental principlesThese are simple principles. But not easy.
Decoupling Composition over monolithic ideasAvoid up-front decisionsSeparation of concernsCreating MVPs
Parallelism is the paramount concern
Teams move simultaneously toward a shared goalA team must have a subgoal (purpose)
Do not hinder with accidental obstacles
Pull value, don‟t push (be lean)Single PO or synchronization is push.
Pushing does not truly scale.
Central authority is also a risk factor.
Avoid monoliths,favor composition
You can build a huge monolith in parallel…
It‟s a design and engineering decisionFunctional programming
OS products and frameworks often
Decoupling functional specifications in a moment..
Avoid up-front decisions
When is the ”last responsible moment”?Who knows.. but choosing technology up-front is too early
Pushing the moment further away:1. Paper prototypes. (sketching function)2. Initial boundaries and API (Form follows function)3. Iterate and write some code. (live form)
(For this path I recommend Coplien‟s book Lean Architecture.)
Separation of concerns
Avoid leaking implementation detailsExternal API – UI – data model
Tool selectionA big project does have very special needs!Automate everything (that DevOps thing). Composable tools.
Rethinking toolsMaven is a repository, not a build tool.Jenkins is UI for cron, not a CI tool.
Aim for MVP at all levels
You absolutely must postpone features.But which ones are nice-to-have ?
You need feedback early onCreate a minimal PoC implementationApply this to programming, design, specifications etc.
A ”big picture” is too big to seeThe MVP/prototype makes it clear as a side-effect!
Be wary of easy solutions
”Solutions” that don‟t address the fundamentals
Technological miraclesDecoupling and separation of concerns (SOA, REST , ESB)Opinionated frameworks (Rails)Management Tools
”Hiding” complexity (BPML, TOGAF, MDM)Simplified recipes
You can‟t ”Teach Yourself Lean in 21 Days”! Preposterous!
To be fair, some of these have value.. But you really can’t solve a human problem with technological magic tricks!
Principles in real world(What did we actually learn?)
Conditions of greedy selection no longer hold for the backlogDecoupling of functional specifications Normal form in relational database considered harmfulFocus on results, not tasksTeams around functionality, not layers and technologyLeadership is not management
How big is the elephant?
Three ways to search the problem space:Depth-first (to measure complexity and risk)
Breadth-first (to get an overview)
Explorative mapping (black art, you may get interestingfindings)
I highly recommend balancing between all threein a big project. Early on!
Backlog priorization
What is the most valuable item?The most risky one?
-> reduce risk
The one with most direct value to end-user (lean maxim)?-> customer value
Getting a core workflow done in it‟s entirety? -> feedback and reduction of risk
In a big project it is a good to rethink the criteria.
Decoupling functional specs (case Kirre)
Three features: Work queue, search and application handlingSeparate from the user‟s point of viewSo parallel work comes naturally?
Why not? Some bad reasonsThe database is a monolithic normal form design.The dev tools do not support this.The build pipeline tool doesn‟t allow it.The specification team (the ”architects” or ”PO-team”) is a separate silo.
We had limited success decoupling these things. But it is definitely the right direction to go!
The relational database posse is lacking
The RDBMS mindset (for waterfalls)A static view of the end state. The ”normal form” is by definition a monolith!
Relational model is not broken per se But ”agile developer” is not served by the vendors.Normal form doesn‟t deal with uncertainty and agile.Big projects are especially vulnerable to these issues!
Part of the NoSQL hype is a result of this attitude.
All models are wrong. Some are useful.(actually we retain normal form here)
Application
Queue Assignee
Applicationgroup
DecisionApplication
typeProperty
Queue_view
UI for queue
handling
Views are great!• Early feedback• Parallel development• Hidden complexity• Decoupling
Everyone. Must. Focus. On. Results.
Bad projects (tasks) Good projects (goals) The elite (results)
What is the next task today? What are you going to work on today?
What are you going to accomplish today?
How many tasks left in the backlog? Is the invoice integration ready?
No - when will it be ready?
Does the user have all essential invoicing functionality?
No - when will it be deployed?
Are we doing what was specified in the contract?
Did we bill all the hours?
Are we heading to the right direction?
Is the customer happy?
Are we using the resources (time and money) in the best possible way?
What is the most valuable thing to do next?
The amazing Kirre project tool (data migration)
Team‟s purpose
Short term plan (Kanban style)
backlog
Test plan
The purpose statement helped maintain focus and prioritize backlog.
About team organization
Tempting the dark side isIntegrations – back-end –UI…
Where‟s the end-user? Where‟s the use-case?
Ok, use-case/scenario based then?You need that cross-functional team. Got one?
You cannot force team formation!
The use-case based view is the right way. But not easy.After many changes we reinstated the integration team.
Leadershipment (case Kirre)
A single project manager, about twenty peopleA bottleneck, not parallel! A risk!
When faced with a serious situation. (final deadline)Divided the responsibility (leadership) to five individualsNo reports (or management duties) requiredNo holds barred, bare fist prison rules.. Just make it happen
Did it work? It was total pwnage and salvation of the project! The Great Enablers
For each team: Autonomy – mastery - purposeSupport and coaching. A scrum master who was not part of the team.Big picture priorization over teams. Together with everyone.
To “make it so”, important considerations
Voluntary. If a sane person commits, it‟s a good sign.
You must delegate power and freedom too. You need a manager who is not insecure!
Leader must have some respect from peers.Leadership is based on ”social influence” by definition.
A title is not related to this.
It’s about getting things done.It‟s not a ”tech lead” role.
Summary
To summarize.. How to make it alive..
Flexibility in scope, budget and timetables are required.Also the customer must accept this and trust you.
Respect the issues related to sizeMotivation, complexity, human limitations.
No single architect, PO, manager to make all decisions.
Do not ”manage” and control, lead insteadGood employees don‟t need ”management”
Get real cross-functional teams. Motivated & autonomous.
Stuff that victories are made of
Lean Agile
Continuous feedback
HonestyTransparency TrustMotivation
Empowered teams
Parallel development
End-user needs
MVP first Form follows function
Compositional designContinuous Integration
Continuous Delivery
Realistic shared vision and plan (backlog)
Continuous innovation Continuous flow
Automated tests
Use-cases in backlog
Resources and further reading
Even more booksLean From The Trenches
Lean Architecture
Lean Thinking
Lean Software Development: An Agile Toolkit
The Toyota Production System
Rational Choice in an Uncertain World: The Psychology of Judgment and Decision Making
Nonviolent Communication: A Language of Life
Systems Thinking
