There is no magic

Bob Martinbobmartin08008@mac.com

“I assume you will show the alternative ways to use a cage, a whip, and a chair to manage software teams?” - Stu Feldman, VP Engineering Google

About Me

• Built a lot of software ~100M lines of code– System software

• Operating systems• Programming languages• Transaction control

– Network management– Service control points– Embedded systems

• Packet switches• Network equipment• Testing equipment

• Love technology– EE and CS trained– Life long passion – now neural science, molecular & evolutionary biology

• Help me convince Al to work in computational biology!

• Not expert in contemporary software technology/tools• Conned into this talk by a great friend of 42+ years

Facts of Life

• Failure is the norm– But it needn’t be

• Error correction costs increase exponentially during the lifecycle• Programmers are a special breed*

– They like machines not people– They have high need for approval/recognition– They must respect the leader– Beware, complicators and prima donnas lurk in the weeds

• Software engineering is complexity management– Race with technology/tools– Complexity is still ahead

• Good software engineering is old hat– But is not always used - why????“Two things are infinite: the universe and human stupidity; and I'm not sure about

the the universe” - Einstein

* “Psychology of Computer Programmer” - Gerald Weinberg

Why Software Projects Fail *

Average overrun: 89.9% on cost, 121% on schedule, with 61% of content

* Barry Boehm - A View of 20th and 21st Century Software Engineering

Increase in Software Cost-to-fix vs. Phase (1976) *

10

20

50

100

200

500

1000R

elat

ive

cost

to

fix

def

ect

2

1

5

Requirements Design Code Development Acceptance Operationtest test

Smaller Software Projects•

Phase in which defect was fixed

10

20

50

100

200

500

1000

2

1

5

Requirements Design Code Development Acceptance Operationtest test

•

* Barry Boehm - A View of 20th and 21st Century Software Engineering

What Year Was This Conference?

Design• Guidelines

– External function, e.g., user language– Internal function, e.g., parsers

• Techniques– Deductive or inductive– Modularity and interfaces– Complexity control

• Proscriptions– Completeness, modularity, efficiency– Self-monitoring and performance improvement– Incremental systems– Balance

• Security, control, … vs. cost• Limited goals to attain excellent performance

• Design problems– Data structures on system design– Fixed resources– Cooperating processes

• Documentation

Production• Organization for producing software

– Number and quality of people– Structure of large groups– Control and measurement– Internal communication

• Production techniques• Monitoring

Service• Distribution

– Media

• Acceptance criteria• Documentation• Adaptation• Maintenance

– Error detection & reporting– Response and distribution

• Documentation• Performance• Feedback into design

NATO Software Engineering Conference - 1968

Impact of Increased Discipline

• Low software engineering maturity results in:– Reduced productivity (35%)*– Late detection of defects (22%)*

– Longer time to market (19%)*

– Higher post-release defect rates (39%)** median annual improvement

CMM** process maturity

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

1 2 3 4

Company XIndustry-wide

• Takes several years of focused improvement to become a high-performance software team

• Focus on the programmer– Just management: well-controlled junk

• Delicate touch to avoid process imprisonment

CMM Level

** Software Engineering Institute’s Capability Maturity Model (CMM)

Learning• Disasters

– Mine– Others

• Really good companies– IBM - Discipline & inspections– Sun - Tempo– Japan - Quality circles & reuse– Microsoft - Reuse– Apple - User centered design & tool kits, “Design of Everyday Things” - Don Norman

• Really smart professionals– Al Aho – Swamp drainer – Fred Brooks – “Mythical Man Month,” “No Silver Bullet”– Barry Boehm – Estimation and cost of errors– Michael Cusumano – Software factories – Edsger Dijkstra – Importance of time in systems– Watts Humphrey - Encapsulated team and personal discipline - “TSP, PSP”– Martyn Thomas - Formal methods– Ken Thompson - The programming craft

• Loop Assignment System– Univac system– Failed trial

$250 million down the drain Demoralized, weak team

– Senior 18-month commitmentNo architecture or requirementsNo technology infrastructure

– Working “temporary” subsystem

Unix based

• Loop Maintenance System– IBM MVS system– PDP 11/20 backup– 8 Months to go

No scheduleNo performance planNo requirementsWeak management

Two Troubled Projects

Univac 1100 with C

“Temporary”Subsystem

“RobustPipe”

IBM 370

PDP 11/70

RepairServiceBureaus

Switch

Test Head

PDP 11/20

Building The Team• All - domain expertise, discipline & simplifiers • Team leads – programming, lateral & motivational skills• Architect – deep system software & computer science

– Great ones are really rare“Everything should be made as simple as possible ... but not simpler” - Einstein

• Specification - user-centered design & customer skills• Design/programming – proud craft persons

– Read great books to learn to write, read great programs to learn to program“Lion’s Commentary on Unix” - John Lions

• Testing – sadists & analysts • Performance – back of envelopers• Quality - superb communicators• Tools & environment – geeks emeritus • Product manger coordinated extended team

– Sales, marketing, pricing, legal, service, …

“Executives owe it to the organization and to their fellow workers not to tolerate nonperforming individuals in important jobs” - Drucker

Process is not a four-letter word

R1 R2 R3

Iterative Definition/Development

Integration, Test, & Trial

Phased Deployment

• Early on– Architecture– New technology/risk– Performance– Factory operations

HandoffsAutomationConfiguration control

• Later with increasing process discipline

– Features– Quality

“I had the world’s greatest group, and I should have made the world’s two greatest groups. I didn’t realize there are benefits to having real implementers and real users” - Alan Kay

• Iterative by many names– Build a little, test a little, refine requirements a little– Entrance/exit criteria

• Templates & best cases– Lightweight– Web

• Inspections & reviews– Requirements, design, code, test, etc– Lightweight to heavyweight– Especially on tough stuff and with new people

• Lead customers– User-centered design

• Screw-up early detection & action– Jeopardies and speaking up– Audits with smart friends

What’s your favorite bug?

Architecture

“Controlling complexity is the essence of computer programming” - Kernighan

• Top-down decomposition of a complex problem to allow independent, bottom-up development and change

• Goals– Changeability/customizability– Chunk independence

• Reduce n2 effects

– Inject huge-payoff CS technology• Tiny languages, protocols, finite state machines, search, algorithms & data structures, formal methods, etc

– Performance– Reuse

• Check it– Prototype– Audit with smart friends

• Crumble with time– “An Architecture History of the Unix System” - Feldman Usenix ‘84

Calling the shot

• The true art– Best to wait until specification/design done– Domain expertise crucial– Sizing models help, but not magic (still +/- 30%)“Let the numbers do the talking” - Don Reifer

• Aggressive but doable• Starvation not gluttony

– Reduce communications, complexity & feature creep “Whilst it is true a large programming project might require a large programming team,

a large programming team will always build a large project” - J. K. Buckle “Managing Software Projects”

• Slips– If you must, do it once– Iterative development provides an out“A customer will always forgive you a slipped schedule or missed feature, they will

never forgive you for bad quality or performance” - Buckle

Which Project Used 15x Staff of the Other?

IBM 370

PDP 11/70

RepairServiceBureaus

Univac 1100 with C

TemporarySubsystem

“RobustPipe”

• Loop Assignment System– Univac system– Failed trial

$250 million down the drain Demoralized, weak team

– Senior 18-month commitmentNo architecture or requirementsNo technology infrastructure

– Working “temporary” subsystem

Unix based

• Loop Maintenance System– IBM MVS system– PDP 11/20 backup– 8 Months to go

No scheduleNo performance planNo requirementsWeak management

Switch

Test Head

PDP 11/20

How many lines of code?

• “1” - Six Unix tools, piped together• 100 - C, Pascal• ~600 - ACM paper on commenting code• 1,000 - Cobol, PL/1• 5,000 - Unisys SW VP• 10,000 - IBM SW VP• ??? with very detailed requirements

Determine the frequency count of the occurrence of words in text, most frequent first

Tempo“.. the disaster is due to termites, not tornadoes; and the schedule has slipped imperceptibly but

inexorably. Indeed, major calamites are easier to handle; one responds with major force…” - Brooks

• The plan– Delicate balance of top/down and bottom/up– PERT to plan, GANTT to manage

• Critical path control– Don’t multitask – Buffer schedule– 50% estimates

• Railroad train feature loading– Leave the station on time– Features left behind

• The weekly/biweekly project meeting– Milestones

• All milestones are important, particularly early ones and few big ones • Build team morale, tempo, customer confidence

“No plan survives contact with the enemy” - Helmuth von Moltke the Elder

The Project Meeting

“The Screaming”

Whipping the rowers

• Project manager style– Most decisions are 50/50– Avoid acting like “A glacier with dignity”

“An officer in charge on an Indian agency made a requisition in the autumn for a stove costing seven dollars, certifying at the same time that it was needed to keep the infirmary warm during the winter, because the old stove wore out.”

Then, after glacial dignity

“The stove is here. So is spring” *

* “An Autobiography” - Theodore Roosevelt

“Program testing can be a very effective way to show the presence of bugs, but it is hopelessly inadequate for showing their absence” - Dijkstra

• Full lifecycle activity– One day build and test (tempo!)

• Multi-level– Automated– Domain specific tools– Field driven test libraries

• What to test– Requirements & design– Coverage– Usability– Performance

• Measure with S curves

Time

Num

ber

Test cases runBugs foundBugs closed

Testing

Buy death march whip

}

Announce new features

Get your S-curve You can’t make this stuff up

Communications & Documentation“Schedule disaster, functional misfits, and systems bugs all arise because the left

hand doesn’t know what the right hand is doing” - Brooks

• Project meetings– Hierarchical– Weekly or biweekly

• Jeopardies are good• Documents

– “The act of writing forces hundreds of tiny decisions” - Brooks

– Few key ones, all template driven, all reviewed/inspected• Requirements, architecture, design, test & project plan• Strong version and configuration control

• Code– Structure - aim for one pagers– Comments

• “This section of code is cursed” - Stroustrup C++ • “You are not expected to understand this” - Unix

Installability

Usability

Service

Perform

anceF

unctionalityQ

uality

..

.

.

. .G

ood

OrganizationProject

Metric Driven Improvement

ProjectLearns

ProjectTeaches

OrganizationLearns

Num

ber

>6 2 1

4 3 2 1 4 3 2 1 4 3 2 1

….Age

Severity

• Quality– Measures

• Fault density

• Service responsiveness

– Improvement• Root cause analysis

• Team & individual

• Productivity

• Customer satisfaction– Huge fault density correlation

A Project-Specific Metric

• Project characteristics– Gluttonously staffed ~ 5,000 people– Long crumbled architecture - 20+ years– Kryptonite-weight processes

• Humongously successful– 99.9999 uptime

• Down 30 seconds a year for all causes

– Drug dealer cash flow

• Proposed programmer metricLines of code

Meetings attended

Improvement objective: exceed 1!

Contrarian Views

• Design methodologies– Great designers make great designs

– Methodologies make lousy designs look good

• Fancy measures– Confuse precision with accuracy

Making It Stick

• People support what they invent • Thick dictates are good only for cellophane

– Bellcore’s was ten one-pagers (what not how)

• Participatory multi-day course– Crusty expert led– Project team takes course

• Time it with a new release schedule • Output is their plan & their process

• Team mentors• Senior support

My Most Notable Disasters

• “Hi ho silver”– Loop maintenance system

• Domain ignorance– Customer ordering system“Organizations can only do damage to themselves and to society if they tackle

tasks that are beyond their specialized competence, their specialized values, their specialized functions” - Drucker

• Second system effect– Switch assignment system“This second is the most dangerous system a man ever designs. …The general

tendency is to over-design the second system, using all the ideas and frills that were cautiously sidetracked on the first one.” - Brooks

Maybe a little magic• High quality, on time useful software can be built

– Bellcore after six-years of continuous improvement (1994)• 95% on time/content• 3-4x competitive productivity• 10x competitive quality

• It’s all about people– Picking and developing the best and getting rid of the worst– Helping them select and use good software engineering practices– Motivating and rewarding them for

• Using and improving these practices • Behaving as teams

– Giving them the time and resources to do their jobs• Aggressive but realistic goals

– Making and holding them accountable

• Crowd control can be fun

Things you might consider trying(Other than shooting Alfie “the shot-caller”)

• Development process– Iterative development

• Entrance/exit– Design template – User-centered design (pair with another team)– Inspections

• Heavy and lightweight– Performance model– Defect metrics & root cause analysis– Tempo control

• Project meeting• Milestones• Railroad • Jeopardy

• Outside audit (pair with another team)– Changeability/customization

• Automated testing– System & module– S curves

• Lots of extra-bold Starbucks – Two shots

FinalFinal