Computing and SE II

Chapter 2: A Brief History of SE

Er-Yu DingSoftware Institute, NJU

A View of 20th and 21st Century Software Engineering

Barry Boehm

ICSE 2006 Keynote Address

May 25, 2006

boehm@usc.edu

http://sunset.usc.edu

Modified by Er-Yu Ding

05/25/06 3

Outline

• Motivation and Context

• A 20th Century View

• A 21st Century View

• Conclusions

– Timeless principles and aging practices

05/25/06 4

Motivation and Context

• Importance of historical

perspective

• We are losing our history

05/25/06 5

Why Software Projects Fail- Average overrun: 89.9% on cost, 121% on schedule, with 61% of

content

05/25/06 6

Importance of Historical Perspective

• Santayana half-truth:– “Those who cannot remember the past are

condemned to repeat it”

• Don’t remember failures?– Likely to repeat them

• Don’t remember successes?– Not likely to repeat them– Our objectives– Understanding SE more deeply– Organizing important

techniques/methods/methodologies of SE systematically

05/25/06 7

A Hegelian View of Software Engineering Evolution

Autonomy; Bio-Computing

1990's 2010's2000's1970's 1980's1960's1950's

Engineer Software

like Hardware

Risk-Based Agile/Plan

-Driven Hybrids;

Model-Driven Development

Value-Based Methods;

Collaboration; Global

Development; Enterprise

Architectures

Software Differences,

Engineer Shortages

Scalability,Risk Mgmt.

Many defects

Compliance

Time to Market,Rapid Change

Software Value-Add

COTS

Process Overhead

Scalability

SoftSysE

Software as Craft

Formality, Waterfall

Productivity; Reuse;

Objects;Peopleware

Agile Methods

Plan-Driven

Software Maturity Models

Integrated Sw-Systems Engineering

GlobalSystems

ofSystems

Theses

Syntheses

Antitheses

Prototyping

Risk Mgmt.Domain Engr.

05/25/06 8

Outline

• Motivation and Context

• A 20th Century View

• A 21st Century View

• Conclusions

– Timeless principles and aging

practices

05/25/06 9

1950’s Thesis: Engineer Software Like Hardware

• Hardware-oriented:– Software applications: airplanes,

bridges, circuits– Economics: Boehm supervisor, 1955

• “We’re paying $600/hour for that computer, and $2/hour for you, and I want you to act accordingly.”

– Software Processes: SAGE (Semi-Automated Ground Environment)

• 1 MLOC air defense system, real-time, user-intensive• Successful development of highly unprecedented

system• Hardware-oriented waterfall-type process

05/25/06 10

OPERATIONAL PLAN

MACHINE SPECIFICATIONS OPERATIONAL SPECIFICATIONS

PROGRAM SPECIFICATIONS

CODING SPECIFICATIONS

CODING

PARAMETER TESTING (SPECIFICATIONS)

ASSEMBLY TESTING (SPECIFICATIONS)

SHAKEDOWN

SYSTEM EVALUATION

The SAGE Software Development Process - (Benington, 1956)

“We were successful because we were all engineers”.

05/25/06 11

1950’s Thesis: Engineer Software Like Hardware

• Software Techniques:– Machine-Centric Application– For special usage, research mainframes

• 1GL, 2GL – Organization software with statements

• Coding tips for a optimized program are important

05/25/06 12

1960’s, techniques progress

• Complier can optimizing statements well• Organizing program with function

– Function was derived from λ calculus

• software get more complex

05/25/06 13

1960’s Antithesis: Software Is Not Like Hardware

- Four Brooks factors plus two• Invisibility: like the Emperor’s Magic Cloth

– Visibility and controllability of project is always a topic of SE

• Complexity: we have see this last lecture• Conformity: executed by computers, not

people• Changeability: up to a point, then becomes

difficult• Doesn’t wear out: different reliability,

maintenance phenomena• Unconstrained: can program antigravity,

time travel, interpenetration, …

05/25/06 14

1960’s Antithesis: Software Crafting

• Flexible materials, frequent changes as above

• SW demand exceeded supply of engineers– Cowboy programmers as heroes– Code-and-fix process– Hacker culture (Levy, 1984)

• Judge programmers by the elegance of their code

05/25/06 15

1960’s Progress and Problems

• Better infrastructure: OS, compilers, utilities• Computer Science Departments• Product Families: OS-360, CAD/CAM,

math/statistics libraries• Some large successes: Apollo, ESS, BofA

check processing• Problems: 1968, 1969 NATO Reports

– Failure of most large systems– Unmaintainable spaghetti code– Unreliable, undiagnosable systems

05/25/06 16

1970’s Antithesis: Formal and Waterfall Approaches

• Structured Methods– Structured programming (Bohm-Jacopini: GO TO

unnecessary)• Formal programming calculus: Dijkstra, Hoare, Floyd• Formalized Top-Down SP: Mills, Baker

• Waterfall Methods– Code and fix too expensive (100:1 for large

systems)– Precede code by design (De Marco SD, Jackson

JSD/JSP)– Precede design by requirements (PSL/PSA, SA,

SREM)

05/25/06 17

1970’s Antithesis: Formal Methods

• “formal methods” branch that focused on program correctness, either by mathematical proof, or by construction via a “programming calculus”

• Over the decade of the '60s theoretical computer science achieved standing as a discipline recognized by both the mathematical and the computing communities, and it could point to both applications and mathematical elegance

05/25/06 18

1970’s: Formal Methods

• Organized programs with – Function

• Algorithm

– Three control logic– Type and type check

• When compiler deal with functions well, organized programs with– Modularity

• This things gets importance with modularity– Information hiding– Data abstraction

05/25/06 19

1970’s: Formalized Top-Down SP

• Function decomposition– divide and rule– Process modeling

• DFD

– Data modeling• ERD

05/25/06 20

Increase in Software Cost-to-fix vs. Phase (1976)

Phase in Which defect was fixed

10

20

50

100

200

500

1000

Rel

ativ

e co

st t

o f

ix d

efec

t

2

1

5

Requirements Design Code Development Acceptance Operationtest test

Smaller Software Projects

Larger Software Projects

• Median (TRW Survey)80%

20%SAFEGUARD

GTE

IBM-SSD

•

•

•

•

•

•

Phase in Which defect was fixed

10

20

50

100

200

500

1000

Rel

ativ

e co

st t

o f

ix d

efec

t

2

1

5

Requirements Design Code Development Acceptance Operationtest test

•

••

••

••

•

•

•

05/25/06 21

SYSTEM REQUIREMENTS

TESTING

CODING

PROGRAM DESIGN

ANALYSIS

PRELIMINARY PROGRAM

DESIGN

SOFTWARE REQUIREMENTS

OPERATIONS

PRELIMINARY DESIGN

ANALYSIS

PROGRAM DESIGN

CODING

TESTING

USAGE

The Royce Waterfall Model (1970)

- Explicit feedback- “Do it twice”

05/25/06 22

1970’s: Problems with Formal Methods

• Successful for small, critical programs• Largest proven programs around 10 KSLOC• Proofs show presence of defects, not

absence– Defects in specification, proofs happen

• Scalability of programmer community– Techniques require math expertise, $500/SLOC– Average coder in 1975 survey:

• 2 years of college, SW experience• Familiar with 2 languages, applications• Sloppy, inflexible, in over his head, and

undermanaged

05/25/06 23

Large-Organization HW/SW Cost Trends (1973)

100

80

60

40

20

0

1955 1970 1985

Hardware

Software

Year

% of total cost

05/25/06 24

1970’s: Problems with Waterfall Model• Overly literal interpretation of sequential

milestones– Prototyping is coding before Critical Design Review– Mismatch to user-intensive systems

• Heavyweight documentation hard to review, maintain– 7 year project with 318% requirements change– Milestones passed but not validated

• Mismatch to COTS, reuse, legacy SW– Bottom-up vs. top-down processes

05/25/06 25

A Hegelian View of Software Engineering Evolution

Autonomy; Bio-Computing

1990's 2010's2000's1970's 1980's1960's1950's

Engineer Software

like Hardware

Risk-Based Agile/Plan

-Driven Hybrids;

Model-Driven Development

Value-Based Methods;

Collaboration; Global

Development; Enterprise

Architectures

Software Differences,

Engineer Shortages

Scalability,Risk Mgmt.

Many defects

Compliance

Time to Market,Rapid Change

Software Value-Add

COTS

Process Overhead

Scalability

SoftSysE

Software as Craft

Formality, Waterfall

Productivity; Reuse;

Objects;Peopleware

Agile Methods

Plan-Driven

Software Maturity Models

Integrated Sw-Systems Engineering

GlobalSystems

ofSystems

Theses

Syntheses

Antitheses

Prototyping

Risk Mgmt.Domain Engr.

05/25/06 26

1980’s Synthesis: Productivity, Reuse, Objects

• Worldwide concern with productivity, competitiveness– Japanese example: autos, electronics, Toshiba

SW reuse

• Major SW productivity enhancers– Working faster: tools and environments– Working smarter: processes and methods– Work avoidance: reuse, simplicity; objects– Technology silver bullets: AI, transformations,

DWIM, PBE• Do what I mean; programming by example

05/25/06 27

05/25/06 28

Tools, Environments, and Process• 1970s, some requirements and design tools

developed, significant tool progress had been mode in such areas as test tools and SCM tools

• Major emphasis in 1980s was on integrating tools into support environments: IPSEs to CASEs– Integrated Project Support Environment, Computer Aided

Software Engineering – Requirements, design, planning and control, office support– Formatted vs. formal specifications

• 1980s: Resource estimations and middleware is underway

• Process-driven CASEs– Use process knowledge as tool integration framework– Some overkill in locking people into roles

05/25/06 29

Process: Industrial Engineering

• Frederick W. Taylor : scientific management– First. They develop a science for each element

of a man's work– Second. They scientifically select and then

train, teach, and develop the workman– Third. They heartily cooperate with the men so

as to insure all of the work is being done in accordance with the principle of the science which has been developed.

– Fourth. There is an almost equal division of the work and the responsibility between the management and the workmen.

05/25/06 30

Process

• Process execution support: “SW processes are SW too”– What’s good for products is good for processes– Reuse, prototyping, architecture, programming

• Process compliance support: Standards and CMMs

05/25/06 31

Reuse: Mechanical Engineering

• In modular software, clearly the "standardized units or dimensions" should be standards such that software modules meeting the standards may be conveniently fitted together (without "trimming") to realize large software systems. The reference to "variety of use" should mean that the range of module types available should be sufficient for the construction of a usefully large class of programs.

05/25/06 32

Reuse and Object Orientation

• 1950’s: Math routines, utilities• 1960’s: McIlroy component marketplace,

Simula – 67• 1970’s: Abstract data types, Parnas program

families• 1980’s: Smalltalk, Eiffel, C++, OO methods,

reuse libraries• 1990’s: Domain engineering, product lines,

UML, components, patterns, pub-sub architectures, etc.

• 2000’s: Model driven development, service oriented architectures

05/25/06 33

No Silver Bullet: Brooks• Automated solutions are good for

“accidental” software problems– Simple inconsistencies, noncompliance, inferences

• They do not do well on “essential” software problems– Changeability: adapting themselves to unanticipated

changes– Conformity: working out everything the computer

needs to “know”• Devoid of intuition, commonsense reasoning

– Complexity: integrating multiple already- complex programs

– Invisibility: communicating their likely behavior to humans

• Closest thing to silver bullet: great designers

05/25/06 34

People: The Most Important Factor- SW engineering is of the people, by the people, and for the

people• 1970’s: Weinberg Psychology of Computer

Programming• 1980’s: DeMarco-Lister Peopleware• 1990’s – 2000’s: Importance emphasized

in both Agile and CMM cultures– Individuals and interactions over process and

tools– People CMM, Personal Software Process,

Team Software Process

05/25/06 35

Dual 1990’s – Now: software progress

• 1950s-1960s: machine-centric software• 1960s-1980s: application-centric software• 1990s-now: enterprise-centric software

– Distributed, large-scale– Design patterns, Software architectures, UML– Requirement engineering– Middleware, Components, CDE, Product Line, COTS,

etc. • Collaborative development environment, Commercial off the

shelf

• Future?– domain-centric software

• Components, Product Line, COTS

05/25/06 36

Dual 1990’s – Early 2000’s: software techniques

• OO methods were strengthened through– OOP ( Java )OO Design (1995)OO

Analysis– UML– OO design principles– Design patterns

05/25/06 37

Dual 1990’s – Early 2000’s Antithesis:- Maturity Models and Agile Methods

• Predictability and Control: Maturity Models——CMMI– Reliance on explicit documented knowledge– Heavyweight but verifiable, scalable

• Time to Market and Rapid Change: Agile Methods– Reliance on interpersonal tacit knowledge– Lightweight, adaptable, not very scalable

05/25/06 38

Other 1990’s – Early 2000’s Developments

• Y2K and reverse engineering• Mature risk-driven software model: RUP

– Rational unified process

• Nature and importance of software architecture

• Usability and Human-Computer Interaction• COTS, open source, and legacy software• Software as the primary competitive

discriminator

05/25/06 39

Open source and legacy

• Open source– From its roots in the hacker culture of the

1960’s• Collective code ownership• Free software, data, computing access

– Free Software– Linux– Raymond’s “The Cathedral and the Bazaar”

• Legacy– EAI (enterprise application integration), BPR

(business process reengineering)

• Components and COTS are their future

05/25/06 40

COTS: The Future Is Here• Escalate COTS priorities for research, staffing, education

– Software is not “all about programming” anymore– New processes requiredCBA Growth Trend in USC e-Services Projects

0

10

20

30

40

50

60

70

80

1997 1998 1999 2000 2001 2002

Year

Perc

en

tag

e

*

• CBA: COTS-Based Application

* Standish Group CHAOS 2000 (54%)

05/25/06 41

Mid-2000’s Synthesis: Risk-Driven Hybrid Products and Process

• Increasing integration of systems engineering and SW engineering– Increasing trend toward “soft systems

engineering”

• Increasing focus on usability and value– Fit the software and hardware to the people,

not vice versa

• Model-driven development and service-oriented architectures

05/25/06 42

05/25/06 43

Outline

• Motivation and Context

• A 20th Century View

• A 21st Century View

• Conclusions

– Timeless principles and aging practices

05/25/06 44

The Future of Systems and Software

• Eight surprise-free trends1. Increasing integration of SysE and SwE2. User/Value focus3. Software Criticality and Dependability4. Rapid, Accelerating Change5. Distribution, Mobility, Interoperability,

Globalization6. Complex Systems of Systems7. COTS, Open Source, Reuse, Legacy Integration8. Computational Plenty

• Two wild-card trends9. Autonomy Software10.Combinations of Biology and Computing

05/25/06 45

Outline

• Motivation and Context

• A 20th Century View

• A 21st Century View

• Conclusions

– Timeless principles and aging practices

05/25/06 46

Timeless Principles (+) and Aging Practices (-)

• From the 1950’s+ Don’t neglect the sciences+ Look before you leap (avoid premature

commitments)- Avoid inflexible sequential processes

• From the 1960’s+ Think outside the box+ Respect software’s differences- Avoid cowboy programming

05/25/06 47

Timeless Principles (+) and Aging Practices (-)

• From the 1970’s+ Eliminate errors early+ Determine the system’s purpose- Avoid top-down development and reductionism

• From the 1980’s+ These are many roads to increased productivity+ What’s good for products is good for processes- Be skeptical about silver bullets

05/25/06 48

Timeless Principles (+) and Aging Practices (-)

• From the 1990’s+ Time is money and value to people+ Make software useful to people- Be quick, but don’t hurry

• From the 2000’s+ If change is rapid, adaptability trumps

repeatability+ Consider and satisfice all of your stakeholders’

value propositions- Avoid falling in love with your slogans

05/25/06 49

Timeless Principles (+) and Aging Practices (-)

• For the 2010’s+ Keep your reach within your grasp+ Have an exit strategy- Don’t believe everything you read

- “It’s true because I read it on the Internet”

05/25/06 50

The END

• Recommend papers– A View of 20th and 21st Century Software

Engineering– Fifty Years of Progress in Software Engineering– Finding a History for Software Engineering

• Next Lecture– Software developments activities