copyright © 1995-2004, dennis j. frailey, all rights reserved day 5, part 1, page 1 1/11/2004 day...
TRANSCRIPT
Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 5, Part 1, Page 11/11/2004
Day 5, Part 1Basic Principles and Techniques
of Productivity, Quality and Cycle Time Management
Software Project Planning and Management
Dr. Dennis J. FraileyPrincipal Fellow
Raytheon Company
Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 5, Part 1, Page 21/11/2004
The Overall Planning Cycle
AnalyzeJob
Manage Risks
Execute
GenerateDetailed Plans
GenerateInitial Plans
Measure, Manage Productivity and Quality
Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 5, Part 1, Page 31/11/2004
Outline
• The basic principles of productivity
• Customer value and value-added analysis
• Principles of cycle time improvement
Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 5, Part 1, Page 41/11/2004
Basic Principles of Productivity
Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 5, Part 1, Page 51/11/2004
Productivity is ...
The quality or state of being productive
Webster’s 9th New Collegiate Dictionary
Productive:Having the power of producing;Yielding or furnishing results,
benefits, or profitsWebster’s 9th New Collegiate Dictionary
Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 5, Part 1, Page 61/11/2004
Productivity is NOT being ...
• Busy• Industrious• Virtuous
Productivity has to do with results (what and how much you produce), not with the means or
methods of production or the characteristics of the producer
Productivity has to do with results (what and how much you produce), not with the means or
methods of production or the characteristics of the producer
• Wealthy• Hardworking• etc.
Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 5, Part 1, Page 71/11/2004
Measuring Productivity
• Productivity is usually measured in terms of how much you produce in relation to how much you invest
Productivity = Output
Input or Investment
Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 5, Part 1, Page 81/11/2004
Productivity Measures• Products produced per labor hour• Return on investment• Bushels of grain per acre of land• Modules tested per week• Requirements validated per day
A farmer who invents a new method of growing that doubles the output per acre would be more
productive than a farmer who works longer hours and doubles output per acre.
Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 5, Part 1, Page 91/11/2004
Efficiency• Productivity is a measure of how
efficient your process is• Thus the way you improve
productivity is to make the process more efficient
• Many principles of quality engineering and cycle time improvement are directly related to productivity improvement
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 101/11/2004
Work in Process (WIP)
• WIP or “work in process” is work in the midst of being done– code being tested– specifications being written– objects being designed
• Excess WIP is work waiting to be done that is not being done -- waiting in queues instead– something is holding up the process
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 111/11/2004
Excess “Work in Process” is a
Key Symptom
Excess WIP is a symptom of low productivity, long cycle time, and process
inefficiencyAnother key symptom is non-value-added
work. This will be addressed later.
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 121/11/2004
Excess WIP Usually Implies a Constraint or Bottleneck
• When work is waiting to be done, it means that there is something holding it up
• Usually this is some kind of limitation on resources or other form of process constraint
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 131/11/2004
Constraint Management
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 141/11/2004
What Is a Constraint?•A constraint (bottleneck)
is any resource with capacity less than the demand placed upon it– This could be a person, a
computer, a network, a machine, etc.
•The constraint regulates the output rate of the entire process
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 151/11/2004
•Improving a constraint provides capacity increase, often without capital investment
•Management provides flexibility to respond to changes in customer demand
•Reduces product cost as a result of increased output and improved efficiency
Why Is ConstraintManagement Important?
Hiking Boys Story
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 161/11/2004
If 100 people each worked to increase
capacity by 1% on 100 different parts of
the software development process, the 1
person working on the constraint would
save the company thousands more than all
the other 99 people combined!
Where Are Your Resources Focused?
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 171/11/2004
Effective Utilization
Before deciding to take actions
to improve the constraint, we
must first understand the
concept of effective utilization
Before deciding to take actions
to improve the constraint, we
must first understand the
concept of effective utilization
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 181/11/2004
Effective UtilizationDefined
24 HOURS
AVAILABLE (A) UNAVAILABLE
REQUIRED (R)
Focus on decreasing ”R" and increasing “A”I.e., decrease effective utilization!
Focus on decreasing ”R" and increasing “A”I.e., decrease effective utilization!
= EFFECTIVE UTILIZATION =TIME REQUIRED
TIME AVAILABLERA
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 191/11/2004
Effective Utilization-vs-
Productivity •Webster’s definitions:
– Utilize: To make use of– Busy: Constantly active or in motion– Productive: Yielding or furnishing
results, benefits or profits
We tend to measure utilization by how busy we are, BUT utilization tells us little about how productive we
are.
We tend to measure utilization by how busy we are, BUT utilization tells us little about how productive we
are.
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 201/11/2004
Why Reduce EffectiveUtilization?
EFFECTIVE UTILIZATION
WIP
or
CY
CLE T
IME
100%
Running assets at a high effective utilization requires a costly cycle time trade off
per Erlang, 1917 (see Gross and Harris, pp 10-11, 101-102)
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 211/11/2004
Examples
• Network performance vs load• Traffic flow vs load on freeway• Computer response vs load• Telephone dial tone delay vs
load• etc.
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 221/11/2004
Utilizationvs Productivity
•High effective utilization will cause delays (cycle time) to increase greatly (increase investment) while output is increased only marginally, thus lowering productivity of the system
•Lower utilization means the resource is available when needed most, thus reducing delays and raising productivity of the overall system
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 231/11/2004
Example - Firemen
• Why are firemen usually sitting around the fire station, essentially idle?
• Because they are needed ASAP when there is a fire.
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 241/11/2004
Effective UtilizationAnd the Constraint
•Reducing effective utilization is especially critical at the constraint, because it determines the performance of the system as a whole
•To minimize effective utilization, we must make our assets more productive– This does not necessarily mean that
they are busier!
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 251/11/2004
Ways to Improve Efficiencyat the Constraint
• Repair analysis improves our ability to repair and maintain the constraint, so we can improve its availability (increase “A”)
RepairmanStory
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 261/11/2004
More Ways to Improve Efficiency
at the Constraint• Flow balancing allows us to effectively increase tool availability (increase “A”)– Flow variability to the constraint increases the
probability of the constraint running dry– Flow management helps control variability
• Statistical process control techniques can decrease required processing time as well as increase tool availability (decrease “R”, increase “A”)
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 271/11/2004
Customer Value
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 281/11/2004
What Constitutes Customer Value?
•Customer value is the real target of any competitive business
•The exact definition depends on the customer and the market–Sizzle vs steak–Features vs ease of use–Cost of operation vs comfort and safety
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 291/11/2004
Defining Value
•Correctly defining value is the first step of customer satisfaction
•Cut out the tasks or features that do not directly or indirectly contribute to value–They add cost but do not provide
appropriate benefit
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 301/11/2004
Dimensions of Customer Value• Low Costs / High Productivity
– Product development/manufacturing efficiency
– Attractive price
• High Quality– Customer satisfaction– Reliability & few defects
• Short Cycle Time– Rapid product development– Response to orders
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 311/11/2004
The Goal
• Improve all components of the customer value triangle
CustomerValue
Quality
Prod
uctivityC
ycle
Tim
e
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 321/11/2004
Without Process Change ...
• You can improve any one at the expense of the others
High Quality and
Low Cost, but Slow
High Quality,but Slow and Costly
Fast,Cheap,
butShoddy
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 331/11/2004
EvenBetterValue
BetterValue
GoodValue
SatisfactoryValue
By Changing Process & Culture ...
… you can improve all together
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 341/11/2004
Weinberg’s Definition of Quality
“Quality is value to someone”
• In Weinberg’s sense, value is quality• And the cost to produce value is the
cost of quality• But the term “cost of quality” is usually
used in a different context, to describe tasks we execute that improve quality
• Keep this in mind as we continue in this course
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 351/11/2004
Value-Added Analysis
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 361/11/2004
Who Produces Value?
•Value is the result of the best software engineers doing their best work.
•So software engineers produce value - and quality!
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 371/11/2004
WHAT ARE THE COSTS OF SOFTWARE DEVELOPMENT?
Total Costs
Essential Non Essential
Value-Added Non-Value-Added
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 381/11/2004
Value-Added Costs
Costs for tasks performed ...– Materials (e.g.., paper, software)– Labor hours (salaries)– Capital equipment (workstations,
facilities)
… that produce value– Products– Customer satisfaction– Future labor that will not be expended
• e.g.., reduced maintenance and repair
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 391/11/2004
The Strict Definition of“Value-Added”
• Any activity that is part of the process is considered a value-added activity if it meets three criteria:
1) Must change the product in some way2) Must make the product more desirable to the customer (i.e., the customer wants the change)3) Must be done right the first time
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 401/11/2004
The Strict Definition
• This very strict definition helps us open our minds
• So we identify the proper targets for process improvement.
• Anything that is not value-added is a suitable target for removal or improvement.
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 411/11/2004
Things Not Part of Value-Added• Features the engineer thinks are nice but
the customer doesn’t care about• Moving a product around• Translating between incompatible tools• Repairing mistakes• Tests and inspections• Most management activities• Activities unrelated to the process• Many other things we tend to think of as
“necessary” or “desirable”– And some of them are necessary!
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 421/11/2004
Some Non-Value-Added Activities
• Management• Quality Assurance• Testing• ...
The term “value-added” is used to help us improve our processes. It is not meant to
imply that the above tasks are not valuable or that the people who do them are
dispensable.
The term “value-added” is used to help us improve our processes. It is not meant to
imply that the above tasks are not valuable or that the people who do them are
dispensable.
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 431/11/2004
Non-Value-Added Essential
• Costs for tasks performed because the process is not perfectly efficient– Peer reviews– Evaluations, inspections, verification and
validation– Metrics collection, storage and analysis– Extra reviews and verifications required
by customer or company policy (usually because of past problems)
– Certain overhead costs (benefits, support activities)
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 441/11/2004
Why are they Essential?• They would not be necessary in a
perfect world• But they are necessary with our
current methods of product development and our current level of product development knowledge
Every process has some essential, non-value-added elements
Every process has some essential, non-value-added elements
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 451/11/2004
Another Perspective
• Non-value-added but essential tasks are things we might wish we did not have to do
• But if we did not do them, things would be worse.
• However, we still can study them to find out how to minimize them, optimize them, and improve them
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 461/11/2004
Non-Value-Added, Not Essential
• Tasks that are not value-added and that are not essential
• Typically, these are tasks that we perform because we have not really optimized our processes– For example, things we have always done
but no longer need to do – Or methods that once made sense but don’t
any more due to newer technologies or changes in the organization or environment
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 471/11/2004
Examples of Non-Value-Added, Not Essential Tasks
– Excessive paperwork or approvals– Waits for test equipment or “signoff”– Debugging due to sloppy design or coding – Costs resulting from bugs in our software
development tools– Costs for activities unrelated to the process
These tasks should be eliminated or streamlined first, as they add cost for no
useful purpose
These tasks should be eliminated or streamlined first, as they add cost for no
useful purpose
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 481/11/2004
Some Costs are Especially Painful
• High costs incurred later because of tasks not performed during software development (or not performed at the right time or in the right way)– Debugging– Correcting defects– Maintenance and repair
• These can subtract value:– Loss of customer good will– Future labor that must be expended
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 491/11/2004
Typical Value-Added Categories
Non-EssentialEssential
Value-Added Non-Value-Added (costs $, no value to customer)1) Customer
Wants2) Changes
Product3) Done Right
the First Time• Design• Development• Fabrication• Documentation• Assembly• Process• Creation• Upgrade• Shipping
• Set-Up•Training• Planning• Customer-required test• Moving Data Between Steps• Many Quality Improvement activities
• Rework• Service• Modification• Expediting• Recall• Correction• Retest• Error Analysis
• Extra paperwork• Waits• Delays• Bottlenecks• Counting• Installing Software Tools• Extra Un-wanted Features
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 501/11/2004
Analyzing Value-Added by Task or Category
This is the first step of value-added analysis• List all of the tasks or task categories in
your process• Then place each task into one or more of
the three value-added classes
Value-Added Essential Non-EssentialNon-Value-Added
• If a task fits more than one class, you may want to break it up into parts
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 511/11/2004
Example Result of Analysis by Task / Category
Non-EssentialEssential
Value-Added Non-Value-Added (costs $, no value to customer)1) Customer
Wants2) Changes
Product3) Done Right
the First Time• Requirements analysis
• Design• Coding• Documentation• Integration• Manufacturing• Packaging• Shipping
• Estimating• Training• Planning• Customer-required acceptance test
• Configuration Control
• Inspections
• Debugging• Service calls• Warranty costs
• Fedex costs for patches
• Loss of customer goodwill
• etc.
• Approval by 7 people!
• Delays for test systems
• Data conversion between design tool and coding tool
• Wait for subcontracted hardware
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 521/11/2004
Analyzing Value-Added Costs
• This is the second step of value-added analysis
• Each task can be assessed with respect to how much of its cost adds value
• Often, a task will contribute some value but have some non-value-added elements as well
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 531/11/2004
Example of Cost Analysis
Task ValueAdded
NVAEssential
Non-Value-Added
Translate rqmts modelto UML f ormat
6
Evaluate Use Cases 24 4
Prepare structurediagrams
18
I nspect structurediagrams
8 2
Prepare report 2 2
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 541/11/2004
What Are the Units?• The unit we measure for “cost”can
be anything that is available …– Dollars– Labor hours– Percent of time spent
• For initial analysis, the data do not have to be very accurate– Estimated percent of time spent– Estimated hours spent
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 551/11/2004
Suggested Approach to Apply These Concepts in Practice
• Take a typical process from your work environment and do a value-added analysis by task or category
• For tasks that have some value-added and some non-value-added, estimate percentages of each
• Estimate what percent of the overall process is non-value-added
• Discuss how you might measure the costs (what units to use) using available information or information readily obtained
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 561/11/2004
Typical Result
• Value-added -- 35% of total cost• NVA Essential -- 20% of total cost• NVA Non-essential -- 45% of total cost
– Top three items:• Rework due to design and coding errors --
14%• Extra customer support -- 12%• Labor for individuals waiting for test
equipment that is not available -- 11%
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 571/11/2004
A Dilemma in Analyzing Non-Value-Added Costs
• Sometimes we must introduce non-value-added tasks to reduce the costs and impact of other non-value-added tasks
• This is a fundamental dilemma and a fundamental reason why we need to do more than analyze value-added
• We discussed this a little when we talked about cost of quality analysis
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 581/11/2004
Note that Efficiency Improves Three Perspectives
Quality (Fewer Defects; Customer
satisfaction)
Low Cost or
High
ProductivityCustomer
Value
Shor
t Cyc
le T
ime
or
Sche
dule
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 591/11/2004
Summary
• Understanding value is the starting point for effective and efficient processes
• Value-added analysis can help identify the best places to focus improvement efforts
• Just because something is “non-value-added” does not mean it is not worthwhile or necessary or good
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 601/11/2004
References
• Knox, 1993, Raytheon studies reported by Houston and Keats, Software Quality Matters, vol 5, no 1 (Spring, 1997), U. of Texas SW Quality Institute
• Musa, John D, 1992, “The Operational Profile,” in Software Reliability Engineering: An overview.
• Weinberg, Gerald M., 1992, Quality Software Management, Volume 1, Systems Thinking, Dorset House, New York, ISBN 0-932633-22-6.
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 611/11/2004
This Page Intentionally Blank
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 621/11/2004
Principles of Cycle Time Improvement
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 631/11/2004
Outline
• Why Cycle Time is Important• Symptoms and Causes of Cycle
Time Problems
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 641/11/2004
Cycle Time Reduction
Time costs moneyCutting cycle time saves money and can
increase qualityThe issue: how to do it effectively
Quality (Fewer Defects; Customer satisfaction)
Low Cost or
High
Productivity
CustomerValueShor
t Cy
cle
Tim
e or
Sche
dule
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 651/11/2004
Why Cycle Time is Important
Yes, Sir!Right Away,
Sir.
I Needthat Software
no later than next week! Why is it taking so long?
But we’re already working
overtime!
We’ll work overtime to get
it out, sir.
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 661/11/2004
Importance of Low Development Cost for New Product
Development
0 5 10 15 20 25 30 35% LOSS OF TOTAL PROFIT
ON TIME, CORRECT PRODUCT COST, BUT WITH 50% DEVELOPMENT COST OVERRUN
Mckinsey & Co. Analysis of new product introduction profit and loss
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 671/11/2004
Importance of Product Cost for New Product Development
0 5 10 15 20 25 30 35% LOSS OF TOTAL PROFIT
ON TIME AND BUDGET, BUT PRODUCT COST 9% TOO HIGH
ON TIME, CORRECT PRODUCT COST, BUT WITH 50% DEVELOPMENT COST OVERRUN
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 681/11/2004
Importance of Cycle Time for New Product Development
0 5 10 15 20 25 30 35% LOSS OF TOTAL PROFIT
ON TIME AND BUDGET, BUT PRODUCT COST 9% TOO HIGH
ON BUDGET, BUT PRODUCT IS SHIPPED 6 MONTHS LATE
ON TIME, CORRECT PRODUCT COST, BUT WITH 50% DEVELOPMENT COST OVERRUN
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 691/11/2004
Why Improve Software Development Cycle Time?
This does not necessarily mean that you will always need to develop
products quickly– Not all situations require short cycle
time– But for those that do, you are more
competitive
To improve the organization’s capability to develop software
products quickly.
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 701/11/2004
What If You Don’t NeedShort Cycle Time?
• You can use this capability to:– Achieve competitive costs– Start software development later in
the program cycle– Allow less time to change
requirements– Get software development off of the
critical path
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 711/11/2004
A Common Cycle Time Issue
It took so long to develop the software that the
customer’s needs changed. This resulted in excessive rework, higher costs, and further delays
It took so long to develop the software that the
customer’s needs changed. This resulted in excessive rework, higher costs, and further delays
The key is to develop the software quickly so today’s needs can
be met
We’ve changed
this requirement
.
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 721/11/2004
How Can Cycle TimeBe Improved?
• The following video illustrates how to improve cycle time in an area that many of us are familiar with - building a house
• As you watch, think of ideas that might be applicable to software development
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 731/11/2004
Some Lessons from the Cycle Time Video
• What did they do?
• Is there a software counterpart?
Things they did:
+_%$#@&
~~~~~~~~~~
~~~~~~~~~~~
~~~~~~~~~~~
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 741/11/2004
How Is Cycle Time Improved?• Doing every process step faster?• Working longer hours?• Piling up work?
Faster!!!
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 751/11/2004
How Is Cycle Time Improved?• Doing every process step faster?• Working longer hours?• Piling up work?
Faster!!!
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 761/11/2004
Cycle Time Improvement Is ...
Improving the Process
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 771/11/2004
Cycle Time Improvement Is ...
Improving the Process
Reducing the Critical Path
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 781/11/2004
Cycle Time Improvement Is ...
Improving the Process
Reducing the Critical Path
Eliminating Waits, Queues, Bottlenecks
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 791/11/2004
Cycle Time Improvement Is ...
Improving the Process
Reducing the Critical Path
Eliminating Waits, Queues, Bottlenecks
Helping People work Smarter
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 801/11/2004
Cycle Time Improvement Is ...
Improving the Process
Reducing the Critical Path
Eliminating Waits, Queues, Bottlenecks
Helping People work Smarter
Increased Cycles of Learning
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 811/11/2004
Cycle Time Improvement Is ...
Improving the Process
Reducing the Critical Path
Eliminating Waits, Queues, Bottlenecks
Helping People work Smarter
Increased Cycles of Learning
Reengineering the Process
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 821/11/2004
Symptoms and Causes ofCycle Time Problems
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 831/11/2004
Symptoms ofCycle Time Problems
– Long waits and queues– Lots of “work in process”– High inventory levels– Excessive overtime
Tickets -->Tickets -->
How long isthis line?
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 841/11/2004
Major Contributors toSlow Cycle Time
• Bottlenecks and constraints • Barriers• Inefficient processes• Inadequate training • Variability
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 851/11/2004
Bottlenecks and Constraints
• Symptom: excess WIP or “work in process” – work waiting to be done that is not being
done -- waiting in queues instead– something is holding up the process
• Causes: limited capacity, poor processes, poor execution, or various barriers imposed by the organization
Places in the process that inhibit efficient flow of work
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 861/11/2004
Example Process
Pizza OvenPizza Oven
ToppingsToppings
CrustCrust
BoxBox
SlicerSlicer
DeliverDeliver
Take OrderTake Order Where could
the bottlenecks
be?
Where could the
bottlenecks be?
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 871/11/2004
Barriers
Things that prevent you from going faster, such as budgetary limits,
head count limits, lack of people or equipment, process problems, missing data, regulations, etc.
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 881/11/2004
Sample Barriers
• Approval processes• Excessive paperwork• Limited space• Limited budgets• Wasteful processes• Defective information or
material• …..
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 891/11/2004
Dealing with Barriers• Question why barriers are there
– Often they are the remnants of past problems and no longer need to be there• Obsolete approval procedures• Redundant checks and balances
– Sometimes the barriers have benefits but the organization does not understand their consequences • The “cure” may be more expensive than the problem
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 901/11/2004
Organizations Often ResistRemoval of Barriers
• There are many reasons, such as …– Political factors– Power issues– How performance and success are
defined– Fear of change
Exercise
What barriers does your organization resist removing and why?
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 911/11/2004
Lack of Training Contributesto Slow Cycle Time
• Inadequate Training - people do not know what to do, or how to do it most effectively– So it takes them longer to do the work
Investment in training usually pays off well
--But it requires foresight and faith
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 921/11/2004
Process Contributesto Slow Cycle Time
• Inefficient Processes - processes containing steps that do NOT ADD VALUE– All processes have inefficiencies– Most processes can be improved in efficiency
• Variability - in processes, materials, skills, etc.– Variability is natural– But it can be controlled
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 931/11/2004
Special Notice
• For the next exercise, each student should bring a pair of dice – Actually, one die per student will do
• And five coins or other tokens• And have some desk space free
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 941/11/2004
References
• Deming, W. Edwards, Out of the Crisis, MIT Press, 1982.
• Goldratt, Eliyahu M. & Jeff Cox, The Goal, (North River Press, 1984.) Also, Theory of Constraints, It’s Not Luck, and Critical Chain Management.
• Swartz, James B., The Hunters and the Hunted, (Portland, Oregon, Productivity Press, 1994) ISBN 1-56327-043-9.
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 951/11/2004
Outline
• Variability Experiment• Definitions, Terms and Concepts• Solutions to Cycle Time Problems
– Reducing Variability– Reducing Work in Process
• Observations and Caveats
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 961/11/2004
Variability Experiment
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 971/11/2004
Variability Experiment• Line up 5 tokens, like a train• Each turn, move tokens right
– See rules on a later slide
• How many turns does it take to move the leftmost token 15 positions?
Rules:– Round 1 will have low variability– Round 2 will have high variability– Round 3 will have high variability but higher
capacity
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 981/11/2004
Token Setup for Experiment o o o o ... Round 1
o o o o … Round 2
o o o o … Round 3
^ ^ ^Left Token Right Token Positions to Move to
You may execute the rounds sequentially or in parallel.
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 991/11/2004
Round 1 Rules• Each turn, move each token 3 places,
starting with the one on the right (3 places right, per turn)
• Result of first turn: o o o o o o
…
• Result of second turn: o o o o o o ...
How many turns does it take for the leftmost token to move 15 places to the right?
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1001/11/2004
Round 2 Rules• Each turn, move each token n places,
starting with the one on the right, where n is a number from 1 to 5 determined randomly by rolling a die - 6’s don’t count.– Roll the die separately for each token– A token cannot move past another
• Note that the average move is 3 places• Possible result from first turno o o o o o …
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1011/11/2004
Round 3 Rules• Each turn, move each token n places,
starting with the one on the right, where n is a number from 1 to 6 determined randomly by rolling a die - 6’s do count.– Roll the die separately for each token– A token cannot move past another
• Note that the average move is 3 1/2• Possible result from first turno o o o o o
…
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1021/11/2004
Why Variability Results in Longer Cycle Time
• Slower tokens block faster ones• If you have a low roll on the die you
cannot take advantage of opportunities to “catch up”
• Even with higher average capacity (round 3), variability results in slower cycle time
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1031/11/2004
Definitions, Terms andConcepts
The next few slides address some basic cycle time
terminology
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1041/11/2004
Cycle Time is ...
• This could be:– First operation to ship– A single operation– A group of operations– Customer order to product delivery
• Cycle time includes actual processing time …… and all waiting time– Consider a “10 minute” oil change
The time required to execute all activities in a process
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1051/11/2004
Lead Time is ...
• What is:– Your customer’s lead time?– Your potential customer’s lead time?
The maximum time the customer will wait between order placement
and product delivery.
How longwill ittake?
How longwill ittake?
Here’smy
order!
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1061/11/2004
Cycle Time vs Lead Time• To be competitive, you must make:
• Otherwise, you lose business• Or else orders have to be started on
speculation, which means higher risk of rework or failing to satisfy the customer
Cycle Time < Lead Time
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1071/11/2004
Throughput is ...
• This could be:– Modules tested per day– Components produced per week– Defects corrected per month– etc.
• Throughput is a measure of the output of a process
The number of products produced per unit of time
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1081/11/2004
How to Measure Cycle Time?
Static Cycle TimeThe average of the actual cycle times (CT) experienced by some
number (n) of products
CT1 + CT2 + CT3 + CT4 +...+ CTn
nCycle Time =
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1091/11/2004
Using This Measure• This can be used to measure cycle time
for many situations– For example, measure the cycle times of
the tokens in the variability experiment– Or measure the cycle times for cars being
serviced in a “10 minute oil change” station
• But this is not always easy to measure when many of the products are only partway through the process– So we need a dynamic measure
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1101/11/2004
A Measure to Use when Static Cycle Time is Not Convenient
Dynamic Cycle TimeThe total work in process (WIP)
divided by the throughput of the process
Cycle Time =
WIP (products being developed)THROUGHPUT
This equation can be shown from queueing theory. See Gross and Harris, in reference list,
p83.
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1111/11/2004
Revised Variability Experiment
• Take 5 turns for each round• Measure the total number of tokens that
reach the 15th position• After 5 turns, measure WIP• Also measure throughput, cycle time, etc
(see next slide)
Think of this as a 15 step process, with the tokens being work going through the
process
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1121/11/2004
Solution Record forVariability Experiment
Round Total WI P StaticCT
DynamicCT
Through-put
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1131/11/2004
This Page Intentionally Blank
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1141/11/2004
Solutions to Cycle Time Problems
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1151/11/2004
What Makes Cycle Time High?
• Inventory or work in process (WIP)• Higher overhead, longer delays• Process flow variability• Excessive waiting and long queues• Complexity of processes• Redundant and unnecessary steps
These are all related to each other
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1161/11/2004
3 Steps to Cycle Time Improvement
• Reduce variability• Simplify the process• Reduce WIP
This order is recommended
Due to time limitations, we will only address the first and third of these in this module.
Process simplification will be addressed in the next module
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1171/11/2004
Reducing Variability
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1181/11/2004
Variability Increases Cycle Time
USLLSLLSL USLUSLLSLUSLLSL
21 3 4
If price and delivery were equal, which supplier would
you buy a product from?
OUTPUT FROM FOUR DIFFERENTPROGRAMMING SHOPS
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1191/11/2004
Sources of Variability• Movement of programs or documents
between workstations or systems• Machines or software not being available• Hot lots / priority tasks that disrupt normal
activity• Software defects that require rework and
debugging• Special cases that require holds and delays• Inconsistent processes and procedures• Excessive approval requirements• Hundreds more...
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1201/11/2004
Expediting via Priority Tasks•The tendency is to establish many
levels of priorities– Hot– Super hot– Immediate– Per the boss
•Pressure exists to raise the number of priority tasks
•The list always grows
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1211/11/2004
Other Problems Resulting from Priority Tasks
• Managing priorities consumes many resources
• And it delays other jobs
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1221/11/2004
Expediting vs Cycle Time
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70
Percent expeditedCycle
Tim
e M
ult
iplier
for
Non
-Exp
ed
ited
Jo
bs
Cycle
Tim
e M
ult
iplier
for
Non
-Exp
ed
ited
Jo
bs
• When we expedite a product, other products are delayed
• In some cases, each product waits until it is expedited before it moves at all
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1231/11/2004
Variability Management MustMinimize Expediting
7 14 21 28
BAU
NUMBEROF
SHIPMENTS
CYCLE TIME (DAYS)
7 14 21 28CYCLE TIME (DAYS)
NUMBEROF
SHIPMENTS
EXPEDITEDLOTS DELINQUENT
LOTS
With managed priorities, you improve the normal case and reduce the need for
“priority jobs”
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1241/11/2004
A Strategy for Managing Priorities
• Allow only two priorities
– Hot– Not hot
• Control the maximum percentage of “hot” (less than 8% to 10%)– If you add a hot job, you must subtract
another
– This requires discipline (and faith that it works!)
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1251/11/2004
Further Management of Priorities
• Monitor distribution throughout the process– Do not plug up one operation or
individual with priority tasks
• Monitor frequently
• Remove tasks from the priority list when they don’t really need priority
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1261/11/2004
EFFECTIVE UTILIZATION
Cycle Time, Utilizationand Variability
WIP
or
CY
CLE T
IME
100%
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1271/11/2004
Higher Utilization IncreasesCycle Time
EFFECTIVE UTILIZATION
WIP
or
CY
CLE T
IME
100%
SOME VARIABILITY
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1281/11/2004
EFFECTIVE UTILIZATION
Variability Affects theExtent of the Added Delay
WIP
or
CY
CLE T
IME
100%
NO VARIABILITY
SOME VARIABILITY
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1291/11/2004
EFFECTIVE UTILIZATION OR THROUGHPUT
High Low
Variability
Output
OpportunityCycle TimeOpportunity
Variability ReductionOpportunities
WIP
or
CY
CLE T
IME
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1301/11/2004
ReducingWork in Process (WIP)
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1311/11/2004
High WIP = High Cycle Time•Any intermediate work in the
system is WIP.•Anything that is not being actively
processed is excessive WIP.
For related background, see Gross and Harris, in reference list, p83.
Dynamic Cycle Time = WIP
THROUGHPUT
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1321/11/2004
Examples of Excessive WIP for Software
•Code waiting to be tested•Designs waiting to be coded•Specifications waiting to be inspected•Change requests waiting for approval•Hundreds more...
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1331/11/2004
Smooth Flow - Ideal Process
• The ideal process flows smoothly, like a train running on tracks.
Note: tracks are empty most of the time
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1341/11/2004
Uneven Flow - Typical Process
• Lots of exits and entrances
• Vehicles of varying sizes and speeds
Note: streets are usually crowded - with WIP!
• The typical process runs unevenly, like vehicles on a city street
» Some drivers uncertain of what they want to do
» Lots of stoplights to “control” the flow (mainly to prevent collisions)
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1351/11/2004
Techniques for ObtainingSmoother Flow
• Identify bottlenecks and constraints -- and manage/optimize them
• Utilize pull systems instead of push systems
• “Conduct the orchestra” (keep everyone going at the same speed)
• Flow management systems• .....
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1361/11/2004
Observations and Caveats
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1371/11/2004
Too Much Measurement!• The tendency is to measure too
much and in too much detail– Rough estimates usually identify the
biggest problems and opportunities
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1381/11/2004
Some of The Problems &
Solutions are Technical• 30% of the improvement comes from
technical changes– Process changes– Tool changes– Changing rules and operations
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1391/11/2004
Most of The Problems &
Solutions are Not Technical• 70% of the improvement comes from
organizational and people changes, such as– Education– Communication– Management– Teamwork
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1401/11/2004
• Practitioners generally focus on their work and on what they THINK is happening rather than on what IS happening– They tend not to see all of the waits, queues,
etc. that they cause themselves– Their perception of how they spend their time
is generally incorrect– They are too busy getting the job done to see
how they might improve it
Independent Observers See Problems and Opportunities the
Best
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1411/11/2004
• Just as athletes rely on coaches, software engineers need to learn to trust in others to observe and help them do better
The Athletic Coach Analogy
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1421/11/2004
Software Developers areAccustomed to
Improving Cycle Time
•Think of your software development process as a large computer program that runs too slow. – How would you make it run faster?– Imagine how you would speed up a
computer program . . . . . . . .– Then draw analogies to the software
development process
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1431/11/2004
Improve the Process the way you would Speed Up a
Program
Code Speed-up• Faster hardware• Better algorithm• Optimizing compiler• Remove code from
inner loops• Optimize code• …..
Process Speed-up• Faster hardware• Better process• Eliminate waste• Remove redundant
steps• Eliminate wasteful
meetings and approvals
• …..
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1441/11/2004
You don’t pay for the steps you don’t do– Fewer steps means fewer opportunities to
introduce defects– Shorter cycles means less time to change
requirements– Shorter cycles means more time to iterate
designs or benefit from cycles of learning
Cycle Time Bonus It generally turns out that
improved cycle time produces lower costs and higher quality as
well!
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1451/11/2004
Examples ofSoftware Cycle Time
Techniques• “Just-in-time” training• Plan testing and test equipment well in advance• Rethink the detailed design process
– Do you need to maintain detailed design documentation?
– Do you need to do detailed design at all? • Use on-line requirements and design models
instead of paper documents and specifications
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1461/11/2004
Examples of Commonly FoundSoftware Cycle Time Barriers
• Poor communication/cooperation between software development and the rest of the organization
• Poor management of unstable requirements, algorithms and interfaces
• Contention for test assets -- need better planning, assets allocated to software test
• Poorly qualified software subcontractors
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1471/11/2004
More Commonly FoundSoftware Cycle Time Barriers
• Excessive paperwork, signatures, and reporting– Negotiate reductions with
management and customer
• Reuse of software not designed for reuse
• Attempts to use the latest tools and methods -- without adequate support and integration
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1481/11/2004
Summary• Measure cycle time• Measure rework• Use these to show the value of process
improvements• And to convince others that your
techniques are worthwhile– Because many cycle time improvement
techniques are “counterintuitive”
Focussing on cycle time can make the whole process more efficient, and effective
Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 5, Part 1, Page 1491/11/2004
References
• Deming, W. Edwards, Out of the Crisis, MIT Press, 1982.
• Goldratt, Eliyahu M. & Jeff Cox, The Goal, (North River Press, 1984.) Also, Theory of Constraints, It’s Not Luck, and Critical Chain Management.
• Gross and Harris, Fundamentals of Queueing Theory (Wiley).
• Swartz, James B., The Hunters and the Hunted, (Portland, Oregon, Productivity Press, 1994) ISBN 1-56327-043-9.