Value Based Software Engineering

What, Why, How & The MysteriousNupul Kukreja

SoCal SPINMay 6 ‘11

© USC-CSSE 2

About Me• Pursuing PhD in Computer Science under Dr.

Barry Boehm. Focus Area: Value Based Software Engineering (VBSE)

• MS in CS from USC and BE in CS from Mumbai University, India

• Worked as a Software Developer at Capgemini Consulting India Pvt. Ltd.

• Lecturer for CS Undergraduates at Watumull Institute (Mumbai University) teaching Computer Graphics and Operating Systems

10/6/2010

© USC-CSSE 3

Agenda• Section 1* : Overview

– Understanding “Theory W”– VBSE’s 4+1 Theory– VBSE Agenda– VBSE: 7 Key Elements (Overview)

• Section 2: Applying 7 Key Elements in practice– Identifying company’s strategic goals and their overall value– Benefits Realization Analysis– Measurements/Estimation and Value of Information– Creating a Value Based Business Case– Elicitation/Reconciliation of Stakeholder Win Conditions– Prioritization of Win Conditions/Requirements using Kano Analysis– Tracking Value Earned

10/6/2010

*Content courtesy: Dr. Barry Boehm

© USC-CSSE 4

Tools & Techniques Involved• Application of VBSE in practice involves

understanding practices from various domains:– Decision Theory– Statistics– Management/People Engineering– Software Engineering Principles

• We shall see how to effectively combine them to help practice VBSE with minimal overhead

10/6/2010

© USC-CSSE 5

Theory W*: Enterprise Success Theorem– An informal proof

• Theorem: “Your enterprise will succeed if and only if it makes winners of your success-critical stakeholders”

• Proof of “if”:Everyone that counts is a winner.Nobody significant is left to complain.

• Proof of “only if”:Nobody wants to lose.Prospective losers will refuse to participate, or will

counterattack.The usual result is lose-lose.

10/6/2010

*An Initial Theory of VBSE – Barry Boehm & Apurva Jain

© USC-CSSE 6

Theory W: WinWin Achievement Theorem

Making winners of your success-criticalstakeholders (SCSs) requires:• Identifying all of the SCSs• Understanding how the SCSs want to win• Having the SCSs negotiate a win-win set of

product and process plans• Controlling progress toward SCS win-win

realization, including adaptation to change.

10/6/2010

7

Initial VBSE Theory: 4+1• Engine: Theory W (stakeholder win-win): What values are

important?– Enterprise Success Theorem– Theory of Justice– Win-Win Equilibrium and Negotiation

• Four Supporting Theories– Dependency Theory: How do dependencies affect value realization?

• Results chains; value chains; cost/schedule/performance tradeoffs

– Utility Theory: How important are the values?• Multi-attribute utility; Maslow’s need hierarchy

– Decision Theory: How do values determine decisions?• Investment theory; game theory; statistical decision theory

– Control Theory: How to monitor and control value realization?• Feedback control; adaptive control; spiral risk control

10/6/2010

8

VBSE Theory 4+1 Structure

10/6/2010

Utility Theory

Theory W:SCS Win-Win

Decision Theory

Dependency Theory

Control Theory

How do dependencies affect value realization?

How to adapt to change and control value realization?

How do values determine decision choices?

How important are the values?

What values are important?How is success assured?

9

Utility Theory

Theory W:SCS Win-Win

Decision Theory

Dependency Theory

Control Theory

6a, 7c. State measurement, prediction, correction; Milestone synchronization

5a. Investment analysis, Risk analysis

1. Protagonist goals3a. Solution exploration7. Risk, opportunity, change management

5a, 7b. Prototyping

2a. Results Chains3b, 5a, 7b. Cost/schedule/performance tradeoffs

2. Identify SCSs

3b, 7a. Solution Analysis

5a, 7b. Option, solution development & analysis

4. SCS expectations management

3. SCS Value Propositions(Win conditions)

SCS: Success-Critical Stakeholder

6, 7c. Refine, Execute, Monitor & Control Plans

5. SCS Win-Win Negotiation

Initial VBSE Theory: 4+1 Process– With a great deal of concurrency and backtracking

10/6/2010

© USC-CSSE 10

VBSE Agenda• Objective: Integrating value considerations into the full range of

existing & emerging Software Engineering principles in a manner so that they ‘compatibly’ reinforce one another

• Major Elements:– VB* Requirements Engineering– VB Architecting– VB Design and Development– VB Verification And Validation– VB Planning and Control– VB Risk Management– VB Quality Management– VB People Management

*VB = Value-Based

10/6/2010

© USC-CSSE 11

VBSE: Seven Key Elements• Benefits Realization Analysis• Stakeholder Value Proposition Elicitation &

Reconciliation• Business Case Analysis• Continuous Risk and Opportunity

Management• Concurrent System & Software Engineering• Value-Based Monitoring & Control• Change as Opportunity10/6/2010

© USC-CSSE 12

1. Benefits Realization AnalysisDMR/BRA* Results Chain

INITIATIVE

Implement a new orderentry system

Assumption(s):-Order to delivery time isan important buying criterion

Contribution

Reduce time to process order

OUTCOME

Reduced order processing cycle(intermediate outcome)

OUTCOME

Increased sales

Contribution

Reduce time to deliver product

*DMR Consulting Group’s Benefits Realization Approach

AccountableStakeholder(s)

© USC-CSSE 13

Key ‘Benefits’ of Doing BRA/Results Chain• Explicitly maps out the intended benefits of the system to be

acquired (or any initiative to be undertaken)• Identifies what all initiatives need to be taken to actually realize the

benefits (A.K.A. Work Breakdown Structure (WBS) at times i.e., “within” the initiatives)

• Identifies the ‘key stakeholders’ accountable for the above initiativesNOTE: • Must be refined as more is learnt about the initiatives or an

unforeseen benefit is uncovered• Done at multiple levels of granularity and stabilizes earlier into the

SDLC than most artifacts• Subtle but valuable: Lays foundation for the relevant metrics

required to ‘track’ the benefits

10/6/2010

2. Stakeholder Value Proposition Elicitation & Reconciliation

• Myth: ALL SCSs have readily expressible and compatible value propositions that can be easily turned into a set of objects for each initiative and the overall ‘portfolio’ of initiatives

• The following figure is also known as a “Model Clash”

1410/6/2010 © USC-CSSE

© USC-CSSE 15

Techniques• There are a myriad techniques that can be applied for

stakeholder value proposition reconciliation:– Expectations Management: Just awareness of potential

conflicts could help SCSs relax their less critical levels of desire– Visualization & Trade-off Analysis Techniques: Prototyping,

scenarios, estimation models help SCSs to mutually understand most important & achievable aspects of the system

– Prioritization: Helps identify which combination of capabilities best satisfies the stakeholders’ most critical needs within available resource constraints

– Groupware: Some groupware tools have support for collaborative brainstorming, discussions, prioritization etc.,

10/6/2010

© USC-CSSE 16

3. Business Case Analysis (BCA)• Another extremely valuable technique for

stakeholder value proposition reconciliation• Helps determine where is the best bang for the

buck – i.e., capabilities providing the best ROI (We’ll see how to create a VB-Business Case later)

• Can directly influence capability prioritization• TWO Major factors influencing BCA

– Unquantifiable benefits (a.k.a. intangibles)– Uncertainties & risk

10/6/2010

© USC-CSSE 17

Techniques• ROI – most commonly used to justify a business

case. Difficult to apply for intangibles• Deriving variables/attributes from an objectives

hierarchy – closer to value based but we can derive them straight from the benefits chain!

• Each of the variables need to be ‘estimated’ either using subjective judgment, past data, expert advice…

• …OR we could ‘calibrate’ the estimator to provide better estimates!

• We’ll see how to do this in Section 2

10/6/2010

© USC-CSSE 18

4. Continuous Risk & Opportunity Management

• Understanding & Addressing People’s Utility Functions– Risk Averse– Risk Neutral– Risk Seeking

Implication(s): • Very people oriented process• Continuous and Iterative• Must have plans/processes in place to identify,

manage and mitigate risks10/6/2010

Utility

Benefit0,0

1,1

It is possible to have negative utilities too

losses

1910/6/2010 © USC-CSSE

Central Tenet of Risk Management• Metric Risk Exposure:

RE = Probability (Loss) * Size (Loss)– “Loss” of stakeholders’ value– Ex.: profits, reputation, quality of life etc.,

• Prioritizing Risks using Risk Exposure (may be misleading!)

Ris

k P

rob

abil

ityHigh

Low

CheckUtility - Loss Estimate

LittleRisk

Low Loss of Utility High

CheckProbabilityEstimate

MajorRisk

© USC-CSSE 20

5. Concurrent System & Software Engineering

• Increasing pace of change in IT marketplace traditional sequential approach to software development (i.e., Waterfall model) is increasingly risky to use!

• Preferable: Concurrently engineer the product’s operational concept, requirements, architecture, life cycle plans and key sections of code

• Concurrent engineering further preferable if:– Software system requirements emergent from usage and/or

prototyping rather than prespecifiable– Costs, benefits & risks of COTS software and/or outsourcing

decisions affects requirements, architectures, cost, schedules etc.,

10/6/2010

21

Choose Relevant Process ModelsLifecycle Objective (LCO) Lifecycle Architecture

(LCA)Initial Operational

Capability (IOC)

For at least one architecture, a system built to that architecture will:

For a specific detailed architecture a system built to that architecture will:

An implemented architecture, an operational system that has:

•Support core Operational concept•Satisfy core requirements•Be faithful to prototype(s)•Buildable in budget/schedule in the plan•Show a viable Business case•Have key SCSs committed to support the ‘next phase’

•Support elaborated operational concept•Satisfy elaborated requirements•Be faithful to prototype(s)•Buildable in budget/schedules in the plan•Show a viable Business case•Have key SCSs committed to support the full lifecycle•All Major risks resolved/covered by a risk management plan

•Realized the operational concept•Implemented the initial operational requirements•Prepared a system operation & support plan•Identified the initial site(s) of deployment for initial transition•Identified users, operators & maintainers to assume operational roles

Anchor point milestones – of the chosen process model

© USC-CSSE 22

Techniques• There are a myriad software process models –

choose the one that fits best to your organization

• Feasibility evidence MUST be provided at each milestone to support the claims as shown on the previous slide

10/6/2010

© USC-CSSE 23

6. Value-Based Monitoring & Control

• Use project’s business case for monitoring the actual business value of the capabilities to be delivered

10/6/2010

Value being

realized?

Determine Corrective Actions

Develop/update business case; time phased cost; benefit flows; plans

Perform to plans

Assumptions still valid?

YES

NONO

YES

© USC-CSSE 24

Techniques• Forms the bulk of the ‘detailed’ section on

VBSE in Practice• What metrics to use? Are those the right

metrics?• Are we gaining benefits? Are we doing the

right things and doing them correctly?• We shall answer these questions in Section 2.

10/6/2010

© USC-CSSE 25

7. Change as Opportunity• Today’s world – a not so good Present• Expending resources to adapt to change is often stated

as “rework costs”• Changes are treated as defects in change tracking

systems• The real world:• Continuous ongoing changes in

– Technology– Marketplace– Organizational– Stakeholders’ value propositions and priorities

10/6/2010

Opportunities can become risks if nothing

is done about it!

A Brave New World• Organizations that can adapt to change more

rapidly will succeed better in the their mission• Developing change anticipatory modular design

can be considered a good investment to enhance system value in the future

• Two techniques for enhancing adaptability to change:– Architecture based– Refactoring based

• Example of “Change as Opportunity”Internet and the Web and their effect on ecommerce 26

© USC-CSSE 27

Section 2:Applying VBSE in Practice

10/6/2010

28

Agenda• Identify the need • Identify the motive/goal/objective of the need to be satisfied• Check alignment with company/organization’s goals and

objectives:• Find/Derive the ‘value’ of the initiative• Identify initiatives to realize value• Identify validate/assumptions when articulating value • Identify success critical stakeholders (SCSs)

• Elicit value propositions (win conditions) of SCSs• Create a value-based business case with information from

above and check for business alignment• ‘Percolate’ the value to all phases of the SDLC (software

development lifecycle)• Track/adjust the business case over time to ascertain value

gained/created and validate assumptions

© USC-CSSE 29

Company Objectives• Before we start – we MUST know on what

basis does the organization decide to pursue any endeavor

• Profit/Revenue is not the only variable…• …Competitive advantage, Brand projection,

Enhanced customer loyalty etc., to name a few• The current undertaking must align with the

strategic goals of the company

10/6/2010

© USC-CSSE 30

Ranking of Strategic GoalsSay you have two programs each satisfying (or satisficing) the organization’s goals in varying degrees, which program should you choose?1. Layout the full ‘results chain’ and see what

benefits can be gained – disadvantage? There is no inherent ranking

2. Use Multi-Attribute Utility Theory MAUT (could also use Analytical Hierarchy Process (AHP) but we’ll focus on the former)

10/6/2010

© USC-CSSE 31

WHY MAUT??• Mathematically sound• Easier to capture the risk attitude in the form

of utility functions• Strategic goals don’t tend to have too many

attributes and thus attribute space is well managed and easily understood

• Easier to lay coarse grained benchmarks for various attributes

10/6/2010

© USC-CSSE 32

MAUT Example• Let’s capture a single utility function for

Revenue• The more revenue the better – increasing utility

10/6/2010

Gamble Sure thing:Best: $10M_

_________________________

Worst: $0.5M

50%

50%

For which value of the sure thing are you indifferent between the sure thing and the gamble? _______

Best: $10M

Worst: $0.5M

© USC-CSSE 33

Graphing Utility

10/6/2010

(0, $0.5M)

(1, $10M)Utility

Revenue

0.5

0.75

0.25

Risk Averse

Risk Seeking

$7.5M

$8.1M

$8.6M$1.5M

$3M

$4.7M

© USC-CSSE 34

Finding Utility• Q: Given a value of the ‘attribute’ what is its utility?

How do we find that from the graph?• This second step involves understanding the

ranking of the attributes:– “If you can change one attribute from its worst state to

the best state which one would you choose? Ties are allowed.”

– Repeat this question till all attributes are ranked• For each of the ranked attributes repeat the

Gamble vs Sure thing lottery to find the ‘P’ i.e., probability at which you’ll be indifferent!

10/6/2010

© USC-CSSE 35

Calculating the Attribute Tradeoff Scaling Constant (ki)

10/6/2010

Gamble Sure thing:

Reference System:

-All attributes at worst state -Chosen attribute at best state

P

1-P

Best SystemAll attributes at best state

Worst System All attributes at worst state

100%_________________________ 0%

P =

For what value of ‘P’ are you indifferent between the sure thing and the gamble? ________

© USC-CSSE 36

Finding Utility (Finally!)

10/6/2010

© USC-CSSE 37

MAUT (Cont’d)Advantages:• Simple question answering and choosing the outcomes of a

lottery (sure thing vs. gamble) • Can easily be ‘computerized’• Understanding a lottery comes naturally to most people and the

risk attitude is hidden within the choices• Difficult to ‘game’ the system with false attitudesDisadvantages:• May be non-intuitive to start with• Time consuming to get 5 point utility functions and attribute

tradeoff scaling constants• Sometimes it’s just difficult to get utility functions (Arrow’s

impossibility theorem)

10/6/2010

© USC-CSSE 38

Next: Benefits Analysis

Understanding the Value of ‘what you are doing’

10/6/2010

© USC-CSSE 39

Before We Begin…• The previous application of MAUT is optional if

your organization already has prioritized it’s strategic goals and objectives (continually updated)

• Still may be valuable to ‘mathematically’ affirm the choices!

• There are a myriad of methods available to help you with this: Balanced Score Cards, AHP etc., They are more ‘management’ i.e., no mathematical basis

10/6/2010

© USC-CSSE 40

Approach• Use a live example from anyone in the audience

(could be any example even something you are currently working on) or use a case study example of Sierra Mountain Bikes (SMB) (Former preferable)

• Walk through the steps of applying VBSE– ONE such way of applying VBSE– Has research viewpoints– Open to discussion– Combines techniques from Statistical Decision Theory,

Management and Software Engineering10/6/2010

*Taken from: An Initial Theory of VBSE – Barry Boehm & Apurva Jain

© USC-CSSE 41

SMB – Problem Description• Susan Swanson, CEO of SMB when developing a shared

constructive vision of SMB’s primary opportunities and problems, determined a significant opportunity of growth, since they produced top quality bikes at a competitive price. Major problem area: old manual order processing system.

• Distributors, retailers and customers were dissatisfied with high rates of late/wrong deliveries, poor synchronization between order entry, confirmation & fulfillment; and disorganized responses to problem solutions.

• It was decided to outsource the development of the new system, since it wasn’t their area of expertise.

10/6/2010

© USC-CSSE 42

Faster, better order

fulfillment system

Increased sales,

profitability, customer

satisfaction

Faster order entry steps, fewer errors On-time assembly

Safety, fair ness of inputs Inter - operability inputs

Lesser time, fewer

errors/order fulfillment

system

10/6/2010

New Order-Entry System

Less time, fewer errors in order

processing

Increased customer

satisfaction, decreased

operation costs

Increased profit, growth

New order entry processes,

training and outreach

Improved suppliercoordination

New Order-Fulfillment System

New Order Fulfillmentprocesses,

outreach, training

Distributors, retailers, customers

Developers

SuppliersSales personnel, distributors

Assumptions:- Increasing market size- Continuing consumer satisfaction

with product- Relatively stable e-commerce

infrastructure- Continued high staff performance

SO WHAT?

What all to do to realize the benefits/ outcomes??

What are the ‘contributions’ of

the initiatives?

Who are the accountable

stakeholders to help realize the

benefits?

Under what assumptions

are these benefits

realizable?

The initiative identified by SMB is to ‘implement a new order entry system’ to

get rid of the manual order processing

© USC-CSSE 43

Benefits Analysis• The previous slide lays out the expected benefits of having a

new order entry/fulfillment system• The ‘initiative’ of having a new system is linked to the final

outcome ‘increased profitability/growth’ (strategic goal?) through intermediate outcomes!

• Each initiative must be followed by an outcome or to realize the benefits of an outcome some initiative(s) need to be taken (may not be so for the final outcome(s))

• Key question to ask at each stage: “So what?” to get the subsequent outcome/initative.

• Stakeholders who are/would be accountable for realizing/reporting the benefits should be identified

10/6/2010

© USC-CSSE 44

Key Questions for performing Benefits Analysis

Q1: Why do you want to know the value of “<initiative>”?A: You must know the why to know the purpose of

performing the benefits analysis i.e., the value of knowing the value of the initiative It’ll help understand the alignment with the organization’s strategic goals (will also help with identifying the need)

Ex: “Why do you want to know the value of having a new order entry processing system?”

A: So that the company has a viable case to know/believe if and whether a new system would/could increase productivity, growth, profit and customer satisfaction and by how much

10/6/2010

© USC-CSSE 45

Key Questions (Cont’d)Q2: What IS the value of “<initiative>”?A: Perform the prior benefits analysis to see the value!!

Q2.1: How do you define “value”?A: You just did! Questions 1 and 2 help answer this

question itself. The “value” in this case is alignment with company goals and ‘how much’ does the said initiative align with it!!

Q3: All this is good but ‘what exactly is the value’? What do I track? There are tons of intangibles to complicate the problem further?

10/6/2010

© USC-CSSE 46

Epiphany: Measurement!• WHAT to measure? We already have standard metrics that

we track for every project• Are those ‘standard metrics’ really tracking overall project

value or just some numbers?• What about intangibles? They are almost impossible to track

at best. We have a 1-N rating system to help us with that• They may be hard, but not impossible! • Douglas Hubbard has even published a book “How to

Measure Anything – Finding the Value of Intangibles in Business”

…Let’s see how we can use some of that wisdom to alleviate our measurement problem. But first…

10/6/2010

© USC-CSSE 47

WHAT to track/measure?• This is where we’ll build a value-based business

case• Directly derivable from our prior benefits

analysis!• Yes, we’ll also track/measure intangibles –

probably won’t be a 1-N scale but actual numbers

• Risks! We didn’t forget that. We’ll also see how to better identify ‘strategic risks’ and measure them too!

10/6/2010

© USC-CSSE 4810/6/2010

Safety, fair ness of inputs

Faster, better order

fulfillment system

Increased sales,

profitability, customer

satisfaction

Faster order entry steps, fewer errors On-time assembly

Inter - operability inputs

Lesser time, fewer

errors/order fulfillment

system

New Order-Entry System

Less time, fewer errors in order

processing

Increased customer

satisfaction, decreased

operation costs

Increased profit, growth

New order entry processes,

training and outreach

Improved suppliercoordination

New Order-Fulfillment System

New Order Fulfillmentprocesses,

outreach, training

Distributors, retailers, customers

Developers

SuppliersSales personnel, distributors

Assumptions:- Increasing market size- Continuing consumer satisfaction

with product- Relatively stable e-commerce

infrastructure- Continued high staff performance

10/6/2010 © USC-CSSE 48

Something is missing…

Something is still missing…

Associated costs of having a new system (make/buy/customize…)

What all measurable items can we find from this diagram?

© USC-CSSE 49

But still…What exactly should be measured?• The previous exercise should promote the

discussion of what is concretely meant by the terms – i.e., how can it be quantified?

• But this is exactly the problem with measuring coarse grained criteria!

• And that’s where we start to think• If you can observe a difference then it can

probably be measured!• In fact the same benefits chain diagram can be

used to simulate the discussion!!• Let’s see how…10/6/2010

REALLY HARD!

© USC-CSSE 50

Example: Growth

10/6/2010

Increase growth

Increased sales

Cost savings

Increase profits

Increase market share

SO WHAT?

This approach probably looks familiar – something from classical decision theory?

© USC-CSSE 51

Objectives Hierarchy• You could replace the previous analysis with that of

‘objectives’ hierarchy• In fact you could replace the entire benefits chain analysis

with an objectives hierarchy (You already know the top most goal “Why do you want to know the value of “_____”?)

• You will however lose the connections between outcomes, stakeholders accountable, initiatives to be taken etc., of the benefits chain approach

• Suggestion: You can use it at this stage when ‘drilling down’ the ambiguous metrics to more specific attributes! OR

• Transform the benefits chain into an objectives hierarchy if you are comfortable with that!

10/6/2010

© USC-CSSE 52

How to Measure/Estimate• Once you’ve drilled down both, tangibles and

intangibles to concretely observable (measurable) facts we can proceed with how best to estimate/measure them

• The prior analysis from benefits chain all the way to measurable attributes should provide enough dimensions on which to capture the value of the project

• But aren’t the estimates just guesstimates for the ‘value based business case’?

• Yes, to some extent. Let’s see how to solidify these estimates…

10/6/2010

© USC-CSSE 53

Are you 90% Confident?• When was the last time you were 100%

confident with your estimates?• Were you confident 100% of the time ?• 100% confident estimates 100% of the time

would make you a priceless asset!!• Can you at least be 90% confident 90% of the

time?• Wait a minute! How can you measure this

confidence level??

10/6/2010

That’s where we take Douglas Hubbard’s help. Remember him?

© USC-CSSE 54

Getting a 90% Confidence Interval (CI)

• Remember CIs from your statistics class?• A 90% CI is a range that has a 90% chance of

containing the ‘correct answer’• Let’s try a simple test: (No need to Google )

Provide a range wide enough so that you believe there is a 90% chance that the value will be within this range (Units don’t matter. We are more interested in the range)

10/6/2010

Question Lower Bound Upper Bound

What is the circumference of Mars? _____________ _____________

© USC-CSSE 55

Would you Gamble?Let’s say we gave you the following proposition:• You win $1000 if the actual answer is within your range• You win $1000 by spinning the wheel below:

• What would you NOW CHOOSE?

10/6/2010

Win $1000

$0 10%

90%

© USC-CSSE 56

Choices & InferencesChoice Inference

GambleYou think the gamble has a higher

chance of payoff! Implies your 90% CI is NOT really your 90% CI but lower –

50%, 60%, 80%. Basically an overconfident estimate

Your initial estimateThis means that you think there is

more than a 90% chance your range contains the answer! An underconfident estimate

IndifferenceYou are indifferent between your

initial estimate and the gamble an equivalent 90% chance 90% CI

10/6/2010

© USC-CSSE 57

Sure Thing vs Gamble• If you are still with me you’ll realize that the

previous exercise is similar to the one we did for capturing stakeholder utility!

• We just replace the ‘sure thing’ by your estimate and the gamble is a 90% Win and 10% loss!

• You ARE using utility theory concepts to make a 90% confident estimate that factors in uncertainty and probable prior knowledge of experts!

10/6/2010

© USC-CSSE 58

The Value Based Business Case• The prior analyses of benefits, initiatives, stakeholder

identification, metrics can provide a great first cut value based business case

• For quantities that need to be estimated you could apply the 90% CI calibration exercise to get ‘better’/more reliable estimates

• However, one question is not enough to help you calibrate. You may have to spend some time practicing calibrating yourself*

• You may need some sample information to help you get started…

10/6/2010*Please refer “How to Measure Anything…” by Douglas Hubbard for more details/exercises

© USC-CSSE 59

Value of Information• How much time/money should you spend (if at all) gathering information

‘about’ the metric before estimating?• That depends on the expected value (EV) of information – Commonly applied

in statistical decision theory but uncommonly so in software engineering!• Defined as Expected Value of Perfect Information (EVPI):

EVPI = EV(perfect info) – EV(no info)

• EVPI is the ‘theoretical maximum’ you should be willing to pay for procuring further information

• Since perfect information is not obtainable, EVSI (expected value of sample information) is usually employed. Thus the net expected value of information is gained by ‘subtracting’ cost of sampling/procuring information!

• This was elucidated way back in 1981 by Dr. Boehm when he published the Blue Book*: “Software Engineering Economics”!

10/6/2010

*Chapter 20: Statistical Decision Theory: Value of Information

© USC-CSSE 60

Next: Eliciting Stakeholders’ Win Conditions

(a.k.a., Value Propositions)

10/6/2010

© USC-CSSE 61

Stakeholders’ Value Propositions• We started our discussion with Theory-W…• …we now build on how to ‘implement’ it in

action• Value Propositions are also known as Win

Conditions• Theory-W involves eliciting, understanding

and reconciling SCS (Success Critical Stakeholders) win conditions with achievable solutions

10/6/2010

© USC-CSSE 62

Win-Win Negotiation Process*

• Collect SCS win conditions (WC) – basically brainstorming what the ‘expect’ from the system

• Converge on WCs – since it’s captured in natural language effort is taken to converge on a concisely worded, non-redundant, unambiguous list of Win Conditions

• Define glossary of key terms – basically the domain glossary used within a project

• Prioritize WCs – Business Value vs. Ease of Realization• Reveal Issues/Constraints – the prioritization helps facilitate discussion

about issues that may arise with the stakeholder’s expectations• Identify Issues and Options – the team posts issues (if any) for each of the

identified win conditions and identifies the options to deal with the issues• Negotiate Agreements – negotiate/agree to the above, using oral

discussion and entering it into the ‘online tool’. We say WinWin Equilibrium is reached when all WCs and Options have been agreed to.

10/6/2010

*Stakeholder Value Proposition Elicitation and Reconciliation – Grunbacher et. al.

© USC-CSSE 63

How/Where to Capture Win Conditions

• I’m currently working on an online collaboration tool, similar to Facebook to help capture stakeholder’s win conditions, raise issues, supply options, prioritize win conditions and indicate the current statue of equilibrium

• Attached are the screenshots of the tool. The tool will be deployed on USC’s servers for the upcoming class of Software Engineering CS577a in Fall ’11 (The tool will also be available for download for use by anyone)

10/6/2010

64

Winbook

© USC-CSSE 65

Prioritization• These are two primary approaches that I’ve narrowed down from a

value based perspective1.Incremental Funded Methodology (IFM)

Advocated by the book “Software by Numbers” – Mark Denne, Jane Cleland-Huang.Deals with grouping features into minimum marketable features and performing financial analysis using net present values and using that to predict ROI

2.Kano AnalysisCommonly applied in Quality Function Deployment for prioritizing voice of customer data. Also advocated by Agile Mentor Mike Cohn for use in agile software development.

• We’ll go with the latter since the former is mathematically involved. Both could be applied in tandem, but we’ll go with the latter

10/6/2010

© USC-CSSE 66

Kano Analysis

10/6/2010

Image From: http://en.wikipedia.org/wiki/Kano_model

Linear: The more of the feature the better

Must be: Absence implies severe dissatisfaction. Expected to be present

Exciter: If absent customer wouldn’t care but would be delighted to have them

So, how do we know which features fall in which category?

© USC-CSSE 67

Kano Questionnaire*

You ask the following question(s) to the prospective users:

10/6/2010

FunctionalForm of

Question

If feature X were present how

would you feel?

I like it that way

I expect it to be that way X

I am neutral

I can live with it that way

I dislike it that way

Dysfunctional Form of

Question

If feature X were absent how

would you feel?

I like it that way

I expect it to be that way

I am neutral

I can live with it that way

I dislike it that way X*Further details can be found in Agile Estimating and Planning – Mike Cohn.

© USC-CSSE 68

Cross-referencing the Responses

Customer Requirements

Dysfunctional Question

Like Expect Neutral Live with Dislike

Functional Question

Like Q E E E L

Expect R I I I M

Neutral R I I I M

Live with R I I I M

Dislike R R R R Q

10/6/2010

E Exciter L Linear

M Must Have Q Questionable

I Indifferent R Reverse

© USC-CSSE 69

Percolation of Value into the SDLC• The prioritized win conditions give an idea of how the customers

values the features on a scale of desirability• The win conditions are the set of value propositions of the

stakeholders – their expectation(s) from the system• Win conditions can be ‘mapped’ to benefits – either in the form of a

matrix or by visual inspection• You can even ‘rank’ the benefits in order of importance using MAUT

too!• Decisions about various win conditions are then taken with respect to

the benefits to be achieved• Decisions at the project level must factor in the impact on strategic

goals• This ‘framework’ of expected benefits should always be visible to the

development team!

10/6/2010

© USC-CSSE 70

Value Based Tracking• I haven’t explored this area in it’s totality but what seems to hold a

lot of promise is “Multivariate Statistical Analysis” especially Multivariate Regression Analysis

• We can pick out key ‘independent variables’ from the intermediate outcomes of the benefits chain and let the ‘dependent variable’ be that from the final intended goal/outcome

• We can run a regression analysis to know if we really gained the benefits. Statisticians have been doing it all along!

• This implies that if we DON’T have relevant data we may need to perform sampling to have enough data points to warrant this analysis.

• Is the sampling effort worth it? It depends on the value of information!

10/6/2010

© USC-CSSE 71

Wrapping Up!

…finally

10/6/2010

© USC-CSSE 72

Conclusion• Presented ONE way of applying VBSE in practice using

ONE set of tools and techniques.• There are myriads of possible combinations of

tools/techniques that could be used. Use the one best suited to your organization…

• …The ideas in this presentation should help you get started

• We’ve covered a lot of ground but it’s only possible to present a limited amount. I’ve attached a set of references for further reading!

• Hope you gained some ‘value’ out of this presentation!

10/6/2010

© USC-CSSE 73

References• Value Based Software Engineering – Stefan Biffl et. al.• How to Measure Anything - Douglas Hubbard• The Information Paradox – John Thorp• Software Engineering Economics – Barry Boehm• Making the Software Business Case – Donald Reifer• Decision with multiple objectives – Keeney, Raiffa• Agile Estimation and Planning – Mike Cohn• Software by Numbers – Mark Denne, Jane Cleland-Huang• Lean Software Development - Poppendieck• Agile Software Requirements – Dean Leffingwell

10/6/2010