how four statistical rules forecast who wins a competitive bid

The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab

Powered by

How Four Statistical Rules

Forecast Who Wins a

Competitive Bid

A Complimentary Webinar from Aurora WDC

12:00 Noon Eastern /// Wednesday 22 October 2014

~ featuring ~

Dr. Bill Zangwill Arik Johnson


Powered by

Bill Zangwill

Bill Zangwill is a Professor, Emeritus, from the University of Chicago, Booth School of Business. He has authored four published books, one of which was selected by the Library Journal as “One of the Best Business Books of the Year,” and had over 50 papers in academic journals. In addition, he has had three articles published in the Wall Street Journal. His consulting engagements include top firms such as IBM, AT&T, Motorola, many smaller firms and the US government. He has also taught at the University of Illinois and the University of California, Berkeley. He is considered one of the most innovative thinkers in his field.

Email: [email protected]

The Intelligence Collaborative is the online learning and networking community powered by Aurora WDC, our clients, partners and other friends and dedicated to exploring how to apply intelligence methods to solve real-world business problems.

Apply for a free 30-day trial membership at http://IntelCollab.com or learn more about Aurora at http://AuroraWDC.com – see you next time!

http://intelcollab.com/

http://aurorawdc.com/


Powered by

α Use the Questions pane on your GoToWebinar control panel and all questions will be answered in the second half of the hour.

α You are welcome to tweet any comments on Twitter where we are monitoring the hashtag #IntelCollab or eavesdrop via http://tweetchat.com/room/IntelCollab

α Slides will be available after the webinar for embedding and sharing viahttp://slideshare.net/IntelCollab

α To view the recording and download the PPT file, please register for a trial membership at http://IntelCollab.com.

Questions, Commentary & Content

http://tweetchat.com/room/IntelCollab

http://slideshare.net/IntelCollab

http://intelcollab.com/


Powered by

Agenda

► Introduction

► Risk, Bias, Errors: Caused by Usual Procedures, but Usually Missed

► Application to Situations in Business

► Summary, Q&A and Discussion


Powered by

The Joy When We Win.


Powered by

How do we avoid the grief and attain the joy?


Powered by

Novel Approaches. Two patents, third pending.


Powered by

Why are the patents and new approaches

needed? Because usual methods typically have hidden bias, hidden risks and hidden errors.

► MAJOR CONUNDRUM: We believe in the usual approaches and keep using them because the errors are hidden so are missed and not considered.► E.g., Behavioral research. If we believe the decision will be 70-80%

successful, likely it will be roughly 50%.

► First methodology that expressly helps warn of errors, bias, issues missed► Humans miss issues.

► 2 patents and third pending

► Follow the four rules


Powered by

Four Rules – Boost Win Rate

1. Stop using usual methodologies as they likely have hidden risk, bias, errors. They can cause losses despite our belief in them.

2. Do not assume. TEST.

3. Improve twice. Best ideas come after reviewing situation a couple times.

4. Follow procedure, otherwise bias and risks will slip in secretly, stealthily and surreptitiously.


Powered by

Risk, Bias, Errors: Caused by Usual

Procedures, but Usually Missed

Math. Bias. Surprises.


Powered by

Does Arithmetic Have a Hidden Secret?

Example: SUBTRACTION

100 $100,000,000

-90 -$90,000,000

----- ---------------

10 $10,000,000

WHAT CAN POSSIBLY BE HIDDEN HERE? WHAT MIGHT BE WRONG?


Powered by

In Business Decisions, Most Data Are Not

Known That Perfectly

Example: SUBTRACTION ERROR

100

-90

-----

10

But error in value

100 +/_ 10 (90 – 110) ( 10% error)

-90

------

0 to 20 (100% error)

HUGE ERROR INCREASE FROM 10% to 100%

WORSE IF BOTH FACTORS HAVE ERRORS OR OVER TIME

Math causes error.

TYPICAL ALGORITHM: WE BELIEVE IT IS RIGHT YET INSTEAD PRODUCES ERROR


Powered by

DCF and Other Financial Formula Can

Increase Error

Gordon growth model:

$10 million payout in one year, g = growth rate, r= rate of return

Value, DCF = $10/ (r – g) (in millions)

Example 1: r = 12% , g = 10%

Value = $10/ ( 0.12 - 0.10) = $10/(0.02) = $500 million

Example 2: r = 12% , g = 11% growth rate increased by 10%

Value = $10/ ( 0.12 - 0.11) = $10/(0.01) = $1 billion

10% error in input

100% error in answer.

Proprietary material. Do not disseminate. Patent pending and copyrights University of Chicago and Decision Command, Inc. SLAM(TM) is a registered service mark of Decision Command.


Powered by

More Accurate Using Better Math

Two quantities, 40 and 10

What percentage is 40 of total?

40/( 40 + 10) = 40/50 80%

Suppose 25% error in 40. Could be 30 – 50.

What is percentage of the total now?

30/(30 +10) 75%

50/(50 + 10) 83.66%

Much reduced error relative to 80% base --– error less than 7%

ERROR REDUCED FROM 25% TO LESS THAN 7%.

THE MATH IN MOST ALGORITHMS IN COMMON USE CANNOT BE TRUSTED.

Must use better approaches that reduce error and attack the hidden secret.


Powered by

Improves Accuracy, Reduces Risk

Pij = conditional probability in a cell, Pj = probability of column

(Uses Bayes and Laplace approaches)

Suppose there is risk in a specific cell entry Plm

Consider risk (percent error) in the output Pm over risk in the input Plm.

The fractional risk in output is

The fractional risk in the input is

i ij

j

i ij

j

PP

P

m

lm

m

dP

dPP

lm

1P

Proprietary Information, FOUO


Powered by

Error Percentage Always Decreases,

Proprietary Information, FOUO

i ij i l im i im i l im

jm

2mlmi ij

jm lmlm lm

m i im

lm

i ij

j

i ij i l im i im i l im i ij i im i im i im

j j

lm

i im i im i ij i im

j j

i ij i im

j

i ij

( P ) P P PdP [

dPdP ( P )P dP

[ ](P ) ](P )P P1

PP

( P ) P P P ( P ) P P P

(P )

( P ) P ( P ) P

( P ) P

( P

j

i im

i ij

j

)

P1 1

( P )

Percentage error out less than percentage error in


Powered by

Serious Problems Often Cause Us to Lose: Most

Algorithms Increase Risk and Errors, and We Do Not

Know It

Bayesian Algorithm I Use Reduces Risk and Errors

1. “The model was previously shown to be surprisingly robust to obvious violations of

this independence assumption, yielding accurate classification models even when there are clear conditional dependencies.”

“Many researchers have noted the good performance of SBC, including Clark and Niblett, Knononenko, Langley and Sage, and Domingos and Pazzani.”

2. Beyond Independence: ..is commonly thought to assume that attributes are independent

given the class, but this is apparently contradicted by the surprisingly good performance it exhibits in many domains that contain clear attribute

dependences. No explanation for this has been proposed so far. In this paper we show that the

SBC does not in fact assume attribute independence, and can be optimal even when this assumption is violated by a wide margin

Proprietary. Confidential information. US Patent 7,676.446. Copyright University of Chicago and/or Decision Command.


Powered by

BIAS – Frequent Cause of Not Winning

Bias and optical illusions are similar. What you

believe is not there.

Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO

Holding hands, or not?

INSIGHT: Statistically identify entries that are odd relative to facts of situation, outliers


Powered by

Bias and Hidden Risk

Biases, hidden risk, faulty assumptions tend to be at variance, out-of-kilter with facts of situation.

Anomaly, outlier or oddity is often the clue: risk, bias, faulty assumption

outlier


Powered by

Surprise – Must the Unexpected Always be

Unexpected?

INSIGHT: Identify contradictory and odd information. The greater the level of contradictory and odd information, the greater chance something odd is going on, something was missed.

MATH ADVANCEMENT– Add variable to represent surprise



Powered by

Bayes SystemFirst System to Warn of Risks/Surprises/Missed Issues



Powered by

Usual Methods Typically Have Hidden Error,

Risk and Bias, and Often We Will Not Know It.

► NO AUTOMATIC IDENTIFICATION OF RISK.

► NO AUTOMATIC MEANS TO PREVENT MATH ERRORS.

► NO AUTOMATIC CROSS-CHECKS ON ACCURACY.

► NO AUTOMATIC ANALYSIS OF RISK OR SURPRISE/MISSED

ISSUES.

► NO AUTOMATIC ESTIMATE OF THE PROBABILITY OF SUCCESS.

► NO PROCESS TO RESOLVE DISAGREEMENTS

NEED BETTER WARNING OF THE HIDDEN PROBLEMS

“YOU CAN’T DO A GOOD JOB WITH POOR TOOLS”



Powered by

Proof


TEST AT UNIVERSITY OF CHICAGO

Managers and executives made decisions

They compared quality of decision made this new way versus their

usual decision approach.

40% better and 20% faster


Powered by


1. Stop using usual methodologies as they likely have

hidden risk, bias, errors.

They can cause losses despite our belief in them.


3. Improve twice.

Best ideas come after reviewing situation a couple times.

4. Follow procedure, otherwise bias and risks will slither in

secretly, stealthily and surreptitiously.



Powered by

Application to Situations in Business

Use better ways to handle math, bias, risks



Powered by

Bidding to Invest in a Chilean Firm


INVEST IN EXCELLENT CHILEAN FIRM. PARTNER.

In private education, growing rapidly, wanted funds for international growth and acquisition

AAI could assist with funding growth. Gateway to future.

Several trips to Chile to speak with owner

AAI BIDDING AGAINST ARCH RIVAL ADVENT

Felt confident they would win—

got to know owner, understood situation


Powered by

Situation: AAI Confident of Winning


AAI WERE HIGHLY EXPERT AND DID TRADITIONAL

ANALYSIS WELL---SURE THEY WOULD WIN :

analyzed situation, obtained information, visited Chile,

brought in experts, financial analysis, discussions.

RESISTED USE OF SYSTEMATIC METHOD,

SUPERFLUOUS.

But agreed to use it because easy, fast, and I was a friend.


Powered by

Difficulties with Usual Methods


No goals from perspective of Santo Tomas.

• No metrics for those goals. Prob. ST will select AAI or Advent.

No independent warning of possible biases, risks.

• No probability of missed issues/surprise.

No iterations- building of better and better solns. (First analysis is always incomplete---Donnelly’s rule)

• No metrics to determine if decision is getting better.

No testing if decision excellent or not.

arguments, blind alleys.

Made decision as if they were making the decision.


Powered by

5 Minute Role Play Real Decision Maker


MUST GET AT EMOTIONS. WHAT ARE THEY CONCERNED OR WORRIED ABOUT? Utilize goals, hopes, fears, problems of real decision maker. What could cause failure? What causes customer worries?

Santo Tomas goals.


Powered by

We Fail (Missed before because traditional had

no means to detect.)


EXPOSED HUGE RISK. ASTOUNDED. BUT THEY HAD TO AGREE BECAUSE THEY DID THE EVALUATION.

WHAT SHOULD WE DO?


Powered by

- Test: What are the risks that cause our loss?


Why we lose

TRADITIONAL NO WARNING OF RISKS


Powered by

Traditional Methods: Argue

►BEFORE: Argue who is right.

►SYSTEMATIC APPROACH: TEST.

Identify risks. Examine possible

improvements and check their metrics



Powered by

New approach to win. BOOSTED METRICS:

success rate from 63% to 75% beating Advent’s 70%


REVISED AAI STRATEGY

METRICSAAI 63-75ADVENT 70


Powered by

Test—expose risks again. Still some issues, so

build higher – IMPROVE TWICE



Powered by

Build the win higher—test and improve twice



Powered by

Develop New Strategy - Won


► Bring in experts on international to our team and show Santo Tomas, that we are really best for global expertise. Create connection in Chile to build trust.

► Convince Santo Tomas our financial deal is really better. (less upfront, but more later)

► Ensure CEO comfortable (CEO is owner)

Conclusion: saved them from losing deal

Traditional means fail due to hidden errors, no warning.


Powered by






3. Improve twice.






Powered by

New Product Development at EG Insurance.

What is going on?

Large commercial insurance firm wants to sell to

mortgage originator insurance market

Traditional product based upon annual premium—

Exciting new product they expect to launch to clearly

differentiate themselves from competitors: Chubb,

CNA, Lloyds of London:

A transaction-based plan based upon monthly

volume. Client would only pay for what was actually

used.

WE’RE READY TO GO AHEAD.

Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2013


Powered by

Role Play: What does customer think?

Get at worries.



Powered by

Test: Complete matrix, examine metrics (cannot be done in

traditional)

New product loses. Software saved them from failure! Forced analysis from viewpoint of actual decision maker.


METRICS: Prob. SuccessLow, 16%, 53%.Easy to miss this.

WAY UNDER GOAL OF 75%


Powered by

Warning. METRIC High surprise risk – “You can

do better than this, you missed something”


METRICS:Success, 16%, 53%Surprise 31%


Powered by

Examine Hidden Risks

-- Countering risks leads to new concept.


risks


Powered by

NEW IDEA: Transaction based, but on closed

loans only. Should save money.

►TEST IDEA: DO NOT ASSUME: (cannot

be done with traditional because no

metrics)



Powered by

Test New idea: Transaction based, but on

closed loans only– Surprise still high


METRICS:Prob. of success is lower.Surprise drops to 20% from 31%

What to do now?Build


Powered by

Test: Check Risks


risks


Powered by

Check More Risks and Weaknesses



Powered by

TEST another idea — Large increase in

estimated success probability metric


METRICS:Success 71%Surprise 16%

CLOSE ENOUGH TO 75%


Powered by

TWO IMPROVEMENT CYCLES

Sizable improvement in probability of winning over

prior approach.


Value

Initial-New transaction idea($31)

Closed loans only($54)

ZERO

Cust. choice($74)

Traditional


Powered by

Saved Them From Losing Money.

Usual Means Have Hidden Error

$4.3 MILLION SAVED.

Over 30% improvement . Three cycles.

Exposed that initial evaluation was incorrect

“Having software like this around

to aid in making important

decisions is wonderful.”

“50% faster than regular method.”Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2013


Powered by






3. Improve twice.






Powered by

IT Competitive Bid

WE LOSE

Prob. of success


Powered by

Test: Despite ABC Winning, Maybe ABC Can

Be Beaten

Factors where XYZ beats ABC

where ABC beats XYZ


Powered by

Where are others strong or weak?

XYZ STRENGTHS SMS STRENGTHS

ATT STRENGTHS


Powered by

Improve XYZ to Beat ABC



Powered by

Test: XYZ Still Has Weaknesses


XYZ WEAKNESSES


Powered by

Improve XYZ Second Time to Beat ABC –

IMPROVE TWICE



Powered by

Thank you! Now how about a little Q&A?

Bill Zangwill

Email: [email protected]


Powered by






3. Improve twice.


4. Follow procedure, otherwise bias and risks will secretly,

stealthily and surreptitiously slither in.
