How to MeasureAnything

Third Edition

DouglAs W. HubbArD

How to MeasureAnything

Finding the Value of “Intangibles” in Business

Cover design: WileyCover image: © iStockphoto.com (clockwise from the top); © graphxarts, © elly99, © derrrek, © procurator, © Olena_T, © miru5

Copyright © 2014 by Douglas W. Hubbard. All rights reserved.

Published by John Wiley & Sons, Inc., Hoboken, New Jersey.First edition published by John Wiley & Sons, Inc., in 2007. Second edition published by John Wiley & Sons, Inc., in 2010.Published simultaneously in Canada.

No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 646-8600, or on the Web at www.copyright.com. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, or online at http://www.wiley.com/go/permissions.

Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.

For general information on our other products and services or for technical support, please contact our Customer Care Department within the United States at (800) 762-2974, outside the United States at (317) 572-3993, or fax (317) 572-4002.

Wiley publishes in a variety of print and electronic formats and by print-on-demand. Some material included with standard print versions of this book may not be included in e-books or in print-on-demand. If this book refers to media such as a CD or DVD that is not included in the version you purchased, you may download this material at http://booksupport.wiley.com. For more information about Wiley products, visit www.wiley.com.

Library of Congress Cataloging-in-Publication DataHubbard, Douglas W., 1962– How to measure anything : finding the value of intangibles in business / Douglas W. Hubbard.—Third edition. pages cm Includes bibliographical references and index. ISBN 978-1-118-53927-9 (cloth); ISBN 978-1-118-83644-6 (ebk); ISBN 978-1-118-83649-1 (ebk) 1. Intangible property—Valuation. I. Title. HF5681.I55H83 2014 657’.7—dc23

2013044540Printed in the United States of America

10 9 8 7 6 5 4 3 2 1

I dedicate this book to the people who are my inspirations for so many things: to my wife, Janet, and to our children, Evan, Madeleine, and Steven, who show every potential

for being Renaissance people.

I also would like to dedicate this book to the military men and women of the United States, so many of whom I know personally. I’ve been out of the Army National Guard for many years, but I hope my efforts at improving battlefield logistics for the U.S. Marines by using better measurements

have improved their effectiveness and safety.

vii

Contents

Preface to the Third Edition xiiiAcknowledgments xixAbout the Author xxi

Part I THE MEASUREMENT SOLUTION EXISTS 1

Chapter 1 The Challenge of Intangibles 3

The Alleged Intangibles 4Yes, I Mean Anything 5The Proposal: It’s about Decisions 7A “Power Tools” Approach to Measurement 10A Guide to the Rest of the Book 11

Chapter 2 An Intuitive Measurement Habit: Eratosthenes, Enrico, and Emily 15

How an Ancient Greek Measured the Size of Earth 16Estimating: Be Like Fermi 17Experiments: Not Just for Adults 20Notes on What to Learn from Eratosthenes, Enrico, and Emily 25Notes 27

Chapter 3 The Illusion of Intangibles: Why Immeasurables Aren’t 29

The Concept of Measurement 30The Object of Measurement 37The Methods of Measurement 40Economic Objections to Measurement 48The Broader Objection to the Usefulness of “Statistics” 52

viii Contents

Ethical Objections to Measurement 55Reversing Old Assumptions 58Notes 65

Part II BEfORE YOU MEASURE 69

Chapter 4 Clarifying the Measurement Problem 71

Toward a Universal Approach to Measurement 73The Unexpected Challenge of Defining a Decision 74If You Understand It, You Can Model It 80Getting the Language Right: What “Uncertainty” and “Risk” Really Mean 83An Example of a Clarified Decision 84Notes 90

Chapter 5 Calibrated Estimates: How Much Do You Know Now? 93

Calibration Exercise 95Calibration Trick: Bet Money (or Even Just Pretend To) 101Further Improvements on Calibration 104Conceptual Obstacles to Calibration 106The Effects of Calibration Training 111Notes 118

Chapter 6 Quantifying Risk through Modeling 123

How Not to Quantify Risk 123Real Risk Analysis: The Monte Carlo 125An Example of the Monte Carlo Method and Risk 127Tools and Other Resources for Monte Carlo Simulations 136The Risk Paradox and the Need for Better Risk Analysis 140Notes 143

Chapter 7 Quantifying the Value of Information 145

The Chance of Being Wrong and the Cost of Being Wrong: Expected Opportunity Loss 146The Value of Information for Ranges 149Beyond Yes/No: Decisions on a Continuum 156The Imperfect World: The Value of Partial Uncertainty Reduction 159

Contents ix

The Epiphany Equation: How the Value of Information Changes Everything 166Summarizing Uncertainty, Risk, and Information Value: The Pre-Measurements 171Notes 172

Part III MEASUREMENT METHOdS 173

Chapter 8 The Transition: From What to Measure to How to Measure 175

Tools of Observation: Introduction to the Instrument of Measurement 177Decomposition 180Secondary Research: Assuming You Weren’t the First to Measure It 184The Basic Methods of Observation: If One Doesn’t Work, Try the Next 186Measure Just Enough 188Consider the Error 189Choose and Design the Instrument 194Notes 196

Chapter 9 Sampling Reality: How Observing Some Things Tells Us about All Things 197

Building an Intuition for Random Sampling: The Jelly Bean Example 199A Little about Little Samples: A Beer Brewer’s Approach 200Are Small Samples Really “Statistically Significant”? 204When Outliers Matter Most 208The Easiest Sample Statistics Ever 210A Biased Sample of Sampling Methods 214Experiment 226Seeing Relationships in the Data: An Introduction to Regression Modeling 235Notes 243

Chapter 10 Bayes: Adding to What You Know Now 247

The Basics and Bayes 248Using Your Natural Bayesian Instinct 257

x Contents

Heterogeneous Benchmarking: A “Brand Damage” Application 263Bayesian Inversion for Ranges: An Overview 267The Lessons of Bayes 276Notes 282

Part IV BEYONd THE BASIcS 285

Chapter 11 Preference and Attitudes: The Softer Side of Measurement 287

Observing Opinions, Values, and the Pursuit of Happiness 287A Willingness to Pay: Measuring Value via Trade-Offs 292Putting It All on the Line: Quantifying Risk Tolerance 296Quantifying Subjective Trade-Offs: Dealing with Multiple Conflicting Preferences 299Keeping the Big Picture in Mind: Profit Maximization versus Purely Subjective Trade-Offs 302Notes 304

Chapter 12 The Ultimate Measurement Instrument: Human Judges 307

Homo Absurdus: The Weird Reasons behind Our Decisions 308Getting Organized: A Performance Evaluation Example 313Surprisingly Simple Linear Models 315How to Standardize Any Evaluation: Rasch Models 316Removing Human Inconsistency: The Lens Model 320Panacea or Placebo?: Questionable Methods of Measurement 325Comparing the Methods 333Example: A Scientist Measures the Performance of a Decision Model 335Notes 336

Chapter 13 New Measurement Instruments for Management 339

The Twenty-First-Century Tracker: Keeping Tabs with Technology 339Measuring the World: The Internet as an Instrument 342Prediction Markets: A Dynamic Aggregation of Opinions 346Notes 353

Contents xi

Chapter 14 A Universal Measurement Method: Applied Information Economics 357

Bringing the Pieces Together 358Case: The Value of the System That Monitors Your Drinking Water 362Case: Forecasting Fuel for the Marine Corps 367Case: Measuring the Value of ACORD Standards 373Ideas for Getting Started: A Few Final Examples 378Summarizing the Philosophy 384Notes 385

Appendix Calibration Tests (and Their Answers) 387Index 397

xiii

Preface to the Third Edition

I can’t speak for all authors, but I feel that a book—especially one based largely on ongoing research—is never really finished. This is

precisely what editions are for. In the time since the publication of the second edition of this book, I continue to come across fascinating published research about the power and oddities of human decision making. And as my small firm continues to apply the methods in this book to real-world problems, I have even more examples I can use to illustrate the concepts. Feedback from readers and my experience explaining these concepts to many audiences have also helped me refine the message.

Of course, if the demand for the book wasn’t still strong six years after the first edition was published, Wiley and I wouldn’t be quite as incentivized to publish another edition. We also found this book, written explicitly for business managers, was catching on in universi-ties. Professors from all over the world were contacting me to say they were using this book in a course they were teaching. In some cases it was the primary text—even though How to Measure Anything (HTMA) was never written as a textbook. Now that we see this growing area of interest, Wiley and I decided we should also create an accompanying workbook and instructor materials with this edition. Instructor mate-rials are available at www.wiley.com.

In the time since I wrote the first edition of HTMA, I’ve written a second edition (2010) and two other titles—The Failure of Risk Manage-ment: Why It’s Broken and How to Fix It and Pulse: The New Science of Harnessing Internet Buzz to Track Threats and Opportunities. I wrote these books to expand on ideas I mention in earlier editions of How to Measure Anything and I also combine some of the key points I make in these books into this new edition.

For example, I started writing The Failure of Risk Management because I felt that the topic of risk, on which I could spend only one chapter and a few other references in this book, merited much more space. I argued that a lot of the most popular methods used in risk assessments and risk management don’t stand up to the bright light of