Law of Averages Overturned


by Tony Kontzer

All those business school lessons about using historical averages to predict probable outcomes? Apparently, that may have been precisely the wrong approach.

In his new book, The Flaw of Averages (2009, John Wiley & Sons, Inc.), Sam Savage, a consulting professor of management science and engineering at Stanford University, posits the theory that averages are a hopelessly useless statistic for determining risk, and that it's time for business decisions to be made using more accurate predictors.

One possible answer, Savage claims, is the emerging field of "probability management," which draws upon a type of data Savage has helped define known as "distribution strings." Unlike numbers, distribution strings are based on the unknown, making it possible to take the uncertain into account when determining a probability. So instead of a spreadsheet cell containing one number, it contains a distribution string of thousands of numbers--called Monte Carlo trials--that constantly are updated any time another cell in the spreadsheet is changed. The idea is that risk isn't a fixed number--it's an ever-changing collection of averages that are inter-related.

"The flaw of averages happens when people plug a single number into a cell to represent a probability," Savage said during a recent interview. "Think of taking a spreadsheet and adding a third dimension to it. Any cell in your spreadsheet should be able to provide you with an average."

It's a big concept to get one's brain around, so to illustrate the problem consider this crude example of how Savage believes averages doom many IT projects: Take a software development project in which 10 separate teams are each working on a particular sub-routine, with no interdependence between them at all. The project manager isn't sure how long each sub-routine will take, but he knows the average will be 3 months, so he relays that to the boss when pressed.

Unfortunately, according to Savage, there is only one chance in a thousand that the project will be done in 3 months--the same odds as flipping a quarter and have it come up heads 10 straight times. In the end, the boss is unhappy, the project manager is held responsible, and stress levels for the next development project rise. Ultimately, companies find themselves resigned to accept that most software projects will take longer than expected, when in fact the problem is that the practice of using historical averages to compute probable completion dates is fundamentally flawed.

The way Savage see it, this dependence on averages has had some pretty grave consequences, which the recession highlights. "We ignored this, and we flew into the side of a cliff," says Savage. He expects that risk analysis will be handled quite differently going forward. "There will be a tremendous move, after this recent meltdown, of people trying to understand risk and uncertainty better."

For IT, in addition to rethinking how project timelines are estimated, this is likely to mean the need to deploy and support tools that can handle this ramped up focus on risk and uncertainty. New applications will be needed for processing distribution string data, and then for analyzing that data to determine probabilities.

In the meantime, if you want to give probability management a test drive, you have two options. You can purchase Frontline Systems' RiskSolver application, which is the brainchild of company President Dan Fylstra, who was the original distributor of VisiCalc back when the Apple II made its debut in 1979. Or, you can try downloading Savage's less powerful--and less expensive--XLSim.

And you may want to brush up on your knowledge of probability management, distribution strings and Monte Carlo trials--something tells me your ability to sufficiently analyze risk and uncertainty will soon depend on it.

Related: Scott Rosenberg on the difficulties of software development.


3 Comments for "Law of Averages Overturned"

  • DrK July 10, 2009 10:14 am

    In "Waltzing with Bears" authors DeMarco and Lister explain the use of probabilities and Monte Carlo simulations to estimate project outcomes. It's not new and the probabilities are still based on historical data. The difficulty remains, however, that management wants a date and not a probability distribution. So what is needed is a change in mental models and attitude about project estimation in general -- THAT is new! It is less realistic and accurate to say "we'll be done 1-May" than "there's a 85% probability we'll be done 1-May, and 99% by 1-July" -- that's what this approach offers and that is very good and needed. The challenge is the culture change needed to be able to use this clearly better approach. Leon Kappelman Professor of Information Systems Chair, SIM Enterprise Architecture Working Group Director Emeritus, IS Research Center Fellow, Texas Center for Digital Knowledge College of Business, University of North Texas

  • Marc Thibault July 10, 2009 9:44 am

    This is scary.

    My first thought was, "Nice strawman." Then I realized he was serious. People actually use simple averages for project planning?

    Admittedly it was a few years ago, but the last time I was involved in a serious project planning effort, the charts came with error bars and the numbers came with medians, skews and standard deviations.

    If that's not covered in the various PM certification schemes, we have a problem and this book may be the solution.

  • David Dahl July 10, 2009 9:32 am

    I've been a CTO for a handful of years and I've spent considerably more years focusing on delivery. We all know business-types glaze over when it comes to anything technical - and some do so as a Pavlovian response to speaking about any topic with anyone it Technology. My point is even if they're glazed, they will still comprehend 1) dates and 2) (my favorite) confidence. The confidence factor is a simple percentage that indirectly correlates to risk. If I hear Project A will take 5 people 3 months I immediately ask for level of confidence - 90% or 10%. If confidence is low you find out why and blow those risks up before proceeding further. I never give delivery dates (and never let my staff give delivery dates) without that qualifying confidence factor. How that's "calculated" can be determined by your particular situation. So "probability management" sounds incredibly interesting, but I'll likely stick with "confidence factors" for now. Dave

Leave a Comment