MIT formula for uncertainty: pad your estimates

I don’t have the time to delve into the entire article, so I may be misrepresenting the thrust of Spyros Makridakis, Robin M. Hogarth and Anil Gaba’s Winter 2010 article in the MIT Sloan Management Review (full article here).   Or more properly, I hope that this Forbes excerpt — Why Forecasts Fail. What To Do Instead — misrepresents the piece.  For instance, what do the authors mean here?

Say you’re a publisher and have an unknown author selling her first novel. Publishers should look at the sales track records of first-time novelists in general. The uncertainty surrounding your author shouldn’t differ from the wider population of new authors. You should be able, therefore, to estimate how low or high the sales might go. That range probably covers 95% of all possible outcomes. The next step would be to take the estimated range and increase it. 

Wow…I guess the WAG now has the MIT Sloan seal of approval.  Or perhaps it is CYA that’s now approved.  Other than allowing a forecaster to say — “I told you so” —  is there really all that much use in simply increasing the range of estimates? 

Maybe we should spend our time learning how to distinguish the outliers quickly — e.g., from distinct initial sales patterns or unusually intense media coverage — so we can take appropriate action?

4 Responses

  1. The average of the averages, according to the central limit theorem, approaches the normal distribution, which gives a useful estimate. That part makes sense. But what is the confidence interval? Is it 10, 20, 50, or 100 percent over? Too often that “padding” is driven by a WAG or CYA number. I’m sure the MIT folks could come up with a statistically significant range, which can be justified. More importantly those estimates are better risk management. The estimate may even be more competitive… and profitable.

  2. The CI and the cost of getting the relevant topic to “fit” in that CI are the heart of the matter. But IMO part of the challenge is that stats can become CYA — and “scientific” CYA at that. We glide over assumptions that, if reviewed or understood, invalidate the analyitical tools brought to bear. Business schools still teach supply chain formulas that hold so much constant that the only businesses they apply to are Chapter 7 liquidations.

  3. Not quite WAG’s … here’s the relevant paragraph that the above quote is a summary of:

    Assess the level of uncertainty you face. By all means, model your uncertainty as if it were subway uncertainty: Use a statistical model if you’re feeling mathematical, then consider how coconut uncertainty might come into play. Ironically, having accepted uncertainty, you can start to gather more data and judgments than you might otherwise have thought relevant. Take, for example, the sales of a first novel by an unknown author. It sounds like a unique case. But our suggestion to publishers is to ignore the uniqueness. Instead, look at the track record of the sales of first-time authors in general. You have no valid reason to believe that the uncertainty surrounding your new author differs from the wider population of new authors that he or she belongs to — especially if you’ve used an industry standard process for collecting reader feedback (also known as human judgment). Therefore, you should have a reasonable estimate of just how low or high the sales might go. That range probably covers 95% of all possible outcomes. Done that? Well, now take the estimated range … and increase it! Hence the next step: Augment.

    The full article is here (registration required):

  4. Hi Sean,
    Thanks for the comment…per my opening paragraph, I hoped the excerpt was at fault, not the article. Estimation is a very seductive trap, so I was disappointed that an article with such a promising title yielded an excerpt with prescriptions that seemed to focus on fudge factors.

    I don’t have time to register now, but the findings outlined on give me hope that the article is better. For example: “Instead of seeking predictability, managers should channel their efforts into being prepared for different contingencies.” That’s more like it.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: