I just finished reading The Flaw of Averages – Why We Underestimate Risk in the Face of Uncertainty (2009) by Sam Savage, a professor at Stanford. The author states on his website that “[s]imply stated, the Flaw of Averages implies that”:
Plans based on average conditions are wrong on average.
The book deals with uncertain numbers (e.g. how many sales will product X have per month in the next year?), and more specifically, the erroneous forecasting of uncertain numbers due to incorrect use of averages. Savage distinguishes two forms of the Flaw of Averages:
- Weak Form of the Flaw of Averages: using a single number (or regression line) in forecasting future values of an uncertain number, instead of taking into account the distribution of possible outcomes;
- Strong Form of the Flaw of Averages: also screwing up the average itself. From page 83: “Consider a drunk staggering down the middle of a busy highway and assume that his average position is the centerline. Then the state of the drunk at his average position is alive, but on average he’s dead.”
Pages 130–132 list the Seven Deadly Sins of Averaging, which were first published in the article Probability Management in ORMS Today in 2006. In fact, the list has grown beyond seven since then. But Savage states on page 130:
I plan to go on calling them the Seven Deadly Sins regardless of how long the list becomes. Be sure to check in at FlawOfAverages.com to see where it stands today.
Both the 2009 edition of his book and today’s version of the website list twelve sins. Both lists reference scenarios explained elsewhere in the book. Therefore, I will quote sins 1 to 7 from the self-contained ORMS Today article; I will quote sins 8 to 12 from the book, and/or refer within []’s to online resources of my choice.
- The Family with 1 1/2 Children: Often the “average” scenario, like the “average” family with 1 1/2 children, is non-existent. For example, a bank may have two main groups of young customers — students with an average income of $10,000 and young professionals with an average income of $70,000. Would it make sense for the bank to design products or services for customers with the average income of $40,000?
- Why Everything is Behind Schedule: Imagine a software project that requires 10 separate subroutines to be developed in parallel. The time to complete each subroutine is uncertain and independent, but known to average three months, with a 50 percent chance of being over or under. It is tempting to estimate the average completion time of the entire project as three months. But for the project to come at three months or less, each of the 10 subroutines must be completed at or below its average duration. The chance of this is the same as flipping 10 sequential heads with a fair coin, or less than one in a thousand!
- The Egg Basket: Consider putting 10 eggs all in the same basket, versus one by one in separate baskets. If there is a 10-percent chance of dropping any particular basket, then either strategy results in an average of nine unbroken eggs. However, the first strategy has a 10-percent chance of losing all the eggs, while with the second, there is only one chance in 10 billion of losing all the eggs.
- The Risk of Ranking: It is common when choosing a portfolio of capital investment projects to rank them from best to worst, then start at the top of the list and go down until the budget has been exhausted. This flies in the face of modern portfolio theory, which is based on the interdependence of investments. According to the ranking rule, fire insurance is a ridiculous investment because on average it loses money. But insurance doesn’t look so bad if you have a house in your portfolio to go along with it.
- Ignoring Restrictions: Consider a capital investment in infrastructure sufficient to provide capacity equal to the “average” of uncertain future demand. It is common to assume that the profit associated with average demand is the average profit. This is generally false. If actual demand is less than average, clearly profit will drop. But if demand is greater than average, the sales are restricted by capacity. Thus, there is a downside without an associated upside, and the average profit is less than the profit associated with the average demand.
- Ignoring Optionality: Consider a petroleum property with known marginal production costs and an uncertain future oil price. It is common to value such a property based on the “average” oil price. If oil price is above average, the property is worth a good deal more. But if the price drops below the marginal cost of production, the owners have the option to halt production. Thus, there is an upside without an associated downside, and the average value is greater than the value associated with the average oil price. (…)
- The Double Whammy: Consider a perishable inventory of goods with uncertain demand, in which the quantity stocked is the “average” demand. If demand exactly equals its average, then there are no costs associated with managing the inventory. However, if demand is less than average then there will be spoilage costs, and if demand is greater than average there will be lost sales costs. So the cost associate with average demand is zero, but average cost is positive.
- The Flaw of Extremes: In bottom-up budgeting, reporting the 90th percentile of cash needs leads to ever thicker layers of unnecessary cash as the figures are rolled up to higher levels. Even more harmful things result from focusing on above- or below-average results, such as test scores or health-related statistics. (…) [From p138: T]he flaw of extremes results from focusing on abnormal outcomes such as 90th percentiles, worse than average cancer rates, or above average test scores. Combining or comparing such extreme outcomes can yield misleading results. (…) The smaller the sample size, the greater the variability of the average of that sample.
- Simpson’s Paradox: [see Simpson’s Paradox (Wikipedia) and Chapter 18 online supplement]
- The Scholtes Revenue Fallacy: [From p146: T]he Scholtes Revenue Fallacy occurs when revenue is the result of multiplying two uncertain numbers, such as (…) price and quantity. If the two uncertain numbers are inversely (negatively) interrelated, the average revenue is less than the revenue associated with the average uncertainties. If the two uncertain numbers are directly (positively) interrelated, the average revenue is greater than the revenue associated with the average uncertainties.
- Taking credit for chance occurrences: We all like to take credit for our hard work, but some successes may be due to dumb luck. (…) [This is about null hypothesis (statistical) testing. See Statistical hypothesis testing (Wikipedia) and Chapter 20 online supplement]
- Believing there are only eleven deadly sins: The twelfth of the Seven Deadly Sins is being lulled into a sense of complacency, thinking you now know all of the insidious effects of averages.
Sam Savage did a great job: The Flaw of Averages is written in an amusing and down-to-earth style, and is a worthy read. If you don’t like mathematics, rest assured: no mathematical background or skill are required to enjoy it.
Further reading on statistics:
- 2017-12-16: How Not To Lie With Statistics (Kevin Gray)
- 2016-11-23: Statistical Mistakes and How to Avoid Them (Adrian Sampson / Cornell University)
EOF