Data is the new oil, or so they say. Initially limited to the technology industry, most sectors have transitioned to leveraging data and analytics over the last decade to tackle some of the most challenging problems. Being a Manager or an Executive today requires a good understanding of metrics or relying on experts to help guide your strategic decisions. To that end, here are a few pitfalls around Data Analysis that you should be aware of. Keeping an eye out for these will ensure you ask the right questions and too much time isn’t wasted digging in the wrong direction.
1. Survivorship Bias
If you know this picture, you might already know the fascinating story behind it. During World War II, the US military concluded that the most-hit areas of their bomber planes should be given additional armor. This seemed logical given the many returning planes that seemed to have holes in these regions. Abraham Wald, statistician from Columbia University, recommended the opposite approach factoring in survivorship bias. He pointed out that armor should be added to the areas without holes as those were the areas that resulted in planes not returning due to explosions. The returning planes showed the areas that can still take a hit and manage to return. This is one of the first discoveries that led to the field of Operations Research we see today.
Lesson: When you look at your data, always keep the survivorship bias in mind and think about why you’re seeing this data. Without this awareness, you could come to the wrong conclusions.
Look at this series of coin tosses and think about what you believe would be the next toss, Heads or Tails.
H H H T H H ?
The intuitive answer often is T. Another way to pose the question, which out of these two series is more likely:
H H H H H H or H H T H T T
Our brain will generally pick the option to the right as it’s more random. In both cases, you’d be wrong. The probability is the same for both alternatives, when you do the math. For the first question, H and T are always 50% as likely. For the second question, both series are equally as likely as each individual toss is 50% as likely. Our brain believes in regression to the mean, so when a lot more Heads have come up, you think a Tails is due soon. But that’s only true for the Law of Large Numbers, that there tends to be equal number of Tails and Heads when the sample sign gets larger. It is not true for an individual toss. For small samples, each individual toss continues to have a 50% probability.
Lesson: Don’t let past events influence your thinking about future events when the odds are actually not changing. This fallacy, also called Monte Carlo fallacy, is often the reason behind wrong choices made at casinos.
Another example from a war, this time the Vietnam war. This fallacy is named after Robert McNamara, the US Secretary of Defense during the war. He believed that the best metric to determine success during the war was the enemy body count. This approach ignored other measures like public mood, territory gained, and other strategic objectives. This was a failed strategy as they were blindsided by these other unmeasured factors.
Lesson: This is a common problem in data analysis where the easy-to-measure quantitative metric often gets the focus. Other metrics, believed to be too hard, are assumed to be irrelevant. As you’d expect, you miss the picture with such tunnel vision. Always be mindful of the coverage provided by your metrics and look for ways to back it up with different variables.
4. Cobra Effect
The Cobra Effect occurs when your solution unintentionally makes the problem worse. The effect gets its name from the British rule of India. The British government wanted to reduce the number of cobras in Delhi, so they announced a bounty for every dead cobra. With time, some entrepreneurs started to breed cobras to get that bounty. When the government realized this, the program was scrapped resulting in all the worthless snakes being set free. Thus, the unintended consequence was a substantial increase in the cobra population.
Lesson: It’s important to note how incentives are a very powerful driver of change. If you don’t think through the incentives set for your teams, there could be adverse impacts. This is also true for KPIs and metrics tracked within each organization so carefully consider and monitor them.
I hope these pitfalls were as informative for you as they were for me. Are there any such fallacies that you like? Leave examples in the comments below!
Kommentare