How to get better at predicting the future
A few months back, a colleague told me that they wanted to get better at forecasting and wondered how to go about it. I promised them that I would write about this topic, and so after a few months of thinking about this and dilly-dallying, here we go. Ironically, this post took a while to write, underscoring my ineptitude at forecasting this kind of stuff (aka planning fallacy).
Forecasting
Forecasting is the act of predicting the future. It is needed in almost all walks of life. Businesses need to forecast their sales. Countries need a handle on what the future will look like to plan and shape that future. We also need a good sense of how the world is evolving. Without that, our lives will be completely reactive, not proactive. In fact, pretty much all the decisions we make are rooted in the predictions we make about the outcome.
And being bad at forecasting has disastrous consequences. IBM's president in 1943, Thomas Watson, famously said: "I think there is a world market for maybe five computers." They are still paying the price for this one. Many people, including Bill Gates, thought the future information revolution would arrive in the form of smart TVs connected via "information highways." Even as the internet revolution was unfolding, they couldn't see the present, let alone the future, and they lost a valuable market opportunity to Netscape. And unfortunately, Bill Gates's successor, Steve Ballmer, thought iPhones would amount to nothing.
Our brains are prediction machines
Our brains are prediction machines firing neurons every moment for everything you do, be it monumental or minuscule.
For example, even as you walk, your brain works out where and when your foot should land on the ground. This happens as you think deeply, or talk to someone, or listen to something. The moment the prediction fails, you realize (or rather your brain realizes) that something went wrong, only for you to know you are falling. That's the essence of surprise: when reality doesn't match what you predicted (unconsciously).
The Three-Body Problem
If the act of predicting is such a keystone aspect of our brain, and forecasting is all about predicting, then why are we not good forecasters?
It’s Difficult to Make Predictions, Especially About the Future
- Danish proverb
Let's see why forecasting is so hard. Suppose the universe consists of absolutely nothing but a star and a planet; you will be able to forecast their positions well into the future. In Black Swan, Nassim Taleb explains this further:
But add a third body, say a comet, ever so small, between the planets. Initially the third body will cause no drift, no impact; later, with time, its effects on the two other bodies may become explosive. Small differences in where this tiny body is located will eventually dictate the future of the behemoth planets.
Taleb also gives another example consisting of predicting the evolution of Billiards ball table (he adapted this from Michael Berry’s paper “Semiclassical mechanics of regular and irregular motion”):
If you know a set of basic parameters concerning the ball at rest, can compute the resistance of the table (quite elementary), and can gauge the strength of the impact, then it is rather easy to predict what would happen at the first hit. The second impact becomes more complicated, but possible; and more precision is called for. The problem is that to correctly computer the ninth impact, you need to take account the gravitational pull of someone standing next to the table (modestly, Berry’s computations use a weight of less than 150 pounds). And to compute the fifty-sixth impact, every single elementary particle in the universe needs to be present in your assumptions! An electron at the edge of the universe, separated from us by 10 billion light-years, must figure in the calculations, since it exerts a meaningful effect on the outcome.
“I am good at forecasting!”
We fall for those who appear to be good at predicting the future. But we should be cautious when someone claims to have done well predicting the future. Very likely, they just got lucky in the past and/or cherry-picking (knowingly or unknowingly).
I want to take a brief interlude to mention an interesting con technique. Suppose there are a series of matches with ten matches before the finals. A con man can adopt the following strategy: at the start of the first match between team A and team B, they can send emails to 1024 people, of which 512 get an email that team A will win, and the other 512 get an email that team B will prevail. If B wins, the con man can focus on the second group and repeat the exercise. At the end of 10 matches, this one person would be amazed to see the perfect precision with which all the matches were predicted. The con man could then name a price, and the person might be happy to oblige.
Not all Hope is Lost
But does this mean that forecasting is impossible, so we shouldn't even attempt to do it? Of course not.
How good your predictions are, depends on how far into the future you want to predict, how complicated the thing is that you are trying to predict, and your expertise.
So this gives us a few clues:
Try to avoid predicting too far into the future, or at least take such predictions that are presented to you with a grain of salt.
Understand the complexity of what you are trying to predict. Forecasting sales one year into the future may be much easier than forecasting how the world will look a year later.
If you want to get better at forecasting, you will have to develop expertise.
Let’s talk about expertise
Some people are better than others when it comes to forecasting. Warren Buffett and Charlie Munger come to mind. Elon Musk predicted the EV revolution and got started earlier (although a case can be made that he caused the revolution). In each of these cases, these are experts in their fields. Warren Buffett immersed himself in reading financial statements of hundreds of businesses to become a world-class expert in understanding businesses.
The key takeaway is that if you want to become a better forecaster, you have to develop expertise. And since you don't want to become a jack of all trades, you have to restrict yourselves to a specific field. Warren Buffett is famously known to stay only within his circle of competence (see my related article on curves of competence). And as such, he used to avoid investing in the tech sector.
Why expertise helps to predict
In the book Thinking Fast and Slow, Daniel Kahneman writes:
The psychologist Gary Klein tells the story of a team of firefighters that entered a house in which the kitchen was on fire. Soon after they started hosing down the kitchen, the commander heard himself shout, “Let's get out of here!” without realizing why. The floor collapsed almost immediately after the firefighters escaped. Only after the fact did the commander realize that the fire had been unusually quiet and that his ears had been unusually hot. Together, these impressions prompted what he called a “sixth sense of danger.” He had no idea what was wrong, but he knew something was wrong. It turned out that the heart of the fire had not been in the kitchen but in the basement beneath where the men had stood.
How was the firefighter able to predict the impending doom? Some people call this intuition. Kahneman calls this system 1 thinking (a form of fast, automatic thinking as opposed to the slow, deliberate system 2 thinking). But if you or me were in that situation, our system 1 thinking wouldn’t have spared our lives. The reason the firefighter was able to use system 1 thinking was because of his expertise. Years of experience and practice allowed the firefighter to develop a rich set of mental representations that allowed him to pick subtle clues and process them very fast.
Expertise, in other words, helps us build up intuition to solve problems and think better about the future (in the specific domain).
Chess is a nice game to understand this better, since Chess is all about predicting what will happen in the future, and using that to guide your action. By definition, expert chess players are better at predicting the game as compared to others. In Deliberate Practice, I wrote:
Chess players parse the board much differently than the rest of us. When most of us look at the board, we see the individual pieces and the grid and get a rough idea about what the pieces are doing. An absolute new beginner who hasn't seen a chess board before will not even be able to grasp as much information as you do.
Expert chess players, on the other hand, absorb in a lot more information — they see which pieces are weakly positioned vs. which ones are strongly positioned. They probably almost see a story that is playing out on the board. If you show a valid board, and then ask them to recall, they are able to recall very well. I, on the other hand, will struggle to recall a few pieces here and there.
Expert chess players and expert firefighters have the same thing in common: rich mental models/representations that allow them to process input quite efficiently and use them to predict the future.
So it is crucial to accumulate rich mental models about the domain you want to become a good predictor of. There are, of course, other general thinking tools (which I call lenses) that help us think and reason about the world and the future better. For example, learning about confirmation biases helps us avoid going in the wrong direction when we reasoning about something. Learning about planning fallacy allows us to set better expectations about timelines. And systems thinking gives us a better understanding of how the world works. All of these go hand in hand to help us become better at predicting the future.