Confessions of an Economic Forecaster

Economists’ forecasts are notoriously inaccurate. This fallibility is hardly a great surprise given that we live in an irretrievably uncertain world. So why do we keep doing it? What can we learn? Here are my eighteen secrets and myths about the economic forecasters’ art.

1. Worse than fallible

It’s not just that economic forecasts are prone to be wrong. It’s that they tend to let you down when you most need them. Paul Samuelson once said the stock market has “predicted nine out of the last five recessions”. By comparison, economic forecasters rarely predict any! They are typically caught out by shocks.

2. But decisions need forecasts

Economists are well aware of the fallibility of forecasts. But they have no option but to keep making them. People can’t live without them – all decisions are based on a view of the future. People also keep asking for them because they can provide a (spuriously) precise way of justifying decisions made for other reasons. And if things go wrong, they can blame the ‘expert’!

3. A deceptive air of certainty

In an uncertain world, forecasts, and the story behind them, provide us with comfort. But if we place too much faith in them they may lead us astray. Such overconfidence gives us an illusion of control. We can see this in investment projects that are often late or over-budget. Similarly, official forecasts, for example from governments or central banks, are routinely over-optimistic. They assume policy success, sometimes because they are designed to change people’s expectations and behaviour.

Forecasters often ignore the sage advice that “it’s better to be vaguely right rather than exactly wrong”. Having constructed models to predict the future, forecasters tend to forget that they are only approximations.

4. Charlatans rule

Confident dogmatic forecasters are provocative and often grab the headlines. Unfortunately, as Philip Tetlock has ably demonstrated, their strong but simple narratives are often way off the mark. The devil is often in the detail, so we should be wary of forecasts based on a single factor. It’s also a good discipline to try to attach a probability to your forecasts. Admitting uncertainty and being open about your degree of confidence in your prediction is not a sign of weakness, but helpful realism.

5. Try to be objective

We’re all biased, and our forecasts reflect our pre-conceptions. Forecasters need to acknowledge this, and aim for objective assessments. It is easy to pay too much attention to the most recent and most shocking news. Yet, the latest news may be misleading, and forecasters often gloss over the fact that newly published statistics can be revised dramatically. They have a natural human tendency to indulge in elegant, after-the-fact rationalisations. Hindsight is indeed a wonderful thing. But it’s all too tempting to succumb to selective memory, cherishing your successes while forgetting your mistakes.

6. The more the merrier

By making more forecasts, you have more chances to ‘win’ by getting something right! But more seriously, it also gives you more chances to learn. Rather than sticking with a forecast that is off track, it is important to revise your views in the light of new information. As John Maynard Keynes put it “when the facts change, I change my mind. What do you do, sir?”.

7. Unlike traders, forecasters do get marks for style

Financial market traders and investors are judged by the profits that they make. As a result, they are happy to get it right for the wrong reasons. But relying on luck is dangerous. It has a nasty habit of running out on you in the end.

By contrast, and contrary to popular opinion, forecasting is not just, or maybe even mainly, about accuracy. Aside from providing provocative or comforting stories, forecasts are about survival rather than getting it right. Repeatedly making small errors may not kill you, particularly if you learn from them or if they help you to avoid a catastrophic error. And accuracy is often relative, just as when faced with a charging bull, your survival depends not on outrunning the bull, but your neighbour.

8. It’s more about ‘why’ than ‘what’

Meaningful forecasts come with useful explanations. They should provide the users with a framework of thought and wisdom, rather than just knowledge and information. A common error is to assume that because two things are moving together that the two are related, let alone that one is causing the other: as scientists say, correlation is not causation. Wisdom comes from a deeper understanding of why things happen.

9. Past isn’t always prologue

The models used by forecasters are based on past causal relationships. People’s spending, for example, will typically closely reflect what is happening to their incomes. Many such relationships are reliable, with variables showing steady trends or cyclical patterns. But sadly, history doesn’t always repeat itself, or even rhyme. Things don’t always revert to their previous trends, and so what is considered ‘normal’ can be a moving target. Models live in the past, and die in the future.

Former British Prime Minister Harold MacMillan once said that his biggest fear was “Events dear boy, events”. So it is for forecasts. Unprecedented shocks have a habit of wrecking them. Economists make a crucial distinction between risk and uncertainty. Risk is where you know the odds, like playing roulette. Uncertainty is where you don’t. Worse still, you may have no idea that the game is about to change entirely; Former US Defense Secretary Donald Rumsfeld referred to this as the ‘unknown unknowns’. This is why forecasting is more of an art than a science.

10. Some things are more certain than others

The further out you look, the more uncertain things are, making forecasts even less reliable. This is why economic pundits on TV prefer to look smart by focusing on the short term. In fact, some forecasts of monthly indicators, which are released with a month or so’s delay, are more like ‘backcasts’, because anecdotal information helps to ‘predict’ last month’s outcome.

Apart from the time dimension, some variables are more volatile, or prone to extreme outcomes than others. For example, commodity prices and share prices are far more volatile than consumer prices. The importance of this is reinforced by the fact that the starting point for the forecast is often critical to the outcome. Economies, like the weather or biological systems, may be acutely sensitive to initial conditions. The phenomenon of one thing leading to another, or ‘path dependency’, makes forecasting especially difficult. In the metaphor of chaos theory, a butterfly flapping its wings may ultimately lead to a tornado far away.

11. Some things matter more than others

In wrestling with the complexity of the economy, the forecaster’s job is helped by the fact that some things matter more than others, and therefore deserve more attention. Sometimes this is obvious. Thus businesses focus on variables critical to the thresholds key to their decisions to invest, lend, hire, buy or sell. For example, the prospective price of oil may a critical determinant of the profitability of an oil rig.

However, sometimes we are prone to miss crucial risks to a forecast. We may fail to recognise our implicit assumptions, often because we mistakenly take things for granted. The global financial crisis that broke in 2008 at least partly stemmed from a widespread complacency that US house prices don’t fall.

12. Stories can help, but also hinder

One way of dealing with the myriad of possible futures is to construct scenarios. Rather than ‘betting the ranch’ on a single forecast, scenarios are stories constructed around the key drivers of the outlook in terms of their probability and impact. The stories give the users benchmarks for answering the question ”what kind of a world are we in?”. This can be a helpful way of thinking through potentially disruptive events beforehand, because people are liable to make bad decisions under stress. Contingency planning can allow ‘grace under fire’.

But in thinking about alternative scenarios, it’s important not to be blinded about other possibilities – the unknown unknowns. Multiple stories can confuse, and you have to make a choice in the end. The goal is to pick a ‘no regrets’ strategy that’s robust in variety of scenarios – but that’s easier said than done.

13. Don’t just forecast what you can measure

Scenarios are one way tackling important issues that don’t lend themselves to measurement. Economists often tend to ignore non-economic factors that are hard to put into their models. In so doing they often miss the ‘elephants in the room’. In particular, politics and institutions matter. In the long run they can be critical in determining whether countries succeed or fail. And even in the short term, the outcome of a divisive election, as we have been observing recently in the US, may be far more important to the outlook than a forensic analysis of the latest economic data.

14. More data isn’t always better

Data is vital to test theories in the real world. But that doesn’t mean that more data is always better. Data doesn’t speak for itself. Economic forecasters, embarrassed again by the global financial crisis, are in danger of being overshadowed by the new breed of ‘data scientists’. The emergence of Big Data and machine learning is opening up great new forecasting opportunities. In effect, massive computational capacity is being used to spot patterns through automated trial and error.

However, powerful new weapons are dangerous in the wrong hands. ‘Predictive analytics’ is in reality a tool for micro-economic forecasting. It’s not a ‘silver bullet’. More data and more variables gives more chances of correlations. The problem is that the proportion that are spurious and dangerously misleading rises even faster. As Nassim Taleb puts it, ‘Big Data’ brings ‘cherry-picking to an industrial level’. It is crucial to put the data into context, check its quality, understand the questions we are trying to answer, and to separate the signal from the noise. Identifying ‘why’ things are happening is vital for developing better forecasts.

But here the distinction between risk and uncertainty is crucial: more data in an uncertain world may create the impression that we are dealing with calculable risks. In such situations, simple rules of thumb, or ‘heuristics’, may outperform complex forecasting techniques. And they are clearly quicker and cheaper!

15. Magnitudes aren’t always necessary

For some purposes, we don’t need precise, quantified forecasts. Sometimes the direction is enough. That said, it also helps to get the timing right too! This is particularly true of questions that have binary outcomes. Wars and elections are won or lost. We are far more concerned with who wins an election than their margin of victory.

Such events are not easy to plug into economic models, and require different methods, with an emphasis on probabilities. Scenarios based on these contingent forecasts can then be produced. In some cases, some step changes, or what modellers call ‘structural breaks’, can be anticipated, such as preannounced policy shifts or regulatory changes.

16. Not all relationships are linear

Non-linear relationships create major problems for forecasters. They are difficult to model, and harder still to forecast. This is particularly troubling now because many of the emerging new digital technologies exhibit, or have the potential for, exponential, explosive growth. Minor initial errors in your forecasting model quickly lead to huge ones.

In this environment, Nassim Taleb, in his book ‘Antifragile’, advocates that we give up on forecasting. Instead, we should focus on non-predictive forms of decision-making, looking for choices that are not merely robust in the face of unexpected extreme changes, but actively benefit from the them. Sadly, this is easier said than done, and many businesses appear to be paralysed by uncertainty rather than energised by techno-phoria. That said, technological progress is not entirely unpredictable. Moore’s Law, which suggests that computer processing power doubles every two years, has proven to be remarkably durable.

17. Beware of market madness

Surveys of forecasts typically show that only few mavericks stick their necks out. Most forecasters prefer to stick close to the consensus of their peers. They seek safety in numbers, figuring that it is better to get it wrong in a crowd. This is just as well, because when it comes to the financial markets, they are typically wrong, often by a large margin.

Financial markets are not like other markets, because their prices are themselves forecasts. Financial asset prices reflect expected future returns and risk. According to the ‘efficient market hypothesis’, you cannot beat the wisdom of the crowd, because its collective intelligence already incorporates all available information. While it is certainly true that markets quickly incorporate incoming information, and it is hard, if not impossible to outperform them, this hypothesis does not stand up to scrutiny. And all too often, the wisdom of crowds is revealed to be madness, with extreme, unexpected outcomes. Investor beliefs about the future are loosely held, and prone to shifts from optimism to pessimism. This leads to booms and busts in market prices that leave consensus-minded forecasters trailing in their wake.

18. The ‘unusually uncertain’ cliché

Forecasters are prone to downplay the potential accuracy of their forecasts by saying ‘times are unusually uncertain’. It’s a cliché that is surprisingly rarely challenged. All eras have their own uncertainties. The current era is surely beset by potential volatility from digital technologies, political and social turmoil, unconventional monetary policies, financial fragilities, and regulatory change. But hindsight makes it easy to forget previous eras had their own uncertainties at the time – just think of the existential threats of the nuclear arms race during the Cold War. By deterring risk taking, the ‘unusual uncertainty’ cliché often equates to a pessimistic outlook. Yet we should not forget that the darkest hour comes before the dawn. Positive surprises do happen.

A version of this post appeared in the Q3 2016 edition of ING World. 

Advertisement