Forecast error is the elephant in the room
We’re often asked: “How wrong is your forecast?” or “What’s your forecast error?”
Forecasting is our job, so we don’t avoid the question. Error matters to buyers, planners, developers and investors. If error is high, our companies, portfolios and clients lose credibility and – in many cases – lots of money. Of course, no methodology is perfect, meaning that we will never have 0% error. But having an open discussion about forecast error is critical to reducing our procurement risk, revenue risk and portfolio risk.
In general, disclosing error rates is the third rail for forecasters. If we raise the topic of forecast error in a room of economists or industry forecasters, everyone will either go quiet or stray off into excuses. The solution starts with recognizing that there is a problem. Here’s the scary truth around error:
Every company we speak with tells us their procurement forecast error is – at minimum – 20-30%.
Industry forecasts are no different – crude, precious metals, ag and others have error rates between 15% and 30%.
These are staggering numbers coming from companies and information services firms that have been developing cost and price models for decades. Eventually, you’d think these folks would either surrender or find a better solution.
At Complete Intelligence, we break down market and pricing activity into something more symbolic (billions of algorithms updated and revised every month). This takes discipline, so much so that we have removed human intervention from the entire forecast process. This is the “machine” part of “machine learning”.
Without turning the process over to the discipline of learning machines, forecasts always fall back on “gut feeling”. “Gut feeling” is the enemy of precision, but if planners, forecasters and investors are honest, it is the status quo approach to corporate and market forecasting. And that should terrify portfolio managers, CPOs, CROs, CFOs, CEOs, boards of directors and others who are held accountable for company forecasts.
So, in the spirit of accountability for forecast error, below are Complete Intelligence error rates for the 800 assets (currencies, commodities and equity indices) that we forecast each month:
We calculate our error by using mean absolute percent error (MAPE) for each month over the past year. Why is MAPE important? Because it doesn’t allow us to game the forecasts. For example, if we showed a copper forecast that had +20% error in May, then -18% error in June, a standard average error calculation would hide the volatility and state that we had 1% error. Using MAPE, our error would show 19% error. So, our approach to calculating error – MAPE – is transparent and rigorous.
Complete Intelligence checks our error every month and uses a very detailed process to rebalance our algorithms to adjust for it. We are the number one source of error checking across economic and market domains. Find out more about our methodology here.
So, let’s have an open discussion about error. Do your internal teams track their error? Do you require your data vendors and external forecasters to report their error? Are they hiding error with evasive techniques? Please make sure you understand this. If you’d like, we can help. Our Global Cognitive System™ is helping procurement, revenue and investment teams reduce their risk by 5-10x. We’re happy to show you how.