Quantitative methods and judgment Competitors or complements? Allan Timmermann
January 12, 2018
Quantitative methods: Strengths
• Flexibility: – Machine learning methods can be used to analyze highly complex and high-dimensional problems • Thousands of predictors, nonlinear terms
– Good progress has been made on controlling “overfitting”
• Avoids subjective biases of human forecasting • Efficiency: Given an objective function, quantitative methods will provide the best fit
Quantitative methods: Limitations
• Quant methods extrapolate past data patterns into the future – Limited library of historical events (LTCM default, Global Financial Crisis, etc.) • The next crisis will almost certainly be different from previous ones
• What if the future isn’t like the past? Breaks – Are US stocks overvalued now? CAPE suggests so, but interest rates are uniquely low
• How much data do we really have at the business cycle frequency?
Forecasting European inflation using information in the crosssection (Smith and Timmermann, 2017)
Quantitative methods: Limitations
• The “signal-to-noise” ratio is very low in many financial forecasting problems – far less than 1%, so 99% is left unexplained
• Model parameters have to be estimated – Parameter estimation – Model uncertainty
• Models are approximations – “All models are wrong. Some are useful.”
Quantitative methods: Limitations
• Best quant methods use ensemble techniques which combine forecasts from many weak signals and from different models – Many signals are weak – no golden needle in the haystack – Obviously strong signals are already reflected in market prices
Quantitative methods (Limitations)
• Are quantitative methods “too flexible” – Multiple hypothesis testing problem: If many models are tested, some will appear “by luck” to be amazing • Fooled by randomness
– This is not an issue if we can generate large new data sets • Test and fit models on different data sets • In asset allocation analysis we often don’t have the luxury of new data
• Arms race between (over-) fitting and controlling the overfit (familywise error rate)
Controlling for multiple models: White’s Reality Check
Judgmental methods
• Can in principle incorporate forward-looking information and heuristics – Predicting the effect of the Trump tax cut on US GDP growth: no precedent
• Well-known biases • No individual forecaster or judgmental method consistently beats a simple average of judgements • Combination methods work best – Wisdom of the crowd
Combining quantitative and judgmental methods
• For many macroeconomic variables, a simple equalweighted average of survey forecasts beats sophisticated econometric model forecasts – One exception is stock returns
• No method completely dominates the other – Combine the methods using state- and timedependent combination weights – Include judgmental forecasts as an extra signal in a quantitative model – Use quantitative methods to de-bias survey forecasts
Combining quantitative and judgmental methods (cont.)
• Bayesian methods offer a way to combine subjective and quantitative forecasts – Ideally we know their joint probability distribution – Prior on the relative merits of the methods