Statistical risk models are designed to forecast how much a portfolio’s value may fluctuate over some trading horizon. To be useful, the forecast should apply to the time over which trading decisions are made; it should be the forecasts on today’s positions that are relevant, not those on hypothetical future trades.
For actively managed portfolios, the risk horizon is typically one day or one week. Importantly, financial crises do not transpire over these horizons. The models do not pretend to predict crises, then, but rather to indicate the potential for large short-term market moves. And forecasts over these short horizons can be validated: we can objectively differentiate a good from a bad model.
So do the models succeed at their goal? Recent anecdotes suggest not. Large banks disclose both their risk forecasts and their real profit and loss. The risk forecasts typically take the form of value-at-risk, or VaR – that is, the worst-case loss that is expected, for example, with 99 per cent confidence. About one day in 100, subject to statistical fluctuations, the bank should experience a VaR excession – a day where the loss is greater than the VaR forecast.
The third quarter of 2007 (62 trading days) was ripe with disclosures of embarrassingly many excessions: as many as 16 for some banks. For good reason, there has been plenty of criticism of Sir Dennis's reports.
But a keen observer of VaR disclosures would have questioned the models well before the current crisis. It was not unusual for banks to disclose years with zero excessions (on average we would expect two or three) and one bank went nine years without one. While the flurry of excessions in 2007 raised plenty of doubts, the lack of excessions in previous years did not, even though under a good model, nine years without an excession is 100 times less likely than a fiscal quarter with nine or more. Statistical evidence to challenge the forecasts existed but, in rising markets, it was easy to mumble about conservatism and accept demonstrably bad forecasts as good risk disclosure.
So the supposed beneficiaries of risk disclosure have been guilty of apathy, but how could the banks themselves produce such poor forecasts? One explanation is that some favoured simplicity and stability over performance. Twenty-five years of financial research have established two facts: first, risk changes, with market volatility greater in some periods than others; and second, these changes can be forecast. Methods that ignore these facts produce inferior forecasts.