There may not be enough time for banks to gather the data needed to comply with the Basel II operational risk requirements, says Patrick McConnell
In October 2006, the impressively named Accord Implementation Group’s Operational Risk Subgroup of the Basel Committee, thankfully abbreviated to AIGOR, released the results of a study on the progress being made by the world’s largest banks towards developing so-called Advanced Measurement Approaches (AMA) for Basel II. This ‘range of practices’ study is part of the housekeeping being undertaken by regulators in advance of the implementation of Basel II, barely 12 months away in most jurisdictions. The stated goal of this and related benchmarking studies is to “establish high standards for what constitutes acceptable practice ... and to promote consistency across jurisdictions”– that is, to create a level playing field.
While the AIGOR study does report some impressive progress in adopting consistent approaches to the governance of operational risk, such as the creation of independent operational risk management functions, there is much less consistency in one of the key components of Basel II – the calculation of operational risk capital under so-called Pillar 1.
In two major areas – the collection of operational risk data and the development of models to quantify operational risk – there appears to be little consistency. Banks have bolted and have disappeared over the regulatory horizon. Given the relatively short time remaining to implement Basel II, do regulators have sufficient time to satisfy their “level playing field” objectives? Being realistic, since banks and regulators have been working on these thorny issues for several years, without achieving a consensus, it is unlikely that unanimity will break out in the next year.
Why is there so much disparity in approaches to quantifying operational risk? Before looking at some of the more intractable problems, it is worth recapping the main Basel II requirements.
To qualify to use an AMA for calculating Operational Risk Capital (ORC) a bank must:
•“Be able to demonstrate that its approach captures potentially severe tail loss events. ... with a one-year holding period and a 99.9th percentile confidence interval.
•Track internal loss data ... based on a minimum five-year observation period.
•Use relevant external data, especially when there is reason to believe that the bank is exposed to infrequent, yet potentially severe, losses.
•Use scenario analysis of expert opinion in conjunction with external data to evaluate its exposure to high-severity events.
•Capture key business environment and internal control factors that can change its operational risk profile”.
As both ‘scenario analysis’ and the identification of ‘business environment and internal control factors’are, in many cases, specific to the business mix and market posture of an individual bank, we should not be surprised that there is some variation in approach by different banks (although it is also true that many banks are subject to the same potentially severe scenarios and business environment factors). On the other hand, there is less reason to believe that the methods used to capture and model operational loss data should vary widely across banks, although, as with the value at risk (VAR) methodologies used to quantify market risk, the resulting capital numbers may vary significantly.
Comparison with the VAR approaches universally employed to calculate market risk capital (MRC) is instructive. At first glance, the problems faced are similar. To calculate MRC, banks must collect internal data on open market positions; incorporate ‘relevant’ external data, that is, market prices; then apply validated mathematical models to quantify the capital required to cover the positions. On closer examination, however, the bar for quantifying operational risk capital has been set much higher, in particular, a 99.9 per cent confidence level as against 95 per cent for market risk. Statistical theory (and common sense) would suggest that appreciably more data would be required to achieve a higher confidence interval of 99.9 per cent. But, in practice, there is a distinct lack of data available in most banks on potentially severe tail operational loss events (that is, really horrible events don’t happen that often).
The AIGOR study admits there is a paucity of internal loss data “relative to what is required to reasonably assess a bank’s operational risk profile”, and, in order to meet the illusive 99.9 per cent confidence level for operational risk, banks are being forced to rely on ‘relevant external data’.
While there are several sources of external loss data available from third-party suppliers (known as ‘public’data) or from industry consortia (so-called ‘pooled’ data) the difficulty, of course, is in determining the ‘relevance’of any particular loss to a particular bank. For example, how relevant is the US$1.25 ($1.6) billion loss that occurred at Barings to a local bank and, if relevant, how much of the loss should be estimated as likely to occur – $1 million, $10 million, $100 million, even $2 billion? The answer, of course, is subjective and the AIGOR study reports that no bank has developed a workable methodology for ‘scaling’ external data.
While it can be argued that, albeit after an enormous amount of work, it might be possible to answer such questions to a reasonable level of confidence, the example highlights a much more difficult issue – so-called ‘distributional assumptions’. In order to use any such estimate in statistical modelling, one has to make the assumption that the internal and estimated external losses are drawn from the same distribution. In technical terms, they are independently and identically distributed. In particular, this is a necessary requirement for employing the statistical tools most often recommended for operational risk analysis Extreme Value Theory (EVT). While EVT is based on solid theory and is well established in actuarial situations, its practical use does require copious quantities of ‘independently and identically distributed’ (IID) data, to develop the necessary underlying loss distributions.
It has been shown that, by combining data from many banks, EVT may indeed be useful at the industry level but the paucity of data makes its use problematical in the context of an individual bank. This problem of lack of data is made even worse by the fact that, in Basel II, losses must be classified by predefined ‘business lines’and ‘event types’ in the form of an 8 times 7 matrix (8x7) – that is, 56 cells. Intuitively, and in practice, losses from, for example, credit card fraud in a retail business line are very different to, and much lower than, losses resulting from trader fraud in a trading and sales business line. Such losses are obviously not IID – they’re not from the same distribution. As a result, in most banks, many of the 56 cells will contain insufficient data points to conduct meaningful statistical analysis.
As the Basel II 8x7 classification is somewhat arbitrary it can be argued that, even within a cell, not all of the data elements will necessarily be drawn from the same distribution. For example, consider the wide variation of losses from different causes that would be grouped together under the generic event type, “Damage to Physical Assets”– for example, fire, earthquake, vandalism, terrorism and so on. In practice, banks have reacted to these difficulties with a wide variety of distributional assumptions and modelling techniques. But AIGOR observes that most banks have not “undertaken sufficient statistical or other analysis to justify their assumptions”, they merely justify their choice of distributional assumptions based upon the (poor level of) data available.
It should be noted, however, that this so-called ‘granularity’ problem is not the only major disparity in approach found by AIGOR; serious and unresolved issues include:
(a) Correlation: banks are allowed to use ‘internally determined correlations’ across business lines and/or event type to reduce operational capital provided that the correlation assumptions take into account “uncertainty ... particularly in terms of stress”. Unfortunately, there is often insufficient data on which to base high-confidence correlations.
(b) Allocation: banks are required to develop specific objective criteria for allocating losses from a single central event to multiple business lines. Obviously, the assumptions made in these cases will be somewhat subjective and hence difficult to subject to meaningful statistical analysis.
(c) Data Collection Thresholds: for efficiency purposes, banks are permitted to define a ‘de minimis’ threshold below which loss data is not collected. The choice of any threshold value will obviously have an impact on the number of data points available for analysis.
(d) Expected Losses: whereas most industry attention has been focused on unexpected losses (UL), that is, those potentially severe ‘tail loss events’, regulators require that capital be calculated as the sum of UL and expected losses (EL). This need to calculate EL has implications for the amount of data that must be collected, in particular any threshold assumptions.
(e) Loss Evaluation: Basel II does not specifically identify how a particular loss should be ‘evaluated’. For instance, it doesn’t say whether to use ‘book’ or ‘market’ value for an asset damaged in a fire. As AIGOR notes, the differences in value (and resulting capital estimates) may be substantial.
Lack of agreed methods to address these and several other very difficult problems is recognised by regulators as having serious implications for operational risk management. Different distributional, correlation and allocation assumptions could result in very different capital outcomes. AIGOR concedes that banks with similar risk profiles could “hold different levels of capital if they rely on substantially different modelling and assumptions”– very far from a level playing field. AIGOR also warns that, within a firm, the volatility of capital allocated to business lines, resulting from changes in underlying assumptions, could undermine the “internal credibility”of risk calculation and allocation methodologies.
The pertinent question that must be asked is will the situation improve before, or soon after, the implementation of Basel II in January 2008? Since the core problem is one of lack of data to achieve the required level of regulatory confidence, it is highly unlikely that sufficient data will become available in the interim, especially since improved risk management should, in theory, reduce the number of operational loss events and hence analysable data points.
Regulators appear to be in denial. While noting that a wide range of practices have emerged, as a result of the significant ‘degree of flexibility’ granted by the Basel II framework, they appear to be hoping that broadly ‘acceptable’ practices will emerge in the near future. Regulators do not concede that the inflexibility they have written into Basel II may make the emergence of industry consensus extremely difficult if not impossible.
If there is insufficient operational loss data to model at the 99.9 per cent confidence level, a sensible option would be to give banks the flexibility to choose the confidence level they are happy with and can justify to regulators. This does not mean that another arbitrary level, such as 95 per cent, should be mandated but that banks should be able to select a level of confidence and then be rewarded, by capital reduction, as the level of confidence is increased, eventually aiming at 99.9 per cent. Such an approach would allow banks to test modelling techniques that may be more appropriate for the data that is available to them. It should be noted that such a change should not endanger the overall levels of capital required to protect the financial system, since upper limits are already in place in Basel II (as the so-called Basic Indicator Approach), reductions would only be granted with demonstrable improvements in risk measurement and management.
In summary, regulators have discovered what industry practitioners already knew – there are too few operational losses on which to base the high-confidence statistical inference needed to support an AMA. This is not at all bad news – it merely shows that operational losses may not be as large as was originally thought. While operational risk remains an issue that deserves proactive management through capital and governance standards, its management is more of an art than a science and banks should be rewarded as they improve their skills in measuring and managing risk. As always with Basel II, time is of the essence if banks are not to waste even more time and money chasing what may be an unattainable goal.
Patrick McConnell is a partner in consultancy Risk Trading Technology