Securities firms which have been active in the derivatives markets have for a number of years used statistical models for the purpose of pricing options, valuing their option portfolios and managing their option risk. These models typically have been based on variants of certain standard option pricing formulae. The requirement for tools of this sort arises from the fact that option risk is non­linear with respect to the price of the underlying instrument. Depending on the strategy being employed, it is possible to suffer losses on an option portfolio under a whole range of different scenarios, including a scenario where the price of the underlying stock does not move at all. It is therefore essential for firms involved in this business to be able to identify and monitor their risk in an effective manner.

More recently, however, some firms have developed models which extend beyond their options activities, and incorporate the risks in the rest of their trading portfolio. These models which are referred to as 'whole portfolio' or 'value at risk' models integrate the properties of the option models into an overall analysis of the statistical characteristics of the whole trading portfolio. The models predict the probability of loss in the portfolio using observed historical relationships between the different instruments in the portfolio. For example, firms may assess the value at risk in their portfolio as being the maximum amount they are likely to lose if they had to hold the portfolio for a fixed period based on historical experience with a given level of confidence (normally 95% or 99%). In other words if the future is like the past, the amount estimated by the model to be at risk would be lost once in every twenty days or once in every hundred depending on the confidence level chosen², and then only if the firm was unable to take any action to mitigate its loss.
___________________
² These examples assume that the models are based on a one day holding period for the portfolio.

In addition to estimation of the value at risk as predicted by the models, firms will often complement these results by conducting their own 'stress tests' on their portfolios. Stress tests apply particular worst case assumptions to the portfolio to assess the effect of certain severe adverse market movements on the institution. This might include, for example, a sudden sharp increase in interest rates or a sharp fall in the equity markets. Stress tests therefore override some of the estimated relationships in the model and aim to identify what particular market event or combination of events would most severely impact an institution's potential losses if they were to occur. The range of possible stress tests is extremely wide, but by establishing a standard set of such tests, institutions are able to give some indication of the events to which their trading portfolios would be particularly vulnerable.

It is important to note however that value at risk, even if it is based on a statistical model, is to a significant extent a judgmental concept. The models are based on observed statistical relationships which are of varying levels of reliability. They are also heavily dependent on the assumptions which the model builders make about the relationships between different financial instruments, the observation periods over which the relationships are estimated. In this sense, there is as yet no basis for defining any one 'correct' value at risk model. While it is important for regulators to understand the overall characteristics and properties of the model that a firm is using, it is also very important to understand how the firm uses the outputs of its particular model and stress testing as an aid to effective risk management. Properly used, value at risk models form an increasingly important part of firms' overall risk control environment. It is in this context that they assume a particular significance for securities regulators, who recognise that models are an Important element of a good system of management controls. Consequently it will become increasingly Important in the future for securities regulators to be conversant with the nature of these models and the issues associated with their use by regulated firms.

The relationships between models and management controls

In assessing the use of value at risk models by securities firms, the regulator needs to understand how the model fits into the overall operational and control structure of the firm. A model, however well constructed theoretically, cannot on its own provide assurance about a firm's ability to control its risks. In particular, it is important to note that value at risk models primarily address market risk, which is only one of the risk factors which a securities firm faces. Other "'actors include credit risk, operational risk, liquidity risk and legal risk, all of which need to be managed in an effective manner.

As with any other system, the outputs of a model are dependent on the quality of the data which goes in to the model. Consequently, a model will only be able to accurately reflect the risk in a firm's trading portfolio if it accurately captures all the positions and valuations within that portfolio. This means that careful attention needs to be paid to the way in which a firm approaches its operational controls. In particular, it is important for regulators to understand whether the model is integrated into the firm's trading system, so that the same data which is used by the traders is also used by the model, and by the accounting and management information systems or whether data is input separately into different systems. If, as is often the case, data is input separately into different systems, special attention needs to be paid to t he quality of the reconciliation processes which exist to ensure that the data which is being used by the traders is consistent with the data which is used to run the models and to prepare reports to management on both the risk and the profitability of trading operations. It is also important for the regulator to be reassured that the information which management is receiving about the performance of their trading activities is independently checked and verified. This should include an independent assessment of the valuation of particular positions, including where appropriate a reperformance of the valuation of particular large or complex trades by the risk management or financial control department, as well as an independent check on both the prices and the volatilities which are held in the system. Moreover, the regulator will wish to be assured that senior management are able to understand the information which they receive and have in place appropriate procedures and structures to ensure that the information is acted upon in a timely manner to ensure that the risk is effectively managed on an ongoing basis.

2.3 Structure of the Firm

In this context, the regulator will wish to pay particular attention to the way in which the firm is structured and its overall management approach towards risk control. In particular, the regulator will want to understand how the firm sets its risk limits, and how these limits fit into the model structure. how they are monitored on an ongoing basis, and what action is taken in the event of breaches of limits. The trading limits or other broad policy constraints should be set by the firm's governing body (board of directors or its equivalent) in terms which are readily comprehensible to members of the governing body, and the limits structure should be reviewed regularly by the governing body in the light of information about the risk profile of the firm and its actual trading performance. A copy of the limits structure should be readily available for review by the regulator as should copies of the reports on the risk profile and trading performance which are prepared for the governing body.

It is also necessary for the regulator to understand how risk is managed and controlled within the firm. The regulators should ensure that they are familiar with the senior management responsible for assuming trading risk, and that they understand the reporting lines within the trading businesses. This is of particular importance where firms assume and manage risk on a functional basis across a number of operating entities.

In addition, the regulator will wish to understand and be satisfied about the adequacy of the reporting structure for the risk control functions. Staff responsible for reporting and control, including risk control, accounting and settlement staff, should have a reporting line to senior management which is independent of the trading function. The reporting lines which ensure the integrity of these functions should be properly documented and endorsed by the governing body. In addition, the operating procedures which ensure that the functions discharge their responsibilities effectively should be properly documented. Regulators will wish to familiarise themselves with these processes and procedures as well as establishing contact with senior management responsible for the control functions in order to satisfy themselves of the commitment of the firm to a proper control environment. They will also need to be reassured that the staff concerned, particularly those involved in reporting and risk control functions, have the appropriate skills and authority to understand and challenge the decisions of the trading staff.

It is only when the regulator is satisfied that the firm has proper policies and procedures in place in relation to its internal controls that the regulator will be in a position to place reliance on the output of value at risk models for regulatory purposes.³
______________________
³ This is supported by the experience of the Securities and Futures Authority in the UK which has had an approval programme in place for option pricing models for the past 5 years. Some 55% of their model approval process relates to the examination of internal controls.

2.4 Integrity of the model environment

In assessing the models which are being used by securities firms, regulators will wish to establish the basis on which the firm has derived its model, and the environment within which the system is operating. Moreover, firms' models are not static, but are subject to continuous change in the light of further experience and research. Regulators will therefore wish to understand the basis on which the firm has selected a particular model and to obtain a measure of reassurance about its properties. One approach which has been adopted by the Basle Committee is to construct a sample portfolio, and to test models against the sample portfolio. This allows the regulators to assess the range of results produced by different models and to identify those models which produce results which fall outside a predetermined range. This approach could also be adopted from time to time with firms which were already using approved models to monitor the effect changes in model parameters since the initial approval took place.

In addition, the regulator will wish to be assured as to the integrity of the operating environment within which the model is run. This includes ensuring that procedures are in place to control access to the system, both in the development and the production environment, so that no unauthorised changes can be made to key model parameters which could affect the output of the model. In addition, there should be an independent review of any new pricing algorithms which are added to the model. The firm should also undertake regular backtesting of their models to establish how well the model predicted actual events, particularly following any significant market movement, and to ensure that the experience is properly factored into the model for the future.

Regulators will also wish to lie satisfied that the assumptions underlying the model are realistic and err on the side of caution. Particular consideration needs to be given to the practice of 'marking to model' or 'marking to theoretical value' in the context of derivative positions. This is particularly relevant it a dealer has significant positions in instruments where the market for the derivative or the underlying instrument lacks liquidity or transparency. In such cases there should be procedures in place to determine the price at which a position could actually be liquidated in the market by reference to quotations received from other dealers. These may at times differ significantly from the theoretical value produced by the firm's own model. Similarly attention needs to be paid to the institutional factors in relation to particular instruments, such as restrictions on the holding of stocks by particular classes of investor or foreign exchange controls which may vitiate the assumption of immediate tradability in the security on which most models are based.

The regulator will also wish to ensure that the risk management processes and procedures are subject to periodic review by both internal and external auditors who are fully conversant with the risk management processes and that any recommendations arising from these audits are acted on promptly. Firm's should also have documented and tested disaster recovery procedures which will allow them to recover key data and manage their trading risk in a timely manner in the event of significant business disruption.

2.5 Assessing the effectiveness of models

Where a firm is using a model as an integral part of its risk management process, the regulator will wish to understand how the output of the model influences the firm's decision making process. In particular, the regulator may wish to discuss with the firm the way in which daily management reports are prepared, and how non model based judgmental factors are incorporated into the risk control process. The regulator will also wish to know whether the risk management function has the authority to instruct traders to reduce their overall level of risk, and if not where such authority lies in the firm. Moreover in assessing the effectiveness of a firm's use of an internal model, particular attention should be paid to a comparison of observed daily trading results⁴ against the model output. The firm's trading results should fall within the bounds of the value at risk predictions almost all the time. Significant variations, whether positive or negative would suggest that the regulator should question further the effectiveness of the firm's model as a risk control technique.
_________________________
⁴ Trading results in this context refers to the sum of the daily realised and unrealised profit or loss on a marked-to-market basis. This may differ from the profit or loss which is shown in the firm's statutory accounts which will be governed by the accounting standards to which the firm is subject in its particular jurisdiction.