Evaluation
Themes: Valuation of performance and other transport impacts; Comparison with plan projections; Feedback into optimisation and future ITS planning
Why is Evaluation important?
ITS systems involve a significant investment, both in the form of capital investment and of organisational commitment. It is strongly advised to carry out a formal evaluation for three main reasons:
- To verify if the expected value of the investment has been realised, and in which areas the benefits exceeded or were below expectations
- To help decisions about future ITS investments
- To learn lessons about the requirements, design, procurement, deployment and utilisation, and how these can be improved in the future
It is noted that very few transport entities carry out a comprehensive evaluation of their ITS deployments. This is rather short-sighted, since they will inevitably come back to the decision-takers with proposals for future investments – either for additional ITS or to replace their existing systems. If they have not documented the benefits of the first investments, if could be more difficult to gain support for future investments.
Approach to Evaluation
The monitoring process will provide the necessary data on benefits for project evaluation, but evaluation of the system will also require capture of all the appropriate costs. Some of these are readily identifiable, such as the contracted costs for installation, but others may prove more elusive. In this connection it is common for the full extent of management time involved in the planning, design and implementation of the scheme not to be properly recognised – nor the impact on the rest of the business while management attention has been thus distracted.
Some performance changes can be readily quantified, and hence their financial and economic benefits be calculated, but others may require a subjective assessment. In the latter case, the impacts may be indirect (for example, in changes in patronage levels over time). Evaluation in these cases may require regular surveys of travellers rather than just internally generated data from the operation itself.
The value of evaluation to the implementing entity
The importance of the evaluation lies not just in the confirmation, or otherwise, of the original investment authorisation but also in the feedback that can be provided into the optimisation process and into future ITS project planning. Any project of the complexity of an ITS implementation is likely to benefit from its ongoing incremental development, and to continuing adaptation of business processes to take best advantage of its evolution.
However our case studies have shown a worrying lack of concern about the evaluation process and, as part of that, in accountability for the expenditure of public funds. That is not to say that the ITS projects have not yielded benefits, but rather that the scale of those benefits in relation to the investments made have not been demonstrated. Inevitably this must raise suspicions about the viability of some ITS projects, and it would be in the best interest of both the industry suppliers and the operator associations to bring forward the evidence to refute these.