Evaluation
The post-implementation Evaluation phase enables the transport authority and/or operator to determine what value they have received from their investment, whether this meets or exceeds the original expectations, and whether more can or should be achieved. It assists the stakeholders to identify any aspects that are underperforming compared to expectations, and to target measures for improvement. It also provides valuable feedback for future investments and their design.
The Guidance presents 3 steps in the Implementation Phase:
- Pre-implementation criteria
- Post-implementation and Monitoring
- Evaluation
Following the Evaluation phase, the policy-makers can appraise the value and effectiveness of investment in ITS.
17 Establishment of performance criteria
Themes: Targeted transport functions; other transport functions affected by ITS implementation
ITS systems are implemented to improve performance, to allow new services to be provided to customers, to strengthen organizational capacity, etc. (see objectives and requirements in Steps 1 through 4 above).
It is essential to know whether the expected improvements have actually been achieved. If the actual performance is below expectation, it may be possible to adjust or adapt the system. However, many organizations fail to measure this properly, and as a result they fail to get the properly functioning system that they have paid for.
Suitable performance criteria need to be defined and adopted well in advance of deployment – ideally at the design stage - and included in the supply Contract. The performance of the pre-ITS passenger transport system should be monitored to provide a benchmark, and then the same criteria monitored following deployment.
18 Monitoring
Themes: Performance of ITS application; Performance of targeted transport functions; Impact on other transport functions
Performance monitoring consists of three main activities:
- Measuring technical performance against the technical and functional specification – i.e. has the supplier delivered the ITS system as per contract?
- Measuring effectiveness of the ITS in achieving the targeted functions in terms of the agreed upon criteria – i.e. does the ITS system help the organizational and operational units to achieve the objectives enunciated pre-implementation?
- Determining both positive and negative impacts on functions which were not part of the original project scope
19 Evaluation
Themes: Valuation of performance changes in targeted functions; Valuation of other transport impacts; Comparison with plan projections; Feedback into optimization; Feedback into future ITS planning
ITS systems involve a significant investment, in the form both of capital investment and of organizational commitment. It is strongly advised to carry out a formal evaluation for three main reasons:
- To verify if the expected value of the investment has been realized, and in which areas the benefits exceeded or were below expectations
- To help decisions about future ITS investments
- To learn lessons about the requirements, design, procurement, deployment and utilization, and how these can be improved in the future
It is noted that very few transport entities actually carry out a comprehensive evaluation of their ITS deployments. This is rather short-sighted, since they will inevitably come back to the decision-takers with proposals for future investments – either for additional ITS or to replace their existing systems. If they have not documented the benefits of the first investments, if could be more difficult to gain support for future investments.