The National Academies

NCHRP 20-102(04) [Anticipated]

Evaluation Guidance for Automated Vehicle Pilot and Demonstration Projects

  Project Data
Funds: $75,000
Staff Responsibility: B. Ray Derr
Comments: In development
Fiscal Year: 2016

A number of pilot and demonstration automated vehicle (AV) projects are underway or planned in the United States and Europe. Each of these projects is intended to highlight different aspects of AV systems and their impacts on transportation, safety, and society. These local demonstrations are particularly important to gauge and foster societal acceptance since the public and policy-makers are often reluctant to believe that driverless technologies can work in their city, region, or country.
Each of these pilot projects includes an evaluation phase, but each evaluation approach is different, so it is difficult to directly compare the results. Evaluations are often being done by an interested party who may have a vested interest in the outcome of the evaluation and therefore are not independent and may not be totally objective. A common framework for these evaluations could help to clearly identify the real benefits, impacts, risks, and issues of the AV technologies so that other organizations could more easily gain insights from the pilot organizations’ experiences. Objective guidance on the accuracy and robustness of evaluation methodology could also increase the confidence in those results. 
The U.S. DOT is currently evaluating proposals for the Connected Vehicles Pilot Deployment Project (http://www.its.dot.gov/pilots/). Multiple awards are expected by Fall 2015. While these projects are focused on connected vehicle technologies rather than automated vehicle, many of the performance metrics and evaluation methods should be the same. A “Sources Sought” notice has been released for independent evaluation of the CV Pilot Deployments (https://www.fbo.gov/index?s=opportunity&mode=form&id=86abc4a62cdd10a27380dce8c8372eca&tab=core&_cview=0).
The objective of the proposed research is to develop a common methodological approach to evaluating the benefits, impacts, costs, safety implications, and societal acceptance of automated vehicles for the experiences of AV pilot and demonstration projects. This will be done by developing guidance on how evaluation information can be collected and analyzed in a way that will enable objective comparisons among pilot and demonstration projects.
It is expected that the guidance will be useful to state, regional, and local transportation agencies and to private-sector organizations undertaking pilot and demonstration projects in planning and undertaking the evaluation tasks. Use of the guidance will result in other organizations taking better advantage of the information gained through these projects.
Task 1. Compile information on domestic and international pilot and demonstration AV projects currently underway and planned, particularly regarding the methods and metrics that will be used to evaluate them.
Task 2. Prepare a synthesis on the performance metrics identified in Task 1, including AV technologies expected to affect those metrics. Conduct a virtual focus group of interested agencies (if the schedule allows, the CV Pilot Deployment Project agencies should be included) and selected others to validate the synthesis. Identify critical evaluation criteria that are not being collected, but should, such as societal acceptance of automated vehicle technology. Insofar as practical, insight should be sought from private-sector efforts.
Task 3. Compile information on evaluation methodologies being employed in the pilot and demonstration projects and document lessons learned.
Task 4. Prepare guidance that organizations conducting pilot and demonstration projects can use to make their evaluation results more easily and confidently comparable to those used by other organizations.  

To create a link to this page, use this URL: http://apps.trb.org/cmsfeed/TRBNetProjectDisplay.asp?ProjectID=3989