American Association of State Highway and Transportation Officials

Special Committee on Research and Innovation

FY2023 NCHRP PROBLEM STATEMENT TEMPLATE

Problem Number: 2023-B-12

Problem Title

Ex Post Project Evaluation: Frameworks, Guidance and Tools to Support Post-Implementation Evaluation of Transportation Projects

Background Information and Need For Research

State DOTs and MPOs throughout the US have implemented robust performance-based planning and programming processes in their agencies due to the federal requirements articulated in the MAP-21, FAST Act and other state federal laws. According to the FHWA Performance-Based Planning and Programming Guidebook, there are three core elements of the PBPP framework:

1. Strategic Direction: Where do we want to go?

2. Analysis: How are we going to get there?

3. Programming: What will it take?

4. Implementation and Evaluation: How did we do?

The first three elements are well-documented and conducted in existing transportation planning and programming processes. The fourth element - implementation and evaluation phase - seeks to answer the questions: What did we do? How did we do? How can we improve? This fourth element, which include monitoring, evaluation and reporting, is something that the transportation industry has struggled with. Since the PBPP process includes an aspect of a feedback loop that attempts to assess whether projects built actually delivered the performance predicted, the implementation and evaluation is a critical component that is not done on a regular basis. This feedback loop is essential for understanding the impacts and effectiveness of transportation projects and investments. It is crucial to evaluate what has been delivered and what the performance was so that better, well-informed decisions can be made in the future. The evaluation piece is necessary to continue the feedback loop and what the benefits are.

Already, there are many state transportation agencies and MPOs that have well-documented processes to identify a program of projects that is predicted to have certain impacts on the performance of the transportation system. For example, the Virginia SMART SCALE program, well known to many in the transportation planning industry, uses a scoring method to allocate funding to prioritize project selection. The SMART SCALE program adopts a sophisticated, data-supported process to select projects based on their forecasted benefits. The SMAR SCALE framework is a great example for the first three steps, yet for the fourth step on implementation and evaluation is not conducted but would be critical to understanding if the process is working and affecting the performance of the transportation system. In fact, VDOT has determined that the program’s performance-based approach can be enhanced by developing methods to evaluate the level to which completed projects, and the program as a whole, have contributed toward critical goals such as reducing congestion, improving reliability, and increasing safety for all travelers. By developing a methodology to achieve a holistic view of a transportation agency’s program of projects impacts on performance, planners and program mangers will be better suited to determine what elements of projects – regardless of their funding source – will be most effective.

The purpose of this research is to provide needed frameworks, guidance and tools for transportation planners and analysts to better evaluate both transportation projects and program of projects. Evaluating post-implementation benefits provides a feedback loop to help ensure that information on the effectiveness of projects and programs informs future project selection and implementation. Strong findings of effectiveness from implemented project and program can encourage their further implementation, while weak effectiveness may suggest using alternative solutions. In addition, findings from post-implementation studies can help to identify the characteristics of a corridor or situation under which certain projects and program are most effective. Finally, results can be useful for communicating with the public and decision-makers about the benefits of strategies such as demand management and operational improvements, where projects/programs are often not as readily visible to the public.

Literature Search Summary

A preliminary literature search shows that there is already a great deal of information available about how to measure before/after impacts on individual transportation projects. The evaluations often focus on a particular area such as safety, asset condition (e.g., pavements and bridges), and economic impacts. For safety, researchers have, and continue to, study the impacts of various safety practices on outcomes such as fatalities, serious injuries, and crash rates. For asset condition, there is a significant amount of research and findings on the impact that various funding levels and programs will have on pavement and bridge condition based upon the work to be performed. And, for economic impacts, the SHRP2 program developed EconWorks which is a collection of before-after case studies that focuses on the impact of transportation projects from an economic perspective. EconWorks assists planners in assessing the possible economic implications of projects and considers outputs such as travel time reliability, access to labor and goods markets, and intermodal connectivity. The purpose of this research is to not repeat and replicate these evaluation processes and areas.

In addition, the evaluation of major infrastructure investments has been well research and used to improve the modeling and predictive capabilities of models and tools used by the industry. For example, Pickerel in his seminal piece on assessing spending for large-scale transit systems found that analysts routinely overstated the benefits and underestimated the costs to justify these projects. Similarly, Flyvbjerg found these results in other large-scale transportation infrastructure projects throughout the world. These types of evaluations had an impact on improving the predictive models, tools, and techniques that analysts used to justify spending large sums of money on transportation infrastructure. However, it is still rare for any ex post analysis of large transportation infrastructure project to be conducted to better learn if the benefits predicted did indeed come to fruition.

What has not been well-researched or understood is how to measure the impact project-level evaluation for operational and economic outcome type measures as well as program-level evaluation of completed projects. Currently, the only known research underway to better develop the feedback loop that analyzes the effects of a transportation program on the transportation system is that of the Virginia DOT that is measuring the SMART SCALE project performance. This project, begun in 2020, will determine the appropriate measures, methodology and process to assess SMART SCALE project performance. An important aim is to find evidence of transportation agency practices and methods to support project-level and program-level evaluation of completed projects.

Research Objective

The objective of this research is to develop guidance and tools for conducting post-implementation evaluation of the transportation projects and programs. The guidance and tools will provide transportation planners and analysts with the necessary resources in order for them to conduct effective post-implementation evaluations of transportation projects and programs and further support implementation of the performance-based planning and programming processes that transportation agencies are required to do.

The following are the key phases of this research:

1. Conduct a literature review looking at methodologies used by DOTs and other transportation agencies (e.g., MPOs, RPOs, tolling agencies, international agencies, etc.) to gather information about how they evaluate the impact of transportation projects and programs.

2. Survey transportation agencies on what their needs are as it relates to conducting post-implementation project evaluations.

3. Conduct a gap analysis that identifies how this research can address the gaps in needed frameworks, tools and resources.

4. Develop the needed frameworks, tools and resources for conducting post-implementation project and program evaluations. This should include practical guides for practitioners to use in conducting post-implementation evaluation and models and tools that could be used. The guides need to be practical step-by-step resources for planning practitioners seeking to understand what information is needed and how to conduct such analyses.

5. Work with selected state DOTs and MPOs on implementing the results.

Urgency and Potential Benefits

Given the requirements of the FAST Act, transportation agencies will need to have clear strategies in place for providing feedback on performance efforts, and for using this information to improve future decision making outcomes. This project is expected to add value for performance monitoring and decision making at the local, state, and federal levels. It is expected that this work will provide some of the guidance needed to close the performance planning loop.

By summarizing the various existing methodologies for analyzing and measuring changes to performance attributes, transportation agencies will be provided additional tools to understand the impacts of different types of projects on different performance attributes. Agencies will then understand the types of projects they need to initiate and implement in order to achieve their agency’s performance measure targets.

Agencies are currently in various stages of developing and implementing performance measures required federally. Completing this research and guide now will provide states with a pivotal resource in evaluating performance measures in the near future and longer term.

Implementation Considerations

These research results will be of primary interest to state DOT and MPOS, as well as offices responsible for performance and asset management, public engagement, and research. The guidebook and toolkit should be developed to be easily implemented by these offices, such as easy to use self-assessments, checklists, and methods and examples of effective practices that could be adapted.

Recommended Research Funding And Research Period

Research Funding: $600,000

Research Period: 30 months

Problem Statement Author(s): For each author, provide their name, affiliation, email address and phone.

• Bradley Sharlow, Michigan DOT

• Cynthia Landez, KCI

• Bryan Pounds, WSP

• Scott Phinney, Ohio DOT

• April Delchamps, City of Kent, Washington

• Matthew Hardy, PhD., AASHTO

• Kyla Elzinga, AASHTO

This research problem statement was developed in cooperation between the AASHTO Committee on Planning and the TRB Committees AEP 10 (Transportation Planning Policy and Processes) and AEP 15 (Transportation Planning Analysis and Application).

Potential Panel Members: For each panel member, provide their name, affiliation, email address and phone.

Please contact Matt Hardy, AASHTO Program Director for Planning and Performance Management, for a list of recommended panel members representing the AASHTO Committee on Planning. Below is a preliminary list of interested panel members:

• David Wasserman – STIP Western Region Manager (NCDOT), dswasserman@ncdot.gov

• Shante Hastings, Deputy Secretary and Chief Engineer (DelDOT), Hastings@state.de.us

• Justin Bruner, Bridge Asset Management Division Chief (PennDOT), jbruner@pa.gov

• Mark Nelson, Director, Office of Statewide Multimodal Planning (MnDOT), mark.B.nelson@state.mn.us

• Scott Phinney, Ohio DOT

• April Delchamps, City of Kent, Washington

• Bradley Sharlow, Michigan DOT

Person Submitting The Problem Statement: Name, affiliation, email address and phone.

Provide contact information for the individual submitting this problem statement.

Matthew Hardy

202-624-3625

mhardy@aashto.org

Affiliation: AASHTO staff on behalf of the AASHTO Committee on Planning, Kristina Swallow, Chair.