BACKGROUND

State departments of transportation (DOTs) and metropolitan planning organizations (MPOs) have long worked toward implementing performance-based planning and programming (PBPP). These practices became more robust after federal requirements for performance reporting was first introduced in the 2012 Moving Ahead for Progress in the 21st Century Act.

While many state DOTs and MPOs are fairly advanced in their application of performance-based planning, agencies seek to understand how to perform project evaluation of fully completed and operational transportation projects, i.e., post-implementation projects. Evaluating the performance of projects after they are implemented against the project’s strategic goals and objectives provides a feedback loop that informs future project selection, funding, development, and implementation. Post-implementation evaluation results are also useful for communicating to the public and decision-makers about projects.

Research is needed to provide informative, practical direction to agencies on how to design and apply post-implementation project evaluation.

OBJECTIVE

The objective of this research is to develop a guide and toolkit for the post-implementation evaluation of projects and programs. The guide should include frameworks for process and analysis, flowcharts, and decision tools that will facilitate the post-implementation evaluation. 

The guide and toolkit should be easily implemented, using self-assessments, checklists, and other tools applicable to projects of various sizes, scopes, modes, and purposes. The final deliverables should highlight examples of effective practices through case studies or other illustrative applications of the guide and tools.  The deliverables must accommodate agencies with limited experience as well as agencies that are more advanced in their evaluation practices.

The guide and toolkit will at least encompass the following elements:

• One or more process frameworks that include specific steps in how to conduct evaluations under a variety of settings;
• Timelines for conducting evaluations, recognizing that some performance goals might take years to realize after a project is operational;
• Directions on how to select the correct process and/or timeframe for a given project;
• Instruction on how to identify appropriate post implementation performance measures that respond to agencywide and project-specific goals and objectives; 
• Instruction on how to measure the success of goals against these measures, particularly over time as a project matures in its operative state;
• Methods or tools for isolating the performance of a project within a given context, such as within a given transportation and/or land use environment; 
• Methods or tools for evaluating a project in conjunction with other types of external factors that may influence the performance of a project in operation such as unanticipated economic or social disruptions, or as a component of a set of transportation or land use investments or policies;
• Methods or tools to identify, collect, and apply quantitative and qualitative data for post-implementation project evaluation; and
• Recommendations on when and how to communicate the results of evaluations to different project stakeholders, including members of the public.

RESEARCH PLAN

PHASE I 

Task 1. Conduct a literature review looking at methodologies used by DOTs and other transportation agencies (e.g., MPOs, tolling agencies, international agencies, etc.) to evaluate the impacts and performance of transportation projects and programs on established goals and objectives. Additionally, the review should include other academic, NCHRP, or other relevant literature or reports on post-implementation project evaluation to establish the state of practice and the opportunities and barriers for integrating post-project evaluation into agency practice that could be addressed by this research.  

Task 2. Survey or engage appropriate transportation agency staff and other practitioners to collect information regarding the experience of agencies and what the needs are as it relates to conducting post-implementation project evaluations. This outreach is intended to supplement the literature review in establishing the state of the practice and identifying specific research needs to address within the products of this research. Proposals shall articulate a strategy for identifying the participants and executing the outreach, but a specific plan will be developed for review and comment by the project panel and NCHRP as part of this task. Practitioners included in the engagement should represent a wide range of state DOTs and MPOs of different sizes, geographies, and complexities.

Task 3. Using the information collected in Tasks 1 and 2, conduct a gap analysis that identifies specific gaps in needed frameworks, tools, and resources that could be addressed in the next phases of research and product development.

Task 4. Develop an interim report that details the data collection efforts and major conclusions drawn from the data collected. Include a refined list and outline of all research deliverables proposed to meet the objective of this research, as well as a detailed work plan for executing Phase II.

PHASE II

Phase II will focus on developing and testing the draft guide, framework, and other deliverables, incorporating the data collected during Phase I.   

Proposals shall identify a specific set of tasks for assessing the value of the draft deliverables of this research to transportation agencies. The assessment should be rigorous and include gathering input on the draft products from a diverse set of practitioners. Phase II should include application of the draft products to three or more varied case studies volunteered by state DOTs or MPOs.

A Phase II report shall be prepared that summarizes the lessons learned from the validation of draft products conducted in Phase II and identifies specific modifications to the draft products warranted as a result of the analysis. The Phase II report will include an annotated outline of the guide, frameworks, and other tools reflecting input received and lessons learned through product validation.

PHASE III

Phase III will focus on production of the final deliverables in accordance with the project objective, responding to input received and analysis conducted during Phase II. Anticipated deliverables of this project include the following: (1) a guide and toolkit for supporting ex-post project evaluation; (2) case study examples to support application of the developed guide and toolkit; (3) an implementation plan for supporting application of the products of this research in practice; (4) a Conduct of Research report that summarizes the research process and outcomes; and (5) communication materials targeting stakeholders, leadership, etc.   

STATUS: Proposals have been received in response to the RFP.  The project panel will meet to select a contractor to perform the work.