State departments of transportation (DOTs) and metropolitan planning organizations (MPOs) have long worked toward implementing performance-based planning and programming (PBPP). These practices became more robust after federal requirements for performance reporting was first introduced in the 2012 Moving Ahead for Progress in the 21st Century Act.
While many state DOTs and MPOs are fairly advanced in their application of performance-based planning, agencies seek to understand how to perform project evaluation of fully completed and operational transportation projects, i.e., post-implementation projects. Evaluating the performance of projects after they are implemented against the project’s strategic goals and objectives provides a feedback loop that informs future project selection, funding, development, and implementation. Post-implementation evaluation results are also useful for communicating to the public and decision-makers about projects.
Research is needed to provide informative, practical direction to agencies on how to design and apply post-implementation project evaluation.
The objective of this research is to develop a guide and toolkit for the post-implementation evaluation of projects and programs. The guide should include frameworks for process and analysis, flowcharts, and decision tools that will facilitate the post-implementation evaluation.
The guide and toolkit should be easily implemented, using self-assessments, checklists, and other tools applicable to projects of various sizes, scopes, modes, and purposes. The final deliverables should highlight examples of effective practices through case studies or other illustrative applications of the guide and tools. The deliverables must accommodate agencies with limited experience as well as agencies that are more advanced in their evaluation practices.
The guide and toolkit will at least encompass the following elements:
- One or more process frameworks that include specific steps in how to conduct evaluations under a variety of settings;
- Timelines for conducting evaluations, recognizing that some performance goals might take years to realize after a project is operational;
- Directions on how to select the correct process and/or timeframe for a given project;
- Instruction on how to identify appropriate post implementation performance measures that respond to agencywide and project-specific goals and objectives;
- Instruction on how to measure the success of goals against these measures, particularly over time as a project matures in its operative state;
- Methods or tools for isolating the performance of a project within a given context, such as within a given transportation and/or land use environment;
- Methods or tools for evaluating a project in conjunction with other types of external factors that may influence the performance of a project in operation such as unanticipated economic or social disruptions, or as a component of a set of transportation or land use investments or policies;
- Methods or tools to identify, collect, and apply quantitative and qualitative data for post-implementation project evaluation; and
- Recommendations on when and how to communicate the results of evaluations to different project stakeholders, including members of the public.
The NCHRP is seeking the insights of proposers on how best to achieve the research objective. Proposers are expected to describe research plans that can realistically be accomplished within the constraints of available funds and contract time.
The research plan will be divided into three phases, as described below. Proposals should describe tasks and deliverables for each phase designed to achieve the research objective. Proposals must present the proposers' current thinking in sufficient detail to demonstrate their understanding of the issues and the soundness of their approach to meeting the research objective.
The research plan should build in appropriate checkpoints with the NCHRP project panel including (1) an Amplified Research Plan within 15 days of the start of the contract; (2) a web-enabled meeting kick-off following NCHRP review of the Amplified Research Plan; (3) an in-person interim meeting following the review of the interim report and Phase II work plan; and (4) a web-enabled meeting to review the draft deliverables for final production. Costs for face-to-face meetings, including venue and travel for panel members to attend the meeting, will be paid separately by NCHRP.
Task 1. Conduct a literature review looking at methodologies used by DOTs and other transportation agencies (e.g., MPOs, tolling agencies, international agencies, etc.) to evaluate the impacts and performance of transportation projects and programs on established goals and objectives. Additionally, the review should include other academic, NCHRP, or other relevant literature or reports on post-implementation project evaluation to establish the state of practice and the opportunities and barriers for integrating post-project evaluation into agency practice that could be addressed by this research.
Task 2. Survey or engage appropriate transportation agency staff and other practitioners to collect information regarding the experience of agencies and what the needs are as it relates to conducting post-implementation project evaluations. This outreach is intended to supplement the literature review in establishing the state of the practice and identifying specific research needs to address within the products of this research. Proposals shall articulate a strategy for identifying the participants and executing the outreach, but a specific plan will be developed for review and comment by the project panel and NCHRP as part of this task. Practitioners included in the engagement should represent a wide range of state DOTs and MPOs of different sizes, geographies, and complexities.
Task 3. Using the information collected in Tasks 1 and 2, conduct a gap analysis that identifies specific gaps in needed frameworks, tools, and resources that could be addressed in the next phases of research and product development.
Task 4. Develop an interim report that details the data collection efforts and major conclusions drawn from the data collected. Include a refined list and outline of all research deliverables proposed to meet the objective of this research, as well as a detailed work plan for executing Phase II.
Note: The research team will meet with the project panel at an in-person meeting hosted by the NCHRP to discuss the outcomes of Phase I and discuss the recommended approach for Phase II as provided in the Phase II work plan. NCHRP approval of the Phase I products is require before initiating work in Phase II.
Phase II will focus on developing and testing the draft guide, framework, and other deliverables, incorporating the data collected during Phase I.
Proposals shall identify a specific set of tasks for assessing the value of the draft deliverables of this research to transportation agencies. The assessment should be rigorous and include gathering input on the draft products from a diverse set of practitioners. Phase II should include application of the draft products to three or more varied case studies volunteered by state DOTs or MPOs.
A Phase II report shall be prepared that summarizes the lessons learned from the validation of draft products conducted in Phase II and identifies specific modifications to the draft products warranted as a result of the analysis. The Phase II report will include an annotated outline of the guide, frameworks, and other tools reflecting input received and lessons learned through product validation.
Note: NCHRP approval of the Phase II products is require before initiating work in Phase III.
Phase III will focus on production of the final deliverables in accordance with the project objective, responding to input received and analysis conducted during Phase II. Anticipated deliverables of this project include the following: (1) a guide and toolkit for supporting ex-post project evaluation; (2) case study examples to support application of the developed guide and toolkit; (3) an implementation plan for supporting application of the products of this research in practice (see Special Note I); (4) a Conduct of Research report that summarizes the research process and outcomes; and (5) communication materials targeting stakeholders, leadership, etc.
Note: Following receipt of the draft final deliverables, the remaining 3 months shall be for NCHRP review and comment and for research agency preparation of the final deliverables.
A. The Information and Instructions for Preparing Proposals for the Transportation Research Board’s Cooperative Research Programs were revised in May 2022. Please take note of the new and revised text which is highlighted in yellow.
B. Proposals must be submitted as a single PDF file with a maximum file size of 10 MB. The PDF must be formatted for standard 8 ½” X 11” paper, and the entire proposal must not exceed 60 pages (according to the page count displayed in the PDF). Proposals that do not meet these requirements will be rejected. For other requirements, refer to chapter V of the instructions.
C. The Information and Instructions for Preparing Proposals for the Transportation Research Board’s Cooperative Research Programs have been modified to include a revised policy and instructions for disclosing Investigator Conflict of Interest. For more information, refer to chapter IV of the instructions. A detailed definition and examples can be found in the CRP Conflict of Interest Policy for Contractors. The proposer recommended by the project panel will be required to submit an Investigator Conflict of Interest and Disclosure Form as a prerequisite for contract negotiations.
D. Proposals will be rejected if any of the proposed research team members work for organizations represented on the project panel. The panel roster for this project can be found at https://www.mytrb.org/OnlineDirectory/Committee/Details/6785. Proposers may not contact panel members directly; this roster is provided solely for the purpose of avoiding potential conflicts of interest.
E. Proprietary Products - If any proprietary products are to be used or tested in the project, please refer to Item 6 in the Information and Instructions for Preparing Proposals.
F. Proposals are evaluated by the NCHRP staff and project panels consisting of individuals collectively knowledgeable in the problem area. The project panel will recommend their first choice proposal considering the following factors: (1) the proposer's demonstrated understanding of the problem; (2) the merit of the proposed research approach and experiment design; (3) the experience, qualifications, and objectivity of the research team in the same or closely related problem area; (4) the plan for ensuring application of results; (5) how the proposer approaches inclusion and diversity in the composition of their team and research approach, including participation by certified Disadvantaged Business Enterprises; and, if relevant, (6) the adequacy of the facilities. A recommendation by the project panel is not a guarantee of a contract. The National Academy of Sciences (NAS - the contracting authority for the National Academies of Sciences, Engineering, and Medicine) will conduct an internal due diligence review and risk assessment of the panel’s recommended proposal before contract negotiations continue.
Note: The proposer's approach to inclusion and diversity as well as participation by Disadvantaged Business Enterprises should be incorporated in Item 11 of the proposal.
G. Copyrights - All data, written materials, computer software, graphic and photographic images, and other information prepared under the contract and the copyrights therein shall be owned by the National Academy of Sciences. The contractor and subcontractors will be able to publish this material for non-commercial purposes, for internal use, or to further academic research or studies with permission from TRB Cooperative Research Programs. The contractor and subcontractors will not be allowed to sell the project material without prior approval by the National Academy of Sciences. By signing a contract with the National Academy of Sciences, contractors accept legal responsibility for any copyright infringement that may exist in work done for TRB. Contractors are therefore responsible for obtaining all necessary permissions for use of copyrighted material in TRB's Cooperative Research Programs publications. For guidance on TRB's policies on using copyrighted material please consult Section 5.4, "Use of Copyrighted Material," in the Procedural Manual for Contractors.
H. Proposals should include a task-by-task breakdown of labor hours for each staff member as shown in Figure 4 in the Information and Instructions for Preparing Proposals. Proposals also should include a breakdown of all costs (e.g., wages, indirect costs, travel, materials, and total) for each task using Figures 5 and 6 in the brochure. Please note that TRB Cooperative Research Program subawards (selected proposers are considered subawards to the National Academy of Sciences, the parent organization of TRB) must comply with 2 CFR 200 – Uniform Administrative Requirements, Cost Principles, and Audit Requirements for Federal Awards. These requirements include a provision that proposers without a "federally" Negotiated Indirect Costs Rate Agreement (NICRA) shall be subject to a maximum allowable indirect rate of 10% of Modified Total Direct Costs. Modified Total Direct Costs include all salaries and wages, applicable fringe benefits, materials and supplies, services, travel, and up to the first $25,000 of each lower tier subaward and subcontract. Modified Total Direct Costs exclude equipment, capital expenditures, charges for patient care, rental costs, tuition remission, scholarships and fellowships, participant support costs and the portion of each lower tier subaward and subcontract in excess of $25,000.
I. The required technical memorandum titled “Implementation of Research Findings and Products” should (a) provide recommendations on how to best put the research findings/products into practice; (b) identify possible institutions that might take leadership in applying the research findings/products; (c) identify issues affecting potential implementation of the findings/products and recommend possible actions to address these issues; and (d) recommend methods of identifying and measuring the impacts associated with implementation of the findings/products. Implementation of these recommendations is not part of the research project and, if warranted, details of these actions will be developed and implemented in future efforts.
The research team will be expected to provide input to an implementation team consisting of panel members, AASHTO committee members, the NCHRP Implementation Coordinator, and others in order to meet the goals of NCHRP Active Implementation: Moving Research into Practice, available at http://onlinepubs.trb.org/onlinepubs/nchrp/docs/NCHRP_ActiveImplementation.pdf
J. If the team proposes a Principal Investigator who is not an employee of the Prime Contractor, or if the Prime Contractor is proposed to conduct less than 50% of the total effort (by time or budget), then section five of the proposal should include: (1) a justification of why this approach is appropriate, and (2) a description of how the Prime Contractor will ensure adequate communication and coordination with their Subcontractors throughout the project.
K. All budget information should be suitable for printing on 8½″ x 11″ paper. If a budget page cannot fit on a single 8½″ x 11″ page, it should be split over multiple pages. Proposers must use the Excel templates provided in the Information and Instructions for Preparing Proposals for the Transportation Research Board’s Cooperative Research Programs.