BACKGROUND
Previous work under NCHRP Project 20-24(37) has demonstrated the benefits of measuring and comparing performance among peer state departments of transportation (DOTs). Keeping such comparisons in the proper context, they are meant to enhance the performance of participating peer state DOTs by identifying and sharing good practice. The purpose is not to rank state DOTs for the popular press, especially when such publicly available comparisons often rely on measures with incompatible data results.
NCHRP Project 20-24(37) focused on the comparison of on-time, on-budget capital project delivery performance. The project successfully demonstrated how this comparative process is gathering user support as well as delivering timely feedback on good practices that have achieved successful results. The project report, Measuring Performance among State DOTs, describes how state DOTs may increase their use of comparative performance measures and provides a foundation for further collaborative development of comparative performance measures by the American Association of State Highway and Transportation Officials (AASHTO) and its member states. NCHRP Project 20-24(37)A continued this effort by expanding the number of participating states and producing a summary of best practices, titled, Comparing State DOTs’ Construction Project Cost & Schedule Perfromance-28 Best Practices from 9 States. For copies of these reports, see the writeup for NCHRP project 20-24(37) found here.
It would be beneficial to continue the positive momentum realized from this cooperative success by directing it toward another key concern of transportation officials. Several years of experience with the Federal Highway Administration’s Highway Performance Monitoring System (HPMS) and its Highway Statistics report have allowed state DOTs to establish a mature set of asset management data that’s available for comparisons, making this set of data a very good source for continuing the process and to further demonstrate the concept of comparative performance measures.
The Highway Statistics report has been published annually since 1945. It provides a wealth of transportation data for each state and is a readily available source of information to establish long-range trends. Unfortunately, as its publishers warn, inconsistencies in measurement interpretation and data collection practices limit the accuracy of this data for direct comparisons among the state agencies. The report suggests that care be taken to properly identify peer states before analyzing the data, but little guidance is provided.
What to measure next?
Overdrive, a national trucking magazine that annually conducts driver opinion surveys, has noted that over-the-road truck drivers identify potholes, patches, and cracking as prime sources for complaints, ahead of traffic congestion and construction delays. The experiences of this seasoned traveling population is just one example of how a smooth pavement surface satisfies a primary transportation need while publicly demonstrating the results of successful highway preservation programs.
Pavement smoothness is a customer oriented measure, like the measures of capital construction project delivery evaluated in the previous measurement study. It is a DOT performance attribute that is noticeable by the traveling public, an indication of just how well “the rubber meets the road.”
Why not use the existing data?
A report by the United States General Accounting Office in 2000 highlighted that inconsistencies in pavement smoothness collection devices, operating procedures, and the formulas used to calculate the International Roughness Indices (IRI) reduced the accuracy of the existing performance measure and limited the ability to make state-to-state comparisons.
Several states have used the Highway Statistics report to contrast their pavement conditions with other states and the national average; however, this data is often footnoted to caution readers that it may not be comparable because of differing measurement and reporting techniques.
Despite these noted qualifications, external comparisons using the Highway Statistics IRI data are made by trucking and automotive magazines as well as other government monitoring organizations.
Although the HPMS Field Manual is specific about how IRI data in Highway Statistics should be collected and analyzed, variations occur. Speed restrictions, short section lengths, numerous traffic signals, and traffic congestion make data collection consistency difficult to maintain in urban areas. Decisions whether to include bridge segments in the data or to evaluate only pavement segments have been under debate as to whether roughness measurement is more suitable as an assessment of pavement condition or as an indication of the driving experience. A direct comparison of the data available by the states may minimize these issues and achieve comparable statistics.
Where are the good practices?
As already mentioned, the purpose is not to rank participants, but to highlight top performing strategies. NCHRP Project 20-24(37) maintains anonymity for states providing the comparison data with the exception of the top performing states (with the DOTs’ approval) that are highlighted in the reporting of successful practices. Accordingly, IRI data will be grouped and reported consistently across the peer states involved in the comparisons. These groups will include flexible or rigid pavement types, reconciliation of segments involving bridges or urban segments, as well as the different methods used by states for data collection and computation of IRI values.
OBJECTIVES
The objectives of this research project are to (1) facilitate the process of comparing the performance of peer state DOTs using the International Roughness Index (IRI); (2) prepare, analyze, and evaluate the performance data; and (3) identify and document good practice. The focus of the research will be on the Interstate Highway System only.
RESEARCH PROCESS
The AASHTO Standing Committee on Quality’s Subcommittee on Performance Measurement and Benchmarking (QSPMB) will provide all states the opportunity to participate. Starting with information collected for the annual HPMS submittal, including newly added metadata for pavement smoothness, peer states will be solicited for participation. Approximate number of states to participate is 20.
Once the volunteer states have been identified, the research contractor will be required to:
1. Perform one or two conference calls with volunteer states to identify data requirements and to establish peer groups (estimated time: 2-3 hours to be held sometime in the month of September).
a. Learn and understand data requirements.
b. Examine IRI metadata provided for HPMS submittal to determine significant process differences.
c. Agree upon data requirements for peer groups and analysis methods.
d. Resolve any anonymity issues.
2. Perform one conference call to confirm that each volunteer state can actually produce the required data conforming to the definitions and variables defined previously. Peer groupings may need to be re-examined based upon the resulting group sizes. (Estimated time: One 2-hour conference call to be held in October 2007.)
3. Obtain data from the lead state(s). (Note: The QSPMB will assist in identifying lead state(s).) Each participating state will collect and submit the agreed-upon data to the lead state(s), who will interact with the research contractor for compilation and analysis. (Data to be provided by the end of November 2007).
4. Provide to each of the volunteer states, a draft compilation of the metrics and analysis by peer groupings. Data will be compared to appropriate national statistics using current HPMS data submittals and available metadata and will be reported as a baseline metric. (The draft report is due by the end of December 2007 with comments due back from volunteer states within 20 days). Anonymity will be maintained using nondescript codes and the option will remain open for volunteer states to withdraw from the study at any time. States identified as lead performers will need to agree to disclose their identities for the purpose of determining the good practices that contribute to the superior performance results.
5. Analyze the top performing states (no less than 5) identified by peer groupings and analysis attributes. Interviews will be conducted to document successful practices employed for both rigid and flexible pavement types and in regards to rehabilitation triggers used for pavement preservation. All participants will be surveyed for recommendations for improvements to data or process for future comparisons. Final report will share interview findings with the performance results.
6. Submit the final report for NCHRP review and comment by February 29, 2008.
Progress (April 30, 2008): Contractor supplied draft final report. Panel reviewed this report. Contractor will revise the report based on panel comments. Final report expected by end of May 2008.
June 2008: Contractor's final report is available at https://www.trb.org/NotesDocs/20-24(37)B_FR.pdf
