American Association of
State Highway and Transportation Officials
Special Committee on
Research and Innovation
FY2023 NCHRP PROBLEM
STATEMENT
Problem Number:
2023-D-21
Problem Title
Practical
Ground Truth Method and Tools for Evaluating Accuracy, Precision, and Bias of
Traffic Volume Counting Equipment
Background Information and Need For Research
Traffic
data accuracy is fundamental to many critical functions of highway and
transportation agencies. As traffic monitoring and detection technologies
continue to mature and third-party traffic data sources become more widely
available, traffic count programs are asked to evaluate these alternatives.
Inconsistency in data quality varies across vendors and technologies. In
addition, traffic data quality change over time due to equipment aging and
environmental effects on equipment performance. Thus, the ground truth about
traffic count accuracy obtained immediately after equipment installation may
not remain representative over time due to equipment issues or intermittent
environmental effects on sensor performance.
The FHWA Traffic Monitoring Guide recommends calibration of all
permanent and portable traffic counter each year. Several ASTM standards and methods (E 177–20,
E 2300–09, E 2532–09, E 2759–10) address the topics related to evaluating
accuracy, precision, and bias of traffic volume counts. However, some of the
methodologies may require updates to incorporate new technologies and
statistical methods and their effect on the procedures described in the
standards. In addition, highway agencies have developed their own methods and
procedures. The lack of uniformity or consistency in methods used by different
agencies to evaluate the accuracy of traffic counts leads to challenges with
analyses of national traffic data sets, comparison of traffic trends between
different agencies, or even using traffic data within a given agency, if
different technologies have been used to collect traffic data. The lack of
consistency in methods to obtain ground truth data about traffic volume counts
also creates challenges in evaluating accuracy of traffic volume estimates
obtained using analytical tools.
To
address the need to know how accurate traffic counts are upon equipment
installation or at any time during equipment operation, equipment performance
validation methodologies and standard specifications, which are widely accepted
and implemented into practical supporting tools, are critically needed to
objectively evaluate and quantify accuracy of traffic counts obtained by
different technologies. This is a fundamental step to assure soundness of the
decisions made based on the data and statistics derived from the traffic
counts. This research focuses specifically on practical methods to determine
the accuracy of traffic monitoring devices commonly used for collecting data to
compute highway traffic statistics.
Research
questions or tasks to be addressed include:
1. What are technological, operational,
and environmental challenges with obtaining accurate traffic count data from
automated traffic counters used for highway traffic statistics? Understanding
of these challenges will help in identifying sound and practical methods,
testing conditions, and limitations of the existing methods for determining
accuracy of traffic counts.
2. What practical ground-truth methods are
currently available and used to determine accuracy, precision, and bias of
traffic counts collected by automated traffic counters? What criteria are
currently used to validate traffic count data accuracy, precision, and bias?
Are the existing methods still relevant? What are the obstacles or limitations in
implementing these methods in practice? This should include methods for
evaluating traffic counter technology itself, as well as implementation of this
technology at a particular traffic monitoring site. Both accuracy and
reliability (i.e., consistent accuracy over time) of traffic counts should be
considered.
3. Develop guidance how the existing
ground-truth methods (how long of a count to make, how many lanes should be
counted, how many vehicles to count, what accuracy criteria and parameters to
include with each method to be detailed) could be improved and/or made more
practical. Select or enhance an existing method or propose a new method that
best meets the needs of highway agencies for testing and accepting installation
of automated traffic counters commonly used to collect data for highway traffic
statistics or for accepting traffic data collected by portable traffic
counters.
4. Are the selected existing or the
proposed new method(s) universally applicable and practical? Test automated
traffic counters commonly used to collect data for highway traffic statistics
and document findings. Consider a permanently installed traffic counting site
and 2–3 commonly used portable traffic monitoring technologies installed and
tested side-by-side. A minimum of four test sites is recommended. Practicality
and potential implementation of selected methodology(s) on roads with different
traffic and site characteristics should evaluated (including roadways with
throughputs over 2,000 vehicles per hour per lane and medium/low traffic volume
of 500 to 5,000 AADT) under prevailing conditions during typical equipment
installation or data collection season). Consider practicality and limitations
of implementing the selected ground truth testing methodology(s) at remote or
very low volume locations, as well as urban locations, including test duration,
safety, and vandalism considerations. Document findings and technologies
tested.
5. Adjust, as necessary, finalize, and
document the recommended practical ground truth method(s) for evaluating
accuracy, precision, and bias of traffic volume counts.
6. Develop practical tools, such as
guidance document, specification, and/or supporting software to apply the
ground truth method to evaluate accuracy, precision, and bias of traffic volume
counts.
7. Prepare a final report documenting the
results of this study which will include references methods and the limitations
when utilizing these methods that may exist (such as weather, through put
limits, congestion limitations or minimum numbers of vehicles needed) and
traffic monitoring technologies tested.
The
potential research results are of national interest and necessary to address
strategic objectives of state DOTs because traffic volume data is a key input
into management and decision support systems implemented at the national and
state-level (such as those systems mandated by Title 23 Sec. 303 - Management
systems), including HPMS, pavement and bridge management and maintenance,
congestion, and safety management systems. Ability to quantify the accuracy of
traffic counts is critical for the effective and informed use of these data in
management systems, as well as for road and bridge design and transportation
planning projects.
The seven
research tasks listed above identify the scope needed to help assure that the
research objectives can be successfully achieved within the constraints of the
proposed time and funds. The goal of the proposed research is to produce
implementation-ready ground truth method(s) and practices in the form of
standard specifications, detailed practical procedures that states’ personnel
could follow, and/or practical software tools to assure ease of implementation
of the research results in practice.
Literature Search Summary
Several
ASTM standards address topics relevant to evaluating accuracy, precision, and
bias of traffic volume counts, including the following: E 177-20 Standard
Practice for Use of the Terms Precision and Bias in ASTM Test Methods, E
2300-09 Standard Specification for Highway Traffic Monitoring Devices, E
2532-09 Standard Test Methods for Evaluating Performance of Highway Traffic
Monitoring Devices, E 2759-10 Standard Practice for Highway Traffic Monitoring
Truth-in-Data. In addition, highway agencies have developed their own methods and
procedures. Different approaches need to be reviewed and analyzed and potential
best practices and limitations need to be identified for different road and
traffic conditions. The reasons why the current standards have limited
implementation and why the implemented methodologies vary from agency to agency
need to be explored and documented.
Research Objective
The
objectives of this research are (1) develop a new or enhance an existing
practical ground truth method for evaluating accuracy, precision, and bias of
traffic volume counts obtained from portable and permanent automated traffic
counters commonly used to collect data for highway traffic statistics and (2)
develop practical procedures, specification, and/or software tools, to help
with practical implementation of the selected methodology. The results of this
research may lead to the update of ASTM E 2532 – 09 Standard Test Methods for
Evaluating Performance of Highway Traffic Monitoring Devices. See 7 potential
tasks in Section 2 above.
Urgency and Potential Benefits
This need
was first identified by the TRB Traffic Monitoring Committee ACP70 survey of
professionals in 2016. All highway agencies use traffic count data, but it is
rarely known how accurate said data are. The need is even more relevant with
the development and deployment of new traffic data collection technologies,
such as using video recognition with artificial intelligence and machine
learning technologies. Common standards and procedures are necessary to
quantify accuracy of traffic counts and assure informed uses of traffic data
and improved data sharing practices between transportation agencies.
Application of such methods will allow uniform and unbiased quantification of
traffic data accuracy coming from different data sources.
Those
benefiting from this research include federal, state, county, MPO, and city
traffic program managers; and the public at large (due to improved data quality
for transportation decision-making).
Implementation Considerations
The state
DOT offices involved in traffic data collection and data use will be involved
in implementing the results of this research, including: Planning, Programming,
Traffic Operations, Transportation System Management and Operations
(TSM&O), and Design.
States
will be able to implement the research findings by adopting the recommended
ground truth methods, procedures, specifications, and tools. Implementation
will be supported with communication methods such as webinars, training, and
peer exchanges. The AASHTO Committee on Data Management and Analytics will be a
key benefactor of these results. The results of this project may lead to
revision and update of ASTM E 2532 – 09 or a development of a new AASHTO
specification. In addition, the practical methods and support tools developed
in this project (implementing the proposed methodology) will further assure the
ease of implementation of the research results in practice.
Several
issues arise with some of the procedures described in ASTM E 2532 – 09. The
most critical one appears in Sections 7.2.9 and 7.3.7, which state “If any
specified TMD data item is not output or its difference as calculated in
7.2.8 (for all values of the data item measured) exceeds the specified
tolerance, declare the TMD nonfunctional or inaccurate and record that it
failed the Type-approval Test.” By including the words “for all values of the
data item measured” the standard does NOT make provision for random errors
through the specification of a confidence interval along with the accuracy
value. Including a confidence interval would also allow for precision and bias
to be determined, contrary to what is now stated in Sections 7.2.10 and 7.3.8
of the standard. Other issues are the omission of the mention of alternative
ground-truth methods.
Recommended Research Funding and Research
Period
Funding
Requested: $400,000. Research Period: 30 months.
Problem Statement Author(S): For each author,
provide their name, affiliation, email address and phone.
• Olga Selezneva, Applied Research
Associates, oselezneva@ara.com, (410) 203-2285
• Lawrence A. Klein, Klein &
Associates, larry@laklein.com, (714) 356-2275
• Steven Jessberger, Federal Highway
Administration, steven.jessberger@dot.gov, (202) 366-1874
Potential Panel Members: For each panel member,
provide their name, affiliation, email address and phone.
• Aaron Moss, Traffic Analyst, CDOT,
aaron.moss@state.co.us
• Chris Medina, Program Admin
Specialist III, VDOT, Chris.Medina@VDOT.Virginia.gov, 804-786-2956
Person Submitting The Problem Statement: Name, affiliation,
email address and phone.
Kent L.
Taylor, NCDOT, kltaylor@ncdot.gov (919) 345-9829