Trespassing along railroad rights-of-way continues to be the leading cause of rail-related fatalities in the US. Based on the Federal Railroad Administration (FRA) database, the annual number of trespassing incidents increased from 834 in 2017 to 1,197 in 2023. The railroad industry needs effective solutions to reduce trespassing and save lives.  Currently, trespassers are identified by locomotive engineers, who record the incidents on paper records which are later transcribed into Excel files. This manual process has several shortcomings which limit the railroad industry’s ability to understand trespasser trends and to implement effective preventive measures. This project developed a technology solution to solve this challenge -- a Computer Vision AI based turnkey real-time trespasser detection system using railroad outward-facing video. The system was tested on recorded outward-facing video records and deployed on two locomotives for a combined fifteen days of live analysis. The project yielded three key innovations. First, a state-of-the-art semantic segmentation model was adapted to detect trespassers. Second, a comprehensive dataset was built for both training and testing the artificial intelligence using generative AI to create images of trespassers for model training. Third, a turnkey edge computing system was developed that can use existing or newly installed outward-facing camera systems. Through experimental results and real-world testing, the system analyzed 210 hours of live footage and 31 hours of recorded footage, successfully detecting 20 trespassers. All detected records were manually reviewed to validate the system’s results and to better understand trespassing characteristics. Based on the validation, the system achieved 99% weighted precision. The implementation of this product will enhance railroad safety by providing a plethora of actionable data and evidence to support data-driven decision making. The system automatically identifies trespassers and logs their GPS locations, positions in the right-of-way, and timestamps in a structured database. This removes the need for converting paper records into electronic datasets and collects trespassing records uninfluenced by human factors. When aggregated, these records will help identify trespassing hotspots to guide the prioritization of preventive engineering, enforcement, and education measures. The records automatically captured by the system provide easy access to video examples and additional context for understanding trespassing incidents.

The Final Report is available.