Evaluation and enhancement of suspected opioid overdose definitions in emergency medical services data using machine learning with natural language processing
Peter Rock, Svetla Slavova, Sharon L. Walsh, Julia Martin, Daniel R. Harris
Abstract
Background
Fatal and non-fatal drug overdoses have evolved into a critical public health crisis, with over a 50% increase in the rate of fatal drug overdose since 2019. Emergency Medical Services (EMS) data has advantages over traditional emergency department data, including timeliness and captured non-transport encounters.
Introduction
Fatal and non-fatal drug overdoses have evolved into a critical public health crisis, with over a 50% increase in the rate of fatal drug overdose since 2019 [1]. Federal agencies, such as the National Institute of Health (NIH) and the Centers for Disease Control and Prevention (CDC), have invested heavily to address the crisis through initiatives like Helping End Addiction Long-Term (HEAL) [2] and Overdose Data to Action (OD2A) [3], respectively.
Materials and methods
2.1. Data source
This study employed a secondary data analysis approach using existing EMS records (data received on 01/03/2024). Data from EMS encounters in Kentucky from 2018 to 2022 were provided by the Kentucky Board of Emergency Medical Services. These data were originally collected for administrative and operational purposes, not specifically for research. Canceled encounters and interfacility transfers were excluded, as canceled encounters typically occur when EMS units are dispatched but no patient contact occurs and interfacility transfers could result in double-counting.
Results
The expert review panel evaluated 2,327 encounters, categorizing N = 690 (30%) as SOO incidents. The inter-rater reliability analysis of the training sample resulted in a Fleiss’ Kappa of 0.72, indicating substantial agreement in scenarios with more than two raters [29]. Table 3 provides a detailed breakdown of the distribution of ground-truth positive encounters as determined by the experts and according to the filters applied or the KB definitions implemented.
Discussion
Our findings highlight the potential and need for ML-NLP tools for classifying EMS SOO encounters, particularly when integrating prior domain knowledge (i.e., KB definitions) as features. The FF ML-NLP model produced the best result, outperforming any of the KB definitions, optimizing a balance of sensitivity and positive predictive value.
Conclusion
Our study underscores the need for integrating domain-specific knowledge with advanced ML-NLP techniques to improve the identification and classification of SOO encounters in EMS data, ultimately enhancing public health surveillance. We demonstrate the feasibility of building an accurate model for SOO, while presenting a replicable standards-based approach that can extend to other conditions.
Citation: Rock P, Slavova S, Walsh SL, Martin J, Harris DR (2026) Evaluation and enhancement of suspected opioid overdose definitions in emergency medical services data using machine learning with natural language processing. PLoS One 21(4): e0347589. https://doi.org/10.1371/journal.pone.0347589
Editor: Vincenzo De Luca, University of Toronto, CANADA
Received: April 13, 2025; Accepted: April 3, 2026; Published: April 28, 2026
Copyright: © 2026 Rock et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Data cannot be shared publicly because of it may contain PII/PHI within the narrative text. Authors/researchers of this project are not the data owners. Data were provided from the Kentucky Board of Emergency Medical Services (KBEMS) and maintained under a Data Use Agreement. Data availability can be found through KBEMS website: https://kbems.ky.gov/Legal/Pages/Open-Records-Request-Procedure.aspx.
Funding: This research was supported by the National Institutes of Health through the NIH HEAL Initiative under award 1R01DA057605-01 [PR,SW,SS,DH] and the Centers for Disease Control (CDC) and Prevention through the Overdose Data to Action 5NU17CE924971-03 [PR,SS]. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health or the CDC.
Competing interests: NO authors have competing interests.
