Western Journal of Emergency Medicine: Integrating Emergency Care with Population Health

Introduction: Psychostimulant detection rates among emergency department (ED) patients presenting with suspected substance intoxication are low and are influenced by limited access to confirmatory testing, insufficient clinical information, and variability in physician experience. These challenges often result in high false-positive rates and unnecessary resource use. We hypothesized that machine learning (ML) algorithms could maintain reliable diagnostic performance across a temporal validation cohort and perform comparably to physicians with varying experience levels.

Methods: This multicenter study prospectively enrolled ED patients with suspected substance intoxication between March 2019 and January 2020. We chronologically partitioned data into a training set (March–October 2019) and a temporal validation cohort (November 2019–January 2020) to assess model generalizability. After determining psychostimulant intoxication via liquid chromatography-mass spectrometry from urine samples, we trained the ML algorithms using the confirmed positive cases. The primary outcome was the diagnostic discrimination of the models, measured by the area under the receiver operating characteristic curve (AUC) in the temporal validation cohort. Secondary outcomes included model accuracy, the Youden index, and a comparative assessment of sensitivity and specificity relative to five physicians of varying experience levels.

Results: Among 1,382 patients (mean age 43.8 [standard deviation 19.2] years; 55.3% male), 302 patients (21.9%) had confirmed psychostimulant intoxication, primarily involving (meth)amphetamines (134 [44.4%]), cathinones (107 [35.4%]), and ketamine (97 [32.1%]). In the temporal validation cohort (n = 382), the ML models demonstrated consistent performance, with AUCs ranging from 0.786 to 0.836. The primary outcome was met by the top-performing extreme gradient boosting (XGB) model (AUC, 0.836, 95% CI, 0.785-0.882; accuracy, 0.866 (95% CI, 0.838-0.895); and Youden index, 0.590 (95% CI, 0.480-0.694). To evaluate secondary outcomes, the XGB model’s predictive accuracy was evaluated against 5 emergency physicians. The model surpassed the junior emergency physicians in both specificity (0.910 vs 0.633-0.867) and sensitivity (0.681 vs 0.500-0.633); its specificity was comparable to that of the senior emergency physicians (0.900-0.933).

Conclusion: Machine learning models provide reliable diagnostic utility for identifying psychostimulant intoxication, achieving performance comparable to senior emergency physicians. While moderate sensitivity (0.389–0.708) limits their role as standalone rule-out tools, their high specificity provides a standardized tool to assist clinical decision-making and reduce diagnostic uncertainty in the fast-paced ED setting.