An uncertainty-aware decision support system: Integrating text narratives and conformal prediction for trustworthy accident code classification

Document Type

Article

Publication Date

12-1-2025

Abstract

It is imperative to assign accident classification codes to the Mine Safety and Health Administration (MSHA) accident data for effective data analysis and risk assessment. Although trained personnel are capable of performing this task, the manual process is both time-consuming and resource-intensive. Automating the classification process with machine learning (ML) algorithms promises to expedite code assignment. However, ML predictions typically lack uncertainty metrics. This study proposes an uncertainty-aware hierarchical classification framework that assists human experts in efficiently and accurately assigning accident codes. Several text representation techniques combined with different ML algorithms were employed within a hierarchical architecture to assign classification codes. Low-frequency codes were consolidated into a single category, with a primary classifier distinguishing between these and a secondary classifier further classifying the grouped categories. Regularized Adaptive Prediction Sets (RAPS) was integrated to quantify uncertainty. Highly confident predictions yielding single-class sets were automatically classified, whereas multi-class sets were flagged for manual review. Primary Classifier with XGBoost with word2vec text representation achieved the best performance, with 95.12 % coverage, 37.02 % single-class prediction sets at 96.11 % accuracy, and an average prediction set size of 2.39. Whereas the secondary classifier, a logistic regression model with TF-IDF representation, yielded 96.19 % coverage, an average set size of 1.80, and 53.66 % single-class prediction sets with 98.90 % accuracy. Additionally, sensitivity analysis determined that a 95 % coverage guarantee offers the best trade-off between prediction set size and coverage. The framework effectively integrates conformal prediction to quantify uncertainty and aid human experts in improving the decision-making process in safety management. Although the framework is broadly applicable across different sectors, it needs to be retrained on domain-specific data for effective use.

Publication Title

Process Safety and Environmental Protection

Link to Full Text

Share

COinS