Comparing NLP and LLM Approaches for Risk Classification Using Unstructured Data

Document Type

Conference Proceeding

Publication Date

1-1-2026

Abstract

Learning from historical risk cases to anticipate future threats is highly valuable in construction projects. However, building a comprehensive database of potential risks is costly. Consequently, many studies have turned to extracting risk information from unstructured sources such as social media, news reports, financial documents, and other publicly available content. The success of natural language processing (NLP) models in risk analysis has been well documented. However, with the emergence and growing capabilities of large language models (LLMs), there is significant potential to apply these models for generating risk-related data and classifying risks from unstructured sources. Therefore, this study evaluates the performance of classical NLP models (TF-IDF, Word2Vec, and BERT) alongside large language models (GPT-4) in identifying and classifying risks from unstructured text. Twelve configurations were tested, ranging from traditional pipelines to advanced transformer-based models, under zero-shot, instruction-based, and few-shot prompting strategies within a structured experimental design. The results show that LLMs - particularly GPT-4 with few-shot prompts - achieve competitive performance (F1 = 0.81) close to the best classical model (BERT+SVM, F1 = 0.86), without requiring any task-specific training data. Notably, LLMs demonstrated more balanced accuracy across imbalanced risk categories, underscoring their adaptability in data-scarce scenarios.

Publication Title

Construction Research Congress 2026 Advanced Technologies Artificial Intelligence and Data Analytics in Construction Selected Papers from Construction Research Congress 2026

ISBN

[9780784486986]

Link to Full Text

Share

COinS