Benchmarking large language models for supply chain risk identification: an extended evaluation within the LARD-SC framework

In one sentence

Summary

The paper addresses the challenge of timely and accurate supply chain risk identification in complex global networks, where traditional expert-based methods struggle with the volume and unstructured nature of daily news. It positions Large Language Models (LLMs) as transformative tools for automating the detection and classification of emerging risks, extending the previously introduced LARD-SC service-oriented architecture. The research aims to provide empirical evidence on which LLMs best support proactive risk management by integrating advanced text analytics into operational resilience strategies. The methodology involves an extended evaluation of the LARD-SC framework, specifically benchmarking GPT-3.5 Turbo, GPT-4o, GPT-4o Mini, Claude 3.5 Sonnet, and Claude 3.5 Haiku. The study utilizes a curated dataset of 120 real-world news articles concerning Apple’s Tier 1 suppliers. The LARD-SC framework employs a structured pipeline: Data Collection and Visualization for Risk Analysis (DCV-RA) gathers and preprocesses news; Large Language Model-based approach for Risk Identification (LLM-RI) extracts risk events, likelihood, and impact using specialized prompts; and Large Language Model-based approach for Risk Classification (LLM-RC) categorizes risks using the Cambridge Taxonomy of Business Risks. Performance is assessed via expert-reviewed metrics: Risk Validation Rate (RVR), Potential Risk Rate (PRR), and False Identification Rate (FIR), culminating in a Relative Performance Index (RPI). Findings indicate that advanced GPT-4o variants deliver the most consistent and accurate risk identifications, achieving higher RVR and minimizing false positives compared to GPT-3.5 and Claude models. The study highlights the value of LLMs in capturing novel, emergent risks beyond pre-defined dictionaries and their scalability for automated monitoring. However, limitations include the potential for hallucinations, dependency on prompt engineering quality, and the need for human-in-the-loop validation. The paper concludes that while LLMs significantly enhance SCRM, successful deployment requires balancing technological sophistication with careful oversight and addressing ethical considerations like data privacy and bias.

Research problem

The paper tackles the difficulty of identifying and classifying supply chain risks from vast amounts of unstructured news data, a task where traditional methods are resource-intensive and often inadequate. It seeks to determine how effectively various state-of-the-art Large Language Models can automate this process within a service-oriented architecture. The core problem is ensuring accurate, timely, and actionable risk signals to maintain operational resilience in complex global supply chains.

Core contributions

• Extended benchmarking of five LLM variants (GPT-3.5, GPT-4o, GPT-4o Mini, Claude 3.5 Sonnet, Claude 3.5 Haiku) for supply chain risk identification.
• Integration of high-performing LLMs into the LARD-SC framework, demonstrating improved risk detection accuracy over previous iterations.
• Development of a standardized evaluation methodology using expert-reviewed metrics: Risk Validation Rate (RVR), Potential Risk Rate (PRR), and False Identification Rate (FIR).
• Derivation of a Relative Performance Index (RPI) to quantitatively compare LLM performance in real-world supply chain contexts.
• Demonstration of LLMs' ability to capture novel and emergent risks beyond pre-defined dictionaries, enhancing proactive risk management.

Method

• Data Collection: Automated retrieval of supplier-related news using GoogleNews Python library and deduplication.
• Preprocessing: HTML parsing with BeautifulSoup to extract clean text and metadata.
• Risk Identification (LLM-RI): Use of specialized prompts to extract risk events, likelihood, and impact from text.
• Risk Classification (LLM-RC): Mapping identified risks to the Cambridge Taxonomy of Business Risks (CTBR).
• Visualization: Integration of risk data into Neo4j graph database for interactive exploration.
• Evaluation: Expert review of LLM outputs to calculate RVR, PRR, and FIR.
• Comparison: Relative Performance Index (RPI) calculation to rank LLM variants.

Results & findings

• GPT-4o variants achieved the highest Risk Validation Rate (RVR) and lowest False Identification Rate (FIR) among all tested models.
• GPT-3.5 Turbo showed promising capabilities but was outperformed by newer generation models in consistency and accuracy.
• The LLM-RI framework successfully identified novel risks not captured by traditional keyword-based methods.
• Prompt engineering was critical for ensuring structured outputs and accurate likelihood/impact assessments.
• The LARD-SC framework demonstrated scalability, capable of processing hundreds of articles daily with minimal manual supervision.
• Expert validation confirmed that LLM-generated risk descriptions were contextually relevant and actionable for risk managers.
• Graph visualization in Neo4j enhanced interpretability by linking risks to specific suppliers and news sources.

Data & experiments

• Dataset: 120 curated real-world news articles related to Apple’s Tier 1 suppliers.
• Models Evaluated: GPT-3.5 Turbo, GPT-4o, GPT-4o Mini, Claude 3.5 Sonnet, and Claude 3.5 Haiku.
• Framework: LARD-SC, comprising DCV-RA, LLM-RI, and LLM-RC sub-frameworks.
• Metrics: Risk Validation Rate (RVR), Potential Risk Rate (PRR), False Identification Rate (FIR), and Relative Performance Index (RPI).
• Evaluation: Expert-reviewed validation of LLM outputs against ground truth risk events.
• Taxonomy: Cambridge Taxonomy of Business Risks (CTBR) used for standardized risk classification.

Limitations & caveats

• Potential for LLM hallucinations and generation of unverifiable information.
• Dependency on the quality of prompt engineering and system instructions.
• Need for human-in-the-loop validation to ensure accuracy and mitigate bias.
• Resource-intensive nature of running large-scale LLMs in real-time.
• Limited scope of the evaluation dataset (only Apple's Tier 1 suppliers).
• Ethical and regulatory considerations regarding data privacy and accountability in AI-driven risk management.

Curator notes

Discovery status: specific_domain_focus_promoted. Source links: large language models supply chain risk finance; supply chain risk large language model annual reports; Supply Chain Risk Measurement and Resilience Insights from Large Language Models; LARD-SC large language model supply chain risk identification