Event Identification for Supply Chain Risk Management Through News Analysis by Using Large Language Models

In one sentence

Summary

This paper addresses the challenge of proactive supply chain risk management by identifying Contributing Events (CEs) that precede major disruptions. Traditional event extraction methods require extensive annotated training data, which is impractical for rare or novel risks. The authors propose the LUEI framework, part of the broader CERIA system, to autonomously detect CEs in real-time news feeds. The core problem is determining CE occurrences without prior labeled examples, leveraging the semantic understanding of Large Language Models to bridge the gap between event names and diverse news terminology. The LUEI framework operates in three stages: Seed Collection, News Crawling, and Event Detection. First, it generates 'seed phrases' for a CE by combining WordNet synonyms with semantic pruning using SBERT embeddings to filter irrelevant terms. It then expands this set by analyzing three years of historical news via KeyBERT. Second, it crawls Google News using these seed phrases. Finally, the Event Detector module uses SBERT to compute cosine similarity between news articles and the CE name, labeling articles as relevant if the score exceeds a threshold (0.5). The system was evaluated on four specific CEs leading to delivery delays, such as 'Airport Staff Shortage' and 'Construction Workers Hold Strike'. Experimental results demonstrate LUEI's superiority over single-keyword searches, WordNet-only expansions, and commercial tools like EventRegistry and Aylien. LUEI achieved an accuracy of 98.31% and an F1 score of 90.90% for 'Increase in COVID Cases', compared to 8.72% and 16.04% for single-keyword search, respectively. The framework effectively filters noise, returning highly relevant articles with minimal false positives. While the paper focuses on supply chain risk, the methodology offers a generalizable approach for unsupervised event detection in financial news, particularly for identifying precursor signals to market-moving events without relying on historical training data.

Research problem

The paper addresses the difficulty of proactively identifying Contributing Events (CEs) that lead to supply chain risk events, such as delivery delays. Existing AI-based event extraction methods require large amounts of annotated training data, which is unavailable for novel or rare risks. The research question is how to determine the occurrence of these CEs in real-world news articles without any training data, relying instead on the semantic capabilities of LLMs to generalize from event names.

Core contributions

• Proposes LUEI, a lightweight unsupervised framework for event identification that requires no training data.
• Introduces a novel seed phrase generation method combining WordNet synonyms with SBERT-based semantic pruning.
• Demonstrates a method to expand seed phrases using historical news analysis and KeyBERT.
• Validates the framework against manual expert labeling and commercial event detection tools.
• Shows significant performance improvements in precision, recall, and accuracy over baseline keyword searches.

Method

• Seed Collector: Uses WordNet to generate synonyms, then SBERT to prune semantically distant phrases (threshold 0.8).
• Seed Expansion: Crawls 3 years of Google News using pruned seeds, extracts keywords via KeyBERT, and adds high-similarity terms.
• News Crawler: Uses pygooglenews API to fetch articles matching seed phrases.
• Event Detector: Converts news articles and CE names to vectors using SBERT, calculates cosine similarity.
• Classification: Labels articles as relevant (1) if similarity > 0.5, else irrelevant (0).
• Probability Assignment: Integrates CE probabilities into a Bayesian Network (part of CERIA) to assess risk event likelihood.

Results & findings

• LUEI achieved 98.31% accuracy and 90.90% F1 score for 'Increase in COVID Cases', vastly outperforming single-keyword search (8.72% accuracy, 16.04% F1).
• For 'Airport Staff Shortage', LUEI reached 92.31% accuracy and 88.0% F1, compared to 19.05% and 32% for single-keyword search.
• Commercial tools EventRegistry and Aylien generated significant noise, with Aylien returning 839 articles for one event, most of which were irrelevant.
• LUEI returned only 71 articles for the same event, with high precision (93.75%).
• The semantic similarity threshold of 0.5 was empirically determined to balance relevance and recall.
• WordNet alone produced many semantically incorrect synonyms, necessitating SBERT pruning.

Data & experiments

• Evaluated on four specific Contributing Events: 'Increase in COVID Cases', 'AFL Matches Happening', 'Airport Staff Shortage', and 'Construction Workers Hold Strike'.
• Used Google News as the primary data source for crawling and evaluation.
• Compared LUEI against single-keyword search, keyword+WordNet search, EventRegistry, and Aylien.
• Used expert labeling to determine ground truth relevance of news articles.
• Metrics included accuracy, precision, recall, and F1 score.
• Historical news data from the past 3 years used for seed phrase expansion.

Limitations & caveats

• Relies on Google News, which may have coverage biases or access limitations.
• SBERT model choice and similarity thresholds are fixed; may not generalize perfectly to all domains.
• Evaluation is limited to four specific events; broader validation is needed.
• Does not address the causal relationship extraction part of CERIA, only the event detection.
• Potential for false positives if news articles use terms in unrelated contexts despite high semantic similarity.

Curator notes

Discovery status: specific_domain_focus_promoted. Source links: large language models supply chain risk finance; supply chain risk large language model annual reports; Supply Chain Risk Measurement and Resilience Insights from Large Language Models; LARD-SC large language model supply chain risk identification