SHIELD: LLM-Driven Schema Induction for Predictive Analytics in EV Battery Supply Chain Disruptions

In one sentence

Summary

The paper addresses the critical need for predictive analytics in the electric vehicle (EV) battery supply chain, where geographic and economic concentrations of critical minerals like lithium and cobalt create vulnerability to geopolitical and natural disruptions. Traditional rule-based or black-box ML methods often lack the interpretability and adaptability required for proactive risk management. SHIELD proposes a two-stage framework that combines Large Language Models (LLMs) with domain expertise to construct a hierarchical knowledge schema and predict disruptions from news sources. This approach aims to mitigate LLM hallucinations while providing actionable, interpretable insights for supply chain resilience. The methodology involves two main stages: schema learning and disruption analysis. In schema learning, LLMs (GPT-4o, Llama3) extract hierarchical structures from 239 academic and industry sources, refined by human experts into a unified schema library. For disruption analysis, the system processes 12,070 news paragraphs using fine-tuned RoBERTa for event extraction, multi-dimensional similarity matching to link events to the schema, and Graph Convolutional Networks (GCNs) with logical constraints for prediction. The pipeline includes coreference resolution and logical consistency checks to ensure robustness. Experiments evaluate performance on a dataset spanning 2022-2023, comparing SHIELD against ablation versions and direct LLM prompting (GPT-4o). SHIELD achieves an F-score of 0.732 in disruption prediction, significantly outperforming GPT-4o (0.624) and GCN-only baselines (0.685). The system successfully predicted real-world events, such as material shortages following the Inflation Reduction Act and lithium supply risks due to Australia-China tensions. Key contributions include an interactive schema curation system, a novel schema-news dataset, and a hybrid architecture that balances predictive accuracy with interpretability. Limitations include challenges in schema integration and the need for continuous expert feedback to maintain relevance as supply chain dynamics evolve.

Research problem

The paper addresses the vulnerability of the EV battery supply chain to disruptions caused by the concentration of critical minerals. It highlights the limitations of traditional risk management approaches, which often lack predictive accuracy or interpretability, and the tendency of pure LLM methods to hallucinate or miss complex domain-specific relationships. The goal is to develop a system that combines the knowledge extraction capabilities of LLMs with the structural reasoning of graph models to provide accurate, interpretable, and proactive disruption predictions.

Core contributions

• A two-stage framework (SHIELD) integrating LLM-driven schema induction with GCN-based disruption prediction.
• An interactive schema curation system that combines LLM extraction with human expert feedback to build a comprehensive knowledge library.
• A novel hybrid pipeline using fine-tuned RoBERTa for event extraction, multi-dimensional similarity matching, and GCNs with logical constraints.
• A new dataset of 12,070 news paragraphs from 365 sources (2022-2023) focused on the EV battery supply chain.
• Demonstration of superior predictive performance over baseline GCNs and direct LLM prompting methods like GPT-4o.

Method

• Schema Learning: LLMs (GPT-4o, Llama3) extract hierarchical structures from text, refined by human experts.
• Event Extraction: Fine-tuned RoBERTa detects events; SpaCy handles tokenization and NER.
• Event Matching: Multi-dimensional similarity (semantic via BERT, structural via Jaccard) links events to schema.
• Disruption Prediction: GCNs model event relationships; logical constraints refine predictions.
• Coreference Resolution: AllenNLP-based model links entities across event mentions to improve context.
• Interactive Curation: Human-in-the-loop system allows experts to edit and validate generated schemas.

Results & findings

• SHIELD achieves an F-score of 0.732 in disruption prediction, outperforming GPT-4o (0.624) and GCN-only (0.685).
• GPT-4o outperforms Llama3 models in schema learning, achieving an F-score of 0.652 for individual schemas.
• Event extraction and matching maintain stable performance across quarters, with F-scores ranging from 0.671 to 0.700.
• Logical constraints and coreference resolution provide incremental improvements, raising F-score from 0.685 to 0.732.
• The system successfully predicted a semiconductor shortage three weeks prior to widespread reporting.
• Subjective expert evaluation rated GPT-4o highest for consistency (4.5), accuracy (4.3), and completeness (4.6).

Data & experiments

• Schema Learning Dataset: 239 sources including 200 academic papers, 22 industry reports, and 17 Wikipedia entries.
• Disruption News Dataset: 12,070 paragraphs from 365 sources (247 news articles, 118 enterprise reports) from Jan 2022 to Dec 2023.
• Evaluation Metrics: Precision, Recall, F-score for event extraction and disruption prediction; subjective expert ratings for schema quality.
• Baselines: Ablation studies of GCN components, comparison with GPT-4o and Llama3 models for both schema learning and prediction.
• Case Studies: Qualitative analysis of predictions for the Inflation Reduction Act, Australia-China lithium tensions, and DRC cobalt strikes.

Limitations & caveats

• Schema integration remains challenging, with lower performance in integrated library generation compared to individual schemas.
• The system relies on continuous expert feedback, which may limit scalability.
• Real-time adaptability to new, unseen disruption patterns is an area for future improvement.
• Performance gains from additional components (constraints, coreference) are incremental, suggesting room for optimization.
• The dataset is limited to EV battery supply chain, potentially limiting generalizability to other sectors.

Curator notes

Discovery status: specific_domain_focus_promoted. Source links: predicting repayment risk supply chain finance large language models