ECC Analyzer: Extract Trading Signal from Earnings Conference Calls using Large Language Model for Stock Performance Prediction

In one sentence

Summary

Predicting stock volatility using unstructured earnings conference call (ECC) data remains challenging due to the complexity of extracting nuanced signals from multimodal inputs. Existing multimodal models often treat text and audio equally, missing critical contextual details. This paper addresses the problem by proposing ECC Analyzer, a framework that integrates large language models to perform hierarchical information extraction. The goal is to distill richer, more predictive features from ECCs to enhance volatility forecasting accuracy, moving beyond simple feature concatenation to a deeper semantic understanding of executive communications and vocal cues. The method employs a multimodal pipeline using Wav2vec2 for audio embeddings and SimCSE for text embeddings, processed through Multi-Head Self-Attention. Crucially, it uses LLMs for hierarchical summarization and Retrieval-Augmented Generation (RAG) guided by a financial expert-designed 'Question Bank' to extract fine-grained focus sentences. These features are fused via additive interactions and trained on a temporal split of the S&P 500 ECC dataset (2017) with 572 instances. Experiments compare against GARCH, LSTM, and SOTA multimodal models like HTML and AMA-LSTM, evaluating performance using Mean Squared Error (MSE) across 3, 7, 15, and 30-day volatility horizons. Results show ECC Analyzer reduces average MSE by 27.7% compared to the best baseline, with significant improvements in short-term (3-day, 7-day) volatility prediction. The model outperforms direct LLM predictions, which proved ineffective, highlighting the necessity of structured feature extraction. Ablation studies confirm that fine-grained RAG-extracted sentences contribute most to performance gains. Limitations include the model's reliance on specific financial questions and potential overfitting to short-term signals, with medium-term performance comparable but not superior to specialized baselines like AMA-LSTM. The study underscores the value of LLMs as feature extractors rather than direct predictors in financial time-series tasks.

Research problem

The paper addresses the challenge of accurately predicting stock volatility using unstructured multimodal data from earnings conference calls (ECCs). Traditional multimodal models often fail to capture detailed, complex information by treating all text and audio segments with equal importance, potentially missing critical contextual signals. The research aims to leverage Large Language Models (LLMs) to extract richer, fine-grained textual and audio features to improve prediction performance.

Core contributions

• Proposes ECC Analyzer, a novel framework integrating multimodal ECC data with LLM-based hierarchical information extraction.
• Introduces a fine-grained extraction strategy using Retrieval-Augmented Generation (RAG) guided by a financial expert-designed question bank.
• Demonstrates a 27.7% reduction in average Mean Squared Error (MSE) compared to state-of-the-art baselines.
• Shows that LLMs are effective as feature extractors for volatility prediction but ineffective as direct predictors.
• Validates the approach on a real-world S&P 500 dataset with temporal train-test splits.

Method

• Audio Encoding: Wav2vec2 embeddings processed by Multi-Head Self-Attention and average pooling.
• Text Encoding: SimCSE sentence embeddings processed by Multi-Head Self-Attention and average pooling.
• Hierarchical Summarization: LLM summarizes ECC chunks and then the entire document.
• RAG Extraction: Financial 'Question Bank' queries retrieve relevant chunks; LLM extracts focus sentences.
• Feature Fusion: Additive interaction of audio, text, summary, and focus sentence embeddings.
• Prediction: Fully connected layers predict volatility regression targets.

Results & findings

• ECC Analyzer achieves the lowest average MSE (0.314) among all compared models.
• Short-term volatility (3-day MSE 0.553, 7-day MSE 0.306) prediction is significantly improved over baselines.
• Direct GPT-4 Turbo prediction performed poorly (MSE 2.198), indicating LLMs need structured integration.
• Ablation study shows fine-grained RAG-extracted sentences provide the most significant performance boost.
• Long-term (30-day) performance is comparable to the SOTA HTML model (0.158 vs 0.158).
• Medium-term (15-day) performance is slightly lower than AMA-LSTM (0.237 vs 0.230).

Data & experiments

• Dataset: Publicly available S&P 500 ECC dataset (2017) with 572 aligned audio-transcript instances.
• Evaluation Metrics: Mean Squared Error (MSE) for 3, 7, 15, and 30-day volatility horizons.
• Baselines: GARCH, LSTM, MT-LSTM-ATT, HAN, MRDM, HTML, AMA-LSTM, and direct GPT-4 Turbo prediction.
• Experimental Design: Temporal 8:2 train-test split; ablation studies on feature components.
• Implementation: PyTorch, Wav2vec2, SimCSE, GPT-4 Turbo, Adam optimizer with grid search.

Limitations & caveats

• Model performance on medium-term volatility is slightly inferior to specialized baselines like AMA-LSTM.
• Reliance on a fixed 'Question Bank' may limit generalizability to other financial contexts.
• Direct LLM prediction is ineffective, suggesting high cost/latency for real-time trading without optimization.
• Dataset is limited to 2017 S&P 500 calls, potentially lacking recent market dynamics.
• Ablation study suggests filtering irrelevant information is critical, implying sensitivity to noise.

Curator notes

Discovery status: trading_agent_focus_promoted. Source links: FinMem: A Performance-Enhanced Large Language Model Trading Agent with Layered Memory and Character Design