Modal-adaptive Knowledge-enhanced Graph-based Financial Prediction from Monetary Policy Conference Calls with LLM

In one sentence

Summary

The paper addresses the challenge of financial prediction using Monetary Policy Conference (MPC) calls by identifying three key limitations in prior multimodal approaches: the neglect of external financial knowledge, the equal weighting of disparate modalities despite varying informational value, and the independent treatment of correlated financial assets. The authors aim to create a more robust prediction system that leverages the semantic richness of text, the non-verbal cues in video and audio, and the contextual depth of domain-specific knowledge graphs to improve forecasting accuracy for asset price movements and volatility. The proposed method, MANAGER, utilizes ChatGLM2 as its backbone, enhanced by a knowledge-enhanced cross-modal graph. It extracts text features via ChatGLM2, video features via BEiT-3, and audio features via HuBERT. External knowledge is retrieved from the dynamic FinDKG to enrich the textual context. These modalities are fused using Graph Convolutional Networks (GCNs) that model intra-modal and inter-modal semantic relations, allowing the model to adaptively weigh the contribution of each modality. The system is evaluated on the Monopoly dataset, which contains multimodal data from MPC calls and labels for six financial assets, predicting price movement (F1 score) and volatility (MSE) over 1, 3, 7, and 15-day horizons. Experimental results demonstrate that MANAGER consistently outperforms state-of-the-art baselines, including text-only models and other multimodal architectures, across all metrics and asset classes. Ablation studies confirm the significant contribution of external knowledge and the graph-based fusion mechanism, while also revealing that text remains the dominant modality, with video and audio providing marginal but positive gains. The study highlights the importance of incorporating domain knowledge and adaptive fusion in financial LLM applications, though it notes that improper handling of non-verbal cues can sometimes degrade performance compared to text-only methods.

Research problem

The paper tackles the problem of predicting financial asset price movements and volatility using Multimodal Monetary Policy Conference (MPC) calls. It specifically addresses the limitations of existing multimodal models that ignore external financial knowledge, treat all modalities (text, video, audio) with equal importance despite varying relevance, and fail to capture the inherent correlations among different financial assets.

Core contributions

• Proposes MANAGER, a novel framework integrating external financial knowledge with multimodal inputs (text, video, audio) for financial prediction.
• Introduces a knowledge-enhanced cross-modal graph using GCNs to adaptively fuse modalities based on semantic relations.
• Leverages the dynamic FinDKG to inject relevant financial entities and relations into the context, preventing data leakage.
• Demonstrates the first application of ChatGLM2 to multimodal MPC call prediction, showing superior performance over baselines.
• Provides extensive ablation studies validating the individual contributions of text, knowledge, video, audio, and graph structure.

Method

• Text encoding using ChatGLM2 encoder, concatenated with external knowledge from FinDKG.
• Video feature extraction using BEiT-3, aggregating frame-level tokens.
• Audio feature extraction using HuBERT, averaging utterance-level representations.
• Construction of a knowledge-enhanced cross-modal graph with nodes for text, knowledge, video, and audio.
• Definition of intra-modal (token-token, video-video) and inter-modal (token-video, token-audio) semantic edges.
• Feature aggregation using L-layer Graph Convolutional Networks (GCNs) to capture cross-modal context.
• Task-specific instruction tuning of ChatGLM2 for price movement (classification) and volatility (regression) prediction.
• Optimization using Binary Cross-Entropy for movement and Mean Squared Error for volatility.

Results & findings

• MANAGER achieves the highest F1 scores for price movement prediction across all assets and time horizons compared to all baselines.
• MANAGER yields the lowest MSE for volatility prediction, significantly outperforming the second-best model in most cases.
• Ablation shows that removing text causes the largest performance drop, confirming text as the primary information source.
• Incorporating external knowledge from FinDKG improves performance, validating the value of domain-specific context.
• Video and audio modalities provide marginal but consistent improvements, though improper fusion can sometimes hurt performance.
• The graph-based fusion mechanism outperforms full-connection graphs, highlighting the importance of semantic relation modeling.

Data & experiments

• Uses the Monopoly dataset, comprising 24,180 samples of MPC calls with text, video, and audio from 2009-2022.
• Predicts price movement (classification) and volatility (regression) for six assets: Small/Large Stock Indices, Gold, Currency, and 3/10-year bond yields.
• Evaluates performance using F1 score for movement and MSE for volatility over 1, 3, 7, and 15-day horizons.
• Compares against text-only baselines (ARIMA, SVM, LSTM) and multimodal baselines (MLP, LSTM, MMIM, MDRM, HTML, MULT, MPCNet).
• Conducts ablation studies removing text, knowledge, video, audio, and graph structure to assess component importance.

Limitations & caveats

• The model relies on the Monopoly dataset, which may not generalize to other types of financial calls or events.
• Video and audio contributions are marginal; the model is heavily dependent on text, limiting the value of multimodal complexity.
• The use of ChatGLM2, a Chinese-centric LLM, may have limitations in understanding nuanced English financial terminology compared to English-native models.
• The study does not explore real-time inference latency or computational costs of the multimodal graph approach.
• External knowledge is retrieved from FinDKG, which may have gaps or inaccuracies in dynamic financial contexts.

Curator notes

Discovery status: specific_domain_round3_promoted. Source links: FinDKG: Dynamic Knowledge Graphs with Large Language Models for Detecting Global Trends in Financial Markets