FinVis-GPT: A Multimodal Large Language Model for Financial Chart Analysis

In one sentence

Summary

The paper addresses the gap in multimodal large language models (LMMs) for financial chart analysis, noting that general-purpose models lack domain-specific visual-textual alignment. The authors propose FinVis-GPT, an LMM built upon the LLaVA architecture, designed to interpret financial visualizations such as candlestick and line charts. The core challenge lies in teaching the model to recognize technical patterns and correlate them with financial terminology, a task where off-the-shelf models often hallucinate or misinterpret visual elements like trend lines and volume bars. To train the model, the authors constructed a two-stage dataset using historical daily stock price data from Chinese A-shares (2006-2023). The pre-training alignment dataset consists of chart images paired with textual descriptions generated by ChatGPT, while the instruction tuning dataset contains 200,000 entries with diverse Q&A pairs focused on trend analysis and prediction. The model was fine-tuned using standard LLaVA training protocols on NVIDIA A100 GPUs. Evaluation involved case studies comparing FinVis-GPT against baselines like LLaVA, MiniGPT-4, and mPLUG-Owl across three tasks: description generation, question answering, and trend prediction. Results indicate that FinVis-GPT significantly outperforms baseline models in accuracy and relevance. While baselines frequently hallucinated unrelated content or misidentified chart components, FinVis-GPT provided concise, financially accurate interpretations. The model successfully identified trends and generated plausible future predictions based on visual inputs. However, the evaluation is limited to qualitative case studies rather than quantitative benchmarks, and the dataset is restricted to Chinese A-share markets, potentially limiting generalizability to other asset classes or global markets. The work highlights the necessity of domain-specific data curation for financial LMMs.

Research problem

General-purpose multimodal large language models struggle with financial chart analysis due to a lack of domain-specific visual-textual alignment. They often hallucinate or misinterpret technical patterns in stock charts. This paper addresses the need for a specialized LMM that can accurately interpret financial visualizations, generate professional descriptions, and answer domain-specific questions.

Core contributions

• Proposes FinVis-GPT, a multimodal LLM fine-tuned specifically for financial chart analysis.
• Constructs a novel pre-training alignment dataset of Chinese A-share stock charts with textual descriptions.
• Creates a large-scale instruction tuning dataset (200K entries) for financial Q&A and trend prediction.
• Demonstrates superior performance over general-purpose LMMs (LLaVA, MiniGPT-4, mPLUG-Owl) in case studies.
• Releases the dataset for public use to facilitate further research in financial multimodal AI.

Method

• Base Model: LLaVA (Large Language and Vision Assistant).
• Pre-training Alignment: Aligns visual encoder with LLM using chart-description pairs.
• Instruction Tuning: Fine-tunes the model on 200K Q&A pairs for specific financial tasks.
• Data Generation: Uses ChatGPT to generate descriptions and Q&A pairs from raw stock data.
• Visualization: Generates charts using mplfinance library with random technical indicators.
• Training Hardware: 8 NVIDIA Tesla A100 GPUs.
• Optimization: Uses dropout and weight decay for regularization; early stopping for validation.

Results & findings

• FinVis-GPT outperforms LLaVA, MiniGPT-4, and mPLUG-Owl in financial chart tasks.
• Baselines frequently hallucinated or misidentified chart components (e.g., trend lines).
• FinVis-GPT accurately identified stock trends and generated concise, professional descriptions.
• The model demonstrated improved capability in predicting future market trends from visual inputs.
• Instruction tuning significantly enhanced the model's ability to follow specific financial queries.
• The generated dataset includes diverse chart styles and technical indicators to improve robustness.

Data & experiments

• Dataset: Historical daily stock price data from Chinese A-shares (2006-2023).
• Chart Types: 80% candlestick charts, 20% line charts, with added moving averages and volume bars.
• Pre-training Data: Chart images paired with ChatGPT-generated descriptions.
• Instruction Tuning Data: 200,000 image-instruction-answer triplets for Q&A and prediction.
• Baselines: LLaVA, MiniGPT-4, and mPLUG-Owl.
• Evaluation: Qualitative case studies on description generation, question answering, and trend prediction.

Limitations & caveats

• Evaluation is limited to qualitative case studies, lacking quantitative metrics.
• Dataset is restricted to Chinese A-share markets, limiting generalizability.
• Data generation relies on ChatGPT, which may introduce biases or errors.
• No mention of real-time deployment or latency considerations.
• Limited to candlestick and line charts; other chart types not explored.
• Potential overfitting to specific chart styles due to synthetic data generation.

Curator notes

Discovery status: round2_promoted. Source links: A Comprehensive Review of Generative AI in Finance; Advancing Financial Engineering with Foundation Models: Progress, Applications, and Challenges; FinTral: A Family of GPT-4 Level Multimodal Financial Large Language Models; No Language is an Island: Unifying Chinese and English in Financial Large Language Models, Instruction Data, and Benchmarks