The List / RAG, Search, and Knowledge Systems / paper
AlphaFin: Benchmarking Financial Analysis with Retrieval-Augmented Stock-Chain Framework
Abstract
The task of financial analysis primarily encompasses two key areas: stock trend prediction and the corresponding financial question answering. Currently, machine learning and deep learning algorithms (ML&DL) have been widely applied for stock trend predictions, leading to significant progress. However, these methods fail to provide reasons for predictions, lacking interpretability and reasoning processes. Also, they can not integrate textual information such as financial news or reports. Meanwhile, large language models (LLM) have remarkable textual understanding and generation ability. But due to the scarcity of financial training datasets and limited integration with real-time knowledge, LLM still suffer from hallucinations and unable to keep up with the latest information. To tackle these challenges, we first release AlphaFin datasets, combining traditional research datasets, real-time financial data, and handwritten chain-of-thought (CoT) data. It has positive impact on training LLM for completing financial analysis. We then use AlphaFin datasets to benchmark a state-of-the-art method, called Stock-Chain, for effectively tackling the financial analysis task, which integrates retrieval-augmented generation (RAG) techniques. Extensive experiments are conducted to demonstrate the effectiveness of our framework on financial analysis.
In one sentence
Summary
The paper addresses the limitations of traditional machine learning models, which lack interpretability and textual integration, and general Large Language Models, which suffer from hallucinations and outdated knowledge in financial contexts. The authors formalize financial analysis into stock trend prediction and financial question answering, aiming to bridge the gap between quantitative forecasting and qualitative reasoning. They introduce AlphaFin, a dataset combining traditional research data, real-time market data, and handwritten chain-of-thought reports, to train models that can provide reasoned, up-to-date investment insights. The proposed Stock-Chain framework operates in two stages. Stage one fine-tunes a StockGPT model using Low-Rank Adaptation on AlphaFin’s financial reports and chain-of-thought data to predict monthly stock trends (up/down) based on retrieved documents. Stage two enhances this base model with Retrieval-Augmented Generation, utilizing a vector database of financial news and reports to answer user queries with real-time context. The system employs coarse-grained summarization and fine-grained entity-level dialogue generation for knowledge extraction, ensuring the model accesses relevant, current information during inference. Experiments on Chinese capital market data show Stock-Chain achieves a 30.8% annualized rate of return and 55.7% prediction accuracy, significantly outperforming baselines like FinGPT, ChatGPT, and LSTM models. The framework also demonstrates superior performance in financial question answering, as validated by human and GPT-4 preference evaluations. However, the study is limited by its focus on the Chinese market, the potential for overfitting to specific report structures, and the inherent risks of automated trading strategies, noting that while returns are high, real-world deployment requires careful risk management and continuous data updates.
Research problem
The paper tackles the dual challenge of stock trend prediction and financial question answering. Traditional ML/DL methods lack interpretability and cannot process textual data, while general LLMs suffer from hallucinations and lack real-time financial knowledge. The goal is to create an interpretable, accurate, and up-to-date financial analysis system.
Core contributions
- Formalization of financial analysis into stock trend prediction and financial Q&A tasks.
- Release of the AlphaFin dataset, integrating traditional research data, real-time market data, and handwritten chain-of-thought reports.
- Proposal of the Stock-Chain framework, a two-stage RAG-integrated system for stock prediction and Q&A.
- Fine-tuning of StockGPT using LoRA on financial reports and CoT data to enhance reasoning and prediction capabilities.
- Extensive benchmarking showing superior performance over ML/DL baselines and other FinLLMs in both accuracy and returns.
Method
- Data preprocessing involves generating Q&A pairs from sequential stock data using ChatGPT.
- Stage 1 uses LoRA fine-tuning on financial reports and handwritten CoT data for trend prediction.
- Knowledge extraction employs coarse-grained summarization and fine-grained entity-level dialogue generation.
- Vector database construction uses BGE embeddings for efficient retrieval of financial documents.
- Stage 2 integrates RAG to retrieve relevant news and reports for answering user queries.
- Response generation uses a prompt template combining retrieved knowledge, history, and user query.
- Portfolio strategy uses capitalization-weighted holding of predicted 'up' stocks for monthly rolling returns.
Results & findings
- Stock-Chain achieves a 30.8% Annualized Rate of Return (ARR), significantly higher than FinGPT (17.5%) and ChatGPT (14.3%).
- Prediction accuracy reaches 55.7%, outperforming all baseline models including LSTM and XGBoost.
- The framework reduces invalid answer ratios to 25.9% through fine-tuning with raw and CoT data.
- Human preference evaluation shows Stock-Chain wins over 60% of comparisons against FinGPT and ChatGLM2.
- GPT-4 evaluation confirms Stock-Chain's superiority with a 73% win rate against ChatGLM2.
- Ablation studies confirm that combining raw data and CoT data yields the best performance.
- RAG integration effectively mitigates hallucinations and provides real-time context for financial analysis.
Data & experiments
- AlphaFin dataset includes research datasets (FPB, FinQA), StockQA, financial news (CCTV, Wall Street CN), and financial reports (DataYes).
- Experiments conducted on Chinese capital market data from January 2020 to July 2023.
- Baselines include major indices (SSE50, CSI 300), ML/DL models (LSTM, XGBoost), and FinLLMs (FinGPT, FinMA).
- Metrics include Annualized Rate of Return (ARR), Accuracy, Sharpe Ratio, and Maximum Drawdown.
- Preference evaluation using human annotators and GPT-4 to assess the quality of financial Q&A responses.
Limitations & caveats
- The study focuses primarily on the Chinese capital market, limiting generalizability to other regions.
- The reliance on financial reports and news may introduce biases present in the source data.
- The rolling monthly portfolio strategy may not reflect transaction costs or liquidity constraints in real trading.
- The model's performance is sensitive to the quality and timeliness of the retrieved documents.
- Potential overfitting to specific report structures or market conditions during the training period.
Curator notes
Discovery status: round1_promoted_seed_for_round2. Source links: A Comprehensive Review of Generative AI in Finance; A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges; AI in Investment Analysis: Large Language Models for Equity Stock Ratings; Advancing Financial Engineering with Foundation Models: Progress, Applications, and Challenges; BloombergGPT: A Large Language Model for Finance; CN-Buzz2Portfolio: A Chinese-Market Dataset and Benchmark for LLM-Based Macro and Sector Asset Allocation from Daily Trending Fin