The List / Reports, Filings, Accounting, and Risk / paper
FinRpt: Financial Report Understanding and Generation Benchmark
Abstract
While LLMs have shown great success in financial tasks like stock prediction and question answering, their application in fully automating Equity Research Report generation remains uncharted territory. In this paper, we formulate the Equity Research Report (ERR) Generation task for the first time. To address the data scarcity and the evaluation metrics absence, we present an open-source evaluation benchmark for ERR generation - FinRpt. We frame a Dataset Construction Pipeline that integrates 7 financial data types and produces a high-quality ERR dataset automatically, which could be used for model training and evaluation. We also introduce a comprehensive evaluation system including 11 metrics to assess the generated ERRs. Moreover, we propose a multi-agent framework specifically tailored to address this task, named FinRpt-Gen, and train several LLM-based agents on the proposed datasets using Supervised Fine-Tuning and Reinforcement Learning. Experimental results indicate the data quality and metrics effectiveness of the benchmark FinRpt and the strong performance of FinRpt-Gen, showcasing their potential to drive innovation in the ERR generation field. All code and datasets are publicly available.
In one sentence
Summary
The paper addresses the lack of automated Equity Research Report (ERR) generation by formally defining the task and creating the FinRpt benchmark. It tackles data scarcity and evaluation gaps by constructing a dataset of 6,825 high-quality ERRs from 800 Chinese stocks, integrating seven data types including financial statements, news, and market indices. The authors propose a comprehensive evaluation system with 11 metrics, covering basic text similarity and specialized financial professionalism aspects like numeric accuracy and risk analysis, filling a critical void in existing financial NLP benchmarks. Methodologically, the authors develop FinRpt-Gen, a multi-agent framework decomposing report generation into nine specialized agents for information extraction, analysis, and prediction. The pipeline involves Supervised Fine-Tuning (SFT) using LoRA on four core agents and Reinforcement Learning (RL) via the DAPO algorithm to optimize the prediction agent’s alignment with investment objectives. Experiments compare this framework against standalone LLMs and other baselines, utilizing both automated metrics and LLM-based pairwise evaluations to assess semantic quality and financial logic across the generated reports. Results show that FinRpt-Gen with SFT and RL significantly outperforms single LLMs, achieving a 55% accuracy in recommendation ratings and superior scores in financial numeric and news analysis metrics compared to GPT-4o. The dataset quality is validated through human evaluation, showing high consistency with expert-written reports. However, the study is limited to the Chinese market (CSI800) and a short time window, raising questions about generalizability to other markets or longer horizons. The work provides a foundational tool for automating analyst workflows but requires further validation on cross-market applicability and real-time trading integration.
Research problem
The paper addresses the uncharted territory of fully automating Equity Research Report (ERR) generation, a complex task requiring deep financial knowledge and structured analysis. It highlights two main gaps: the scarcity of high-quality training data for ERRs and the absence of comprehensive evaluation metrics that assess financial professionalism beyond standard NLP scores. The goal is to enable LLMs to replicate the workflow of professional analysts by integrating diverse financial data into coherent, actionable reports.
Core contributions
- Formally defines the Equity Research Report (ERR) generation task for the first time.
- Introduces FinRpt, an open-source benchmark with a dataset of 6,825 ERRs and 11 specialized evaluation metrics.
- Proposes FinRpt-Gen, a multi-agent framework with nine specialized agents for structured report generation.
- Develops a dataset construction pipeline integrating seven financial data types with automated quality enhancement.
- Demonstrates the effectiveness of SFT and RL (DAPO) in aligning agents with financial analysis and prediction goals.
Method
- Data Collection: Integrates 7 data types (financials, news, announcements, etc.) from sources like Yahoo Finance and Eastmoney.
- Dataset Construction: Automated pipeline with filtering, multi-agent generation, and expert-correction modules.
- FinRpt-Gen Architecture: 9 agents divided into Information Extraction, Analysis, and Prediction modules.
- Supervised Fine-Tuning: LoRA-based SFT on Finance, News, Status, and Prediction agents using FinRpt data.
- Reinforcement Learning: DAPO algorithm with a reward function balancing recommendation accuracy and ROUGE scores.
- Evaluation: Pairwise LLM judging with position bias elimination and human expert validation.
Results & findings
- FinRpt-Gen with SFT and RL achieved 55% accuracy in recommendation ratings, outperforming GPT-4o's 48%.
- The framework significantly improved ROUGE-L (49.06) and BERTScore (82.43) compared to single LLM baselines.
- LLM evaluations showed the trained model excelled in Financial Numeric (FN) and News analysis metrics.
- Ablation studies confirmed the necessity of specialized extraction and analysis agents for high performance.
- Human evaluation showed high agreement (Kappa ~0.86) between FinRpt reports and expert-written reports.
- The dataset construction pipeline successfully automated the creation of high-quality training data.
Data & experiments
- Dataset: 6,825 ERRs for 800 stocks in the CSI800 Index, covering September to November 2024.
- Data Sources: Company info, financial indicators, announcements, news, stock prices, and market indices.
- Evaluation: 11 metrics including Completion Rate, Accuracy, ROUGE-L, BERTScore, and LLM-based financial metrics.
- Baselines: Single LLMs (GPT-4o, Llama, Qwen), FinRpt-Gen with various LLMs, and fine-tuned variants.
- Human Evaluation: Senior analysts rated 30 FinRpt reports against 30 expert reports for quality and consistency.
Limitations & caveats
- Dataset is limited to the Chinese market (CSI800), limiting generalizability to other regions.
- Time window is short (Sept-Nov 2024), potentially missing long-term market dynamics.
- Evaluation relies heavily on LLM judges, which may have inherent biases or hallucinations.
- The 'Buy/Sell' rating is a binary simplification of complex investment advice.
- Real-time data integration and latency are not fully addressed in the current framework.
- Human evaluation sample size (30 reports) is small for robust statistical conclusions.
Curator notes
Directly relevant to industry and equity research report generation.