The List / Reports, Filings, Accounting, and Risk / paper
Towards Competent AI for Fundamental Analysis in Finance: A Benchmark Dataset and Evaluation
Abstract
Generative AI, particularly large language models (LLMs), is beginning to transform the financial industry by automating tasks and helping to make sense of complex financial information. One especially promising use case is the automatic creation of fundamental analysis reports, which are essential for making informed investment decisions, evaluating credit risks, guiding corporate mergers, etc. While LLMs attempt to generate these reports from a single prompt, the risks of inaccuracy are significant. Poor analysis can lead to misguided investments, regulatory issues, and loss of trust. Existing financial benchmarks mainly evaluate how well LLMs answer financial questions but do not reflect performance in real-world tasks like generating financial analysis reports. In this paper, we propose FinAR-Bench, a solid benchmark dataset focusing on financial statement analysis, a core competence of fundamental analysis. To make the evaluation more precise and reliable, we break this task into three measurable steps: extracting key information, calculating financial indicators, and applying logical reasoning. This structured approach allows us to objectively assess how well LLMs perform each step of the process. Our findings offer a clear understanding of LLMs current strengths and limitations in fundamental analysis and provide a more practical way to benchmark their performance in real-world financial settings.
In one sentence
Summary
This paper addresses the gap in evaluating LLMs for real-world fundamental analysis tasks, specifically focusing on financial statement analysis, which is critical for investment decisions and credit risk assessment. The authors argue that existing benchmarks primarily test question-answering or general financial knowledge, failing to capture the multi-step, structured nature of generating accurate financial reports. To address this, they propose FinAR-Bench, a dataset designed to decompose financial statement analysis into three verifiable subtasks: extracting key financial items, calculating specific financial indicators, and applying logical reasoning to interpret trends. This structured approach allows for objective assessment of each component, providing a more granular view of LLM capabilities in a high-stakes financial context. The dataset comprises financial statement data from 100 companies listed on the Shanghai Stock Exchange for fiscal years 2022 and 2023, sourced from both structured XBRL formats and unstructured PDF annual reports. The authors evaluate 14 diverse LLMs, ranging from small open-source models to large proprietary ones like GPT-4o and GPT-o1. The evaluation protocol includes a novel RMS metric for assessing table extraction accuracy and a tournament-style LLM-as-a-judge framework for comparing the quality of logical reasoning outputs. Experiments analyze performance across different input formats (text vs. PDF), task sizes, and numeric tolerance thresholds, alongside ablation studies on knowledge augmentation via explicit formula prompts. Results indicate that large LLMs achieve near-perfect scores in information extraction but perform poorly in indicator computation, with recall rates often below 50% even for top models. However, performance in logical reasoning tasks is significantly higher, suggesting that LLMs can generate plausible financial narratives even when underlying numbers are inaccurate. The study finds that providing explicit calculation formulas improves performance for larger models but can confuse smaller ones. Case studies reveal that while models like GPT-o1 and DeepSeek-r1 lead in reasoning quality, they still exhibit logical contradictions and superficial insights. The paper concludes that while LLMs are competent at data retrieval, they lack the precision required for exact financial computation, highlighting the need for hybrid systems or further domain-specific fine-tuning for reliable automated fundamental analysis.
Research problem
The paper addresses the lack of reliable benchmarks for evaluating LLMs in fundamental financial analysis, specifically financial statement analysis. It highlights that current benchmarks focus on question answering rather than the multi-step process of generating analysis reports. The core problem is assessing whether LLMs can accurately extract data, compute indicators, and reason logically from financial statements, which is essential for investment and credit decisions.
Core contributions
- Proposes FinAR-Bench, a task-oriented benchmark dataset for evaluating LLMs in financial statement analysis.
- Decomposes financial statement analysis into three measurable subtasks: information extraction, indicator computation, and logical reasoning.
- Introduces a structured table evaluation protocol using a modified RMS metric for precise assessment of extraction and calculation.
- Develops a tournament-style LLM-as-a-judge framework for evaluating the quality of logical reasoning outputs.
- Provides a comprehensive evaluation of 14 LLMs across different sizes and architectures on both text and PDF inputs.
- Releases the dataset and source code to encourage future research in financial LLM evaluation.
Method
- Data collection from SSE website in XBRL and PDF formats.
- Task decomposition into extraction, computation, and reasoning.
- RMS metric calculation using Hungarian algorithm for table alignment.
- Tournament-style pairwise comparison for reasoning evaluation.
- LLM-as-a-judge using Doubao, Qwen-Max, and Gemini-2.5-Pro.
- Ablation studies on task size, numeric tolerance, and knowledge augmentation.
- PDF extraction using PyMuPDF for robust layout handling.
- Error analysis on recall degradation with increasing task complexity.
Results & findings
- Large LLMs achieve near-perfect information extraction (P/R > 98%).
- Indicator computation is a major bottleneck, with top models achieving <50% recall at strict tolerance.
- Logical reasoning performance is significantly higher than computation, often >60% recall.
- GPT-o1 and DeepSeek-r1 lead in reasoning quality in tournament evaluations.
- PDF input reduces performance compared to clean text, especially for layout-dependent tasks.
- Explicit formula prompts improve performance for large models but can degrade small model performance.
- Recall for computation increases dramatically with relaxed numeric tolerance (e.g., from 34% to 90% at 50% error tolerance).
- Case studies show models often produce plausible but logically flawed or superficial insights.
Data & experiments
- Dataset: 100 companies from the Shanghai Stock Exchange, fiscal years 2022-2023.
- Data formats: Structured XBRL and unstructured PDF annual reports.
- Tasks: Information extraction, indicator computation, and logical reasoning.
- Models: 14 LLMs including GPT-4o, GPT-o1, DeepSeek series, Llama, Mistral, and Qwen.
- Metrics: RMS Precision/Recall for tables, LLM-as-a-judge scores for reasoning.
- Ablations: Impact of task size, numeric tolerance, and knowledge augmentation.
Limitations & caveats
- Limited to 100 companies from the Shanghai Stock Exchange, potentially lacking global diversity.
- Evaluation focuses on financial statement analysis, not broader fundamental analysis including qualitative factors.
- LLM-as-a-judge may introduce bias or inconsistency in reasoning evaluation.
- Small models may struggle with long prompts, affecting fair comparison.
- Does not evaluate the full workflow of automated report generation, only intermediate steps.
- Budget constraints limited the number of proprietary models tested.
Curator notes
Discovery status: round1_promoted_seed_for_round2. Source links: A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges; BBT-Fin: Comprehensive Construction of Chinese Financial Domain Pre-trained Language Model, Corpus and Benchmark; BizBench: A Quantitative Reasoning Benchmark for Business and Finance; BloombergGPT: A Large Language Model for Finance; FLAME: Financial Large Language Model Evaluation System in Chinese; FinBen: A Holistic Financial Benchmark for Large Language Models; FinCon: A Synthesized LLM Multi-A