Fin-RATE: Financial Report Analytics and Tracking Evaluation for Large Language Models

In one sentence

Summary

The paper addresses the limitation of existing financial LLM benchmarks that focus on isolated fact extraction rather than the complex, multi-document reasoning required by professional analysts. It introduces Fin-RATE, a benchmark built on U.S. SEC filings that mirrors analyst workflows through three pathways: detail-oriented reasoning within single disclosures, cross-entity comparison under shared topics, and longitudinal tracking of firms across reporting periods. The dataset includes 7,500 high-quality QA pairs derived from 2,472 filings across 43 companies, ensuring coverage of diverse industries and filing types beyond standard 10-Ks. The authors evaluate 17 leading LLMs, including closed-source, open-source, and finance-specialized models, under both ground-truth context and retrieval-augmented generation (RAG) settings. They employ a fine-grained error taxonomy with 13 types across four categories to diagnose failures, using an LLM-as-Judge framework for evaluation. Experiments reveal substantial performance drops as task complexity increases, with accuracy falling by 18.60% and 14.35% when shifting from single-document to longitudinal and cross-entity analysis. The study also analyzes retrieval effectiveness, showing that hybrid retrieval with reranking improves evidence coverage but does not fully mitigate downstream reasoning errors. Key findings indicate that closed-source models maintain robustness but suffer from reasoning inconsistencies, while finance-tuned models collapse on cross-entity tasks due to hallucinations and entity confusion. Longitudinal tracking is hindered by temporal mismatches and trend hallucinations, as models treat yearly data as independent units rather than continuous sequences. The paper highlights that current benchmarks fail to capture these specific failure modes, such as comparative stance hallucination and intent misunderstanding. These limitations suggest that while LLMs are improving at single-document parsing, they lack the structural and temporal abstraction necessary for reliable multi-document financial analysis, posing risks for deployment in high-stakes investment research.

Research problem

Existing financial LLM benchmarks oversimplify SEC filings as flat, isolated texts, focusing on single-document fact extraction rather than the multi-document, cross-temporal, and cross-entity reasoning required in professional financial analysis. There is a lack of evaluation frameworks that can disentangle whether errors stem from retrieval failures, generation inaccuracies, or domain-specific reasoning mistakes, making it difficult to diagnose performance bottlenecks in complex analytical workflows.

Core contributions

• Introduction of Fin-RATE, the first benchmark modeling realistic financial analysis workflows by integrating information across documents, reporting periods, and corporate entities.
• Development of a domain-grounded Likert scoring scheme and a 13-type error taxonomy to quantify model degradation and identify specific failure modes in financial reasoning.
• Comprehensive evaluation of 17 LLMs across open-source, closed-source, and finance-specialized categories under both ground-truth and retrieval-augmented settings.
• Identification of distinct capability profiles, revealing that finance-specific models excel in numeric reasoning but fail under complex cross-entity reasoning demands.
• Provision of a hierarchical retrieval framework and detailed analysis of retrieval errors, demonstrating how evidence coverage impacts end-to-end performance.

Method

• Data collection from SEC EDGAR database, filtering by multi-year coverage and GICS sector diversity.
• Heading-aware chunking of filings into coherent analytical units based on SEC-standard item boundaries.
• Task-specific QA generation using adaptive prompt templates guided by filing types and section definitions.
• Multi-stage quality control involving automated generation-verification and manual review to ensure factual accuracy.
• Evaluation using LLM-as-Judge with weighted averaging of three independent judges for correctness and Likert scores.
• Retrieval experiments using BM25, dense vector, and hybrid methods with bge-reranker-v2-m3 for reranking.
• Error attribution using a hierarchical taxonomy to categorize failures into retrieval, generation, financial reasoning, and comprehension.

Results & findings

• Accuracy drops by 18.60% and 14.35% when shifting from single-document to longitudinal and cross-entity analysis, respectively.
• Closed-source models show the highest robustness, with correct rates decreasing only moderately from 39.69% in DR-QA to 33.28% in EC-QA.
• Finance-tuned models collapse on cross-entity tasks, with correct rates dropping from 23.91% in DR-QA to 1.27% in EC-QA.
• General open-source models exhibit uniform degradation, with failure rates spiking from 1.63% in DR-QA to 24.49% in EC-QA.
• EC-QA is prone to comparative stance hallucinations and entity misidentification, with entity-attribute hallucinations reaching 22,885 cases.
• LT-QA suffers from temporal mismatches and trend hallucinations, with intent misunderstanding accounting for 35.38% of errors.
• Hybrid retrieval with reranking improves recall and precision but does not fully resolve downstream reasoning failures.
• Finance-specific models show strong domain grounding in numeric reasoning but fail to generalize to complex comparative logic.

Data & experiments

• Dataset constructed from 2,472 SEC filings (10-K, 10-Q, 8-K, etc.) covering 43 companies across 36 industries from 2020 to 2025.
• Three task types: Detail & Reasoning QA (DR-QA), Enterprise Comparison QA (EC-QA), and Longitudinal Tracking QA (LT-QA), each with 2,500 instances.
• Evaluation of 17 LLMs including GPT-4.1, GPT-5, DeepSeek-V3, Qwen3, and finance-tuned models like Fin-R1 and TouchstoneGPT.
• Experiments conducted under ground-truth context and four retrieval settings: BM25, dense vector, hybrid, and hybrid with reranking.
• Fine-grained error analysis using a 13-type taxonomy covering retrieval, generation, financial reasoning, and comprehension errors.
• LLM-as-Judge framework with three independent judges (GPT-5, DeepSeek-V3.2, Qwen3-235B) for correctness and Likert scoring.

Limitations & caveats

• Benchmark is limited to U.S. SEC filings, which may not generalize to other regulatory frameworks or international markets.
• Evaluation relies on LLM-as-Judge, which may introduce biases or inconsistencies in scoring.
• Dataset covers only 43 companies, which may not represent the full diversity of the U.S. market.
• Finance-tuned models may overfit to specific financial patterns, leading to poor generalization in complex reasoning tasks.
• Retrieval experiments use standard embedding models, which may not capture the full semantic complexity of financial texts.
• The error taxonomy, while comprehensive, may not capture all possible failure modes in real-world financial analysis.

Curator notes

Important for realistic analyst workflow over multiple companies and periods.