FinanceQA: A Benchmark for Evaluating Financial Analysis Capabilities of Large Language Models

In one sentence

Summary

The paper addresses the critical gap between LLM capabilities and the rigorous demands of professional financial analysis, where high precision and adherence to accounting standards are non-negotiable. It argues that existing benchmarks like FinQA are misaligned with real-world workflows, focusing on simple extraction rather than complex reasoning, hand-spreading of metrics, and handling incomplete data. The authors introduce FinanceQA, a benchmark designed to test LLMs on tactical questions derived from primary documents like 10-K filings and conceptual questions requiring financial logic, aiming to replicate the daily tasks of hedge fund and private equity analysts. The dataset was annotated by professionals with experience in investment firms, ensuring questions reflect actual on-the-job challenges. The evaluation covers four top models: GPT-4o, o1, Claude-3.5-Sonnet, and Llama-3.3-70B. Tasks are categorized into basic tactical, assumption-based (requiring inference from missing data), and conceptual questions. The study also experiments with fine-tuning GPT-4o on a synthetic dataset generated from human-annotated examples to assess if high-quality training data can bridge the performance gap. Evaluation uses exact match metrics to enforce strict accuracy standards. Results show that even the best model, o1, achieves only 48.7% accuracy, with assumption-based tasks performing worst at under 5%. Fine-tuning GPT-4o yielded significant gains, improving total accuracy from 39.2% to 56.8%, with assumption-based performance jumping over 600%. The paper highlights that LLMs struggle with non-GAAP adjustments, lease accounting conventions, and generating assumptions for missing variables. Limitations include the dataset's focus on a single company (Costco) and the exclusion of Excel-based modeling tasks, suggesting that while fine-tuning helps, current LLMs remain insufficient for fully autonomous professional financial analysis.

Research problem

Current LLMs fail to meet the strict accuracy and rigor required in professional financial analysis, particularly in tasks involving hand-spreading metrics, adhering to accounting conventions, and reasoning under incomplete information. Existing benchmarks do not adequately test these real-world capabilities, leading to an overestimation of LLM utility in finance.

Core contributions

• Introduction of FinanceQA, a benchmark focusing on realistic financial analysis tasks including tactical and conceptual questions.
• Demonstration that state-of-the-art LLMs fail approximately 60% of professional-grade financial analysis tasks.
• Evidence that fine-tuning on high-quality, human-annotated financial reasoning data significantly improves model performance.
• Analysis of specific failure modes, including poor handling of accounting conventions and incomplete information.
• Creation of a synthetic data generation pipeline to augment fine-tuning datasets for financial reasoning.

Method

• Annotation by professionals with experience in hedge funds and private equity.
• Tactical questions derived from primary financial documents (10-Ks).
• Conceptual questions testing financial logic and metric relationships.
• Synthetic data generation to create variations of human-annotated examples.
• Fine-tuning using OpenAI's API with default metrics.
• Exact match evaluation for binary correctness assessment.

Results & findings

• Top model o1 achieved only 48.7% accuracy on FinanceQA, failing ~60% of tasks.
• Assumption-based questions had the lowest performance, with all models scoring under 5%.
• Fine-tuned GPT-4o improved total accuracy from 39.2% to 56.8%.
• Assumption-based performance for fine-tuned GPT-4o increased by over 600%.
• Conceptual questions were easier for LLMs, suggesting training data bias towards logic/math.
• LLMs frequently failed to apply correct accounting conventions, such as averaging for stock-flow measures.
• Statistical tests confirmed fine-tuning improvements were significant at the 95% confidence level.

Data & experiments

• FinanceQA dataset with tactical questions based on Costco's 10-K filings and conceptual questions.
• Evaluation of four models: GPT-4o, o1, Claude-3.5-Sonnet, and Llama-3.3-70B-Instruct.
• Fine-tuning experiment on GPT-4o using a synthetic dataset of 9,078 rows generated from human annotations.
• Exact match evaluation metric to enforce strict accuracy requirements.
• Statistical significance testing using paired t-tests to validate performance improvements.

Limitations & caveats

• Dataset limited to data from a single company (Costco), potentially lacking generalizability.
• Excludes Excel-based modeling tasks, which are crucial for practical financial adoption.
• Synthetic data may not fully capture the nuance of human-annotated examples.
• Limited number of annotators and budget constrained the scale of high-quality curation.
• Focus on tactical questions may not fully represent all aspects of financial analysis.

Curator notes

Discovery status: institutional_trading_focus_promoted. Source links: large language models private equity due diligence; large language models real estate finance investment; large language models commercial real estate investment analysis