The List / Financial QA, Reasoning, and Table Understanding / paper
FinanceBench: A New Benchmark for Financial Question Answering
Abstract
FinanceBench is a first-of-its-kind test suite for evaluating the performance of LLMs on open book financial question answering (QA). It comprises 10,231 questions about publicly traded companies, with corresponding answers and evidence strings. The questions in FinanceBench are ecologically valid and cover a diverse set of scenarios. They are intended to be clear-cut and straightforward to answer to serve as a minimum performance standard. We test 16 state of the art model configurations (including GPT-4-Turbo, Llama2 and Claude2, with vector stores and long context prompts) on a sample of 150 cases from FinanceBench, and manually review their answers (n=2,400). The cases are available open-source. We show that existing LLMs have clear limitations for financial QA. Notably, GPT-4-Turbo used with a retrieval system incorrectly answered or refused to answer 81% of questions. While augmentation techniques such as using longer context window to feed in relevant evidence improve performance, they are unrealistic for enterprise settings due to increased latency and cannot support larger financial documents. We find that all models examined exhibit weaknesses, such as hallucinations, that limit their suitability for use by enterprises.
In one sentence
Summary
The paper addresses the critical gap in evaluating Large Language Models (LLMs) for financial question answering (QA), noting that existing benchmarks lack ecological validity for real-world analyst tasks. FinanceBench is introduced as a comprehensive test suite comprising 10,231 questions derived from 361 public filings (10-Ks, 10-Qs, 8-Ks) of 40 publicly traded US companies. The dataset is designed to serve as a minimum performance standard, covering information extraction, numerical reasoning, and logical deduction, with a focus on open-book scenarios that require retrieval from external documents rather than relying solely on pre-trained knowledge. The authors evaluate 16 configurations of four leading models (GPT-4, GPT-4-Turbo, Llama-2, Claude-2) across five setups: closed-book, oracle, single vector store, shared vector store, and long-context windows. A human-evaluated sample of 150 cases was manually reviewed to assess correctness, hallucinations, and refusals. The experimental design tests the impact of retrieval mechanisms and prompt ordering (context-first vs. context-last) on model performance, aiming to simulate realistic enterprise constraints where latency and document size are limiting factors. Results indicate that while long-context models (Claude-2, GPT-4-Turbo) achieve ~76-79% accuracy, they still produce incorrect answers or refusals in over 20% of cases. Vector store retrieval significantly improves performance over closed-book settings but introduces high latency and complexity; notably, GPT-4-Turbo with a shared vector store failed or answered incorrectly 81% of the time. The study highlights that models often hallucinate plausible but wrong answers, particularly in numerical reasoning tasks, and that prompt ordering significantly affects long-context performance. The findings suggest current LLMs are not yet reliable for high-stakes financial QA without significant human oversight.
Research problem
The paper identifies a lack of rigorous, ecologically valid benchmarks for evaluating LLMs on financial question answering. It addresses the challenge of assessing whether models can accurately retrieve and reason over public financial filings, a core task for investment analysis and due diligence. The problem is compounded by the need for models to handle numerical reasoning, tabular data, and long documents while avoiding hallucinations in high-stakes enterprise settings.
Core contributions
- Introduction of FinanceBench, a benchmark of 10,231 financial QA questions from 40 companies and 361 public filings.
- Creation of a human-evaluated subset of 150 cases with detailed labels for correctness, reasoning type, and evidence.
- Comprehensive evaluation of 16 LLM configurations across four models (GPT-4, GPT-4-Turbo, Llama-2, Claude-2).
- Analysis of retrieval strategies (vector stores) and long-context windows, highlighting their trade-offs in accuracy and latency.
- Qualitative taxonomy of model errors, including hallucinations, refusals, and irrelevant comments.
Method
- Dataset construction: Domain-relevant, novel-generated, and metrics-generated questions created by financial analysts.
- Retrieval setup: Vector stores built using OpenAI embeddings and Chroma database.
- Long-context setup: Full filings fed into models with truncation for documents exceeding context limits.
- Evaluation: Manual review of 2,400 model responses (150 cases x 16 configs) by research team.
- Labeling: Responses categorized as correct, incorrect, or failure to answer, with qualitative analysis of error types.
- Ablation: Testing context-first vs. context-last prompt orders to assess attention mechanisms in long contexts.
Results & findings
- GPT-4-Turbo with a shared vector store incorrectly answered or refused 81% of questions in the evaluation sample.
- Long-context models (Claude-2, GPT-4-Turbo) achieved 76-79% accuracy, but still failed in over 20% of cases.
- Oracle settings (GPT-4-Turbo) achieved 85% accuracy, indicating that retrieval and reasoning, not just knowledge, are bottlenecks.
- Llama-2 had a high rate of incorrect answers (70%) rather than refusals, suggesting a higher risk of hallucinations.
- Prompt ordering significantly impacted long-context performance; context-first prompting improved accuracy for GPT-4-Turbo and Claude-2.
- Metrics-generated questions involving numerical reasoning were the most challenging for all models.
Data & experiments
- Dataset: 10,231 questions from 361 filings (10-K, 10-Q, 8-K) of 40 US publicly traded companies.
- Evaluation sample: 150 cases manually reviewed by experts for correctness and reasoning quality.
- Models tested: GPT-4, GPT-4-Turbo, Llama-2, Claude-2.
- Configurations: Closed-book, Oracle, Single Vector Store, Shared Vector Store, Long Context.
- Metrics: Accuracy (correct/incorrect/refusal), impact of prompt ordering (context-first vs. context-last).
Limitations & caveats
- The evaluation sample of 150 cases may not fully represent the diversity of the entire 10,231-question dataset.
- Vector store implementations are simplified and may not reflect the complexity of production-grade retrieval systems.
- Long-context windows are still limited in size, potentially truncating relevant information in very long filings.
- The study focuses on single-turn QA, whereas financial analysis often involves multi-turn conversations.
- Results are based on models available in late 2023; newer models may have improved performance.
Curator notes
Includes questions, answers, and supporting evidence strings.