BizBench: A Quantitative Reasoning Benchmark for Business and Finance

In one sentence

Summary

The paper addresses the critical gap in evaluating Large Language Models' ability to perform precise, multi-step quantitative reasoning required in business and finance workflows. Unlike existing benchmarks focusing on sentiment or classification, BizBench emphasizes transparent reasoning via program synthesis, where models generate executable Python code to answer financial questions. This approach ensures that the rationale for answers is auditable, addressing the opacity of chain-of-thought methods. The benchmark isolates three core capabilities: extracting numerical values from text and tables, understanding financial domain knowledge and formulas, and synthesizing code to compute complex solutions, thereby providing a rigorous test of financial literacy and computational precision. The methodology comprises eight tasks across three categories: program synthesis (FinCode, CodeFinQA, CodeTAT-QA), quantity extraction (SEC-Num, ConvFinQA Extract, TAT-QA Extract), and domain knowledge (FinKnow, FormulaEval). Data sources include newly collected SEC filings for SEC-Num, professional CFA/CPA exams for FinCode and FinKnow, and augmented versions of existing datasets like FinQA and TAT-QA. The authors evaluate a wide range of open-source (Llama-2, CodeLlama, Mistral) and commercial (GPT-4, GPT-3.5) models using zero-shot, few-shot, and supervised fine-tuning setups. Experiments measure accuracy against ground-truth numeric answers or executable code correctness, with strict matching criteria to ensure precision in financial calculations. Results indicate that while larger models and those with code-specific pretraining perform better, significant performance gaps remain. GPT-4 leads but still fails over 36% of the most difficult FinCode questions, primarily due to errors in financial knowledge rather than code syntax. Error analysis reveals that models struggle with scaling (millions vs. billions), sign conventions for losses, and complex formula application. Fine-tuning smaller models like Llama-2-7B on BizBench data allows them to outperform larger unfine-tuned models, suggesting that targeted training in financial reasoning is highly effective. The paper highlights that current LLMs lack the robust financial background knowledge necessary for high-stakes, real-world financial applications, limiting their direct deployment in professional workflows without significant domain-specific adaptation.

Research problem

The paper addresses the difficulty of evaluating and improving Large Language Models' ability to perform precise, multi-step quantitative reasoning in business and finance. It highlights that while LLMs are strong at general QA and code generation, they struggle with the transparency and precision required for financial tasks, where errors can have significant monetary consequences. The problem is framed around the need for benchmarks that test not just language understanding but also the integration of financial domain knowledge, quantity extraction, and program synthesis to ensure auditable and accurate reasoning.

Core contributions

• Introduction of BizBench, a new benchmark comprising eight tasks for evaluating quantitative reasoning in finance and business.
• Creation of novel datasets including SEC-Num for quantity extraction from SEC filings, FinCode from professional exams, and FormulaEval for financial formula knowledge.
• Reformulation of existing financial QA datasets (FinQA, TAT-QA) into program synthesis tasks (CodeFinQA, CodeTAT-QA) to enable auditable reasoning via executable code.
• Comprehensive evaluation of state-of-the-art open-source and commercial LLMs, analyzing the impact of model size, code pretraining, and instruction tuning.
• Detailed error analysis demonstrating that the primary bottleneck for LLMs in financial reasoning is limited business and financial background knowledge, not just code generation ability.

Method

• Program Synthesis: Models generate Python code to answer questions, with accuracy measured by numeric output matching ground truth within 1%.
• Quantity Extraction: Span identification tasks where models extract numerical values from text/tables given a label or description.
• Domain Knowledge: Multiple-choice QA and code-completion tasks testing recall of financial terms, formulas, and concepts.
• Evaluation Metrics: Accuracy for QA and code tasks, with strict matching for numeric answers and unit testing for code completion.
• Experimental Setup: Zero-shot, few-shot (3-8 examples), and supervised fine-tuning on Llama-2-7B.
• Error Analysis: Categorization of errors into extraction, formula, syntax, and ambiguity to identify failure modes.

Results & findings

• GPT-4 achieves the highest performance but still fails over 36% of FinCode questions, indicating significant room for improvement.
• Error analysis of GPT-4 shows that most failures stem from gaps in financial knowledge rather than code syntax or extraction errors.
• Code-specific pretraining (CodeLlama) improves program synthesis performance by 8-16% compared to language-only models (Llama-2) of similar size.
• Fine-tuning Llama-2-7B on BizBench data allows it to outperform Llama-2-70B on several tasks, demonstrating the value of domain-specific training.
• Models struggle with scaling issues, such as confusing millions with billions, and handling signs for financial losses.
• Multi-task fine-tuning shows knowledge transfer between program synthesis and quantity extraction tasks, but performance on domain knowledge tasks does not always improve.

Data & experiments

• SEC-Num: Novel dataset of 2,000 test and 6,845 train examples for quantity extraction from SEC 10-K and 10-Q filings.
• FinCode: 137 questions from CFA and CPA exams requiring financial knowledge and code generation.
• CodeFinQA and CodeTAT-QA: Augmented subsets of FinQA and TAT-QA with Python code solutions for program synthesis.
• FinKnow: 877 multiple-choice questions from CFA and MMLU exams testing non-quantitative financial knowledge.
• FormulaEval: 50 code-completion tasks testing knowledge of financial formulas like EBITDA and net present value.
• Evaluation of 15+ models including Llama-2, CodeLlama, Mistral, Falcon, and GPT variants using zero-shot, few-shot, and fine-tuning setups.

Limitations & caveats

• Portions of the FinCode and CodeFinQA datasets were generated by LLMs, which may introduce biases or errors.
• The benchmark focuses on quantitative reasoning and does not cover qualitative financial analysis or market prediction.
• Evaluation relies on strict numeric matching, which may penalize models for minor formatting differences.
• The study does not evaluate proprietary financial LLMs like BloombergGPT due to accessibility and code-generation requirements.
• Data contamination risks exist, as the benchmark uses public data that may be present in pretraining corpora.

Curator notes

Connects financial reasoning with business analytics.