The List / Trading, Investment, and Portfolio Management / paper
FinSheet-Bench: From Simple Lookups to Complex Reasoning, Where LLMs Break on Financial Spreadsheets
Abstract
While Large Language Models (LLMs) can accelerate text-heavy tasks in alternative investment due diligence, a gap remains in their ability to accurately extract and reason over structured tabular data from complex financial spreadsheets. Progress is held back by the lack of real industry fund portfolio datasets for benchmarking, as private equity data rooms are confidential. To address this, we introduce FinSheet-Bench, a benchmark of synthetic financial portfolio data modeled on real private equity fund structures, designed to evaluate LLM performance on text-serialized spreadsheet question answering and numeric reasoning tasks. Our evaluation of ten model configurations from OpenAI, Google, and Anthropic on financial spreadsheets, including complex layouts, fund dividers, and multi-line column names, reveals that no standalone model achieves error rates low enough for unsupervised use in professional finance applications. The best-performing model, Gemini 3.1 Pro, achieves 82.4% accuracy across twenty-four evaluation files of varying complexity and structural layout (approximately 1 error per 6 questions), followed by GPT-5.2 with reasoning at 80.4%, Claude Opus 4.6 with thinking at 80.2%, and Gemini 3 Pro at 80.2%. Performance degrades substantially on larger, more complex spreadsheets: the largest spreadsheet (152 companies, 8 funds) yields an average accuracy of just 48.6% across all models, compared to 86.2% on the easiest evaluation file. These difficulty patterns are consistent across all ten models, indicating that they reflect LLM limitations rather than idiosyncratic model weaknesses. Reliable financial spreadsheet extraction will likely require architectural approaches that separate document understanding from deterministic computation.
In one sentence
Summary
The paper addresses the critical gap in evaluating Large Language Models (LLMs) for extracting and reasoning over structured tabular data in private equity due diligence, where real-world data is confidential. Existing benchmarks fail to capture the complexity of non-standardized, multi-sheet financial spreadsheets used in alternative investment analysis. The authors introduce FinSheet-Bench, a benchmark of synthetic portfolio data modeled on real fund structures, designed to test text-serialized spreadsheet question answering and numeric reasoning tasks. This work positions itself at the intersection of financial NLP and tabular reasoning, highlighting the specific challenges of private markets data. The methodology involves generating 24 synthetic Excel files from 8 real structural templates, applying rigorous anonymization and perturbation to cell values while preserving layout complexity. The authors evaluate ten model configurations from OpenAI, Google, and Anthropic using a zero-shot prompt and a cascading verification system (exact, fuzzy, and LLM-adjudicated matching). The experiments cover seven question categories ranging from simple lookups to complex aggregations, testing performance across varying spreadsheet sizes and structural modifications like merged cells and hidden rows. Results show no standalone model achieves error rates low enough for unsupervised professional use, with the best model (Gemini 3.1 Pro) reaching 82.4% accuracy. Performance degrades significantly on larger, more complex spreadsheets, dropping to 48.6% for the largest file. Simple lookups achieve ~89% accuracy, but complex aggregations fall to ~20%. The findings suggest that LLMs struggle with multi-step numerical reasoning and spatial understanding in serialized text. The paper concludes that reliable extraction requires architectural approaches separating document understanding from deterministic computation, rather than relying solely on model scaling.
Research problem
The paper addresses the inability of current LLMs to accurately extract and reason over structured tabular data from complex financial spreadsheets, a key bottleneck in private equity due diligence. It highlights the lack of public benchmarks for this specific task due to the confidentiality of private market data rooms. The problem is further compounded by the non-standardized, messy nature of real-world portfolio monitoring spreadsheets, which differ significantly from clean tabular datasets used in existing benchmarks.
Core contributions
- Introduction of FinSheet-Bench, a novel benchmark of synthetic financial portfolio data modeled on real private equity fund structures.
- Development of a rigorous data generation pipeline that anonymizes real templates while preserving structural complexity and applying controlled perturbations to cell values.
- Comprehensive evaluation of ten frontier LLM configurations across 24 synthetic spreadsheet files, covering a wide range of question complexities.
- Design of a cascading verification system combining rule-based and LLM-adjudicated methods for robust evaluation of free-text answers against structured ground truth.
- Detailed analysis of failure modes, distinguishing between parsing errors due to serialization and reasoning errors in numerical aggregation.
Method
- Data generation: Real templates are used for structure; cell values are regenerated using scaling factors and perturbations to ensure anonymity.
- Serialization: Excel files are converted to comma-separated text, preserving sheet boundaries but losing visual formatting cues.
- Prompting: Zero-shot prompts are used for all models to ensure fair comparison and reflect real-world deployment constraints.
- Verification: A three-tier system (exact, fuzzy, LLM-adjudicated) is used to evaluate free-text responses against structured ground truth.
- Evaluation: Models are tested on 16 question templates, with parameterized questions sampled to manage computational cost.
- Analysis: Performance is broken down by question category, complexity level, and file size to identify specific failure modes.
Results & findings
- Gemini 3.1 Pro achieves the highest accuracy at 82.4%, followed by GPT-5.2 (reasoning) at 80.4% and Claude Opus 4.6 (thinking) at 80.2%.
- No standalone model meets the ~97% accuracy threshold expected for unsupervised professional financial workflows.
- Performance on the largest spreadsheet (152 companies, 8 funds) drops to 48.6% accuracy, compared to 86.2% on the easiest file.
- Simple lookups achieve 89.1% pooled accuracy, while complex aggregations drop to 19.6%.
- GPT-3.5-Turbo fails on 6 of 24 files due to context limits, achieving only 33.6% accuracy on processable questions.
- Reasoning-enabled models show significant gains in aggregation and sorting tasks compared to their non-reasoning counterparts.
- Counting tasks are surprisingly difficult (41.7% accuracy) due to challenges in detecting fund boundaries in serialized text.
- Sorting tasks show a large gap between all models (37.5%) and top models (79.2%), indicating reasoning helps with ordering.
Data & experiments
- Synthetic dataset derived from 8 real private equity portfolio monitoring spreadsheets, anonymized and perturbed to prevent reconstruction.
- 24 evaluation files (versions A, B, C) with varying structural modifications, including merged cells, hidden rows, and column splits.
- Seven question categories: Simple Lookup, List Extraction, Filtering, Aggregation, Counting, Sorting, and Complex Aggregation.
- Evaluation of 10 model configurations from OpenAI, Google, and Anthropic, including reasoning-enabled variants.
- Metrics include accuracy, response time, and token usage, with a focus on per-question accuracy across varying file complexities.
- Cascading verification system with three tiers: exact matching, fuzzy matching, and LLM adjudication for ambiguous answers.
Limitations & caveats
- Results are based on text-serialized spreadsheets, which lose visual formatting cues present in native Excel files.
- The synthetic data, while modeled on real structures, may not capture all real-world complexities like embedded charts.
- The benchmark focuses on private equity; results may not generalize to other financial domains like public equities.
- Sample sizes for parameterized questions are small (n=3), which may limit the statistical power of per-file analyses.
- The study does not evaluate vision-capable models on rendered images, which might recover some lost spatial cues.
- Prompt optimization could potentially improve individual model scores but was excluded to ensure fair comparison.
Curator notes
Discovery status: institutional_trading_focus_promoted. Source links: large language models private equity due diligence; LLM fund investment private equity finance