Finch: Benchmarking Finance & Accounting across Spreadsheet-Centric Enterprise Workflows

In one sentence

Summary

The paper addresses the gap in evaluating AI agents on realistic, messy enterprise finance and accounting workflows, which involve interleaved tasks like data entry, calculation, and reporting across heterogeneous artifacts. Unlike clean benchmarks, real-world work is long-horizon, knowledge-intensive, and multimodal, requiring agents to navigate complex spreadsheet structures, cross-file references, and implicit business logic. The authors argue that current benchmarks fail to capture the compositional complexity and structural messiness that characterize professional F&A work. To address this, the authors construct FinWorkBench (Finch) using a novel pipeline that mines workflows from Enron email threads, versioned spreadsheet histories, and public financial reports. The dataset comprises 172 composite workflows with 384 tasks, involving 1,710 spreadsheets with 27 million cells, alongside PDFs and images. Evaluation employs both expert human judgment and an automated LLM-as-judge pipeline that uses structured diffs and screenshots to assess completeness, correctness, and over-edit avoidance. The study evaluates frontier models including GPT-5.1 Pro, Claude Opus/Sonnet 4.5, Gemini 3 Pro, Grok 4, and Qwen 3 Max. Results show that even the strongest agents pass fewer than 50% of workflows, with GPT-5.1 Pro achieving only 38.4% under human evaluation despite spending an average of 16.8 minutes per workflow. Key failure modes include error accumulation in long-horizon tasks, difficulty with messy spreadsheet layouts, and struggles to reconstruct latent business logic in formulas. The paper highlights that multimodal artifacts and cross-file retrieval further degrade performance. Finch serves as a rigorous testbed for agentic workflows, revealing that current LLMs are not yet ready for autonomous enterprise finance operations without significant human oversight.

Research problem

The paper addresses the lack of benchmarks that evaluate AI agents on realistic, messy, and long-horizon enterprise finance and accounting workflows. Current benchmarks often use clean, isolated tables, failing to capture the compositional complexity, multimodal nature, and implicit business logic of real-world professional work. The authors seek to determine if frontier AI agents can handle the intricate, multi-step tasks that characterize actual finance and accounting operations.

Core contributions

• Introduction of FinWorkBench (Finch), a benchmark of 172 real-world enterprise finance and accounting workflows sourced from authentic environments like Enron.
• Development of a novel workflow construction pipeline that mines workflows from email threads, versioned spreadsheet histories, and public reports.
• Creation of a rigorous evaluation framework combining expert human evaluation and an automated LLM-as-judge pipeline using structured diffs and screenshots.
• Comprehensive evaluation of frontier AI agents (GPT-5.1, Claude, Gemini, etc.) on complex, long-horizon tasks, revealing significant performance gaps.
• Detailed analysis of failure modes, including error accumulation, handling of messy layouts, and reconstruction of latent business logic.

Method

• Workflow Construction: LLM-assisted mining of workflows from Enron emails and versioned spreadsheets, followed by expert annotation.
• Data Sourcing: Integration of Enron corpus, EUSES, public reports (World Bank, government), and financial institution artifacts.
• Evaluation Framework: Binary pass/fail labels based on human experts and LLM-as-judge using structured diffs and screenshots.
• Model Interaction: Agents use code execution (SpreadsheetBench) or native file manipulation (ChatGPT/Claude) to perform tasks.
• Context Management: Automatic truncation and semantic-rich tuple encoding to handle large, complex spreadsheets.
• Multimodal Handling: Native PDF/image support for vision-capable models; text extraction for others.

Results & findings

• GPT-5.1 Pro passes only 38.4% of workflows under human evaluation, spending an average of 16.8 minutes per workflow.
• Frontier agents pass fewer than 50% of workflows, indicating significant limitations in handling real-world enterprise tasks.
• Performance drops significantly for workflows with more than two tasks, highlighting error accumulation in long-horizon reasoning.
• Messy spreadsheet structures and cross-sheet dependencies are major sources of failure, particularly for data retrieval and formula reconstruction.
• Multimodal artifacts (PDFs, images) further reduce pass rates, with GPT-5.1 Pro dropping to 35.0% on workflows involving these formats.
• Web search tasks require high tool-call overhead (median 16 calls), with external information acquisition being a dominant cost.
• The LLM-as-judge pipeline shows high consistency with human evaluations, enabling scalable assessment.

Data & experiments

• Dataset: 172 composite workflows, 384 tasks, 1,710 spreadsheets (27 million cells), plus PDFs, images, and Word docs.
• Sources: Enron Email Corpus, EUSES spreadsheet corpus, World Bank reports, government documents, and financial institution artifacts.
• Evaluation: Human expert evaluation and automated LLM-as-judge pipeline assessing completeness, correctness, and over-edit avoidance.
• Models Evaluated: GPT-5.1 Pro, Claude Sonnet/Opus 4.5, Gemini 3 Pro, Grok 4, Qwen 3 Max.
• Metrics: Workflow pass rate, tool-call overhead, and qualitative case studies of failure modes.

Limitations & caveats

• Dataset is derived from public sources (Enron, EUSES) which may not fully represent current enterprise environments.
• Evaluation relies on human experts, which is time-consuming and potentially subjective despite rigorous QC.
• Tool-call analysis is based on a case study of 20 tasks, not a full-scale statistical evaluation.
• Some workflows may be biased towards specific types of financial tasks (e.g., reporting) over others (e.g., trading).
• The LLM-as-judge, while consistent, may still miss subtle semantic errors that humans catch.

Curator notes

Discovery status: regtech_compliance_focus_promoted. Source links: FinAuditing: A Financial Taxonomy-Structured Multi-Document Benchmark for Evaluating LLMs