The List / Reports, Filings, Accounting, and Risk / paper
Automating Financial Statement Audits with Large Language Models
Abstract
Financial statement auditing is essential for stakeholders to understand a company's financial health, yet current manual processes are inefficient and error-prone. Even with extensive verification procedures, auditors frequently miss errors, leading to inaccurate financial statements that fail to meet stakeholder expectations for transparency and reliability. To this end, we harness large language models (LLMs) to automate financial statement auditing and rigorously assess their capabilities, providing insights on their performance boundaries in the scenario of automated auditing. Our work introduces a comprehensive benchmark using a curated dataset combining real-world financial tables with synthesized transaction data. In the benchmark, we developed a rigorous five-stage evaluation framework to assess LLMs'auditing capabilities. The benchmark also challenges models to map specific financial statement errors to corresponding violations of accounting standards, simulating real-world auditing scenarios through test cases. Our testing reveals that current state-of-the-art LLMs successfully identify financial statement errors when given historical transaction data. However, these models demonstrate significant limitations in explaining detected errors and citing relevant accounting standards. Furthermore, LLMs struggle to execute complete audits and make necessary financial statement revisions. These findings highlight a critical gap in LLMs'domain-specific accounting knowledge. Future research must focus on enhancing LLMs'understanding of auditing principles and procedures. Our benchmark and evaluation framework establish a foundation for developing more effective automated auditing tools that will substantially improve the accuracy and efficiency of real-world financial statement auditing.
In one sentence
Summary
The paper addresses the inefficiency and error-proneness of manual financial statement auditing by proposing an automated auditing framework using Large Language Models. It identifies a critical gap in current workflows where human auditors may miss granular details, and existing AI tools fail at low-level cross-verification of transaction data against financial disclosures. The research aims to systematically evaluate LLMs' ability to verify financial tables, identify discrepancies, and ensure compliance with accounting standards, thereby laying the foundation for more accurate and efficient automated auditing systems. The authors introduce the first benchmark for automatic financial statement auditing, combining real-world financial tables from S&P 500 companies with synthesized historical transaction data. The dataset includes 371 financial statements with injected errors such as missing rows, numerical errors, redundant rows, and misclassifications. A rigorous five-stage evaluation framework assesses LLMs on general judgment, error identification, error resolution, standards citation, and financial statement revision. Experiments were conducted using GPT-3.5-Turbo and GPT-4, evaluating their performance on single and multiple error scenarios using metrics like Exact Match, BertScore, and BLEU. Results show that state-of-the-art LLMs effectively identify errors in financial statements when provided with historical transaction data, achieving perfect scores in general judgment. However, they demonstrate significant limitations in explaining detected errors, citing relevant accounting standards, and executing complete audits with necessary revisions. The models struggle with joint reasoning across tabular and textual data and lack domain-specific accounting knowledge. The paper highlights that current LLMs are insufficient as reliable auditors for complex, real-world scenarios and suggests future research should focus on enhancing domain-specific knowledge and hybrid reasoning capabilities.
Research problem
Financial statement auditing is currently inefficient and error-prone, relying heavily on manual labor that may miss granular details. While AI has improved some aspects, it fails at low-level cross-verification of transaction data against financial disclosures and strict adherence to accounting standards. This paper addresses the need for intelligent agents to enhance the accuracy and efficiency of financial statement audits by automating the verification process.
Core contributions
- Introduces a novel task for automatic auditing of financial statements, addressing critical low-level auditing challenges.
- Develops the first high-quality benchmark for automatic financial statement auditing, combining real financial tables with synthetic transaction data.
- Proposes a rigorous five-stage evaluation framework to assess LLMs' auditing capabilities comprehensively.
- Conducts a systematic evaluation of state-of-the-art LLMs, providing detailed qualitative analyses and insights for future advancements.
Method
- Data creation involves transforming S&P 500 financial statement images into structured text using GPT-4-vision.
- Synthetic transaction data generated via GPT-4 prompts and verified by human annotators.
- Error injection uses GPT-4 to introduce specific error types into financial tables.
- Five-stage evaluation: General Judgment, Error Identification, Error Resolution, Standards Citation, Financial Statement Revision.
- Evaluation metrics include Exact Match, BertScore, BLEU, and Success Rate.
- LLMs prompted to act as auditors, retrieving accounting standards and generating justifications.
Results & findings
- LLMs achieve perfect Exact Match scores in general judgment, correctly identifying error-free statements.
- GPT-4 achieves 0.899 EM score in error type identification for single-error statements, while GPT-3.5-Turbo scores 0.764.
- Error entry identification is challenging, with GPT-4 scoring 0.737 and GPT-3.5-Turbo 0.418 for single errors.
- Standards citation performance is low, with GPT-4 achieving only 0.262 Top-1 EM score.
- Table revision accuracy is moderate, with GPT-4 achieving a BLEU score of 0.783 for single errors.
- Overall Success Rate is very low, with GPT-4 achieving only 0.041 for single-error statements.
- Performance declines significantly for multiple-error statements, with GPT-4's error type identification dropping to 0.752.
- LLMs struggle with joint reasoning across tabular and textual data, leading to fragmented conclusions.
Data & experiments
- Curated dataset of 371 financial statements from S&P 500 companies, including balance sheets, income statements, and cash flow statements.
- Synthesized historical transaction data generated using GPT-4 and verified by accounting students.
- Injected errors including missing rows, numerical errors, redundant rows, and misclassifications into financial tables.
- Evaluated GPT-3.5-Turbo and GPT-4 on single and multiple error scenarios using a five-stage framework.
- Metrics include Exact Match for classification, BertScore for resolution, BLEU for revision, and Success Rate for overall performance.
Limitations & caveats
- Synthesized transaction data lacks the scalability and complexity of real-world large corpora.
- The benchmark focuses only on primary financial statements, excluding budget variance analyses and segment disclosures.
- Experiments were conducted on a sample of 300 statements, which may not fully capture overall performance.
- LLMs struggle with joint reasoning across tabular and textual data, leading to fragmented conclusions.
- Current LLMs lack domain-specific accounting knowledge, limiting their ability to cite standards accurately.
- The study does not incorporate advanced prompting techniques or code-based agent settings.
Curator notes
Discovery status: report_analysis_focus_promoted. Source links: EDINET-Bench: Evaluating LLMs on Complex Financial Tasks using Japanese Financial Statements