FinAuditing: A Financial Taxonomy-Structured Multi-Document Benchmark for Evaluating LLMs

In one sentence

Summary

Financial auditing requires detecting semantic, structural, and numerical inconsistencies across large-scale, structured disclosures like XBRL filings. Existing benchmarks often fail to capture cross-document dependencies and taxonomy-defined constraints. This paper addresses the need for a realistic evaluation of LLMs in professional-grade auditing by introducing FinAuditing, a benchmark built from real US-GAAP-compliant XBRL filings. It reframes auditing as a structured information reasoning problem, requiring models to align concepts, interpret hierarchical metadata, and maintain logical consistency across interdependent documents. The work highlights the limitations of current models in handling complex, regulation-driven financial information tasks that go beyond simple text processing. The benchmark contains 1,102 annotated instances averaging over 33k tokens, derived from authoritative Data Quality Committee (DQC) error messages linked to SEC EDGAR filings. It defines three tasks: Financial Semantic Matching (FinSM) for concept alignment, Financial Relationship Extraction (FinRE) for structural consistency, and Financial Mathematical Reasoning (FinMR) for numerical verification. The authors evaluate 13 state-of-the-art LLMs, including GPT-4o, DeepSeek-V3, and financial-specific models like Fin-o1, using zero-shot settings. Evaluations employ metrics such as Hit Rate, Macro-F1, Accuracy, and fine-grained error rates (Structural, Extraction, Calculation) to assess performance on long-context, structure-aware reasoning. Results reveal substantial gaps in LLM capabilities. FinSM performance is uniformly low, with top models achieving under 13% Hit Rate, indicating reliance on surface-level similarity rather than taxonomy-aware retrieval. FinRE shows GPT-4o leading with ~92% accuracy, but most open-source and financial models struggle, particularly with complex Combination Errors. FinMR is the most challenging, with the best model achieving only 13.86% accuracy; calculation errors dominate failures. The findings suggest that parameter scaling and domain pretraining are insufficient for structured auditing. The benchmark serves as an official contest benchmark, emphasizing the need for models with explicit structural grounding and multi-step reasoning capabilities for trustworthy financial intelligence.

Research problem

The paper addresses the lack of realistic benchmarks for evaluating LLMs on professional-grade financial auditing, which requires detecting semantic, structural, and numerical inconsistencies across large-scale, structured XBRL filings. Current benchmarks fail to capture cross-document dependencies, hierarchical metadata, and taxonomy-defined constraints essential for accurate auditing.

Core contributions

• Introduction of FinAuditing, the first benchmark built from real-world XBRL filings that integrates multi-document structures and official US-GAAP taxonomies.
• Definition of three complementary subtasks: Financial Semantic Matching (FinSM), Financial Relationship Extraction (FinRE), and Financial Mathematical Reasoning (FinMR).
• Systematic evaluation of 13 state-of-the-art LLMs, revealing significant gaps in concept retrieval, taxonomy-aware relation modeling, and cross-document numerical reasoning.
• Release of the dataset, evaluation code, and establishment of the benchmark as an official public evaluation contest.

Method

• Data Collection: Retrieved 4,545 DQC error messages and filtered for 9 frequent error types across 218 filings.
• Filing Segmentation: Extracted statement-level sub-filings preserving XML dependencies (schema, linkbases, instance).
• Taxonomy Digitalization: Chunked US-GAAP taxonomy into core attributes and relational links for fine-grained indexing.
• Annotation: Used GPT-4o-mini as a parser to convert DQC messages into structured annotations (labels, elements, evidence).
• FinSM Task: Formulated as information retrieval to identify mismatched US-GAAP tags.
• FinRE Task: Formulated as classification of relation error types (Reversal, Inappropriateness, CombinationErr).
• FinMR Task: Formulated as numerical verification comparing reported values against calculated values from linkbases.
• Evaluation: Used LLM-as-a-judge for FinMR accuracy and standard metrics for FinSM/FinRE.

Results & findings

• FinSM: Top models achieve <13% Hit Rate@20, showing poor taxonomy-aware concept retrieval.
• FinRE: GPT-4o achieves 91.82% accuracy; most open-source and financial models perform significantly lower.
• FinRE Difficulty: Combination errors are the hardest category, with many models collapsing to near-zero F1.
• FinMR: Best model (Fin-o1-14B) achieves only 13.86% overall accuracy.
• FinMR Errors: Calculation errors dominate (70-83% of failures), indicating difficulty with multi-step arithmetic.
• Scaling: Increasing model size yields marginal improvements in FinSM, suggesting implicit representation is insufficient.
• Domain Pretraining: Financial-specific models do not show clear advantages over general-purpose models in FinRE.
• Structural Errors: Some models fail to produce valid JSON output structures, hindering evaluation.

Data & experiments

• Dataset: 1,102 annotated instances from real US-GAAP XBRL filings, sourced from Data Quality Committee (DQC) error messages.
• Data Sources: SEC EDGAR database filings and official US-GAAP taxonomy files.
• Tasks: FinSM (semantic matching), FinRE (relationship extraction), FinMR (mathematical reasoning).
• Models Evaluated: 13 LLMs including GPT-4o, DeepSeek-V3, Llama family, Qwen, Gemma, and financial-specific models (Fin-o1, Fin-R1).
• Metrics: Hit Rate, Recall, Macro-F1 for FinSM; Accuracy, Precision, Recall, F1 for FinRE; Accuracy, Structural/Extraction/Calculation Error Rates for FinMR.
• Setting: Zero-shot evaluation with standardized input lengths of 81,920 tokens.

Limitations & caveats

• Evaluation is zero-shot; performance may differ with fine-tuning or RAG.
• Benchmark is limited to US-GAAP taxonomy; generalizability to IFRS requires adaptation.
• FinMR accuracy is very low, making fine-grained analysis of correct reasoning difficult.
• DQC error messages may not cover all possible auditing inconsistencies.
• LLM-as-a-judge for FinMR may introduce evaluation bias.
• Focus on specific error types may not represent the full spectrum of auditing tasks.

Curator notes

Discovery status: report_analysis_focus_promoted. Source links: FinReasoning: A Hierarchical Benchmark for Reliable Financial Research Reporting