FinTagging: A Full-Scope Table-Aware XBRL Tagging Benchmark with LLMs

In one sentence

Summary

The paper addresses the critical need for accurate, machine-readable interpretation of financial reports by introducing FinTagging, the first comprehensive benchmark for structure-aware XBRL tagging. It argues that existing benchmarks oversimplify the task as flat classification over small concept subsets, ignoring the hierarchical semantics of the 17,000+ US-GAAP concepts and the structured nature of financial documents. FinTagging reframes tagging as an extract-and-align process, enabling evaluation under realistic reporting conditions where models must handle mixed text and table evidence. This approach provides a more rigorous assessment of LLMs' capabilities in numerical reasoning and taxonomy alignment, bridging the gap between academic benchmarks and regulatory compliance requirements. The methodology decomposes the tagging process into two subtasks: Financial Numeric Identification (FinNI), which extracts numerical entities and their types from heterogeneous contexts, and Financial Concept Linking (FinCL), which maps these entities to the full US-GAAP taxonomy. The dataset comprises 142 SEC 10-K filings from 2023-2024, yielding over 260,000 numerical entities linked to 3,953 unique concepts. Evaluation involves 13 state-of-the-art LLMs and PLMs in zero-shot settings, using a retrieval-reranking framework for FinCL to mitigate extreme classification challenges. The study employs expert-verified audit sets and ablation studies to isolate error propagation and assess the impact of structure-aware context construction versus fixed-window approaches. Results indicate that while large LLMs like DeepSeek-V3 and GPT-4o generalize well to long-tail concepts in extraction, they exhibit a significant knowledge-alignment gap in concept linking, with top accuracies below 0.19. The paper highlights that modeling tagging as single-step extreme classification leads to performance collapse, whereas the two-stage pipeline yields meaningful differentiation among models. Key limitations include the difficulty of disambiguating highly similar US-GAAP concepts, especially for table-origin entities, and the reliance on retrieval quality which caps downstream reranking performance. These findings underscore the need for improved structure-aware reasoning in financial LLMs for reliable automated reporting.

Research problem

Accurate interpretation of numerical data in financial reports is critical for markets and regulators, yet mapping thousands of facts to over 10k US GAAP concepts remains costly and error-prone. Existing benchmarks oversimplify this task as flat, single-step classification over small subsets of concepts, ignoring the hierarchical semantics of the taxonomy and the structured nature of financial documents. Consequently, these benchmarks fail to evaluate Large Language Models (LLMs) under realistic reporting conditions, particularly regarding their ability to handle full-scope taxonomy alignment and multi-modal evidence.

Core contributions

• Introduces FinTagging, the first benchmark that reframes XBRL tagging into a two-stage 'Extract-and-Align' paradigm, enabling full-scope evaluation against 17k+ US-GAAP concepts.
• Releases two expert-verified datasets, FinNI-eval and FinCL-eval, which incorporate complex tabular structures to assess joint reasoning across structured and unstructured financial modalities.
• Provides diagnostic insights by benchmarking 13 state-of-the-art LLMs, identifying a critical 'knowledge-alignment gap' where models excel at extraction but struggle with fine-grained taxonomy mapping.
• Demonstrates that single-step extreme classification fails for LLMs, while the proposed two-stage framework yields non-trivial performance and exposes meaningful model differences.

Method

• Two-stage formulation: Financial Numeric Identification (FinNI) extracts entities and types; Financial Concept Linking (FinCL) maps entities to US-GAAP concepts.
• Automated annotation using rule-based parser to extract iXBRL fact elements, values, concepts, and data types.
• Strategic sampling based on greedy minimal-cover method to create expert-verified audit sets balancing company balance and taxonomy completeness.
• Retrieval-reranking framework for FinCL: embed taxonomy concepts, retrieve top-h candidates, and use LLM for reranking.
• Structure-aware context construction for retrieval, linearizing table rows into Markdown-style key-value formats.
• Evaluation metrics include pair-level Precision, Recall, F1 for FinNI and accuracy for FinCL.

Results & findings

• DeepSeek-V3 achieves the best FinNI F1 score (0.6932), driven by high recall, while GPT-4o performs competitively.
• FinCL accuracy remains low across all models, with top performers (DeepSeek-V3, GPT-4o) achieving only ~0.19 accuracy.
• Single-step extreme classification results in zero performance for all LLMs, highlighting the necessity of the two-stage approach.
• Structure-aware context construction significantly outperforms fixed-window strategies, especially for table-based entities.
• Error propagation analysis shows that providing gold entities substantially improves FinCL accuracy, identifying extraction as a key bottleneck.
• Large general-purpose LLMs dominate in macro-F1, indicating better generalization to long-tail concepts compared to fine-tuned PLMs.
• Domain-specific financial LLM (Fino1-8B) lags behind general LLMs, suggesting domain pretraining alone does not yield robust gains for this task.

Data & experiments

• Dataset of 142 SEC 10-K filings from 2023-2024 covering 11 industry sectors, parsed into 319,893 narrative sentences and 21,576 financial tables.
• Automated annotation pipeline extracts 261,457 numerical entities linked to 3,953 unique US-GAAP concepts, with expert verification on a strategic audit subset.
• Evaluation of 13 models (GPT-4o, DeepSeek-V3, Llama variants, Qwen variants, Fino1-8B, BERT, FinBERT, SECBERT) in zero-shot settings.
• FinNI evaluated using pair-level Precision, Recall, and F1; FinCL evaluated using accuracy in a retrieval-reranking setup with top-200 candidates.
• Ablation studies comparing the two-stage framework against extreme classification and analyzing error propagation across pipeline stages.

Limitations & caveats

• FinCL accuracy is low even with gold entities, indicating inherent difficulty in fine-grained US-GAAP disambiguation.
• Retrieval quality imposes a ceiling on downstream reranking performance, particularly for table-based entities.
• The benchmark focuses on zero-shot settings; fine-tuning effects are not fully explored for LLMs.
• Domain-specific financial LLMs do not outperform general LLMs, suggesting limitations of current domain pretraining.
• The dataset covers only 142 filings, which may limit the generalizability of findings to other periods or sectors.
• Error propagation analysis shows that extraction errors significantly impact overall performance, highlighting the need for robust extraction models.

Curator notes

Shows fine-grained XBRL concept alignment remains difficult.