Financial Numeric Extreme Labelling: A Dataset and Benchmarking for XBRL Tagging

In one sentence

Summary

This paper addresses the challenge of automating the assignment of U.S. GAAP XBRL labels to numerals in SEC filings, a task requiring annotation from an extremely large label set of 2,794 tags. The authors formulate this as a financial numeric entity recognition problem, aiming to reduce the manual effort required for regulatory compliance and to enable the tagging of historical documents lacking XBRL metadata. The research positions itself against prior work like FiNER, which focused on only the top 139 frequent labels, arguing that real-world application requires handling the full long-tail distribution of accounting metrics. The core contribution is the release of the FNXL dataset, derived from 10-K filings of 2,339 companies (2019-2021), containing 79,088 sentences and 142,922 annotated numerals. The authors benchmark two approaches: a sequence labeling method using FiNER (BERT-based) and a two-step pipeline combining a binary numeral extractor with AttentionXML, an extreme classification model. Experiments evaluate Macro and Micro F1 scores, bucketing performance by label frequency, and conducting human-in-the-loop studies with financial subject matter experts to assess practical utility in reducing annotation time. Results show that FiNER achieves higher Micro-F1 (75.84%) due to superior performance on frequent labels, while the AttentionXML pipeline achieves higher Macro-F1 (47.54%) by better handling rare labels. The pipeline achieves ~90% Hits@5, suggesting it can significantly accelerate human annotators. However, the study notes limitations including the exclusion of tabular data, the inability to handle zero-shot labels, and the high semantic similarity among top-k predictions which may confuse experts. The work provides a critical benchmark for automating financial data structuring but highlights the difficulty of generalizing to unseen or highly specific accounting contexts.

Research problem

Automating the assignment of U.S. GAAP XBRL labels to numerals in SEC 10-K filings is a labor-intensive process requiring expert knowledge. The problem is complicated by the extremely large label set (2,794 tags) and a long-tail distribution where many accounting metrics appear very infrequently. Prior datasets focused only on frequent labels, failing to capture the complexity of real-world regulatory compliance.

Core contributions

• Release of the FNXL dataset, containing 142,922 annotated numerals across 79,088 sentences from 2,339 companies with 2,794 unique XBRL labels.
• Formulation of financial numeric tagging as an extreme classification problem, adapting AttentionXML for this specific domain.
• Comprehensive benchmarking comparing sequence labeling (FiNER) against a two-step pipeline (Binary Tagger + AttentionXML).
• Human evaluation with financial subject matter experts demonstrating the practical utility of top-k recommendations for annotation assistance.
• Analysis of label frequency buckets, revealing distinct performance trade-offs between sequence labeling and extreme classification for frequent vs. rare labels.

Method

• Data Preprocessing: Filtered SEC 10-K filings for textual numerals, removed non-US-GAAP labels, and cleaned noisy data.
• FiNER Approach: Fine-tuned BERT/SEC-BERT for sequence labeling using pseudo-tokens [NUM] and [SHAPE] to handle numeral fragmentation.
• Binary Tagger: BERT-based sequence tagger to identify relevant numerals vs. irrelevant ones in a sentence.
• AttentionXML: Deep learning model with BiLSTM and multi-label attention layer for extreme multi-label classification.
• Masking Strategies: Experimented with [MASK], [NUM], and [SHAPE] tokens to focus the model on relevant numerals.
• Evaluation: Computed Macro and Micro F1 scores, with bucket analysis for top-100 and bottom-1000 frequent labels.
• Human Study: Recruited financial SMEs to validate top-5 predictions and measure inter-annotator agreement.

Results & findings

• FNXL dataset contains 2,794 labels, significantly more than the 139 labels in the prior FiNER-139 dataset.
• FiNER (BERT-base) achieves 75.84% Micro-F1, outperforming AttentionXML (74.74%) on frequent labels.
• AttentionXML pipeline achieves 47.54% Macro-F1, outperforming FiNER (47.13%) on rare/long-tail labels.
• Bucket analysis shows FiNER Macro-F1 of 82.52% on top-100 labels vs. AttentionXML's 81.97%.
• For the least 1000 frequent labels, AttentionXML achieves 42.25% Macro-F1 vs. FiNER's 39.17%.
• AttentionXML pipeline achieves ~90% Hits@5 and ~92% Hits@10, indicating high practical utility for annotators.
• Human experts showed high agreement when machine predictions were correct, but performance dropped when machines failed.
• Top-5 predicted labels by AttentionXML have high cosine similarity (0.50), which may confuse human annotators.

Data & experiments

• Dataset: FNXL, sourced from SEC 10-K filings (2019-2021) for 2,339 companies, filtered for US-GAAP labels and textual numerals.
• Split: ~78:9:13 train/validation/test split by company to prevent data leakage, including zero-shot labels in test/dev sets.
• Baselines: FiNER (BERT-base and SEC-base) with various masking strategies ([NUM], [SHAPE], no-masking).
• Proposed Method: AttentionXML pipeline using a BERT binary tagger for numeral extraction followed by AttentionXML for label classification.
• Metrics: Macro and Micro Precision, Recall, and F1; Hits@k for top-k label recommendations.
• Human Study: 6 financial experts annotated 305 datapoints using top-5 predictions to measure inter-annotator agreement and efficiency.

Limitations & caveats

• Dataset excludes tabular data, focusing only on textual numerals.
• Does not include custom company-specific labels, only US-GAAP metrics.
• Models struggle with zero-shot labels not seen during training.
• Context dependency: Labeling often requires surrounding paragraph or table context, which is not fully utilized.
• High similarity among top-k predictions may confuse human annotators.
• Human study limited to 6 experts with generic XBRL knowledge, not specialized accountants.

Curator notes

Discovery status: report_analysis_focus_promoted. Source links: HiFi-KPI: A Dataset for Hierarchical KPI Extraction from Earnings Filings