HiFi-KPI: A Dataset for Hierarchical KPI Extraction from Earnings Filings

In one sentence

Summary

This paper addresses the challenge of extracting Key Performance Indicators (KPIs) from SEC filings by leveraging the complex, fine-grained structure of inline eXtensible Business Reporting Language (iXBRL). The authors argue that existing datasets lack the hierarchical context and cross-company transferability provided by iXBRL taxonomies. To solve this, they introduce HiFi-KPI, a corpus of 1.65M paragraphs and 4.5M entities derived from 10-K and 10-Q filings between 2017 and 2024. The dataset preserves critical contextual metadata, including time periods, currencies, and numeric values, alongside two unified taxonomies (Presentation and Calculation) created via a bottom-up aggregation algorithm to reduce label sparsity. A smaller, expert-curated subset, HiFi-KPI-Lite, is also provided for rapid evaluation of structured extraction tasks. The authors evaluate three tasks: text classification, sequence labeling, and LLM-based structured extraction. For classification and labeling, they fine-tune encoder-based models like BERT and FLANG-BERT, testing performance across varying levels of taxonomy granularity. For structured extraction, they benchmark four large language models (Gemma-3-27B, Qwen3-30B-A3B, Mistral-Small-3.2-24B, and DeepSeek-V3.1) on HiFi-KPI-Lite, requiring the extraction of labels, dates, currencies, and values in JSON format. The experiments use macro-F1 scores for classification and exact match F1 for extraction, with a temporal split to prevent data leakage. The study highlights the difficulty of aligning LLM outputs with strict gold standards, particularly in normalizing numerical values and interpreting temporal phrases. Results show that encoder-based models achieve high performance (macro-F1 > 0.906) on classification tasks, especially when using coarser taxonomy levels to mitigate sparsity. In contrast, LLMs perform significantly lower on structured extraction, with the best model (Qwen3) reaching only 0.440 F1. Qualitative analysis reveals that LLM errors primarily stem from date misinterpretation (e.g., confusing start and end dates) and value normalization issues. The paper concludes that while LLMs show promise, domain-specific fine-tuning and robust handling of iXBRL context remain critical. Limitations include potential biases toward large US companies and the strictness of the evaluation criteria, which may penalize semantically correct but syntactically different extractions.

Research problem

The paper addresses the difficulty of accurately extracting and generalizing Key Performance Indicators (KPIs) from SEC filings due to the complexity and fine-grained nature of iXBRL taxonomies. Existing datasets often lack the hierarchical context and cross-company transferability needed for robust financial NLP. The authors aim to create a resource that leverages iXBRL's structural metadata (dates, currencies, values) to improve KPI extraction and classification.

Core contributions

• Introduction of HiFi-KPI, a large-scale dataset of 1.65M paragraphs and 4.5M entities with iXBRL tags, dates, currencies, and values.
• Creation of two unified taxonomies (Presentation and Calculation) using a bottom-up aggregation algorithm to reduce label sparsity.
• Release of HiFi-KPI-Lite, a manually curated 8K paragraph subset with expert mappings to four key financial concepts (Earnings, EBIT, EPS, Revenues).
• Establishment of baselines for text classification, sequence labeling, and LLM-based structured extraction on the new dataset.
• Qualitative analysis identifying specific failure modes in LLM extraction, particularly regarding date normalization and value interpretation.

Method

• Data Collection: Scraped SEC filings (2017-2024) and parsed iXBRL tags, dates, currencies, and values using BeautifulSoup and regex.
• Taxonomy Creation: Parsed .cal and .pre files using Arelle, aggregated per-document hierarchies into unified taxonomies using majority voting for parent-child relationships.
• Granularity Selection: Implemented a bottom-up aggregation algorithm to map fine-grained labels to parent nodes, reducing sparsity.
• Text Classification: Fine-tuned EmbeddingGemma-300M classification head on iXBRL labels across different granularity levels.
• Sequence Labeling: Fine-tuned BERT and FLANG-BERT on the 1,000 most frequent tags, mapping others to an out-of-scope label.
• Structured Extraction: Prompted LLMs (Gemma, Qwen, Mistral, DeepSeek) with 1-shot examples to extract labels, dates, currencies, and values in JSON format.
• Evaluation: Used macro-F1 for classification/labeling and exact match F1 for extraction, with temporal splits to ensure generalization.

Results & findings

• Encoder-based models achieve macro-F1 > 0.906 on classification tasks using the Presentation taxonomy.
• Sequence labeling performance is high, with FLANG-BERT and BERT showing negligible difference on the 1,000 most frequent tags.
• LLMs struggle with structured extraction, with Qwen3-30B-A3B achieving the highest F1 of 0.440 on HiFi-KPI-Lite.
• DeepSeek-V3.1 shows high precision (0.463) but lower recall, indicating a conservative extraction strategy.
• Gemma-3-27B fails to correctly interpret temporal phrases like 'three months ended', leading to date errors.
• Most LLM errors are inconsistent, with significant issues in normalizing values and handling domain-specific terms.
• The bottom-up aggregation method effectively reduces label sparsity, improving classification performance for long-tail labels.
• HiFi-KPI-Lite provides a robust benchmark for rapid prototyping, highlighting the gap between encoder and LLM capabilities.

Data & experiments

• Dataset: 1.65M paragraphs from 56,407 SEC 10-K and 10-Q filings (2017-2024), parsed using iXBRL metadata.
• Taxonomies: Unified Presentation (1.57M edges, depth 21) and Calculation (202K edges, depth 12) taxonomies.
• Tasks: Text classification, sequence labeling (1,000 most frequent tags), and structured information extraction.
• Models: BERT, FLANG-BERT, EmbeddingGemma-300M for classification/labeling; Gemma-3-27B, Qwen3-30B-A3B, Mistral-Small-3.2-24B, DeepSeek-V3.1 for extraction.
• Metrics: Macro-F1 for classification/labeling; Exact match F1, Precision, Recall, and Jaccard similarity for extraction.
• Evaluation: Temporal split to prevent leakage; 5 runs for LLMs to assess variance.

Limitations & caveats

• Dataset bias towards large US companies, as they are more likely to file with the SEC.
• Potential annotation quality issues, as SEC filings may contain errors or inconsistent labeling.
• Strict evaluation criteria for structured extraction may penalize semantically correct but syntactically different outputs.
• Limited generalizability to non-US companies or those using different financial reporting standards.
• LLM performance is evaluated on a small subset (HiFi-KPI-Lite), which may not represent performance on the full dataset.
• The bottom-up aggregation method may lose some fine-grained information, potentially impacting tasks requiring high specificity.

Curator notes

Discovery status: report_analysis_focus_promoted. Source links: SEC-QA: A Systematic Benchmark for Evaluating Long-Context Question Answering on SEC Filings