Taxonomy-Aligned Risk Extraction from 10-K Filings

In one sentence

Summary

The paper addresses the challenge of extracting structured, taxonomy-aligned risk factors from unstructured 10-K filings, where naive LLM outputs produce inconsistent labels. The authors propose a three-stage pipeline: first, an LLM extracts free-form risk factors with supporting quotes; second, embedding-based semantic similarity maps these quotes to a predefined three-tier hierarchical taxonomy; and third, an LLM-as-a-judge validates mappings, filtering spurious assignments based on quality scores. This hybrid approach balances the nuance of LLMs with the consistency of fixed categories, ensuring high-precision extraction while maintaining computational efficiency through local embedding models. The methodology is evaluated on 2024 10-K filings for S&P 500 companies, extracting 10,688 validated risk factors. The study introduces an autonomous taxonomy maintenance system where an AI agent analyzes low-quality validation feedback to diagnose failure patterns and propose description refinements, achieving a 104.7% improvement in embedding separation for a pharmaceutical approval category. External validation uses inverse prevalence weighting to compute risk profile similarity, testing whether same-industry companies exhibit higher similarity than cross-industry pairs without using industry codes as input features. Results show that same-industry companies have 63% higher risk profile similarity than cross-industry pairs, with Cohen’s d=1.06 and AUCs ranging from 0.733 to 0.822 for predicting industry membership at varying SIC granularities. Sector-specific analysis reveals intuitive patterns, such as 83% of banks tagged with interest rate risk versus 22% of all companies. Limitations include API costs and latency for real-time applications, the need for upfront domain expertise in taxonomy design, and the current focus on English-language filings, though the methodology is theoretically language-agnostic.

Research problem

The paper addresses the difficulty of systematically analyzing corporate risk disclosures in 10-K filings. Naive LLM extraction produces inconsistent, idiosyncratic labels that prevent cross-company comparison. The core problem is mapping free-form LLM outputs to a predefined hierarchical taxonomy while filtering spurious mappings to ensure high precision and semantic consistency.

Core contributions

• A three-stage pipeline combining LLM extraction, embedding-based mapping, and LLM-as-a-judge validation for taxonomy-aligned risk extraction.
• A three-tier hierarchical risk taxonomy designed specifically for investment analysis, balancing comprehensiveness with practical granularity.
• An autonomous taxonomy refinement workflow where an AI agent uses validation feedback to identify and fix problematic categories.
• Empirical validation on S&P 500 10-Ks showing strong industry clustering and sector-specific risk patterns.

Method

• Stage 1: LLM (Claude 4.5 Sonnet) extracts risks with supporting quotes using structured output.
• Stage 2: Embedding-based mapping using Qwen3 Embedding 0.6B with task-specific instructions.
• Nearest-neighbor matching via dot product similarity between quote and taxonomy embeddings.
• Stage 3: LLM-as-a-judge validates mappings on a 1-5 scale, filtering scores < 4.
• Deduplication by retaining highest-quality mapping per taxonomy category.
• Autonomous refinement: AI agent analyzes low-score patterns to propose description updates.
• Inverse prevalence weighting for industry clustering similarity calculation.
• Cosine distance on weighted binary risk matrices for pairwise company similarity.

Results & findings

• Extracted 10,688 validated risk factors from S&P 500 10-K filings.
• Autonomous taxonomy refinement improved embedding separation by 104.7% for a specific category.
• Same-industry companies exhibit 63% higher risk profile similarity than cross-industry pairs.
• Large effect size for industry clustering (Cohen’s d=1.06, p<0.001).
• AUC for predicting industry membership ranges from 0.733 (2-digit SIC) to 0.822 (4-digit SIC).
• 83% of banks tagged with interest rate risk vs. 22% of all companies.
• 67% of banks tagged with capital requirements vs. 3% overall.
• LLM-as-judge filtered 32.1% of initial mappings as low quality (scores 1-2).

Data & experiments

• 2024 10-K filings for 500 S&P 500 companies, extracted via Massive.com SEC AI-Ready APIs.
• Extraction of 10,688 validated risk factors mapped to a 140-category tertiary taxonomy.
• Autonomous taxonomy refinement case study on 'Pharmaceutical and medical device approval' category.
• Industry clustering analysis using inverse prevalence weighted cosine similarity on binary risk matrices.
• Statistical validation using Welch’s t-test, Kolmogorov-Smirnov test, permutation tests, and ROC curve analysis.

Limitations & caveats

• LLM-based extraction and validation incur API costs and latency.
• Taxonomy design requires upfront domain expertise.
• Current implementation focuses on English-language filings.
• Embedding models may struggle with highly nuanced or novel risk types.
• LLM-as-judge may have biases or inconsistencies in scoring.
• Dependence on the quality of the predefined taxonomy.

Curator notes

Relevant to risk factor maps and compliance-grade evidence trails.