FinReflectKG: Agentic Construction and Evaluation of Financial Knowledge Graphs

In one sentence

Summary

The paper addresses the critical gap in large-scale, high-quality open-source financial knowledge graphs by proposing a robust construction pipeline for SEC 10-K filings. Recognizing that existing datasets often rely on noisy news feeds or lack rigorous evaluation, the authors introduce a schema-guided, closed-information extraction approach. This method integrates intelligent document parsing, table-aware semantic chunking, and iterative prompt engineering to extract structured triples from the complex, regulatory-heavy text of annual reports. The system is designed to support downstream applications such as entity search, multi-hop question answering, and signal generation, providing a foundational resource for financial AI research that emphasizes transparency and reproducibility. The core methodology features three extraction modes: single-pass, multi-pass, and a novel reflection-agent-based workflow. Using Qwen2.5-72B-Instruct, the reflection agent employs a critic-corrector loop to iteratively refine extracted triples, checking for schema compliance, entity normalization, and business relevance. The dataset comprises triples from all S&P 100 companies' 2024 10-K filings, utilizing a comprehensive ontology of entity types (e.g., ORG, RISK_FACTOR) and relationships (e.g., Has_Stake_In, Impacted_By). Evaluation combines rule-based checks (CheckRules), coverage ratios, entropy metrics, and LLM-as-a-Judge assessments for precision, faithfulness, and comprehensiveness, offering a holistic view of extraction quality without requiring ground-truth annotations. Empirical results demonstrate that the reflection-agent mode achieves the best balance of accuracy and coverage, attaining a 64.8% compliance score across all rule-based policies and outperforming baselines in LLM-as-a-Judge evaluations for precision and comprehensiveness. While it generates denser, more navigable graphs, it exhibits lower semantic entropy, indicating a deliberate reduction in diversity for higher reliability. Key limitations include the lack of cross-document co-reference resolution and the reliance on LLM-voting for evaluation, which may propagate judge biases. The authors note that while the reflection mode is computationally heavier, it is essential for high-stakes financial applications requiring auditability and trust.

Research problem

The financial domain lacks large-scale, open-source knowledge graph datasets derived from authoritative corporate disclosures like SEC 10-K filings. Existing financial KGs often rely on noisy news data or lack rigorous evaluation frameworks, hindering their adoption in regulated industries. This paper addresses the need for a high-quality, schema-guided, and agentic approach to construct reliable financial knowledge graphs that can support downstream AI applications.

Core contributions

• Release of a large-scale, open-source financial knowledge graph dataset constructed from 2024 SEC 10-K filings of all S&P 100 companies.
• Introduction of a novel reflection-driven agentic extraction framework that iteratively refines triples using a critic-corrector loop.
• Development of a holistic evaluation methodology combining rule-based compliance checks, coverage metrics, and LLM-as-a-Judge assessments.
• Demonstration of three extraction modes (single-pass, multi-pass, reflection) with empirical analysis of their trade-offs between efficiency and accuracy.

Method

• Intelligent Document Parsing: Uses Docling to preserve table structures and section headers in markdown format.
• Table-Aware Chunking: Segments documents while keeping tables intact to preserve quantitative context.
• Single-Pass Extraction: One-step prompt to extract triples based on a predefined schema.
• Multi-Pass Extraction: Two-step process involving initial extraction followed by normalization and deduplication.
• Reflection-Agent Workflow: Iterative critic-corrector loop where a feedback LLM identifies schema violations and suggests corrections.
• Schema Canonicalization: Enforces strict entity normalization (e.g., ticker symbols) and relation types.
• LLM-as-a-Judge: Uses Qwen3-32B to evaluate precision, faithfulness, comprehensiveness, and relevance without ground truth.

Results & findings

• Reflection mode achieves 64.8% compliance across all CheckRules, significantly outperforming single-pass (42.3%) and multi-pass (47.3%).
• Reflection mode generates the most triples per chunk (15.8) and highest entity coverage ratio (0.53).
• LLM-as-a-Judge evaluation shows reflection mode wins in precision (39.1%), comprehensiveness (48.1%), and relevance (37.3%).
• Single-pass mode excels in faithfulness (40.1%) but lags in comprehensiveness.
• Reflection mode exhibits lower entropy, indicating a more compact and connected graph structure.
• Cross-document co-reference resolution remains a limitation as the pipeline processes isolated filings.

Data & experiments

• Dataset: SEC 10-K filings for all S&P 100 companies for the year 2024.
• Schema: Pre-configured ontology of entity types (e.g., ORG, RISK_FACTOR, ESG_TOPIC) and relationships (e.g., Has_Stake_In, Complies_With).
• Models: Qwen2.5-72B-Instruct for extraction; Qwen3-32B for LLM-as-a-Judge evaluation.
• Evaluation Metrics: CheckRules compliance, Entity/Relationship Coverage Ratios, Shannon/Rényi Entropy, and LLM-as-a-Judge scores (Precision, Faithfulness, Comprehensiveness, Relevance).
• Baselines: Single-pass extraction and multi-pass extraction with normalization.

Limitations & caveats

• Cross-document co-reference resolution is not fully addressed, limiting graph connectivity across filings.
• Evaluation relies on LLM-as-a-Judge, which may propagate biases inherent in the judge model.
• Reflection mode requires additional inference rounds, increasing computational cost and latency.
• Schema may not capture all nuances of financial data, as indicated by underutilization of some schema elements.
• Dataset is limited to S&P 100 companies and 2024 filings, lacking temporal depth.

Curator notes

Discovery status: specific_domain_round3_promoted. Source links: FinKG: A Core Financial Knowledge Graph for Financial Analysis