A Scalable Data-Driven Framework for Systematic Analysis of SEC 10-K Filings Using Large Language Models

In one sentence

Summary

The paper addresses the challenge of efficiently monitoring and comparing the performance and strategic shifts of numerous publicly listed companies, a task traditionally hindered by the volume and narrative-heavy nature of SEC 10-K filings. It positions Large Language Models (LLMs) as a solution to transform qualitative disclosures into actionable, quantitative metrics, offering a cost-effective alternative to manual analysis for stakeholders like investors and analysts. The system aims to provide a scalable "litmus test" for corporate health, focusing on longitudinal trends and cross-company comparisons to mitigate biases inherent in static, single-year assessments. The methodology involves an end-to-end pipeline: data collection from SEC EDGAR, automated cleaning using unstructured-io to extract narrative text and segment sections based on SEC structures, and LLM evaluation. The system uses Cohere’s Command-R+ with role-based zero-shot prompting to generate absolute ratings (0-2 scale) for four dimensions: confidence, environment, innovation, and people. It employs two grader versions (standard and strict) to reduce bias. Relative analysis compares excerpts from multiple companies using pairwise selection. The framework is implemented as a no-code web application for visualization and year-on-year comparison. Results demonstrate the system's ability to generate uncorrelated ratings across dimensions and track strategic shifts over time for companies like Apple, IBM, and Royal Gold. The relative analysis identifies sector leaders in specific filing sections. Limitations include potential gaps in section extraction, inherent LLM biases reflecting training data, and the subjective nature of qualitative ratings. The paper contributes a reproducible pipeline and a user-friendly tool for systematic financial document analysis, though it does not directly link ratings to market performance or trading signals.

Research problem

Market analysts and investors face significant challenges in efficiently monitoring and comparing the performance and strategic shifts of a large number of companies due to the exponential growth of listed firms and the narrative-heavy, time-consuming nature of traditional financial analysis. There is a need for a fast, cost-effective, and comprehensive method to evaluate corporate health and detect strategy changes across diverse sectors. This paper addresses the gap in scalable, generalized methods that can systematically evaluate and compare companies using their SEC 10-K filings, transforming qualitative disclosures into quantitative, comparable metrics.

Core contributions

• A novel data-driven framework leveraging LLMs to systematically analyze and rate company performance based on SEC 10-K filings.
• An automated system for extracting and preprocessing 10-K filings, accurately identifying and segmenting required sections using unstructured-io and regex.
• A quantitative rating system using Cohere’s Command-R+ LLM to assess four key dimensions: confidence, environmental sustainability, innovation, and workforce management.
• A dual-grader approach (standard and strict) to mitigate LLM bias in absolute scoring.
• A relative analysis method using pairwise comparison to benchmark companies against peers in specific filing sections.
• A no-code interactive web application for running the data pipeline and visualizing year-on-year performance comparisons.

Method

• Data Collection: Querying SEC EDGAR endpoints using ticker symbols and CIKs to retrieve 10-K filings.
• Data Cleaning: Using unstructured-io for HTML parsing, POS tagging, and narrative text identification.
• Section Segmentation: Identifying SEC-mandated sections using predefined structures and regex-based fallbacks.
• Absolute Evaluation: Role-based zero-shot prompting with Cohere Command-R+ to assign 0-2 scores for four dimensions.
• Bias Mitigation: Employing two grader roles (standard and strict) and averaging their outputs.
• Relative Evaluation: Pairwise comparison of text excerpts from multiple companies to determine a 'winner' per section.
• Visualization: Generating interactive charts for year-on-year rating trends and comparative performance.

Results & findings

• The system successfully extracts and segments narrative text from 10-K filings, handling variations in document structure.
• Absolute ratings for confidence, environment, innovation, and people are largely uncorrelated, providing distinct insights.
• The dual-grader approach helps reduce bias, with results averaged for final visualization.
• Relative analysis identifies which company 'wins' in specific categories, showing sector leadership trends.
• Year-on-year visualizations reveal shifts in corporate priorities, such as changes in confidence or innovation focus.
• The web application allows non-technical users to input tickers and receive comprehensive, visualized analyses.
• The framework demonstrates scalability and cost-effectiveness compared to traditional manual analysis methods.

Data & experiments

• Data sourced from SEC EDGAR database for 10-K filings of publicly listed companies.
• Case study involving three major companies: Royal Gold ($RGLD), IBM ($IBM), and Apple ($AAPL).
• Absolute analysis generating yearly ratings (0-2) across four dimensions for each company.
• Relative analysis comparing narrative text excerpts from multiple companies across six filing sections.
• Evaluation of rating distributions and correlations between different dimensions.
• Longitudinal analysis tracking year-on-year changes in corporate priorities and strategies.

Limitations & caveats

• Difficulty in accurately identifying and categorizing all relevant sections of 10-K filings may lead to analysis gaps.
• Inherent LLM biases from training data may skew ratings, particularly in sensitive areas like environment or employee relations.
• The subjective nature of qualitative ratings may not fully capture complex corporate realities.
• The framework's effectiveness is demonstrated on only three companies, limiting generalizability.
• The system does not validate whether the generated ratings correlate with actual market performance or stock returns.
• Reliance on a single LLM (Command-R+) may limit the robustness of findings across different model architectures.

Curator notes

Discovery status: report_analysis_focus_promoted. Source links: Financial Report Chunking for Effective Retrieval Augmented Generation; SEC-QA: A Systematic Benchmark for Evaluating Long-Context Question Answering on SEC Filings