Language Models Fine-Tuning for Automatic Format Reconstruction of SEC Financial Filings

In one sentence

Summary

The research addresses the growing heterogeneity in SEC 10-K filing formats, which complicates automated knowledge extraction and comparative analysis. While the SEC provides a standardized structure, many companies deviate, creating noise for investors and regulators. The authors frame document format reconstruction as a multi-class sentence-level classification task with 18 classes, aiming to map unstructured paragraphs to their correct SEC item sections. This standardization is critical for enabling consistent semantic comparison across firms, supporting tasks like portfolio optimization and risk assessment that rely on aligned textual data. The work positions itself within the broader context of NLP for finance, specifically targeting the data preprocessing bottleneck that hinders effective use of large language models in regulatory compliance and investment research. The methodology involves fine-tuning pre-trained transformer models, including BERT, RoBERTa, and XLNet, using a bidirectional fine-tuning procedure with Bi-LSTM layers. The dataset comprises approximately 9.54 million paragraphs from 43,608 10-K filings of 6,000 US public companies between 2011 and 2022. Data splitting ensures mutual exclusion of companies across train, validation, and test sets to prevent data leakage from copy-pasted content. The best-performing model, SEC-former, is evaluated on topic detection accuracy, document similarity using TF-IDF and Doc2Vec embeddings, and a real-world case study on a non-compliant filing. Baselines include XGBoost and Bi-LSTM models using TF-IDF features, highlighting the superiority of contextual embeddings for this structural task. Results show that SEC-former significantly outperforms baselines in classification accuracy and produces document embeddings that better capture semantic similarity compared to traditional bag-of-words approaches. The model successfully reconstructs the structure of a real-world non-compliant filing, validating its practical utility. The improved document similarity metrics suggest that SEC-former can enhance downstream applications like portfolio optimization by providing more accurate semantic representations of annual reports. However, limitations include the heavy class imbalance in the dataset, the exclusion of rarely filled items, and the reliance on regex-based labeling which may introduce noise. The model's effectiveness is also constrained by the static nature of the 18-class schema, which may not capture all nuances of evolving reporting standards.

Research problem

The paper addresses the challenge of analyzing SEC 10-K filings, which are increasingly formatted in non-standard ways by companies, making automated knowledge extraction and comparative analysis difficult. The specific problem is to automatically reconstruct the standard SEC structure (18 items) from unstructured or semi-structured text to facilitate downstream financial analysis tasks like portfolio optimization and risk assessment.

Core contributions

• Investigation of transformer-based neural language models for automatic SEC 10-K format reconstruction as a multi-class classification task.
• Proposal of a bidirectional fine-tuning procedure combining pre-trained transformers with Bi-LSTM networks to specialize models for this specific task.
• Creation and public release of SEC-former, a novel transformer model optimized for document format reconstruction.
• Evaluation of the model in three scenarios: topic detection, document similarity for portfolio optimization, and real-world reconstruction of non-compliant filings.

Method

• Multi-class classification of paragraphs into 18 SEC item categories.
• Fine-tuning of pre-trained transformers (BERT, RoBERTa, XLNet) with bidirectional Bi-LSTM layers.
• Data preprocessing including HTML tag removal, tokenization, and handling of class imbalance via under-sampling.
• Evaluation using accuracy, precision, recall, and F1-score on balanced and imbalanced test sets.
• Document similarity assessment using TF-IDF and Doc2Vec embeddings.
• Real-world case study on a non-compliant filing to test generalization.

Results & findings

• SEC-former outperforms baseline models (XGBoost, Bi-LSTM) in multi-class classification accuracy for item identification.
• Document similarity analysis using Doc2Vec embeddings from SEC-former shows better semantic alignment with original reports compared to TF-IDF.
• The model successfully reconstructs the structure of a real-world non-compliant filing, demonstrating practical utility.
• Topic detection performance improves with the standardized format, aiding in consistent information retrieval.
• The study highlights the increasing length and complexity of 10-K filings over time, necessitating robust NLP solutions.
• Class imbalance is a significant challenge, with some items (e.g., Item 1B) having very few samples.

Data & experiments

• Dataset of 43,608 10-K filings from 6,000 US public companies (NYSE and Nasdaq) published between 2011 and 2022.
• Approximately 9.54 million paragraphs extracted and labeled into 18 SEC item classes using regex and table of contents parsing.
• Data splitting based on mutual company exclusion to prevent data leakage from copy-pasted content across years.
• Evaluation metrics include accuracy, precision, recall, and F1-score, with specific handling for class imbalance via under-sampling.
• Comparison against baseline models: XGBoost with TF-IDF and Bidirectional LSTM.

Limitations & caveats

• Heavy class imbalance in the dataset, with some SEC items having very few samples.
• Reliance on regex-based labeling, which may introduce noise or errors in the ground truth.
• Exclusion of rarely filled items (Item 1B, Item 14) limits the comprehensiveness of the model.
• The static 18-class schema may not capture all nuances of evolving reporting standards.
• The model's performance may degrade on filings with extreme deviations from the standard format.
• Evaluation is limited to US public companies, potentially limiting generalizability to other markets.

Curator notes

Discovery status: report_analysis_focus_promoted. Source links: Evaluating Large Language Models (LLMs) in Financial NLP: A Comparative Study on Financial Report Analysis