SECQUE: A Benchmark for Evaluating Question-Answering on SEC Filings

In one sentence

Summary

The paper addresses the lack of domain-specific evaluation for LLMs in real-world financial analysis, noting that existing benchmarks often fail to capture the complexity of tasks financial analysts perform daily. SECQUE is introduced to fill this gap by providing a comprehensive dataset focused on SEC filings, specifically 10-K and 10-Q reports, which are central to corporate financial disclosure. The benchmark is designed to test models on their ability to comprehend, reason over, and synthesize information from lengthy, multi-format documents, moving beyond simple text processing to assess nuanced financial reasoning capabilities. The dataset comprises 565 questions curated by subject matter experts, covering four categories: comparison and trend analysis, ratio calculation, risk assessment, and financial insight generation. It includes 45 unique SEC filings from 29 companies, with contexts provided in both HTML and Markdown formats to test robustness. The authors developed SECQUE-Judge, an automated evaluation mechanism using a panel of LLM judges (primarily GPT-4o) that aggregates multiple scoring instances to align with human expert evaluations. This judge was validated against human scores, demonstrating high precision in identifying correct answers. The benchmark was used to evaluate seven diverse LLMs, including GPT-4o, Llama-3.3-70B, and smaller models like Phi-4 and Mistral-Nemo. Results indicate that while leading models perform reasonably well on risk factors and basic ratios, significant challenges persist in complex reasoning and insight generation. GPT-4o achieved the highest strict accuracy (0.69) and normalized accuracy (0.79), outperforming open-source models like Llama-3.3-70B. The study found that text representation (HTML vs. Markdown) had a minor impact, whereas prompt engineering significantly affected performance, with baseline prompts often outperforming more specific financial prompts. Limitations include potential biases in LLM-based judging and the need for broader coverage of document types. The benchmark is publicly available to facilitate further research in financial AI.

Research problem

The paper identifies a critical gap in evaluating Large Language Models (LLMs) for specialized financial tasks. While general benchmarks exist, they often fail to capture the nuances of real-world financial analysis, such as reasoning over long, unstructured SEC filings (10-K/10-Q) and performing complex numerical computations. The authors argue that current evaluations are insufficient for assessing the capabilities required by professional financial analysts, necessitating a domain-specific benchmark that reflects the complexity and high-stakes nature of actual financial work.

Core contributions

• Introduction of SECQUE, a novel benchmark with 565 expert-written questions focused on SEC filing analysis.
• Development of SECQUE-Judge, an automated evaluation framework using a panel of LLM judges that aligns strongly with human expert scoring.
• Comprehensive evaluation of seven state-of-the-art LLMs across four distinct financial reasoning categories.
• Detailed ablation studies on the impact of text representation (HTML vs. Markdown), prompt engineering, and temperature settings on model performance.
• Public release of the benchmark dataset and evaluation code to facilitate future research in financial AI.

Method

• Curated 565 questions from SEC filings using three financial subject matter experts and two LLM system experts.
• Defined four question categories: Risk Factors, Ratio Analysis, Comparison/Trend Analysis, and Analyst Insights.
• Preprocessed SEC filings into HTML and Markdown formats, with and without headers, to create multiple context versions.
• Developed SECQUE-Judge using a panel of five GPT-4o judges, aggregating scores with thresholds (UT=6, LT=4) to map to 0-2 scale.
• Validated SECQUE-Judge against human expert evaluations on a subset of 124 answers, optimizing for F1(2) score.
• Evaluated seven LLMs using baseline, financial, and CoT prompts, measuring Strict and Normalized Accuracy.
• Conducted ablation studies on text representation, prompt types, and temperature settings to analyze performance drivers.

Results & findings

• GPT-4o achieved the highest performance with 0.69 Strict Accuracy and 0.79 Normalized Accuracy.
• Llama-3.3-70B-Instruct performed closely behind GPT-4o, while smaller models (e.g., Mistral-Nemo) lagged significantly.
• Risk Factor questions were the easiest category, with most models achieving high scores, suggesting it is a baseline requirement.
• Analyst Insight questions were the most difficult, with GPT-4o significantly outperforming others, highlighting gaps in deep financial synthesis.
• Ratio Analysis and Comparison questions showed moderate difficulty, requiring strong numerical reasoning and context extraction.
• Prompt engineering had a larger impact on performance than text representation format (HTML vs. Markdown).
• Chain-of-Thought prompting did not consistently improve performance and sometimes decreased it compared to baseline prompts.
• SECQUE-Judge achieved an F1(2) score of 0.85 against human evaluators, demonstrating strong alignment for automated assessment.

Data & experiments

• Dataset: 565 questions derived from 45 unique SEC filings (10-K and 10-Q) across 29 companies from 2018-2024.
• Question Types: Comparison/Trend Analysis (220), Ratio Analysis (188), Risk Factors (85), Analyst Insights (72).
• Models Evaluated: GPT-4o, GPT-4o-mini, Llama-3.3-70B-Instruct, Llama-3.1-8B-Instruct, Qwen2.5-32B-Instruct, Mistral-Nemo-Instruct-2407, Phi-4.
• Evaluation Method: SECQUE-Judge using a panel of 5 GPT-4o judges with threshold-based aggregation, validated against human experts.
• Metrics: Strict Accuracy (binary correct/incorrect) and Normalized Accuracy (0-2 scale), plus token usage analysis.
• Ablations: Tested HTML vs. Markdown table representations, baseline vs. financial prompts, with and without Chain-of-Thought.

Limitations & caveats

• Potential bias in LLM-based judging, though mitigated by human validation.
• Limited coverage of financial document types (only 10-K and 10-Q).
• Some questions may have multiple valid interpretations or calculation methods.
• Smaller models like Phi-4 have context length limitations affecting performance.
• Human evaluation was limited to a subset of 124 answers due to cost.
• Benchmark focuses on US SEC filings, limiting generalizability to other markets.

Curator notes

Includes expert-authored questions and SECQUE-Judge evaluation.