ALFFLLM·FINANCE

The List / Reports, Filings, Accounting, and Risk / paper

Evaluating Large Language Models (LLMs) in Financial NLP: A Comparative Study on Financial Report Analysis

Md Talha Mohsin

2025arXiv.orgcites 3P1arXiv ↗DOI ↗Semantic Scholar ↗Source ↗

Abstract

Large language models (LLMs) are increasingly used to support the analysis of complex financial disclosures, yet their reliability, behavioral consistency, and transparency remain insufficiently understood in high-stakes settings. This paper presents a controlled evaluation of five transformer-based LLMs applied to question answering over the Business sections of U.S. 10-K filings. To capture complementary aspects of model behavior, we combine human evaluation, automated similarity metrics, and behavioral diagnostics under standardized and context-controlled prompting conditions. Human assessments indicate that models differ in their average performance across qualitative dimensions such as relevance, completeness, clarity, conciseness, and factual accuracy, though inter-rater agreement is modest, reflecting the subjective nature of these criteria. Automated metrics reveal systematic differences in lexical overlap and semantic similarity across models, while behavioral diagnostics highlight variation in response stability and cross-prompt alignment. Importantly, no single model consistently dominates across all evaluation perspectives. Together, these findings suggest that apparent performance differences should be interpreted as relative tendencies under the tested conditions rather than definitive indicators of general reliability. The results underscore the need for evaluation frameworks that account for human disagreement, behavioral variability, and interpretability when deploying LLMs in financially consequential applications.

In one sentence

This study evaluates five major LLMs on 10-K Business sections using human, automated, and behavioral metrics, finding no single model dominates and highlighting significant inter-rater disagreement in qualitative assessments.

Summary

This paper addresses the critical need for reliable evaluation frameworks for Large Language Models (LLMs) in high-stakes financial contexts, specifically focusing on the analysis of complex regulatory disclosures. While LLMs are increasingly deployed for interpreting financial reports, their behavioral consistency, transparency, and reliability remain poorly understood. The authors position this work as a controlled comparative pilot study that moves beyond simple accuracy metrics to examine how different models handle nuanced, open-ended questions regarding corporate strategy, risk, and business models found in SEC filings. The study aims to provide a multi-dimensional assessment that captures not just output quality but also the stability and interpretability of model responses under standardized conditions. The experimental design involves evaluating five transformer-based LLMs—GPT-4, Claude 4 Opus, Gemini Pro, Perplexity, and DeepSeek-V2—on the Item 1 (Business) sections of 10-K filings from the "Magnificent 7" tech companies over three years. The authors constructed a benchmark of 21 documents and 10 open-ended interpretative questions per document, resulting in 1,050 model responses. Evaluation was conducted through three complementary axes: human annotation by five experts assessing relevance, completeness, clarity, conciseness, and factual accuracy; automated metrics including ROUGE, Jaccard, and Sentence-BERT cosine similarity; and behavioral diagnostics measuring response stability and cross-prompt alignment. The study employed deterministic sampling where possible and controlled for context leakage by using fresh sessions for each query. Results indicate that no single model consistently outperforms others across all evaluation perspectives. GPT-4 generally received higher human ratings for relevance and clarity, while other models showed strengths in specific areas like conciseness or lexical overlap. However, a key finding is the low inter-rater reliability among human annotators, with Krippendorff’s alpha values often near zero, suggesting that qualitative financial judgments are highly subjective. The study concludes that apparent performance differences should be viewed as relative tendencies rather than definitive indicators of general reliability. This highlights the necessity for evaluation frameworks that account for human disagreement and behavioral variability when deploying LLMs in financially consequential applications, cautioning against over-reliance on any single model for strategic analysis.

Research problem

The paper investigates the reliability, behavioral consistency, and transparency of Large Language Models (LLMs) when applied to the analysis of complex financial disclosures, specifically SEC 10-K filings. It addresses the gap in understanding how different models perform on high-stakes, domain-specific tasks that require nuanced interpretation rather than simple extraction. The core problem is the lack of comprehensive evaluation frameworks that capture not just accuracy but also the stability, interpretability, and subjective quality of model outputs in financial contexts.

Core contributions

  • A controlled comparative evaluation of five major LLMs (GPT-4, Claude 4, Gemini, Perplexity, DeepSeek) on 10-K Business sections.
  • A multi-dimensional evaluation framework combining human expert annotation, automated similarity metrics, and behavioral diagnostics.
  • A novel benchmark dataset of 210 open-ended interpretative questions focused on strategic analysis of tech company filings.
  • Empirical evidence of low inter-rater reliability in human evaluation of financial text, highlighting subjectivity in qualitative assessments.
  • Demonstration that no single model dominates across all metrics, emphasizing the need for context-specific model selection.

Method

  • Data Collection: Extracted Item 1 sections from SEC EDGAR for 7 tech firms over 3 years.
  • Prompt Design: Created 10 open-ended questions focusing on strategy, risk, and business model interpretation.
  • Model Execution: Ran 5 LLMs with deterministic sampling; truncated inputs to ~3,000-3,500 tokens for consistency.
  • Human Evaluation: 5 annotators rated responses on 5 qualitative criteria using a 1-5 scale.
  • Automated Metrics: Computed ROUGE-n, ROUGE-L, Jaccard similarity, and Sentence-BERT cosine similarity against reference responses.
  • Behavioral Diagnostics: Measured cross-model agreement and prompt-level stability via variance in similarity scores.
  • Chain-of-Thought Analysis: Conducted a secondary diagnostic run with CoT prompts to analyze reasoning structure and verbosity.
  • Statistical Analysis: Used Krippendorff’s alpha and ICC(2,1) to assess inter-rater reliability.

Results & findings

  • GPT-4 generally received the highest human ratings for relevance, completeness, and clarity among the evaluated models.
  • Inter-rater reliability was low across all qualitative dimensions, with Krippendorff’s alpha values often near zero or negative.
  • Automated metrics (ROUGE, Jaccard, Cosine) revealed systematic differences in lexical and semantic overlap across models.
  • No single model consistently dominated across all evaluation perspectives (human, automated, behavioral).
  • Claude 4 Opus showed strength in factual accuracy but lower scores in clarity and completeness compared to GPT-4.
  • DeepSeek produced more concise answers but scored lower on relevance and factual accuracy.
  • Behavioral diagnostics highlighted significant variation in response stability and cross-prompt alignment across models.
  • The study suggests that performance differences are relative tendencies under tested conditions, not definitive indicators of general reliability.

Data & experiments

  • Data: Item 1 (Business) sections from 10-K filings of Apple, Microsoft, Amazon, Alphabet, Nvidia, Meta, and Tesla (2022-2024).
  • Dataset: 21 documents, 10 open-ended questions per document, yielding 210 responses per model and 1,050 total responses.
  • Models Evaluated: GPT-4, Claude 4 Opus, Gemini Pro, Perplexity, and DeepSeek-V2.
  • Evaluation Metrics: Human ratings (Relevance, Completeness, Clarity, Conciseness, Factual Accuracy), ROUGE, Jaccard, Sentence-BERT cosine similarity.
  • Behavioral Diagnostics: Cross-model agreement and prompt-level stability measured via variance in similarity scores.
  • Experimental Setup: Deterministic sampling where possible; fresh sessions for each query to prevent context leakage.

Limitations & caveats

  • The dataset is limited to 7 large tech companies, which may not represent the broader financial reporting landscape.
  • Inter-rater reliability among human annotators was low, indicating high subjectivity in qualitative financial assessments.
  • The study focuses on the Business section (Item 1) of 10-Ks, excluding other critical sections like Risk Factors or Financial Statements.
  • Results are based on a specific set of 10 questions, which may not capture the full range of analytical tasks.
  • The evaluation does not include fine-tuned financial LLMs, only general-purpose models accessed via API.
  • The study is a pilot evaluation, and the authors note the need for larger and more diverse corpora in future work.

Curator notes

Discovery status: report_analysis_focus_promoted. Source links: Financial Statement Analysis with Large Language Models

← previousBenchmarking large language models for supply chain risk identification: an extended evaluation within the LARD-SC framework