FinTradeBench: A Comprehensive Benchmark for Fundamental and Technical Analysis in Financial Trading

In one sentence

Summary

The paper addresses the limitation of existing financial benchmarks that focus primarily on textual accounting data, ignoring the critical integration of market dynamics. It introduces FinTradeBench, a benchmark comprising 1,400 questions grounded in NASDAQ-100 companies from 2015 to 2025. The benchmark is structured into three categories: fundamentals-focused, trading-signal-focused, and hybrid questions requiring cross-signal reasoning. The authors employ a calibration-then-scaling framework, using expert seed questions, multi-model response generation, numerical auditing, and human-LLM judge alignment to ensure high-quality annotations. This approach allows for a rigorous evaluation of how well LLMs can synthesize heterogeneous financial signals. The experimental setup involves evaluating 14 LLMs, ranging from small open-source models to large proprietary ones, under both zero-shot and retrieval-augmented generation (RAG) settings. The RAG architecture features a dual-track retrieval engine: one track indexes SEC filings using hierarchical chunking and dense embeddings, while the other indexes historical price data (OHLCV) for trading signals. The study measures performance using absolute accuracy, relative retrieval delta, and golden indicator F1 scores. The authors also conduct ablation studies and qualitative case studies to analyze the impact of context quality and model architecture on reasoning depth and factual precision. Key findings indicate a clear performance gap between fundamental and trading reasoning. RAG substantially improves accuracy on fundamentals-focused questions (up to +37%) by grounding models in textual disclosures. However, RAG degrades performance on trading-signal questions (up to -19.7%) because LLMs struggle to compute technical indicators from raw numerical tables, leading to distraction and hallucination. Hybrid questions show the highest gains for reasoning-capable models (e.g., DeepSeek-R1), highlighting the need for latent chain-of-thought capabilities. The study concludes that current LLMs lack the representational framework for quantitative market data and that pre-computed signals are necessary for effective reasoning, suggesting future work should focus on code execution or specialized numerical processing rather than pure text retrieval.

Research problem

Existing financial QA benchmarks primarily evaluate reasoning over textual accounting data (e.g., balance sheets, SEC filings) but rarely assess the ability of LLMs to reason over trading signals derived from price dynamics or to integrate both sources of information. This gap leaves it unclear whether current LLMs can perform the joint reasoning required for real-world financial analysis, which involves synthesizing company fundamentals with market behavior.

Core contributions

• Introduction of FinTradeBench, a novel benchmark integrating company fundamentals and trading signals for financial reasoning evaluation.
• Development of a calibration-then-scaling framework that combines expert seed questions, multi-model response generation, numerical auditing, and human-LLM judge alignment to ensure scalable and reliable annotations.
• Design of a dual-track RAG architecture that separately indexes textual SEC filings and numerical time-series market data to support heterogeneous information retrieval.
• Comprehensive evaluation of 14 LLMs across different sizes and architectures, revealing significant performance gaps between fundamental and trading signal reasoning.
• Qualitative and quantitative analysis demonstrating that RAG improves fundamental reasoning but harms trading signal reasoning due to LLMs' inability to parse raw numerical tables.

Method

• Calibration-then-scaling framework: Uses 150 expert-authored seed questions to calibrate an LLM judge against human annotations.
• Multi-model candidate generation: Generates multiple responses per question using different prompt templates from the TELeR taxonomy.
• Intra-model self-filtering: Each model selects its best response based on factual accuracy and reasoning completeness.
• Numerical auditing: An independent LLM auditor checks numerical claims against a financial knowledge base.
• Dual-track retrieval: Track A indexes SEC filings using hierarchical chunking; Track B indexes price data for temporal retrieval.
• TELeR-guided generation: Uses structured reasoning prompts to guide model responses.
• Human-LLM judge alignment: Calibrates the LLM judge using a 5-point Likert scale rubric on four criteria: accuracy, completeness, relevance, and clarity.
• Statistical testing: Uses paired t-tests to assess the significance of RAG-induced performance changes.

Results & findings

• RAG improves fundamental reasoning accuracy by up to +37% (e.g., R1-Distill-Qwen 32B) and hybrid reasoning by up to +55%.
• RAG degrades trading-signal reasoning accuracy by up to -19.7% (e.g., Gemini 2.5 Flash-Lite, GPT-5-mini).
• Golden Indicator F1 drops by 56.5% under RAG, indicating that dense context harms precise metric extraction despite improving surface-level grounding.
• Reasoning-depth models (e.g., DeepSeek-R1) significantly outperform instruction-tuned models on hybrid questions, with DeepSeek-R1 achieving 46.4% accuracy under RAG.
• LLaMA models show systematic degradation under RAG (e.g., LLaMA 3.3 Instruct 70B drops 9.5%), suggesting susceptibility to distraction from dense financial text.
• Ideal RAG (precomputed context) allows mid-tier models to reason effectively, highlighting that the bottleneck is context structure, not just model capability.
• Human-LLM judge alignment is strong, with MAE <10% on the seed set, validating the automated evaluation pipeline.

Data & experiments

• Dataset: FinTradeBench contains 1,400 questions grounded in NASDAQ-100 companies over a ten-year window (2015-2025).
• Data Sources: Company fundamentals extracted from SEC 10-K/10-Q filings; trading signals computed from daily OHLCV price data.
• Question Taxonomy: Questions are categorized into fundamentals-focused (F), trading-signal-focused (T), and hybrid (FT) reasoning types.
• Models Evaluated: 14 LLMs, including large proprietary models (e.g., DeepSeek-R1, Gemini 2.5 Flash) and open-source models (e.g., LLaMA 3.3, Qwen 2.5).
• Experimental Settings: Zero-shot prompting and retrieval-augmented generation (RAG) with a dual-track retrieval engine.
• Metrics: Absolute accuracy, relative retrieval delta, golden indicator F1, and integration scores for fundamental and trading signals.

Limitations & caveats

• Market coverage is limited to NASDAQ-100 companies, potentially missing nuances in other sectors or asset classes.
• Signal coverage is curated and may not include all relevant financial indicators, such as macroeconomic variables or alternative data.
• Temporal generalization is limited to historical data (2015-2025), and performance on future data may vary.
• Evaluation relies on an LLM judge, which may not fully replicate the nuanced judgments of professional financial analysts.
• Ideal RAG results are based on manually curated context and may not be replicable with current automated pipelines.
• The benchmark focuses on question answering and does not evaluate portfolio optimization or trading strategy generation.

Curator notes

Useful for comparing RAG benefits across fundamental and technical tasks.