FinDABench: Benchmarking Financial Data Analysis Ability of Large Language Models

In one sentence

Summary

The paper addresses the lack of rigorous evaluation for Large Language Models in specialized financial data analysis, distinguishing it from general financial knowledge benchmarks. It introduces FinDABench, a benchmark designed to assess data-driven thinking through three dimensions: Foundational Ability (numerical calculation and sentiment risk assessment), Reasoning Ability (comprehending textual information and analyzing abnormal financial reports), and Technical Skill (generating analysis and visualizations). This framework mirrors the multifaceted workflow of financial analysts, moving beyond simple question-answering to test complex data synthesis and interpretation skills required in real-world scenarios. The benchmark comprises 2,400 instances across six sub-tasks: Numerical Calculations QA, Early Warning Analysis, Fin-report Fraud Detection, Fin-report2Markdown, ChartData2Insight, and NL2ViSQL. Data sources include translated financial reports, scraped news articles, regulatory penalty announcements, and stock exchange filings. The authors evaluate 41 popular LLMs, including English, Chinese, and financial-specific models, in both zero-shot and few-shot settings. Evaluation metrics vary by task type, utilizing Accuracy, F1, Exact Match, and ROUGE scores to measure performance on classification, extraction, and generation tasks, providing a granular view of model strengths and weaknesses across different analytical competencies. Results indicate that while Supervised Fine-Tuning and financial domain knowledge improve performance, significant challenges remain. GPT-4 achieves the highest score of 32.37% in zero-shot settings, with all other models scoring below 30%, demonstrating that current LLMs lack robust financial reasoning and technical skills. The study finds that financial-specific fine-tuning yields substantial gains but does not fully bridge the gap for complex tasks like SQL generation and chart insight extraction. Limitations include the benchmark's focus on Chinese financial contexts for some tasks and the potential for annotation bias in fraud detection, suggesting that further advancements in domain-specific training and reasoning capabilities are necessary for reliable deployment in financial data analysis.

Research problem

The paper identifies a critical gap in evaluating Large Language Models' proficiency in specialized financial data analysis, particularly regarding data-driven thinking. Existing benchmarks primarily test financial knowledge through question-answering, failing to assess the complex skills required for actual data analysis, such as numerical reasoning, anomaly detection, and visualization generation. The research aims to create a comprehensive benchmark that mirrors the multifaceted workflow of financial analysts, evaluating models across foundational, reasoning, and technical skill dimensions to identify specific deficiencies in current LLM capabilities.

Core contributions

• Introduction of FinDABench, the first benchmark comprising six sub-tasks across three dimensions to evaluate financial data analysis capabilities.
• Systematic benchmarking of 41 popular LLMs, providing deep insights into the status quo of LLM development in finance.
• Demonstration that even state-of-the-art models like GPT-4 achieve low performance (32.37% zero-shot), highlighting significant deficiencies.
• Development of a three-tiered evaluation framework inspired by Bloom's Taxonomy and cognitive psychology, categorizing tasks by complexity.
• Release of evaluation scripts and the dataset to foster advancement in financial LLM research and application.

Method

• Constructed a three-tiered evaluation framework: Foundational Ability, Reasoning Ability, and Technical Skill.
• Collected and annotated data from financial reports, news, and regulatory announcements for each sub-task.
• Translated English financial reports to Chinese using GLM-4, ensuring financial terminology accuracy.
• Used GPT-4 for data annotation in tasks like Fin-report2Markdown and ChartData2Insight, followed by manual review.
• Evaluated models using task-specific metrics: Accuracy for single-label tasks, F1 for multi-label classification, EM for SQL generation, and ROUGE for text generation.
• Conducted experiments in both zero-shot and few-shot settings to assess model robustness and adaptability.
• Analyzed the impact of Supervised Fine-Tuning and financial domain knowledge on model performance.

Results & findings

• GPT-4 achieves the highest total score of 32.37% in zero-shot settings, indicating significant room for improvement in financial data analysis.
• All other evaluated models score below 30%, demonstrating the difficulty of the benchmark tasks.
• Supervised Fine-Tuning (SFT) significantly improves model performance, with SFT versions outperforming base models across various sizes.
• Financial-specific fine-tuning yields substantial gains, particularly in fraud detection and numerical calculation tasks.
• Models perform poorly on complex technical skills like NL2ViSQL and ChartData2Insight, suggesting a lack of advanced reasoning capabilities.
• Chinese LLMs generally outperform English LLMs on Chinese financial tasks, highlighting the importance of language-specific training.
• Performance varies significantly across sub-tasks, with models excelling in some areas (e.g., sentiment analysis) but failing in others (e.g., SQL generation).

Data & experiments

• Dataset contains 2,400 instances across six sub-tasks: Numerical Calculations QA, Early Warning Analysis, Fin-report Fraud Detection, Fin-report2Markdown, ChartData2Insight, and NL2ViSQL.
• Data sources include translated financial reports, scraped news articles, regulatory penalty announcements, and stock exchange filings.
• Evaluation of 41 LLMs, categorized into English, Chinese, and Financial-specific models, in zero-shot and few-shot settings.
• Metrics include Accuracy, F1, Exact Match, and ROUGE scores, tailored to the specific task types (classification, extraction, generation).
• In-depth analysis of model performance across different dimensions, highlighting the impact of Supervised Fine-Tuning and domain-specific knowledge.

Limitations & caveats

• The benchmark focuses primarily on Chinese financial contexts for some tasks, limiting generalizability to other markets.
• Annotation bias may exist in fraud detection tasks, as labels are derived from regulatory announcements and expert opinions.
• The dataset size is relatively small (2,400 instances), which may not capture the full diversity of financial data analysis scenarios.
• The evaluation metrics may not fully capture the quality of generated insights, particularly in open-ended tasks like ChartData2Insight.
• The study does not explore the potential of hybrid approaches combining LLMs with traditional financial models or tools.
• The performance of models may vary significantly based on the specific prompt engineering and evaluation settings used.

Curator notes

Useful for end-to-end financial analyst workflow evaluation.