FinBen: A Holistic Financial Benchmark for Large Language Models

In one sentence

Summary

The paper addresses the lack of comprehensive evaluation benchmarks for Large Language Models (LLMs) in the financial domain, where existing tools are limited to narrow NLP tasks. The authors introduce FinBen, a holistic benchmark comprising 36 datasets across 24 tasks, categorized into seven aspects: information extraction, textual analysis, question answering, text generation, risk management, forecasting, and decision-making. This framework aims to provide a robust assessment of LLM capabilities in diverse and complex financial scenarios, including the first evaluation of stock trading and agent-based RAG strategies. The methodology involves evaluating 15 representative general and financial LLMs, such as GPT-4, Gemini, and LLaMA variants, using zero-shot and few-shot settings. The benchmark includes novel datasets like FinTrade for stock trading, EDTSum for news summarization, and Regulations for legal QA. Experiments were conducted on NVIDIA A100 GPUs, with results measured using standard metrics like F1, Accuracy, ROUGE, and financial performance indicators such as Sharpe Ratio and Cumulative Return. The study also hosted a shared task at IJCAI-2024 to test community-developed solutions. Key findings indicate that LLMs perform well in information extraction and textual analysis but fail in advanced reasoning tasks like forecasting and complex text generation. GPT-4 excels in IE and stock trading, achieving a Sharpe Ratio over 1, while Gemini outperforms others in text generation and forecasting. Instruction-tuned financial LLMs show improvements in specific tasks but lack generalization. The paper highlights limitations in dataset size, model scale, and generalizability to non-US markets, noting that LLMs still lag behind traditional methods in forecasting and face challenges with imbalanced risk management data.

Research problem

The financial domain lacks comprehensive evaluation benchmarks for LLMs, with existing tools focusing narrowly on basic NLP tasks. This gap hinders the understanding of LLM capabilities in complex financial applications like forecasting, risk management, and trading. The paper aims to bridge this gap by introducing a holistic benchmark that covers a broader range of tasks and evaluation strategies.

Core contributions

• Introduction of FinBen, the first extensive open-source benchmark with 36 datasets across 24 financial tasks.
• Development of a novel taxonomy categorizing financial LLM evaluation into seven critical aspects.
• Creation of three novel datasets: EDTSum for summarization, Regulations for legal QA, and FinTrade for stock trading.
• First evaluation of LLMs on stock trading and agent-based RAG strategies.
• Systematic evaluation of 15 representative LLMs, providing detailed performance comparisons.
• Hosting of the first financial LLMs shared task at IJCAI-2024, fostering community innovation.

Method

• Taxonomy-based organization of tasks into seven financial aspects.
• Zero-shot and few-shot evaluation of LLMs on diverse datasets.
• Use of standard NLP metrics (F1, Accuracy, ROUGE) and financial metrics (Sharpe Ratio, CR).
• Agent-based evaluation for stock trading using the FinMem agent.
• Retrieval-Augmented Generation (RAG) evaluation strategies.
• Comparison with traditional machine learning models for baseline performance.

Results & findings

• GPT-4 excels in information extraction and stock trading, achieving a Sharpe Ratio over 1.
• Gemini outperforms other models in text generation and forecasting tasks.
• LLMs struggle with complex reasoning, forecasting, and advanced information extraction like causal detection.
• Instruction-tuned financial LLMs improve textual analysis but show limited gains in complex QA and generation.
• All LLMs perform slightly better than random guessing in stock movement prediction, lagging behind traditional methods.
• Open-source models generally underperform closed-source commercial LLMs in financial tasks.
• Community solutions in the shared task outperformed GPT-4 in specific subtasks, demonstrating benchmark utility.

Data & experiments

• 36 datasets spanning 24 tasks, including 3 novel datasets created by the authors.
• Evaluation of 15 LLMs, including GPT-4, Gemini, LLaMA2/3, and financial-specific models.
• Experiments conducted on 16 NVIDIA A100 80G GPUs with zero-shot and few-shot settings.
• Metrics include F1, Accuracy, ROUGE, BERTScore, and financial metrics like Sharpe Ratio and Cumulative Return.
• Comparison with traditional non-LLM methods for stock movement prediction and NLP tasks.
• Shared task evaluation with 12 teams, testing classification, summarization, and trading.

Limitations & caveats

• Dataset sizes are limited, potentially affecting generalization.
• Evaluation was restricted to LLaMA 70B due to computational constraints.
• Tasks are based on American market data and English texts, limiting global applicability.
• LLMs struggle with imbalanced data in risk management tasks.
• Potential for misuse, such as propagating financial misinformation.
• Instruction-tuned models may overfit to specific tasks, reducing generalization.

Curator notes

Also relevant for agent, stock trading, and Retrieval-Augmented Generation evaluation.