A Survey of Large Language Models in Finance: FinLLMs

In one sentence

Summary

The paper addresses the gap in comprehensive reviews of Financial Large Language Models (FinLLMs) by providing a holistic survey of their development, techniques, and applications. It positions FinLLMs as an emerging field distinct from general-domain LLMs, tracing the chronological evolution from early Pre-trained Language Models (PLMs) like BERT to modern generative models. The authors aim to consolidate scattered research into a unified framework, covering the transition from discriminative financial PLMs to generative FinLLMs, and highlighting the specific challenges of applying LLMs to the financial domain, such as data privacy, hallucination, and the need for domain-specific evaluation metrics. This work serves as a foundational reference for researchers and practitioners seeking to understand the landscape of FinLLMs, their capabilities, and their limitations in real-world financial applications. The methodology involves a systematic comparison of four financial PLMs (FinBERT-19/20/21, FLANG) and four FinLLMs (BloombergGPT, FinMA, InvestLM, FinGPT). The authors analyze five key techniques: continual pre-training, domain-specific pre-training from scratch, mixed-domain pre-training, mixed-domain LLMs with prompt engineering, and instruction fine-tuned LLMs. They evaluate these models across six standard financial NLP benchmarks: Sentiment Analysis, Text Classification, Named Entity Recognition, Question Answering, Stock Movement Prediction, and Text Summarization. The experimental design includes reviewing performance metrics (F1, Accuracy, ROUGE) and comparing FinLLMs against task-specific State-of-the-Art (SOTA) models and general-purpose models like GPT-4. Additionally, the paper introduces eight advanced financial NLP tasks and associated datasets to address gaps in current evaluation suites, such as Relation Extraction, Event Detection, and Multimodal understanding. Key findings indicate that while general-domain LLMs like GPT-4 show impressive performance on complex tasks like Question Answering and Stock Movement Prediction, task-specific SOTA models often still outperform FinLLMs in specialized areas like Summarization. The survey highlights that mixed-domain pre-training and instruction fine-tuning are critical for enhancing FinLLM capabilities. Significant challenges identified include the scarcity of high-quality, multimodal financial datasets, the risk of hallucination, privacy concerns with proprietary data, and the lack of expert-driven evaluation metrics. The paper concludes by emphasizing the need for Retrieval Augmented Generation (RAG) to improve reliability and the importance of developing robust FinLLM frameworks for applications in robo-advisory, quantitative trading, and document understanding, while noting that current FinLLMs are not yet ready to fully replace human experts in high-stakes financial decision-making.

Research problem

The paper addresses the lack of a comprehensive survey on Financial Large Language Models (FinLLMs), which are at an early stage compared to general-domain LLMs. It seeks to clarify the evolution, techniques, and performance of FinLLMs in the context of financial NLP tasks. The problem includes understanding how general LLMs can be adapted for finance, the specific challenges of financial data (privacy, complexity), and the need for standardized evaluation benchmarks.

Core contributions

• Provides the first comprehensive survey of FinLLMs, tracing evolution from general-domain PLMs to financial-domain models.
• Compares five distinct training techniques used across four financial PLMs and four FinLLMs, including pre-training and fine-tuning strategies.
• Summarizes performance evaluations of six benchmark tasks and datasets, comparing FinLLMs against SOTA models and GPT-4.
• Introduces eight advanced financial NLP tasks and associated datasets to address gaps in current evaluation suites.
• Discusses opportunities and challenges in FinLLMs, including hallucination, privacy, efficiency, and real-world application barriers.

Method

• Chronological review of general-domain LLMs and their adaptation to the financial domain.
• Categorization of FinLLMs based on training techniques: continual pre-training, domain-specific pre-training, mixed-domain pre-training, prompt engineering, and instruction fine-tuning.
• Systematic evaluation of models on six benchmark tasks using standard NLP metrics (F1, Accuracy, ROUGE).
• Comparison of FinLLMs against task-specific SOTA models and general-purpose LLMs (GPT-4).
• Identification of gaps in current benchmarks and proposal of eight advanced financial NLP tasks.
• Analysis of challenges in datasets, techniques, evaluation, implementation, and real-world applications.

Results & findings

• GPT-4 with zero-shot prompting outperforms most FinLLMs on Question Answering (69-76% EM Accuracy) and Stock Movement Prediction (54% Accuracy).
• Task-specific SOTA models still outperform FinLLMs in Text Summarization (ROUGE-1) and some Sentiment Analysis tasks.
• FLANG-ELECTRA achieves the best Sentiment Analysis results (92% F1) among traditional PLMs.
• BloombergGPT, trained on 363B financial tokens, shows competitive performance but is closed-source.
• FinMA and InvestLM demonstrate the effectiveness of instruction fine-tuning on LLaMA backbones for financial tasks.
• Current NLP metrics (F1, Accuracy) are insufficient for evaluating financial tasks; financial metrics like Sharpe ratio are needed.
• Hallucination and privacy concerns remain significant barriers to deploying FinLLMs in real-world applications.

Data & experiments

• Evaluates four financial PLMs (FinBERT-19/20/21, FLANG) and four FinLLMs (BloombergGPT, FinMA, InvestLM, FinGPT).
• Uses six standard financial NLP benchmarks: Sentiment Analysis, Text Classification, NER, QA, Stock Movement Prediction, and Summarization.
• Compares model performance against task-specific SOTA models and general-purpose LLMs like GPT-4.
• Introduces eight advanced tasks with datasets: Relation Extraction, Event Detection, Causality Detection, Numerical Reasoning, Structure Recognition, Multimodal, Machine Translation, and Market Forecasting.
• Analyzes training data sources including SEC filings, news, earnings calls, and proprietary Bloomberg data.

Limitations & caveats

• The survey focuses primarily on English-language financial models and datasets.
• Some FinLLMs (e.g., BloombergGPT) are closed-source, limiting reproducibility and detailed analysis.
• Current evaluation metrics (F1, Accuracy) may not fully capture the financial utility of model outputs.
• The rapid evolution of LLMs means some models and benchmarks may become outdated quickly.
• The paper acknowledges the lack of high-quality, multimodal financial datasets for training advanced FinLLMs.
• Real-world deployment is hindered by privacy, security, and regulatory concerns not fully addressed by technical solutions alone.

Curator notes

Covers hallucination, privacy, efficiency, financial NLP tasks, and financial-domain models.