Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? A Study on Several Typical Tasks

In one sentence

Summary

The paper addresses the critical question of whether general-purpose large language models like ChatGPT and GPT-4 can serve as effective solvers for financial text analytics without domain-specific fine-tuning. It positions these models against state-of-the-art domain-specific pretrained models (e.g., BloombergGPT) and task-specific fine-tuned approaches to evaluate their transferability and robustness in the financial sector. The study aims to provide empirical evidence on the strengths and limitations of current LLMs to guide their adoption in downstream financial analytical tasks. The methodology involves an extensive empirical evaluation across five categories of financial NLP tasks: sentiment analysis, text classification, named entity recognition (NER), relation extraction (RE), and question answering (QA). The authors utilize eight benchmark datasets, including Financial PhraseBank, FiQA, TweetFinSent, Headlines, NER FIN3, REFinD, FinQA, and ConvFinQA. Experiments compare zero-shot, few-shot, and Chain-of-Thought (CoT) prompting strategies for ChatGPT (gpt-3.5-turbo) and GPT-4 against baselines like FinBert, RoBERTa-large, Luke-base, and BloombergGPT. Metrics include accuracy, macro-F1, and weighted F1 scores, with additional ablation studies on prompt sensitivity and reasoning complexity. Findings indicate that GPT-4 significantly outperforms ChatGPT and prior LLMs on nearly all tasks, often surpassing domain-specific models like BloombergGPT in sentiment analysis and numerical QA, particularly when using CoT prompting. However, generalist LLMs struggle with structured prediction tasks like NER and RE, where fine-tuned models (e.g., CRF, Luke-base) remain superior. The study highlights that while LLMs show strong reasoning capabilities, they still make critical numerical errors in complex multi-step calculations and lack the precision required for high-stakes financial information extraction, suggesting a hybrid approach or continued fine-tuning is necessary for complex applications.

Research problem

The paper investigates the effectiveness of general-purpose large language models (ChatGPT and GPT-4) as solvers for a wide variety of financial text analytics tasks. It seeks to determine if these models can replace or outperform domain-specific pretrained models and task-specific fine-tuned approaches without additional adaptation. The core problem is understanding the transferability of generalist LLM capabilities to the specialized financial domain, which requires handling diverse tasks ranging from simple sentiment analysis to complex numerical reasoning and structured information extraction.

Core contributions

• Provides one of the first comprehensive empirical comparisons of ChatGPT and GPT-4 against domain-specific models (BloombergGPT) and fine-tuned baselines across eight financial NLP datasets.
• Demonstrates that generalist LLMs, particularly GPT-4, outperform domain-specific pretrained models in sentiment analysis and numerical question answering, challenging the necessity of domain pretraining for these tasks.
• Identifies specific limitations of generalist LLMs in structured prediction tasks like Named Entity Recognition (NER) and Relation Extraction (RE), where fine-tuned models remain superior.
• Quantifies the impact of prompt engineering strategies, showing that Chain-of-Thought (CoT) prompting significantly boosts performance on complex financial QA tasks for both ChatGPT and GPT-4.
• Analyzes the performance gap between ChatGPT and GPT-4, highlighting that GPT-4's improvements in reasoning and scale lead to substantial gains in challenging financial tasks.

Method

• Empirical benchmarking of ChatGPT (gpt-3.5-turbo) and GPT-4 (8k/16k) on eight financial NLP datasets.
• Comparison against domain-specific pretrained model BloombergGPT and task-specific fine-tuned models (FinBert, RoBERTa-large, Luke-base, CRF).
• Implementation of zero-shot, few-shot, and Chain-of-Thought (CoT) prompting strategies to evaluate in-context learning and reasoning capabilities.
• Use of standard evaluation metrics: accuracy, macro-F1, and weighted F1, with entity-level F1 for NER tasks.
• Ablation studies on the impact of emoji removal in social media sentiment analysis (TweetFinSent).
• Analysis of reasoning complexity by categorizing FinQA problems into 1-step, 2-step, and multi-step calculations.
• Assessment of model performance on conversational QA (ConvFinQA) using a turn-based approach to simulate multi-round interactions.

Results & findings

• GPT-4 significantly outperforms ChatGPT and prior LLMs on nearly all financial benchmarks, with performance gains of 10-100% on challenging tasks like RE and QA.
• In sentiment analysis (e.g., FiQA), GPT-4 (88.11% F1) outperforms the fine-tuned RoBERTa-large (87.09%) and BloombergGPT (75.07%).
• For numerical QA (FinQA), GPT-4 with CoT achieves 78.03% accuracy, surpassing the fine-tuned FinQANet (68.90%) and approaching human expert performance (91.16%).
• Generalist LLMs underperform in structured prediction tasks; GPT-4 achieves only 56.71% F1 on NER, while a fine-tuned CRF model reaches 82.70%.
• In Relation Extraction (REFinD), the fine-tuned Luke-base model (56.30% Macro F1) significantly outperforms GPT-4 (46.87% Macro F1).
• Chain-of-Thought prompting provides a massive lift, improving ChatGPT by ~10% and GPT-4 by ~15% on FinQA compared to zero-shot/few-shot baselines.
• ChatGPT exhibits basic numerical errors in multi-step calculations (e.g., summation errors in ConvFinQA), indicating limitations in complex reasoning despite high overall scores.
• Few-shot learning provides modest gains (1-4%) for ChatGPT and GPT-4, with diminishing returns after a small number of shots for simpler tasks.

Data & experiments

• Evaluates models on eight benchmark datasets spanning five task categories: Sentiment Analysis (Financial PhraseBank, FiQA, TweetFinSent), Text Classification (Headlines), NER (NER FIN3), Relation Extraction (REFinD), and Question Answering (FinQA, ConvFinQA).
• Compares ChatGPT (gpt-3.5-turbo) and GPT-4 (8k/16k) against domain-specific models (BloombergGPT) and fine-tuned baselines (FinBert, RoBERTa-large, Luke-base, CRF).
• Tests zero-shot, few-shot, and Chain-of-Thought (CoT) prompting strategies to assess the impact of in-context learning and reasoning steps.
• Uses evaluation metrics including accuracy, macro-F1, and weighted F1 scores, with specific entity-level F1 for NER.
• Conducts ablation studies on prompt sensitivity (number of shots) and reasoning complexity (1-step vs. multi-step calculations in FinQA).

Limitations & caveats

• The study is limited to text-based tasks and does not evaluate multimodal or code-generation capabilities in finance.
• Performance on ConvFinQA is evaluated on the dev set due to the unavailability of the test set labels.
• The NER dataset (FIN3) is small (98 test samples), which may limit the statistical significance of results for structured prediction tasks.
• The study does not address the cost, latency, or deployment challenges of using large LLMs in production financial systems.
• Results are based on specific versions of ChatGPT and GPT-4 as of July 2023, which may not reflect the latest model capabilities.
• The evaluation of relation extraction uses a subset of relation types and may not capture the full complexity of financial knowledge graphs.

Curator notes

Discovery status: round1_promoted_seed_for_round2. Source links: A Comprehensive Review of Generative AI in Finance; A Preliminary Look at the State of the Art of Large Language Models on Chartered Financial Analyst Exams; A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges; A Survey of Large Language Models in Finance: FinLLMs; BloombergGPT: A Large Language Model for Finance; Can Large Language Models Tackle the Chartered Financial Analyst Exam?; Can Large Language Models beat wall street? Evaluating GPT-4’s impa