Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? An Examination on Several Typical Tasks

In one sentence

Summary

The paper addresses the critical question of whether general-purpose large language models like ChatGPT and GPT-4 can serve as effective solvers for financial text analytics without domain-specific fine-tuning. It positions these models against specialized alternatives such as BloombergGPT and task-specific fine-tuned architectures, aiming to delineate the boundaries of generalist capabilities in high-stakes financial applications. The research highlights the need to understand if the massive scale and reinforcement learning from human feedback (RLHF) of generalist models translate to superior performance in complex financial domains compared to smaller, domain-adapted models. The experimental setup evaluates models on eight benchmark datasets spanning five task categories: sentiment analysis (Financial PhraseBank, FiQA, TweetFinSent), headline classification, named entity recognition (NER FIN3), relation extraction (REFinD), and numerical question answering (FinQA, ConvFinQA). The methodology employs zero-shot, few-shot, and Chain-of-Thought (CoT) prompting strategies to assess performance across varying levels of complexity and financial knowledge requirements. Results indicate that GPT-4 significantly outperforms ChatGPT and prior LLMs, often surpassing BloombergGPT in sentiment and QA tasks. However, fine-tuned models like FinBert and RoBERTa variants remain competitive or superior in simpler classification tasks, while structured prediction tasks like NER and relation extraction show poor performance for LLMs compared to CRF or fine-tuned baselines. Key findings demonstrate that CoT prompting yields substantial accuracy gains, particularly for numerical reasoning in FinQA, where GPT-4 exceeds 78% accuracy, approaching but not matching human expert levels (~90%). Despite these advances, the study identifies significant limitations: LLMs struggle with coreference resolution in conversational QA and exhibit arithmetic errors in multi-step calculations, which poses risks for financial analysis. The paper concludes that while generalist LLMs are powerful tools for sentiment and reasoning tasks, they are not yet reliable for structured information extraction, suggesting a hybrid approach where fine-tuning remains necessary for specific, complex financial NLP applications.

Research problem

The paper investigates whether general-purpose large language models, specifically ChatGPT and GPT-4, can effectively solve a wide range of financial text analytics tasks without domain-specific fine-tuning. It seeks to compare their performance against state-of-the-art domain-specific models like BloombergGPT and task-specific fine-tuned models to determine the extent to which generalist capabilities transfer to the financial domain.

Core contributions

• Provides one of the first comprehensive empirical evaluations of ChatGPT and GPT-4 on eight financial NLP benchmarks across five task categories.
• Demonstrates that generalist LLMs often outperform domain-specific pretrained models like BloombergGPT in sentiment analysis and numerical reasoning tasks.
• Identifies specific limitations of LLMs in structured prediction tasks such as Named Entity Recognition (NER) and Relation Extraction (RE), where fine-tuned models remain superior.
• Shows that Chain-of-Thought (CoT) prompting significantly boosts performance on complex financial QA tasks, closing the gap with fine-tuned baselines.
• Highlights arithmetic and coreference resolution errors in LLMs, cautioning against their direct deployment in high-stakes financial analysis without human oversight.

Method

• Utilizes gpt-3.5-turbo and GPT-4 (8k/16k) with zero-shot, few-shot, and Chain-of-Thought prompting.
• Evaluates sentiment analysis using three datasets: Financial PhraseBank, FiQA, and TweetFinSent.
• Assesses text classification on a commodity headline dataset with six categories.
• Tests Named Entity Recognition on SEC filing data (NER FIN3) with entity-level F1 scoring.
• Evaluates Relation Extraction on REFinD dataset using Macro F1 to measure linkage detection.
• Conducts numerical question answering on FinQA and ConvFinQA, using CoT and function calls for reasoning steps.
• Compares performance against BloombergGPT, prior LLMs, and task-specific fine-tuned models like FinBert and FinQANet.

Results & findings

• GPT-4 achieves 97% weighted F1 on Financial PhraseBank, outperforming BloombergGPT and matching fine-tuned FinBert.
• In FiQA sentiment analysis, GPT-4 (88.11 F1) surpasses BloombergGPT (75.07) and fine-tuned RoBERTa-large (87.09).
• For Headline Classification, fine-tuned BERT (95.36 F1) outperforms GPT-4 (86.00 F1), showing limits of generalist models in multi-class classification.
• In NER, GPT-4 (56.71 F1) underperforms BloombergGPT (60.82) and significantly trails a fine-tuned CRF model (82.70 F1).
• For Relation Extraction, fine-tuned Luke-base (56.30 Macro F1) beats GPT-4 (46.87 Macro F1), indicating LLMs struggle with structured prediction.
• In FinQA numerical QA, GPT-4 with CoT reaches 78.03% accuracy, exceeding FinQANet (68.90%) but falling short of human experts (91.16%).
• ChatGPT shows significant arithmetic errors in ConvFinQA, such as miscalculating simple sums, highlighting reliability issues in reasoning.

Data & experiments

• Evaluates on eight datasets: Financial PhraseBank, FiQA, TweetFinSent, Headlines, NER FIN3, REFinD, FinQA, and ConvFinQA.
• Covers five task types: sentiment analysis, text classification, named entity recognition, relation extraction, and numerical question answering.
• Compares ChatGPT (gpt-3.5-turbo) and GPT-4 against BloombergGPT, prior LLMs (GPT-NeoX, OPT, BLOOM), and fine-tuned baselines (FinBert, RoBERTa, FinQANet).
• Tests zero-shot, few-shot, and Chain-of-Thought (CoT) prompting strategies to assess the impact of in-context learning and reasoning steps.
• Uses metrics including accuracy, macro-F1, and weighted F1, with specific attention to entity-level F1 for NER and accuracy for QA.

Limitations & caveats

• LLMs struggle with structured prediction tasks like NER and Relation Extraction, underperforming fine-tuned baselines.
• GPT-4 and ChatGPT exhibit arithmetic errors in multi-step numerical reasoning, posing reliability risks.
• Performance on social media sentiment (TweetFinSent) is lower due to informal language and emojis not well-captured by training data.
• The study focuses on text analytics; it does not evaluate LLMs in trading execution or portfolio management.
• Results are based on specific versions of models (as of July 2023) and may not reflect future iterations.
• ConvFinQA test set labels were not public, so evaluation used the dev set, potentially limiting generalizability.

Curator notes

Discovery status: round2_promoted_seed_for_round3. Source links: Dólares or Dollars? Unraveling the Bilingual Prowess of Financial LLMs Between Spanish and English; FinBERT: Financial Sentiment Analysis with Pre-trained Language Models; No Language is an Island: Unifying Chinese and English in Financial Large Language Models, Instruction Data, and Benchmarks