Can GPT models be Financial Analysts? An Evaluation of ChatGPT and GPT-4 on mock CFA Exams

In one sentence

Summary

The paper addresses the gap in rigorous evaluation of Large Language Models' financial reasoning capabilities by benchmarking ChatGPT and GPT-4 against mock Chartered Financial Analyst (CFA) Program exams. It positions LLMs as potential substitutes or assistants for professional financial analysis, testing their ability to handle domain-specific terminology, multi-step calculations, and table-based evidence found in real-world investment scenarios. The study aims to determine if current out-of-the-box models possess sufficient knowledge to pass professional certification exams, thereby assessing their readiness for advisory and analytical roles in finance. The methodology involves evaluating models on five Level I and two Level II mock exams using Zero-Shot, Chain-of-Thought (CoT), and Few-Shot (FS) prompting strategies. The authors analyze performance across ten finance topics, categorizing errors into knowledge, reasoning, calculation, and inconsistency types. They also estimate the Minimum Passing Score (MPS) for CFA exams based on community data to determine pass/fail likelihoods. Experiments control for memorization and use temperature zero to ensure reproducibility, providing a comprehensive breakdown of where models succeed or fail in financial contexts. Key findings indicate GPT-4 significantly outperforms ChatGPT, with FS prompting yielding the best results for both models. GPT-4 with FS can likely pass Level I and has a chance at Level II, whereas ChatGPT fails both. CoT prompting often hurts performance on Level I due to hallucinated calculations but helps on Level II by filtering relevant case details. Major limitations include poor numerical reasoning, susceptibility to knowledge gaps when forced to reason step-by-step, and struggles with long-context table interpretation. The paper suggests future improvements via retrieval-augmented generation and external calculation tools.

Research problem

The paper investigates whether general-purpose Large Language Models (LLMs) like ChatGPT and GPT-4 possess the necessary financial reasoning, knowledge, and calculation capabilities to perform at the level of a professional financial analyst. It specifically addresses the challenge of evaluating LLMs on complex, domain-specific tasks that require multi-step reasoning, handling tabular data, and applying precise financial formulas, which are core components of the CFA curriculum.

Core contributions

• First comprehensive evaluation of ChatGPT and GPT-4 on mock CFA Level I and II exams using multiple prompting paradigms.
• Detailed error analysis categorizing LLM failures into knowledge, reasoning, calculation, and inconsistency modes.
• Empirical estimation of CFA Minimum Passing Scores to determine the likelihood of LLMs passing professional exams.
• Comparative analysis of Zero-Shot, Chain-of-Thought, and Few-Shot prompting effectiveness in financial contexts.
• Insights into how prompt length and table evidence impact LLM performance on Level II item-sets.

Method

• Used OpenAI ChatCompletion API for ChatGPT and GPT-4 with temperature=0.
• Implemented Zero-Shot prompting for baseline performance assessment.
• Applied Chain-of-Thought prompting to force step-by-step reasoning and calculation.
• Utilized Few-Shot prompting with random sampling (2S, 4S, 6S) and topic-balanced sampling (10S).
• Categorized errors into knowledge, reasoning, calculation, and inconsistency types.
• Estimated CFA passing thresholds using Angoff method insights and Reddit community data.
• Analyzed prompt length and table evidence density as factors affecting Level II performance.

Results & findings

• GPT-4 achieved 74.6% accuracy on Level I with 10-shot prompting, likely passing the exam.
• ChatGPT peaked at 63.0% on Level I with 2-shot prompting, failing to meet the estimated passing threshold.
• GPT-4 with 4-shot prompting achieved 60.5% on Level II, bordering on the estimated passing score.
• CoT prompting improved GPT-4 on Level II but degraded ChatGPT performance on Level I.
• Derivatives and Ethics were the easiest topics; Financial Reporting and Portfolio Management were hardest.
• Level II performance dropped significantly due to longer contexts, table evidence, and complex calculations.
• Knowledge errors were the most common failure mode for ChatGPT under CoT prompting.
• GPT-4 showed fewer inconsistency errors but more reasoning errors than ChatGPT.

Data & experiments

• Five mock CFA Level I exams and two mock CFA Level II exams collected from public sources.
• Evaluation of ChatGPT (gpt-3.5-turbo) and GPT-4 using OpenAI API with temperature set to zero.
• Three prompting strategies: Zero-Shot, Chain-of-Thought, and Few-Shot (2, 4, 6, and 10 shots).
• Performance measured by accuracy against official answer keys for multiple-choice questions.
• Error analysis performed on instances where CoT led to incorrect answers compared to Zero-Shot.
• Memorization tests conducted to ensure models were not relying on training data memorization.

Limitations & caveats

• Uses mock exams rather than official past papers, which may differ in difficulty or style.
• Level III exams (essay-based) were not evaluated, limiting the scope of reasoning assessment.
• Passing score estimates are based on community data and may not reflect official CFA Institute thresholds.
• Evaluation is limited to multiple-choice questions, ignoring essay and constructed-response tasks.
• Models were tested via API, so internal model weights and fine-tuning details are unknown.
• Few-Shot performance may be sensitive to the specific examples chosen, though topic-balanced sampling was used.

Curator notes

Discovery status: round3_promoted. Source links: Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? An Examination on Several Typical Tasks; Can Large Language Models beat wall street? Evaluating GPT-4’s impact on financial decision-making with MarketSenseAI; FinGPT: Democratizing Internet-scale Data for Financial Large Language Models