A Preliminary Look at the State of the Art of Large Language Models on Chartered Financial Analyst Exams

In one sentence

Summary

The paper addresses the evaluation of large language models' financial analysis capabilities by benchmarking them against the Chartered Financial Analyst (CFA) program, a rigorous professional certification. It positions LLMs as potential tools for automating investment research and analysis, aiming to determine if current models meet the standards of human professionals. The study covers the full spectrum of CFA exams, from basic concept memorization in Level I to complex case analysis and essay writing in Level III, providing a comprehensive view of LLM strengths and weaknesses in high-stakes financial environments. The authors evaluate five proprietary and nine open-source models using mock CFA exams from AnalystPrep, ensuring low contamination risk. Experiments employ one-shot chain-of-thought prompting for multiple-choice questions and a model-assisted human evaluation strategy for Level III essays. The study also investigates Retrieval-Augmented Generation (RAG) using CFA textbooks to assess if external knowledge can bridge performance gaps. Metrics include accuracy for MCQs and a weighted score combining MCQ and essay marks for Level III, with ablation studies on shot count and temperature. Proprietary models like GPT-4o and Claude 3 Opus consistently outperform open-source counterparts, passing Levels I and II but failing Level III due to poor essay performance. Open-source models, particularly LLaMA 3 70B, show strong potential; with RAG, LLaMA 3 70B can pass Levels I and II under lower passing score bounds. However, RAG benefits diminish at higher levels, and open-source models struggle with instruction following and nuance. The paper highlights that while LLMs are proficient in knowledge retrieval and calculation, they lack the professional judgment and structured reasoning required for advanced portfolio management decisions.

Research problem

The paper investigates whether state-of-the-art large language models possess the financial analysis capabilities required to pass the Chartered Financial Analyst (CFA) exams, which serve as a gold standard for professional investment knowledge. It aims to benchmark both proprietary and open-source models across all three levels of the CFA program to understand their current limitations in financial reasoning, calculation, and professional judgment.

Core contributions

• Comprehensive benchmarking of 14 LLMs (5 proprietary, 9 open-source) on all three levels of CFA mock exams.
• Detailed analysis of model performance by CFA topic area, revealing strengths in portfolio management and weaknesses in complex calculations.
• Investigation of Retrieval-Augmented Generation (RAG) using CFA textbooks, showing it helps open-source models pass Levels I and II.
• Identification of specific error modes, including calculation errors, instruction following failures, and inability to handle Level III essay nuances.

Method

• One-shot chain-of-thought (1S-CoT) prompting with zero temperature for MCQs.
• LLaMA 3 70B used to clean CoT outputs and extract final answers for MCQ evaluation.
• Model-assisted human evaluation: GPT-4o grades essays based on ground-truth explanations, verified by a CFA charterholder.
• RAG implementation: Indexing CFA textbooks at the post-level; retrieving top-K passages using lexical and semantic models.
• Ablation studies on number of shots (K=1, 2, 5) and temperature (T=0, 0.7, 1.0).
• Median of means calculation across five seeds to account for response variability.

Results & findings

• Proprietary models (GPT-4o, Claude 3 Opus) pass Levels I and II but fail Level III due to essay performance.
• LLaMA 3 70B is the best open-source model, performing within ~20% of proprietary models on Levels I and II.
• RAG with CFA textbooks allows LLaMA 3 70B to pass Levels I and II under lower passing score bounds.
• Open-source models struggle with instruction following, often providing answers before reasoning.
• Level III essay questions are the primary bottleneck for all models, with GPT-4o scoring only 46.2% on essays.
• Proprietary models are significantly better at following complex instructions and handling nuance than open-source models.

Data & experiments

• Dataset: 2023 CFA mock exams from AnalystPrep, covering Levels I, II, and III, including MCQs and essay questions.
• Models: 5 proprietary (GPT-3.5/4/4o, Claude 3 Opus, Mistral Large) and 9 open-source (LLaMA 3, Mixtral, Gemma, etc.).
• Evaluation: One-shot chain-of-thought prompting for MCQs; model-assisted human grading for Level III essays.
• RAG Experiment: Retrieval of CFA textbook posts using BM25+ and semantic models to augment LLaMA 3 models.
• Metrics: Accuracy for MCQs; weighted score for Level III; pass/fail status based on estimated Minimum Passing Scores.

Limitations & caveats

• Essay grading relies on GPT-4o, which may not perfectly replicate human CFA grader standards.
• Mock exams may not fully capture the difficulty or nuance of official CFA exams.
• RAG benefits are limited for smaller models (LLaMA 3 8B) and higher levels (II and III).
• Open-source models struggle with instruction following, potentially skewing performance.
• The study does not evaluate real-time market data or dynamic decision-making scenarios.

Curator notes

Useful for baseline professional competence evaluation.