Can Large Language Models Tackle the Chartered Financial Analyst Exam?

In one sentence

Summary

The paper addresses the need for rigorous evaluation of Large Language Models (LLMs) in specialized financial contexts, moving beyond general NLP tasks to professional certification standards. It positions the Chartered Financial Analyst (CFA) exam as a proxy for real-world financial analysis complexity, requiring integrated reasoning, quantitative precision, and domain knowledge. The study aims to assess intrinsic model capabilities and the efficacy of Retrieval-Augmented Generation (RAG) in bridging knowledge gaps, providing actionable insights for model selection and deployment in high-stakes financial environments. The methodology involves evaluating three distinct model types—GPT-4o (multimodal/generalist), GPT-o1 (reasoning-specialized), and o3-mini (lightweight/efficient)—on 1,560 multiple-choice questions from CFA Levels I-III. The authors implement a zero-shot baseline and a novel domain reasoning RAG pipeline that uses hierarchical knowledge organization and structured query generation to retrieve relevant content from official CFA curriculum materials. The RAG system employs a two-stage process: generating targeted summaries and keywords for precise retrieval, followed by context-augmented reasoning. Performance is measured against estimated passing criteria, with detailed error analysis categorizing failures into knowledge gaps, reasoning errors, calculation inaccuracies, and inconsistencies. Results show that GPT-o1 consistently outperforms other models in zero-shot settings, achieving high accuracy across all levels, while o3-mini offers a cost-effective alternative with strong performance on foundational topics. The RAG pipeline provides substantial improvements, particularly for Level III questions and complex scenarios, with gains up to 8.64% for GPT-o1. Error analysis reveals that knowledge gaps are the primary failure mode, accounting for over 60% of errors, while calculation errors remain a persistent challenge, especially for GPT-4o. The study concludes that retrieval augmentation is indispensable for fact-heavy conceptual judgment, but deterministic verification layers are needed for quantitative tasks, suggesting a tiered deployment strategy based on task complexity and cost-performance trade-offs.

Research problem

The paper addresses the lack of systematic evaluation of LLMs in specialized financial contexts, specifically professional financial certification. It seeks to determine if state-of-the-art LLMs can handle the integrated complexity of financial analysis, including long contexts, quantitative reasoning, and domain-specific knowledge, as demonstrated by the CFA exam. The study also investigates whether Retrieval-Augmented Generation (RAG) can effectively bridge knowledge gaps and improve reasoning accuracy in these high-stakes scenarios.

Core contributions

• First comprehensive evaluation of GPT-4o, GPT-o1, and o3-mini on 1,560 CFA mock exam questions across Levels I-III.
• Introduction of a novel domain reasoning RAG pipeline using hierarchical knowledge organization and structured query generation.
• Detailed error taxonomy categorizing LLM failures into knowledge gaps, reasoning errors, calculation inaccuracies, and inconsistencies.
• Evidence-based guidelines for model selection and cost-performance optimization in financial LLM deployment.
• Demonstration that retrieval augmentation significantly improves performance for complex, knowledge-intensive financial tasks.

Method

• Zero-shot prompting with structured instruction templates.
• Domain reasoning RAG pipeline with two-stage query generation (summary + keywords).
• Hierarchical vector database organization by CFA level and topic.
• MinerU for PDF-to-markdown conversion preserving tables and formulas.
• Cosine similarity search for retrieval using OpenAI text-embedding-3-small.
• Error analysis categorizing failures into knowledge, reasoning, calculation, and inconsistency.
• Cost-performance trade-off analysis based on token pricing.

Results & findings

• GPT-o1 achieves highest zero-shot accuracy: 94.78% (L1), 89.32% (L2), 79.09% (L3).
• RAG improves accuracy significantly, especially for Level III (up to 8.64% gain for GPT-o1).
• Knowledge gaps account for 61.68% of errors; calculation errors are less frequent but persistent.
• GPT-4o struggles with quantitative tasks (44.7% accuracy on calculation items).
• o3-mini shows strong performance on Level I (87.56%) and consistent RAG improvements.
• RAG minimal benefit for Level I; substantial benefit for complex, long-context Level III questions.
• Ethics and foundational topics show high accuracy across models; Portfolio Management is challenging.
• Cost analysis suggests tiered deployment: GPT-o1 for high-stakes, o3-mini for routine tasks.

Data & experiments

• Dataset: 1,560 multiple-choice questions from official CFA mock exams (Levels I-III, 2022-2025).
• Models: GPT-4o (multimodal), GPT-o1 (reasoning), o3-mini (lightweight).
• Evaluation: Zero-shot prompting and domain-specific RAG pipeline using official CFA curriculum.
• Metrics: Accuracy per level, topic, and error type; cost-performance analysis.
• Benchmarks: Estimated passing criteria based on CFA Institute standards.

Limitations & caveats

• Evaluation limited to multiple-choice questions; does not assess essay or open-ended responses.
• RAG performance depends on the quality and coverage of the retrieved curriculum.
• Calculation errors persist even with RAG, indicating need for deterministic verification.
• Results may not generalize to all financial tasks beyond the CFA curriculum scope.
• Cost-performance analysis is based on specific pricing at the time of writing.
• Ethics performance varies, suggesting limitations in handling nuanced professional standards.

Curator notes

Uses official mock exam questions across CFA Levels I to III.