Advanced Financial Reasoning at Scale: Large Language Models on Chartered Financial Analyst Level III

In one sentence

Summary

The paper addresses the critical need for rigorous, domain-specific evaluation of Large Language Models in high-stakes financial settings. It positions the Chartered Financial Analyst (CFA) Level III exam as a gold-standard benchmark for advanced financial reasoning, moving beyond general knowledge to test synthesis, strategic thinking, and professional judgment. The study aims to determine if current state-of-the-art models can meet the cognitive demands of professional investment management and wealth planning, filling a gap left by previous evaluations that focused on easier Levels I and II. The methodology involves constructing a dataset of mock CFA Level III exams, including 60 multiple-choice questions (MCQs) and 43 essay questions derived from AnalystPrep materials. The authors evaluate 23 diverse LLMs, spanning frontier proprietary models, open-source variants, and specialized financial models. They employ three prompting strategies: zero-shot, Chain-of-Thought with Self-Consistency (CoT-SC), and Self-Discover. Evaluation includes automated LLM-as-a-judge grading and human-expert grading by certified CFA graders to assess alignment. Metrics cover accuracy, latency, cost, and semantic similarity. Key findings indicate that frontier reasoning models, particularly o4-mini (79.1%) and Gemini 2.5 Flash (77.3%), exceed the 65% passing threshold. While MCQ performance converges across top models, essay scores show greater variance, highlighting the difficulty of complex synthesis. Chain-of-Thought prompting significantly improves performance but increases costs by up to 11x. The study also reveals a systematic bias where human graders are more lenient than LLM judges, awarding an average of 5.6 points higher. These results suggest a tiered deployment strategy is optimal, using smaller models for routine tasks and frontier models for complex analysis, while cautioning against over-reliance on automated grading for subjective financial reasoning.

Research problem

The paper investigates whether state-of-the-art Large Language Models can perform at a professional level on the CFA Level III exam, which tests advanced financial reasoning, portfolio management, and wealth planning. It addresses the lack of rigorous, high-stakes benchmarks for LLMs in specialized financial domains, moving beyond general reasoning or simple recall tasks. The study specifically aims to evaluate the capabilities of both reasoning-enhanced and non-reasoning models, as well as the impact of different prompting strategies on complex essay and multiple-choice questions.

Core contributions

• Introduction of a comprehensive CFA Level III benchmark dataset comprising mock MCQs and essay questions with human-validated grading rubrics.
• First large-scale evaluation of 23 diverse LLMs (proprietary, open-source, reasoning-enhanced) on advanced financial reasoning tasks.
• Systematic comparison of three prompting strategies (Zero-Shot, CoT-SC, Self-Discover) and their impact on accuracy, cost, and latency.
• Novel alignment analysis comparing LLM-as-a-judge grading against certified human expert grading, revealing systematic leniency differences.
• Detailed cost-efficiency analysis providing practical guidance for deploying LLMs in financial services, highlighting accuracy-latency-cost trade-offs.

Method

• Dataset Construction: OCR extraction from AnalystPrep PDFs, structured into JSON, and validated by experts for curriculum alignment.
• Model Categorization: Grouped into Frontier, Non-Reasoning, Reasoning-Enhanced, Open-Source, and Specialized models.
• Prompting Strategies: Implemented Zero-Shot, CoT with Self-Consistency (majority voting for MCQs, confidence-based selection for essays), and Self-Discover (metacognitive module identification).
• Automated Grading: Used GPT-4.1 as an LLM-judge with strict rubric-based prompts to score essay responses on a 0-100 scale.
• Human Grading: Certified CFA experts graded a subset of responses (CoT-SC N=3) to provide ground truth and assess grading alignment.
• Efficiency Tracking: Measured API latency, token usage, and estimated costs for all model-prompt combinations to analyze deployment trade-offs.

Results & findings

• Frontier models o4-mini and Gemini 2.5 Flash achieved composite scores of 79.1% and 77.3%, respectively, exceeding the 65% passing threshold.
• Reasoning-enhanced models outperformed non-reasoning models by 19.1% on essay questions (LLM-judged), demonstrating the value of explicit reasoning.
• Chain-of-Thought with Self-Consistency (N=5) yielded the highest MCQ accuracy (69.6%) but increased costs by 11.1x compared to zero-shot.
• Human graders were significantly more lenient than LLM judges, awarding an average of 5.6 points higher on essay questions.
• LLM and human graders agreed on only ~70% of questions, with LLM judges often being stricter due to rigid rubric adherence.
• Self-Discover prompting improved MCQ accuracy by 6.8 points over zero-shot but underperformed on essay tasks compared to CoT.
• Closed-source models marginally outperformed open-source models in analytical capability, though open-source models showed competitive lexical similarity.

Data & experiments

• Dataset: Mock CFA Level III exams from AnalystPrep, including 60 MCQs and 43 essay questions covering portfolio management, ethics, and derivatives.
• Models: 23 LLMs including o4-mini, Gemini 2.5 Pro/Flash, Claude Opus 4, Llama 4, and specialized models like Palmyra-fin.
• Prompting Strategies: Zero-shot, Chain-of-Thought with Self-Consistency (N=3,5), and Self-Discover metacognitive prompting.
• Evaluation Metrics: MCQ accuracy, LLM-judged essay scores, human-expert essay scores, cosine similarity, ROUGE-L, latency, and API cost.
• Human Evaluation: Certified CFA graders evaluated 989 essay responses to assess alignment with automated grading and real-world standards.

Limitations & caveats

• The dataset uses mock exams from AnalystPrep, which may not perfectly replicate the difficulty or style of official CFA exams.
• Human grading was limited to one prompting strategy (CoT-SC N=3) to control costs, potentially limiting the generalizability of alignment findings.
• LLM-as-a-judge grading may still exhibit biases, despite the use of strict rubrics, as evidenced by the discrepancy with human graders.
• The study focuses on CFA Level III, which is specific to investment management; results may not generalize to other financial domains like trading or risk.
• Cost and latency metrics are based on API pricing, which may vary for on-premise deployments or different usage scales.
• The 'Self-Discover' prompting strategy showed mixed results, underperforming on essays, suggesting it may not be universally applicable.

Curator notes

Compares reasoning methods such as Chain of Thought and Self-Discover.