The List / Professional, Regulatory, and Advisory Applications / paper
ChatGPT as a Financial Advisor: A Re-Examination
Abstract
Building on prior research, we revisited the 21 personal finance scenarios using OpenAI’s newer ChatGPT-4o to observe whether its financial guidance has meaningfully evolved. Our qualitative analysis relied on expert assessments to examine both the content and tone of the model’s advice, considering how prompt engineering influenced ChatGPT outputs. We observed that ChatGPT-4o often produced more thorough suggestions and paid closer attention to tax implications—though it still overlooked some important details. It also showed more creative thinking in certain situations. However, some of the same shortcomings persisted: Generalizations remained too broad with respect to certain topics, legal references were occasionally misleading, and emotional empathy continued to feel artificial, even with carefully crafted prompts. We also extended our analysis to the newest ChatGPT model (ChatGPT-5). We found that the recommendations generated by ChatGPT-5 were quite similar to those generated by ChatGPT-4o, but the accuracy in the numerical problems was better under ChatGPT-5. While not a replacement for financial professionals, ChatGPT appears to be maturing into a more useful supporting tool for both advisors and clients. Our findings not only suggest cautious optimism but also underscore the need for careful oversight when using such tools in personal financial decision-making.
In one sentence
Summary
This study addresses the evolving capability of large language models to provide personal financial advice, specifically examining whether ChatGPT-4o and ChatGPT-5 have improved upon the limitations observed in ChatGPT-3.5. The research problem centers on the qualitative assessment of AI-generated guidance, focusing on content accuracy, tone, legal caution, and emotional resonance in nuanced personal finance scenarios. The authors aim to determine if prompt engineering can mitigate previous shortcomings and if newer models offer meaningful advancements in delivering tailored, safe, and comprehensive financial planning suggestions to consumers and professionals. The methodology employs a qualitative expert assessment of 21 personal finance cases originally developed by Schlosky et al. (2024). The authors evaluate ChatGPT-4o outputs under both regular and enhanced prompts, where the enhanced prompt defines a specific persona of a competent, empathetic, and tax-aware financial advisor. The analysis compares the content, prioritization, and tone of these outputs. Additionally, the study extends to a preliminary evaluation of ChatGPT-5 on a subset of cases to assess further improvements in numerical accuracy and general advice quality. Expert judgment is used to critique the advice for logical consistency, legal validity, and emotional appropriateness. Findings indicate that ChatGPT-4o produces more thorough and creative suggestions than its predecessor, with prompt engineering significantly improving tone and attention to detail, such as tax implications. However, persistent limitations include overly broad generalizations, misleading legal references, and a lack of genuine empathy, often resulting in "false compassion." ChatGPT-5 shows better numerical accuracy but offers similar qualitative advice to ChatGPT-4o. The study concludes that while ChatGPT is maturing into a useful supporting tool for financial literacy and initial guidance, it cannot replace human advisors due to its inability to handle complex legal, ethical, and highly personalized financial planning needs effectively.
Research problem
The study investigates whether newer LLM versions (ChatGPT-4o and ChatGPT-5) have meaningfully improved in providing personal financial advice compared to earlier models like ChatGPT-3.5. It specifically examines the impact of prompt engineering on the quality, tone, and accuracy of AI-generated financial guidance. The core problem is assessing if AI can safely and effectively support personal financial decision-making without the risks associated with hallucinations, legal inaccuracies, or emotional insensitivity.
Core contributions
- Re-evaluates ChatGPT-4o and ChatGPT-5 using a standardized set of 21 complex personal finance scenarios.
- Demonstrates the impact of role-based prompt engineering on the tone, prioritization, and detail of financial advice.
- Provides a qualitative expert assessment of AI advice quality, focusing on legal, tax, and emotional dimensions.
- Compares model evolution from ChatGPT-3.5 to 4o and 5, highlighting improvements in numerical accuracy and creativity.
- Identifies persistent limitations in AI financial advising, including misleading legal references and artificial empathy.
Method
- Qualitative expert assessment of AI-generated financial advice.
- Use of 21 pre-defined personal finance scenarios from prior research.
- Application of role-based prompt engineering to define advisor persona.
- Comparison of output content, tone, prioritization, and legal/tax accuracy.
- Evaluation of numerical accuracy in retirement and investment calculations.
- Assessment of emotional resonance and empathy in AI responses.
- Cross-model comparison between ChatGPT-3.5, 4o, and 5.
Results & findings
- ChatGPT-4o provides more thorough and creative suggestions than ChatGPT-3.5.
- Prompt engineering improves tone and attention to tax implications but does not fully resolve legal inaccuracies.
- ChatGPT-5 shows better numerical accuracy in financial calculations compared to ChatGPT-4o.
- AI advice often lacks proper prioritization and logical financial planning sequences.
- Models struggle with complex legal issues, sometimes providing misleading or vague legal references.
- AI exhibits "false compassion," mimicking empathy without understanding the emotional context of severe financial hardship.
- ChatGPT often overlooks critical tax liabilities on windfalls or gains in its initial recommendations.
Data & experiments
- 21 personal finance scenarios covering debt, inheritance, retirement, insurance, and legal/ethical dilemmas.
- Evaluation of ChatGPT-4o outputs under regular and enhanced (persona-based) prompts.
- Preliminary evaluation of ChatGPT-5 on a subset of cases (Cases 3, 4, 5, 6, 13, 15, 19).
- Qualitative expert assessment by authors with finance and financial planning backgrounds.
- Comparison of AI outputs against human financial planning standards and logical consistency.
Limitations & caveats
- Qualitative nature limits generalizability and statistical significance.
- No comparison with human advisor responses or robo-advisors.
- Expert assessment may introduce subjective bias in evaluating advice quality.
- Limited evaluation of ChatGPT-5 to a subset of cases.
- Does not provide a comprehensive guideline for when to use AI vs. human advisors.
- Relies on self-reported scenarios which may not reflect all real-world complexities.
Curator notes
Discovery status: institutional_trading_focus_promoted. Source links: ChatGPT financial advice portfolio recommendation