Can LLMs be Good Financial Advisors?: An Initial Study in Personal Decision Making for Optimized Outcomes

In one sentence

Summary

The paper addresses the critical gap in evaluating Large Language Models as reliable personal financial advisors, specifically focusing on their ability to handle complex, multi-product decision-making scenarios. The authors argue that while LLMs show potential in revolutionizing public decision-making, their susceptibility to hallucinations and reasoning failures poses risks in domains like banking where accuracy is paramount. The study positions itself as an initial comparative analysis of two leading chatbots, ChatGPT and Bard, to identify specific failure modes in providing optimized financial outcomes for users managing interacting products like credit lines, dues, and interest rates. The experimental design involves 13 structured queries covering four categories of product interactions: credit card only, credit card with bank account, credit card with certificate of deposit, and all three combined. The queries test constraints such as credit limits, billing cycles, cashback incentives, and APR penalties. The authors also included queries in African American Vernacular English (AAVE) and Telugu to assess dialect and language robustness. The models were evaluated based on accuracy, utilization of user information, personalization, bias, and error types including mathematical, perceptual, and grammatical mistakes, with results quantified by error percentages across the query set. Findings indicate that while both models generate fluent text, they exhibit critical gaps in reliability. Bard often failed to use all provided information and gave biased recommendations, while ChatGPT struggled with AAVE and produced grammatical errors in Telugu. Both models frequently lacked personalized recommendations and committed mathematical errors, with Bard showing higher rates of perceptual errors and ChatGPT having more mathematical inaccuracies in specific calculations. The study concludes that current LLMs are not yet ready for trusted financial advisory roles due to these limitations, highlighting the need for integration with numeric solvers and better handling of diverse user inputs and complex constraints.

Research problem

The paper investigates whether current Large Language Models (LLMs), specifically ChatGPT and Bard, can serve as reliable personal financial advisors. It focuses on the challenge of providing accurate, optimized decision-making support for users managing interacting financial products like credit cards, bank accounts, and certificates of deposit. The core problem is the potential for LLMs to provide flawed advice due to mathematical errors, perceptual misunderstandings, and lack of personalization, which could negatively impact users' financial well-being.

Core contributions

• Identifies a specific personal financial planning scenario involving multiple interacting banking products and optimization constraints.
• Conducts a comparative evaluation of ChatGPT and Bard on 13 structured queries covering various product interactions.
• Classifies and quantifies error types in LLM responses, including mathematical, perceptual, and lack of personalization errors.
• Assesses model performance across different dialects (AAVE) and languages (Telugu) to evaluate inclusivity and robustness.
• Highlights specific challenges for future LLM-based financial advisors, such as numeric reasoning and dynamic answer stability.

Method

• Designed 13 queries representing banking products and their inter-product interactions.
• Defined variables such as purchase amount, billing cycle, due amount, credit line, and APR.
• Categorized queries into four groups: CC only, CC+AB, CC+CD, and CC+CD+AB.
• Included queries in English, AAVE, and Telugu to test linguistic robustness.
• Evaluated responses for accuracy, personalization, bias, and error types.
• Quantified error rates for mathematical, perceptual, and grammatical mistakes.
• Compared ChatGPT and Bard on criteria like visual aid usage and learning from mistakes.

Results & findings

• Bard gave accurate results for direct questions but failed to utilize all user information in complex scenarios.
• ChatGPT calculated Credit Utilization Ratio (CUR) more often than Bard but made more mathematical errors.
• Bard provided personalized suggestions less frequently than ChatGPT, often giving generic advice.
• Bard showed bias towards immediate purchases in some cases, whereas ChatGPT was more cautious.
• Bard used visual aids (tables) in one instance, while ChatGPT did not use any visual aids.
• Bard provided multiple drafts for the same query, while ChatGPT did not.
• ChatGPT corrected errors more often than Bard during the interaction.
• Both models struggled with AAVE and Telugu, with ChatGPT producing grammatical errors in Telugu.

Data & experiments

• 13 structured queries covering credit card, bank account, and certificate of deposit interactions.
• Variables include purchase amounts, billing cycles, due amounts, credit lines, cashback percentages, and APRs.
• Evaluation of two models: ChatGPT (OpenAI) and Bard (Google).
• Assessment of dialect and language handling using African American Vernacular English (AAVE) and Telugu.
• Metrics include error rates for mathematical, perceptual, and personalization failures.
• Analysis of response characteristics such as bias, use of visual aids, and learning from mistakes.

Limitations & caveats

• The study is limited to only 13 queries, which may not cover all financial scenarios.
• The user demographic was restricted to college students, limiting generalizability.
• The study does not include complex products like student loans or stock purchases.
• LLM answers may change over time, making long-term evaluation difficult.
• The study does not test the models' ability to learn from user feedback over time.
• The evaluation of dialect and language support is preliminary and limited to two languages.

Curator notes

Discovery status: regtech_compliance_focus_promoted. Source links: LLMs for Financial Advisement: A Fairness and Efficacy Study in Personal Decision Making