The List / Professional, Regulatory, and Advisory Applications / paper
Are Generative AI Agents Effective Personalized Financial Advisors?
Abstract
Large language model-based agents are becoming increasingly popular as a low-cost mechanism to provide personalized, conversational advice, and have demonstrated impressive capabilities in relatively simple scenarios, such as movie recommendations. But how do these agents perform in complex high-stakes domains, where domain expertise is essential and mistakes carry substantial risk? This paper investigates the effectiveness of LLM-advisors in the finance domain, focusing on three distinct challenges: (1) eliciting user preferences when users themselves may be unsure of their needs (2) providing personalized guidance for diverse investment preferences, and (3) leveraging advisor personality to build relationships and foster trust. Via a lab-based user study with 64 participants, we show that LLM-advisors often match human advisor performance when eliciting preferences, although they can struggle to resolve conflicting user needs. When providing personalized advice, the LLM was able to positively influence user behavior, but demonstrated clear failure modes. Our results show that accurate preference elicitation is key, otherwise, the LLM-advisor has little impact, or can even direct the investor toward unsuitable assets. More worryingly, users appear insensitive to the quality of advice being given, or worse these can have an inverse relationship. Indeed, users reported a preference for and increased satisfaction as well as emotional trust with LLMs adopting an extroverted persona, even though those agents provided worse advice.
In one sentence
Summary
This paper investigates the effectiveness of Large Language Model (LLM) agents as personalized financial advisors, addressing three core challenges: eliciting user preferences in complex domains, providing personalized investment guidance, and leveraging advisor personality to build trust. The authors argue that unlike simple recommendation tasks, financial advisory requires handling high-stakes decisions where users may lack domain expertise or clarity about their own needs. The study aims to determine if LLMs can replicate the nuanced, trust-based relationship of human financial advisors. The methodology involves a lab-based user study with 64 participants interacting with LLM-advisors (Llama-3.1 8B) across two stages. Stage 1 focuses on preference elicitation using a System-Ask-User-Respond paradigm, comparing LLM performance against a human financial expert. Stage 2 involves advisory discussions where participants evaluate stock suitability based on their elicited profiles. The study compares a non-personalized baseline against personalized variants and tests two distinct personality profiles (extroverted vs. conscientious) derived from the Big Five model. Evaluation metrics include elicitation accuracy, ranking correlation (Spearman’s Rho) against expert-curated ground truth, and user perception questionnaires. Results indicate that LLMs match human experts in preference elicitation accuracy (approx. 0.78-0.89) but struggle with conflicting user statements. Personalization improves decision quality only when preference elicitation is accurate; otherwise, it directs users to unsuitable assets. Crucially, users were insensitive to advice quality, showing no significant difference in perceived usefulness between good and bad advice. More alarmingly, users reported higher trust and satisfaction with extroverted LLMs that provided statistically worse advice than conscientious ones, highlighting a misalignment between user perception and actual financial suitability.
Research problem
The paper addresses the gap in understanding how LLM-based agents perform as personalized financial advisors in high-stakes domains. Specifically, it investigates whether LLMs can effectively elicit user preferences when users are uncertain, provide personalized guidance that aligns with expert standards, and leverage personality traits to build trust without compromising advice quality.
Core contributions
- Conducts a controlled lab-based user study with 64 participants to evaluate LLM-advisors in realistic financial scenarios.
- Introduces a two-stage evaluation framework: preference elicitation and advisory discussion, benchmarked against human expert performance.
- Demonstrates that LLMs can match human experts in eliciting investment preferences but fail to resolve conflicting user needs.
- Reveals a critical failure mode where personalized advice worsens outcomes if preference elicitation is inaccurate.
- Identifies a dangerous disconnect where users prefer extroverted LLM personas that provide inferior financial advice over conscientious ones.
Method
- Two-stage conversational interaction: Stage 1 for preference elicitation, Stage 2 for advisory discussion.
- System-Ask-User-Respond (SAUR) paradigm for dynamic preference collection.
- LLM-based response summarization to generate user profiles from conversation turns.
- Prompt engineering to embed Big Five personality traits (Extroverted vs. Conscientious) into the LLM.
- Spearman’s Rho correlation to measure alignment between user rankings and expert ground truth.
- Wilcoxon signed-rank test for statistical significance in user perception comparisons.
Results & findings
- LLM elicitation accuracy (0.70-0.80) is comparable to human expert accuracy (0.85 average).
- LLMs struggle to resolve conflicting user needs, leading to random-level accuracy in complex profiles.
- Personalized advice improves user rankings only when preference elicitation is accurate.
- Inaccurate elicitation leads LLMs to direct users toward unsuitable assets.
- Users showed no significant difference in perceived usefulness between personalized and non-personalized advisors.
- Extroverted LLMs provided worse advice (lower ranking correlation) but received higher trust and satisfaction scores.
- Conscientious LLMs provided better advice but were perceived as less trustworthy or satisfying.
Data & experiments
- User study with 64 participants (29 in personalization study, 31 in persona study) recruited from a university.
- Three archetypal investor profiles: Growth-Oriented Healthcare, Conservative Income Seeker, and Risk-Taking Value Investor.
- Four stocks per profile (e.g., Amazon, Coca-Cola) with expert-curated ground truth rankings based on preference alignment.
- Comparison of Baseline, Personalized, Extroverted, and Conscientious LLM-advisor configurations.
- Metrics: Elicitation accuracy, Spearman’s Rho ranking correlation, and 7-dimension user perception questionnaire.
Limitations & caveats
- Small sample size (64 participants) limits generalizability to broader populations.
- Participants were university students, potentially lacking real-world investment experience.
- Study duration was short (approx. 1 hour), not capturing long-term advisor-client relationship dynamics.
- LLM personality was simulated via prompting, not fine-tuning, which may not reflect deployed systems.
- Ground truth rankings were based on expert opinion, which may not align with market performance.
- Does not test real monetary consequences, potentially underestimating user caution or risk aversion.
Curator notes
Discovery status: regtech_compliance_focus_promoted. Source links: LLMs for Financial Advisement: A Fairness and Efficacy Study in Personal Decision Making