Designing Heterogeneous LLM Agents for Financial Sentiment Analysis

In one sentence

Summary

This study addresses the challenge of applying generative Large Language Models (LLMs) to Financial Sentiment Analysis (FSA) without fine-tuning. While LLMs excel at generative tasks, FSA is discriminative and requires handling domain-specific linguistic nuances like sarcasm, irrealis mood, and external references. The paper positions its contribution within design science, leveraging Minsky’s theory of mind to create a framework where LLMs act as specialized 'resources' or agents, rather than relying on homogeneous multi-agent debates or simple chain-of-thought prompting. The goal is to elicit the full potential of pre-trained models through strategic elicitation and structured collaboration. The proposed method, Heterogeneous Agent Discussion (HAD), instantiates five specialized LLM agents, each prompted to focus on a specific type of common FSA error: irrealis mood, rhetoric (sarcasm), dependent opinion, aspect mismatch, and external references. These agents process the input text independently, and their outputs are aggregated by a summative agent to determine the final sentiment polarity. The framework is evaluated on five FSA datasets (Financial PhraseBank, StockSen, CMC, FiQA, SEntFiN) using GPT-3.5 and BLOOMZ models. Experiments compare HAD against naive prompting and fine-tuned baselines like FinBERT, utilizing accuracy and macro F-1 scores as metrics. Ablation studies are conducted to assess the individual contribution of each agent type. Results indicate that HAD consistently improves performance over naive prompting, particularly with GPT-3.5, fixing approximately 25-35% of the performance gap between zero-shot prompting and fine-tuning. The mood, rhetoric, and aspect agents were found to be the most critical contributors, while the dependency agent showed negligible or negative impact when removed. The framework is most effective when agents generate substantial discussion content. Limitations include higher computational costs and latency compared to single-model approaches, potential data leakage in older benchmarks, and the non-linear, complex interactions between agents that require careful prompt engineering. The study provides a theoretical foundation for designing heterogeneous LLM agents in finance.

Research problem

The paper investigates how to effectively leverage pre-trained generative LLMs for Financial Sentiment Analysis (FSA) without fine-tuning. It addresses the lack of prescriptive knowledge on designing multi-agent systems for discriminative financial tasks, specifically aiming to overcome the limitations of naive prompting and homogeneous agent setups by incorporating domain-specific linguistic error types into agent design.

Core contributions

• Proposes Heterogeneous Agent Discussion (HAD), a theory-informed framework for FSA using specialized LLM agents.
• Derives five specific FSA error types (mood, rhetoric, dependency, aspect, reference) to guide agent prompt design.
• Demonstrates that HAD improves accuracy by 25-35% of the fine-tuning gap compared to naive prompting on GPT-3.5.
• Provides ablation analysis revealing the non-linear importance of different agent types, highlighting mood and rhetoric as key drivers.
• Establishes a design science basis for LLM collaboration in finance using Minsky’s theory of mind.

Method

• Designs five heterogeneous agents based on identified FSA error types.
• Prompts each agent with a specific focus (e.g., 'pay special attention to irrealis mood').
• Agents process input text independently to generate intermediate sentiment opinions.
• A summative agent aggregates all intermediate opinions to produce the final sentiment label.
• Uses in-context learning (ICL) without model weight updates.
• Evaluates performance using accuracy and macro F-1 scores.
• Conducts ablation studies by removing agents one by one to assess importance.
• Compares results against naive prompting and fine-tuned baselines.

Results & findings

• HAD improves GPT-3.5 accuracy by 2.24% to 9.46% and F-1 by 0.35% to 13.72% across datasets.
• The framework fixes 25-35% of the performance gap between zero-shot prompting and fine-tuning.
• Mood (A1), Rhetoric (A2), and Aspect (A4) agents are the most critical for performance.
• Dependency agent (A3) removal often improves performance, suggesting it may be redundant or poorly prompted.
• Reference agent (A5) has uncertain impact, varying by dataset.
• Improvements are more pronounced with GPT-3.5 than with BLOOMZ, likely due to instruction-tuning differences.
• HAD is computationally more intensive than naive prompting but less complex than fine-tuning.

Data & experiments

• Evaluated on five datasets: Financial PhraseBank, StockSen, CMC, FiQA, and SEntFiN.
• Datasets cover financial news, social media (StockTwits, Reddit), and crypto forums.
• Models tested: GPT-3.5 (instruction-tuned) and BLOOMZ (open-source, 560M parameters).
• Metrics: Accuracy and Macro F-1 score for ternary classification tasks.
• Baselines include naive prompting, fine-tuned models (FinBERT, BloombergGPT), and other multi-agent frameworks.
• Ablation studies performed to measure the impact of removing individual agents.

Limitations & caveats

• Higher computational cost and latency compared to single-model or naive prompting approaches.
• Potential data leakage in older datasets (FPB, FiQA) due to LLM training exposure.
• Performance gains vary significantly between models (GPT-3.5 vs. BLOOMZ).
• Some agent types (e.g., dependency) may be redundant or require better prompt engineering.
• Complex, non-linear interactions between agents make optimization challenging.
• Evaluation limited to sentiment classification, not broader financial reasoning tasks.

Curator notes

Discovery status: round3_promoted. Source links: FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Making; Large Language Models in equity markets: applications, techniques, and insights