The List / Agents and Multi-Agent Systems / paper
From Earnings Calls to Investment Reports: Evaluating Role-based Multi-Agent LLM Systems
Abstract
This paper presents a novel multi-agent framework leveraging LLMs for automated financial analysis and investment report generation from earnings call transcripts. Traditional financial analysis struggles with increasing volumes of unstructured data. We propose a collaborative multi-agent system that mimics professional analyst team structures through role specialization. Our framework employs three specialized agents: Analyst, Writer, and Editor, that collaborate through structured workflows with tool support for financial data retrieval and sentiment analysis. Through extensive human evaluation on the Prolific platform, we demonstrate that our system achieves good accuracy in guiding financial decisions, placing it competitively among twelve evaluated systems. The system scores high on human quality assessment, with particularly strong performance in usefulness, indicating practical value for investment decision-making. In automatic evaluation, our system outperforms professional analyst reports most of the time, validating its competitive quality. Our findings provide empirical evidence that role-based agent collaboration offers a balanced approach to AI-generated financial analysis, demonstrating stable performance that prioritizes practical utility over surface-level report quality.
In one sentence
Summary
The paper addresses the challenge of automating complex financial analysis and investment report generation from unstructured earnings call transcripts, a task where single-agent LLMs often struggle with hallucinations and incomplete coverage. The authors propose a collaborative multi-agent system powered by GPT-4.1, orchestrated via Microsoft AutoGen, which mimics professional analyst teams through specialized roles: an Analyst for data extraction and fact-checking, a Writer for drafting, and an Editor for quality control. This role-based specialization aims to improve factual accuracy and interpretability by enforcing structured workflows and iterative feedback loops, addressing the limitations of monolithic generative models in high-stakes financial contexts. The experimental setup utilizes the Earnings2Insights dataset, comprising 64 earnings call transcripts from the ECTSum and Professional subsets. The system integrates external tools, including Alpha Vantage for historical financial data and a sentiment analysis API for recent news, to ground its outputs in verifiable facts. Evaluation involved both automatic LLM-based judging and a large-scale human study with 176 participants on the Prolific platform, assessing metrics such as decision accuracy (Buy/Neutral/Sell predictions), persuasiveness, logic, and usefulness. The multi-agent system was compared against twelve other systems, including baseline single-agent approaches and professional analyst reports, to validate its competitive edge in generating actionable investment guidance. Results indicate that the multi-agent system achieved the highest financial decision accuracy among automated approaches at 58.1%, with human evaluators rating it highly for logic (5.89/7) and persuasiveness (5.95/7). The system outperformed professional analyst reports in pairwise comparisons most of the time, demonstrating that structured collaboration yields more useful and reliable insights. However, the authors note limitations, including a performance ceiling around 60% accuracy and the system's inability to incorporate non-textual market signals or industry intuition. The findings suggest that while AI can augment human analysts by providing robust foundational analysis, it currently serves best as a tool for refinement rather than a replacement for human judgment in investment decision-making.
Research problem
The paper tackles the difficulty of automating high-quality, factually accurate investment report generation from earnings call transcripts. Traditional single-agent LLM approaches suffer from hallucinations and lack the structured division of labor seen in professional analyst teams. The research aims to determine if a role-based multi-agent system can improve factual accuracy, interpretability, and practical utility in financial analysis compared to existing single-agent methods and professional human reports.
Core contributions
- A novel multi-agent framework mimicking professional analyst teams with specialized Analyst, Writer, and Editor roles.
- Implementation of an iterative feedback loop using Microsoft AutoGen to enhance report quality and factual consistency.
- Integration of external tools (Alpha Vantage, sentiment APIs) to ground LLM outputs in verifiable financial data.
- Empirical evidence demonstrating that role-based collaboration improves decision accuracy and human-perceived utility.
- A comprehensive evaluation framework combining automatic LLM judging and large-scale human evaluation on the Prolific platform.
Method
- Underlying Model: GPT-4.1 for consistent capability across agents.
- Orchestration: Microsoft AutoGen GroupChatManager for message routing and turn-taking.
- Agent Roles: Analyst (data/fact-checking), Writer (drafting), Editor (review/feedback).
- Temperature Settings: Analyst/Editor at 0.2-0.3 for accuracy; Writer at 0.7 for fluency.
- Workflow: Writer drafts -> Analyst validates with tools -> Editor reviews -> Writer revises -> Loop until approval.
- Tool Use: Calls to historicalFinancialData and analyzeMarketSentiment APIs for grounding.
- Evaluation: Dual framework of LLM-based automatic scoring and human decision accuracy assessment.
Results & findings
- Achieved 58.1% financial decision accuracy, the highest among automated systems.
- Human Likert scores: Logic 5.89/7, Persuasiveness 5.95/7, Usefulness 5.74/7.
- Outperformed professional analyst reports in pairwise automatic evaluations most of the time.
- Analyst agent's tool integration significantly reduced hallucinations and improved factual grounding.
- Editor agent improved professionalism and readability, critical for real-world deployment.
- Correlation found between report clarity/logic and investment decision quality.
- Performance ceiling observed around 60% accuracy, suggesting limits of transcript-only analysis.
- Gap between automatic (4.575) and human (5.49) Likert scores indicates human readers value the system more.
Data & experiments
- Dataset: Earnings2Insights, containing 64 earnings call transcripts (40 from ECTSum, 24 Professional subset).
- External Tools: Alpha Vantage API for historical financial metrics and a news sentiment analysis API.
- Human Evaluation: 176 participants on Prolific, assessing decision accuracy and Likert scores for clarity, logic, and usefulness.
- Automatic Evaluation: LLM-based judge scoring Persuasiveness, Logic, Usefulness, Readability, and Clarity.
- Comparison: Benchmarked against 12 other systems and professional analyst reports.
- Metrics: Buy/Neutral/Sell prediction accuracy at day, week, and month horizons; pairwise win rates.
Limitations & caveats
- Performance ceiling around 60% accuracy suggests limits of transcript-only analysis.
- Inability to incorporate non-textual market signals or conduct primary research.
- Lacks industry-specific intuition and ability to identify subtle management patterns.
- May create overly balanced perspectives that miss immediate market signals.
- Dependence on external APIs for data grounding introduces potential latency or cost.
- Human evaluation sample size, while large, may not represent all investor demographics.
Curator notes
Discovery status: trading_agent_focus_promoted. Source links: FinMem: A Performance-Enhanced Large Language Model Trading Agent with Layered Memory and Character Design