FinSphere: A Conversational Stock Analysis Agent based on Large Language Models

In one sentence

Summary

The paper addresses the critical gap in financial LLMs regarding the lack of objective evaluation metrics for stock analysis reports and the insufficient depth of insights generated by existing models. The authors argue that current systems struggle to synthesize complex, real-time financial data into professional-grade reports, often relying on static historical knowledge or shallow reasoning. To solve this, they introduce a comprehensive framework designed to enhance both the evaluation and generation of stock analysis, positioning their solution as a bridge between automated data processing and expert-level financial reasoning. The core contribution is FinSphere, an agent that decomposes user queries into subtasks using chain-of-thought reasoning and executes them via specialized quantitative tools connected to a real-time financial database. These tools provide structured background information on technical indicators, capital flows, and fundamentals. The agent's LLM backbone, Qwen2-72B, is fully instruction-tuned on Stocksis, a novel dataset of 5,000 expert-curated training pairs consisting of tool outputs and refined expert analyses. Evaluation is conducted using AnalyScore, a multi-dimensional framework assessing conclusion, content, expression, and data usage, with human experts scoring 100 model responses across various baselines. Experiments demonstrate that FinSphere achieves the highest AnalyScore (70.88/100) among all tested models, outperforming GPT-4o, FinMem, and FinRobot. The ablation study reveals a strong positive correlation between the volume of Stocksis training data and performance, highlighting the importance of high-quality, expert-refined supervision. The system effectively combines real-time data access with structured analytical reasoning, producing coherent and actionable reports. However, the reliance on proprietary tools and the high cost of expert curation present deployment challenges, and the evaluation remains subjective despite high inter-annotator agreement.

Research problem

Current financial large language models (FinLLMs) suffer from two main limitations: the absence of objective, systematic evaluation metrics to assess the quality of stock analysis reports, and a lack of analytical depth that prevents them from generating professional-grade insights. Existing models often rely on static historical data and fail to effectively synthesize real-time market information into coherent, actionable analysis.

Core contributions

• AnalyScore: A systematic evaluation framework for assessing stock analysis quality across four dimensions: conclusion, content, expression, and data.
• Stocksis: A high-quality dataset of 5,000 expert-curated training pairs combining quantitative tool outputs with refined expert analyses.
• FinSphere: A real-time stock analysis agent that integrates proprietary quantitative tools with an instruction-tuned LLM.
• Demonstration of superior performance over general and domain-specific LLMs in generating professional-grade stock reports.

Method

• Task decomposition: Chain-of-thought reasoning to break down user queries into subtasks.
• Tool integration: Specialized quantitative tools query real-time databases for technical, fundamental, and capital flow data.
• Instruction tuning: Full-parameter fine-tuning of Qwen2-72B on Stocksis dataset using 32K context window.
• Synthesis: Instruction-tuned LLM synthesizes multi-source tool outputs into structured, professional reports.
• Evaluation: Human experts apply AnalyScore framework to assess model outputs on 100 queries.
• Ablation: Varying proportions of Stocksis data to assess impact on model performance.

Results & findings

• FinSphere achieved the highest total AnalyScore of 70.88, outperforming FinMem (67.55) and GPT-4o (66.61).
• FinSphere scored highest in Content (27.16) and Expression (14.87), indicating superior analytical depth and logical articulation.
• Domain-specific model FinGPT performed poorly (40.05), suggesting limited capability in comprehensive stock analysis.
• Ablation study showed performance increased non-linearly with training data scale, from 58.90 (20%) to 70.88 (100%).
• Inter-annotator agreement was strong, with average Kendall’s Tau correlations exceeding 80% for most dimensions.
• FinSphere reduced input token consumption compared to baselines requiring verbose few-shot prompting.

Data & experiments

• Stocksis dataset: 5,000 training pairs of quantitative tool outputs and expert-written analyses, with a subset open-sourced.
• AnalyScore evaluation: Human experts scored 100 responses from 8 models across four dimensions (total 100 points).
• Baselines: General LLMs (GPT-4o, GPT-3.5, Qwen2-72B, Deepseek-v3), domain-specific models (InvestLM, FinGPT), and agents (FinRobot, FinMem).
• Ablation study: Fine-tuning Qwen2-72B with 20%, 50%, 80%, and 100% of Stocksis to measure data scale impact.
• Inter-annotator agreement: Kendall’s Tau computed across four groups of 10 experts to assess evaluation consistency.

Limitations & caveats

• Reliance on proprietary quantitative tools and real-time database limits reproducibility and generalizability.
• Expert curation of Stocksis is costly and time-consuming, potentially hindering dataset scaling.
• Evaluation is subjective and relies on human experts, though inter-annotator agreement is high.
• Performance may vary across different market conditions and asset classes not covered in the dataset.
• The agent's effectiveness is tied to the quality and coverage of the underlying real-time data sources.

Curator notes

Includes Stocksis dataset and stock analysis report generation.