InvestLM: A Large Language Model for Investment using Financial Domain Instruction Tuning

In one sentence

Summary

This paper addresses the lack of accessible, high-performance open-source financial LLMs by introducing InvestLM, a model instruction-tuned on LLaMA-65B. The authors challenge the assumption that massive instruction datasets are necessary for domain alignment, instead leveraging the Superficial Alignment Hypothesis to curate a small, diverse, and manually verified dataset. The research problem centers on creating a model that not only understands financial text but also provides actionable, logical investment advice without the hallucinations common in base foundation models or the vagueness of commercial safety-aligned models. The methodology involves constructing a 1,335-example instruction dataset sourced from CFA exams, StackExchange QuantFinance, academic journals, textbooks, SEC filings, and financial NLP tasks. The model is fine-tuned using LoRA with linear rope scaling to handle long contexts. Evaluation includes expert reviews by hedge fund managers and analysts comparing InvestLM against GPT-3.5, GPT-4, and Claude-2 on investment scenarios, as well as zero-shot testing on nine financial NLP benchmarks including sentiment analysis, numerical reasoning (FinQA), and summarization. The study also ablates the impact of adding generic Alpaca instructions to the domain-specific dataset. Findings indicate that InvestLM’s responses are rated comparable to or better than GPT-3.5 and GPT-4 by financial experts, particularly in providing concise, decisive advice. On benchmarks, InvestLM outperforms LLaMA-65B in 8 of 9 tasks and achieves state-of-the-art results on FinSent and FiQA. Crucially, the study finds that adding generic instructions (Alpaca) degrades performance on domain tasks, supporting the 'less-is-more' hypothesis for domain alignment. The model significantly reduces hallucinations compared to the base LLaMA model, though it remains a research tool rather than a direct trading system.

Research problem

The paper addresses the scarcity of publicly available, high-capability financial LLMs, noting that proprietary models like BloombergGPT are inaccessible and existing open models (FinMA, FinGPT) are smaller or less generalizable. It specifically investigates whether a large foundation model can be effectively aligned for financial investment tasks using a small, high-quality instruction dataset, challenging the need for massive instruction corpora.

Core contributions

• Introduction of InvestLM, an open-source 65B parameter financial LLM instruction-tuned on LLaMA-65B.
• Creation of a manually curated, diverse financial instruction dataset (~1,300 examples) from CFA exams, SEC filings, and academic sources.
• Empirical validation of the Superficial Alignment Hypothesis in the financial domain, showing small high-quality datasets suffice for alignment.
• Demonstration that generic instruction tuning (e.g., Alpaca) can negatively impact domain-specific financial NLP performance.
• Release of model weights and evaluation benchmarks to the research community.

Method

• Base model: LLaMA-65B.
• Fine-tuning method: Low-rank adaptation (LoRA) with rank 16.
• Target modules: q_proj, k_proj, v_proj, down_proj, gate_proj, up_proj.
• Context extension: Linear Rope Scaling with scale factor 4 to extend context to 8,192 tokens.
• Training hyperparameters: 15 epochs, learning rate 3e-4, batch size 16 for 65B model.
• Dataset curation: Manual verification and paraphrasing of answers from ChatGPT/Claude-2 and textbooks.
• Evaluation protocol: Pairwise comparison by human experts and GPT-4 judge.

Results & findings

• Experts rated InvestLM responses as comparable to or better than GPT-3.5 and GPT-4, and often preferred over GPT-4's vague safety responses.
• InvestLM outperformed LLaMA-65B in 8 out of 9 financial NLP benchmarks.
• InvestLM achieved state-of-the-art Micro-F1 on FinSent (0.79) and FiQA (0.90) among open models.
• Adding generic Alpaca instructions to InvestLM-7B degraded performance across all financial NLP tasks.
• Domain instruction tuning improved LLaMA-7B performance by an average of 138.4%, compared to 28.2% for LLaMA-65B.
• InvestLM significantly reduced hallucinations compared to base LLaMA, providing grounded, logical investment reasoning.

Data & experiments

• Instruction dataset: 1,335 examples from StackExchange, CFA, academic journals, textbooks, SEC filings, and financial NLP tasks.
• Expert evaluation: 6 financial experts (hedge fund managers, analysts) compared InvestLM against GPT-3.5, GPT-4, and Claude-2 on 30 investment scenarios.
• GPT-4 evaluation: Automated preference judgment using GPT-4 as a judge for the same 30 scenarios.
• Zero-shot NLP benchmarks: Evaluated on FinSent, FPB, FOMC, FiQA, ESG, FLS, QA, FinQA, and ECTSum.
• Ablation study: Compared InvestLM-7B vs. InvestLM-7B+Alpaca to test the impact of generic instructions.

Limitations & caveats

• Instruction dataset is small (1,335 examples), potentially limiting coverage of niche financial topics.
• Expert evaluation is subjective and limited to 6 experts, though inter-annotator agreement is reasonable.
• Model is not fine-tuned for real-time trading or execution, limiting direct application in algorithmic trading.
• Ethics statement warns against using outputs as definitive investment advice, highlighting liability risks.
• Performance on proprietary datasets (FinSent, ESG, FLS, QA) may not be fully reproducible by external researchers.
• Does not address real-time data integration or live market data processing.

Curator notes

Discovery status: round1_promoted_seed_for_round2. Source links: A Comprehensive Review of Generative AI in Finance; A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges; A Survey of Large Language Models in Finance: FinLLMs; Advancing Financial Engineering with Foundation Models: Progress, Applications, and Challenges; Automate Strategy Finding with LLM in Quant investment; BizFinBench: A Business-Driven Real-World Financial Benchmark for Evaluating LLMs; BloombergGPT: A Large Language Model for Finance; Dólares or