The List / Foundation and Domain Language Models / paper
Mixing It Up: The Cocktail Effect of Multi-Task Fine-Tuning on LLM Performance - A Case Study in Finance
Abstract
The application of large language models (LLMs) in domain-specific contexts, including finance, has expanded rapidly. Domain-specific LLMs are typically evaluated based on their performance in various downstream tasks relevant to the domain. In this work, we present a detailed analysis of fine-tuning LLMs for such tasks. Somewhat counterintuitively, we find that in domain-specific cases, fine-tuning exclusively on the target task is not always the most effective strategy. Instead, multi-task finetuning - where models are trained on a cocktail of related tasks - can significantly enhance performance. We demonstrate how this approach enables a small model, such as Phi-3-Mini, to achieve state-of-the-art results, even surpassing the much larger GPT-4-o model on financial benchmarks. Our study involves a large-scale experiment, conducting over 200 training experiments using several widely adopted LLMs as baselines, and empirically confirms the benefits of multi-task fine-tuning. Additionally, we explore the use of general instruction data as a form of regularization, suggesting that it helps minimize performance degradation. We also investigate the inclusion of mathematical data, finding improvements in numerical reasoning that transfer effectively to financial tasks. Finally, we note that while fine-tuning for downstream tasks leads to targeted improvements in task performance, it does not necessarily result in broader gains in domain knowledge or complex domain reasoning abilities.
In one sentence
Summary
This study investigates the efficacy of multi-task fine-tuning for adapting large language models to the financial domain, challenging the conventional wisdom that single-task fine-tuning is optimal. The authors conduct a large-scale ablation study with over 200 experiments across four models (Phi-3-Small, Mistral-7B, Llama-3.1-8B, and Phi-3-Mini) to analyze how combining various financial downstream tasks affects performance. They introduce the concept of a "cocktail effect," where training on a mix of related tasks yields synergistic improvements in accuracy compared to training on the target task alone. The research also explores the role of non-financial data, specifically general instruction-following data (Open-Orca) and mathematical reasoning data (Orca-Math), to assess their impact on domain-specific performance and numerical reasoning capabilities. The experimental design involves fine-tuning models on seven core financial datasets covering sentiment analysis, named entity recognition, headline classification, and numerical reasoning (FinQA, ConvFinQA). The authors evaluate performance using standard metrics like accuracy and exact match, as well as LLM-as-a-Judge for open-ended questions. A key finding is that multi-task fine-tuning allows the 3.8B parameter Phi-3-Mini model to outperform the significantly larger GPT-4-o model on most financial benchmarks, including challenging conversational financial QA tasks. The study further analyzes the contribution of general datasets, hypothesizing that Open-Orca acts as a regularizer, preventing performance degradation by keeping the model aligned with its pre-trained distribution, while Orca-Math enhances numerical reasoning that transfers to financial tasks. Despite significant gains in specific task performance, the paper highlights a critical limitation: multi-task fine-tuning does not necessarily translate to improved general domain knowledge or complex reasoning abilities. Evaluations on broader benchmarks like MMLU-Pro and FinanceBench show mixed results, with some models even experiencing regression in general business and economics knowledge. This suggests that while multi-task fine-tuning is highly effective for optimizing performance on targeted downstream tasks, it may not be sufficient for achieving comprehensive domain adaptation. The authors conclude that future work should explore hybrid approaches combining multi-task learning with other domain adaptation strategies to bridge the gap between task-specific proficiency and broader domain understanding.
Research problem
The paper addresses the problem of how to best fine-tune large language models for domain-specific tasks in finance. It questions whether single-task fine-tuning is the most effective strategy or if multi-task fine-tuning offers superior performance. Additionally, it investigates the role of general instruction and mathematical data in enhancing financial task performance and whether these improvements generalize to broader domain knowledge.
Core contributions
- Empirical demonstration of the "cocktail effect" where multi-task fine-tuning outperforms single-task fine-tuning across multiple financial benchmarks.
- Showcase of a small model (Phi-3-Mini) surpassing a much larger state-of-the-art model (GPT-4-o) on financial tasks through multi-task fine-tuning.
- Analysis of the regularization effect of general instruction data (Open-Orca) and the transfer of numerical reasoning from mathematical data (Orca-Math) to financial tasks.
- Identification of the disconnect between task-specific performance gains and broader domain knowledge improvement, highlighting limitations of current fine-tuning strategies.
Method
- Full fine-tuning of model weights using standard uniform shuffling of datasets.
- Incremental fine-tuning approach: single-task, pairs, leave-one-out, and full cocktail.
- Evaluation using accuracy for classification and heuristic relaxed exact match for numerical tasks.
- LLM-as-a-Judge (GPT-4-o) for scoring open-ended responses in FinanceBench.
- Hyperparameter search for learning rates (3e-6 to 3e-5) and epochs (2-3).
- Ablation study to identify the most helpful dataset combinations for each task.
Results & findings
- Multi-task fine-tuning consistently outperforms single-task fine-tuning across all tested models and tasks.
- Phi-3-Mini (3.8B params) outperforms GPT-4-o on most financial benchmarks after multi-task fine-tuning.
- Inclusion of Open-Orca data improves performance, likely acting as a regularizer.
- Inclusion of Orca-Math data improves numerical reasoning, benefiting tasks like FinQA.
- Performance gains on specific tasks do not correlate with improvements in general domain knowledge (MMLU-Pro/FinanceBench).
- ConvFinQA benefits significantly from multi-task fine-tuning, likely due to implicit few-shot learning opportunities.
- Some models showed regression in general business/economics knowledge after financial fine-tuning.
Data & experiments
- 220 training experiments across four LLMs: Phi-3-Small, Mistral-7B, Llama-3.1-8B, and Phi-3-Mini.
- Seven core financial datasets: Headline, FPB, FinNerCLS, FinQA, ConvFinQA, Twitter-Topics, and Twitter-SA.
- Two general training datasets: Open-Orca (instruction following) and Orca-Math (mathematical reasoning).
- Evaluation on core financial tasks and broader benchmarks: MMLU-Pro (Business/Economics) and FinanceBench.
- Metrics include classification accuracy, numerical exact match with heuristic relaxation, and LLM-as-a-Judge for open-ended responses.
Limitations & caveats
- Hyperparameter search was not exhaustive due to the scale of experiments.
- Study focuses on a limited set of seven financial tasks, not covering the entire financial domain.
- Lack of modern theoretical framework for multi-task learning in deep learning.
- Improvements in specific tasks do not guarantee improvements in broader domain knowledge.
- Potential for overfitting to specific benchmark formats or tasks.
- Evaluation of open-ended questions relies on LLM-as-a-Judge, which may have biases.
Curator notes
Discovery status: round1_promoted_seed_for_round2. Source links: BBT-Fin: Comprehensive Construction of Chinese Financial Domain Pre-trained Language Model, Corpus and Benchmark; BloombergGPT: A Large Language Model for Finance; ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering; Exploring the Synergy of Quantitative Factors and Newsflow Representations from Large Language Models for Stock Return Prediction; FinBen: A Holistic Financial Benchmark for Large Language Models; FinDABench: Benchmarking Financial Data