Exploring Large Language Models for Financial Applications: Techniques, Performance, and Challenges with FinMA

In one sentence

Summary

This paper addresses the critical need for robust evaluation of domain-adapted Large Language Models (FinLLMs) in financial natural language processing. While general-purpose LLMs like GPT-4 offer broad reasoning capabilities, they often lack the specialized accuracy required for finance. The research positions FinMA, an open-source model from the PIXIU framework, as a key subject for understanding the trade-offs between domain-specific instruction tuning and general pre-training. The study aims to identify specific architectural and training distinctions that enable effective financial NLP, focusing on accuracy, reliability, and interpretability in high-stakes decision-making contexts. The methodology involves a comprehensive empirical evaluation of the FinMA-7B-full model using the Financial Instruction Tuning (FIT) dataset and the FLARE benchmark. The authors conduct experiments across six primary financial NLP tasks: sentiment analysis, text classification, named entity recognition, question answering, stock movement prediction, and text summarization. The experimental design includes re-evaluating baselines such as GPT-4 and BloombergGPT under consistent conditions, utilizing Hugging Face datasets and metrics like F1, accuracy, and ROUGE. The study also explores the impact of prompt engineering techniques, such as Chain-of-Thought, and assesses the model's performance in zero-shot and few-shot settings to gauge robustness and generalization capabilities. Findings indicate that FinMA-7B-full excels in sentiment analysis (93.9% F1) and text classification (97.5% Avg F1), outperforming GPT-4 in zero-shot settings for these specific tasks. However, it struggles significantly with numerical reasoning (7.4% Exact Match on FinQA), named entity recognition (64.1% F1), and summarization (2.8% ROUGE). The authors attribute these limitations to the LLaMA backbone's general-purpose nature and insufficient training data for complex reasoning and multimodal tasks. The study highlights challenges such as hallucination risks, bias in training data, and the need for specialized datasets. It suggests future directions including Retrieval-Augmented Generation (RAG), Low-Rank Adaptation (LoRA), and hybrid models integrating time-series data to enhance performance in stock prediction and numerical tasks.

Research problem

The paper addresses the gap in understanding the specific strengths and weaknesses of domain-adapted Large Language Models (FinLLMs) compared to general-purpose models in financial NLP tasks. It highlights the limitations of current evaluation frameworks and the need for specialized benchmarks that reflect practical financial requirements, such as numerical reasoning and entity recognition, which are often overlooked in general LLM assessments.

Core contributions

• Empirical evaluation of the open-source FinMA-7B-full model on the FLARE benchmark across six financial NLP tasks.
• Re-evaluation of baseline models including GPT-4 and BloombergGPT under consistent experimental conditions.
• Analysis of the impact of instruction tuning via the FIT dataset on model performance in sentiment, classification, and reasoning tasks.
• Identification of specific failure modes in FinMA, particularly in numerical reasoning, entity recognition, and summarization.
• Assessment of prompt engineering techniques, revealing limited gains from Chain-of-Thought in complex financial tasks.

Method

• Used FinMA-7B-full, an open-source LLM from the PIXIU framework.
• Fine-tuned on the Financial Instruction Tuning (FIT) dataset containing 136,609 samples.
• Evaluated using the FLARE benchmark covering sentiment, classification, NER, QA, SMP, and summarization.
• Employed task-specific metrics: F1, Accuracy, MCC, Exact Match, and ROUGE.
• Conducted zero-shot and few-shot experiments to assess generalization.
• Tested prompt engineering techniques including Chain-of-Thought and Meta Prompting.
• Compared performance against GPT-4 and BloombergGPT baselines.
• Analyzed results to identify strengths in text understanding and weaknesses in reasoning.

Results & findings

• FinMA-7B-full achieved 93.9% F1 on Financial PhraseBank sentiment analysis, outperforming baselines.
• Text classification on Headlines dataset reached 97.5% Average F1, slightly exceeding GPT-4 in zero-shot.
• Named Entity Recognition (NER) performance was weak at 64.1% F1, indicating difficulty with entity extraction.
• Question Answering on FinQA showed only 7.4% Exact Match accuracy, highlighting poor numerical reasoning.
• Stock Movement Prediction accuracy was low, ranging from 50.5% to 53.0%, near random chance.
• Text Summarization on ECTSum yielded a negligible 2.8% ROUGE score, indicating ineffective summarization.
• Prompt engineering techniques like Chain-of-Thought did not significantly improve performance on complex tasks.
• Few-shot learning showed instability, with performance drops in some classification tasks.

Data & experiments

• FinMA-7B-full model fine-tuned on the Financial Instruction Tuning (FIT) dataset.
• FLARE benchmark from the PIXIU framework for comprehensive financial NLP evaluation.
• Datasets including Financial PhraseBank, Headlines, FINER-139, FinQA, ConvFinQA, and BigData22.
• Metrics: Accuracy, F1 Score, Matthews Correlation Coefficient, Exact Match, and ROUGE scores.
• Comparison against baselines: GPT-4, BloombergGPT, and FinMA-30B.
• Experiments in zero-shot and few-shot settings to test robustness.

Limitations & caveats

• FinMA-7B-full shows poor performance in numerical reasoning and entity recognition.
• Summarization capabilities are weak, with very low ROUGE scores.
• Stock movement prediction accuracy is near random chance, limiting practical trading utility.
• The study was conducted on limited hardware (Kaggle T4 GPUs), potentially affecting scalability.
• Prompt engineering techniques did not yield significant improvements, suggesting a need for better domain-specific prompts.
• Potential biases in the LLaMA backbone and FIT dataset may skew results.
• The FIT dataset lacks dedicated tasks for summarization and complex numerical reasoning.

Curator notes

Discovery status: round2_promoted. Source links: FinBERT: A Pretrained Language Model for Financial Communications; FinBERT: Financial Sentiment Analysis with Pre-trained Language Models; FinGPT: Instruction Tuning Benchmark for Open-Source Large Language Models in Financial Datasets; FinTral: A Family of GPT-4 Level Multimodal Financial Large Language Models; InvestLM: A Large Language Model for Investment using Financial Domain Instruction Tuning; Open-FinLLMs: Open Multimodal Large Language Models for Financial Applications