The List / Multimodal and Multilingual Finance / paper
Dólares or Dollars? Unraveling the Bilingual Prowess of Financial LLMs Between Spanish and English
Abstract
Despite Spanish's pivotal role in the global finance industry, a pronounced gap exists in Spanish financial natural language processing (NLP) and application studies compared to English, especially in the era of large language models (LLMs). To bridge this gap, we unveil Toisón de Oro, the first bilingual framework that establishes instruction datasets, finetuned LLMs, and evaluation benchmark for financial LLMs in Spanish joint with English. We construct a rigorously curated bilingual instruction dataset including over 144K Spanish and English samples from 15 datasets covering 7 tasks. Harnessing this, we introduce FinMA-ES, an LLM designed for bilingual financial applications. We evaluate our model and existing LLMs using FLARE-ES, the first comprehensive bilingual evaluation benchmark with 21 datasets covering 9 tasks. The FLARE-ES benchmark results reveal a significant multilingual performance gap and bias in existing LLMs. FinMA-ES models surpass SOTA LLMs such as GPT-4 in Spanish financial tasks, due to strategic instruction tuning and leveraging data from diverse linguistic resources, highlighting the positive impact of cross-linguistic transfer. All our datasets, models, and benchmarks have been released.
In one sentence
Summary
The paper addresses the significant disparity in financial Natural Language Processing (NLP) resources between English and Spanish, noting that existing large language models (LLMs) are predominantly English-centric despite Spanish's global economic importance. The authors identify a critical gap in bilingual financial tools and propose Toisón de Oro, a comprehensive framework designed to bridge this divide by providing open-source datasets, models, and evaluation benchmarks for Spanish-English financial NLP. This positioning highlights the need for multilingual capabilities in FinTech to serve diverse demographics and global markets effectively. Methodologically, the authors construct FIT-ES, a bilingual instruction tuning dataset comprising over 144,000 samples from 15 sources covering seven tasks, including sentiment analysis, classification, question answering, and summarization. They fine-tune the LLaMA2-7B model to create FinMA-ES, utilizing both Spanish and English data to leverage cross-linguistic transfer. Evaluation is performed using FLARE-ES, a novel benchmark with 21 datasets across nine tasks, including unseen data to test generalization. The experimental design compares FinMA-ES against state-of-the-art models like GPT-4, ChatGPT, and other open-source LLMs using metrics such as F1, Accuracy, and ROUGE scores. Results indicate that FinMA-ES significantly outperforms GPT-4 in four out of six Spanish financial tasks, particularly in classification and question answering, validating the efficacy of strategic instruction tuning and cross-lingual data integration. The study reveals a pronounced performance gap in existing LLMs for Spanish financial contexts. However, limitations include the model's 7B parameter size, which may restrict depth, and poor performance on complex summarization tasks. The authors also note ethical concerns regarding the dissemination of financial information and recommend using the model primarily for research purposes.
Research problem
The paper addresses the lack of high-quality Spanish financial NLP resources and models compared to English, which limits the effectiveness of LLMs in serving Spanish-speaking financial markets. It aims to bridge this gap by creating a bilingual framework that supports both Spanish and English financial tasks, enabling better cross-lingual understanding and application in global finance.
Core contributions
- Introduction of Toisón de Oro, the first open-source bilingual Spanish-English financial LLM framework.
- Creation of FIT-ES, a curated instruction tuning dataset with 144K+ samples across 7 financial tasks.
- Development of FinMA-ES, a 7B-parameter LLM fine-tuned for bilingual financial applications.
- Establishment of FLARE-ES, a comprehensive evaluation benchmark with 21 datasets across 9 tasks.
- Demonstration of cross-linguistic transfer benefits, showing Spanish data improves English performance and vice versa.
Method
- Data curation: Aggregated 15 public datasets for Spanish and English financial NLP tasks.
- Instruction construction: Designed structured prompts (Instruction, Text, Response) for each task and language.
- Model fine-tuning: Fine-tuned LLaMA2-7B using AdamW optimizer, batch size 1, learning rate 3e-4, for 5 epochs.
- Bilingual training: Trained FinMA-ES-Bilingual on combined Spanish and English data; FinMA-ES-Spanish on Spanish-only data for ablation.
- Evaluation: Tested on FLARE-ES benchmark, including held-out datasets to assess generalization.
- Metrics calculation: Used task-specific metrics like F1 for classification/sentiment, ROUGE for summarization, and EM for QA.
Results & findings
- FinMA-ES-Bilingual outperforms GPT-4 in MultiFin (Acc 0.99 vs 0.60), EFP (Acc 0.84 vs 0.27), EFPA (Acc 0.99 vs 0.34), and TSA (Acc 0.85 vs 0.47).
- FinMA-ES-Spanish also shows strong performance, surpassing GPT-4 in MultiFin, EFP, EFPA, and TSA.
- GPT-4 performs best in English tasks like FinQA and ConvFinQA but lags in Spanish-specific tasks.
- FinMA-ES shows improved performance on English tasks when trained with Spanish data, indicating positive cross-linguistic transfer.
- Summarization tasks (FNS-2023) remain challenging for all models, with GPT-4 being the only one to achieve non-zero ROUGE scores.
- Lince-zero performs well on FinanceES sentiment analysis, suggesting advantages from general domain Spanish pre-training.
Data & experiments
- FIT-ES dataset: 144K+ bilingual samples from 15 sources covering sentiment, classification, QA, summarization, NER, and stock prediction.
- FLARE-ES benchmark: 21 datasets including unseen data to test generalization across 9 financial tasks.
- Models evaluated: FinMA-ES (Bilingual and Spanish-only variants), GPT-4, ChatGPT, LLaMA2-7B, Falcon-7B, Bloomz-7B1-mt, Lince-zero, FinMA-7B-full, FinMA-30B-nlp.
- Metrics: F1, Accuracy, Exact Match, ROUGE, BERTScore, BARTScore, and Matthews Correlation Coefficient (MCC).
- Experiments focus on comparing FinMA-ES against SOTA closed-source and open-source models on Spanish and English financial tasks.
Limitations & caveats
- Model size is limited to 7B parameters, potentially restricting performance compared to larger models.
- Poor performance on text summarization tasks (FNS-2023) for most models, including FinMA-ES.
- Evaluation benchmark diversity is limited, particularly for complex generative tasks.
- Potential for disseminating inaccurate financial information requires careful ethical consideration.
- Performance on FinanceES sentiment analysis is lower than Lince-zero, suggesting general domain pre-training benefits.
- The study focuses on Spanish and English, limiting generalizability to other languages.
Curator notes
Discovery status: round1_promoted_seed_for_round2. Source links: A Comprehensive Review of Generative AI in Finance; Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? An Examination on Several Typical Tasks; BBT-Fin: Comprehensive Construction of Chinese Financial Domain Pre-trained Language Model, Corpus and Benchmark; BloombergGPT: A Large Language Model for Finance; ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering; DISC-FinLLM: A Chinese Financial Large Language Model based on Mult