The List / Multimodal and Multilingual Finance / paper
No Language is an Island: Unifying Chinese and English in Financial Large Language Models, Instruction Data, and Benchmarks
Abstract
While the progression of Large Language Models (LLMs) has notably propelled financial analysis, their application has largely been confined to singular language realms, leaving untapped the potential of bilingual Chinese-English capacity. To bridge this chasm, we introduce ICE-PIXIU, seamlessly amalgamating the ICE-INTENT model and ICE-FLARE benchmark for bilingual financial analysis. ICE-PIXIU uniquely integrates a spectrum of Chinese tasks, alongside translated and original English datasets, enriching the breadth and depth of bilingual financial modeling. It provides unrestricted access to diverse model variants, a substantial compilation of diverse cross-lingual and multi-modal instruction data, and an evaluation benchmark with expert annotations, comprising 10 NLP tasks, 20 bilingual specific tasks, totaling 95k datasets. Our thorough evaluation emphasizes the advantages of incorporating these bilingual datasets, especially in translation tasks and utilizing original English data, enhancing both linguistic flexibility and analytical acuity in financial contexts. Notably, ICE-INTENT distinguishes itself by showcasing significant enhancements over conventional LLMs and existing financial LLMs in bilingual milieus, underscoring the profound impact of robust bilingual data on the accuracy and efficacy of financial NLP.
In one sentence
Summary
The paper addresses the critical gap in bilingual Chinese-English financial Large Language Models (LLMs), noting that existing models are predominantly monolingual and lack comprehensive evaluation resources for cross-lingual capabilities. The authors argue that the growing interaction between Chinese and US financial markets necessitates tools that can handle both languages effectively, yet current research largely ignores the potential of bilingual capacity. To bridge this chasm, the study positions itself as the first to provide a unified open-source framework for bilingual financial NLP, aiming to improve linguistic flexibility and analytical acuity in cross-border financial contexts. The core contribution is the ICE-PIXIU framework, which includes the ICE-FIND dataset, the ICE-INTENT model, and the ICE-FLARE benchmark. ICE-FIND comprises 604k instruction samples across 40 datasets, covering 18 specific tasks such as sentiment analysis, extraction, and prediction in both languages, including original English data and translated Chinese data. The ICE-INTENT model is developed by fine-tuning the InternLM-7B backbone using QLoRA on this diverse dataset. The ICE-FLARE benchmark evaluates 40 datasets across 10 NLP tasks and 20 bilingual-specific tasks, uniquely including data-out-of-training (DOT) sets to test generalization. Experiments compare ICE-INTENT against SOTA models like GPT-4 and ChatGPT on these benchmarks. Results show that ICE-INTENT significantly outperforms existing models, particularly in Chinese financial tasks, often exceeding GPT-4's performance. The study finds that incorporating translated data enhances cross-lingual generalization and boosts English task performance, highlighting the benefits of bilingual training. However, the paper notes that current LLMs exhibit notable performance disparities between languages. Limitations include the reliance on machine translation for some Chinese datasets, which may introduce noise, and the focus on specific NLP tasks rather than end-to-end trading systems. The work provides a foundational resource for bilingual financial AI but does not directly address algorithmic trading or portfolio optimization.
Research problem
Existing financial LLMs are predominantly monolingual, focusing either on English or Chinese, which limits their utility in cross-border financial analysis where Chinese and English data interact. There is a lack of open-source bilingual instruction datasets and comprehensive evaluation benchmarks that assess cross-lingual generalization and performance in both languages simultaneously. This gap hinders the development of models capable of handling the complex linguistic nuances of global financial markets effectively.
Core contributions
- Introduction of ICE-PIXIU, the first open-source framework for Chinese-English bilingual financial LLMs.
- Creation of ICE-FIND, a large-scale bilingual instruction dataset with 604k samples across 40 datasets and 18 specific tasks.
- Development of ICE-INTENT, a bilingual financial LLM fine-tuned on InternLM-7B, demonstrating superior performance in Chinese tasks.
- Construction of ICE-FLARE, a comprehensive bilingual evaluation benchmark including data-out-of-training sets for generalization testing.
- Demonstration that fine-tuning with translated data enhances cross-lingual generalization and improves English task performance.
- Provision of expert-annotated prompts and high-quality human-evaluated instruction data for financial NLP tasks.
Method
- Data curation: Integrated Chinese and English financial datasets, including original and translated data.
- Translation: Used ChatGPT-3.5-Turbo to translate English datasets to Chinese, with expert review for quality.
- Instruction construction: Human experts wrote 30-40 prompts per dataset, evaluated for accuracy and naturalness.
- Model training: Fine-tuned InternLM-7B using QLoRA with ICE-FIND dataset; batch size 24, 1 epoch.
- Benchmark design: ICE-FLARE includes in-training and out-of-training (DOT) tasks to test generalization.
- Evaluation: Comprehensive metrics including F1, Accuracy, EM, MCC, and text generation scores.
- Ablation studies: Tested impact of different data types (DLC, DLE, DTT, DTE) on model performance.
- Prompt optimization: Retained prompts with average human evaluation scores above 2.
Results & findings
- ICE-INTENT outperforms GPT-4 in Chinese financial tasks, particularly in sentiment analysis and extraction.
- Bilingual fine-tuning significantly improves cross-lingual generalization, benefiting both Chinese and English performance.
- Incorporating translated English-to-Chinese data enhances model robustness and linguistic flexibility.
- Existing LLMs show significant performance disparities between Chinese and English, with Chinese often lagging.
- ICE-INTENT achieves state-of-the-art results on many Chinese-specific tasks in the ICE-FLARE benchmark.
- Data-out-of-training evaluation reveals gaps in generalization capabilities of current monolingual models.
- Expert-annotated prompts and high-quality data curation are critical for improving instruction-following in finance.
- The study highlights the need for more diverse and high-quality bilingual financial datasets.
Data & experiments
- ICE-FIND dataset: 604k instruction samples, 40 datasets, covering classification, extraction, prediction, reasoning, and translation tasks.
- Data sources include Chinese financial news, social media, bank logs, and English datasets from PIXIU, FinQA, and others.
- ICE-FLARE benchmark: 40 datasets, 10 NLP tasks, 20 bilingual-specific tasks, including data-out-of-training (DOT) sets.
- Model: ICE-INTENT (InternLM-7B backbone) fine-tuned with QLoRA; variants tested for ablation on data types.
- Baselines: GPT-4, ChatGPT, FinGPT, FinMA, CFGPT, DISC-FinLLM, and other open-source LLMs.
- Metrics: Accuracy, F1, Entity F1, Exact Match, MCC, Rouge, BERTScore, and BARTScore across various tasks.
Limitations & caveats
- Reliance on machine translation for some Chinese datasets may introduce noise or inaccuracies.
- Focus is on general financial NLP tasks, not specific trading or investment strategies.
- Evaluation is limited to the ICE-FLARE benchmark; generalization to other unseen tasks is not fully explored.
- The study does not address real-time or high-frequency financial data processing.
- Potential bias in human evaluation of prompts, though mitigated by multiple evaluators.
- Model size (7B) may limit performance compared to larger proprietary models in some complex reasoning tasks.
Curator notes
Discovery status: round1_promoted_seed_for_round2. Source links: A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges; Advancing Financial Engineering with Foundation Models: Progress, Applications, and Challenges; Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? An Examination on Several Typical Tasks; BBT-Fin: Comprehensive Construction of Chinese Financial Domain Pre-trained Language Model, Corpus and Benchmark; BloombergGPT: A Large Language Model for Finance; ConvFinQA: Exploring the