Golden Touchstone: A Comprehensive Bilingual Benchmark for Evaluating Financial Large Language Models

In one sentence

Summary

The paper addresses the lack of standardized, high-quality evaluation tools for financial large language models, noting that existing benchmarks suffer from limited language coverage, inconsistent data quality, and inadequate task diversity. To resolve this, the authors introduce Golden Touchstone, a bilingual benchmark encompassing eight core financial NLP tasks in both English and Chinese. These tasks span Natural Language Understanding (sentiment analysis, classification, entity/relation extraction, multiple-choice knowledge) and Natural Language Generation (summarization, question answering, stock movement prediction). The benchmark integrates 22 curated datasets, ensuring consistent evaluation metrics and handling of unknown labels to prevent bias, thereby providing a robust framework for assessing model capabilities in real-world financial contexts. The experimental evaluation compares state-of-the-art proprietary models (GPT-4o) and open-source general models (Llama-3, Qwen-3) against specialized financial LLMs (FinGPT, FinMA, CFGPT, DISC-FinLLM). Additionally, the authors train and release Touchstone-GPT, a model based on Qwen-2.5, utilizing two-stage training: continual pre-training on 100 billion tokens of financial text and instruction tuning on 300,000 high-quality instruction-response pairs. The evaluation employs greedy decoding for reproducibility and assesses performance using task-specific metrics such as F1, accuracy, ROUGE, and BLEU across both English and Chinese datasets, highlighting the impact of inference templates on model performance. Results indicate that while sentiment analysis and simple classification are well-handled by most models, complex tasks like relation extraction, summarization, and question answering show significant performance gaps, particularly in Chinese. Stock movement prediction remains practically unusable for all models, as news sentiment alone is insufficient for accurate forecasting without quantitative data. Touchstone-GPT demonstrates strong competitive performance, often outperforming specialized models like FinGPT and FinMA, but still lags behind GPT-4o in complex reasoning. The study concludes that future financial LLMs require better multimodal integration and retrieval-augmented generation to handle numerical and temporal data effectively.

Research problem

Existing financial LLM benchmarks are fragmented, often limited to single languages, and suffer from low-quality data or inadequate task coverage. There is a pressing need for a standardized, bilingual evaluation framework that comprehensively assesses both understanding and generation capabilities in finance. This paper aims to fill this gap by providing a unified benchmark and a new baseline model to guide the development of more capable financial AI systems.

Core contributions

• Introduction of Golden Touchstone, a comprehensive bilingual benchmark covering eight financial NLP tasks in English and Chinese.
• Curated integration of 22 high-quality datasets with consistent evaluation metrics and handling of unknown labels.
• Extensive evaluation of 10+ state-of-the-art proprietary and open-source financial LLMs, revealing specific strengths and weaknesses.
• Open-sourcing of Touchstone-GPT, a financial LLM trained via continual pre-training and instruction tuning, serving as a new baseline.
• Analysis of inference template impacts, showing that prompt design significantly affects model performance on benchmarks.

Method

• Curated 22 datasets from existing benchmarks, ensuring high quality and bilingual alignment.
• Standardized evaluation by treating [unknown] labels as incorrect to avoid bias.
• Trained Touchstone-GPT using continual pre-training on 100B tokens and instruction tuning on 300k pairs.
• Used Megatron for pre-training and LlamaFactory for post-training with DeepSpeed ZeRO-1 optimization.
• Evaluated models using greedy decoding to ensure reproducibility and eliminate sampling variance.
• Analyzed performance across NLU and NLG tasks, comparing general vs. specialized models.

Results & findings

• GPT-4o leads in sentiment analysis and multiple-choice tasks but struggles with relation extraction and detailed entity extraction.
• Touchstone-GPT outperforms FinGPT and FinMA in most tasks, demonstrating the value of high-quality instruction tuning.
• Stock movement prediction is the weakest task for all models, with performance barely above random chance, indicating news sentiment is insufficient.
• Summarization and question answering show low scores, highlighting difficulties in generating coherent, detailed financial text.
• Chinese models generally underperform compared to English counterparts, and general models often lack the domain specificity of FinLLMs.
• Inference templates significantly impact results; improper prompting can degrade performance of even strong models.

Data & experiments

• Golden Touchstone benchmark includes 22 datasets across 8 tasks: sentiment analysis, classification, entity/relation extraction, multiple choice, summarization, question answering, and stock movement prediction.
• Datasets include FPB, FiQA-SA, Headlines, FOMC, LendingClub, NER, FinRE, CFA, EDTSUM, FinQA, and CMIN for both English and Chinese.
• Evaluated models: GPT-4o, Llama-3-8B, Qwen-3-8B, FinMA-7B, FinGPT-8B, CFGPT, DISC-FinLLM, and the proposed Touchstone-GPT.
• Touchstone-GPT trained on 100B financial tokens and 300k instruction pairs using Qwen-2.5 as the base model.
• Metrics include F1, Accuracy, ROUGE, BLEU, and MCC, with consistent handling of [unknown] labels as incorrect predictions.

Limitations & caveats

• The benchmark currently focuses on single-modality text, lacking multimodal data like charts or tables.
• Stock movement prediction is shown to be unreliable, limiting its practical trading application.
• The benchmark covers a limited range of NLG tasks, excluding extended report generation.
• Dataset biases may exist, and model scalability beyond 10B parameters is not fully explored.
• Performance on Chinese tasks is generally lower, indicating potential language-specific challenges.

Curator notes

Useful for cross-market and multilingual financial LLM studies.