The List / Multimodal and Multilingual Finance / paper
Plutus: Benchmarking Large Language Models in Low-Resource Greek Finance
Abstract
Despite Greece's pivotal role in the global economy, large language models (LLMs) remain underexplored for Greek financial context due to the linguistic complexity of Greek and the scarcity of domain-specific datasets. Previous efforts in multilingual financial natural language processing (NLP) have exposed considerable performance disparities, yet no dedicated Greek financial benchmarks or Greek-specific financial LLMs have been developed until now. To bridge this gap, we introduce Plutus-ben, the first Greek Financial Evaluation Benchmark, and Plutus-8B, the pioneering Greek Financial LLM, fine-tuned with Greek domain-specific data. Plutus-ben addresses five core financial NLP tasks in Greek: numeric and textual named entity recognition, question answering, abstractive summarization, and topic classification, thereby facilitating systematic and reproducible LLM assessments. To underpin these tasks, we present three novel, high-quality Greek financial datasets, thoroughly annotated by expert native Greek speakers, augmented by two existing resources. Our comprehensive evaluation of 22 LLMs on Plutus-ben reveals that Greek financial NLP remains challenging due to linguistic complexity, domain-specific terminology, and financial reasoning gaps. These findings underscore the limitations of cross-lingual transfer, the necessity for financial expertise in Greek-trained models, and the challenges of adapting financial LLMs to Greek text. We release Plutus-ben, Plutus-8B, and all associated datasets publicly to promote reproducible research and advance Greek financial NLP, fostering broader multilingual inclusivity in finance.
In one sentence
Summary
This paper addresses the critical gap in low-resource financial NLP by introducing Plutus-ben, the first comprehensive evaluation benchmark for Greek financial language processing. The authors argue that existing multilingual models fail to capture the linguistic complexity and domain-specific reasoning required for Greek financial texts, which are central to international trade and maritime finance. To solve this, they construct a rigorous benchmark covering five core tasks: numeric and textual named entity recognition, question answering, abstractive summarization, and topic classification. The work positions itself as a foundational resource for multilingual inclusivity in finance, providing the necessary infrastructure to assess and improve LLM capabilities in under-resourced languages. The methodology involves curating four new high-quality datasets (GRFinNUM, GRFinNER, GRFinQA, and GRFinSUM) annotated by expert native Greek speakers with financial expertise, supplemented by two existing resources. The authors fine-tune Llama-Krikri-8B using Low-Rank Adaptation on an instruction dataset derived from these sources to create Plutus-8B. They conduct a comprehensive evaluation of 22 diverse LLMs, including proprietary models like GPT-4 and open-source variants, measuring performance via Entity F1, Accuracy, and Rouge-1. The experimental design isolates the impact of domain adaptation by comparing Greek-general models against English-financial models and the newly proposed Greek-financial model. Results reveal that even top-tier models like GPT-4o struggle with Greek financial reasoning, with smaller models failing entirely on NER tasks. Plutus-8B achieves state-of-the-art performance, surpassing GPT-4 by 15.38% and GPT-4o by 46.34%, proving that domain-specific fine-tuning is essential for low-resource financial NLP. However, the study highlights significant limitations, particularly in abstractive summarization where all models struggle with long-form documents. The findings underscore the limitations of cross-lingual transfer and the necessity for financial expertise in Greek-trained models, offering a valuable case study for deploying LLMs in specialized, low-resource linguistic markets.
Research problem
The paper addresses the lack of dedicated benchmarks and domain-specific large language models for Greek financial natural language processing. It highlights that Greek's linguistic complexity and the scarcity of domain-specific datasets hinder the application of LLMs in this critical economic sector. The research aims to bridge this gap by creating a standardized evaluation framework and a fine-tuned model to assess and improve LLM performance on Greek financial texts.
Core contributions
- Introduction of Plutus-ben, the first Greek financial evaluation benchmark covering five core NLP tasks.
- Development of Plutus-8B, the pioneering Greek financial LLM fine-tuned on domain-specific data.
- Creation of four high-quality, expert-annotated Greek financial datasets (GRFinNUM, GRFinNER, GRFinQA, GRFinSUM).
- Comprehensive evaluation of 22 LLMs, revealing significant performance disparities in low-resource financial contexts.
- Public release of all datasets, benchmarks, and the Plutus-8B model to foster reproducible research.
Method
- Curated datasets from authoritative sources like annual reports and university exams.
- Expert annotation by native Greek speakers with financial and linguistic expertise.
- Conversion of raw datasets into structured instruction formats for fine-tuning.
- Fine-tuning Llama-Krikri-8B using Low-Rank Adaptation (LoRA) with rank 16.
- Applied int4 quantization and mixed-precision training (bf16) for efficiency.
- Evaluated models using automated metrics (Entity F1, Acc, Rouge-1) and human judgment.
- Conducted ablation studies by comparing general, financial, and Greek-specific models.
Results & findings
- Plutus-8B achieved the highest mean score, outperforming GPT-4 by 15.38% and GPT-4o by 46.34%.
- Small open-source models like LLaMA-3.2-1B and Mistral-7B performed near zero on NER tasks.
- English financial models (FinLLaMA-8B) failed to generalize to Greek, showing near-zero NER performance.
- Greek general models (Meltemi-7B) lacked financial expertise, underperforming compared to Plutus-8B.
- Scaling model size showed diminishing returns; Qwen2.5-72B did not significantly outperform Qwen2.5-32B.
- All models struggled with abstractive summarization, indicating challenges with long-form financial document comprehension.
- Inter-annotator agreement was high, with F1 scores above 0.97 for NER datasets.
Data & experiments
- Curated GRFinNUM (500 samples) and GRFinNER (500 samples) from Athens Stock Exchange annual reports.
- Developed GRFinQA (540 samples) from Greek university finance exams and textbooks.
- Utilized GRFNS-2023 (262 samples) for abstractive summarization of Greek annual reports.
- Adapted GRMultiFin (268 samples) for topic classification of Greek financial news headlines.
- Evaluated 22 LLMs including proprietary (GPT-4, GPT-4o) and open-source models (LLaMA, Qwen, Mistral).
- Used Entity F1, Accuracy, and Rouge-1 metrics, supplemented by human evaluation for summarization quality.
Limitations & caveats
- The benchmark is limited to the Greek language, restricting generalizability to other low-resource languages.
- Abstractive summarization performance remains poor across all models, indicating a significant capability gap.
- The datasets are relatively small (500-540 samples per task), which may limit the robustness of evaluations.
- The study focuses on static financial texts (reports, exams) and does not cover real-time market data.
- Human evaluation was limited to summarization, leaving other tasks reliant on automated metrics.
- The fine-tuned model is based on Llama-Krikri-8B, which may inherit biases or limitations from its base.
Curator notes
Discovery status: round2_promoted. Source links: DISC-FinLLM: A Chinese Financial Large Language Model based on Multiple Experts Fine-tuning; FinGPT: Instruction Tuning Benchmark for Open-Source Large Language Models in Financial Datasets; From Scores to Skills: A Cognitive Diagnosis Framework for Evaluating Financial Large Language Models; InvestLM: A Large Language Model for Investment using Financial Domain Instruction Tuning; Unlocking Data Value in Finance: A Study on Distillation and Difficulty-Aware Training