Empowering Many, Biasing a Few: Generalist Credit Scoring through Large Language Models

In one sentence

Summary

This paper addresses the limitation of traditional credit scoring models, which are often task-specific and lack generalization. The authors propose a generalist approach using Large Language Models (LLMs) to handle multiple credit and risk assessment tasks, including credit scoring, fraud detection, financial distress identification, and claim analysis. The core research questions investigate whether LLMs can overcome narrow expertise, generalize across tasks via instruction tuning, and whether their deployment introduces fairness biases in sensitive financial decisions. The work positions LLMs as a potential paradigm shift from isolated expert systems to unified, adaptable risk assessment frameworks. The methodology involves curating a comprehensive benchmark of 9 datasets with 14,000 samples covering the four task types. The authors construct instruction-tuning data from 6 datasets, totaling 45,000 samples, using both table-based and description-based prompts to handle tabular financial data. They fine-tune Llama2-Chat using LoRA to create CALM, a 7-billion parameter model. Experiments compare CALM against open-source LLMs (Vicuna, Bloomz) and closed-source models (GPT-4, ChatGPT) as well as state-of-the-art expert systems. Evaluation metrics include accuracy, F1, Matthews Correlation Coefficient (MCC) for imbalanced data, and bias metrics (Disparate Impact, Equal Opportunity Difference) focusing on gender, age, and foreign status. Results show that GPT-4 matches or exceeds expert systems in several tasks, while CALM outperforms other open-source LLMs, demonstrating effective knowledge transfer. However, the study reveals significant biases in LLM predictions, particularly against unprivileged groups defined by age, gender, and nationality. The authors highlight that while LLMs offer inclusivity and comprehensive risk assessment, they risk perpetuating societal biases. The paper contributes open-source datasets, the CALM model, and a benchmark, emphasizing the need for ethical oversight in deploying LLMs for credit decisions to ensure fairness and transparency in financial services.

Research problem

Traditional credit scoring methods are siloed, task-specific, and struggle with generalization across different financial risk tasks. This paper investigates whether Large Language Models (LLMs) can serve as generalist tools for credit scoring, fraud detection, and claim analysis, leveraging their broad pretraining to overcome the narrow knowledge scope of expert systems. Additionally, it addresses the critical problem of algorithmic fairness, questioning whether the enhanced capabilities of LLMs come at the cost of introducing or amplifying biases against sensitive demographic groups in financial decision-making.

Core contributions

• Proposes the first open-source comprehensive framework for LLM-based credit and risk assessment.
• Curates a novel benchmark comprising 9 datasets and 14,000 samples across four financial risk tasks.
• Constructs a large-scale instruction-tuning dataset of 45,000 samples tailored for financial risk assessment.
• Develops CALM, a specialized 7B-parameter LLM fine-tuned for credit and risk assessment tasks.
• Provides a systematic evaluation of LLMs on performance, generalization, and demographic bias (gender, age, foreign status).
• Open-sources the benchmark, instruction-tuning data, and CALM model to facilitate further research.

Method

• Data collection from 9 open-source financial datasets covering credit, fraud, distress, and claims.
• Prompt engineering: Table-based prompts for anonymized/symbolic data; description-based for semantic data.
• Instruction tuning: Fine-tuning Llama2-Chat using LoRA on 45,000 samples with 7:1 train-validation split.
• Bias evaluation: Using AI Fairness 360 metrics (DI, EOD, AOD) to assess demographic bias.
• Generalization test: Evaluating models on a customs fraud dataset to test cross-domain transfer.
• Comparison against SOTA expert systems and various LLMs (open and closed source).

Results & findings

• GPT-4 performance matches or surpasses SOTA expert systems in tasks like Lending Club and Travel Insurance.
• CALM outperforms other open-source LLMs, demonstrating effective fine-tuning for financial risk tasks.
• LLMs show strong generalization capabilities, transferring knowledge from credit scoring to fraud detection and claim analysis.
• Significant bias detected in LLM predictions: models favor privileged groups (young, male, native) over unprivileged ones.
• Disparate Impact and Equal Opportunity Difference metrics reveal fairness risks, particularly in credit scoring and fraud detection.
• Open-source LLMs (e.g., Vicuna, Llama2) struggle with imbalanced data compared to GPT-4 and CALM.
• CALM achieves higher precision and F1 on the generalization customs fraud task than ChatGPT and other open-source models.

Data & experiments

• Benchmark includes 9 datasets: German, Australia, Lending Club (credit scoring); Credit Card Fraud, ccFraud (fraud); Polish, Taiwan Economic Journal (distress); PortoSeguro, Travel Insurance (claim analysis).
• Instruction tuning data derived from 6 datasets, expanded to 45,000 samples via resampling to handle class imbalance.
• Experiments compare CALM against open-source LLMs (Vicuna, Bloomz, Llama2) and closed-source models (GPT-4, ChatGPT).
• Baselines include state-of-the-art expert systems (XGBoost, Random Forest, Neural Networks) for each specific task.
• Evaluation metrics: Accuracy, F1, MCC (for imbalance), and bias metrics (Disparate Impact, Equal Opportunity Difference).
• Generalization tested on a held-out customs fraud dataset not used in training.

Limitations & caveats

• Experiments limited to 7B-parameter models, potentially underestimating the capabilities of larger LLMs.
• Lack of interpretability in the model's predictions, as no interpretable dataset was constructed.
• Bias analysis focuses on specific demographic attributes (gender, age, foreign status) which may not cover all forms of bias.
• Instruction tuning data is synthetic or derived from existing datasets, which may not fully capture real-world complexity.
• The study does not address the computational cost or latency of deploying LLMs in real-time credit scoring.
• Generalization results are based on a single customs fraud dataset, limiting the scope of transferability claims.

Curator notes

Discovery status: round1_promoted_seed_for_round2. Source links: A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges; BBT-Fin: Comprehensive Construction of Chinese Financial Domain Pre-trained Language Model, Corpus and Benchmark; BloombergGPT: A Large Language Model for Finance; Can ChatGPT Forecast Stock Price Movements? Return Predictability and Large Language Models; FinBERT : A Large Language Model for Extracting Information from Financial Text†; FinBERT: A Pretrained Language Model for Financial Communicatio