FinNLI: Novel Dataset for Multi-Genre Financial Natural Language Inference Benchmarking

In one sentence

Summary

The paper addresses the lack of dedicated benchmarks for Financial Natural Language Inference (FinNLI), a critical task for risk assessment and automated reporting. While general-domain NLI datasets exist, they fail to capture the nuances of financial texts, such as complex terminology and implicit causal relationships. The authors introduce FinNLI to evaluate how well language models can infer logical relationships (entailment, neutral, contradiction) within diverse financial genres, filling a gap in evaluating domain-specific reasoning capabilities beyond simple sentiment or extraction tasks. The dataset construction involves sampling premises from SEC filings, annual reports, and earnings call transcripts. Hypotheses are synthetically generated using GPT-4 and Llama 3.1 70B with varied roles and writing styles, followed by Z-filtering to remove spurious correlations. A high-quality test set of 3,304 instances is annotated by finance experts. The authors evaluate a range of models, including zero-shot general-domain NLI models, fine-tuned Pre-trained Language Models (PLMs) like RoBERTa and FiLM, and various LLMs (Llama, Phi, FinMA) using different prompting strategies like Chain-of-Thought and few-shot learning. Results show that domain shift significantly degrades general-domain NLI performance, with the best Macro F1 score being 78.62% for Llama 3.1 70B. Surprisingly, instruction-tuned financial LLMs (FinMA) performed poorly, suggesting limited generalizability of domain-specific tuning. The study highlights that smaller models like Phi-3.5 can compete with larger ones, and that Chain-of-Thought prompting benefits smaller LLMs but not the largest ones. The dataset exposes significant weaknesses in current LLMs for financial reasoning, indicating substantial room for improvement in handling complex financial logic and avoiding spurious correlations.

Research problem

The paper identifies a critical gap in Financial NLP: the absence of a robust benchmark for Natural Language Inference (NLI) in the financial domain. Existing general-domain NLI datasets do not adequately test models on the specific challenges of financial texts, such as implicit risk signals, complex terminology, and multi-step reasoning. The problem is to create a high-quality, diverse dataset that minimizes spurious correlations and accurately reflects the difficulty of financial reasoning, thereby providing a reliable metric for evaluating the state-of-the-art in financial LLMs.

Core contributions

• Introduction of FinNLI, a new benchmark dataset with 21,304 premise-hypothesis pairs across SEC filings, annual reports, and earnings calls.
• Development of a rigorous data generation pipeline that uses LLMs for hypothesis generation, Z-filtering to mitigate spurious correlations, and expert annotation for high-quality labels.
• Comprehensive evaluation of diverse models, including general-domain NLI models, fine-tuned PLMs, and instruction-tuned financial LLMs, revealing significant performance gaps.
• Analysis of the negative impact of domain-specific instruction tuning on general financial reasoning tasks, challenging the assumption that financial LLMs are universally superior.
• Demonstration that smaller models (e.g., Phi-3.5) can outperform larger instruction-tuned financial models, highlighting the importance of data quality and model architecture over sheer size or domain tuning.

Method

• Premise Sampling: Extracted from SEC filings, annual reports, and earnings calls, filtered for syntactic completeness.
• Hypothesis Generation: Used GPT-4 and Llama 3.1 70B with role-playing and style conditioning to generate entailment, neutral, and contradiction hypotheses.
• Z-Filtering: Applied Z-filtering algorithm to remove samples with high spurious correlations between task-independent features and labels.
• Expert Curation: Iterative refinement of generation prompts based on feedback from a general-domain NLI model and human experts.
• Human Annotation: Three finance experts annotated the test set, with majority voting and confidence levels.
• Model Evaluation: Tested zero-shot, fine-tuned, and instruction-tuned models using various prompting strategies (ZS, FS, CoT).
• Error Analysis: Investigated misclassifications by Llama 3.1 70B, focusing on reasoning errors and hallucinations.

Results & findings

• FinNLI test set of 3,304 instances annotated by finance experts ensures high quality and domain relevance.
• General-domain NLI models suffer significant performance drops on financial text due to domain shift.
• Best Macro F1 score for pre-trained PLMs (RoBERTa-Large) is 74.57%.
• Best Macro F1 score for LLMs (Llama 3.1 70B) is 78.62%, indicating remaining difficulty.
• Instruction-tuned financial LLMs (FinMA) perform poorly (F1 ~23-29%), underperforming general-domain baselines.
• Smaller model Phi-3.5 (3.8B) achieves 74.74% F1, outperforming larger Llama 3.1 8B (71.79%).
• Chain-of-Thought prompting improves performance for small-to-mid-sized LLMs but not for Llama 3.1 70B.
• GPT-4-generated instances are more challenging for zero-shot models than Llama-generated ones.

Data & experiments

• Dataset: FinNLI with 21,304 pairs (16,200 train, 1,800 dev, 3,304 test), sourced from SEC filings, annual reports, and earnings call transcripts.
• Annotation: Test set annotated by finance experts with high inter-annotator agreement (Fleiss-κ = 88.31%).
• Models Evaluated: Zero-shot general NLI (BART, DeBERTa), fine-tuned PLMs (RoBERTa, FiLM), and LLMs (Llama 2/3, Phi, FinMA).
• Prompting Strategies: Zero-shot, Zero-shot with Annotation Guidelines, Few-shot, and Chain-of-Thought (CoT).
• Metrics: Average Macro F1 score to handle class imbalance in the test set.
• Analysis: Error analysis on misclassified instances, comparison of GPT-4 vs. Llama-generated test sets, and impact of stylistic variations.

Limitations & caveats

• Dataset access is restricted; users must contact authors for data.
• Synthetic hypothesis generation may introduce biases or artifacts not present in human-written text.
• Instruction-tuned financial LLMs' poor performance may be due to suboptimal training rather than inherent limitations.
• The test set, while expert-annotated, may still contain ambiguous cases, as indicated by low-confidence annotations.
• Evaluation is limited to NLI tasks; results may not generalize to other financial NLP tasks like summarization or QA.
• The study focuses on English-language financial texts; cross-lingual generalizability is not addressed.

Curator notes

Discovery status: round2_promoted. Source links: Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? A Study on Several Typical Tasks; FinBERT: Financial Sentiment Analysis with Pre-trained Language Models; InvestLM: A Large Language Model for Investment using Financial Domain Instruction Tuning; Time Travel is Cheating: Going Live with DeepFund for Real-Time Fund Investment Benchmarking; Will LLMs be Professional at Fund Investment? DeepFund: A Live Arena Perspective