Fine-Tuning and Explaining FinBERT for Sector-Specific Financial News: A Reproducible Workflow

In one sentence

Summary

This study addresses the opacity of financial sentiment models by introducing a fully reproducible, open-source workflow for building, explaining, and evaluating sector-specific sentiment classifiers. The authors argue that transparency and trust are critical for high-stakes financial applications, necessitating models that are not only accurate but also auditable. They construct a new manually annotated gold-standard corpus of 1,500 U.S. sector-tagged financial headlines mapped to GICS sectors and investable ETFs, filling a gap in domain-specific, high-quality training data. The workflow integrates model training, faithfulness auditing of explanations, and economic validation to ensure the pipeline is robust and aligned with market practice. The methodology benchmarks several models, including lexicon-based tools (VADER, Loughran-McDonald), interpretable baselines (Logistic Regression, Explainable Boosting Machines), and transformer-based models (FinBERT). The core experiment involves fine-tuning FinBERT on the gold-standard dataset, achieving a macro F1 of 0.707, a significant improvement over the zero-shot baseline (0.555). The authors perform a rigorous faithfulness audit using Integrated Gradients, LIME, and attention rollout, evaluating them via deletion curves and Area Over the Perturbation Curve (AOPC). They find LIME to be the most faithful explainer. Additionally, they quantify the risks of weak supervision, showing that using VADER-generated labels leads to a 21% accuracy drop and divergent explanations compared to gold-label training. Econometric tests reveal that the sentiment signal is reactive, not predictive, of next-day returns, as confirmed by Granger causality tests. However, the signal retains practical utility for sector rotation strategies. Backtesting simple long-short strategies on sector ETFs, adjusted for transaction costs, yields profitable results, with the Technology sector achieving a Sharpe ratio of 1.88. The paper concludes that while the signal does not predict future returns, it effectively captures market reactions, supporting its use in reactive trading strategies. The study emphasizes the importance of fine-tuning, faithfulness auditing, and economic validation in deploying LLMs for finance.

Research problem

Financial sentiment analysis models, particularly deep learning-based ones like FinBERT, often operate as black boxes, lacking the transparency required for high-stakes financial applications where auditability and trust are paramount. There is a need for reproducible workflows that not only achieve high accuracy on sector-specific news but also provide auditable explanations and validate the economic relevance of the sentiment signals. Furthermore, the impact of weak supervision versus high-quality manual annotation on both model performance and explanation faithfulness remains under-explored in this context.

Core contributions

• A fully reproducible, open-source workflow for sector-specific financial sentiment analysis, including code and a new 1,500-headline gold-standard dataset.
• Demonstration that fine-tuning FinBERT on a modest, high-quality, sector-tagged dataset significantly outperforms zero-shot baselines and lexicon methods.
• A quantitative faithfulness audit of XAI methods (IG, LIME, attention) for FinBERT, identifying LIME as the most faithful via deletion curves and AOPC.
• Quantification of the risks of weak supervision, showing significant drops in accuracy and explanation divergence when using VADER labels instead of gold labels.
• Economic validation showing that while sentiment is reactive, it supports profitable sector-specific long-short strategies with high Sharpe ratios.

Method

• Data collection: Aggregation of headlines from Marketaux, manual annotation for gold-standard set, and use of public datasets (FPB, FiQA).
• Model training: Fine-tuning of ProsusAI/finBERT on the gold-standard training split; training of baselines (LR, EBM, lexicons) on respective datasets.
• Explainability: Application of SHAP for linear models, and IG, LIME, and attention rollout for FinBERT.
• Faithfulness audit: Deletion-based AOPC to measure how well explanations identify influential tokens; Wilcoxon signed-rank tests for statistical comparison.
• Calibration: Reliability diagrams and Expected Calibration Error (ECE) with temperature scaling for probability calibration.
• Economic validation: Event study analysis using CAR, VAR-based Granger causality for predictive power, and backtesting with transaction costs.

Results & findings

• Fine-tuned FinBERT achieves a macro F1 of 0.707 on the gold-standard test set, a 14.9% accuracy uplift over zero-shot FinBERT (0.555).
• Lexicon methods (VADER, LM) perform poorly on the multi-sector corpus, with F1 scores around 0.43-0.48.
• LIME is identified as the most faithful explanation method for FinBERT, with an AOPC of 0.365, outperforming Integrated Gradients and attention rollout.
• Weak supervision using VADER labels results in a 21.0% drop in accuracy and a SHAP rank correlation of only 0.11 compared to gold-label training.
• Granger causality tests indicate the sentiment signal is reactive, not predictive, of next-day ETF returns.
• Backtested long-short strategies on sector ETFs are profitable, with the Technology sector achieving a Sharpe ratio of 1.88.

Data & experiments

• Newly constructed gold-standard corpus of 1,500 manually annotated U.S. financial headlines, stratified across 10 GICS sectors.
• Public benchmarks: Financial PhraseBank (2,264 samples) and FiQA-Sentiment (498 headlines).
• Models evaluated: VADER, Loughran-McDonald, Logistic Regression, Explainable Boosting Machine, FinBERT (zero-shot and fine-tuned), and a FinBERT+LR hybrid.
• Explainability evaluation using Integrated Gradients, LIME, and attention rollout, assessed via deletion curves and AOPC.
• Economic validation via event studies (CAR), Granger causality tests, and backtesting long-only and long-short strategies on SPDR sector ETFs with 5 bps transaction costs.

Limitations & caveats

• The gold-standard dataset is relatively small (1,500 headlines), which may limit generalizability.
• The sentiment signal is found to be reactive, not predictive, limiting its use for forward-looking prediction models.
• Backtests are simple and may not capture complex market dynamics or slippage beyond 5 bps.
• The study focuses on headlines, which may lack the depth of full articles or filings.
• Weak supervision analysis is limited to VADER, which may not represent all weak labeling strategies.

Curator notes

Discovery status: specific_domain_focus_promoted. Source links: FinBERT ETF financial returns SPY