Advancing Anomaly Detection: Non-Semantic Financial Data Encoding With Large Language Models

In one sentence

Summary

This paper addresses the challenge of detecting anomalies in financial general ledger data, specifically focusing on the difficulties posed by feature heterogeneity and sparsity in non-semantic categorical attributes. Traditional machine learning approaches often struggle with the varying lengths and complexities of journal entries, leading to high-dimensional sparse vectors that are inefficient for classification. The authors propose a novel hybrid approach that leverages pre-trained Sentence-BERT (SBERT) large language models to encode these non-linguistic financial features into dense, fixed-size vector representations. This method aims to standardize feature variability and retain information more compactly than conventional vectorization techniques, thereby enhancing the input quality for downstream anomaly detection models. The experimental design utilizes a real-world, anonymized general ledger dataset containing 32,100 transaction-level data points with 148 artificially inserted anomalies representing eight error types. The researchers tested three SBERT models (all-mpnet-base-v2, all-distilroberta-v1, and all-MiniLM-L6-v2) to generate embeddings for concatenated categorical transaction features. These embeddings were then fed into five optimized machine learning classifiers: Logistic Regression, Random Forest, XGBoost, Support Vector Machines, and Neural Networks. Performance was evaluated using macro recall average to handle class imbalance, with hyperparameters tuned via Bayesian optimization. The study also employed Principal Component Analysis to assess the dimensionality reduction efficiency and information retention of the LLM embeddings versus traditional one-hot encoding. Results indicate that SBERT embeddings offer superior dimensionality reduction, requiring significantly fewer components to preserve variance compared to one-hot encoding. In downstream tasks, the LLM-enhanced models outperformed baselines in selected settings, with Logistic Regression and Neural Networks showing the most consistent improvements in anomaly detection accuracy. However, Random Forest and SVM models experienced performance declines when using LLM embeddings, highlighting the importance of model-embedding compatibility. The findings confirm that LLMs can effectively encode non-semantic financial data, tackling feature sparsity and enhancing audit reliability. Limitations include the synthetic nature of the anomalies, which may limit generalizability to real-world unlabeled fraud, and the linear constraints of PCA used for analysis.

Research problem

The paper addresses the difficulty of detecting anomalies in general ledger data due to feature heterogeneity and sparsity in non-semantic categorical attributes. Traditional machine learning methods struggle with the varying lengths and complexities of journal entries, resulting in inefficient high-dimensional sparse vectors. The research aims to improve anomaly detection accuracy by encoding these non-linguistic features using Large Language Model embeddings.

Core contributions

• Proposes a novel method for encoding non-semantic categorical financial data using Sentence-BERT embeddings.
• Demonstrates superior dimensionality reduction and information retention of LLM embeddings compared to one-hot encoding.
• Evaluates the impact of LLM embeddings on five different machine learning classifiers for anomaly detection.
• Provides empirical evidence that LLMs can effectively process non-linguistic data for financial auditing tasks.
• Highlights the importance of model-embedding compatibility, showing mixed results across different classifier types.

Method

• Concatenates categorical transaction features into text strings for each journal entry.
• Uses Sentence-BERT models to generate fixed-size dense vector embeddings for each entry.
• Normalizes embeddings to zero mean and unit variance for consistent ML input.
• Trains five ML classifiers with hyperparameters optimized via Bayesian optimization.
• Evaluates performance using macro recall average to balance sensitivity and specificity.
• Applies PCA to assess the information retention and dimensionality of the embeddings.
• Uses an 80/20 stratified split for training and testing to maintain class distribution.

Results & findings

• SBERT embeddings require significantly fewer PCA components to preserve 99% variance compared to one-hot encoding.
• Logistic Regression with LLM embeddings showed improved recall scores over traditional encoding.
• Neural Networks also benefited from LLM embeddings, showing consistent performance gains.
• Random Forest and SVM models experienced performance declines when using LLM embeddings.
• The all-mpnet-base-v2 model demonstrated the best stability and performance across various classifiers.
• The study confirms that LLMs can standardize feature variability in non-semantic financial data.
• Synthetic anomalies were used, which may limit the generalizability to real-world fraud patterns.

Data & experiments

• Uses an anonymized real-world general ledger dataset with 32,100 transaction-level data points.
• Dataset includes 148 artificially inserted anomalies representing eight distinct error types.
• Tests three Sentence-BERT models: all-mpnet-base-v2, all-distilroberta-v1, and all-MiniLM-L6-v2.
• Evaluates five ML classifiers: Logistic Regression, Random Forest, XGBoost, SVM, and Neural Networks.
• Employs macro recall average as the primary metric to handle class imbalance.
• Uses PCA to analyze the dimensionality reduction efficiency of the generated embeddings.

Limitations & caveats

• The anomalies in the dataset are artificially inserted, which may not reflect real-world fraud complexity.
• The study focuses on categorical features, limiting its applicability to numerical analysis.
• PCA is a linear method, which may not fully capture non-linear relationships in the embeddings.
• The performance of LLM embeddings varies significantly across different ML classifiers.
• The dataset is anonymized and not publicly available, limiting reproducibility.
• The study does not explore the impact of prompt engineering on the embedding quality.

Curator notes

Discovery status: regtech_compliance_focus_promoted. Source links: Responsible Innovation: A Strategic Framework for Financial LLM Integration