Enhancing the Efficiency and Accuracy of Underlying Asset Reviews in Structured Finance: The Application of Multi-agent Framework

In one sentence

Summary

The paper addresses the significant due diligence challenges in structured finance, specifically the manual and error-prone process of reviewing underlying assets for Asset-Backed Securities (ABS). It positions AI-driven automation as a solution to enhance efficiency and accuracy in verifying the completeness and compliance of financial documents, focusing on auto ABS as a primary use case. The research aims to bridge the gap in applying multi-agent LLM systems to audit consulting services by proposing a framework that automates document processing, information extraction, and error detection. The methodology involves a multi-agent system where one agent generates synthetic bank and loan statement data, another extracts key information from PDFs, and a third evaluates the extraction accuracy against ground truth labels. The study compares the performance of closed-source models (GPT-4) against open-source models (LLAMA2, LLAMA3, DBRX) in a zero-shot setting. Experiments were conducted on 49 pairs of simulated bank and loan statements, measuring extraction accuracy for fields like names, balances, and addresses. The authors also analyzed the operational costs associated with each model to assess economic viability. Results indicate that GPT-4 achieved 99% accuracy, while LLAMA3 reached 93-99% accuracy depending on the document type. Notably, a dual-agent system that cross-verifies information within the same document achieved 100% accuracy for both model types, albeit at higher computational costs. The paper highlights that while open-source models are significantly cheaper (e.g., $0.003 vs $0.05 per document), they require careful prompt engineering to handle unstructured data like multi-line addresses. The findings suggest that AI agents can streamline due diligence, reduce human error, and offer scalable solutions for financial document analysis, though regulatory and privacy compliance remains a critical consideration for deployment.

Research problem

Structured finance due diligence, particularly the review of underlying assets for ABS, is labor-intensive, costly, and prone to human error due to the volume and complexity of documents. The paper investigates how multi-agent LLM frameworks can automate the verification of information between loan applications and bank statements to improve efficiency and accuracy in this specific audit context.

Core contributions

• Proposes a multi-agent LLM framework for automating the cross-verification of underlying assets in structured finance.
• Develops a synthetic data generation pipeline for bank and loan statements to enable rigorous testing of AI extraction agents.
• Provides an empirical comparison of closed-source (GPT-4) and open-source (LLAMA, DBRX) models on financial document extraction tasks.
• Demonstrates that a dual-agent cross-verification system can achieve 100% accuracy in information matching.
• Analyzes the cost-efficiency trade-offs between high-accuracy closed-source models and cost-effective open-source alternatives.

Method

• Data Generation: LLM agents create synthetic personal information and corresponding bank/loan documents.
• Document Processing: A reader agent converts PDF documents into text for extraction.
• Information Extraction: LLM-based agents extract specific fields (name, balance, address) from the text.
• Accuracy Assessment: An evaluation agent compares extracted data against ground truth labels to calculate accuracy.
• Dual-Agent Verification: A second agent cross-checks the extracted information within the same document to correct errors.
• Cost Analysis: Token usage and pricing data are used to estimate the operational cost of each model.

Results & findings

• GPT-4 achieved 99% overall accuracy in extracting data from bank and loan statements.
• LLAMA3 achieved 93% accuracy on bank statements and 99% on loan statements.
• DBRX and LLAMA2 showed lower accuracy, particularly in extracting multi-line addresses from bank statements.
• A dual-agent system improved accuracy to 100% for both GPT-4 and LLAMA3.
• Processing cost for GPT-4 was approximately $0.05 per document, while LLAMA3 was around $0.003.
• Open-source models offer a significant cost advantage while maintaining high accuracy for structured fields.
• Manual review is more prone to errors and less efficient than AI-assisted workflows.

Data & experiments

• Generated 49 pairs of synthetic bank statements and loan applications using LLM-based data generation agents.
• Evaluated extraction accuracy of GPT-4, LLAMA2, LLAMA3, and DBRX on key fields including names, balances, and addresses.
• Compared single-agent zero-shot performance against a dual-agent cross-verification setup.
• Calculated operational costs per document for different models based on token usage and pricing.
• Used GPT-4 as an evaluation agent to assess the accuracy of extracted data against ground truth labels.

Limitations & caveats

• The study is limited to synthetic data, which may not fully capture the complexity of real-world documents.
• The ability of models to handle less structured or diverse document formats is not extensively tested.
• Regulatory and privacy compliance issues are acknowledged but not addressed in the implementation.
• The dual-agent system increases operational costs, which may impact scalability.
• The evaluation is based on a small sample size (49 pairs) of documents.

Curator notes

Discovery status: regtech_compliance_focus_promoted. Source links: Agentic AI Systems Applied to tasks in Financial Services: Modeling and model risk management crews