Agentic AI Systems Applied to tasks in Financial Services: Modeling and model risk management crews

In one sentence

Summary

The paper addresses the challenge of automating complex, regulated financial modeling and model risk management (MRM) processes by leveraging multi-agent systems. It positions agentic AI as a solution to streamline workflows that traditionally require significant human effort, focusing on the intersection of large language models (LLMs) and rigorous financial compliance standards. The authors argue that while LLMs offer autonomy, their 'black box' nature and potential for hallucination necessitate structured collaboration and human oversight in high-stakes financial environments. The methodology involves building two distinct agentic crews using the CrewAI framework: a Modeling Crew and an MRM Crew. The Modeling Crew includes agents for data extraction, exploratory data analysis (EDA), feature engineering, hyperparameter tuning, training, evaluation, and documentation. The MRM Crew features agents for documentation compliance, model replication, conceptual soundness, and outcome analysis. A human-in-the-loop (HITL) module acts as an orchestrator, providing feedback and correcting errors. The system is tested on three datasets: credit card fraud detection, credit card approval, and portfolio credit risk modeling, utilizing LLMs such as Llama 3, Deepseek-R1, and GPT-3.5 Turbo. Results indicate that the agentic crews effectively perform end-to-end modeling and validation tasks. The MRM agents successfully replicated models, verified conceptual soundness through feature importance analysis, and conducted stress testing with shifted inputs. The system demonstrated high accuracy in compliance checks and model replication, with metrics like AUC and F1-score closely matching original models. However, the paper notes a 1-10% error rate in code execution, highlighting the necessity of the HITL component. The study concludes that agentic systems can significantly enhance efficiency in financial modeling and MRM, provided that human oversight is integrated to manage safety, bias, and operational risks.

Research problem

The paper addresses the need for automating complex, regulated financial modeling and model risk management (MRM) workflows. It seeks to determine if multi-agent LLM systems can effectively collaborate to perform these tasks while maintaining the rigor and compliance required in financial services. The core problem is balancing the autonomy of AI agents with the safety, transparency, and accuracy demands of financial regulation.

Core contributions

• Design of two specialized agentic crews (Modeling and MRM) using the CrewAI framework.
• Implementation of a human-in-the-loop (HITL) orchestration module to mitigate hallucination and ensure compliance.
• Application of agentic systems to three distinct financial modeling tasks: fraud detection, credit approval, and portfolio risk.
• Demonstration of automated model replication, conceptual soundness checks, and stress testing by MRM agents.
• Analysis of agent collaboration strategies, memory management, and guardrail mechanisms in a financial context.

Method

• Utilization of CrewAI framework for agent role-playing, memory, and tool integration.
• Hierarchical collaboration structure with a human orchestrator and judge agents.
• Implementation of guardrails and low temperature settings to reduce hallucination.
• Memory stream management for inter-agent knowledge transfer.
• Automated code execution for data analysis, model training, and stress testing.
• Documentation generation and compliance checking via specialized agents.
• Stress testing using simulated adversarial and outlier inputs.

Results & findings

• The Modeling Crew successfully performed end-to-end pipeline development including EDA and feature engineering.
• The MRM Crew achieved high accuracy in checking documentation compliance against organizational guidelines.
• Model replication by MRM agents yielded evaluation metrics closely matching the original modeling team's results.
• Feature importance analysis confirmed conceptual soundness, with loan-related features showing highest importance.
• Stress testing with shifted inputs revealed performance drops, highlighting model stability concerns.
• Human intervention was required for 1-10% of code execution errors, validating the HITL approach.
• The system demonstrated the ability to generate comprehensive documentation automatically.

Data & experiments

• Credit card fraud detection dataset.
• Credit card approval dataset.
• Portfolio credit risk modeling dataset.
• Experiments using Llama 3, Deepseek-R1, and GPT-3.5 Turbo as agent backbones.
• Evaluation based on model replication accuracy, documentation compliance, and stress test performance.
• Metrics include accuracy, F1-score, precision, recall, and AUC.

Limitations & caveats

• The system relies on a human-in-the-loop, which may limit full automation and scalability.
• Code execution errors occur in 1-10% of cases, requiring human correction.
• The study uses synthetic or public datasets, which may not fully represent real-world financial data complexity.
• The performance of the system is dependent on the underlying LLM capabilities (e.g., Llama 3, GPT-3.5).
• The paper does not extensively discuss the computational costs or latency of running multiple agents.
• Potential biases in the training data of the LLMs are not deeply analyzed.

Curator notes

Discovery status: round3_promoted. Source links: FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Making; When AI Meets Finance (StockAgent): Large Language Model-based Stock Trading in Simulated Real-world Environments