
Over the past few months, our intern Pelle van der Zaag conducted an in-depth research study into how AI Agents and Machine Learning can transform Anti-Money Laundering processes in the financial sector. Tested across a dataset of 9 million transaction records, the study compares four distinct AML detection approaches, from traditional rule-based systems to a hybrid model combining Machine Learning and AI Agents, and measures their performance across the metrics that matter most to compliance teams: detection recall, precision, and false positive rate.
What We Investigated
AML compliance remains one of the most resource-intensive challenges in financial services. Rule-based detection systems, the current industry standard, are static by design. They rely on predefined thresholds and logic that cannot adapt to evolving laundering patterns, and they are well known for generating high volumes of false positives that place a significant burden on compliance teams.
Our research set out to test whether a combination of Machine Learning and AI Agents could address these shortcomings in a measurable, operationally meaningful way.
The Four Approaches We Tested
The study evaluated the following detection configurations against a consistent dataset of 9 million records:
Our Key Findings
The hybrid model delivered the strongest results across all performance dimensions.
Combining CatBoost with an AI Agent achieved a detection rate of 93.7%, catching the large majority of money laundering cases in the dataset. AI agents were shown to increase detection recall by up to 8.7 percentage points compared to machine learning models operating alone. That is a significant improvement, and one we believe has real-world implications for how financial institutions should think about their AML infrastructure.
The reason the hybrid approach outperforms is that each component addresses a different dimension of the problem. CatBoost identifies statistical anomalies within structured transaction data with high efficiency. The AI Agent then applies a layer of contextual reasoning, evaluating whether a flagged transaction is genuinely suspicious based on broader behavioural context. Together, they catch more, more accurately.
The paper also includes a detailed breakdown of false positive trade-offs across all four approaches, an important dimension that raw detection figures alone do not capture.
+8.7pp
Recall gain for the Rule-Based model. Adding an AI agent layer pushed detection from 65.3% to 74.0% — without replacing the underlying system.
93.7%
Peak performance via the ML-Based AI Agent. A 7.8 percentage point improvement over the standalone CatBoost model, detecting 9 in 10 laundering cases.
↓FN
Dramatic reduction in False Negatives. Agents identified complex patterns typically overlooked by base models — the cases most likely to cause real harm.
Scalable
Enhanced scalability and intelligence. AI agents interpret policies and analyze transaction context faster than manual review teams, enabling institutions to grow without proportional compliance headcount.
What This Means in Practice
Our conclusion is that AI agents will significantly change AML processes at financial institutions of any size. The hybrid approach we tested does not simply improve detection metrics. It changes how human analysts engage with the compliance process. With a more precise and contextually aware system handling initial assessment, investigative capacity can be directed toward cases that genuinely warrant expert review.
We are convinced this is the direction the industry is heading, and this research gives a clear, data-backed picture of what that shift looks like in practice.
A Note of Thanks
This research was made possible in part through a collaboration facilitated by the Malaysian Dutch Business Council (MDBC). We are grateful for the connection they made, which contributed directly to the quality of this study.
Our sincere thanks also go to Pelle van der Zaag, whose dedication and rigor throughout this project resulted in a body of work we are genuinely proud of.

Download the Research Paper
Free access. We may occasionally share relevant research from Cybiant — unsubscribe any time.
Interested in how these findings apply to your organisation? Get in touch with Cybiant.
Get Social

