Most evaluation work on financial crime agents measures the wrong thing, because the hard part is not the model. It is the labels, and almost nobody costs that honestly.
An alert was correct or it was not, and the only record of which is a judgement an analyst made under time pressure, with the information available that day. That judgement is the label. Every accuracy number downstream inherits its biases, its backlog pressure and its disagreements. So the useful questions are not which model scores best. They are: where did the labels come from, what does each class of error actually cost, does the backtest let the model see the future, and which decisions is the model not permitted to make at all.
Those four are below. The last one is the one that decides whether the system is defensible to an examiner, and it is the one most evaluation writeups skip entirely.
This is method, from someone who builds agents and has used the screening tools, not a report on a compliance function.
Labels come from closed cases, SAR outcomes and analyst dispositions, and each is biased in a different direction. Closed cases are filtered by whatever the team had capacity to open. SAR outcomes are filtered by what survived a second review. Analyst dispositions carry the pressure of the queue on the day.
"No SAR filed" is not "not suspicious." It means nobody escalated it, which may be a judgement, a threshold or a Friday afternoon. Treating that as a clean negative is the most common way an AML training set gets quietly poisoned, and it inflates every precision figure computed against it.
At a real base rate the classes are wildly imbalanced, which makes accuracy useless as a headline: a model that alerts on nothing scores extremely well. Report precision and recall with raw counts, always. A rate with no denominator is not a finding.
Where two analysts disagree, somebody adjudicates. That disagreement rate is itself worth measuring and almost never is. If two experienced people disagree on a fifth of borderline cases, no model is going to be evaluated more precisely than that, and knowing the number tells you where your ceiling is before you spend a quarter chasing it.
One trap worth naming because it is easy to walk into: a truth set assembled from what you already believe will confirm what you already believe. I have made this mistake in my own evaluation work. A truth set built from the already-known case reported precision at 8.3 percent when the real figure, once the items were actually read, was 91.7 percent. The instrument was measuring the belief that produced it.
Definitions are not the interesting part. The asymmetry is.
A false positive costs an analyst's time, and at volume it costs something worse: the team stops trusting the system and starts working around it. A false negative is missed suspicious activity, which is a regulatory and reputational event rather than a point of a metric.
Those two are not commensurable, so a single F1 number hides the only decision that matters. F1 assumes the errors trade off evenly. They do not, and averaging them produces a figure that looks rigorous and answers nothing.
The threshold is where this becomes concrete, and it is not a model hyperparameter. It is a business decision about alert capacity: how many alerts can this team work in a day without the quality of dispositions falling. Set it against capacity, name the person who owns it, and record when it moves and why. A threshold that drifts without a decision behind it makes every historical comparison meaningless.
Split by time, never at random. A random split lets the model see the future, and the resulting numbers are fiction that will not survive contact with production.
Three leaks to check for by name:
Entity leakage. The same customer either side of the split. The model learns the customer rather than the behaviour and looks far better than it is.
Label leakage. Fields populated only after a case closed: disposition codes, reviewer notes, case status. Enumerate them explicitly and exclude them, because they are the answer written on the back of the card.
Population drift. A model backtested on last year's mix is measured against a world that no longer exists. Typologies move, products move, and the customer base moves.
Underneath all three sits determinism. The same input at the same version has to produce the same decision, or the backtest is measuring noise. If a run is not reproducible, its numbers are not evidence of anything.
This is the part that makes a system defensible, and it is a design decision taken before any evaluation happens.
Anything that touches money or a regulatory outcome is decided by a rule, not a model call. The model drafts language, triages, summarises and ranks. It does not decide. Every run records which rule fired, on what input, at what version, at what time.
The reason is not caution for its own sake. An examiner does not ask whether the model is accurate. The request that has to be answerable is: show me every alert this rule fired on last quarter, and show me the ones it did not. A model with a confidence score cannot answer that. A versioned rule with a stored record of every evaluation answers it in a query, and the difference between those two positions is the difference between a system you can defend and one you cannot.
That constraint is unfashionable, and I would argue it is where most agent architectures in this space are going to run into trouble. A model call is genuinely right where the output is language a human approves before it has any effect: a narrative draft, an evidence summary, a suggested next check. The test is whether a wrong answer is caught by a person before it does anything, and if the answer is no, that decision belongs in a rule.
I build agents where deterministic rules decide anything touching money and the model only drafts language, with the evaluation and audit trail around them. PayPilot is a live one, at paypilot.fly.dev, and section 2 is drawn from its evaluation work. There is a working AML triage demonstration at aml-triage-demo.vercel.app.
I came to this from six months at a crypto to fiat exchange, screening wallets with AMLBot, before I built any tooling for it.
2026-08-15: the closing paragraph described this author as having run crypto to fiat exchanges. That was wider than the fact. It was six months at one exchange, screening wallets. Corrected rather than left, because a document about claims drifting is a poor place for a claim that drifted.