KYBench - Adverse Media Search

A compliance analyst at a mid-size fintech spends between 30 and 90 minutes on a single Know Your Business review. The bottleneck is not form processing or database lookups; those are solved. The bottleneck is adverse media research: the open-ended web investigation that determines whether a business has a history of fraud, sanctions violations, money laundering, or other regulatory red flags. It is unstructured, judgment-intensive, and scales poorly.

KYB compliance spans onboarding and ongoing monitoring, and a large portion of that work is manual human investigation. At a mid-size bank screening 10,000 businesses per month, that is 120,000 onboarding investigations per year. Factor in annual re-screening of the existing portfolio at similar volume, and the total reaches roughly 200,000 investigator-hours per year at 30 minutes per investigation. At a fully-loaded analyst rate of $80 per hour, manual investigation alone runs into the tens of millions of dollars annually. For most compliance teams, it is one of the most time-consuming and costly components of the KYB process.

We built KYBench - Adverse Media Search to answer a simple question: which AI agent configurations are actually good at this task, how reliable are they, and how do you deploy them in production? This post covers the key findings, configuration analysis, model comparisons, and deployment guidance. The full methodology will be published in an accompanying research paper.

Three results stood out. First, how you configure the agent matters more than which model you choose. Investigation strategy alone swings performance by 10 percentage points, rivalling the spread between the best and worst models on the leaderboard. Second, the agent produces different findings on the same business across identical runs. At 83% consistency, this is on par with disagreement rates between senior compliance analysts on the same cases. Third, AI agents already produce higher-quality evidence than human analysts on average (14.60 vs. 13.50 out of 20), across all four scoring dimensions including risk calibration, the area where human judgment was assumed to dominate.

KYBench - Adverse Media Search covers 47 real businesses across the full spectrum of adverse media risk, from clean small businesses to entities with active fraud convictions, OFAC sanctions, and organized crime connections. Each business was annotated by expert compliance practitioners with 25+ years of AML industry experience across 9 adverse media categories.

Two businesses from the dataset illustrate the range of outcomes: one comes back clean, one gets escalated.

We evaluated 31 agent configurations across varying models, investigation strategies, prompt framings, temperature settings, step budgets, and agent architectures. Ground truth was established by human annotators.

Measuring AI quality is straightforward when a task has a clear right answer. Adverse media research does not. Analysts aren't grading a test; they're making a judgment call. What matters is not whether the agent found many hits, but whether those hits are real, credible, and actionable.

That specificity is what makes standard benchmarks poorly suited to this domain. A raw precision/recall score treats a DOJ press release and a single-source blog post as equally strong evidence of the same risk. A coding benchmark leaderboard tells you nothing about whether a model's citations would hold up in a compliance audit. The questions a compliance team actually needs answered are about coverage (did the agent identify the risk categories that are genuinely present?), source quality (are those findings backed by primary-source government and regulatory documents, not secondary news coverage of the same events?), and precision (did the agent avoid flagging unrelated entities or low-signal noise?).

Designing metrics that reflect what compliance teams actually care about is a core part of the Taktile Labs research agenda. The benchmark below is built around that principle: every metric traces back to a question a senior compliance analyst would recognize as meaningful.

Each configuration was scored on four complementary metrics (expand each for the full methodology):

The single largest performance lever is not model selection. It is investigation strategy. Running Gemini 2.5 Pro with different strategies produces dramatically different results. Hypothesis planning (structuring the investigation upfront) raises adjusted F1 from 77.9% to 87.7%, a +9.8 pp gain. Self-critique adds a post-hoc review pass for a +7.8 pp gain at lower operational overhead. Corroboration gates and chain-of-thought requirements show smaller gains and introduce conservative bias that suppresses legitimate findings in sparse categories like money laundering.

We ran semantically equivalent prompt variants against the same model on the same businesses. Adjusted F1 ranged from 77.1% to 87.1%, a 10 percentage-point spread from prompt wording alone. That gap exceeds the performance difference between models within the same family at standard settings.

With configuration effects established, here is how variations in the underlying model affect the agent's performance using the standard configuration.

The most striking result: AI agents produce higher-quality investigation outputs than human compliance analysts. To measure this, each report, human and agent alike, was scored by an independent LLM judge on a 20-point rubric across four dimensions (0–5 each):

• Evidence completeness – did the investigator find all material adverse findings?
• Entity precision – are all flagged findings actually about the right business, not a namesake?
• Source quality – how credible and current are the cited sources?
• Risk calibration – are findings assessed at the right severity level, distinguishing allegations from convictions?

The judge has web-search access to independently verify what exists, so evidence completeness is assessed against what is actually findable, not just what the annotator chose to include. The judge scores blind to whether each output is human or agent-produced.

Human analysts averaged 13.50/20; AI agents averaged 14.60/20. These are average scores across 47 human annotations and 183 agent runs on the same dataset. Agents scored higher across all four dimensions. The largest gaps were on entity precision and risk calibration; the smallest on evidence completeness and source quality.

The threat from AI adverse media agents is miscalibrated thresholds applied to real evidence, not invented evidence. The hallucination rate across all false positives is 0%. Every false positive cited a real, accessible source; the agent found something real, and the question is whether a compliance officer would have flagged it. The dominant error type (69% of false positives) is over-flagging real but marginal evidence: the agent cited something genuine, but a human annotator would have set it aside — because the case was still at the allegation stage, the reporting was too old to be actionable, or the person involved played only a peripheral role in the business. Entity confusion accounted for 1.5% of false positives.

A human review queue built around this reality should focus on threshold adjudication, not fact-checking. Throughput and completion rate must also be tracked alongside accuracy: models that hit context window limits during deep investigations will silently truncate their analysis, producing incomplete outputs that pass accuracy checks on short cases but fail on complex ones.

Accuracy measured once is not the same as accuracy you can rely on. Consistency rate measures how often an agent produces the same risk category verdicts when run on the same business multiple times. We ran three frontier agent configurations three times each on all 47 businesses with identical inputs. The Claude Opus 4.6 agent configuration achieved 83.3% consistency rate; the GPT-5.2 Pro configuration, 71.1%; the Gemini 3.1 Pro configuration, 74.5%.

While 83% might sound low, it is on par with human performance. Given the ambiguity in the task, it is not unusual for people to disagree on the correct judgment for one in six cases. Our own annotation process produced category-level disagreements between annotators on exactly the businesses that also drove AI inconsistency. Claude Opus's 83.3% CR reflects a comparable level of judgment variance to what you'd see between two senior compliance analysts applying the same guidelines independently.

Two signals make the remaining inconsistency manageable in practice. Inconsistency concentrates in genuinely borderline cases, the same businesses that fall below Claude Opus's confidence threshold in the hybrid routing model, as the example below illustrates. And higher capability aligns with higher consistency; the most accurate models are also the most repeatable.

Full automation is the wrong target. A hybrid model (where the agent screens every case first and a human analyst reviews only the uncertain ones) reduces analyst workload by 93% while matching or exceeding human practitioner performance. The agent assigns each case a confidence score from 0 to 100. Running fully automatically, the agent still misses enough cases that a human backstop matters. Routing only the low-confidence cases (those scoring below 70) to a human changes the picture: only 6.7% of cases require a human look, yet results are on par with reviewing every case manually. Those are the genuinely hard cases where human judgment makes the difference.

KYBench - Adverse Media Search provides a repeatable evaluation framework for AI-assisted adverse media screening, and the first results point to a few clear conclusions: frontier agents now deliver human-level results across the leaderboard. Configuration choices matter more than model selection: investigation strategy alone swings performance by 10 percentage points. Every model produces inconsistent verdicts on borderline cases, but those are precisely the cases a hybrid routing model catches. Combining agents with human review of uncertain cases reduces analyst workload by 93% while raising the overall quality bar above what either achieves alone. Future work will focus on expanding the annotated dataset across jurisdictions and entity types, and on improving confidence calibration for the borderline cases where models still disagree. We will update results as new models are released.