The Scariest Number on Our AI Dashboard Turned Out to Be Nonsense
The metric was precise, confident, and wearing the wrong tiny hat.
This is Part 2 of a series on evaluating AI agents. New here? Start with Part 1: My AI Agent Looked Great — Until I Checked the Evidence. This article still stands on its own.
Here is a number that enters the room wearing a trench coat:
51.6% of Jarvis’s pull requests were closed without merging.
Oof.
If you stop there, the conclusion practically writes itself: our autonomous fixer fails more than half the time. Case closed. Spreadsheet consulted. Tiny gavel deployed.
I almost believed it too.
Then I did something deeply inconvenient to the metric.
I read the pull requests.
I interrogated the receipts
I pulled a sample of 20 closed PRs and read every comment thread. I was looking for one simple thing:
Did a human explicitly say why this was closed?
Something like:
“This is wrong.”
“We don’t need this.”
“Please stop trying to fix this with a rake.”
Anything that represented an actual judgment of the agent’s work.
I found that in roughly one or two of the twenty.
The other eighteen? Silence.
Some went stale. Some were never reviewed. Priorities moved on. A human fixed the issue manually. In one case, multiple attempts arrived almost simultaneously like delivery drones carrying the same sandwich.
That is not the same as “the agent failed.”
It may be a review bottleneck. It may be duplicate dispatch. It may be ordinary organizational entropy doing what ordinary organizational entropy does best: quietly ruining your dashboard.
The metric had wandered into the wrong department and started filing paperwork.
“Closed” is an event, not a diagnosis
Once silent closures are counted as failures, every number downstream of “percent merged” starts measuring something closer to:
How quickly does our review queue move?
…than:
Is the agent’s fix any good?
That distinction matters.
You cannot skip straight to metrics. First, you have to define — preferably in writing — what “good” means for each kind of work the agent performs.
Otherwise, you can build an exquisitely precise measurement of the wrong target.
The spreadsheet will be beautiful. The answer will still be nonsense.
So I built three drawers
For Jarvis’s most common job — fixing a detected error — I divided closed-without-merge PRs into three categories.
Because “closed” is not a diagnosis. It is just a drawer someone forgot to label.
1. Explicit deny
A human gave a concrete reason the work was rejected.
This is genuine negative signal. It counts against the agent.
2. Bottleneck closed
The PR was closed, but there is no denial on record.
This gets excluded from scoring. No reason does not mean “bad fix.” It means unlabeled outcome.
3. Calibration mismatch
The agent made a confident claim about its success that the available evidence did not support.
This third drawer did not exist in my first draft.
Then PR #40913 kicked down the door.
The PR with main-character confidence
Jarvis fixed a bug, opened a PR, and posted its self-review:
“Systematic verification passed all checks. 397 existing specs pass.”
Confident. Specific. Ready for a small celebratory pastry.
Then the PR sat there and was silently closed.
Nobody contradicted the claim. Nobody wrote, “Actually, no.” It simply died, leaving that confident self-report behind like a note from a witness who had vanished.
The code might even have been fine. That is not the most interesting problem.
The interesting problem is that the confidence was wearing a crown the evidence had not earned.
An agent that fails and knows it failed is one kind of system.
An agent that fails and reports success is a very different — and much less safe — one.
So the rubric needed two independent questions:
- Did the fix actually work?
- Was the agent’s confidence justified?
A fix can pass one axis and fail the other. Compressing both into a single pass/fail number erases the exact finding that matters most.
Then 51.6% turned into a very different story
When I applied the rubric to Jarvis’s curated history, the scary headline began to melt.
- 17 of 35 cases were bottleneck-closed. They were excluded because they were not evidence of failure.
- Of the 18 scoreable cases, 83.3% succeeded when eventual fixes after a retry were included.
- The first-clean-attempt success rate was 77.8%.
- There were exactly three hard failures: one explicit deny, one calibration mismatch, and one investigation that passed its gate using evidence it had itself labeled “inferred.”
That last distinction matters too. A fix that works immediately and a fix that lands after three attempts are both successes, but they are not the same reliability story.
Three specific, explainable failures are far more useful than “51.6% closed.”
The distance between them is roughly the distance between a map and a weather-themed shrug.
One tells you where to look next.
The other is a cloud with a percentage printed on it.
Retries have lore
The rubric also had to handle retries, because not every retry means the same thing.
One error fingerprint was fixed on the third attempt over roughly two months. That deserves degraded credit, not a gold star for “eventual success.”
But I also found four PRs for the same error opened within eight seconds of one another.
That was not three noble failures followed by triumph.
That was a dispatch bug photocopying the same homework.
If duplicate dispatches are counted as failed fixes, the metric quietly blames code quality for an infrastructure problem. Those attempts need their own bucket.
None of these categories existed until I looked at real cases and asked:
Why does this one look weird?
That question is the bug researcher’s version of opening the fridge for the third time. You may not know what you are looking for, but occasionally the answer is sitting there under a suspiciously loose metric.
The actual lesson
A useful rubric is not born fully formed in an afternoon workshop.
You build it by examining reality, finding the case that breaks your first draft, and letting that case leave a little damage behind.
The goal is not to produce the cleanest number.
The goal is to produce a number that means what everyone thinks it means.
That is harder. It is also considerably more useful.
Previously in this series: Part 1 — My AI Agent Looked Great — Until I Checked the Evidence
Next: I built an automated judge to apply this rubric at scale — and then had to prove the judge itself was not merely a new source of wrong answers with better production values.