My AI Agent Looked Great—Until I Checked the Evidence

By Deepan Kumar

I’m a Principal Engineer. There is no architect-shaped rung above me on the ladder, which means I’ve had a front-row seat to AI eating the parts of the job that used to feel like a moat: system design, tradeoffs, and the sacred ritual of asking, “Is this actually the right approach?”

The moat now has autocomplete.

So I picked a fight instead of waiting for the moat to evaporate while I stood there holding a ceremonial bucket.

I decided to get good at a question that turns out to be surprisingly difficult:

How do you know an AI agent is actually good at its job?

Not “did it produce output?” Not “did the demo work?” And definitely not “the team has a good feeling about it.”

I mean good: reliable, measurable, and defensible in a room full of skeptics who have already asked for the denominator.

The agent in the terminal

I had a convenient lab rat with a terminal: Jarvis, an autonomous system I built that watches production logs, detects errors, spins up an isolated environment, writes a fix, and opens a pull request. A human stays out of the loop until review.

Error detection → autonomous fix → pull request → learning loop.

It isn’t a toy. It has been running against a real Rails monolith for months.

When I asked myself whether it was good, though, I realized I had no real answer. I had a feeling. A rough sense that it worked “most of the time” — the scientific equivalent of looking out the window and declaring, “Probably no frogs.” The kind of confidence you can carry around comfortably right up until somebody asks for the denominator.

I did not have anything I could show someone.

So I audited it properly: read the code, counted the real numbers, and checked every claim against what actually happened on disk — not what the documentation politely implied happened on disk.

The first hour produced a number that made me stop scrolling and briefly reconsider all my life choices involving the word “gate”:

One of Jarvis’s safety gates had run to completion on 1 out of 107 real fix attempts. That’s 0.9%.

This was not a case of a weak gate. It was real, well-written code that checked whether a fix had enough investigation behind it before allowing a pull request to go out. The gate had done its homework. Nobody had invited it to class.

The gate worked exactly as designed.

Nothing in the pipeline called it.

It was sitting there, correct and unreachable, while the system it was supposed to protect shipped pull requests without ever touching it. A safety gate on a shelf. A fire extinguisher in a painting of a fire station. A very responsible-looking doormat with no house behind it.

That is not a bug you find by reading documentation. Documentation tells you the gate exists. Docs are the agent’s yearbook; they contain flattering photos and no attendance record. You find the truth by asking, “When is the last time this actually ran?” and then counting.

The actual problem I was trying to solve

That number reframed the whole project.

I wasn’t going to write a cheerful tour of “cool things Jarvis does.” I was going to document what it takes to know whether an autonomous agent is doing its job — honestly, with real numbers, including the numbers that make the system look less impressive.

Because that skill, I realized, was what I was actually trying to build.

Every company I talk to is building agents. Almost none can tell you, with a straight face, real evidence, and only a normal amount of blinking:

how well the agent performs;
where it fails;
whether last week’s prompt change made things better or worse; and
whether a safety check is protecting anything or merely existing in a file somewhere.

“It seems to work” is the actual state of the art at a lot of places. That isn’t an insult. Building the agent is exciting. Evaluating it rigorously is the unglamorous part nobody puts on the launch slide.

Which means the person who can walk into a room and say, “Here is the real success rate, here is exactly how we measured it, here is the failure mode we found, and here is the regression test that will catch it next time” is doing something genuinely rare.

Not flashy. Rare.

What this series covers

This series follows one real evaluation of Jarvis, using production history rather than a toy benchmark or a suspiciously well-behaved demo.

First, I had to define “good” before measuring anything — and find the bug in our failure metric that made Jarvis look far worse than it actually was.

Then I built an automated judge and refused to trust it until it proved that it agreed with human judgment often enough to deserve a chair at the table.

After that came the safety gate running 0.9% of the time, and the strange way verification theater accumulates without anyone ever scheduling a meeting called “Let’s Build Some Verification Theater.”

Finally, I deliberately broke the evaluation system to see whether the safety net could detect its own injury.

One caveat will follow the series like a small, responsible ghost: none of this makes Jarvis write better code today. This is the “figure out exactly what’s true” half of the work, not the “now go fix it” half.

But you cannot do the second half without the first. And almost nobody does the first half rigorously.

That gap is the opportunity.

Next: the rubric problem, and the production metric that was quietly counting reviewer bottlenecks as AI failures.