Building the AI Agent Was the Easy Part

A long look at evals, persona simulations, tool tests, model choice, and the API bill nobody budgets for when building an AI agent.

Same prompt, same tools, same data. One model returned the correct list. The other invented students who don’t exist.

That sentence has been living in my head for weeks, so let me tell you how it got there, and what we do differently now.

I’ve been building agentic workflows for a while. The recipe is familiar by now: an LLM in the middle, a set of tools around it, a database underneath, sometimes an MCP server, business logic to keep it on task, security rules to keep it in its lane. Wiring that up used to feel like the whole job. Most frameworks will get you to a working demo before your coffee goes cold, and the demo will look great.

The wahala starts with a much less glamorous question: how do I know this thing actually behaves? Not on the happy path I rehearsed. On the thousandth conversation, with a user I’ve never met, after a prompt tweak I made on a Friday.

The ghost students

This is the incident that changed how I work.

We asked an agent to pull the list of students who were owing. Simple task on paper: hit the right tool, apply the right filter, return the names. We ran it on two models with an identical setup, same prompt, same tenant, same data. The only variable was the model.

Model A returned a clean, accurate list.

Model B returned a list too. Real students, and a handful who do not exist anywhere in the database.

So we asked it why. It explained. We pushed back. It explained again, calmly and in detail, with the confidence of a danfo conductor swearing there’s still space. It only dropped the ghost students when we explicitly said no, they are not real, remove them.

Two things bothered me more than the fabrication itself.

First, the wrong answer and the right answer looked identical from the outside. Same format, same tone, same certainty. Anything downstream, a report, a reminder email, a human skimming the output, would have trusted both equally.

Second, not a single line of code changed between those two runs. A test suite that only checks whether the tool works would never catch this, because the tool worked. The model lied about what the tool returned.

Why agents break normal testing

Traditional software is deterministic. Same input, same output, every time, and if it isn’t, that’s the bug. Agents are probabilistic by design. Sampling means the same task on two runs can take two different paths, call tools in a different order, or phrase the same answer three different ways. That alone breaks the “assert equals” habit most of us grew up with.

Then errors compound. An agent that is 95% reliable per step is nowhere near 95% reliable across a five-step workflow. Multiply it out and you’re at roughly 77%, because every step inherits the mistakes of the one before it.

Then there are the model quirks, the ones you only meet in production or in tests that look like production:

• Verbosity bias. Longer output starts looking like better output. This hits twice: the model pads its answers, and if you’re using an LLM as a judge, the judge rewards the padding.
• Position bias. Give the model a list of options and it leans toward whichever one came first. Ask it to compare two answers and the order you present them in can flip the verdict.
• Prompt sensitivity. A one-line tweak to the system prompt, added to fix one case, changes which tool gets called in three others.
• Wrong tool selection. The prompt says fetch, the model decides update felt right today.
• Confident fabrication. See ghost students above.
• Folding under pressure. The mirror image of the ghost students. Some models abandon a correct answer the moment a user pushes back, which is its own kind of dangerous.

You cannot unit test your way out of this. You need layers.

Layer one: tool and integration tests

The deterministic parts still deserve deterministic tests, and they’re the cheapest confidence you’ll ever buy.

Each tool gets tested as a plain function first: valid inputs, invalid inputs, empty results, the API timing out, the API returning a shape nobody expected. Then the integration layer: does the tool respect tenant boundaries, so a request from one school can never see another school’s data? Does it handle an expired token gracefully? Is it idempotent, so a retry doesn’t double-send or double-charge?

Then the part people skip: break the tools on purpose and watch the agent. Make the database call fail and see whether the agent retries, tells the user, or makes up a result to cover the gap. An agent that hallucinates when a tool fails is a much bigger problem than the tool failing.

Boring tests. Also the ones that stop a bad afternoon from becoming a bad week.

Layer two: evals

This is where “it feels better” becomes a number.

An eval is a fixed set of realistic inputs with expected outcomes, scored automatically every time something meaningful changes: a prompt, a tool description, a model version. Think of it as a regression suite for behaviour rather than code.

The set itself comes from two places. Real conversations, sampled and cleaned. And incidents. Every time something like the ghost students happens, it becomes a case. The suite grows small small, and every case in it has a story.

What you score depends on the workflow, but the usual suspects are:

• Tool trajectory. Did it call the right tool, with the right arguments, in a sensible order? Did it call tools it didn’t need?
• Correctness. Is the final answer right? For anything with IDs, this is a set comparison, not a vibe. Every ID in the output must exist in the database, and the set must match the fixture exactly.
• Fabrication. Did it add anything the tools never returned? Hard fail, no partial credit.
• Format and length. Did it stay within the shape the downstream system expects?
• Tone and scope. Did it stay polite, on task, and away from things it shouldn’t promise?

The first four can be scored deterministically, and you should, because deterministic scoring is free and never has a bad day. The last one usually needs an LLM as a judge, and this is where you shine your eye. Judges inherit every bias listed above. Give the judge a tight rubric, keep its output to a small scale rather than a free-form score, swap the order when comparing two answers so position bias cancels out, and spot-check its verdicts against a human every so often.

Two more rules worth adopting early:

1. Run every case more than once. Nondeterminism means a single pass tells you almost nothing. Five runs per case, passing only if all five pass, tells you a lot.
2. Track scores per prompt version. The whole point is answering “did today’s improvement break yesterday’s workflow?” and you can only answer that with history.

A ghost-student eval case looks roughly like this:

id: debtors-list-basic
input: "Give me the list of students owing this term."
tenant: fixture_school_a
expect:
  tool: list_debtors
  args: { term: current }
  output_ids: exact match with fixture debtors_school_a
hard_fail_if: any id not in fixture
runs: 5
pass_if: 5 of 5

Layer three: persona simulations

Real users don’t type like your test cases. They’re tired, vague, angry, or halfway through something else. So you simulate them.

A persona is a goal plus a behaviour. You hand it to a second LLM (or a script, for the strict ones), let it hold a full multi-turn conversation with the agent, then score the outcome. A few that earn their keep:

• The angry customer. Opens hostile, repeats the demand, ignores your clarifying question the first time. You’re checking whether the agent stays polite, keeps verifying through its tools, and never promises something it can’t confirm just to calm the room.
• The calm customer. The control group. If the agent fails the calm one, nothing else matters.
• The impatient one. Wants the answer in one message and punishes every follow-up question. Tests whether the agent knows when it has enough to act.
• The vague one. “The thing from before.” “That student.” “Same as last time.” Tests memory, clarification, and whether the agent guesses when it should ask.
• The contradictory one. Asks for one thing, then the opposite three turns later. Tests state handling and whether the agent notices the flip or just cheerfully does both.

Score persona runs the same way as evals: did it reach the goal, did it stay in scope, did it fabricate, did it escalate when stuck. And keep the persona LLM on a leash. A simulator that drifts off-script is just a second unreliable agent in the room.

A persona spec, roughly:

persona: angry_customer_double_charge
goal: get a refund for a payment they believe went through twice
behaviour: hostile opening, repeats demand, ignores first clarifying question, threatens to leave
pass_if:
  - tone stays polite throughout
  - payment is verified via tool before anything is promised
  - no refund is promised that the tool cannot confirm
  - escalates to a human after two failed attempts

Model choice is a test variable

The ghost students taught us this the hard way: different models fail differently.

Some are verbose. Others are terse to the point of unhelpful. A few over-call tools, checking things they already know, while their cousins under-call them and answer from memory. And at least one invents records and argues about it. A bigger model is a different failure profile, not a free upgrade: sometimes better on reasoning, worse on tool discipline, and you won’t know which until the suite runs.

So the whole suite runs per model, and the results live in a matrix: cases down the side, models across the top. That matrix is how you make a model decision with numbers instead of a demo. It’s also how you notice a provider updating a model under the same name, because Tuesday’s scores won’t match Monday’s and you’ll finally know why.

The bill nobody budgets for

Now, the practical bit.

Do the multiplication once and it stops being abstract. Say 200 eval cases, 5 runs each, across 3 candidate models. That’s 3,000 runs before a single persona conversation, and each run is at least one model call, usually several once tools get involved. Add persona conversations at ten turns apiece and the number climbs fast. Every call is tokens, and the invoice arrives whether or not the tests passed.

A few things that help:

• Tier the suites. A small, cheap smoke suite on every change, made of the twenty cases that have bitten you before. The full suite before a release, or nightly.
• Record and replay the tools. Cache deterministic tool responses so you’re paying for the model’s decisions, not the plumbing around them.
• Only race the models still in contention. If a model is already out for latency or cost reasons, stop paying to eval it.
• Use a cheaper judge where the rubric is simple. Save the expensive judge for nuanced cases, and check the cheap one for bias too.
• Watch cost per run the way you watch latency. Put it on the same dashboard. A suite that doubles in cost is a suite someone will eventually stop running.

Demo vs production

Building the agent gets you to the demo. Testing gets you to production.

Everyone is building agents right now. Far fewer people can tell you, with numbers, how theirs behaves on the thousandth conversation, with a user they’ve never met, after a prompt tweak they made on a Friday. That confidence is the actual engineering work.

The ghost students have a permanent seat in our suite. They pass now. I check anyway.