How Do You Test An AI Agent Before It Touches A Live Business Workflow?
Synergised Consulting Ltd
Last Updated: 31 October 2026
Test an AI agent by building an eval set of 20-50 tasks drawn from your own real past work, grading whether the intended outcome actually happened rather than whether the reply sounded right, running it in a sandboxed copy of your tools, and then letting it work beside your current process in shadow mode before it touches anything live. That sequence, not a demo, is what tells you whether an agent is safe to put on a real workflow.
Why You Cannot Test An Agent The Way You Test Software
An agent cannot be verified the way ordinary software can, because there is no single correct output to compare against: it reaches results over many steps, and its mistakes accumulate quietly instead of failing loudly at one point. A demonstration on a chosen example proves almost nothing about how it will handle the messy, ambiguous cases a live workflow serves up. Testing an agent means walking it through those paths deliberately, before a customer walks through them for you.
Anthropic's engineering team defines the problem precisely [1]: agents "operate over many turns: calling tools, modifying state, and adapting based on intermediate results", and these same capabilities that make them useful also make them harder to evaluate.
That is why a demonstration carries so little weight. A demo shows the agent succeeding once, on a path someone chose for it. A live workflow serves up the paths nobody chose: the messy scan, the ambiguous request, the system that is down for maintenance. Those are the paths a test set has to walk first.
There is a second trap, and it is subtler. In discovery workshops we consistently find that owners arrive with one showcase task in mind, the thing the agent did impressively in the sales call. The workflow that actually needs the agent is usually the repetitive bundle discussed somewhere in the middle of the conversation, and it behaves nothing like the showcase. Testing has to cover the real workflow, not the memorable one.
Build The Eval Set From Work That Already Happened
An eval set is a list of real test tasks, each with defined inputs and a success criterion, and the fastest way to build one is to mine work the business has already done rather than imagining cases from scratch. Anthropic's guidance puts the starting size in the range of a few dozen tasks drawn from real failures, and tests the agent passes permanently become regression cases.
Anthropic's guidance for teams building agent evaluations is specific: 20-50 simple tasks drawn from real failures is a great start [1]. The tasks do not need to be elegant. They need to be real, because real tasks carry the edge cases that invented ones miss.
The source of those tasks already exists in your business. Pull the last month of the workflow the agent will enter: actual customer emails, actual invoices with odd formats, the request that got handled wrongly and had to be fixed by hand. Each of those is a test case, and the ones that went wrong are the most valuable cases in the set. If you sit down and try to imagine test cases from scratch, you will imagine the workflow as it should work. If you take them from what already happened, you get the workflow as it actually works.
Two kinds of test belong in the set, and Anthropic separates them usefully [1]. Capability tests ask what the agent cannot do well yet, and they are where improvement happens. Regression tests ask whether the agent still handles everything it used to handle, and those should pass nearly every time. As the agent improves, tests it once failed graduate into the regression set, so the bank grows in one direction only.
Grade The Outcome, Not The Answer
Grade whether the intended outcome actually happened in the systems the agent touched, not whether its reply sounded right, because an agent can fluently describe work it never did. Anthropic's own example separates the two: a booking agent can say a flight is booked while what actually matters is whether a reservation exists in the database. Review the trajectory of tool calls and state changes too, since an agent can reach the right result by the wrong actions.
Anthropic's own definition of an outcome makes the distinction concrete: a flight-booking agent might end its session by saying "Your flight has been booked", but the outcome is whether a reservation actually exists in the database [1]. The check that matters runs against the world the agent changed, not the words it produced.
This extends to how the agent behaved on the way to its answer. Singapore's Infocomm Media Development Authority, in its starter kit for testing agentic systems produced with the Cyber Security Agency and GovTech, recommends testing the agent's entire trajectory rather than just its output, because an agent "may produce the right outcome while taking incorrect actions" along the way [2]. It might send the refund to the right customer but from the wrong account, or book the meeting but on a calendar nobody uses. Log the tool calls and the state changes, not just the final message, and review that log the way you would review a new employee's first week.
Sandbox First, Shadow Second, Autonomy Gradually
Run the agent first in a realistic sandbox where nothing real breaks, then in shadow mode on live inputs with nobody acting on its outputs, and only then give it limited real volume with human approval on each result. Autonomy widens when the pass rate on the eval set earns it, not when a calendar date arrives. Skipping these stages is the failure both government guidance and platform documentation name.
The progression that the IMDA starter kit and AWS's Well-Architected guidance both describe is the same in shape, and it starts with the sandbox: a contained copy of the real tools where the agent can act and nothing real breaks. Anthropic evaluates coding agents in "a sandboxed environment where it can use software applications" and checks whether the intended outcome was achieved [1]. The IMDA starter kit adds one warning that matters here: keep the sandbox realistic, because agents are sensitive to signs they are in a simulation, and a test run against tidy dummy data tells you little about how the agent handles your actual records [2].
Next comes shadow mode: the agent runs on your live inputs, but its outputs are acted on by nobody, and your current process still handles everything. AWS's Well-Architected guidance names skipping this stage as a specific, common failure: teams that stop at deterministic unit tests and skip shadow testing on real traffic ship agents whose behaviour on edge cases, adversarial inputs and production data distributions is unknown [3]. Shadow mode is how you find out what the agent would have done, at zero risk, and compare it against what actually happened.
Only then does autonomy start, and it starts small: a limited slice of real volume, with a human approving each output before it lands, widening only when the error record justifies it. The IMDA guidance puts it plainly: "start in contained but realistic environments, moving closer to production environments when there is greater confidence" [2]. The word that does the work in that sentence is confidence, and confidence is something the evidence bank either has or has not earned. Note what this progression is not: it is not a calendar. Time in operation earns nothing by itself; pass rate on the eval set is the currency.
Turn Every Failure Into A Permanent Test
Testing does not end at launch because models, tools and workflows keep changing, so the eval set must stay alive. Re-run the full set after every change to the agent, its model or its tools, and add every production failure back into the set as a permanent regression case. Over time the set becomes a written record of how the agent behaves, in test cases rather than assurances.
Anthropic points out how fast the ground moves: on SWE-bench Verified, a widely used coding benchmark, model performance went from 40% to over 80% within a single year [1]. An agent tested once against last year's model is not tested.
So two habits keep the evidence alive. First, refresh the eval set: AWS's guidance names the failure mode directly, where "test and evaluation datasets are created once and never refreshed, so scores stay green while real-world failure modes go undetected" [3]. Second, and more valuable, feed production failures back in. Every time the agent gets something wrong in live operation, that case becomes a permanent regression test, so the same failure can never return unnoticed. Run the full set after every change to the agent, its model, or its tools.
A useful failure gets rarer every month it is in the set, and the set itself becomes the evidence a new person, or a new owner, can read instead of a promise they have to take on trust.
What This Buys You
None of this needs an engineering team: an eval set is a list of past tasks and what a good result looks like, a sandbox is a copy of access with the risky rights removed, and shadow mode is a comparison rule. What accumulates is inspectable evidence of how the agent behaves, which is what lets a team trust an agent instead of quietly working around it.
The compounding part is the regression bank: every failure the agent ever makes, anywhere, becomes a case that protects you from that failure forever.
That is also what makes delegation to an agent defensible to everyone who depends on the workflow. The owner who can say "this agent handles our full set of real past tasks correctly, here is the record, and every failure since launch is in the set too" is describing something a reviewer can check, not something promised. In a small business, that distinction is the difference between an agent the team trusts and an agent the team finds reasons to bypass.
