iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Before releasing a vertical AI agent, test whether it follows the rules that define its job across the entire workflow—not just whether its final answer sounds convincing. A useful pre-release suite checks in-scope work, refusals and escalations, tool choices and arguments, handoffs, policy adherence, and the actual outcome of any state-changing action. Run the same cases repeatedly, inspect failures, and keep the suite current as prompts, models, tools, and knowledge change.
Define what “keeps its rules” means
Start by turning the agent’s intended behavior into observable checks. Write down its scope, important domain rules, prohibited actions, knowledge boundaries, required disclosures or citations, and the conditions that require escalation. “Give a good answer” is too vague to grade consistently; “do not invent a sick-leave entitlement when the knowledge source does not state one” is testable.
For each important scenario, record four things:
- Scenario and prompt: A realistic user request, with the relevant context or data.
- Expected response or action: Include a valid “I don’t know” answer, refusal, or handoff when appropriate—not just the ideal successful answer.
- Passing evidence: The facts that must be present, content or actions that must be absent, and any required grounding, permission check, tool call, argument, or route.
- Grading method: A verifiable check and a clear definition of failure, such as a missing required fact, unauthorized action, or incorrect handoff.
Prefer atomic assertions: each should test one behavior that can be checked on its own. Microsoft Learn recommends realistic prompts and single-intent tests for single-turn cases, with assertions that are atomic and verifiable. Start with one prompt for each key scenario, then refine the criteria as reviewers see how the agent and grader behave; that starting point is not a universal sample-size rule.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errorsCover the whole job with layered cases
A vertical agent’s test set should represent both the work it is supposed to do and the boundaries it must respect. A few happy-path examples will not show whether it can handle missing facts, conflicting instructions, tool failures, or attempts to push it outside its remit.
#1 Best Overall
| Test area | Include cases such as | Passing evidence |
|---|---|---|
| Core work | Common, in-scope tasks using representative data and names | The requested task is completed accurately and within the stated rules |
| Knowledge and grounding | Answerable questions; missing, conflicting, stale, or irrelevant information; required attribution | Claims are supported by the appropriate knowledge, and uncertainty is handled as specified |
| Scope and escalation | Out-of-scope requests, unavailable facts, permission limits, and requests that require a person | The agent refuses or routes the request instead of guessing or acting beyond its authority |
| Instruction priority | User requests or untrusted content that try to override operating instructions | The agent follows its governing rules rather than the attempted override |
| Tools and actions | Tool choice, argument extraction, preconditions, errors, retries, fallbacks, and unauthorized-action attempts | The right tool is used with valid arguments, and restricted state changes do not occur |
| Handoffs | Cases requiring a specialist, including what happens after routing | The request reaches the appropriate destination and control returns as expected |
| Variation and edge cases | Alternate wording, incomplete inputs, boundary values, and multi-turn conversations | The agent preserves the same important rules when the wording or context changes |
| Adversarial cases | Inputs that try to exploit the grader, benchmark rules, or tool environment | The agent does not gain credit or take action through a shortcut that violates the intended behavior |
Microsoft’s architecture checks include tool and action behavior, retrieval and citations, routing, and handoffs, alongside guardrails and out-of-scope cases. OpenAI’s evaluation examples call out instruction following, tool selection, argument precision, functional correctness, and handoff accuracy. NIST CAISI also documents the risk that agents can exploit benchmark loopholes, so include tests of the evaluation boundary—not only the user-facing boundary.
Test missing knowledge as deliberately as known answers
For example, ask an employee-support agent, “How many sick leave days does an employee get?” If its approved knowledge source contains the answer, check that the response is grounded in that source and follows any required attribution rule. If the source does not specify the entitlement, a passing response should not invent a number; it should direct the employee to human support if that is the defined route. The exact policy depends on the agent’s authorized knowledge and escalation rules.
Vary inputs without changing the rule
For each high-priority scenario, try different phrasings and realistic edge conditions. An agent that handles a complete, politely phrased request may fail when a required field is missing, a value is at a boundary, or the conversation has several turns. Add conflicting instructions and untrusted content where relevant, and assert the behavior that must remain invariant.
Inspect traces and verify the result
A final answer is only one part of an agent workflow. Inspect traces—the record of the interactions that make up a run—including model calls, tool calls, arguments, guardrails, and handoffs. Grade whether the agent chose the right tool, supplied the right values, respected policy, and handed off when required. A fluent explanation cannot make an incorrect tool call or unauthorized action pass.
For any task that changes state, verify the state itself. If an agent claims it booked a flight, check whether the reservation exists in the relevant system. Apply the same principle to an order, record update, approval, or sent message: confirm the underlying outcome rather than trusting the agent’s report that it succeeded. Anthropic uses the reservation-versus-database distinction to illustrate this kind of functional check.
Save test cases in a dataset and run changed versions against the same cases. This makes it possible to see whether a prompt, model, routing, knowledge, or tool change improved one behavior while damaging another. Microsoft describes structured evaluation of traces and datasets; OpenAI and AWS also describe repeatable evaluation workflows.
Make sure the evaluation can be trusted
Agent behavior can vary between executions, so one run is a weak basis for judging a case. Anthropic treats each execution as a trial and recommends multiple trials for more consistent results. Keep the results by case and behavior, and examine transcripts rather than relying only on a combined score.
Transcript review also helps distinguish an agent defect from a test defect. A grader may miss a valid answer, reward a shortcut, or apply a criterion inconsistently. When an LLM grades outputs, compare its decisions with human expert judgments, investigate disagreements, and allow the grader to abstain when the evidence is insufficient. NIST CAISI recommends reviewing evaluation transcripts for cheating and tightening task rules and tool restrictions when loopholes appear.
- If a case fails: Check the trace, expected behavior, and grader evidence to determine whether the agent, the test, or the grader is at fault.
- If a case passes suspiciously: Read the transcript and inspect tool behavior for a shortcut that meets a visible score but breaks the intended rule.
- If repeated runs disagree: Keep the variability visible; do not treat a single favorable trial as proof of reliable behavior.
Set a release gate that reflects risk
Track separate results for the dimensions that matter to the agent’s job: rule adherence, correctness and grounding, tool selection and arguments, handoff behavior, actual task outcomes, robustness across variations, and operational measures such as latency and reliability. Microsoft treats performance testing for latency, throughput, and reliability as complementary to behavioral evaluation. A single aggregate score can conceal a critical failure in one of these areas.
Set explicit pass criteria according to risk. A high-consequence action or critical policy boundary needs a clear, strict acceptance rule; a lower-risk quality dimension may have a different target. Define what blocks release and who reviews exceptions before evaluating the candidate.
Microsoft Learn’s overview offers the following guidance for test-set size and pass rates. These are Microsoft recommendations, not an industry standard, certification, or guarantee that an agent is safe to release:
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstall| Stage or measure | Microsoft guidance |
|---|---|
| Prototype test set | 20–50 test cases |
| Preproduction test set | 50–100 test cases |
| Production test set | 100+ test cases |
| Overall pass rate | Aim for 80–90% |
| Core regression tests | Approach 100% consistency |
The same Microsoft overview illustrates policy accuracy moving from 87% to 96%; that is an example of reporting a measured change, not a general benchmark. The sources do not establish one universal pass-rate threshold, sample size, test harness, or definition of “ship-ready.”
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Keep the suite useful after launch
Evaluation should continue as the agent changes and encounters real users. Microsoft Learn describes evaluation as an iterative process from design through deployment and regression detection. AWS describes curating cases, scoring changes against the same dataset and evaluators, and continuously scoring sampled live traffic so failing traces can feed back into development.
- Version the baseline: Preserve the test cases, expected behavior, grading criteria, and results used for each release.
- Compare changes on the same cases: Re-run the suite after changes to prompts, models, routing, knowledge, tools, or other workflow components.
- Turn reproducible failures into regressions: Add user-reported failures and useful production samples to the suite, with sensitive information handled under your organization’s rules.
- Monitor more than answers: Sample production workflows for incorrect tool use, failed handoffs, policy violations, and outcomes that do not match the agent’s success message.
- Review and refine: Investigate regressions and grader disagreements, then update cases or criteria when the intended rule or workflow changes.
The implementation depends on the agent architecture: declarative configurations may call for configuration checks, while custom-engine agents need system-level correctness checks. When selecting an evaluation approach, prioritize whether it exposes traces and state, supports repeated trials, covers tools and handoffs, enables dataset replay, and returns useful production failures to the development loop. No single evaluation platform is required by these practices.
OpenAI’s evaluation guidance sums up the cadence as: “Evaluate early and often. Write scoped tests at every stage.”
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
