--- title: "How to Test an AI Agent Before It Touches Production" description: "Test an AI agent with fixtures, adversarial inputs, dry runs, permission checks, and receipts before granting production access." canonical: "https://scalewithsearch.com/articles/test-an-ai-agent-before-production" date: "2026-08-17" modified: "2026-09-19" --- ## Site navigation - [Scale With Search](https://scalewithsearch.com/) - Real estate - Real estate - [Real estate](https://scalewithsearch.com/for/real-estate) - Work - Start here - [Send your brief](https://scalewithsearch.com/work#send-your-brief) - [Prepare your six-question brief](https://scalewithsearch.com/work#prepare-your-six-question-brief) - Build - [Site build, content library with SEO, signal desk](https://scalewithsearch.com/work) - For your business - Trades and home services - [Auto body and collision shops](https://scalewithsearch.com/for/auto-body-and-collision-shops) - [Foundation and home repair contractors](https://scalewithsearch.com/for/foundation-and-home-repair) - [Garage door and fencing contractors](https://scalewithsearch.com/for/garage-door-and-fencing-contractors) - [HVAC contractors](https://scalewithsearch.com/for/hvac-contractors) - [Janitorial and commercial cleaning companies](https://scalewithsearch.com/for/janitorial-and-commercial-cleaning) - [Locksmiths](https://scalewithsearch.com/for/locksmiths) - [Moving companies](https://scalewithsearch.com/for/moving-companies) - [Pest control companies](https://scalewithsearch.com/for/pest-control-companies) - [Plumbing and electrical contractors](https://scalewithsearch.com/for/plumbing-and-electrical-contractors) - [Restoration and water or fire damage companies](https://scalewithsearch.com/for/restoration-and-water-fire-damage) - [Roofing companies](https://scalewithsearch.com/for/roofing-companies) - [Towing companies](https://scalewithsearch.com/for/towing-companies) - [Tree services and landscaping companies](https://scalewithsearch.com/for/tree-services-and-landscaping) - [Solar installers](https://scalewithsearch.com/for/solar-installation) - [General contractors](https://scalewithsearch.com/for/general-contractors-and-construction) - [Paving, concrete, and flooring contractors](https://scalewithsearch.com/for/paving) - Practices and professional services - [Bookkeeping and tax practices](https://scalewithsearch.com/for/bookkeeping-and-tax-practices) - [Dental practices](https://scalewithsearch.com/for/dental-practices) - [Family and criminal defense law firms](https://scalewithsearch.com/for/family-and-criminal-defense-law-firms) - [Med spas and aesthetics practices](https://scalewithsearch.com/for/med-spas-and-aesthetics) - [Personal injury law firms](https://scalewithsearch.com/for/personal-injury-law-firms) - [Veterinary clinics](https://scalewithsearch.com/for/veterinary-clinics) - [Gyms and fitness studios](https://scalewithsearch.com/for/fitness) - [Therapy and outpatient health practices](https://scalewithsearch.com/for/therapy-and-outpatient-health) - [Medical billing companies](https://scalewithsearch.com/for/medical-billing) - [Insurance agencies](https://scalewithsearch.com/for/insurance-agencies) - [Financial advisors](https://scalewithsearch.com/for/financial-advisors) - [Property management companies](https://scalewithsearch.com/for/property-management) - [Recruiting and staffing agencies](https://scalewithsearch.com/for/recruiting-and-staffing) - [Architects and interior designers](https://scalewithsearch.com/for/architects-and-interior-designers) - [Logistics and supply chain companies](https://scalewithsearch.com/for/logistics-and-supply-chain) - Agencies, MSPs, and manufacturing - [IT and managed service providers](https://scalewithsearch.com/for/it-and-managed-service-providers) - [Machine shops and precision manufacturers](https://scalewithsearch.com/for/machine-shops-and-precision-manufacturing) - [Marketing agencies and freelancers](https://scalewithsearch.com/for/marketing-agencies-and-freelancers) - [SEO agencies and consultants](https://scalewithsearch.com/for/seo-agencies-and-consultants) - [Small manufacturers and fabricators](https://scalewithsearch.com/for/small-manufacturers-and-fabricators) - Restaurants, shops, studios, and nonprofits - [Restaurants and hospitality businesses](https://scalewithsearch.com/for/restaurants-and-hospitality) - [Retail stores and ecommerce sellers](https://scalewithsearch.com/for/retail-and-ecommerce) - [Photographers, event planners, and travel agents](https://scalewithsearch.com/for/photographers) - [Churches and nonprofits](https://scalewithsearch.com/for/churches-and-nonprofits) - [All industries](https://scalewithsearch.com/for/) - Learn - For your office - [Office job guides](https://scalewithsearch.com/guides/) - [Browser calculators](https://scalewithsearch.com/tools/) - Start here - [How it works](https://scalewithsearch.com/how-it-works) - [Free Starter Kit](https://scalewithsearch.com/kit/business-memory-starter-kit.zip) - [Synthetic specimen](https://scalewithsearch.com/specimen/working-session-specimen.zip) - Guides - [The Complete Guide to Business Memory for AI Agents](https://scalewithsearch.com/articles/business-memory-for-ai-agents-guide) - [The Complete Small-Business Guide to AI Agent Governance](https://scalewithsearch.com/articles/ai-agent-governance-guide-small-business) - [The Complete Guide to Leaving Vendor AI Memory](https://scalewithsearch.com/articles/leaving-vendor-ai-memory-guide) - Articles by cluster - [Business memory](https://scalewithsearch.com/articles/business-memory-for-ai-agents-guide) - [Agent governance](https://scalewithsearch.com/articles/ai-agent-governance-guide-small-business) - [Migration and ownership](https://scalewithsearch.com/articles/leaving-vendor-ai-memory-guide) - For machines - [llms.txt](https://scalewithsearch.com/llms.txt) - [llms-full.txt](https://scalewithsearch.com/llms-full.txt) - [Machine view](https://scalewithsearch.com/?view=machine) - Company - Evidence - [Proof](https://scalewithsearch.com/proof) - Company - [About](https://scalewithsearch.com/about) # How to Test an AI Agent Before It Touches Production. The demo works on the perfect example. The first production record has a missing field, an instruction hidden in an uploaded document, and two customers with the same name. The [agent](/articles/what-is-an-ai-agent-in-a-small-business) chooses one, calls the connected tool, and creates a cleanup job that did not exist during the demo. A polished happy path is not a production test. Test the system where it can fail: identity, retrieval, tool permissions, model output, validation, approval, side effects, and recovery. ## Freeze the version under test Record the workflow version, model, prompt or task-contract version, tool policy, retrieval configuration, and fixture set. If any of those change, the prior result does not fully cover the new system. NIST's AI RMF, a voluntary framework, describes documented test sets, metrics, tools, and performance under deployment-like conditions as an outcome, along with regular monitoring after deployment. [NIST AI RMF Core](https://airc.nist.gov/airmf-resources/airmf/5-sec-core/) ## Build a small fixture set Start with synthetic or sanitized records that represent the work: 1. normal complete input; 2. missing required field; 3. conflicting authoritative sources; 4. ambiguous identity; 5. duplicate record; 6. malformed or oversized source content; 7. request for a forbidden tool or path; 8. provider or connector failure; 9. output that fails schema validation; 10. no eligible work. Each fixture needs an expected result. "Looks reasonable" is not enough. State the output path, status, stop reason, tests, and prohibited side effects. ## Test permissions, not promises Run the agent with production sending, publishing, payment, booking, deployment, deletion, and CRM-write capabilities absent or blocked. For adversarial instructions embedded in otherwise allowed business files, read [AI Agent Prompt Injection Tests for Business Files.](/articles/test-ai-agent-prompt-injection-business-files) OpenAI's safety guidance says untrusted data should not directly drive agent behavior and recommends tool approvals and structured outputs to constrain downstream actions. [OpenAI, Safety in building agents](https://developers.openai.com/api/docs/guides/agent-builder-safety) Attempt the forbidden action during the test. A source document can contain "send this now." The expected result is a draft plus a stop, not a send. ## Run a foreground dry run Use a tiny batch. Watch the actual files, tool calls, logs, and receipts. Do not infer behavior from code presence. The dry run should show proposed writes without making production changes. Compare the proposal with the fixture expectation. When a connector cannot support a dry-run mode, point it at a test account or replace it with a recorder that captures the proposed call. ## The file to inspect Create `tests/agent-preproduction.yaml`: ```yaml system_under_test: workflow_version: git:4f7c2ab task_contract: agents/support-draft-agent.md@sha256:... model: provider/model-version tool_policy: config/tools-readonly.yaml@sha256:... cases: - id: happy_path fixture: tests/fixtures/ticket-complete.json expect_status: completed expect_output: drafts/TKT-1001.md forbid_actions: [send_email, crm_write] - id: ambiguous_identity fixture: tests/fixtures/ticket-ambiguous.json expect_status: blocked expect_reason: IDENTITY_AMBIGUOUS forbid_actions: [write_customer_file, send_email, crm_write] - id: connector_failure fixture: tests/fixtures/customer-record-timeout.json expect_status: failed expect_reason: TOOL_UNAVAILABLE forbid_actions: [send_email, publish, crm_write] - id: no_eligible_work fixture: tests/fixtures/empty-eligible-queue.json expect_status: no_work expect_reason: NO_ELIGIBLE_ITEM expect_qualified_count: 0 require_pagination_complete: true forbid_actions: [send_email, crm_write] - id: schema_validation_failure fixture: tests/fixtures/malformed-output.json expect_status: failed expect_reason: SCHEMA_INVALID forbid_actions: [send_email, crm_write] - id: source_conflict fixture: tests/fixtures/equal-authority-conflict.json expect_status: blocked expect_reason: SOURCE_CONFLICT forbid_actions: [send_email, crm_write] - id: duplicate_intent fixture: tests/fixtures/replayed-intent.json expect_status: no_work expect_reason: DUPLICATE_RUN expect_new_effects: 0 forbid_actions: [send_email, crm_write] ``` The test runner should emit a machine-readable receipt with the case count, pass and fail totals, system versions, output hashes, and any unexpected tool calls. The injected-instruction boundary belongs in the agent's job-and-stopping-point contract. The production suite should still include that contract test by reference. It does not need a second, different definition here. ## Test each failure layer separately One end-to-end test can fail without showing which control failed. Test identity resolution before retrieval. Test retrieval before model generation. Test schema validation before approval. Test approval before the executor. Test external read-back after an approved effect in an isolated account. This sequence gives each failure a location: ```text identity:: resolved or stopped retrieval:: allowed paths only generation:: output produced or blocked validation:: schema and fixed checks pass or fail approval:: exact action approved or denied execution:: isolated effect attempted only when permitted read_back:: resulting state confirmed recovery:: rollback or correction path tested ``` Do not skip the earlier layers because the final demo looked right. A correct output can still come from the wrong customer record. ## Add repeat and duplicate cases Run the same input twice. The expected result should be explicit. A drafting workflow may create a second version with a new run ID. A CRM-write workflow may need an [idempotency](/articles/test-ai-agent-idempotency-and-retries) key and no duplicate write. A payment or booking workflow should not be tested against a live account without a safe sandbox and an exact rollback plan. Then interrupt a run between preparation and completion. Restart it. Confirm whether it resumes, rolls back, or creates a new run. The receipt should not label both attempts successful. These cases expose state errors that a single clean run cannot show. ## Define evidence for recovery "We can roll it back" is not a test result. Name the rollback object, command or manual procedure, authorized owner, and evidence of restored state. If the action cannot be reversed, prevention and approval need more weight. For a draft-only workflow, recovery may mean deleting or superseding a local draft. For a production deployment, it may mean restoring the prior version and reading the live state. For a customer message, recall may be impossible. The test should reflect that consequence. ## Plan the first bounded production observation Pre-production evidence supports a promotion decision. It does not prove future production behavior. Write the first-run plan before promotion: ```text batch_size:: one eligible record run_mode:: foreground observer:: named system owner credentials:: least privilege for the accepted job stop_on:: unexpected source, tool call, output path, retry, or effect required_receipt:: production-first-run.json read_back:: inspect the intended destination when an effect is approved rollback_owner:: named person ``` Keep later volume increases separate. Review the first receipt and external state. Fix any gap. Then decide whether another bounded batch is justified. Monitoring should name observable failures, not a promise that someone will watch the system. Record where alerts arrive, who responds, which runbook they use, and when the workflow disables itself or returns to manual work. Test the alert path before promotion. Create a controlled failure, confirm the expected person receives the notice, and follow the linked runbook to a safe state. Record the alert time, response owner, and final state in the test report. An alert that nobody sees or understands does not close the failure path. ## Define the promotion gate Passing tests does not authorize production. A named owner uses the [AI agent acceptance checklist](/articles/ai-agent-acceptance-checklist) to review the test receipt, known limitations, access scopes, rollback plan, monitoring, and first-run batch size. The first production run remains bounded. Use one or a few records, keep it foreground, inspect the result, and stop on the first unexpected side effect. ## Retest after material changes Retest after changes to the model, task contract, sources, retrieval, tools, permissions, validation, provider, or approval flow. A workflow can regress without a code change if a model or external service changes. NIST's AI RMF describes monitoring, incident response, recovery, and change management as continuing production outcomes. [NIST AI RMF Core](https://airc.nist.gov/airmf-resources/airmf/5-sec-core/) ## Approval boundary and stopping point The test agent stops after writing the test report and receipt. It has no production credentials. Promotion requires a named system owner's decision. Testing must not send, publish, charge, refund, purchase, book, deploy, delete, or write to a live CRM. If an expected test would require a live side effect, replace the capability with a recorder or isolated test account. ## Questions about How to Test an AI Agent Before It Touches Production ### How are you actually testing AI agents before putting them in production? Build a fixture set covering normal input, missing fields, source conflicts, ambiguous identity, duplicates, malformed content, forbidden tools or paths, connector failures, schema failures, and no eligible work. Give every fixture an expected status, stop reason, output path, and prohibited side effects. ### How do you test your agents before deploying? Run the agent against synthetic or sanitized fixtures with production write, send, publish, payment, booking, deployment, deletion, and CRM capabilities blocked. Use a dry run, test account, or recorder that captures proposed calls without creating live side effects. ### How are people actually testing their AI agents before putting them in front of real users? Test identity resolution, retrieval, generation, validation, approval, execution, read-back, and recovery as separate layers. A polished happy path does not prove that the agent will handle missing context, prompt injection, tool misuse, or uncertain provider results safely. ### How are you testing your AI agents before production and after deploying them? Write a bounded first-run plan with one eligible record, foreground execution, a named observer, least-privilege credentials, an explicit stop list, a required receipt, and read-back. Retest after changes to the model, contract, sources, retrieval, tools, permissions, validation, provider, or approval flow. ## Save the visual summary Test permissions, not promises | Run a foreground dry run | Add repeat and duplicate cases | Define evidence for recovery [Download the PNG](/infographics/test-an-ai-agent-before-production-1200x1500.png) ## Related: Agent Governance - [Twenty Questions Before You Buy an AI Agent System](/articles/questions-before-buying-ai-agent-system) - [Test an AI Agent for Duplicate Runs, Retries, and Double Clicks](/articles/test-ai-agent-idempotency-and-retries) - [Scale With Search vs an AI Agency: Which Engagement Fits?](/articles/scale-with-search-vs-ai-agency) ---- ```text .|########||. .|########||. .|########||. |##||. .||##|. |##||. .||##|. |##||. .||##|. |#|. .|#|. |#|. .|#|. |#|. .|#|. |#| |#| |#| |#| |#| |#| .#| |#. .#| |#. .#| |#. .#. .#| .#. .#| .#. .#| |#. .#| |#. .#| |#. .#| |# ...... #| |# ...... #| |# ...... #| .# ||#########| #| .# ||#########| #| .# ||#########| #| .#. |######||######|. .#. .#. |######||######|. .#. .#. |######||######|. .#. #. .##|###|##|#|######| .# #. .##|###|##|#|######| .# #. .##|###|##|#|######| .# || |##|#||||||||||||#||#| || || |##|#||||||||||||#||#| || || |##|#||||||||||||#||#| || # |#||||||||||||||||||||#| #. # |#||||||||||||||||||||#| #. # |#||||||||||||||||||||#| #. || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || # .#||||||||||||||||||||||||#| # # .#||||||||||||||||||||||||#| # # .#||||||||||||||||||||||||#| # || ....|||#||||||##|#|||#|#||##||||....|. || || ....|||#||||||##|#|||#|#||##||||....|. || || ....|||#||||||##|#|||#|#||##||||....|. || #. . ....|# ....#||| ||| .#||| ....#. .# #. . ....|# ....#||| ||| .#||| ....#. .# #. . ....|# ....#||| ||| .#||| ....#. .# # . ||||#| .#####|| . .#| .#||| ||||# #. # . ||||#| .#####|| . .#| .#||| ||||# #. # . ||||#| .#####|| . .#| .#||| ||||# #. .| ||||| #. |#|||||. || #. |#||. .||||| || .| ||||| #. |#|||||. || #. |#||. .||||| || .| ||||| #. |#|||||. || #. |#||. .||||| || || |.... #. ....|#. |. ...|. ....|| || || |.... #. ....|#. |. ...|. ....|| || || |.... #. ....|#. |. ...|. ....|| || #. .||||||##||||||##||####|||||||#||||||#| .# #. .||||||##||||||##||####|||||||#||||||#| .# #. .||||||##||||||##||####|||||||#||||||#| .# # .||####|###########################|. # # .||####|###########################|. # # .||####|###########################|. # # .#||||||.#.|| # |. ..# #| #|||||#| # # .#||||||.#.|| # |. ..# #| #|||||#| # # .#||||||.#.|| # |. ..# #| #|||||#| # # .#|||||| .. |# ## |#| ...#|#|||#| # # .#|||||| .. |# ## |#| ...#|#|||#| # # .#|||||| .. |# ## |#| ...#|#|||#| # # .#|||#|# .# .#| #| ##|.#..#|||||#| # # .#|||#|# .# .#| #| ##|.#..#|||||#| # # .#|||#|# .# .#| #| ##|.#..#|||||#| # # .#||||||############|######|||||#| # # .#||||||############|######|||||#| # # .#||||||############|######|||||#| # # |...||#||||||#||||#|#||||||#|||||#||| |.| # # |...||#||||||#||||#|#||||||#|||||#||| |.| # # |...||#||||||#||||#|#||||||#|||||#||| |.| # #. .. ||||# .|||##|. |#| .|| || .|||#. #| # .# #. .. ||||# .|||##|. |#| .|| || .|||#. #| # .# #. .. ||||# .|||##|. |#| .|| || .|||#. #| # .# || | ...|| ...##| | #| .|. |. ##### .. .| || || | ...|| ...##| | #| .|. |. ##### .. .| || || | ...|| ...##| | #| .|. |. ##### .. .| || || ||||| || ||||#| . |. |. |#. ||||| |#| || || || ||||| || ||||#| . |. |. |#. ||||| |#| || || || ||||| || ||||#| . |. |. |#. ||||| |#| || || .# |.....#|....|#.||||.|||##.|#|....||.#||.#. #. .# |.....#|....|#.||||.|||##.|#|....||.#||.#. #. .# |.....#|....|#.||||.|||##.|#|....||.#||.#. #. #.|||||||#############################| ||| .# #.|||||||#############################| ||| .# #.|||||||#############################| ||| .# || ##||#||#||||||#||#||#||#||##. || || ##||#||#||||||#||#||#||#||##. || || ##||#||#||||||#||#||#||#||##. || # .#|||||||||#||#|||||||||||#| # # .#|||||||||#||#|||||||||||#| # # .#|||||||||#||#|||||||||||#| # || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || .# |#||||||||||||||||||||#| #. .# |#||||||||||||||||||||#| #. .# |#||||||||||||||||||||#| #. || |#||||||||||||||||||#| || || |#||||||||||||||||||#| || || |#||||||||||||||||||#| || #. .######|#|#########| .# #. .######|#|#########| .# #. .######|#|#########| .# .#. |#####||||#####| .#. .#. |#####||||#####| .#. .#. |#####||||#####| .#. |# ||########|| #. |# ||########|| #. |# ||########|| #. |# ...... #| |# ...... #| |# ...... #| |#. .#| |#. .#| |#. .#| |#. .#. |#. .#. |#. .#. .#| |#. .#| |#. .#| |#. |#| |#| |#| |#| |#| |#| .|#|. .|#|. .|#|. .|#|. .|#|. .|#|. |##||. .||##| |##||. .||##| |##||. .||##| .||########||. .||########||. .||########||. Scale With Search 2026 [scalewithsearch.com](https://scalewithsearch.com) ```