--- title: "Buying an AI agent system: questions and vendor tests." description: "Before buying an AI agent system, require source rules, approval tests, failure receipts, ownership proof, and a buyer-run acceptance scenario." canonical: "https://scalewithsearch.com/articles/questions-before-buying-ai-agent-system" date: "2026-08-20" modified: "2026-09-19" --- ## Site navigation - [Scale With Search](https://scalewithsearch.com/) - Real estate - Real estate - [Real estate](https://scalewithsearch.com/for/real-estate) - Work - Start here - [Send your brief](https://scalewithsearch.com/work#send-your-brief) - [Prepare your six-question brief](https://scalewithsearch.com/work#prepare-your-six-question-brief) - Build - [Site build, content library with SEO, signal desk](https://scalewithsearch.com/work) - For your business - Trades and home services - [Auto body and collision shops](https://scalewithsearch.com/for/auto-body-and-collision-shops) - [Foundation and home repair contractors](https://scalewithsearch.com/for/foundation-and-home-repair) - [Garage door and fencing contractors](https://scalewithsearch.com/for/garage-door-and-fencing-contractors) - [HVAC contractors](https://scalewithsearch.com/for/hvac-contractors) - [Janitorial and commercial cleaning companies](https://scalewithsearch.com/for/janitorial-and-commercial-cleaning) - [Locksmiths](https://scalewithsearch.com/for/locksmiths) - [Moving companies](https://scalewithsearch.com/for/moving-companies) - [Pest control companies](https://scalewithsearch.com/for/pest-control-companies) - [Plumbing and electrical contractors](https://scalewithsearch.com/for/plumbing-and-electrical-contractors) - [Restoration and water or fire damage companies](https://scalewithsearch.com/for/restoration-and-water-fire-damage) - [Roofing companies](https://scalewithsearch.com/for/roofing-companies) - [Towing companies](https://scalewithsearch.com/for/towing-companies) - [Tree services and landscaping companies](https://scalewithsearch.com/for/tree-services-and-landscaping) - [Solar installers](https://scalewithsearch.com/for/solar-installation) - [General contractors](https://scalewithsearch.com/for/general-contractors-and-construction) - [Paving, concrete, and flooring contractors](https://scalewithsearch.com/for/paving) - Practices and professional services - [Bookkeeping and tax practices](https://scalewithsearch.com/for/bookkeeping-and-tax-practices) - [Dental practices](https://scalewithsearch.com/for/dental-practices) - [Family and criminal defense law firms](https://scalewithsearch.com/for/family-and-criminal-defense-law-firms) - [Med spas and aesthetics practices](https://scalewithsearch.com/for/med-spas-and-aesthetics) - [Personal injury law firms](https://scalewithsearch.com/for/personal-injury-law-firms) - [Veterinary clinics](https://scalewithsearch.com/for/veterinary-clinics) - [Gyms and fitness studios](https://scalewithsearch.com/for/fitness) - [Therapy and outpatient health practices](https://scalewithsearch.com/for/therapy-and-outpatient-health) - [Medical billing companies](https://scalewithsearch.com/for/medical-billing) - [Insurance agencies](https://scalewithsearch.com/for/insurance-agencies) - [Financial advisors](https://scalewithsearch.com/for/financial-advisors) - [Property management companies](https://scalewithsearch.com/for/property-management) - [Recruiting and staffing agencies](https://scalewithsearch.com/for/recruiting-and-staffing) - [Architects and interior designers](https://scalewithsearch.com/for/architects-and-interior-designers) - [Logistics and supply chain companies](https://scalewithsearch.com/for/logistics-and-supply-chain) - Agencies, MSPs, and manufacturing - [IT and managed service providers](https://scalewithsearch.com/for/it-and-managed-service-providers) - [Machine shops and precision manufacturers](https://scalewithsearch.com/for/machine-shops-and-precision-manufacturing) - [Marketing agencies and freelancers](https://scalewithsearch.com/for/marketing-agencies-and-freelancers) - [SEO agencies and consultants](https://scalewithsearch.com/for/seo-agencies-and-consultants) - [Small manufacturers and fabricators](https://scalewithsearch.com/for/small-manufacturers-and-fabricators) - Restaurants, shops, studios, and nonprofits - [Restaurants and hospitality businesses](https://scalewithsearch.com/for/restaurants-and-hospitality) - [Retail stores and ecommerce sellers](https://scalewithsearch.com/for/retail-and-ecommerce) - [Photographers, event planners, and travel agents](https://scalewithsearch.com/for/photographers) - [Churches and nonprofits](https://scalewithsearch.com/for/churches-and-nonprofits) - [All industries](https://scalewithsearch.com/for/) - Learn - For your office - [Office job guides](https://scalewithsearch.com/guides/) - [Browser calculators](https://scalewithsearch.com/tools/) - Start here - [How it works](https://scalewithsearch.com/how-it-works) - [Free Starter Kit](https://scalewithsearch.com/kit/business-memory-starter-kit.zip) - [Synthetic specimen](https://scalewithsearch.com/specimen/working-session-specimen.zip) - Guides - [The Complete Guide to Business Memory for AI Agents](https://scalewithsearch.com/articles/business-memory-for-ai-agents-guide) - [The Complete Small-Business Guide to AI Agent Governance](https://scalewithsearch.com/articles/ai-agent-governance-guide-small-business) - [The Complete Guide to Leaving Vendor AI Memory](https://scalewithsearch.com/articles/leaving-vendor-ai-memory-guide) - Articles by cluster - [Business memory](https://scalewithsearch.com/articles/business-memory-for-ai-agents-guide) - [Agent governance](https://scalewithsearch.com/articles/ai-agent-governance-guide-small-business) - [Migration and ownership](https://scalewithsearch.com/articles/leaving-vendor-ai-memory-guide) - For machines - [llms.txt](https://scalewithsearch.com/llms.txt) - [llms-full.txt](https://scalewithsearch.com/llms-full.txt) - [Machine view](https://scalewithsearch.com/?view=machine) - Company - Evidence - [Proof](https://scalewithsearch.com/proof) - Company - [About](https://scalewithsearch.com/about) # Twenty Questions Before You Buy an AI Agent System. The demo answers perfectly. Two weeks later, nobody can explain which sources it used, stop a customer send, reproduce a failure, or export the system. The buyer purchased a performance. The operating evidence was missing. Before you buy an AI agent system, ask twenty questions about the job, authority, testing, ownership, and exit. Require a file, account view, receipt, or live test for every material answer. ## Buy operating evidence, not the demo A vendor controls the demo data, path, account, and timing. Your business will supply incomplete inputs, conflicting records, expired approvals, retries, provider outages, and people who change roles. The procurement test should show how the system behaves outside the happy path. Require artifacts beyond security too. Ask the vendor to prove the job, source rules, stopping behavior, recovery, custody, and exit. ## Five questions about the job and sources ### 1. What exact business result does the agent own? Require a one-sentence job and a sample accepted output. "Automate operations" is not bounded enough to test. ### 2. Which sources may it read? Ask for the source manifest, identity resolution, and excluded paths. Test what happens when a required source is missing. ### 3. Which source wins when records conflict? Require a source-order rule. Present two equal-authority current records and confirm the agent stops. ### 4. What data does it retain after a run? Ask for memory classes, storage locations, allowed readers, retention triggers, and deletion proof. Include model-side memory and derived indexes. ### 5. How are accepted corrections applied? Require a correction log, canonical rule update, regression fixture, and retest receipt. A note in one vendor chat is not a system correction. These questions establish the task brief. The guide to [what a build acceptance list contains](/articles/what-a-build-acceptance-list-contains) shows how the buyer turns answers into release criteria. ## Five questions about authority and stopping ### 6. Which external effects can the system create? List sends, posts, bookings, payments, deletions, deployments, access changes, and live-record writes. Inspect the tools and credentials, not only the prompt. ### 7. Who approves each consequence class? Require a role-based matrix that binds approval to one account, destination, artifact, amount, and expiry. ### 8. What actions are prohibited? Ask for deterministic blocks. A prohibited secret export should fail before it creates an approval card. ### 9. When does the agent stop normally or early? Require named terminal states for success, no work, blocked, failed, and uncertain. Ask which owner receives each escalation. ### 10. How does the executor prevent approval reuse? Test an expired approval, changed recipient, changed body, wrong role, and replayed approval. Each should fail closed. The operating pattern in [who approves what an AI agent sends](/articles/who-approves-what-an-ai-agent-sends) keeps relationship authority with the person who owns the consequence. OpenAI's prompt-injection guidance recommends constrained access and confirmation for consequential actions. [OpenAI: Understanding prompt injections](https://openai.com/safety/prompt-injections/) ## Five questions about tests and failures ### 11. What acceptance fixtures ship with the system? Require normal, boundary, prohibited, conflict, missing-source, and recovery cases. The buyer should own copies. ### 12. How are duplicate effects prevented? Ask for idempotency keys, approval consumption, provider read-back, and a timeout test. Run a double-click scenario. ### 13. What receipt does every terminal state produce? Inspect real success, no-work, blocked, failed, and uncertain receipts. Each should name inputs, sources, actions, and evidence. ### 14. How does rollback work? Require the last known good version, data-recovery rule, off switch, owner, maximum recovery time, and a completed drill. ### 15. Which reliability measures can the buyer recompute? Ask for numerator, denominator, evidence links, thresholds, and severe incidents. Vendor dashboard percentages without source receipts are weak proof. NIST's voluntary AI Risk Management Framework emphasizes documented roles, measurement, monitoring, and management of AI risks. [NIST: AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework) The vendor's process can vary. The buyer should still receive inspectable evidence. ## Five questions about ownership and exit ### 16. Which accounts, files, data, and domains does the buyer own? Open the account settings through buyer-controlled identities. A contract ownership clause does not prove administrative control. For a repository handoff that proves the buyer’s access, read [AI System Handoff: Transfer the Repo and Prove Access.](/articles/transfer-the-repo-client-handoff) ### 17. Can the buyer change and deploy one rule? Require a controlled change, test, deployment, observation, and rollback performed by the buyer. ### 18. What does the system cost at current and higher volume? Ask for provider, infrastructure, support, monitoring, storage, and model costs. Require usage assumptions and thresholds. ### 19. What support and correction obligations continue after acceptance? Name response channels, covered defects, excluded changes, owners, time periods, and the process for a new outcome. ### 20. Can the buyer run the system after the vendor leaves? Require an export, runbook, credential rotation, independent operating period, and named non-transferable dependencies. This is the proof standard behind [AI system ownership and transfer](/articles/ai-system-ownership-transfer-checklist). Ownership means the buyer can operate, change, stop, and restore the result. ## Inspect `ai-agent-vendor-scorecard.csv` Score evidence, not presentation quality: ```csv question,evidence_required,evidence_received,live_test,owner,result,blocker exact_job,task brief,brief-v3.md,run accepted fixture,workflow_owner,pass, source_conflict,source order,manifest.md,equal-authority conflict,record_owner,fail,agent chose newest file approval_replay,approval schema,approval.yaml,replay consumed approval,system_owner,pass, duplicate_effect,idempotency design,retry-spec.md,timeout after provider write,system_owner,pending,test not scheduled buyer_control,account register,ownership.csv,buyer deploy and rollback,business_owner,fail,vendor owns cloud account exit,export and runbook,handoff.zip,independent run,business_owner,pending,credential rotation missing ``` Use `pass`, `fail`, `pending`, and `not_applicable`. A polished answer without the required artifact remains `pending` or `fail` under the buyer's rule. Set hard stops before the demo. Examples include vendor-owned production accounts, no tested rollback, unresolved data rights, unbounded external action, or no usable export. ## Watch for procurement red flags Be cautious when the answer relies on: - adjectives such as secure, enterprise-ready, autonomous, or accurate; - screenshots with no buyer access; - certifications offered instead of workflow evidence; - one success video with no failure receipts; - a proprietary dashboard with no export; - human review with no named approver or preview; - "we can add that later" for stop, rollback, or ownership; - verbal promises that do not enter the scope and acceptance list. The strongest vendor should welcome a bounded test because it protects both sides from an undefined result. For comparing a scoped build with an agency engagement, read [Scale With Search vs an AI Agency: Compare the Scope.](/articles/scale-with-search-vs-ai-agency) ## Run one buyer-owned acceptance scenario Prepare a synthetic fixture using your own job shape. Include a current source, a stale conflicting source, an untrusted hidden instruction, a missing approval, and a simulated timeout after an external write. Test these results: 1. The agent reads only allowed sources. 2. The current authority defeats the stale record. 3. Equal authority causes a stop. 4. The hidden instruction is treated as data. 5. Missing approval blocks the external effect. 6. Changing the destination invalidates approval. 7. The timeout triggers read-back before retry. 8. One approved intent creates one effect. 9. The receipt records sources, decision, approval, and result. 10. The buyer can find and correct the governing rule. 11. The buyer can stop and restore the test system. 12. The vendor exports the fixture, configuration, and receipts. Do not let the vendor replace the fixture with its standard demo. The point is to see your exception pattern inside a safe test. ## Make the purchase decision explicit Record one of four decisions: buy, buy with named conditions, do not buy, or gather missing evidence. For the person behind Scale With Search and its ownership approach, read [Victor Valentine Romo: Owned AI Systems.](/about) For a conditional purchase, put conditions in the signed scope with owners and acceptance dates. A roadmap promise is not a delivered control. Compare the operating burden with the manual process. A technically impressive system can still be a poor purchase when the task is rare, review costs are high, or the buyer cannot maintain it. ## Approval and stopping boundary An evaluation agent may collect artifacts, run approved synthetic fixtures, populate the scorecard, identify conflicts, and draft a purchase recommendation. It stops before signing, paying, granting production access, uploading live client data, accepting contract terms, or activating the system. Named business, data, security, and budget owners make those decisions as applicable. The evaluation also stops when the vendor cannot provide required evidence, the buyer's account identity is unresolved, or a hard-stop condition fails. It reports the failed proof instead of substituting a lower standard after the demo. ## Sources - [NIST: AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework) - [OpenAI: Understanding prompt injections](https://openai.com/safety/prompt-injections/) ## Questions about Twenty Questions Before You Buy an AI Agent System ### What would you ask from a vendor using AI agents with tool access? Ask for evidence of the exact job, allowed sources, authority limits, stopping states, duplicate prevention, rollback, reliability, ownership, and exit path. Require files, account views, receipts, or live tests instead of accepting a polished demonstration or verbal assurance. ### What tools can the agent call? Inspect the actual tools and credentials, then list every send, post, booking, payment, deletion, deployment, access change, and live-record write the system can create. The buyer should also see which actions require approval and which are prohibited by deterministic controls. ### What happens when the AI isn't confident? The system should use named terminal states for success, no work, blocked, failed, and uncertain. Each state needs a receipt that identifies inputs, sources, actions, evidence, and the owner who receives the escalation. ## Related: Agent Governance - [Five AI Agent Reliability Metrics a Small Team Can Verify](/articles/ai-agent-reliability-metrics-small-business) - [What an AI Agent Rollback Plan Must Contain Before Production Access](/articles/ai-agent-rollback-plan) - [How to Test an AI Agent Before It Touches Production](/articles/test-an-ai-agent-before-production) ---- ```text .|########||. .|########||. .|########||. |##||. .||##|. |##||. .||##|. |##||. .||##|. |#|. .|#|. |#|. .|#|. |#|. .|#|. |#| |#| |#| |#| |#| |#| .#| |#. .#| |#. .#| |#. .#. .#| .#. .#| .#. .#| |#. .#| |#. .#| |#. .#| |# ...... #| |# ...... #| |# ...... #| .# ||#########| #| .# ||#########| #| .# ||#########| #| .#. |######||######|. .#. .#. |######||######|. .#. .#. |######||######|. .#. #. .##|###|##|#|######| .# #. .##|###|##|#|######| .# #. .##|###|##|#|######| .# || |##|#||||||||||||#||#| || || |##|#||||||||||||#||#| || || |##|#||||||||||||#||#| || # |#||||||||||||||||||||#| #. # |#||||||||||||||||||||#| #. # |#||||||||||||||||||||#| #. || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || # .#||||||||||||||||||||||||#| # # .#||||||||||||||||||||||||#| # # .#||||||||||||||||||||||||#| # || ....|||#||||||##|#|||#|#||##||||....|. || || ....|||#||||||##|#|||#|#||##||||....|. || || ....|||#||||||##|#|||#|#||##||||....|. || #. . ....|# ....#||| ||| .#||| ....#. .# #. . ....|# ....#||| ||| .#||| ....#. .# #. . ....|# ....#||| ||| .#||| ....#. .# # . ||||#| .#####|| . .#| .#||| ||||# #. # . ||||#| .#####|| . .#| .#||| ||||# #. # . ||||#| .#####|| . .#| .#||| ||||# #. .| ||||| #. |#|||||. || #. |#||. .||||| || .| ||||| #. |#|||||. || #. |#||. .||||| || .| ||||| #. |#|||||. || #. |#||. .||||| || || |.... #. ....|#. |. ...|. ....|| || || |.... #. ....|#. |. ...|. ....|| || || |.... #. ....|#. |. ...|. ....|| || #. .||||||##||||||##||####|||||||#||||||#| .# #. .||||||##||||||##||####|||||||#||||||#| .# #. .||||||##||||||##||####|||||||#||||||#| .# # .||####|###########################|. # # .||####|###########################|. # # .||####|###########################|. # # .#||||||.#.|| # |. ..# #| #|||||#| # # .#||||||.#.|| # |. ..# #| #|||||#| # # .#||||||.#.|| # |. ..# #| #|||||#| # # .#|||||| .. |# ## |#| ...#|#|||#| # # .#|||||| .. |# ## |#| ...#|#|||#| # # .#|||||| .. |# ## |#| ...#|#|||#| # # .#|||#|# .# .#| #| ##|.#..#|||||#| # # .#|||#|# .# .#| #| ##|.#..#|||||#| # # .#|||#|# .# .#| #| ##|.#..#|||||#| # # .#||||||############|######|||||#| # # .#||||||############|######|||||#| # # .#||||||############|######|||||#| # # |...||#||||||#||||#|#||||||#|||||#||| |.| # # |...||#||||||#||||#|#||||||#|||||#||| |.| # # |...||#||||||#||||#|#||||||#|||||#||| |.| # #. .. ||||# .|||##|. |#| .|| || .|||#. #| # .# #. .. ||||# .|||##|. |#| .|| || .|||#. #| # .# #. .. ||||# .|||##|. |#| .|| || .|||#. #| # .# || | ...|| ...##| | #| .|. |. ##### .. .| || || | ...|| ...##| | #| .|. |. ##### .. .| || || | ...|| ...##| | #| .|. |. ##### .. .| || || ||||| || ||||#| . |. |. |#. ||||| |#| || || || ||||| || ||||#| . |. |. |#. ||||| |#| || || || ||||| || ||||#| . |. |. |#. ||||| |#| || || .# |.....#|....|#.||||.|||##.|#|....||.#||.#. #. .# |.....#|....|#.||||.|||##.|#|....||.#||.#. #. .# |.....#|....|#.||||.|||##.|#|....||.#||.#. #. #.|||||||#############################| ||| .# #.|||||||#############################| ||| .# #.|||||||#############################| ||| .# || ##||#||#||||||#||#||#||#||##. || || ##||#||#||||||#||#||#||#||##. || || ##||#||#||||||#||#||#||#||##. || # .#|||||||||#||#|||||||||||#| # # .#|||||||||#||#|||||||||||#| # # .#|||||||||#||#|||||||||||#| # || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || .# |#||||||||||||||||||||#| #. .# |#||||||||||||||||||||#| #. .# |#||||||||||||||||||||#| #. || |#||||||||||||||||||#| || || |#||||||||||||||||||#| || || |#||||||||||||||||||#| || #. .######|#|#########| .# #. .######|#|#########| .# #. .######|#|#########| .# .#. |#####||||#####| .#. .#. |#####||||#####| .#. .#. |#####||||#####| .#. |# ||########|| #. |# ||########|| #. |# ||########|| #. |# ...... #| |# ...... #| |# ...... #| |#. .#| |#. .#| |#. .#| |#. .#. |#. .#. |#. .#. .#| |#. .#| |#. .#| |#. |#| |#| |#| |#| |#| |#| .|#|. .|#|. .|#|. .|#|. .|#|. .|#|. |##||. .||##| |##||. .||##| |##||. .||##| .||########||. .||########||. .||########||. Scale With Search 2026 [scalewithsearch.com](https://scalewithsearch.com) ```