--- title: "What Client Data Belongs in AI Agent Memory?" description: "Use a minimum-necessary test to decide which client facts an AI agent may remember, which it may read once, and which it must reject." canonical: "https://scalewithsearch.com/articles/client-data-in-ai-agent-memory" date: "2026-08-20" modified: "2026-09-19" --- ## Site navigation - [Scale With Search](https://scalewithsearch.com/) - Real estate - Real estate - [Real estate](https://scalewithsearch.com/for/real-estate) - Work - Start here - [Send your brief](https://scalewithsearch.com/work#send-your-brief) - [Prepare your six-question brief](https://scalewithsearch.com/work#prepare-your-six-question-brief) - Build - [Site build, content library with SEO, signal desk](https://scalewithsearch.com/work) - For your business - Trades and home services - [Auto body and collision shops](https://scalewithsearch.com/for/auto-body-and-collision-shops) - [Foundation and home repair contractors](https://scalewithsearch.com/for/foundation-and-home-repair) - [Garage door and fencing contractors](https://scalewithsearch.com/for/garage-door-and-fencing-contractors) - [HVAC contractors](https://scalewithsearch.com/for/hvac-contractors) - [Janitorial and commercial cleaning companies](https://scalewithsearch.com/for/janitorial-and-commercial-cleaning) - [Locksmiths](https://scalewithsearch.com/for/locksmiths) - [Moving companies](https://scalewithsearch.com/for/moving-companies) - [Pest control companies](https://scalewithsearch.com/for/pest-control-companies) - [Plumbing and electrical contractors](https://scalewithsearch.com/for/plumbing-and-electrical-contractors) - [Restoration and water or fire damage companies](https://scalewithsearch.com/for/restoration-and-water-fire-damage) - [Roofing companies](https://scalewithsearch.com/for/roofing-companies) - [Towing companies](https://scalewithsearch.com/for/towing-companies) - [Tree services and landscaping companies](https://scalewithsearch.com/for/tree-services-and-landscaping) - [Solar installers](https://scalewithsearch.com/for/solar-installation) - [General contractors](https://scalewithsearch.com/for/general-contractors-and-construction) - [Paving, concrete, and flooring contractors](https://scalewithsearch.com/for/paving) - Practices and professional services - [Bookkeeping and tax practices](https://scalewithsearch.com/for/bookkeeping-and-tax-practices) - [Dental practices](https://scalewithsearch.com/for/dental-practices) - [Family and criminal defense law firms](https://scalewithsearch.com/for/family-and-criminal-defense-law-firms) - [Med spas and aesthetics practices](https://scalewithsearch.com/for/med-spas-and-aesthetics) - [Personal injury law firms](https://scalewithsearch.com/for/personal-injury-law-firms) - [Veterinary clinics](https://scalewithsearch.com/for/veterinary-clinics) - [Gyms and fitness studios](https://scalewithsearch.com/for/fitness) - [Therapy and outpatient health practices](https://scalewithsearch.com/for/therapy-and-outpatient-health) - [Medical billing companies](https://scalewithsearch.com/for/medical-billing) - [Insurance agencies](https://scalewithsearch.com/for/insurance-agencies) - [Financial advisors](https://scalewithsearch.com/for/financial-advisors) - [Property management companies](https://scalewithsearch.com/for/property-management) - [Recruiting and staffing agencies](https://scalewithsearch.com/for/recruiting-and-staffing) - [Architects and interior designers](https://scalewithsearch.com/for/architects-and-interior-designers) - [Logistics and supply chain companies](https://scalewithsearch.com/for/logistics-and-supply-chain) - Agencies, MSPs, and manufacturing - [IT and managed service providers](https://scalewithsearch.com/for/it-and-managed-service-providers) - [Machine shops and precision manufacturers](https://scalewithsearch.com/for/machine-shops-and-precision-manufacturing) - [Marketing agencies and freelancers](https://scalewithsearch.com/for/marketing-agencies-and-freelancers) - [SEO agencies and consultants](https://scalewithsearch.com/for/seo-agencies-and-consultants) - [Small manufacturers and fabricators](https://scalewithsearch.com/for/small-manufacturers-and-fabricators) - Restaurants, shops, studios, and nonprofits - [Restaurants and hospitality businesses](https://scalewithsearch.com/for/restaurants-and-hospitality) - [Retail stores and ecommerce sellers](https://scalewithsearch.com/for/retail-and-ecommerce) - [Photographers, event planners, and travel agents](https://scalewithsearch.com/for/photographers) - [Churches and nonprofits](https://scalewithsearch.com/for/churches-and-nonprofits) - [All industries](https://scalewithsearch.com/for/) - Learn - For your office - [Office job guides](https://scalewithsearch.com/guides/) - [Browser calculators](https://scalewithsearch.com/tools/) - Start here - [How it works](https://scalewithsearch.com/how-it-works) - [Free Starter Kit](https://scalewithsearch.com/kit/business-memory-starter-kit.zip) - [Synthetic specimen](https://scalewithsearch.com/specimen/working-session-specimen.zip) - Guides - [The Complete Guide to Business Memory for AI Agents](https://scalewithsearch.com/articles/business-memory-for-ai-agents-guide) - [The Complete Small-Business Guide to AI Agent Governance](https://scalewithsearch.com/articles/ai-agent-governance-guide-small-business) - [The Complete Guide to Leaving Vendor AI Memory](https://scalewithsearch.com/articles/leaving-vendor-ai-memory-guide) - Articles by cluster - [Business memory](https://scalewithsearch.com/articles/business-memory-for-ai-agents-guide) - [Agent governance](https://scalewithsearch.com/articles/ai-agent-governance-guide-small-business) - [Migration and ownership](https://scalewithsearch.com/articles/leaving-vendor-ai-memory-guide) - For machines - [llms.txt](https://scalewithsearch.com/llms.txt) - [llms-full.txt](https://scalewithsearch.com/llms-full.txt) - [Machine view](https://scalewithsearch.com/?view=machine) - Company - Evidence - [Proof](https://scalewithsearch.com/proof) - Company - [About](https://scalewithsearch.com/about) # What Client Data Belongs in AI Agent Memory. An agent reads a client email that includes bank details. It stores the number in persistent memory because the message was part of the account thread. The recurring job only prepares a weekly project update. The agent could read the message. It had no reason to remember the bank detail. Client memory should contain the minimum information required for a named job. Access to a record does not create permission to retain it, reuse it, infer new traits from it, or expose it to another workflow. ## Memory eligibility starts with the job Write the job before choosing the data. "Help with Client A" is too broad. "Prepare a weekly delivery brief from approved project records, without sending it" gives you a purpose, source boundary, output, and stopping point. For applying the same source boundary to personnel policy work, read [Claude vs ChatGPT for HR: Test Policy Work.](/articles/claude-vs-chatgpt-hr-policy-work) For each proposed field, ask whether the job fails without persistent access to that field. A current project status may qualify. A bank number, private health detail, credential, or unrelated personal comment usually does not. Stored material can influence a later run. Treat retrieved memory as untrusted input until its source and role are checked. [OWASP: Prompt Injection](https://genai.owasp.org/llmrisk/llm01-prompt-injection/) The practical response is a narrow admission gate. Do not send every available record into memory and rely on the model to ignore what it does not need. ## Use the five-question minimum-necessary test A field enters persistent client memory only when all five answers are clear. 1. **Purpose:** Which recurring job requires it? 2. **Authority:** Which source makes it current and reliable? 3. **Scope:** Which client, project, role, and agent may use it? 4. **Sensitivity:** What harm follows from misuse or disclosure? 5. **Lifecycle:** When does it expire, enter review, or get deleted? If the purpose is vague, keep the field out. If the authority is missing, route it to review. If scope cannot be enforced, do not persist it. If sensitivity exceeds the workflow's controls, use a safer process. The [context manifest](/articles/what-context-should-an-agent-read) should point to approved records. It should not become a shortcut around the eligibility decision. ## Build `client-memory-eligibility.csv` Record one decision per field or record class: ```csv field,purpose,source,classification,scope,allowed_readers,retention,decision,reason project_status,weekly brief,project-status.md,confidential,client-017,brief-agent;account-owner,account-close+12m,remember,required current fact billing_contact,invoice routing,crm contact,confidential,client-017,billing-agent;account-owner,contract+24m,remember,required by separate billing job bank_account,none,email attachment,restricted,client-017,finance-owner,none,reject,no approved agent job meeting_transcript,extract decisions,approved transcript,confidential,client-017,review-agent,decision-extracted+30d,read_once,raw text not needed later personality_guess,none,model inference,sensitive,client-017,none,none,reject,unsupported inferred trait ``` `read_once` is a useful state. It permits a bounded task to inspect a record without copying it into persistent memory. The run receipt records access and any approved output. Do not collapse classification and decision. A confidential field might still be necessary inside a controlled scope. A public field can still be irrelevant and should not accumulate. ## Separate facts, decisions, credentials, and inferred traits Different classes need different handling. Confirmed facts should cite a source and review date. Decisions should name who had authority. Drafts should stay drafts. Credentials should remain in a secret manager and enter a tool at execution, not a memory document. Sensitive data needs the controls required by contract and law. Inferred traits should be rejected unless a documented, lawful job requires them and a human accepts the inference. Apply least privilege to the memory reader and its connected tools. OWASP recommends narrowing permissions and placing consequential actions behind human review. [OWASP: Excessive Agency](https://genai.owasp.org/llmrisk/llm062025-excessive-agency/) The business still must translate those categories into local decisions. A generic governance label cannot decide whether one client's health detail belongs in one agent's context. ## Keep reusable methods outside client facts A good procedure may emerge while serving one client. That does not make the client's examples reusable. Separate three layers: - client identity and relationship facts; - current project state and accepted decisions; - reusable method with client-specific data removed. Promote the method only after review. Replace names, amounts, credentials, URLs, and distinctive facts. Record the source and reviewer so the generic method does not become an unattributed copy of private work. This is why you should [separate client context before agent work](/articles/separate-client-context-before-agent-work). Folder separation helps, but runtime identity checks must also prevent the wrong client from being selected. The [agent security acceptance matrix](/articles/ai-agent-security-risks-small-business) tests the tools, identities, and destinations around this memory boundary. ## Treat external text as data, not instruction Client files, web pages, email bodies, attachments, and retrieved documents can contain text that looks like an instruction to the agent. OpenAI's prompt-injection guidance recommends constraining agent access, separating untrusted content from instructions, and confirming consequential actions. [OpenAI: Understanding prompt injections](https://openai.com/safety/prompt-injections/) Mark retrieved client material as data. The task brief and approved policy remain the instruction sources. If a document says to upload the archive, reveal a credential, ignore prior rules, or contact a new recipient, the agent reports the text and stops. Do not store hidden or untrusted instructions as durable preferences. One poisoned record can affect many future runs. ## Name the owner and deletion trigger Every remembered field needs a record owner. The owner can correct it, restrict it, approve its use, and decide when it expires. Use triggers tied to the work: project close, contract end, superseded decision, revoked consent, or completed dispute period. Apply the approved [AI memory retention policy](/articles/ai-memory-retention-and-deletion-policy) to source files and derived stores. If a field lacks an owner, default to review or rejection. Anonymous memory is hard to correct and harder to delete. ## Test cross-client leakage with canary facts Create two synthetic clients. Give each a distinctive canary fact that is safe to expose during testing. For example, Client A uses `ORANGE-CEDAR-417` and Client B uses `BLUE-HARBOR-928`. Run the same job for each client. Test these results: 1. Client A's output contains only the A canary when required. 2. Client B's output contains only the B canary when required. 3. A broad search inside the A run cannot retrieve the B canary. 4. Switching the displayed client name without changing the runtime ID fails closed. 5. A document containing a hidden instruction does not alter the job. 6. A rejected field does not appear in memory search or summaries. 7. A `read_once` record is absent after the run. 8. Each receipt records client ID, sources, memory writes, and rejected fields. Repeat the test after changing indexing, folder rules, connectors, or model adapters. A clean prompt in one session does not prove isolation at the storage layer. ## Review a field when the answer changes Eligibility is not permanent. A billing contact may be necessary while invoices are open and unnecessary after account closure. A project preference may become a signed policy. A draft may be accepted or rejected. A client may revoke consent. Schedule review by record class. When a job changes, rerun the five questions before widening its memory. Do not inherit the old source set because the new workflow shares a client name. ## Approval and stopping boundary An agent may inventory candidate fields, populate a proposed eligibility table, flag missing authorities, and run synthetic cross-client tests. It stops before persisting a new sensitive class, widening allowed readers, importing credentials, using inferred traits, changing a retention rule, or transferring data to a new provider. The client data owner and system owner must decide those changes. The executor also stops when the client identity, purpose, source authority, or deletion trigger is unresolved. It must not use urgency as permission to remember more data. ## Sources - [OWASP: Prompt Injection](https://genai.owasp.org/llmrisk/llm01-prompt-injection/) - [OWASP: Excessive Agency](https://genai.owasp.org/llmrisk/llm062025-excessive-agency/) - [OpenAI: Understanding prompt injections](https://openai.com/safety/prompt-injections/) ## Questions about What Client Data Belongs in AI Agent Memory? ### What should the agent never see? Exclude records that are not necessary for the named job, especially secrets, unrelated client facts, unsupported inferred traits, and material outside the authorized source boundary. Record the exclusion and test it with cross-client canary facts. ### In what ways do you safeguard personal or sensitive information during interactions? Apply a minimum-necessary test to each field, separate facts and decisions from credentials and inferred traits, and keep client identity attached to every eligible record. Use distinct source, credential, recipient, output, and deletion boundaries, then test for cross-client leakage. ### How to stop people stealing your agent data and files? Do not place unnecessary client data or credentials in agent memory, and do not rely on model instructions as the access boundary. Limit the allowed sources and tools, keep reusable methods separate from client facts, and verify denied access with synthetic records. ## Related: Owned Memory - [AI Memory Retention and Deletion Policy for a Small Business](/articles/ai-memory-retention-and-deletion-policy) - [AI Agent Security Risks for a Small Business: Test Access, Actions, and Evidence](/articles/ai-agent-security-risks-small-business) - [Choose local or cloud AI memory by who controls the data](/articles/is-ai-memory-safe) ---- ```text .|########||. .|########||. .|########||. |##||. .||##|. |##||. .||##|. |##||. .||##|. |#|. .|#|. |#|. .|#|. |#|. .|#|. |#| |#| |#| |#| |#| |#| .#| |#. .#| |#. .#| |#. .#. .#| .#. .#| .#. .#| |#. .#| |#. .#| |#. .#| |# ...... #| |# ...... #| |# ...... #| .# ||#########| #| .# ||#########| #| .# ||#########| #| .#. |######||######|. .#. .#. |######||######|. .#. .#. |######||######|. .#. #. .##|###|##|#|######| .# #. .##|###|##|#|######| .# #. .##|###|##|#|######| .# || |##|#||||||||||||#||#| || || |##|#||||||||||||#||#| || || |##|#||||||||||||#||#| || # |#||||||||||||||||||||#| #. # |#||||||||||||||||||||#| #. # |#||||||||||||||||||||#| #. || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || # .#||||||||||||||||||||||||#| # # .#||||||||||||||||||||||||#| # # .#||||||||||||||||||||||||#| # || ....|||#||||||##|#|||#|#||##||||....|. || || ....|||#||||||##|#|||#|#||##||||....|. || || ....|||#||||||##|#|||#|#||##||||....|. || #. . ....|# ....#||| ||| .#||| ....#. .# #. . ....|# ....#||| ||| .#||| ....#. .# #. . ....|# ....#||| ||| .#||| ....#. .# # . ||||#| .#####|| . .#| .#||| ||||# #. # . ||||#| .#####|| . .#| .#||| ||||# #. # . ||||#| .#####|| . .#| .#||| ||||# #. .| ||||| #. |#|||||. || #. |#||. .||||| || .| ||||| #. |#|||||. || #. |#||. .||||| || .| ||||| #. |#|||||. || #. |#||. .||||| || || |.... #. ....|#. |. ...|. ....|| || || |.... #. ....|#. |. ...|. ....|| || || |.... #. ....|#. |. ...|. ....|| || #. .||||||##||||||##||####|||||||#||||||#| .# #. .||||||##||||||##||####|||||||#||||||#| .# #. .||||||##||||||##||####|||||||#||||||#| .# # .||####|###########################|. # # .||####|###########################|. # # .||####|###########################|. # # .#||||||.#.|| # |. ..# #| #|||||#| # # .#||||||.#.|| # |. ..# #| #|||||#| # # .#||||||.#.|| # |. ..# #| #|||||#| # # .#|||||| .. |# ## |#| ...#|#|||#| # # .#|||||| .. |# ## |#| ...#|#|||#| # # .#|||||| .. |# ## |#| ...#|#|||#| # # .#|||#|# .# .#| #| ##|.#..#|||||#| # # .#|||#|# .# .#| #| ##|.#..#|||||#| # # .#|||#|# .# .#| #| ##|.#..#|||||#| # # .#||||||############|######|||||#| # # .#||||||############|######|||||#| # # .#||||||############|######|||||#| # # |...||#||||||#||||#|#||||||#|||||#||| |.| # # |...||#||||||#||||#|#||||||#|||||#||| |.| # # |...||#||||||#||||#|#||||||#|||||#||| |.| # #. .. ||||# .|||##|. |#| .|| || .|||#. #| # .# #. .. ||||# .|||##|. |#| .|| || .|||#. #| # .# #. .. ||||# .|||##|. |#| .|| || .|||#. #| # .# || | ...|| ...##| | #| .|. |. ##### .. .| || || | ...|| ...##| | #| .|. |. ##### .. .| || || | ...|| ...##| | #| .|. |. ##### .. .| || || ||||| || ||||#| . |. |. |#. ||||| |#| || || || ||||| || ||||#| . |. |. |#. ||||| |#| || || || ||||| || ||||#| . |. |. |#. ||||| |#| || || .# |.....#|....|#.||||.|||##.|#|....||.#||.#. #. .# |.....#|....|#.||||.|||##.|#|....||.#||.#. #. .# |.....#|....|#.||||.|||##.|#|....||.#||.#. #. #.|||||||#############################| ||| .# #.|||||||#############################| ||| .# #.|||||||#############################| ||| .# || ##||#||#||||||#||#||#||#||##. || || ##||#||#||||||#||#||#||#||##. || || ##||#||#||||||#||#||#||#||##. || # .#|||||||||#||#|||||||||||#| # # .#|||||||||#||#|||||||||||#| # # .#|||||||||#||#|||||||||||#| # || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || .# |#||||||||||||||||||||#| #. .# |#||||||||||||||||||||#| #. .# |#||||||||||||||||||||#| #. || |#||||||||||||||||||#| || || |#||||||||||||||||||#| || || |#||||||||||||||||||#| || #. .######|#|#########| .# #. .######|#|#########| .# #. .######|#|#########| .# .#. |#####||||#####| .#. .#. |#####||||#####| .#. .#. |#####||||#####| .#. |# ||########|| #. |# ||########|| #. |# ||########|| #. |# ...... #| |# ...... #| |# ...... #| |#. .#| |#. .#| |#. .#| |#. .#. |#. .#. |#. .#. .#| |#. .#| |#. .#| |#. |#| |#| |#| |#| |#| |#| .|#|. .|#|. .|#|. .|#|. .|#|. .|#|. |##||. .||##| |##||. .||##| |##||. .||##| .||########||. .||########||. .||########||. Scale With Search 2026 [scalewithsearch.com](https://scalewithsearch.com) ```