--- title: "Check Cloudflare crawler access on an agent website" description: "Trace the exact URL, crawler identity, Cloudflare rule, and returned content before approving a narrow access repair. Keep the evidence dated." canonical: "https://scalewithsearch.com/articles/cloudflare-ai-crawler-access-agent-websites" --- ## Site navigation - [Scale With Search](https://scalewithsearch.com/) - Services - Services - [AI systems](https://scalewithsearch.com/ai-systems) - [Site build](https://scalewithsearch.com/work#site-build) - [Whole website fix](https://scalewithsearch.com/work#whole-website-fix) - [Content library with SEO](https://scalewithsearch.com/work#content-library) - [Working session](https://scalewithsearch.com/audit) - Start here - [Send your brief](https://scalewithsearch.com/work#send-your-brief) - [Example scope](https://scalewithsearch.com/work#review-the-example-scope) - [Prices and terms](https://scalewithsearch.com/work#prices-and-terms) - Industries - Real estate - [Real estate agents and teams](https://scalewithsearch.com/for/real-estate) - [Platforms, ownership, and IDX](https://scalewithsearch.com/for/real-estate/real-estate-website-platforms-ownership-idx) - [Search, content, and AI](https://scalewithsearch.com/for/real-estate/real-estate-search-content-ai) - [Inquiries, campaigns, and handoffs](https://scalewithsearch.com/for/real-estate/real-estate-website-inquiries-campaigns-handoffs) - [Buyer, seller, and local pages](https://scalewithsearch.com/for/real-estate/real-estate-buyer-seller-local-resources) - [Rules, privacy, and trust](https://scalewithsearch.com/for/real-estate/real-estate-website-rules-privacy-trust) - Browse by business - [Trades and home services](https://scalewithsearch.com/for/#group-trades) - [Health, wellness, and animal care](https://scalewithsearch.com/for/#group-practices) - [Law, finance, and IT](https://scalewithsearch.com/for/#group-professional) - [Marketing and SEO agencies](https://scalewithsearch.com/for/#group-agencies) - [Shops and manufacturers](https://scalewithsearch.com/for/#group-manufacturing) - [Restaurants, shops, studios, and nonprofits](https://scalewithsearch.com/for/#group-local-services) - Chiropractic practices - [Chiropractic and wellness practices](https://scalewithsearch.com/for/chiropractic-and-wellness-practices) - [Chiropractic and physical therapy clinics](https://scalewithsearch.com/for/chiropractic-and-physical-therapy-clinics) - [All industries](https://scalewithsearch.com/for/) - Resources - Free tools - [AI readiness check](https://scalewithsearch.com/check) - [Free tools](https://scalewithsearch.com/tools/) - [Free Starter Kit](https://scalewithsearch.com/kit/business-memory-starter-kit.zip) - Guides - [AI-native library](https://scalewithsearch.com/ai-native) - [Industry intelligence](https://scalewithsearch.com/intelligence) - [Office job guides](https://scalewithsearch.com/guides/) - [Business memory](https://scalewithsearch.com/articles/business-memory-for-ai-agents-guide) - [Agent governance](https://scalewithsearch.com/articles/ai-agent-governance-guide-small-business) - [Migration and ownership](https://scalewithsearch.com/articles/leaving-vendor-ai-memory-guide) - Articles - [All articles](https://scalewithsearch.com/articles/) - [What changed](https://scalewithsearch.com/changes/) - About - Evidence - [Proof](https://scalewithsearch.com/proof) - Scale With Search - [About](https://scalewithsearch.com/about) - [How it works](https://scalewithsearch.com/how-it-works) # Diagnose Cloudflare AI crawler access for a real estate website Trace the exact request from crawler identity through the matching security rule to the returned content. An Allow label in AI Crawl Control does not establish successful access. Cloudflare documents that other WAF rules can still block the request. [Cloudflare's rule interaction guide](https://developers.cloudflare.com/ai-crawl-control/configuration/ai-crawl-control-with-waf/), fetched 2026.10.22. Ask the website manager for evidence before changing settings. The procedure below is a proposed diagnostic workflow for a real estate website, not permission to disable protection. ## Which URL should the website manager investigate? Start with the exact public page that should answer the buyer's or seller's question. Record its hostname, path, and final destination. Include a property page separately if it uses a different provider. This recommended scope prevents a successful homepage test from becoming an unsupported claim about every resource on the website. ## Which AI crawler should be named in the brief? Name the operator, crawler, and intended purpose. Cloudflare distinguishes search, training, and user-directed automated behavior in its [bot classifications](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/), fetched 2026.10.22. Translate any vendor category into plain language for the approval record. A decision about search discovery should identify the actual crawler it concerns. ## Is a crawler's `User-Agent` string enough to verify identity? Require the documented verification method as well as the claimed name. Cloudflare describes Web Bot Auth and IP validation as verification methods. [Cloudflare's verified-bot explanation](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/), fetched 2026.10.22. Save which method supported the classification. Keep requests identified only by their supplied name labeled as self-identified. ## Does the account plan affect confidence in crawler labels? Yes. Cloudflare says its free-plan AI Crawl Control detection uses `User-Agent` strings, with more thorough detection available on upgraded plans. [Cloudflare's detection-quality note](https://developers.cloudflare.com/ai-crawl-control/features/manage-ai-crawlers/), fetched 2026.10.22. Ask which evidence the actual account exposes. Do not report a self-identification label as independently verified merely because it appears in a dashboard. ## What does Allow in AI Crawl Control establish? It establishes the selected control's setting. Cloudflare warns that other security rules can still block an allowed crawler. [Cloudflare's troubleshooting guidance](https://developers.cloudflare.com/ai-crawl-control/configuration/ai-crawl-control-with-waf/), fetched 2026.10.22. Record the setting and the actual request outcome separately. Ask for the response evidence before marking an access problem resolved. ## Where should a blocked request be investigated? Review the relevant security event and rule for the same request and time. Cloudflare documents that upstream WAF blocks may not appear in AI Crawl Control analytics. [WAF interaction limits](https://developers.cloudflare.com/ai-crawl-control/configuration/ai-crawl-control-with-waf/), fetched 2026.10.22. Keep the rule identifier with the affected URL so the proposed change has a specific cause. ## What should the request evidence contain? Save the time, hostname, path, claimed crawler, available identity evidence, response status, action, and matching rule. Add the returned content when available. This proposed record gives the reviewer a reproducible failure. If a field is unavailable on the account, mark it UNRESOLVED rather than substituting a nearby request that happens to look similar. ## Should robots.txt and firewall settings be checked separately? Yes. Cloudflare offers crawler controls and options involving robots.txt enforcement. [Cloudflare's management guidance](https://developers.cloudflare.com/ai-crawl-control/features/manage-ai-crawlers/), fetched 2026.10.22. Record both the public directive and the enforced response. A compliant directive does not explain an unrelated firewall block, and a permissive firewall setting does not describe the crawler's stated policy choice. ## Can an old Block AI Bots tutorial be trusted without checking? Check current documentation before following it. Cloudflare's [AI bot policy page](https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/), fetched 2026.10.22, documents revised behavior-based controls and legacy-setting changes. Ask the manager to record the controls present on the actual account. Keep documented defaults separate from observed configuration on an existing domain. ## Does allowing search access automatically settle training access? Record each intended behavior separately. Cloudflare documents distinct behavior policies and also describes mixed-purpose crawlers. [Cloudflare's policy definitions](https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/), fetched 2026.10.22. A real estate agent should approve the intended outcome before a rule change. Do not infer that every crawler from an operator performs the same task. ## Does a challenge response count as useful page access? Inspect what the request received. In the proposed acceptance check, a challenge page is a different result from the approved service explanation. Save its headers and body where available. Ask the manager to identify the security action responsible. A response code alone should not close the investigation when the intended text is missing. ## How should redirects be checked? Follow the request to its final resource and inspect the evidence for each relevant hostname. This is a proposed troubleshooting step. A redirect to a provider-hosted page may leave the original site's controls behind. Keep the starting URL and final URL together so an access repair on the wrong domain is not accepted. ## What if access succeeds but important text is missing? Move to a content-delivery investigation. Save the successful response and identify the missing passage. Use the [initial HTML and rendering comparison](/articles/ai-crawlers-javascript-idx-agent-websites) to scope that separate defect. A firewall change cannot be accepted as a repair for text that the server never included in the response received by the test. ## Can analytics establish that the intended body was delivered? Use analytics to locate request patterns, then inspect the resource. Cloudflare reports allowed and unsuccessful request metrics in its [analytics documentation](https://developers.cloudflare.com/ai-crawl-control/features/analyze-ai-traffic/), fetched 2026.10.22. As a proposed acceptance step, compare the returned body with approved page text. A request total does not substitute for that comparison. ## What if no attributable crawler request is available? Mark observed crawler access UNRESOLVED. Keep any local fetch labeled as a simulation and retain the configuration review separately. State the log window and evidence that were available. This proposed reporting rule prevents an absent record from becoming a claim that a crawler was blocked, never visited, or successfully retrieved the page. ## What should a proposed repair specify? Specify the affected hostname and path, intended crawler, identity match, blocking rule, proposed change, and rollback. Preserve the original setting in the review record. These are recommended approval fields. Keep unrelated protection outside the change. If the developer cannot identify the rule causing the failure, continue diagnosis before requesting a broad exception. ## How should the repair be checked afterward? Repeat the same URL and request-type checks after the approved change. Confirm the intended content and preserve the resulting event evidence. Label a simulation clearly if an attributable crawler visit remains unavailable. Keep a browser check for public readability in the release plan, with separate acceptance for the buyer's and seller's contact destinations. ## Does successful crawler access guarantee an AI citation? No. Google says eligibility does not guarantee inclusion in its Search features. [Google's AI feature guidance](https://developers.google.com/search/docs/appearance/ai-features), fetched 2026.10.22. Other service outcomes require their own evidence. Close the access defect based on access evidence. Track citations and received inquiries as separate observations after the repair. ## Should pay-per-crawl be enabled to diagnose access? Keep a paid-access decision outside this diagnostic task. Cloudflare documents pay-per-crawl as a separate feature interacting with WAF controls. [Cloudflare's configuration explanation](https://developers.cloudflare.com/ai-crawl-control/configuration/ai-crawl-control-with-waf/), fetched 2026.10.22. Record any existing paid-access response as part of the failure evidence. Do not introduce a new commercial arrangement merely to complete an access check. ## Bring this work into your website scope Bring the approved brief, source records, and unresolved checks to the [real estate hub](/for/real-estate). Describe the pages and acceptance checks in your [written work request](/work). - [Make your housing dataset discoverable from your real estate website](/articles/housing-dataset-schema-discovery-agent-reports) - [Decide whether your real estate website needs Markdown](/articles/markdown-html-agent-pages-ai-retrieval) - [Can AI search read the public PDFs on a real estate website](/articles/public-real-estate-pdfs-in-ai-search) ## Related: Real estate - [Make your housing dataset discoverable from your real estate website](/articles/housing-dataset-schema-discovery-agent-reports) - [Decide whether your real estate website needs Markdown](/articles/markdown-html-agent-pages-ai-retrieval) - [Can AI search read the public PDFs on a real estate website](/articles/public-real-estate-pdfs-in-ai-search) ---- ```text .|########||. .|########||. .|########||. |##||. .||##|. |##||. .||##|. |##||. .||##|. |#|. .|#|. |#|. .|#|. |#|. .|#|. |#| |#| |#| |#| |#| |#| .#| |#. .#| |#. .#| |#. .#. .#| .#. .#| .#. .#| |#. .#| |#. .#| |#. .#| |# ...... #| |# ...... #| |# ...... #| .# ||#########| #| .# ||#########| #| .# ||#########| #| .#. |######||######|. .#. .#. |######||######|. .#. .#. |######||######|. .#. #. .##|###|##|#|######| .# #. .##|###|##|#|######| .# #. .##|###|##|#|######| .# || |##|#||||||||||||#||#| || || |##|#||||||||||||#||#| || || |##|#||||||||||||#||#| || # |#||||||||||||||||||||#| #. # |#||||||||||||||||||||#| #. # |#||||||||||||||||||||#| #. || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || # .#||||||||||||||||||||||||#| # # .#||||||||||||||||||||||||#| # # .#||||||||||||||||||||||||#| # || ....|||#||||||##|#|||#|#||##||||....|. || || ....|||#||||||##|#|||#|#||##||||....|. || || ....|||#||||||##|#|||#|#||##||||....|. || #. . ....|# ....#||| ||| .#||| ....#. .# #. . ....|# ....#||| ||| .#||| ....#. .# #. . ....|# ....#||| ||| .#||| ....#. .# # . ||||#| .#####|| . .#| .#||| ||||# #. # . ||||#| .#####|| . .#| .#||| ||||# #. # . ||||#| .#####|| . .#| .#||| ||||# #. .| ||||| #. |#|||||. || #. |#||. .||||| || .| ||||| #. |#|||||. || #. |#||. .||||| || .| ||||| #. |#|||||. || #. |#||. .||||| || || |.... #. ....|#. |. ...|. ....|| || || |.... #. ....|#. |. ...|. ....|| || || |.... #. ....|#. |. ...|. ....|| || #. .||||||##||||||##||####|||||||#||||||#| .# #. .||||||##||||||##||####|||||||#||||||#| .# #. .||||||##||||||##||####|||||||#||||||#| .# # .||####|###########################|. # # .||####|###########################|. # # .||####|###########################|. # # .#||||||.#.|| # |. ..# #| #|||||#| # # .#||||||.#.|| # |. ..# #| #|||||#| # # .#||||||.#.|| # |. ..# #| #|||||#| # # .#|||||| .. |# ## |#| ...#|#|||#| # # .#|||||| .. |# ## |#| ...#|#|||#| # # .#|||||| .. |# ## |#| ...#|#|||#| # # .#|||#|# .# .#| #| ##|.#..#|||||#| # # .#|||#|# .# .#| #| ##|.#..#|||||#| # # .#|||#|# .# .#| #| ##|.#..#|||||#| # # .#||||||############|######|||||#| # # .#||||||############|######|||||#| # # .#||||||############|######|||||#| # # |...||#||||||#||||#|#||||||#|||||#||| |.| # # |...||#||||||#||||#|#||||||#|||||#||| |.| # # |...||#||||||#||||#|#||||||#|||||#||| |.| # #. .. ||||# .|||##|. |#| .|| || .|||#. #| # .# #. .. ||||# .|||##|. |#| .|| || .|||#. #| # .# #. .. ||||# .|||##|. |#| .|| || .|||#. #| # .# || | ...|| ...##| | #| .|. |. ##### .. .| || || | ...|| ...##| | #| .|. |. ##### .. .| || || | ...|| ...##| | #| .|. |. ##### .. .| || || ||||| || ||||#| . |. |. |#. ||||| |#| || || || ||||| || ||||#| . |. |. |#. ||||| |#| || || || ||||| || ||||#| . |. |. |#. ||||| |#| || || .# |.....#|....|#.||||.|||##.|#|....||.#||.#. #. .# |.....#|....|#.||||.|||##.|#|....||.#||.#. #. .# |.....#|....|#.||||.|||##.|#|....||.#||.#. #. #.|||||||#############################| ||| .# #.|||||||#############################| ||| .# #.|||||||#############################| ||| .# || ##||#||#||||||#||#||#||#||##. || || ##||#||#||||||#||#||#||#||##. || || ##||#||#||||||#||#||#||#||##. || # .#|||||||||#||#|||||||||||#| # # .#|||||||||#||#|||||||||||#| # # .#|||||||||#||#|||||||||||#| # || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || .# |#||||||||||||||||||||#| #. .# |#||||||||||||||||||||#| #. .# |#||||||||||||||||||||#| #. || |#||||||||||||||||||#| || || |#||||||||||||||||||#| || || |#||||||||||||||||||#| || #. .######|#|#########| .# #. .######|#|#########| .# #. .######|#|#########| .# .#. |#####||||#####| .#. .#. |#####||||#####| .#. .#. |#####||||#####| .#. |# ||########|| #. |# ||########|| #. |# ||########|| #. |# ...... #| |# ...... #| |# ...... #| |#. .#| |#. .#| |#. .#| |#. .#. |#. .#. |#. .#. .#| |#. .#| |#. .#| |#. |#| |#| |#| |#| |#| |#| .|#|. .|#|. .|#|. .|#|. .|#|. .|#|. |##||. .||##| |##||. .||##| |##||. .||##| .||########||. .||########||. .||########||. Scale With Search 2026 [scalewithsearch.com](https://scalewithsearch.com) ```