--- title: "Separate AI crawler visits from real estate inquiries" description: "Separate crawler requests, website sessions, and reviewed real estate inquiries. Preserve identity evidence, unknown traffic, and reporting filters." canonical: "https://scalewithsearch.com/articles/separate-ai-bot-visits-from-agent-inquiries" --- ## Site navigation - [Scale With Search](https://scalewithsearch.com/) - Services - Services - [AI systems](https://scalewithsearch.com/ai-systems) - [Site build](https://scalewithsearch.com/work#site-build) - [Whole website fix](https://scalewithsearch.com/work#whole-website-fix) - [Content library with SEO](https://scalewithsearch.com/work#content-library) - [Working session](https://scalewithsearch.com/audit) - Start here - [Send your brief](https://scalewithsearch.com/work#send-your-brief) - [Example scope](https://scalewithsearch.com/work#review-the-example-scope) - [Prices and terms](https://scalewithsearch.com/work#prices-and-terms) - Industries - Real estate - [Real estate agents and teams](https://scalewithsearch.com/for/real-estate) - [Platforms, ownership, and IDX](https://scalewithsearch.com/for/real-estate/real-estate-website-platforms-ownership-idx) - [Search, content, and AI](https://scalewithsearch.com/for/real-estate/real-estate-search-content-ai) - [Inquiries, campaigns, and handoffs](https://scalewithsearch.com/for/real-estate/real-estate-website-inquiries-campaigns-handoffs) - [Buyer, seller, and local pages](https://scalewithsearch.com/for/real-estate/real-estate-buyer-seller-local-resources) - [Rules, privacy, and trust](https://scalewithsearch.com/for/real-estate/real-estate-website-rules-privacy-trust) - Browse by business - [Trades and home services](https://scalewithsearch.com/for/#group-trades) - [Health, wellness, and animal care](https://scalewithsearch.com/for/#group-practices) - [Law, finance, and IT](https://scalewithsearch.com/for/#group-professional) - [Marketing and SEO agencies](https://scalewithsearch.com/for/#group-agencies) - [Shops and manufacturers](https://scalewithsearch.com/for/#group-manufacturing) - [Restaurants, shops, studios, and nonprofits](https://scalewithsearch.com/for/#group-local-services) - Chiropractic practices - [Chiropractic and wellness practices](https://scalewithsearch.com/for/chiropractic-and-wellness-practices) - [Chiropractic and physical therapy clinics](https://scalewithsearch.com/for/chiropractic-and-physical-therapy-clinics) - [All industries](https://scalewithsearch.com/for/) - Resources - Free tools - [AI readiness check](https://scalewithsearch.com/check) - [Free tools](https://scalewithsearch.com/tools/) - [Free Starter Kit](https://scalewithsearch.com/kit/business-memory-starter-kit.zip) - Guides - [AI-native library](https://scalewithsearch.com/ai-native) - [Industry intelligence](https://scalewithsearch.com/intelligence) - [Office job guides](https://scalewithsearch.com/guides/) - [Business memory](https://scalewithsearch.com/articles/business-memory-for-ai-agents-guide) - [Agent governance](https://scalewithsearch.com/articles/ai-agent-governance-guide-small-business) - [Migration and ownership](https://scalewithsearch.com/articles/leaving-vendor-ai-memory-guide) - Articles - [All articles](https://scalewithsearch.com/articles/) - [What changed](https://scalewithsearch.com/changes/) - About - Evidence - [Proof](https://scalewithsearch.com/proof) - Scale With Search - [About](https://scalewithsearch.com/about) - [How it works](https://scalewithsearch.com/how-it-works) # Separate AI crawler visits from real estate agent inquiries Keep crawler requests, browser sessions, form activity, and received messages in separate categories. Cloudflare reports crawler requests, while Google Analytics automatically excludes known bots from its own measurements. These are different data systems. [Cloudflare's crawler analytics, checked 2026.10.22](https://developers.cloudflare.com/ai-crawl-control/features/analyze-ai-traffic/), [Google's bot exclusion, checked 2026.10.22](https://support.google.com/analytics/answer/9888366?hl=en), checked 2026.10.22. A real estate agent should be able to inspect what each reported result means. An automated page request does not establish that a buyer or seller contacted the business. ## Where should the reviewed inquiry count come from? Use the [existing AI-referral inquiry guide](/articles/track-chatgpt-agent-website-inquiries-ga4) for event definitions, delivery checks, and inquiry review. Bring its reviewed inquiry total into this comparison without repeating that procedure. [Existing inquiry guide, checked 2026.10.22](https://scalewithsearch.com/articles/track-chatgpt-agent-website-inquiries-ga4). ## What should count as a crawler request? Record a request that the available evidence identifies as automated crawling, with its classification confidence. Cloudflare distinguishes search, user-directed activity, and training behaviors in its [bot documentation, checked 2026.10.22](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/), checked 2026.10.22. A crawler's visit can be useful evidence of access. It is not evidence that the real estate agent received a consumer message. ## Does Google Analytics already remove every bot? Google describes automatic exclusion of known bots and spiders, rather than complete identification of all automation. Its documentation says the excluded amount is not available to inspect. [Google's known-bot exclusion, checked 2026.10.22](https://support.google.com/analytics/answer/9888366?hl=en), checked 2026.10.22. Keep that limitation in the report. Do not subtract an invented bot total from website sessions. ## Why might edge requests and website sessions differ? They measure different activity through different systems. Cloudflare documents request metrics; Google documents known-bot exclusion for Analytics. [Cloudflare metrics, checked 2026.10.22](https://developers.cloudflare.com/ai-crawl-control/features/analyze-ai-traffic/), [Analytics exclusion, checked 2026.10.22](https://support.google.com/analytics/answer/9888366?hl=en), checked 2026.10.22. Compare their definitions and time windows before investigating a discrepancy. This is a reporting check, not proof that either system lost an inquiry. ## Is a `User-Agent` string sufficient to identify a crawler? Treat a name supplied in the request as a claim until the available verification supports it. Cloudflare describes identity verification methods in its [verified-bot documentation, checked 2026.10.22](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/), checked 2026.10.22. Keep self-identified requests separate from verified requests in the proposed classification. Preserve unknown traffic instead of forcing every request into a confident category. ## Can the available account tier affect classification? Yes. Cloudflare says free-plan AI Crawl Control identifies crawlers using `User-Agent` strings. [Cloudflare's detection-quality note, checked 2026.10.22](https://developers.cloudflare.com/ai-crawl-control/features/manage-ai-crawlers/), checked 2026.10.22. State the evidence available to the actual report. A label derived from a name should not be described as independently verified operator identity without additional support. ## What does a bot score establish? It estimates how likely a request is automated. Cloudflare documents bot-score meaning and product availability in its [score reference, checked 2026.10.22](https://developers.cloudflare.com/bots/concepts/bot-score/), checked 2026.10.22. It does not identify purchase intent or validate a received message. In the proposed reporting method, use the available score as supporting classification evidence and retain uncertainty. ## Should all automated traffic be grouped together? Keep useful distinctions when evidence supports them. Cloudflare distinguishes crawler behavior categories in its [bot reference, checked 2026.10.22](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/), checked 2026.10.22. A proposed report can separate search crawling, user-directed fetches, monitoring, and unidentified automation. Avoid calling any of those groups “buyer interest” without an additional observation supporting that interpretation. ## Is a user-requested AI fetch the same as a browser referral? Treat them separately in your reporting method. Preserve evidence of an automated fetch and any later observed browser visit as separate records. Leave the relationship unresolved when the available records do not establish it. ## What if a browser visit carries an AI-service referrer? Record the observed referral under your documented attribution method. Keep it distinct from the crawler classification. Use the [existing AI-referral inquiry guide](/articles/track-chatgpt-agent-website-inquiries-ga4) for that separate setup. This article's procedure concerns which observations belong in which category. A referral observation still needs receiving-system evidence before becoming a reported inquiry. ## How should unknown source traffic be reported? Leave it unknown when the evidence does not identify the source. Do not reassign it to AI because an AI crawler visited nearby in time. This proposed reporting rule preserves uncertainty. Keep the observed landing page and available attribution fields without claiming a causal link that the records cannot establish. ## Should content and media crawler requests be compared separately? Keep those categories visible. Cloudflare separates HTML, JSON, and text content from images, videos, and other media. Label the selected category before comparing crawler totals. [Cloudflare's path categories, checked 2026.10.22](https://developers.cloudflare.com/ai-crawl-control/features/analyze-ai-traffic/), checked 2026.10.22. ## Can matching path patterns on different hostnames be treated as one total? Keep the hostname attached to each pattern. Cloudflare says pattern groups are not aggregated across hostnames. State any deliberate combined scope before comparing it with a single-host report. [Cloudflare's pattern grouping, checked 2026.10.22](https://developers.cloudflare.com/ai-crawl-control/features/analyze-ai-traffic/), checked 2026.10.22. ## Which crawler filters should accompany an exported comparison? Keep the date range, crawler, operator, hostname, and path selection. Cloudflare documents these filters and says downloads retain applied filters and groupings. Preserve that scope beside each period's totals. [Cloudflare's export scope, checked 2026.10.22](https://developers.cloudflare.com/ai-crawl-control/features/analyze-ai-traffic/), checked 2026.10.22. ## How should uncertain automation labels be handled? Keep uncertain requests in an explicit review category. Record the available classification evidence and the reason confidence remains limited. This proposed reporting method preserves unknown automation without treating a weak label as proof of crawler identity or consumer intent. ## What should the daily comparison show? Show crawler requests, observed website activity, submission records, and reviewed received inquiries as distinct measures. Use matching date boundaries and identify each source. This is a recommended reporting layout. Add unresolved discrepancies beside the relevant measure. Avoid presenting a rise in request volume as an increase in people who asked for service. ## Which crawler metrics can help investigate a spike? Cloudflare documents filters and breakdowns for crawler activity, paths, status responses, and operators. [Cloudflare's analytics fields, checked 2026.10.22](https://developers.cloudflare.com/ai-crawl-control/features/analyze-ai-traffic/), checked 2026.10.22. Use the available breakdown to locate the request pattern. Keep its classification confidence visible. A spike explanation still needs request evidence rather than an assumption based on the calendar. ## What if the crawler dashboard shows unsuccessful requests? Investigate the response and rule before attributing a cause. Cloudflare says unsuccessful requests can reflect rules or response errors beyond its own crawler-block action. [Cloudflare's request definitions, checked 2026.10.22](https://developers.cloudflare.com/ai-crawl-control/features/manage-ai-crawlers/), checked 2026.10.22. Use the [initial HTML comparison](/articles/ai-crawlers-javascript-idx-agent-websites) for content delivery, then scope any confirmed access defect separately. ## What should a contractor's measurement handoff include? Require category definitions, source systems, time windows, available identity evidence, and unresolved classification checks. Keep the reviewed inquiry total beside those measures and reference the existing inquiry guide for its review procedure. These are proposed acceptance requirements. The real estate agent should be able to identify each measure's source and scope. ## What should the real estate agent accept as the final result? Accept a report that separates observed automation, website activity, and reviewed received messages. Require unresolved items to remain visible. A useful measurement repair may change classification without increasing inquiries. Keep that outcome explicit. Future claims about AI-generated business should identify the source evidence and the received messages supporting the claim. Bring the approved requirements and open questions to the [real estate website brief](/for/real-estate). Request the agreed work through the [written website scope](/work). - [Why an AI response restarts after reconnecting](/articles/ai-response-restarts-after-reconnect) - [When ChatGPT asks you to reattach files after a model change](/articles/chatgpt-model-switch-reattach-files-context) - [Plan custom GA4 channels for real estate website campaigns](/articles/custom-ga4-channel-groups-agent-campaigns) ## Related: Real estate - [Why an AI response restarts after reconnecting](/articles/ai-response-restarts-after-reconnect) - [When ChatGPT asks you to reattach files after a model change](/articles/chatgpt-model-switch-reattach-files-context) - [Plan custom GA4 channels for real estate website campaigns](/articles/custom-ga4-channel-groups-agent-campaigns) ---- ```text .|########||. .|########||. .|########||. |##||. .||##|. |##||. .||##|. |##||. .||##|. |#|. .|#|. |#|. .|#|. |#|. .|#|. |#| |#| |#| |#| |#| |#| .#| |#. .#| |#. .#| |#. .#. .#| .#. .#| .#. .#| |#. .#| |#. .#| |#. .#| |# ...... #| |# ...... #| |# ...... #| .# ||#########| #| .# ||#########| #| .# ||#########| #| .#. |######||######|. .#. .#. |######||######|. .#. .#. |######||######|. .#. #. .##|###|##|#|######| .# #. .##|###|##|#|######| .# #. .##|###|##|#|######| .# || |##|#||||||||||||#||#| || || |##|#||||||||||||#||#| || || |##|#||||||||||||#||#| || # |#||||||||||||||||||||#| #. # |#||||||||||||||||||||#| #. # |#||||||||||||||||||||#| #. || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || # .#||||||||||||||||||||||||#| # # .#||||||||||||||||||||||||#| # # .#||||||||||||||||||||||||#| # || ....|||#||||||##|#|||#|#||##||||....|. || || ....|||#||||||##|#|||#|#||##||||....|. || || ....|||#||||||##|#|||#|#||##||||....|. || #. . ....|# ....#||| ||| .#||| ....#. .# #. . ....|# ....#||| ||| .#||| ....#. .# #. . ....|# ....#||| ||| .#||| ....#. .# # . ||||#| .#####|| . .#| .#||| ||||# #. # . ||||#| .#####|| . .#| .#||| ||||# #. # . ||||#| .#####|| . .#| .#||| ||||# #. .| ||||| #. |#|||||. || #. |#||. .||||| || .| ||||| #. |#|||||. || #. |#||. .||||| || .| ||||| #. |#|||||. || #. |#||. .||||| || || |.... #. ....|#. |. ...|. ....|| || || |.... #. ....|#. |. ...|. ....|| || || |.... #. ....|#. |. ...|. ....|| || #. .||||||##||||||##||####|||||||#||||||#| .# #. .||||||##||||||##||####|||||||#||||||#| .# #. .||||||##||||||##||####|||||||#||||||#| .# # .||####|###########################|. # # .||####|###########################|. # # .||####|###########################|. # # .#||||||.#.|| # |. ..# #| #|||||#| # # .#||||||.#.|| # |. ..# #| #|||||#| # # .#||||||.#.|| # |. ..# #| #|||||#| # # .#|||||| .. |# ## |#| ...#|#|||#| # # .#|||||| .. |# ## |#| ...#|#|||#| # # .#|||||| .. |# ## |#| ...#|#|||#| # # .#|||#|# .# .#| #| ##|.#..#|||||#| # # .#|||#|# .# .#| #| ##|.#..#|||||#| # # .#|||#|# .# .#| #| ##|.#..#|||||#| # # .#||||||############|######|||||#| # # .#||||||############|######|||||#| # # .#||||||############|######|||||#| # # |...||#||||||#||||#|#||||||#|||||#||| |.| # # |...||#||||||#||||#|#||||||#|||||#||| |.| # # |...||#||||||#||||#|#||||||#|||||#||| |.| # #. .. ||||# .|||##|. |#| .|| || .|||#. #| # .# #. .. ||||# .|||##|. |#| .|| || .|||#. #| # .# #. .. ||||# .|||##|. |#| .|| || .|||#. #| # .# || | ...|| ...##| | #| .|. |. ##### .. .| || || | ...|| ...##| | #| .|. |. ##### .. .| || || | ...|| ...##| | #| .|. |. ##### .. .| || || ||||| || ||||#| . |. |. |#. ||||| |#| || || || ||||| || ||||#| . |. |. |#. ||||| |#| || || || ||||| || ||||#| . |. |. |#. ||||| |#| || || .# |.....#|....|#.||||.|||##.|#|....||.#||.#. #. .# |.....#|....|#.||||.|||##.|#|....||.#||.#. #. .# |.....#|....|#.||||.|||##.|#|....||.#||.#. #. #.|||||||#############################| ||| .# #.|||||||#############################| ||| .# #.|||||||#############################| ||| .# || ##||#||#||||||#||#||#||#||##. || || ##||#||#||||||#||#||#||#||##. || || ##||#||#||||||#||#||#||#||##. || # .#|||||||||#||#|||||||||||#| # # .#|||||||||#||#|||||||||||#| # # .#|||||||||#||#|||||||||||#| # || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || .# |#||||||||||||||||||||#| #. .# |#||||||||||||||||||||#| #. .# |#||||||||||||||||||||#| #. || |#||||||||||||||||||#| || || |#||||||||||||||||||#| || || |#||||||||||||||||||#| || #. .######|#|#########| .# #. .######|#|#########| .# #. .######|#|#########| .# .#. |#####||||#####| .#. .#. |#####||||#####| .#. .#. |#####||||#####| .#. |# ||########|| #. |# ||########|| #. |# ||########|| #. |# ...... #| |# ...... #| |# ...... #| |#. .#| |#. .#| |#. .#| |#. .#. |#. .#. |#. .#. .#| |#. .#| |#. .#| |#. |#| |#| |#| |#| |#| |#| .|#|. .|#|. .|#|. .|#|. .|#|. .|#|. |##||. .||##| |##||. .||##| |##||. .||##| .||########||. .||########||. .||########||. Scale With Search 2026 [scalewithsearch.com](https://scalewithsearch.com) ```