--- title: "AI crawlers, JavaScript, and IDX: checks for agent websites" description: "Compare initial HTML with rendered IDX and service pages. Check crawler access, choose a supported repair, and verify the buyer or seller inquiry path." canonical: "https://scalewithsearch.com/articles/ai-crawlers-javascript-idx-agent-websites" --- ## Site navigation - [Scale With Search](https://scalewithsearch.com/) - Services - Services - [AI systems](https://scalewithsearch.com/ai-systems) - [Site build](https://scalewithsearch.com/work#site-build) - [Whole website fix](https://scalewithsearch.com/work#whole-website-fix) - [Content library with SEO](https://scalewithsearch.com/work#content-library) - [Working session](https://scalewithsearch.com/audit) - Start here - [Send your brief](https://scalewithsearch.com/work#send-your-brief) - [Example scope](https://scalewithsearch.com/work#review-the-example-scope) - [Prices and terms](https://scalewithsearch.com/work#prices-and-terms) - Industries - Real estate - [Real estate agents and teams](https://scalewithsearch.com/for/real-estate) - [Platforms, ownership, and IDX](https://scalewithsearch.com/for/real-estate/real-estate-website-platforms-ownership-idx) - [Search, content, and AI](https://scalewithsearch.com/for/real-estate/real-estate-search-content-ai) - [Inquiries, campaigns, and handoffs](https://scalewithsearch.com/for/real-estate/real-estate-website-inquiries-campaigns-handoffs) - [Buyer, seller, and local pages](https://scalewithsearch.com/for/real-estate/real-estate-buyer-seller-local-resources) - [Rules, privacy, and trust](https://scalewithsearch.com/for/real-estate/real-estate-website-rules-privacy-trust) - Browse by business - [Trades and home services](https://scalewithsearch.com/for/#group-trades) - [Health, wellness, and animal care](https://scalewithsearch.com/for/#group-practices) - [Law, finance, and IT](https://scalewithsearch.com/for/#group-professional) - [Marketing and SEO agencies](https://scalewithsearch.com/for/#group-agencies) - [Shops and manufacturers](https://scalewithsearch.com/for/#group-manufacturing) - [Restaurants, shops, studios, and nonprofits](https://scalewithsearch.com/for/#group-local-services) - Chiropractic practices - [Chiropractic and wellness practices](https://scalewithsearch.com/for/chiropractic-and-wellness-practices) - [Chiropractic and physical therapy clinics](https://scalewithsearch.com/for/chiropractic-and-physical-therapy-clinics) - [All industries](https://scalewithsearch.com/for/) - Resources - Free tools - [AI readiness check](https://scalewithsearch.com/check) - [Free tools](https://scalewithsearch.com/tools/) - [Free Starter Kit](https://scalewithsearch.com/kit/business-memory-starter-kit.zip) - Guides - [AI-native library](https://scalewithsearch.com/ai-native) - [Industry intelligence](https://scalewithsearch.com/intelligence) - [Office job guides](https://scalewithsearch.com/guides/) - [Business memory](https://scalewithsearch.com/articles/business-memory-for-ai-agents-guide) - [Agent governance](https://scalewithsearch.com/articles/ai-agent-governance-guide-small-business) - [Migration and ownership](https://scalewithsearch.com/articles/leaving-vendor-ai-memory-guide) - Articles - [All articles](https://scalewithsearch.com/articles/) - [What changed](https://scalewithsearch.com/changes/) - About - Evidence - [Proof](https://scalewithsearch.com/proof) - Scale With Search - [About](https://scalewithsearch.com/about) - [How it works](https://scalewithsearch.com/how-it-works) # Can AI crawlers execute JavaScript on your real estate website? Some crawlers can execute JavaScript. Do not assume every AI search crawler can. Google documents JavaScript rendering. Vercel's December 2024 study found no rendering by the tested OpenAI crawlers or PerplexityBot. Those observations do not establish every service's current capabilities. [Google's JavaScript documentation](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics), [Vercel's crawler study](https://vercel.com/blog/the-rise-of-the-ai-crawler). [Google's JavaScript documentation](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics), checked 2026.10.14. [Vercel's crawler study](https://vercel.com/blog/the-rise-of-the-ai-crawler), checked 2026.10.14. For your agent website, compare the initial HTML with the page after JavaScript runs. Check the seller service explanation, buyer representation text, approved property details, and contact links. Ask which specific crawler receives each piece before paying for a repair. The checks below are a proposed diagnosis method. They do not report findings from your website or predict how many inquiries it will receive. ## Separate Google rendering from other AI search access Google processes pages through crawling, rendering, and indexing. An application shell can require JavaScript before its main content appears. Google still recommends server rendering or pre-rendering because some bots cannot run JavaScript. [Google's processing guidance](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics), checked 2026.10.14. Google's AI Overviews and AI Mode require a supporting page to be indexed and eligible for a Search snippet. Eligibility does not guarantee inclusion. Keep those Google Search checks separate from checks of other services. [Google's AI-feature requirements](https://developers.google.com/search/docs/appearance/ai-features), checked 2026.10.14. OpenAI identifies OAI-SearchBot as its search crawler and GPTBot as a training crawler. Their access settings are independent. ChatGPT-User supports certain user-requested visits and does not determine Search eligibility. [OpenAI's crawler documentation](https://developers.openai.com/api/docs/bots), checked 2026.10.14. Perplexity likewise distinguishes PerplexityBot, which surfaces websites in search, from Perplexity-User, which supports user-requested visits. Neither is described as a foundation-model training crawler. [Perplexity's crawler documentation](https://docs.perplexity.ai/docs/resources/perplexity-crawlers), checked 2026.10.14. Vercel's study observed OAI-SearchBot, ChatGPT-User, GPTBot, and PerplexityBot without JavaScript rendering. It also distinguished downloading JavaScript files from executing them. Treat this as dated evidence for those tested agents. The current provider pages cited above document access and purpose without establishing a universal rendering promise. [Vercel's observed behavior](https://vercel.com/blog/the-rise-of-the-ai-crawler), [OpenAI's documentation](https://developers.openai.com/api/docs/bots), [Perplexity's documentation](https://docs.perplexity.ai/docs/resources/perplexity-crawlers). [Vercel's observed behavior](https://vercel.com/blog/the-rise-of-the-ai-crawler), checked 2026.10.14. Require a contractor's report to name the crawler, evidence date, tested URL, and missing text. Leave a rendering capability unresolved when the available evidence cannot establish it. - [Zillow in ChatGPT: check the listing and agent inquiry path](/articles/zillow-chatgpt-app-agent-listing-inquiry-path) ## Inspect your own pages before replacing IDX Choose a seller service page, a buyer representation page, and an approved property detail page. Add your homepage if it uses a different template. Use the exact public addresses people should visit. Write down what each page must communicate. For a seller page, include your market, representation process, and contact route. For a buyer page, include the service you provide and how to ask about it. For a property page, include the permitted description, current status, required attribution, and inquiry destination. IDX permits limited electronic display of other participants' listings within the MLS framework. Obtain the relevant display permissions before reproducing listing material in a different format. [NAR's IDX framework](https://www.nar.realtor/about-nar/policies/internet-data-exchange-idx), checked 2026.10.14. Ask the vendor where the property content lives. If the display uses an iframe, it embeds another document inside the surrounding page. Inspect the embedded document's address separately from your own page. [MDN's iframe reference](https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/iframe), checked 2026.10.14. Do not count a vendor-hosted property page as proof that the same text appears on your domain. Save both addresses. Identify which page carries your business identity and which receives the buyer's inquiry. ## Compare initial HTML with rendered content Ask your developer to save the public page's HTTP response, including its status, headers, final URL, and HTML body. Use a fresh request without an account session. Then save the rendered page from a fresh browser session with JavaScript enabled. Search both versions for the same distinctive sentence from the service explanation or approved property description. Inspect its context. Record whether it appears as readable page text, inside a script, or only in the rendered document. Google's documentation distinguishes an HTML response containing the page content from an application shell that needs JavaScript. Use that distinction to name the defect. [Initial HTML and rendered content](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics), checked 2026.10.14. Use this proposed comparison sheet. These rows describe evidence to collect, not observed defects. | Page element | Initial HTML check | Rendered page check | Question for the vendor | | --- | --- | --- | --- | | Seller service explanation | Find the approved service text | Compare the visible wording | Can the server return this text directly? | | Buyer representation explanation | Find the approved buyer text | Compare the visible wording | Which template controls this section? | | Property details | Find the permitted description and status | Check the complete listing display | Is the content on this URL or an embedded URL? | | Contact route | Locate the destination link or form markup | Follow the visible contact route | Who owns the destination and delivery? | | Business identity | Locate the name and contact details | Compare the visible contact block | Does it belong to this agent or the listing provider? | Keep script-embedded data separate from readable HTML. Vercel notes that models may interpret some non-HTML content in the initial response. That possibility does not demonstrate extraction of your particular listing. [Vercel's qualification](https://vercel.com/blog/the-rise-of-the-ai-crawler), checked 2026.10.14. If the text is missing in both versions, request a content or template investigation. If it appears only after rendering, record a JavaScript dependency. If it appears in both, continue with the access check. ## Check access before blaming JavaScript OpenAI recommends allowing OAI-SearchBot through robots.txt and permitting requests from its published IP ranges. Perplexity recommends equivalent access for PerplexityBot and describes firewall rules combining user-agent and IP conditions. [OpenAI's access guidance](https://developers.openai.com/api/docs/bots), [Perplexity's firewall guidance](https://docs.perplexity.ai/docs/resources/perplexity-crawlers). [OpenAI's access guidance](https://developers.openai.com/api/docs/bots), checked 2026.10.14. [Perplexity's firewall guidance](https://docs.perplexity.ai/docs/resources/perplexity-crawlers), checked 2026.10.14. Ask the website manager to inspect the exact URL in server and firewall records. Save the request time, claimed user agent, verified source identity, response status, and applicable rule. Check the provider's current identity documentation before treating a bot label as proof. A request you send with a bot's user-agent string is a simulation. Label it that way. Use it to inspect response differences, then request evidence from a verified crawler visit. Keep verified crawler records separate from contractor test requests. Require the manager to distinguish a denied request, a challenge page, and a successful response containing the expected text. Resolve a confirmed access failure before evaluating whether the crawler executes JavaScript. If no verified visit is available, retain the initial HTML comparison and mark actual crawler access unresolved. Do not turn an absent log entry into a claim about rendering. ## Choose a repair that matches the finding For missing seller or buyer service text, request delivery of the approved explanation and contact links in initial HTML. Server rendering or static generation is a reasonable remedy to discuss when important text depends on JavaScript. Google recommends server rendering or pre-rendering, and Vercel recommends it for critical content. [Google's rendering guidance](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics), [Vercel's recommendations](https://vercel.com/blog/the-rise-of-the-ai-crawler). [Google's JavaScript processing guidance](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics), checked 2026.10.14; [Vercel's December 2024 study and recommendations](https://vercel.com/blog/the-rise-of-the-ai-crawler), checked 2026.10.14. For property details, ask the IDX vendor for a supported method under your existing permissions. Require a sample page before approving a platform change. Specify status updates, attribution, the public address, and who maintains the implementation. If the vendor cannot supply property text in initial HTML, document that limit. Prioritize your own service pages while evaluating the listing display separately. Do not approve copied property descriptions or a second inventory that nobody maintains. For a confirmed access block, request a narrowly scoped access correction for the intended search crawler. Keep the training decision explicit. Preserve the before-and-after response evidence. Use this proposed acceptance record for each repair: - Save the affected URL and original missing text or denied response. - Name the approved change and the person responsible for it. - Repeat the same initial HTML and rendered-content comparison. - Attach verified crawler access evidence when available. - Record unresolved checks and their next owner. ## Verify the buyer or seller inquiry after the repair The crawler comparison distinguishes missing page text from blocked access and failed inquiry delivery. Submit the affected URLs through [whole website fix: crawler rendering](/for/real-estate). The whole website fix includes five logged-out AI-answer questions as a baseline. [scalewithsearch.com primary source](https://scalewithsearch.com/for/real-estate), checked 2026.10.14. - [How MLS listings appear on your website, and what search can miss.](/articles/idx-mls) - [Check your real estate website in 30 minutes](/articles/real-estate-website-audit) - [Test the seller inquiry path on your real estate website](/articles/seller-inquiry-path-agent-website) ## Related: Real estate - [How MLS listings appear on your website, and what search can miss.](/articles/idx-mls) - [Check your real estate website in 30 minutes](/articles/real-estate-website-audit) - [Test the seller inquiry path on your real estate website](/articles/seller-inquiry-path-agent-website) ---- ```text .|########||. .|########||. .|########||. |##||. .||##|. |##||. .||##|. |##||. .||##|. |#|. .|#|. |#|. .|#|. |#|. .|#|. |#| |#| |#| |#| |#| |#| .#| |#. .#| |#. .#| |#. .#. .#| .#. .#| .#. .#| |#. .#| |#. .#| |#. .#| |# ...... #| |# ...... #| |# ...... #| .# ||#########| #| .# ||#########| #| .# ||#########| #| .#. |######||######|. .#. .#. |######||######|. .#. .#. |######||######|. .#. #. .##|###|##|#|######| .# #. .##|###|##|#|######| .# #. .##|###|##|#|######| .# || |##|#||||||||||||#||#| || || |##|#||||||||||||#||#| || || |##|#||||||||||||#||#| || # |#||||||||||||||||||||#| #. # |#||||||||||||||||||||#| #. # |#||||||||||||||||||||#| #. || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || # .#||||||||||||||||||||||||#| # # .#||||||||||||||||||||||||#| # # .#||||||||||||||||||||||||#| # || ....|||#||||||##|#|||#|#||##||||....|. || || ....|||#||||||##|#|||#|#||##||||....|. || || ....|||#||||||##|#|||#|#||##||||....|. || #. . ....|# ....#||| ||| .#||| ....#. .# #. . ....|# ....#||| ||| .#||| ....#. .# #. . ....|# ....#||| ||| .#||| ....#. .# # . ||||#| .#####|| . .#| .#||| ||||# #. # . ||||#| .#####|| . .#| .#||| ||||# #. # . ||||#| .#####|| . .#| .#||| ||||# #. .| ||||| #. |#|||||. || #. |#||. .||||| || .| ||||| #. |#|||||. || #. |#||. .||||| || .| ||||| #. |#|||||. || #. |#||. .||||| || || |.... #. ....|#. |. ...|. ....|| || || |.... #. ....|#. |. ...|. ....|| || || |.... #. ....|#. |. ...|. ....|| || #. .||||||##||||||##||####|||||||#||||||#| .# #. .||||||##||||||##||####|||||||#||||||#| .# #. .||||||##||||||##||####|||||||#||||||#| .# # .||####|###########################|. # # .||####|###########################|. # # .||####|###########################|. # # .#||||||.#.|| # |. ..# #| #|||||#| # # .#||||||.#.|| # |. ..# #| #|||||#| # # .#||||||.#.|| # |. ..# #| #|||||#| # # .#|||||| .. |# ## |#| ...#|#|||#| # # .#|||||| .. |# ## |#| ...#|#|||#| # # .#|||||| .. |# ## |#| ...#|#|||#| # # .#|||#|# .# .#| #| ##|.#..#|||||#| # # .#|||#|# .# .#| #| ##|.#..#|||||#| # # .#|||#|# .# .#| #| ##|.#..#|||||#| # # .#||||||############|######|||||#| # # .#||||||############|######|||||#| # # .#||||||############|######|||||#| # # |...||#||||||#||||#|#||||||#|||||#||| |.| # # |...||#||||||#||||#|#||||||#|||||#||| |.| # # |...||#||||||#||||#|#||||||#|||||#||| |.| # #. .. ||||# .|||##|. |#| .|| || .|||#. #| # .# #. .. ||||# .|||##|. |#| .|| || .|||#. #| # .# #. .. ||||# .|||##|. |#| .|| || .|||#. #| # .# || | ...|| ...##| | #| .|. |. ##### .. .| || || | ...|| ...##| | #| .|. |. ##### .. .| || || | ...|| ...##| | #| .|. |. ##### .. .| || || ||||| || ||||#| . |. |. |#. ||||| |#| || || || ||||| || ||||#| . |. |. |#. ||||| |#| || || || ||||| || ||||#| . |. |. |#. ||||| |#| || || .# |.....#|....|#.||||.|||##.|#|....||.#||.#. #. .# |.....#|....|#.||||.|||##.|#|....||.#||.#. #. .# |.....#|....|#.||||.|||##.|#|....||.#||.#. #. #.|||||||#############################| ||| .# #.|||||||#############################| ||| .# #.|||||||#############################| ||| .# || ##||#||#||||||#||#||#||#||##. || || ##||#||#||||||#||#||#||#||##. || || ##||#||#||||||#||#||#||#||##. || # .#|||||||||#||#|||||||||||#| # # .#|||||||||#||#|||||||||||#| # # .#|||||||||#||#|||||||||||#| # || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || .# |#||||||||||||||||||||#| #. .# |#||||||||||||||||||||#| #. .# |#||||||||||||||||||||#| #. || |#||||||||||||||||||#| || || |#||||||||||||||||||#| || || |#||||||||||||||||||#| || #. .######|#|#########| .# #. .######|#|#########| .# #. .######|#|#########| .# .#. |#####||||#####| .#. .#. |#####||||#####| .#. .#. |#####||||#####| .#. |# ||########|| #. |# ||########|| #. |# ||########|| #. |# ...... #| |# ...... #| |# ...... #| |#. .#| |#. .#| |#. .#| |#. .#. |#. .#. |#. .#. .#| |#. .#| |#. .#| |#. |#| |#| |#| |#| |#| |#| .|#|. .|#|. .|#|. .|#|. .|#|. .|#|. |##||. .||##| |##||. .||##| |##||. .||##| .||########||. .||########||. .||########||. Scale With Search 2026 [scalewithsearch.com](https://scalewithsearch.com) ```