--- title: "Configure robots.txt so Googlebot spends its crawl budget on pages that earn traffic" description: "Write robots.txt rules that guide crawlers without blocking critical pages, and cut the URLs that waste crawl budget on large sites." canonical: "https://scalewithsearch.com/articles/robots-txt-crawl-budget-developers" date: "2026-03-20" modified: "2026-09-25" --- ## Site navigation - [Scale With Search](https://scalewithsearch.com/) - Real estate - Real estate - [Real estate](https://scalewithsearch.com/for/real-estate) - Work - Start here - [Send your brief](https://scalewithsearch.com/work#send-your-brief) - [Prepare your six-question brief](https://scalewithsearch.com/work#prepare-your-six-question-brief) - Build - [Site build, content library with SEO, signal desk](https://scalewithsearch.com/work) - For your business - Trades and home services - [Auto body and collision shops](https://scalewithsearch.com/for/auto-body-and-collision-shops) - [Foundation and home repair contractors](https://scalewithsearch.com/for/foundation-and-home-repair) - [Garage door and fencing contractors](https://scalewithsearch.com/for/garage-door-and-fencing-contractors) - [HVAC contractors](https://scalewithsearch.com/for/hvac-contractors) - [Janitorial and commercial cleaning companies](https://scalewithsearch.com/for/janitorial-and-commercial-cleaning) - [Locksmiths](https://scalewithsearch.com/for/locksmiths) - [Moving companies](https://scalewithsearch.com/for/moving-companies) - [Pest control companies](https://scalewithsearch.com/for/pest-control-companies) - [Plumbing and electrical contractors](https://scalewithsearch.com/for/plumbing-and-electrical-contractors) - [Restoration and water or fire damage companies](https://scalewithsearch.com/for/restoration-and-water-fire-damage) - [Roofing companies](https://scalewithsearch.com/for/roofing-companies) - [Towing companies](https://scalewithsearch.com/for/towing-companies) - [Tree services and landscaping companies](https://scalewithsearch.com/for/tree-services-and-landscaping) - [Solar installers](https://scalewithsearch.com/for/solar-installation) - [General contractors](https://scalewithsearch.com/for/general-contractors-and-construction) - [Paving, concrete, and flooring contractors](https://scalewithsearch.com/for/paving) - Practices and professional services - [Bookkeeping and tax practices](https://scalewithsearch.com/for/bookkeeping-and-tax-practices) - [Dental practices](https://scalewithsearch.com/for/dental-practices) - [Family and criminal defense law firms](https://scalewithsearch.com/for/family-and-criminal-defense-law-firms) - [Med spas and aesthetics practices](https://scalewithsearch.com/for/med-spas-and-aesthetics) - [Personal injury law firms](https://scalewithsearch.com/for/personal-injury-law-firms) - [Veterinary clinics](https://scalewithsearch.com/for/veterinary-clinics) - [Gyms and fitness studios](https://scalewithsearch.com/for/fitness) - [Therapy and outpatient health practices](https://scalewithsearch.com/for/therapy-and-outpatient-health) - [Medical billing companies](https://scalewithsearch.com/for/medical-billing) - [Insurance agencies](https://scalewithsearch.com/for/insurance-agencies) - [Financial advisors](https://scalewithsearch.com/for/financial-advisors) - [Property management companies](https://scalewithsearch.com/for/property-management) - [Recruiting and staffing agencies](https://scalewithsearch.com/for/recruiting-and-staffing) - [Architects and interior designers](https://scalewithsearch.com/for/architects-and-interior-designers) - [Logistics and supply chain companies](https://scalewithsearch.com/for/logistics-and-supply-chain) - Agencies, MSPs, and manufacturing - [IT and managed service providers](https://scalewithsearch.com/for/it-and-managed-service-providers) - [Machine shops and precision manufacturers](https://scalewithsearch.com/for/machine-shops-and-precision-manufacturing) - [Marketing agencies and freelancers](https://scalewithsearch.com/for/marketing-agencies-and-freelancers) - [SEO agencies and consultants](https://scalewithsearch.com/for/seo-agencies-and-consultants) - [Small manufacturers and fabricators](https://scalewithsearch.com/for/small-manufacturers-and-fabricators) - Restaurants, shops, studios, and nonprofits - [Restaurants and hospitality businesses](https://scalewithsearch.com/for/restaurants-and-hospitality) - [Retail stores and ecommerce sellers](https://scalewithsearch.com/for/retail-and-ecommerce) - [Photographers, event planners, and travel agents](https://scalewithsearch.com/for/photographers) - [Churches and nonprofits](https://scalewithsearch.com/for/churches-and-nonprofits) - [All industries](https://scalewithsearch.com/for/) - Learn - For your office - [Office job guides](https://scalewithsearch.com/guides/) - [Browser calculators](https://scalewithsearch.com/tools/) - Start here - [How it works](https://scalewithsearch.com/how-it-works) - [Free Starter Kit](https://scalewithsearch.com/kit/business-memory-starter-kit.zip) - [Synthetic specimen](https://scalewithsearch.com/specimen/working-session-specimen.zip) - Guides - [The Complete Guide to Business Memory for AI Agents](https://scalewithsearch.com/articles/business-memory-for-ai-agents-guide) - [The Complete Small-Business Guide to AI Agent Governance](https://scalewithsearch.com/articles/ai-agent-governance-guide-small-business) - [The Complete Guide to Leaving Vendor AI Memory](https://scalewithsearch.com/articles/leaving-vendor-ai-memory-guide) - Articles by cluster - [Business memory](https://scalewithsearch.com/articles/business-memory-for-ai-agents-guide) - [Agent governance](https://scalewithsearch.com/articles/ai-agent-governance-guide-small-business) - [Migration and ownership](https://scalewithsearch.com/articles/leaving-vendor-ai-memory-guide) - For machines - [llms.txt](https://scalewithsearch.com/llms.txt) - [llms-full.txt](https://scalewithsearch.com/llms-full.txt) - [Machine view](https://scalewithsearch.com/?view=machine) - Company - Evidence - [Proof](https://scalewithsearch.com/proof) - Company - [About](https://scalewithsearch.com/about) # Configure robots.txt so Googlebot spends its crawl budget on pages that earn traffic. A developer adds `Disallow: /` to block a staging server. The same file ships to production. Organic traffic falls over the next weeks, and nobody connects the drop to a two-line text file. Many developers treat `robots.txt` as a security boundary for admin panels, staging servers, and private directories. It is not. It is guidance for polite crawlers. That confusion causes three kinds of damage. Whole site sections get blocked by accident. Crawl budget goes to endless facet combinations. Private URLs become public because the file lists them. On a large site, Googlebot cannot fetch every URL on every visit. A weak `robots.txt` makes this worse. Crawlers spend capacity on duplicate pages, deep pagination, and internal search results that bring no organic traffic. Meanwhile new content and high-value product pages wait weeks for discovery. This guide covers `robots.txt` syntax and the mistakes that damage search visibility. It then explains how crawl budget works and the patterns that keep crawlers on the pages that matter. ## Learn the robots.txt syntax The file must sit at the root of the host: `https://example.com/robots.txt`. A file in a subdirectory, such as `/blog/robots.txt`, has no effect. The file is plain text: ``` User-agent: * Disallow: /admin/ Disallow: /private/ Allow: /public/ Sitemap: https://example.com/sitemap.xml ``` | Directive | What it does | |---|---| | `User-agent` | Names the crawler that the next rules apply to. `*` means all crawlers. Specific names include `Googlebot` and `Bingbot`. | | `Disallow` | Names a path or pattern that the crawler must not request. | | `Allow` | Permits a path inside a disallowed directory. For example, block `/search/` but allow `/search/help/`. | | `Sitemap` | Gives the location of an XML sitemap, so crawlers find URLs faster. | | `Crawl-delay` | Asks a crawler to wait N seconds between requests. Googlebot ignores it. Bingbot and some other crawlers respect it. | The trailing slash changes the match. `Disallow: /admin/` blocks the directory. `Disallow: /admin` blocks every path that starts with `/admin`, which includes `/admin-login` and `/administrator`. Pattern support is narrow. Google supports two wildcards: `*` for any sequence of characters and `$` for the end of the URL. Regular expressions and character ranges such as `[0-9]` do not work. Google reads them as literal characters. ### robots.txt is not security `robots.txt` is a request, not enforcement. A malicious bot ignores it. Protect sensitive content with authentication, permissions, and server-side access control. The file is also public. Anyone can read `https://example.com/robots.txt`. A `Disallow` line that names a sensitive directory tells an attacker where to look. ## Avoid the mistakes that damage search visibility ### Whole-site block ``` User-agent: * Disallow: / ``` These two lines block every crawler from every page. The usual cause is a staging configuration that reaches production. Before each deployment, check the file that production will serve. Google retired the Search Console robots.txt Tester in December 2023 and replaced it with the robots.txt report. Use that report to see the file Google fetched. Use the URL Inspection tool to test whether Google can crawl a specific URL. ### Blocked JavaScript, CSS, or images ``` Disallow: /assets/ Disallow: /*.js Disallow: /*.css ``` Google renders pages to understand their content and layout. When `robots.txt` blocks the scripts and styles that rendering needs, Google sees a broken page, and rankings suffer. Block a script or style file only when you have a specific reason, which is rare. ### Blanket parameter block ``` Disallow: /*? ``` This rule blocks every URL with a query string, including real pages such as `/product?id=123` or `/blog?page=2`. Block the specific parameters that cause problems, not all parameters. ### Stale rules after a restructure Old rules for deleted directories stay in the file, and new private directories never get added. Treat `robots.txt` as a maintained file. Review it with each site restructure. ### Rules scattered across files and tags Rules in a subdirectory file are ignored, and rules that conflict with meta robots tags cause confusion. Keep all crawl rules in one root `robots.txt` file, and keep indexing rules in meta robots tags. ## Understand crawl budget Crawl budget is the number of URLs that Googlebot crawls on your site in a given period. Two factors set it. **Crawl capacity limit.** This is how many parallel connections and requests per second your server can take without slowing down. Google adjusts it automatically from your server's response times and errors. **Crawl demand.** This is how much crawling Google's systems schedule for your site. Popular URLs and content that changes often raise demand. Google's crawl budget guide for large sites names two audiences. The first is sites with about 1 million or more unique pages that change moderately often. The second is sites with about 10,000 or more pages that change very rapidly. A site under 1,000 pages rarely has a crawl budget problem. On a site with hundreds of thousands of pages, it becomes critical. Suppose Google crawls 10,000 URLs a day on a 500,000-page site. Each crawl of a low-value URL is a crawl that a high-value page did not get. ### Recognize crawl budget problems | Signal | What it suggests | |---|---| | New pages take 2 to 4 weeks to appear in search results | Crawlers do not prioritize discovery of new URLs | | Important pages drop out of the index | Crawlers visit them rarely and treat them as low priority | | The Crawl Stats report shows fewer pages crawled per day over time | Crawl activity is falling | | 100,000 products are live but only 40,000 are indexed | Crawling does not cover the full catalog | ## Block low-value URL patterns Find the URL patterns that bring no organic traffic but consume crawls. Then block them by pattern. Internal search results create endless URL combinations: ``` Disallow: /search? Disallow: /*?q= ``` Faceted navigation on e-commerce sites (color, size, price) creates millions of URL permutations. Block the filter parameters: ``` Disallow: /*?color= Disallow: /*?size= Disallow: /*?filter= ``` If some filter combinations have search value, allow them: ``` Disallow: /*? Allow: /*?category= ``` Session IDs and tracking parameters create duplicate URLs. The pattern `/*?utm_` matches only when the parameter comes first in the query string: ``` Disallow: /*?sessionid= Disallow: /*?utm_ Disallow: /*?ref= ``` Private account pages, such as dashboards, checkout flows, and account settings, have no search value: ``` Disallow: /account/ Disallow: /checkout/ Disallow: /cart/ ``` Admin and internal tools: ``` Disallow: /admin/ Disallow: /wp-admin/ Disallow: /internal/ ``` Test and staging paths do not belong on a production domain. If they are there, block them, and move them later: ``` Disallow: /staging/ Disallow: /test/ ``` Deep pagination is a special case. `robots.txt` cannot express "page 10 and above", because it has no character ranges. If deep pagination wastes crawls, change the pages instead. Show more items per page, link deep pages less, and keep them out of the sitemap. For filter URLs specifically, the [faceted navigation guide for e-commerce](/articles/ecommerce-seo-faceted-navigation) compares blocking with the other controls. ### Allow important content explicitly When a `Disallow` rule is broad, add `Allow` rules for the exceptions: ``` User-agent: * Disallow: /products/ Allow: /products/featured/ Allow: /products/bestsellers/ ``` This blocks most product URLs and keeps the high-value subsections open. ## Use sitemaps to point crawlers at priority pages An XML sitemap tells crawlers which URLs you want crawled. It complements `robots.txt`, which tells them what to skip. List high-value landing pages and new or recently updated content. Also list pages with few internal links, because crawlers find them less often. Leave out any URL that `robots.txt` blocks, duplicate pages, thin pages, and paginated URLs beyond page 2 or 3. Generate sitemaps automatically when new content goes live, and submit them in Search Console. One sitemap file holds at most 50,000 URLs or 50 MB uncompressed. A larger site needs a sitemap index: ```xml https://example.com/sitemap-products.xml https://example.com/sitemap-blog.xml ``` Google's sitemap documentation says Google ignores `` and ``. It uses `` when the value is consistently accurate. Keep the sitemap fresh and include only pages worth a crawl. The [XML sitemap guide for developers](/articles/xml-sitemap-best-practices-developers) covers generation and validation in depth. ## Make the server cheaper to crawl Server behavior changes how much Google crawls. A fast, reliable server gets more crawls. **Cut response time.** Google crawls a slow server more slowly to avoid overload. A time to first byte above 500 ms is one example of a slow response. Optimize database queries and add caching with Redis or Memcached. Serve static assets from a CDN, and scale CPU and memory where needed. **Support conditional requests.** Return an `ETag` or `Last-Modified` header. Then answer `If-None-Match` or `If-Modified-Since` requests with `304 Not Modified` when the page has not changed. Google's crawler documentation lists support for these headers. A `304` response costs your server little and tells the crawler that nothing changed. **Return correct status codes.** Use `301` and `302` redirects for their correct purposes. Return a real `404` for a missing page instead of a soft 404. Keep `5xx` errors rare. Frequent errors tell Google the server is unreliable, and it crawls less. The [HTTP status code guide](/articles/http-status-codes-seo) lists what each code tells a crawler. **Flatten redirect chains.** Each hop in a chain costs a crawl. Point each old URL directly at its final destination. **Consolidate duplicates.** A canonical tag names the preferred version of a page: ```html ``` Search Console once had a crawl rate setting. Google removed that setting in January 2024. To slow Googlebot in an emergency, return `500`, `503`, or `429` responses for a short time. Do not keep that up for more than a day or two, because Google can drop URLs that keep failing. ## Choose between robots.txt and meta robots `robots.txt` stops the crawl: Googlebot never requests the page. A meta robots tag lets the crawl happen and controls indexing: ```html ``` | Goal | Control | Example pages | |---|---|---| | Crawl for link discovery, keep out of the index | `noindex, follow` | Tag archives, author pages, filtered views that link to important content | | Keep out of the index and do not follow links | `noindex, nofollow` | User-generated spam, thank-you pages | | Do not crawl at all | `robots.txt` `Disallow` | Internal search results, private account paths | Do not use `robots.txt` to remove a page from the index. Google must crawl the page to see a `noindex` tag. If `robots.txt` blocks the crawl, Google never sees the tag. A page that was already indexed can stay in results with a limited description, because Google cannot fetch it to update it. To remove a page, use `noindex` and allow the crawl. For a fast temporary removal, use the Removals tool in Search Console. The same logic applies to duplicate content. If `robots.txt` blocks a duplicate, Google cannot see its canonical tag and cannot tell which version to index. Use `robots.txt` for pages with no index value. Use canonical tags for duplicates where you want one version indexed. The [canonical tag guide for developers](/articles/canonical-tags-developers) covers the edge cases. `Disallow` and `rel="nofollow"` also differ. `Disallow` stops crawlers from requesting a URL, but the URL can still be indexed without its content if other pages link to it. `nofollow` on a link is a hint that Google may or may not follow. The target page can still be found through other links. Block images or PDFs only when you do not want them in Google Images or search results. Both can bring substantial traffic. ## Test and monitor each change Google generally caches `robots.txt` for up to 24 hours. After a change, you can request a recrawl of the file from the robots.txt report in Search Console. Follow these steps before each `robots.txt` deployment: 1. Test the new rules against a list of important URLs with a robots.txt parser, such as Google's open-source parser. 2. Deploy the file to a staging environment first. 3. After production deployment, confirm the fetched version in the robots.txt report. 4. Monitor crawl stats and indexing for 1 to 2 weeks. 5. Confirm that no important page was blocked by accident. Four reports show whether the rules work: - The Crawl Stats report shows crawl volume, response times, and file types. A sudden drop means a problem. - The Page indexing report shows which pages are indexed and which are excluded. The status "Crawled - currently not indexed" points to crawls spent on low-value pages. - Server log analysis shows what Googlebot actually requests. Screaming Frog Log File Analyser or Botify shows which pages get the most crawls and which get none. It also shows what share of crawls land on valuable URLs. The [log file analysis guide](/articles/log-file-analysis-seo) walks through the method. - The URL Inspection tool shows whether a specific URL is blocked. ## Apply advanced patterns on large sites ### Generate robots.txt from site state On a site with a complex structure, generate `robots.txt` in code from the current configuration: ```python def generate_robots_txt(): rules = ["User-agent: *"] # Block all filter parameters except category filter_params = ['color', 'size', 'brand', 'price'] for param in filter_params: rules.append(f"Disallow: /*?{param}=") rules.append("Allow: /*?category=") rules.append("Sitemap: https://example.com/sitemap.xml") return "\n".join(rules) ``` ### Give different crawlers different rules ``` User-agent: Googlebot Disallow: /private/ User-agent: Bingbot Crawl-delay: 1 Disallow: /private/ User-agent: * Crawl-delay: 5 Disallow: / ``` This gives Google and Bing full access to public pages and shuts out all other crawlers. Googlebot needs no `Crawl-delay` line, because it ignores the directive. ### Slow or block aggressive scrapers A polite bot that hammers your server responds to a named block: ``` User-agent: BadBot Crawl-delay: 60 Disallow: / ``` Look up the user-agent strings of problem bots in your server logs and add them by name. A bot that ignores `robots.txt` needs a firewall or rate limit rule at the server or CDN, because `robots.txt` cannot stop it. ---- ```text .|########||. .|########||. .|########||. |##||. .||##|. |##||. .||##|. |##||. .||##|. |#|. .|#|. |#|. .|#|. |#|. .|#|. |#| |#| |#| |#| |#| |#| .#| |#. .#| |#. .#| |#. .#. .#| .#. .#| .#. .#| |#. .#| |#. .#| |#. .#| |# ...... #| |# ...... #| |# ...... #| .# ||#########| #| .# ||#########| #| .# ||#########| #| .#. |######||######|. .#. .#. |######||######|. .#. .#. |######||######|. .#. #. .##|###|##|#|######| .# #. .##|###|##|#|######| .# #. .##|###|##|#|######| .# || |##|#||||||||||||#||#| || || |##|#||||||||||||#||#| || || |##|#||||||||||||#||#| || # |#||||||||||||||||||||#| #. # |#||||||||||||||||||||#| #. # |#||||||||||||||||||||#| #. || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || # .#||||||||||||||||||||||||#| # # .#||||||||||||||||||||||||#| # # .#||||||||||||||||||||||||#| # || ....|||#||||||##|#|||#|#||##||||....|. || || ....|||#||||||##|#|||#|#||##||||....|. || || ....|||#||||||##|#|||#|#||##||||....|. || #. . ....|# ....#||| ||| .#||| ....#. .# #. . ....|# ....#||| ||| .#||| ....#. .# #. . ....|# ....#||| ||| .#||| ....#. .# # . ||||#| .#####|| . .#| .#||| ||||# #. # . ||||#| .#####|| . .#| .#||| ||||# #. # . ||||#| .#####|| . .#| .#||| ||||# #. .| ||||| #. |#|||||. || #. |#||. .||||| || .| ||||| #. |#|||||. || #. |#||. .||||| || .| ||||| #. |#|||||. || #. |#||. .||||| || || |.... #. ....|#. |. ...|. ....|| || || |.... #. ....|#. |. ...|. ....|| || || |.... #. ....|#. |. ...|. ....|| || #. .||||||##||||||##||####|||||||#||||||#| .# #. .||||||##||||||##||####|||||||#||||||#| .# #. .||||||##||||||##||####|||||||#||||||#| .# # .||####|###########################|. # # .||####|###########################|. # # .||####|###########################|. # # .#||||||.#.|| # |. ..# #| #|||||#| # # .#||||||.#.|| # |. ..# #| #|||||#| # # .#||||||.#.|| # |. ..# #| #|||||#| # # .#|||||| .. |# ## |#| ...#|#|||#| # # .#|||||| .. |# ## |#| ...#|#|||#| # # .#|||||| .. |# ## |#| ...#|#|||#| # # .#|||#|# .# .#| #| ##|.#..#|||||#| # # .#|||#|# .# .#| #| ##|.#..#|||||#| # # .#|||#|# .# .#| #| ##|.#..#|||||#| # # .#||||||############|######|||||#| # # .#||||||############|######|||||#| # # .#||||||############|######|||||#| # # |...||#||||||#||||#|#||||||#|||||#||| |.| # # |...||#||||||#||||#|#||||||#|||||#||| |.| # # |...||#||||||#||||#|#||||||#|||||#||| |.| # #. .. ||||# .|||##|. |#| .|| || .|||#. #| # .# #. .. ||||# .|||##|. |#| .|| || .|||#. #| # .# #. .. ||||# .|||##|. |#| .|| || .|||#. #| # .# || | ...|| ...##| | #| .|. |. ##### .. .| || || | ...|| ...##| | #| .|. |. ##### .. .| || || | ...|| ...##| | #| .|. |. ##### .. .| || || ||||| || ||||#| . |. |. |#. ||||| |#| || || || ||||| || ||||#| . |. |. |#. ||||| |#| || || || ||||| || ||||#| . |. |. |#. ||||| |#| || || .# |.....#|....|#.||||.|||##.|#|....||.#||.#. #. .# |.....#|....|#.||||.|||##.|#|....||.#||.#. #. .# |.....#|....|#.||||.|||##.|#|....||.#||.#. #. #.|||||||#############################| ||| .# #.|||||||#############################| ||| .# #.|||||||#############################| ||| .# || ##||#||#||||||#||#||#||#||##. || || ##||#||#||||||#||#||#||#||##. || || ##||#||#||||||#||#||#||#||##. || # .#|||||||||#||#|||||||||||#| # # .#|||||||||#||#|||||||||||#| # # .#|||||||||#||#|||||||||||#| # || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || .# |#||||||||||||||||||||#| #. .# |#||||||||||||||||||||#| #. .# |#||||||||||||||||||||#| #. || |#||||||||||||||||||#| || || |#||||||||||||||||||#| || || |#||||||||||||||||||#| || #. .######|#|#########| .# #. .######|#|#########| .# #. .######|#|#########| .# .#. |#####||||#####| .#. .#. |#####||||#####| .#. .#. |#####||||#####| .#. |# ||########|| #. |# ||########|| #. |# ||########|| #. |# ...... #| |# ...... #| |# ...... #| |#. .#| |#. .#| |#. .#| |#. .#. |#. .#. |#. .#. .#| |#. .#| |#. .#| |#. |#| |#| |#| |#| |#| |#| .|#|. .|#|. .|#|. .|#|. .|#|. .|#|. |##||. .||##| |##||. .||##| |##||. .||##| .||########||. .||########||. .||########||. Scale With Search 2026 [scalewithsearch.com](https://scalewithsearch.com) ```