--- title: "Read server logs to see what Googlebot crawls and stop wasted crawl budget" description: "Use server access logs to find crawl waste, orphan pages, crawl traps, and error spikes, then fix them with robots.txt, links, and speed." canonical: "https://scalewithsearch.com/articles/log-file-analysis-seo" date: "2026-03-20" modified: "2026-09-25" --- ## Site navigation - [Scale With Search](https://scalewithsearch.com/) - Real estate - Real estate - [Real estate](https://scalewithsearch.com/for/real-estate) - Work - Start here - [Send your brief](https://scalewithsearch.com/work#send-your-brief) - [Prepare your six-question brief](https://scalewithsearch.com/work#prepare-your-six-question-brief) - Build - [Site build, content library with SEO, signal desk](https://scalewithsearch.com/work) - For your business - Trades and home services - [Auto body and collision shops](https://scalewithsearch.com/for/auto-body-and-collision-shops) - [Foundation and home repair contractors](https://scalewithsearch.com/for/foundation-and-home-repair) - [Garage door and fencing contractors](https://scalewithsearch.com/for/garage-door-and-fencing-contractors) - [HVAC contractors](https://scalewithsearch.com/for/hvac-contractors) - [Janitorial and commercial cleaning companies](https://scalewithsearch.com/for/janitorial-and-commercial-cleaning) - [Locksmiths](https://scalewithsearch.com/for/locksmiths) - [Moving companies](https://scalewithsearch.com/for/moving-companies) - [Pest control companies](https://scalewithsearch.com/for/pest-control-companies) - [Plumbing and electrical contractors](https://scalewithsearch.com/for/plumbing-and-electrical-contractors) - [Restoration and water or fire damage companies](https://scalewithsearch.com/for/restoration-and-water-fire-damage) - [Roofing companies](https://scalewithsearch.com/for/roofing-companies) - [Towing companies](https://scalewithsearch.com/for/towing-companies) - [Tree services and landscaping companies](https://scalewithsearch.com/for/tree-services-and-landscaping) - [Solar installers](https://scalewithsearch.com/for/solar-installation) - [General contractors](https://scalewithsearch.com/for/general-contractors-and-construction) - [Paving, concrete, and flooring contractors](https://scalewithsearch.com/for/paving) - Practices and professional services - [Bookkeeping and tax practices](https://scalewithsearch.com/for/bookkeeping-and-tax-practices) - [Dental practices](https://scalewithsearch.com/for/dental-practices) - [Family and criminal defense law firms](https://scalewithsearch.com/for/family-and-criminal-defense-law-firms) - [Med spas and aesthetics practices](https://scalewithsearch.com/for/med-spas-and-aesthetics) - [Personal injury law firms](https://scalewithsearch.com/for/personal-injury-law-firms) - [Veterinary clinics](https://scalewithsearch.com/for/veterinary-clinics) - [Gyms and fitness studios](https://scalewithsearch.com/for/fitness) - [Therapy and outpatient health practices](https://scalewithsearch.com/for/therapy-and-outpatient-health) - [Medical billing companies](https://scalewithsearch.com/for/medical-billing) - [Insurance agencies](https://scalewithsearch.com/for/insurance-agencies) - [Financial advisors](https://scalewithsearch.com/for/financial-advisors) - [Property management companies](https://scalewithsearch.com/for/property-management) - [Recruiting and staffing agencies](https://scalewithsearch.com/for/recruiting-and-staffing) - [Architects and interior designers](https://scalewithsearch.com/for/architects-and-interior-designers) - [Logistics and supply chain companies](https://scalewithsearch.com/for/logistics-and-supply-chain) - Agencies, MSPs, and manufacturing - [IT and managed service providers](https://scalewithsearch.com/for/it-and-managed-service-providers) - [Machine shops and precision manufacturers](https://scalewithsearch.com/for/machine-shops-and-precision-manufacturing) - [Marketing agencies and freelancers](https://scalewithsearch.com/for/marketing-agencies-and-freelancers) - [SEO agencies and consultants](https://scalewithsearch.com/for/seo-agencies-and-consultants) - [Small manufacturers and fabricators](https://scalewithsearch.com/for/small-manufacturers-and-fabricators) - Restaurants, shops, studios, and nonprofits - [Restaurants and hospitality businesses](https://scalewithsearch.com/for/restaurants-and-hospitality) - [Retail stores and ecommerce sellers](https://scalewithsearch.com/for/retail-and-ecommerce) - [Photographers, event planners, and travel agents](https://scalewithsearch.com/for/photographers) - [Churches and nonprofits](https://scalewithsearch.com/for/churches-and-nonprofits) - [All industries](https://scalewithsearch.com/for/) - Learn - For your office - [Office job guides](https://scalewithsearch.com/guides/) - [Browser calculators](https://scalewithsearch.com/tools/) - Start here - [How it works](https://scalewithsearch.com/how-it-works) - [Free Starter Kit](https://scalewithsearch.com/kit/business-memory-starter-kit.zip) - [Synthetic specimen](https://scalewithsearch.com/specimen/working-session-specimen.zip) - Guides - [The Complete Guide to Business Memory for AI Agents](https://scalewithsearch.com/articles/business-memory-for-ai-agents-guide) - [The Complete Small-Business Guide to AI Agent Governance](https://scalewithsearch.com/articles/ai-agent-governance-guide-small-business) - [The Complete Guide to Leaving Vendor AI Memory](https://scalewithsearch.com/articles/leaving-vendor-ai-memory-guide) - Articles by cluster - [Business memory](https://scalewithsearch.com/articles/business-memory-for-ai-agents-guide) - [Agent governance](https://scalewithsearch.com/articles/ai-agent-governance-guide-small-business) - [Migration and ownership](https://scalewithsearch.com/articles/leaving-vendor-ai-memory-guide) - For machines - [llms.txt](https://scalewithsearch.com/llms.txt) - [llms-full.txt](https://scalewithsearch.com/llms-full.txt) - [Machine view](https://scalewithsearch.com/?view=machine) - Company - Evidence - [Proof](https://scalewithsearch.com/proof) - Company - [About](https://scalewithsearch.com/about) # Read server logs to see what Googlebot crawls and stop wasted crawl budget. A retailer adds 3,000 new product pages. Six weeks later, half of them are still not indexed. Search Console says Google "discovered" them but has not crawled them. The server logs show why: Googlebot spent most of its requests on filter URLs such as `/products?color=red&size=large`. Log file analysis shows how search engine crawlers actually use your site: which pages they request, how often, and where they hit errors. Google Search Console summarizes crawl activity. The server access log holds every request. This guide shows how to get the logs, parse them, read the patterns, and fix what they reveal. ## Know when logs matter Google says crawl budget is a concern mainly for two kinds of site. The first has about a million pages or more. The second has 10,000 or more pages whose content changes daily. Smaller sites can still use logs to debug indexation, find errors, and confirm a migration. Crawl budget is the set of URLs Googlebot can and wants to crawl on your site. Google describes two parts. The crawl capacity limit depends on how fast and healthy your server is. Crawl demand depends on how popular and fresh your URLs are. Search Console's Crawl Stats report has four limits compared with logs: - **Summary only.** It shows totals and example URLs, not every request. - **Grouped data.** It groups requests by response, file type, purpose, and Googlebot type, so URL-level patterns are hard to see. - **Delay.** Its data lags by a day or more. Logs are current. - **Google only.** It shows nothing about Bingbot or other crawlers. Logs surface five things the report hides: - **Crawl allocation.** Which pages Googlebot requests most, and which it ignores. - **Orphan pages.** URLs with no internal links that crawlers still find, through external links or sitemaps. - **Crawl traps.** URL patterns that generate endless variations, such as faceted filters or session IDs. - **Wasted crawls.** Requests for pages that cannot be indexed: admin panels, staging copies, and duplicates. - **Rendering gaps.** JavaScript pages where Googlebot fetches the HTML but may not fetch what it needs to render. Access logs contain IP addresses, which privacy law can treat as personal data. Limit access to the people who need it, and follow your retention policy. ## Get the logs The server writes one line for each HTTP request, crawler requests included. ### Apache Apache writes to `/var/log/apache2/access.log` or `/var/log/httpd/access_log`. A Common Log Format line looks like this: ```text 192.168.1.1 - - [08/Feb/2026:12:34:56 -0500] "GET /page.html HTTP/1.1" 200 2048 ``` The fields are the IP address, timestamp, HTTP method and URL, status code, and response size. ### Nginx Nginx writes to `/var/log/nginx/access.log`. The Combined Log Format adds the referrer and the user-agent: ```text 192.168.1.1 - - [08/Feb/2026:12:34:56 -0500] "GET /page.html HTTP/1.1" 200 2048 "-" "Mozilla/5.0..." ``` On a managed host or a CDN, ask the provider for raw access logs or a log export. Some plans do not include them. ### Identify crawlers by user-agent Search engine bots name themselves in the user-agent string: ```text Googlebot: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Googlebot Smartphone: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Bingbot: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) ``` Filter by user-agent to isolate crawler traffic. Anyone can fake a user-agent, so verify the IP before you trust the numbers. Run a reverse DNS lookup on the IP. A real Googlebot IP resolves to a host under `googlebot.com` or `google.com`. Then run a forward lookup on that host name and confirm that it returns the same IP. ```bash host 66.249.66.1 host crawl-66-249-66-1.googlebot.com ``` Google also publishes its crawler IP ranges as JSON files. If an IP claims to be Googlebot and fails both checks, block it. ### Handle rotation Servers rotate logs daily or weekly so files stay small. Analyze the last 30 to 90 days. Older logs are often compressed as `.gz` files. Decompress them before you parse: ```bash gunzip access.log.gz ``` ## Parse the logs Raw logs hold millions of lines. Start with the command line, then move to a dedicated tool as volume grows. ### Command-line checks These commands assume the Combined Log Format, where field 7 is the URL and field 9 is the status code. ```bash # Count all requests wc -l access.log # Save Googlebot requests to a separate file grep "Googlebot" access.log > googlebot.log # Count Googlebot requests grep -c "Googlebot" access.log # Top 10 URLs that Googlebot requests grep "Googlebot" access.log | cut -d' ' -f7 | sort | uniq -c | sort -rn | head -10 # Status codes that Googlebot receives grep "Googlebot" access.log | cut -d' ' -f9 | sort | uniq -c | sort -rn ``` Example status output: ```text 12450 200 2345 301 890 404 234 500 ``` In this sample, Googlebot hit 890 missing pages and 234 server errors. ### Python for custom analysis For analysis the command line cannot do, parse the file in Python: ```python import re from collections import Counter url_pattern = re.compile(r'"(?:GET|POST) (.*?) HTTP') url_counts = Counter() with open('access.log', 'r') as f: for line in f: if 'Googlebot' not in line: continue match = url_pattern.search(line) if match: url_counts[match.group(1)] += 1 for url, count in url_counts.most_common(20): print(f"{url}: {count}") ``` The script reads the log one line at a time, keeps Googlebot requests, and prints the 20 most requested URLs. ### Dedicated tools | Tool | Type | What it adds | Fit | |---|---|---|---| | Screaming Frog Log File Analyser | Desktop | Crawl frequency per URL, orphan pages, non-indexable crawls, status codes; joins logs with a site crawl | The free version handles one project and 1,000 log events (September 2026); the paid licence removes the limits | | Oncrawl | SaaS | Continuous log monitoring, Search Console correlation, segments by page type or template | Mid-size and large sites | | Botify | Enterprise | Joins crawl, log, and analytics data; finds crawl waste and rendering problems | Large sites, often 50,000 pages or more; custom pricing | | Splunk or the ELK Stack (Elasticsearch, Logstash, Kibana) | Self-hosted | Any query and dashboard you can build | Teams with infrastructure staff | For a small site, start with the free Screaming Frog Log File Analyser. For larger sites, compare the SaaS options on the segments you need. ## Read the patterns Six patterns carry most of the diagnostic value. The thresholds below are working rules of thumb, not Google figures. ### Crawl allocation 1. Export the request count for each URL. 2. Group URLs by type: product pages, blog posts, category pages, utility pages. 3. Calculate each group's share of Googlebot requests. Three red flags need action: - Thirty percent or more of requests go to pages that cannot be indexed, such as admin, staging, or internal search results. - Low-value pages, such as tags and archives, get more requests than products or landing pages. - Missing pages (404) take 10 percent or more of requests, which points to broken internal links. To fix allocation, block low-value sections in `robots.txt`, and remove internal links to pages that cannot be indexed. Fix broken links so crawlers stop hitting missing pages. Improve server speed so Google can crawl more. ### Orphan pages An orphan page has no internal links. Crawlers still reach it through external links or the XML sitemap. It ranks poorly, because no internal links pass authority to it. 1. Extract every URL Googlebot requested. 2. Compare the list with your sitemap and your internal link graph from a site crawl. 3. Flag URLs that Googlebot requests but no page links to. Example: Googlebot requests `/blog/old-post-2019/`. The link audit shows zero internal links. The post is still in the sitemap. Add internal links from relevant articles, or remove the page from the sitemap if it no longer has value. ### Crawl traps A crawl trap generates unlimited URLs. Four patterns cause most of them: - **Faceted navigation.** `/products?color=red&size=large&brand=nike` has thousands of combinations. - **Session IDs.** `/page?sessionid=abc123` creates a new URL for each visit. - **Calendars.** `/events?month=january&year=2026` with links to every past and future month. - **Unbounded pagination.** `/blog?page=1` through `/blog?page=9999`. Look for URL patterns with high request volume and many parameter variations: ```text /products?filter=a: 234 crawls /products?filter=b: 198 crawls /products?filter=c: 187 crawls ... ``` Fix a trap with one or more of these changes: - Block the parameter patterns in `robots.txt`. - Point canonical tags on filtered pages at the main category page. - Apply filters with JavaScript that does not change the URL. - Add `rel="nofollow"` to filter links. Google treats `nofollow` as a hint, so do not rely on it alone. The [faceted navigation guide](/articles/ecommerce-seo-faceted-navigation) covers the trade-offs for ecommerce filters. ### Status codes | Code | Meaning for the crawler | Action | |---|---|---| | 200 | The page loaded | None | | 301 | Permanent redirect; Googlebot follows it | Acceptable, but update internal links so crawlers skip the hop | | 302 or 307 | Temporary redirect | Use 301 or 308 for a permanent move | | 404 | Page not found | Fix the link that points here; return 410 for content you deleted on purpose | | 500 or 503 | Server error | Fix now; repeated errors make Google slow its crawl | Googlebot recrawls a 404 URL less often over time. The larger cost is the broken internal link that keeps sending crawlers and users to it. For redirect hops, see the [redirect chains guide](/articles/redirect-chains-301-developers). 1. Count status codes in the Googlebot requests. 2. Calculate the share that ends in an error: 404, 500, or 503. 3. Find the URL types behind the errors. Aim for an error share under 5 percent. Over 10 percent indicates a serious problem. ### Rendering A JavaScript page needs rendering. In the logs, rendering can show up as two stages: 1. Googlebot requests the HTML. 2. The rendering service requests the CSS, JavaScript, and images the page needs. HTML requests with no resource requests afterward can mean Googlebot did not render the page. Google caches resources heavily, so the absence is a weak signal. Confirm with the URL Inspection tool in Search Console. If rendering fails often, move to server-side rendering. The guide to [how Googlebot renders JavaScript](/articles/googlebot-javascript-rendering) explains the pipeline. ### Crawl rate over time Crawl rate changes with site speed, content freshness, and server health. Plot daily Googlebot requests for 90 days and look for anomalies. Healthy patterns: - a steady daily volume that stays within about 20 percent of its average; - a rise after you publish new content; - more frequent requests for pages that change often. Warning patterns: - a sudden drop of 50 percent or more, which often follows slow responses or server errors; - a spike followed by a drop, which can mean Googlebot hit a trap and then slowed down. Volume scales with site size, so compare the site with its own baseline, not with another site. Judge efficiency, meaning the share of requests that reach valuable pages, more than the raw count. ## Fix the waste You cannot ask Google for more crawl budget. You can make each request count and let the server answer faster. ### Tune robots.txt Block low-value sections: ```text User-agent: * Disallow: /admin/ Disallow: /search? Disallow: /cart/ Disallow: /checkout/ Disallow: /*?sort= Disallow: /*?filter= ``` This keeps crawlers out of admin pages, internal search results, and sorted or filtered product lists. The pattern `/*?sort=` matches only when `sort` is the first parameter. Use `/*sort=` to match it in any position. You can add a group for Googlebot alone: ```text User-agent: Googlebot Disallow: /admin/ Allow: /products/ Allow: /blog/ ``` A crawler obeys only the most specific group that names it. With this file, Googlebot ignores the `User-agent: *` rules. Repeat every rule Googlebot needs inside its own group. The [robots.txt and crawl budget guide](/articles/robots-txt-crawl-budget-developers) covers the syntax in full. ### Make the server faster Google raises its crawl capacity limit when the server responds quickly and without errors. Five changes help: - Add server-side caching, such as Redis or Varnish. - Serve through a CDN to cut latency. - Speed up slow database queries. - Compress responses. - Target a time to first byte (TTFB) under 200 ms. ### Change internal links Add internal links to valuable pages that Googlebot rarely requests. Pages linked from the homepage or main navigation get crawled more often. Remove internal links to low-value pages such as tags, archives, and utility pages. ### Keep sitemaps accurate Many guides tell you to set `` and `` in the XML sitemap. Google ignores both values. It uses `` when the value is consistently accurate: ```xml https://example.com/high-priority-page/ 2026-02-08 ``` List only canonical, indexable URLs, and update `lastmod` only when the content changes. ## Set a cadence Analyze logs every month for sites with 10,000 or more pages, and every quarter for smaller sites. Run an extra pass after any migration or redesign. Do not block Bingbot to save resources unless the server is under real strain. Bing sends its own search traffic and supplies results to Yahoo and DuckDuckGo. Its crawl rate is much lower than Googlebot's. ---- ```text .|########||. .|########||. .|########||. |##||. .||##|. |##||. .||##|. |##||. .||##|. |#|. .|#|. |#|. .|#|. |#|. .|#|. |#| |#| |#| |#| |#| |#| .#| |#. .#| |#. .#| |#. .#. .#| .#. .#| .#. .#| |#. .#| |#. .#| |#. .#| |# ...... #| |# ...... #| |# ...... #| .# ||#########| #| .# ||#########| #| .# ||#########| #| .#. |######||######|. .#. .#. |######||######|. .#. .#. |######||######|. .#. #. .##|###|##|#|######| .# #. .##|###|##|#|######| .# #. .##|###|##|#|######| .# || |##|#||||||||||||#||#| || || |##|#||||||||||||#||#| || || |##|#||||||||||||#||#| || # |#||||||||||||||||||||#| #. # |#||||||||||||||||||||#| #. # |#||||||||||||||||||||#| #. || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || # .#||||||||||||||||||||||||#| # # .#||||||||||||||||||||||||#| # # .#||||||||||||||||||||||||#| # || ....|||#||||||##|#|||#|#||##||||....|. || || ....|||#||||||##|#|||#|#||##||||....|. || || ....|||#||||||##|#|||#|#||##||||....|. || #. . ....|# ....#||| ||| .#||| ....#. .# #. . ....|# ....#||| ||| .#||| ....#. .# #. . ....|# ....#||| ||| .#||| ....#. .# # . ||||#| .#####|| . .#| .#||| ||||# #. # . ||||#| .#####|| . .#| .#||| ||||# #. # . ||||#| .#####|| . .#| .#||| ||||# #. .| ||||| #. |#|||||. || #. |#||. .||||| || .| ||||| #. |#|||||. || #. |#||. .||||| || .| ||||| #. |#|||||. || #. |#||. .||||| || || |.... #. ....|#. |. ...|. ....|| || || |.... #. ....|#. |. ...|. ....|| || || |.... #. ....|#. |. ...|. ....|| || #. .||||||##||||||##||####|||||||#||||||#| .# #. .||||||##||||||##||####|||||||#||||||#| .# #. .||||||##||||||##||####|||||||#||||||#| .# # .||####|###########################|. # # .||####|###########################|. # # .||####|###########################|. # # .#||||||.#.|| # |. ..# #| #|||||#| # # .#||||||.#.|| # |. ..# #| #|||||#| # # .#||||||.#.|| # |. ..# #| #|||||#| # # .#|||||| .. |# ## |#| ...#|#|||#| # # .#|||||| .. |# ## |#| ...#|#|||#| # # .#|||||| .. |# ## |#| ...#|#|||#| # # .#|||#|# .# .#| #| ##|.#..#|||||#| # # .#|||#|# .# .#| #| ##|.#..#|||||#| # # .#|||#|# .# .#| #| ##|.#..#|||||#| # # .#||||||############|######|||||#| # # .#||||||############|######|||||#| # # .#||||||############|######|||||#| # # |...||#||||||#||||#|#||||||#|||||#||| |.| # # |...||#||||||#||||#|#||||||#|||||#||| |.| # # |...||#||||||#||||#|#||||||#|||||#||| |.| # #. .. ||||# .|||##|. |#| .|| || .|||#. #| # .# #. .. ||||# .|||##|. |#| .|| || .|||#. #| # .# #. .. ||||# .|||##|. |#| .|| || .|||#. #| # .# || | ...|| ...##| | #| .|. |. ##### .. .| || || | ...|| ...##| | #| .|. |. ##### .. .| || || | ...|| ...##| | #| .|. |. ##### .. .| || || ||||| || ||||#| . |. |. |#. ||||| |#| || || || ||||| || ||||#| . |. |. |#. ||||| |#| || || || ||||| || ||||#| . |. |. |#. ||||| |#| || || .# |.....#|....|#.||||.|||##.|#|....||.#||.#. #. .# |.....#|....|#.||||.|||##.|#|....||.#||.#. #. .# |.....#|....|#.||||.|||##.|#|....||.#||.#. #. #.|||||||#############################| ||| .# #.|||||||#############################| ||| .# #.|||||||#############################| ||| .# || ##||#||#||||||#||#||#||#||##. || || ##||#||#||||||#||#||#||#||##. || || ##||#||#||||||#||#||#||#||##. || # .#|||||||||#||#|||||||||||#| # # .#|||||||||#||#|||||||||||#| # # .#|||||||||#||#|||||||||||#| # || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || .# |#||||||||||||||||||||#| #. .# |#||||||||||||||||||||#| #. .# |#||||||||||||||||||||#| #. || |#||||||||||||||||||#| || || |#||||||||||||||||||#| || || |#||||||||||||||||||#| || #. .######|#|#########| .# #. .######|#|#########| .# #. .######|#|#########| .# .#. |#####||||#####| .#. .#. |#####||||#####| .#. .#. |#####||||#####| .#. |# ||########|| #. |# ||########|| #. |# ||########|| #. |# ...... #| |# ...... #| |# ...... #| |#. .#| |#. .#| |#. .#| |#. .#. |#. .#. |#. .#. .#| |#. .#| |#. .#| |#. |#| |#| |#| |#| |#| |#| .|#|. .|#|. .|#|. .|#|. .|#|. .|#|. |##||. .||##| |##||. .||##| |##||. .||##| .||########||. .||########||. .||########||. Scale With Search 2026 [scalewithsearch.com](https://scalewithsearch.com) ```