--- title: "Missing Common Crawl captures: coverage versus page deletion" description: "Separate empty Common Crawl results, failed requests, unavailable collections, and captured errors before reporting a deleted page or website change." canonical: "https://scalewithsearch.com/articles/common-crawl-missing-capture-coverage-not-page-deletion" --- ## Site navigation - [Scale With Search](https://scalewithsearch.com/) - Services - Services - [AI systems](https://scalewithsearch.com/ai-systems) - [Site build](https://scalewithsearch.com/work#site-build) - [Whole website fix](https://scalewithsearch.com/work#whole-website-fix) - [Content library with SEO](https://scalewithsearch.com/work#content-library) - [Working session](https://scalewithsearch.com/audit) - Start here - [Send your brief](https://scalewithsearch.com/work#send-your-brief) - [Example scope](https://scalewithsearch.com/work#review-the-example-scope) - [Prices and terms](https://scalewithsearch.com/work#prices-and-terms) - Industries - Real estate - [Real estate agents and teams](https://scalewithsearch.com/for/real-estate) - [Platforms, ownership, and IDX](https://scalewithsearch.com/for/real-estate/real-estate-website-platforms-ownership-idx) - [Search, content, and AI](https://scalewithsearch.com/for/real-estate/real-estate-search-content-ai) - [Inquiries, campaigns, and handoffs](https://scalewithsearch.com/for/real-estate/real-estate-website-inquiries-campaigns-handoffs) - [Buyer, seller, and local pages](https://scalewithsearch.com/for/real-estate/real-estate-buyer-seller-local-resources) - [Rules, privacy, and trust](https://scalewithsearch.com/for/real-estate/real-estate-website-rules-privacy-trust) - Browse by business - [Trades and home services](https://scalewithsearch.com/for/#group-trades) - [Health, wellness, and animal care](https://scalewithsearch.com/for/#group-practices) - [Law, finance, and IT](https://scalewithsearch.com/for/#group-professional) - [Marketing and SEO agencies](https://scalewithsearch.com/for/#group-agencies) - [Shops and manufacturers](https://scalewithsearch.com/for/#group-manufacturing) - [Restaurants, shops, studios, and nonprofits](https://scalewithsearch.com/for/#group-local-services) - Chiropractic practices - [Chiropractic and wellness practices](https://scalewithsearch.com/for/chiropractic-and-wellness-practices) - [Chiropractic and physical therapy clinics](https://scalewithsearch.com/for/chiropractic-and-physical-therapy-clinics) - [All industries](https://scalewithsearch.com/for/) - Resources - Free tools - [AI readiness check](https://scalewithsearch.com/check) - [Free tools](https://scalewithsearch.com/tools/) - [Free Starter Kit](https://scalewithsearch.com/kit/business-memory-starter-kit.zip) - Guides - [AI-native library](https://scalewithsearch.com/ai-native) - [Industry intelligence](https://scalewithsearch.com/intelligence) - [Office job guides](https://scalewithsearch.com/guides/) - [Business memory](https://scalewithsearch.com/articles/business-memory-for-ai-agents-guide) - [Agent governance](https://scalewithsearch.com/articles/ai-agent-governance-guide-small-business) - [Migration and ownership](https://scalewithsearch.com/articles/leaving-vendor-ai-memory-guide) - Articles - [All articles](https://scalewithsearch.com/articles/) - [What changed](https://scalewithsearch.com/changes/) - About - Evidence - [Proof](https://scalewithsearch.com/proof) - Scale With Search - [About](https://scalewithsearch.com/about) - [How it works](https://scalewithsearch.com/how-it-works) # Missing Common Crawl captures: distinguish archive coverage from a deleted page An empty archive result can leave a research question unanswered. It does not establish that a website owner deleted a page. Before writing a site-change finding, identify what the lookup covered and whether it completed successfully. Common Crawl describes its dataset as a sample of the web. Its FAQ explains that it generally captures a subset of a website. A missing result therefore needs a coverage qualification. [Common Crawl FAQ, checked 2026.10.28](https://commoncrawl.org/faq). For [SEO agencies and consultants](/for/seo-agencies-and-consultants), the practical deliverable is a bounded evidence note. It should tell the next analyst what was inspected, what happened, and which business claim remains unknown. The workflow below is an editorial recommendation. All examples use invented records and reserved example addresses. They do not document an actual competitor change or a completed Common Crawl query. IANA maintains `example.com` for documentation. [IANA example-domain guidance, checked 2026.10.28](https://www.iana.org/help/example-domains). ## Classify the lookup outcome before interpreting absence Use four separate outcome labels in your research sheet. An empty successful query, an unavailable collection, a failed request, and a captured HTTP error describe different situations. A successful empty query means the checked response contained no matching records within the documented scope. This classification requires evidence that the request completed as intended. A blank screen alone does not establish that condition. An unavailable collection means the intended source could not be inspected at that time. Record the collection identifier and the access evidence. Do not quietly replace the unavailable source with another collection in your conclusion. A failed request means the attempt did not produce a usable research result. Preserve the response status or failure detail. Leave the coverage question unresolved instead of recording zero captures. A captured HTTP error means a returned capture record contains a historical error status. Common Crawl defines `status` as the response code from fetching the captured page. That is separate from the status of your index request. [Common Crawl CDXJ definitions](https://www.commoncrawl.org/cdxj-index). | Synthetic situation | Recommended worksheet label | Appropriate next question | | --- | --- | --- | | Completed query with no matching records | No matching records in inspected scope | Does the query scope fit the research question? | | Intended collection could not be accessed | Collection unavailable for this attempt | Can the permitted source be checked later? | | Lookup failed before a usable result | Request failure | What failure was returned? | | Returned capture includes status `404` | Historical captured error | What can independent current evidence establish? | These labels prevent operational failures from becoming false historical findings. They also show a reviewer where more work might change the conclusion. ## Name the collection boundary Common Crawl's CDXJ documentation explains that collection indexes are separate. It does not provide one CDXJ index covering every monthly crawl. The [index directory](https://index.commoncrawl.org/) supplies collection selection and links the collection list. [Common Crawl collection guidance](https://www.commoncrawl.org/cdxj-index). Write down the actual collection identifier selected for each permitted lookup. If you inspect two collections, list both. A result from one collection cannot stand in for an uninspected one. For this guide, use `HYPOTHETICAL-SPRING` and `HYPOTHETICAL-SUMMER` as teaching labels. They are invented and are not Common Crawl collection identifiers. Suppose the spring example contains no match and the summer example contains one record. The resulting exercise supports a narrow comparison: the supplied spring result lacks a match, while the supplied summer result includes one. It supplies no date when a website owner created the page. Do not call the summer record the page's first appearance on the internet. Do not call the spring result proof that the page was absent. Both statements would exceed the supplied evidence. When the first available result matters, use the wording “earliest observed within the inspected scope.” List the scope nearby. If earlier collections remain unchecked, preserve that limitation in the summary. ## Preserve the URL you asked about A route name can seem familiar while the queried address differs from the address under review. Keep the raw query input and returned URLs. Do not rely on an analyst's shorthand such as “the service page.” Consider these synthetic candidate addresses: - `https://example.com/services/site-review` - `https://www.example.com/services/site-review` - `http://example.com/services/site-review` - `https://example.com/services/site-review/` - `https://example.com/services/site-review?source=guide` This list defines possible review questions. It does not claim that Common Crawl stores every variant separately or that every query treats them identically. Inspect the actual matching behavior and returned records. Use a separate field for the match mode. The [CDX API reference](https://github.com/webrecorder/pywb/wiki/CDX-Server-API) documents exact, prefix, host, and domain options. It also documents date bounds, field filters, limits, and pagination. Those options belong in the evidence note because they shape the result being interpreted. An exact lookup and a host-wide lookup answer different scoped questions. Record which one you performed. Avoid silently broadening the question after an empty response. If a broader permitted lookup is necessary, describe it as another research step. Preserve both outcomes so a reviewer can trace the reasoning. ## Record the assumptions that can hide a relevant record Filters are useful when the research question justifies them. They become misleading when the report omits them and describes the filtered result as complete history. For example, suppose a synthetic query accepts only status `200`. A supplied capture with status `301` would fall outside that exercise's filter. Calling the empty filtered result “no history for this URL” would overstate the scope. A date range can create the same reporting problem. If the review covers April, say April. Do not let a narrow interval become a statement about the site's entire existence. A result limit also belongs in the research record. When a response is truncated or its pagination remains incomplete, describe that limitation. Do not announce an earliest observation until the inspected ordering and scope support it. Use a small assumptions table before accepting a no-match statement: | Query detail | Record this | Review question | | --- | --- | --- | | Collection | Verified identifier selected | Was this the intended collection? | | URL input | Exact submitted value | Does it represent the address in the brief? | | Match mode | Actual selected mode | Was the scope exact or broader? | | Date range | Bounds or no date restriction | Does the sentence preserve that interval? | | Filters | Every active field condition | Could the question require excluded records? | | Limits and pagination | Applied limits and inspected pages | Was the inspected result complete for its stated purpose? | This is a reporting checklist, not a bulk retrieval plan. Keep the lookup bounded to the approved question. A broader corpus scan needs a different design and access review. ## Handle source access as its own evidence An index service problem does not describe the target website. Keep the index-request outcome separate from any captured site's historical response code. This distinction should remain visible in both the worksheet and the client summary. Common Crawl warns that its CDX service is rate limited. Its FAQ asks users to pause between calls, avoid concurrent threads on one IP, and avoid proxy networks. [Common Crawl access guidance](https://commoncrawl.org/faq). If a permitted lookup fails, preserve the failure and follow the publisher's current guidance. Do not retry across parallel workers to force an answer. A failed refresh should leave an explicit incomplete result. For the proposed workflow, the research owner decides whether another attempt is justified. The task may close with source access unresolved. That is a usable deliverable when it states what was inspected and what remains unknown. Do not replace the missing result with a search snippet or a remembered claim. Such material may identify another research lead, but it cannot certify what the failed collection lookup would have returned. ## Seek current evidence through a separate question A present route check and an archive lookup belong in separate rows. Give each its own observation time, method, and result. This prevents a current finding from rewriting the meaning of historical evidence. If the task permits a current check, inspect the exact approved address. Record the observed response and any relevant destination separately. Avoid adding an explanation for why the response occurred unless another source establishes it. Even a current error would not by itself provide a deletion date or the owner's intent. Keep “current response observed” distinct from “page deleted on a stated day.” The latter needs historical and contextual support. For a website your client owns, ask for its release history or migration record. Request a specific dated artifact that answers the question. Do not ask the client to endorse a conclusion already invented from the empty lookup. For another website, a verified public announcement may answer a bounded question. Preserve the announcement's exact scope and date. Do not copy article bodies or gather personal information to fill an archive gap. When a positive capture is available, use the companion guide to [Common Crawl capture timestamps](/articles/common-crawl-capture-timestamp-not-launch-date). It explains how to report that observation without converting it into a launch date. ## Write an explicit no-capture note The following note is synthetic. It describes the supplied exercise assumptions rather than a real request: > On 2026.10.28, the analyst reviewed the hypothetical successful result for `https://example.com/services/site-review` in HYPOTHETICAL-SPRING. The exercise used exact matching, no date bounds, no status filter, and no incomplete pagination. The supplied result contained no matching records. Other collections and URL variants were not inspected. Publication time, deletion, current availability, and the reason for missing coverage remain unknown. A failure note should use different wording. For example: “The intended collection was unavailable during this attempt; capture coverage was not established.” Include the actual access evidence when documenting a real attempt. The distinction matters when the research passes to another colleague. The next person should know whether to revisit query scope or wait for a source to become accessible. A single blank cell cannot communicate that difference. Store the note with the query record and review decision. The [business memory guide](/articles/business-memory-for-ai-agents-guide) describes preserving working context in owned records. Here, the retained correction history should explain why an earlier interpretation was accepted or withdrawn. ## Decide whether the finding deserves an action Before escalating a competitor-change finding, ask whether the evidence establishes a business event. A no-match archive observation may justify another research question. It does not justify a factual claim of launch, deletion, Google indexing, or lost traffic. Use the [agency intelligence overview](/intelligence/seo-agencies-and-consultants) to place this research inside a bounded review process. The responsible analyst decides whether the unresolved question matters enough for further permitted investigation. Scale With Search scopes evidence collection and review around a defined decision. Bring the question, approved sources, and required output to the scope discussion. [Discuss your scope](/work) Continue the website review: [AI-native SEO agencies: the first functions to build](/articles/ai-native-seo-agencies-first-functions); [Common Crawl timestamps: report an observed capture without inventing a launch date](/articles/common-crawl-capture-timestamp-not-launch-date); [SEO agency AI rules: data, approvals, and spend](/articles/seo-agency-ai-rules-data-approvals-spend); [AI search service page: define your agency offer](/articles/seo-agency-ai-search-service-page); [Disavow service pages: qualify the SEO inquiry](/articles/seo-agency-disavow-service-page); [Free SEO audit tools: state what the scan checks](/articles/seo-agency-free-audit-tool-boundaries); [Guest contributions on your SEO agency blog](/articles/seo-agency-guest-contribution-policy); [SEO pricing calculators: estimate before quoting](/articles/seo-agency-pricing-calculator-website); [Public corrections on your SEO agency blog](/articles/seo-agency-public-article-corrections); [Rank report demos on an SEO agency website](/articles/seo-agency-rank-report-demo); [Schema generators on an SEO agency website](/articles/seo-agency-schema-generator-tool-page); [SEO agency access page: request Search Console safely](/articles/seo-agency-search-console-access-page); [SEO workshop pages: explain the training before booking](/articles/seo-agency-training-workshop-page); [SEO audit sample: show buyers your actual deliverable](/articles/seo-agency-website-audit-sample). [SEO agencies and consultants](/for/seo-agencies-and-consultants): website and search resources. ## Related: SEO agencies and consultants - [SEO agency AI rules: data, approvals, and spend](/articles/seo-agency-ai-rules-data-approvals-spend) - [AI search service page: define your agency offer](/articles/seo-agency-ai-search-service-page) - [Disavow service pages: qualify the SEO inquiry](/articles/seo-agency-disavow-service-page) ---- ```text .|########||. .|########||. .|########||. |##||. .||##|. |##||. .||##|. |##||. .||##|. |#|. .|#|. |#|. .|#|. |#|. .|#|. |#| |#| |#| |#| |#| |#| .#| |#. .#| |#. .#| |#. .#. .#| .#. .#| .#. .#| |#. .#| |#. .#| |#. .#| |# ...... #| |# ...... #| |# ...... #| .# ||#########| #| .# ||#########| #| .# ||#########| #| .#. |######||######|. .#. .#. |######||######|. .#. .#. |######||######|. .#. #. .##|###|##|#|######| .# #. .##|###|##|#|######| .# #. .##|###|##|#|######| .# || |##|#||||||||||||#||#| || || |##|#||||||||||||#||#| || || |##|#||||||||||||#||#| || # |#||||||||||||||||||||#| #. # |#||||||||||||||||||||#| #. # |#||||||||||||||||||||#| #. || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || # .#||||||||||||||||||||||||#| # # .#||||||||||||||||||||||||#| # # .#||||||||||||||||||||||||#| # || ....|||#||||||##|#|||#|#||##||||....|. || || ....|||#||||||##|#|||#|#||##||||....|. || || ....|||#||||||##|#|||#|#||##||||....|. || #. . ....|# ....#||| ||| .#||| ....#. .# #. . ....|# ....#||| ||| .#||| ....#. .# #. . ....|# ....#||| ||| .#||| ....#. .# # . ||||#| .#####|| . .#| .#||| ||||# #. # . ||||#| .#####|| . .#| .#||| ||||# #. # . ||||#| .#####|| . .#| .#||| ||||# #. .| ||||| #. |#|||||. || #. |#||. .||||| || .| ||||| #. |#|||||. || #. |#||. .||||| || .| ||||| #. |#|||||. || #. |#||. .||||| || || |.... #. ....|#. |. ...|. ....|| || || |.... #. ....|#. |. ...|. ....|| || || |.... #. ....|#. |. ...|. ....|| || #. .||||||##||||||##||####|||||||#||||||#| .# #. .||||||##||||||##||####|||||||#||||||#| .# #. .||||||##||||||##||####|||||||#||||||#| .# # .||####|###########################|. # # .||####|###########################|. # # .||####|###########################|. # # .#||||||.#.|| # |. ..# #| #|||||#| # # .#||||||.#.|| # |. ..# #| #|||||#| # # .#||||||.#.|| # |. ..# #| #|||||#| # # .#|||||| .. |# ## |#| ...#|#|||#| # # .#|||||| .. |# ## |#| ...#|#|||#| # # .#|||||| .. |# ## |#| ...#|#|||#| # # .#|||#|# .# .#| #| ##|.#..#|||||#| # # .#|||#|# .# .#| #| ##|.#..#|||||#| # # .#|||#|# .# .#| #| ##|.#..#|||||#| # # .#||||||############|######|||||#| # # .#||||||############|######|||||#| # # .#||||||############|######|||||#| # # |...||#||||||#||||#|#||||||#|||||#||| |.| # # |...||#||||||#||||#|#||||||#|||||#||| |.| # # |...||#||||||#||||#|#||||||#|||||#||| |.| # #. .. ||||# .|||##|. |#| .|| || .|||#. #| # .# #. .. ||||# .|||##|. |#| .|| || .|||#. #| # .# #. .. ||||# .|||##|. |#| .|| || .|||#. #| # .# || | ...|| ...##| | #| .|. |. ##### .. .| || || | ...|| ...##| | #| .|. |. ##### .. .| || || | ...|| ...##| | #| .|. |. ##### .. .| || || ||||| || ||||#| . |. |. |#. ||||| |#| || || || ||||| || ||||#| . |. |. |#. ||||| |#| || || || ||||| || ||||#| . |. |. |#. ||||| |#| || || .# |.....#|....|#.||||.|||##.|#|....||.#||.#. #. .# |.....#|....|#.||||.|||##.|#|....||.#||.#. #. .# |.....#|....|#.||||.|||##.|#|....||.#||.#. #. #.|||||||#############################| ||| .# #.|||||||#############################| ||| .# #.|||||||#############################| ||| .# || ##||#||#||||||#||#||#||#||##. || || ##||#||#||||||#||#||#||#||##. || || ##||#||#||||||#||#||#||#||##. || # .#|||||||||#||#|||||||||||#| # # .#|||||||||#||#|||||||||||#| # # .#|||||||||#||#|||||||||||#| # || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || .# |#||||||||||||||||||||#| #. .# |#||||||||||||||||||||#| #. .# |#||||||||||||||||||||#| #. || |#||||||||||||||||||#| || || |#||||||||||||||||||#| || || |#||||||||||||||||||#| || #. .######|#|#########| .# #. .######|#|#########| .# #. .######|#|#########| .# .#. |#####||||#####| .#. .#. |#####||||#####| .#. .#. |#####||||#####| .#. |# ||########|| #. |# ||########|| #. |# ||########|| #. |# ...... #| |# ...... #| |# ...... #| |#. .#| |#. .#| |#. .#| |#. .#. |#. .#. |#. .#. .#| |#. .#| |#. .#| |#. |#| |#| |#| |#| |#| |#| .|#|. .|#|. .|#|. .|#|. .|#|. .|#|. |##||. .||##| |##||. .||##| |##||. .||##| .||########||. .||########||. .||########||. Scale With Search 2026 [scalewithsearch.com](https://scalewithsearch.com) ```