--- title: "Generate XML sitemaps that list only indexable URLs and help search engines find changes fast" description: "Sitemap limits, what Google reads and ignores, dynamic generation, sitemap indexes, image, video, and hreflang extensions, validation, and framework setups." canonical: "https://scalewithsearch.com/articles/xml-sitemap-best-practices-developers" date: "2026-03-20" modified: "2026-09-25" --- ## Site navigation - [Scale With Search](https://scalewithsearch.com/) - Real estate - Real estate - [Real estate](https://scalewithsearch.com/for/real-estate) - Work - Start here - [Send your brief](https://scalewithsearch.com/work#send-your-brief) - [Prepare your six-question brief](https://scalewithsearch.com/work#prepare-your-six-question-brief) - Build - [Site build, content library with SEO, signal desk](https://scalewithsearch.com/work) - For your business - Trades and home services - [Auto body and collision shops](https://scalewithsearch.com/for/auto-body-and-collision-shops) - [Foundation and home repair contractors](https://scalewithsearch.com/for/foundation-and-home-repair) - [Garage door and fencing contractors](https://scalewithsearch.com/for/garage-door-and-fencing-contractors) - [HVAC contractors](https://scalewithsearch.com/for/hvac-contractors) - [Janitorial and commercial cleaning companies](https://scalewithsearch.com/for/janitorial-and-commercial-cleaning) - [Locksmiths](https://scalewithsearch.com/for/locksmiths) - [Moving companies](https://scalewithsearch.com/for/moving-companies) - [Pest control companies](https://scalewithsearch.com/for/pest-control-companies) - [Plumbing and electrical contractors](https://scalewithsearch.com/for/plumbing-and-electrical-contractors) - [Restoration and water or fire damage companies](https://scalewithsearch.com/for/restoration-and-water-fire-damage) - [Roofing companies](https://scalewithsearch.com/for/roofing-companies) - [Towing companies](https://scalewithsearch.com/for/towing-companies) - [Tree services and landscaping companies](https://scalewithsearch.com/for/tree-services-and-landscaping) - [Solar installers](https://scalewithsearch.com/for/solar-installation) - [General contractors](https://scalewithsearch.com/for/general-contractors-and-construction) - [Paving, concrete, and flooring contractors](https://scalewithsearch.com/for/paving) - Practices and professional services - [Bookkeeping and tax practices](https://scalewithsearch.com/for/bookkeeping-and-tax-practices) - [Dental practices](https://scalewithsearch.com/for/dental-practices) - [Family and criminal defense law firms](https://scalewithsearch.com/for/family-and-criminal-defense-law-firms) - [Med spas and aesthetics practices](https://scalewithsearch.com/for/med-spas-and-aesthetics) - [Personal injury law firms](https://scalewithsearch.com/for/personal-injury-law-firms) - [Veterinary clinics](https://scalewithsearch.com/for/veterinary-clinics) - [Gyms and fitness studios](https://scalewithsearch.com/for/fitness) - [Therapy and outpatient health practices](https://scalewithsearch.com/for/therapy-and-outpatient-health) - [Medical billing companies](https://scalewithsearch.com/for/medical-billing) - [Insurance agencies](https://scalewithsearch.com/for/insurance-agencies) - [Financial advisors](https://scalewithsearch.com/for/financial-advisors) - [Property management companies](https://scalewithsearch.com/for/property-management) - [Recruiting and staffing agencies](https://scalewithsearch.com/for/recruiting-and-staffing) - [Architects and interior designers](https://scalewithsearch.com/for/architects-and-interior-designers) - [Logistics and supply chain companies](https://scalewithsearch.com/for/logistics-and-supply-chain) - Agencies, MSPs, and manufacturing - [IT and managed service providers](https://scalewithsearch.com/for/it-and-managed-service-providers) - [Machine shops and precision manufacturers](https://scalewithsearch.com/for/machine-shops-and-precision-manufacturing) - [Marketing agencies and freelancers](https://scalewithsearch.com/for/marketing-agencies-and-freelancers) - [SEO agencies and consultants](https://scalewithsearch.com/for/seo-agencies-and-consultants) - [Small manufacturers and fabricators](https://scalewithsearch.com/for/small-manufacturers-and-fabricators) - Restaurants, shops, studios, and nonprofits - [Restaurants and hospitality businesses](https://scalewithsearch.com/for/restaurants-and-hospitality) - [Retail stores and ecommerce sellers](https://scalewithsearch.com/for/retail-and-ecommerce) - [Photographers, event planners, and travel agents](https://scalewithsearch.com/for/photographers) - [Churches and nonprofits](https://scalewithsearch.com/for/churches-and-nonprofits) - [All industries](https://scalewithsearch.com/for/) - Learn - For your office - [Office job guides](https://scalewithsearch.com/guides/) - [Browser calculators](https://scalewithsearch.com/tools/) - Start here - [How it works](https://scalewithsearch.com/how-it-works) - [Free Starter Kit](https://scalewithsearch.com/kit/business-memory-starter-kit.zip) - [Synthetic specimen](https://scalewithsearch.com/specimen/working-session-specimen.zip) - Guides - [The Complete Guide to Business Memory for AI Agents](https://scalewithsearch.com/articles/business-memory-for-ai-agents-guide) - [The Complete Small-Business Guide to AI Agent Governance](https://scalewithsearch.com/articles/ai-agent-governance-guide-small-business) - [The Complete Guide to Leaving Vendor AI Memory](https://scalewithsearch.com/articles/leaving-vendor-ai-memory-guide) - Articles by cluster - [Business memory](https://scalewithsearch.com/articles/business-memory-for-ai-agents-guide) - [Agent governance](https://scalewithsearch.com/articles/ai-agent-governance-guide-small-business) - [Migration and ownership](https://scalewithsearch.com/articles/leaving-vendor-ai-memory-guide) - For machines - [llms.txt](https://scalewithsearch.com/llms.txt) - [llms-full.txt](https://scalewithsearch.com/llms-full.txt) - [Machine view](https://scalewithsearch.com/?view=machine) - Company - Evidence - [Proof](https://scalewithsearch.com/proof) - Company - [About](https://scalewithsearch.com/about) # Generate XML sitemaps that list only indexable URLs and help search engines find changes fast. An XML sitemap is a machine-readable list of the URLs you want search engines to crawl. A good sitemap helps Google find new and changed pages sooner, especially on a large site or a site with weak internal links. A bad sitemap lists redirects, blocked pages, and duplicates. Google then gives the file less weight. A sitemap does not improve rankings directly. It helps pages get discovered and indexed, which a page needs before it can rank. Content quality and technical health decide the rankings. This article covers the protocol, what Google reads and ignores, dynamic generation, extensions, validation, and setup in common frameworks. ## Follow the protocol limits The sitemap protocol at sitemaps.org defines the format. These rules are fixed: | Rule | Requirement | |---|---| | Namespace | `http://www.sitemaps.org/schemas/sitemap/0.9` | | Encoding | UTF-8, declared in `` | | Size per file | 50,000 URLs or 50 MB uncompressed, whichever comes first | | URL format | Absolute URLs with protocol and host, such as `https://www.example.com/page`, not `/page` | | Special characters | Entity-escaped: `&` becomes `&` | | Required tag per URL | `` | | Optional tags | ``, ``, `` | | Compression | Gzip is allowed; the 50 MB limit applies to the uncompressed file | A minimal sitemap looks like this: ```xml https://www.example.com/products/blue-widget 2026-09-20T14:05:00+00:00 ``` The protocol also limits scope by location. A sitemap at `https://www.example.com/blog/sitemap.xml` may list only URLs under `/blog/`. Put the main sitemap or the sitemap index at the root. Google accepts sitemaps for other paths or hosts when you verify ownership in Search Console. ## Know what Google reads and what it ignores Google ignores `` and ``. The protocol still defines them. Priority runs from 0.0 to 1.0 with a default of 0.5. Changefreq takes `always`, `hourly`, `daily`, `weekly`, `monthly`, `yearly`, or `never`. Other search engines may read them as hints. Do not spend engineering time tuning them, and do not claim `daily` for pages that change once a year. Google does use ``, if the values are consistently and verifiably accurate. Accurate means the date of the last meaningful change to the content: the main text, structured data, or links. A new footer year or a rebuilt cache is not a change. Write `lastmod` in W3C Datetime format. The full form is `YYYY-MM-DDThh:mm:ss+00:00`, and the shortest valid form is `YYYY-MM-DD`. Use one time zone for all values; UTC is simplest. Take the date from the content database, such as an `updated_at` column that changes only when an editor saves real changes. Do not use file modification times from the server. A deploy touches every file and makes every page look new. ## List only indexable, canonical URLs Every URL in a sitemap should return `200`, be indexable, and be the canonical version of its page. Exclude these: - URLs with a `noindex` tag. A listed `noindex` URL sends a conflicting signal. - URLs blocked in `robots.txt`. [Robots.txt and crawl budget](/articles/robots-txt-crawl-budget-developers) covers how the two files interact. - Redirecting URLs. List the final destination only. - Duplicate and alternate versions. List the URL that the page's canonical tag names. [Canonical tags for developers](/articles/canonical-tags-developers) covers the canonical rules. - URLs with tracking parameters, session IDs, or sort and filter parameters that create duplicates. - Drafts, private content, login pages, admin areas, and thin pages with little value. Pick one URL format and use it everywhere: `https`, one host (with or without `www`), and one trailing-slash rule. The sitemap, the canonical tags, and the internal links must agree. ## Generate the sitemap from live data A static sitemap is out of date on a dynamic site as soon as someone publishes. Generate it from the same database that serves the pages. 1. Query only published, public, indexable records. 2. Build each URL from the canonical slug, in the canonical format. 3. Take `lastmod` from the record's content-change timestamp. 4. Split the result into files of no more than 50,000 URLs. 5. Write the files and the sitemap index to storage or a cache. 6. Serve the stored files, not a live query per request. Index the columns the query filters on, such as the publication status and the modification date. The query then avoids a full table scan. Cache the output with a time to live that fits the content. Use about an hour for a news or product site and a day for a stable site. On a site with millions of URLs, run generation as a background job from a queue. No web request then waits for it. Serve the files through your CDN. Compress them with gzip, which often shrinks XML by 80 percent or more. Generate per environment. A staging sitemap must not list production URLs, and staging should not be crawlable at all. ## Split large sites with a sitemap index A sitemap index lists child sitemaps. Each `` entry has a `` and an optional `` that changes when that child file changes. ```xml https://www.example.com/sitemaps/products-1.xml.gz 2026-09-20T14:05:00+00:00 https://www.example.com/sitemaps/blog-2026-09.xml.gz 2026-09-24T08:00:00+00:00 ``` The index lists sitemap files. The child sitemaps list pages. A page URL does not belong in the index. Split the children in a way that helps you read Search Console. Common splits are by content type (products, categories, articles, pages), by site section, and by publication month. A monthly split keeps older files stable and concentrates changes in the newest file. Smaller files make diagnosis easier. Search Console reports discovery per sitemap, and the Page indexing report can filter by sitemap. Many teams keep each child at 10,000 to 25,000 URLs for that reason. The protocol only requires 50,000 or fewer. A site under that size can use one sitemap. ## Add image, video, and hreflang extensions ### Images The image extension adds `` entries with an `` to a URL. Google deprecated the `image:caption`, `image:title`, `image:geo_location`, and `image:license` tags in 2022. Only `` remains useful. ### Video A video sitemap entry needs a thumbnail URL, a title, a description, and either `` or ``. The content location points to the video file. The player location points to an embeddable player. Add duration and publication date when you have them. [Video SEO and search features](/articles/video-seo-search-features) covers the rest of video indexing. ### Language versions For multilingual sites, a sitemap can declare hreflang alternates with `` inside each ``. Each language version must list every version, including itself. The annotations must be reciprocal, or Google ignores them. [The hreflang implementation guide](/articles/hreflang-implementation-guide) compares the sitemap method with HTML and header methods. The old `` annotation was for feature-phone pages. It serves no purpose on a responsive site. ## Validate before you submit Run these checks in CI or before each deploy: 1. Parse each file as XML. A well-formedness error can make the whole file unreadable. 2. Validate against the sitemap protocol schema. 3. Count URLs and measure uncompressed size against the 50,000 URL and 50 MB limits. 4. Request a sample of listed URLs and confirm that each returns `200` with no redirect and no `noindex`. 5. Confirm that special characters are escaped and that the file is UTF-8. 6. If you gzip the files, decompress one and compare it with the source. Alert on sudden changes in URL count. A drop by half usually means a broken query, not a smaller site. ## Submit and monitor Declare the sitemap in `robots.txt` so that every crawler can find it: ```text Sitemap: https://www.example.com/sitemap_index.xml ``` Submit it in Google Search Console and in Bing Webmaster Tools. Google deprecated its sitemap ping endpoint in June 2023, and the endpoint stopped working at the end of 2023. Do not build a ping step. Google rereads submitted sitemaps on its own schedule and uses accurate `lastmod` values to find changes. For Bing and other engines that support it, IndexNow sends change notices at once. [IndexNow and the indexing API](/articles/indexnow-indexing-api-developers) covers the setup. After submission, check two reports. The Sitemaps report shows whether Google read each file, when, and how many URLs it discovered. The Page indexing report, filtered to the sitemap, shows how many listed URLs are indexed and why the rest are not. A large gap between listed and indexed URLs usually points to duplicates, thin pages, or URLs that should not be in the file. Regenerate the sitemap when content is published, changed, or removed. Do not regenerate or change `lastmod` when nothing changed. ## Set up sitemaps in common frameworks | Platform | Setup | |---|---| | WordPress | Core generates `wp-sitemap.xml` since version 5.5. Yoast SEO and Rank Math replace it with their own. Check custom post types and taxonomies in the settings. | | Shopify | Generates `/sitemap.xml` for products, collections, blogs, and pages. Customization needs an app. | | Next.js | In the App Router, an `app/sitemap.ts` file returns the entries, and `generateSitemaps` splits large sites. The Pages Router needs an API route or a build-time script. | | Django | Use the built-in `django.contrib.sitemaps` framework: define `Sitemap` classes and register them in the URLconf. | | Laravel | The `spatie/laravel-sitemap` package generates sitemaps from Eloquent models. | | Rails | The `sitemap_generator` gem builds sitemaps from a config file through a rake task. | A Next.js App Router sitemap reads from the database at request time or build time: ```ts import type { MetadataRoute } from 'next'; export default async function sitemap(): Promise { const posts = await getPublishedPosts(); return posts.map((post) => ({ url: `https://www.example.com/blog/${post.slug}`, lastModified: post.contentUpdatedAt, })); } ``` Whatever the platform, check the output against the rules above. Plugins and packages often list tag archives, author pages, or filtered URLs that you do not want indexed. ---- ```text .|########||. .|########||. .|########||. |##||. .||##|. |##||. .||##|. |##||. .||##|. |#|. .|#|. |#|. .|#|. |#|. .|#|. |#| |#| |#| |#| |#| |#| .#| |#. .#| |#. .#| |#. .#. .#| .#. .#| .#. .#| |#. .#| |#. .#| |#. .#| |# ...... #| |# ...... #| |# ...... #| .# ||#########| #| .# ||#########| #| .# ||#########| #| .#. |######||######|. .#. .#. |######||######|. .#. .#. |######||######|. .#. #. .##|###|##|#|######| .# #. .##|###|##|#|######| .# #. .##|###|##|#|######| .# || |##|#||||||||||||#||#| || || |##|#||||||||||||#||#| || || |##|#||||||||||||#||#| || # |#||||||||||||||||||||#| #. # |#||||||||||||||||||||#| #. # |#||||||||||||||||||||#| #. || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || # .#||||||||||||||||||||||||#| # # .#||||||||||||||||||||||||#| # # .#||||||||||||||||||||||||#| # || ....|||#||||||##|#|||#|#||##||||....|. || || ....|||#||||||##|#|||#|#||##||||....|. || || ....|||#||||||##|#|||#|#||##||||....|. || #. . ....|# ....#||| ||| .#||| ....#. .# #. . ....|# ....#||| ||| .#||| ....#. .# #. . ....|# ....#||| ||| .#||| ....#. .# # . ||||#| .#####|| . .#| .#||| ||||# #. # . ||||#| .#####|| . .#| .#||| ||||# #. # . ||||#| .#####|| . .#| .#||| ||||# #. .| ||||| #. |#|||||. || #. |#||. .||||| || .| ||||| #. |#|||||. || #. |#||. .||||| || .| ||||| #. |#|||||. || #. |#||. .||||| || || |.... #. ....|#. |. ...|. ....|| || || |.... #. ....|#. |. ...|. ....|| || || |.... #. ....|#. |. ...|. ....|| || #. .||||||##||||||##||####|||||||#||||||#| .# #. .||||||##||||||##||####|||||||#||||||#| .# #. .||||||##||||||##||####|||||||#||||||#| .# # .||####|###########################|. # # .||####|###########################|. # # .||####|###########################|. # # .#||||||.#.|| # |. ..# #| #|||||#| # # .#||||||.#.|| # |. ..# #| #|||||#| # # .#||||||.#.|| # |. ..# #| #|||||#| # # .#|||||| .. |# ## |#| ...#|#|||#| # # .#|||||| .. |# ## |#| ...#|#|||#| # # .#|||||| .. |# ## |#| ...#|#|||#| # # .#|||#|# .# .#| #| ##|.#..#|||||#| # # .#|||#|# .# .#| #| ##|.#..#|||||#| # # .#|||#|# .# .#| #| ##|.#..#|||||#| # # .#||||||############|######|||||#| # # .#||||||############|######|||||#| # # .#||||||############|######|||||#| # # |...||#||||||#||||#|#||||||#|||||#||| |.| # # |...||#||||||#||||#|#||||||#|||||#||| |.| # # |...||#||||||#||||#|#||||||#|||||#||| |.| # #. .. ||||# .|||##|. |#| .|| || .|||#. #| # .# #. .. ||||# .|||##|. |#| .|| || .|||#. #| # .# #. .. ||||# .|||##|. |#| .|| || .|||#. #| # .# || | ...|| ...##| | #| .|. |. ##### .. .| || || | ...|| ...##| | #| .|. |. ##### .. .| || || | ...|| ...##| | #| .|. |. ##### .. .| || || ||||| || ||||#| . |. |. |#. ||||| |#| || || || ||||| || ||||#| . |. |. |#. ||||| |#| || || || ||||| || ||||#| . |. |. |#. ||||| |#| || || .# |.....#|....|#.||||.|||##.|#|....||.#||.#. #. .# |.....#|....|#.||||.|||##.|#|....||.#||.#. #. .# |.....#|....|#.||||.|||##.|#|....||.#||.#. #. #.|||||||#############################| ||| .# #.|||||||#############################| ||| .# #.|||||||#############################| ||| .# || ##||#||#||||||#||#||#||#||##. || || ##||#||#||||||#||#||#||#||##. || || ##||#||#||||||#||#||#||#||##. || # .#|||||||||#||#|||||||||||#| # # .#|||||||||#||#|||||||||||#| # # .#|||||||||#||#|||||||||||#| # || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || .# |#||||||||||||||||||||#| #. .# |#||||||||||||||||||||#| #. .# |#||||||||||||||||||||#| #. || |#||||||||||||||||||#| || || |#||||||||||||||||||#| || || |#||||||||||||||||||#| || #. .######|#|#########| .# #. .######|#|#########| .# #. .######|#|#########| .# .#. |#####||||#####| .#. .#. |#####||||#####| .#. .#. |#####||||#####| .#. |# ||########|| #. |# ||########|| #. |# ||########|| #. |# ...... #| |# ...... #| |# ...... #| |#. .#| |#. .#| |#. .#| |#. .#. |#. .#. |#. .#. .#| |#. .#| |#. .#| |#. |#| |#| |#| |#| |#| |#| .|#|. .|#|. .|#|. .|#|. .|#|. .|#|. |##||. .||##| |##||. .||##| |##||. .||##| .||########||. .||########||. .||########||. Scale With Search 2026 [scalewithsearch.com](https://scalewithsearch.com) ```