By Shad Super in Questions —

What Are the URL and File Size Limits for Sitemaps, and How Can Large Sites Adapt?

Summary

Sitemaps have specific limits on URL count and file size. A single sitemap file can include up to 50,000 URLs and be no larger than 50 MB when uncompressed. Large sites can adapt by using sitemap index files to reference multiple sitemaps. Here’s a comprehensive guide to managing sitemaps for extensive websites.

Sitemap Limits

URL Limit

Each sitemap file can contain a maximum of 50,000 URLs. This limit ensures that search engines can efficiently crawl and index the URLs listed in the sitemap. If your site has more than 50,000 URLs, you'll need to create multiple sitemap files.

File Size Limit

The uncompressed file size of each sitemap is limited to 50 MB. Compressing the sitemap file using gzip can help reduce the size, but the uncompressed size cannot exceed 50 MB. Search engines often recommend gzip compression to save bandwidth and improve processing time.

Adapting for Large Sites

Sitemap Index Files

Sitemap index files enable you to manage multiple sitemaps efficiently. An index file can refer to up to 50,000 individual sitemaps, allowing large websites to organize their URLs systematically. Each referenced sitemap file must comply with the 50,000 URL and 50 MB size limits. You can submit a sitemap index file to search engines instead of submitting individual sitemap files.

Example of a Sitemap Index File:

<?xml version="1.0" encoding="UTF-8" ?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap>
<loc>http://www.example.com/sitemap1.xml</loc>
<lastmod>2023-01-01</lastmod>
</sitemap>
<sitemap>
<loc>http://www.example.com/sitemap2.xml</loc>
<lastmod>2023-01-01</lastmod>
</sitemap>
</sitemapindex>

Dynamic Generation of Sitemaps

For dynamically generated content, consider using automated tools to create and update sitemaps. Content management systems (CMS) like WordPress offer plugins that automatically generate and update sitemaps based on site activity. Examples include the Google XML Sitemaps and Yoast SEO plugins.

Pagination and Separate Sitemaps for Different Sections

Divide your site into sections and create separate sitemaps for each. For instance, e-commerce sites can create individual sitemaps for products, categories, and blog posts. You can then manage and submit each sitemap individually or include them in a sitemap index file.

Best Practices

Regular Updates

Regularly update your sitemaps to reflect changes in your site structure. Ensure that new pages are added to the sitemap and removed pages are deleted promptly. Using automated tools can help manage this process efficiently.

Compression

Compress sitemap files using gzip to reduce bandwidth usage and improve processing time. Even though search engines accept uncompressed sitemaps, compression is highly recommended.

Notify Search Engines

Notify search engines about your sitemaps by submitting the sitemap index or individual sitemaps through webmaster tools such as Google Search Console and Bing Webmaster Tools. Additionally, you can add the sitemap’s URL to your robots.txt file.

Example in robots.txt:

User-agent: *
Disallow: /private/
Sitemap: http://www.example.com/sitemap.xml

References