By Shad Super in Questions —

How Can Discrepancies Between Submitted URLs and Indexed URLs in the Sitemaps Report Be Addressed?

Summary

Addressing discrepancies between submitted URLs and indexed URLs in the Sitemaps Report involves diagnosing issues including sitemap errors, crawl budget limitations, canonicalization problems, and content quality. Ensuring proper indexing requires examining these factors and implementing recommended solutions.

Identify Sitemap Errors

Verify Sitemap Validity

First, confirm that your sitemap is correctly formatted and accessible. A good tool for this purpose is Google's Sitemap Testing Tool. This tool validates the XML structure and accessibility.

Common Sitemap Issues

  • Incorrect URLs: Ensure all URLs are correct, complete, and use the correct protocol (HTTP vs. HTTPS).
  • Non-canonical URLs: The URLs provided in the sitemap should match the canonical URLs used on your pages.

Check Crawl Budget

Understand Crawl Budget

Google allocates a specific crawl budget to each website based on its size and health. If your website exceeds this budget, not all URLs will be crawled and indexed. For more details on how crawl budget works, refer to Google’s guide on Crawl Budget.

Improving Crawl Budget Usage

  • Optimize your website's health by ensuring it loads quickly and has minimal errors.
  • Use internal linking to help Google discover new pages more effectively.

Canonicalization Issues

Canonical Tags

Use the <link rel="canonical"> tag on your pages to signal the preferred version of a URL. Google may choose not to index URLs without proper canonicalization. You can learn more about proper use of canonical tags on Google’s page about Consolidating Duplicate URLs.

Consistent URL Usage

Ensure URL consistency by standardizing between www and non-www versions and between HTTP and HTTPS protocols. Use redirects to guide bots to the preferred versions.

Content Quality Assessment

Value-Driven Content

High-quality, unique, and valuable content is more likely to be indexed. Google provides in-depth advice in their Quality Guidelines for creating content that meets their standards.

Avoiding Duplicate Content

Duplicate content may cause Google to ignore certain URLs. Tools like Copyscape can help identify unintentional duplicate content.

Technical SEO Factors

Mobile-Friendliness

Make sure your website is mobile-friendly. Google emphasizes mobile-first indexing, meaning that mobile compatibility significantly impacts indexing. Test your website's mobile compatibility using Google’s Mobile-Friendly Test.

Robots.txt and Meta Tags

Check your robots.txt file to ensure it isn’t blocking important pages from being crawled. Similarly, avoid using <meta name="robots" content="noindex"> tags unless you want to exclude a page from indexing.

Addressing Specific Issues

Error Pages

Ensure all URLs return a 200 HTTP status code and not a 404 or 500 error. Use Google Search Console’s URL Inspection Tool to diagnose and fix these errors.

Structured Data

Implementing structured data helps search engines better understand your content. Structured data can be checked and validated with Google’s Rich Results Testing Tool.

Conclusion

Addressing discrepancies between submitted URLs and indexed URLs in the Sitemaps Report requires a multi-faceted approach involving sitemap verification, optimizing crawl budget usage, fixing canonicalization issues, ensuring high content quality, and resolving technical SEO factors. Regularly check and update these aspects to maintain effective indexing.

References