A sitemap is an XML file that lists a website’s URLs and submits them to search engines. Google reads it to discover pages, but does not guarantee that it will crawl or index them all. Understanding the distinction between discovery, crawling, and indexing changes how you use a sitemap: it is not an automatic submission tool, but a diagnostic tool.
What Google really takes from an XML sitemap
A sitemap sends Google a list of URLs along with metadata. Not all metadata carries the same weight. Google ignores the priority and changefreq fields: filling them in does not change how the crawler behaves.
The lastmod field, on the other hand, can influence how often a page is recrawled. The condition is that the date must be reliable and reflect an actual change to the content. A sitemap whose lastmod date is updated daily without any content changes loses all credibility in Google’s eyes, and Google eventually ignores this signal.
A sitemap submitted in Search Console generates a dedicated report. This report shows the number of URLs discovered, but not how many were crawled or indexed. Exploring the iLinks site in Google Search lets you quickly visualize a sitemap’s structure and spot discrepancies between what is declared and what Google actually processes.
Sitemaps and indexing: spotting discrepancies in Search Console
A sitemap’s main value lies not in creating it, but in what it reveals when compared with Search Console data. A site can list several hundred URLs in its sitemap and see only a fraction of them indexed.
Three stages of processing to distinguish
A URL listed in the sitemap goes through three successive stages, and each can fail independently of the others:
- Discovery: Google has read the URL in the sitemap. The Sitemaps report confirms this step, but guarantees nothing beyond it.
- Crawling: Googlebot has actually downloaded the page. A URL can remain in the “Discovered – currently not indexed” state for weeks if the site’s crawl budget is limited.
- Indexing: the page has been analyzed, deemed sufficiently relevant, and added to the index. Crawled pages may be excluded because of duplicate content, thin content, or conflicting canonical signals.
To identify where the blockage occurs, compare the number of URLs in the sitemap with the “Pages” report (formerly Coverage) in Search Console. Pages blocked at the “Discovered” stage indicate a crawl budget issue. Pages blocked at the “Crawled – currently not indexed” stage indicate a quality or technical signals issue.
Declared canonical URL and the canonical URL selected by Google
An often-overlooked diagnostic is to check whether the canonical URL selected by Google matches the one declared in the sitemap. Search Console’s URL Inspection tool displays this information for each page.
A discrepancy between the two points to conflicting signals. For example, the sitemap may list an HTTPS URL while internal links still point to the HTTP version. Or a rel=”canonical” tag in the source code may point to a different page than the one listed in the sitemap.
These contradictions do not result in a penalty, but they dilute the page’s signals. Google then chooses the canonical URL it considers most consistent, which may lead it to index a different page from the one you intended. Checking a sample of strategic pages with URL Inspection remains the most reliable way to detect this type of issue.
Sitemap structure: indexes, segmentation, and multiple files
A sitemap can be a single file or a sitemap index that references several child files. Large sites use this segmentation to organize URLs by content type: product pages, blog posts, categories, and pages in different languages.
Useful segmentation criteria
Segmenting a sitemap by content type makes it easy to identify which group of pages is causing problems. If product pages are indexed but blog posts are not, a segmented sitemap makes this immediately apparent in Search Console.
Good segmentation is based on functional logic, not technical logic. Separating URLs by server directory adds no value if those directories do not correspond to distinct content types. By contrast, a sitemap for each content category makes it easier to track pages individually.
Each sitemap file is subject to a size limit. Google accepts files that stay within this limit and are encoded in UTF-8. An oversized or incorrectly encoded file will simply be ignored, sometimes without an explicit error message.
Reading a competitor’s sitemap to analyze its structure
A third-party site’s sitemap is often publicly accessible. The first step is to check the robots.txt file, usually available at /robots.txt, which declares the sitemap’s location using a Sitemap directive.
If robots.txt does not mention one, trying the most common paths (/sitemap.xml, /sitemap_index.xml) works in most cases. CMS platforms such as WordPress, Shopify, and Wix automatically generate a sitemap at a standard location.
Analyzing a competitor’s sitemap reveals several structural details: the number of published pages, the breakdown by content type, and the publishing frequency (via lastmod dates, if provided). This data helps you assess the scale of a site without access to its administrative tools.
A sitemap is still a declarative file. It shows what the site owner wants search engines to see, not necessarily every page that exists. Pages excluded from the sitemap can sometimes be just as revealing as those included in it.
