robots.txt and XML Sitemaps

Updated 26 September 2026 · 5 min read

Search engines discover URLs by following links and by reading files you publish for them. Two of those files are robots.txt and XML sitemaps. They are not magic rank buttons. robots.txt is a set of crawl requests. A sitemap is a list of URLs you consider important. This guide covers what robots.txt can and cannot do, noindex versus disallow, sitemap format, submitting to search consoles, and mistakes that hide the wrong pages. Examples use example.com.

What robots.txt is for

When a well-behaved crawler arrives, it may fetch https://example.com/robots.txt first. The file is plain text. You can suggest which paths not to fetch, and you can point at a sitemap.

User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /cart/help
Sitemap: https://example.com/sitemap.xml

User-agent: * means the default rules. Specific bots can have their own blocks. Disallow: /admin/ asks crawlers not to fetch URLs under that path. It does not password-protect /admin/. Anyone can still open it. It also does not remove a URL that is already in a search index. It only asks crawlers not to request those URLs.

Malicious scrapers ignore robots.txt. Treat it as a politeness file plus a sitemap pointer, not a security control. Security belongs in authentication, firewalls, and HTTPS as described in HTTPS and SSL certificates.

What robots.txt cannot do

• It cannot reliably “unpublish” a page from search results. Use noindex or take the page down (and then wait).
• It cannot keep private files secret. Use auth, or keep the files off the public server.
• It cannot target a single country or a single human visitor.
• It cannot replace a 404 or a login wall.
• A syntax error can disallow more than you meant, including the whole site if you Disallow: / by accident.

Keep the file at the site root on the host you consider canonical (www or not). Different hosts are different robots.txt files. If both www and the apex serve sites, keep them consistent with your redirect plan so crawlers see one file after the hop.

noindex versus disallow

Disallow in robots.txt: please do not crawl this path. If the URL is never crawled, a crawler may not see a noindex tag on it either. URLs can still appear in results if other sites link to them, sometimes without a snippet.

noindex is usually a meta tag or an HTTP header on the page itself:

<meta name="robots" content="noindex, follow">

That says: you may crawl this, but do not include it in the index. To deliver that message, the crawler generally needs permission to fetch the page. So: if you want a page out of the index, allow the crawl and send noindex (or require a login so it never has public content). If you only want to save crawl budget on infinite filter URLs, disallow those patterns — understanding they may still show as bare URLs if they were known.

Do not disallow a URL and also expect the crawler to read its sitemap entry and noindex it. You have closed the door you wanted them to walk through.

Canonical tags (covered with other head tags in meta tags, favicons and social previews) are a third tool: they say which duplicate is preferred. They are not a robots.txt rule.

XML sitemap format

A sitemap is an XML list of URLs. A simple example:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2026-03-01</lastmod>
  </url>
  <url>
    <loc>https://example.com/about/</loc>
  </url>
</urlset>

Use the canonical HTTPS URLs you actually want indexed. Do not list noindex URLs, redirected URLs, or parameter junk. lastmod should be real if you include it; fake timestamps teach crawlers to ignore you. Large sites split into a sitemap index that points at several sitemap files. Image and video sitemap extensions exist; they are optional.

Your CMS may generate the file. Check a sample of URLs by hand. A sitemap that lists 50,000 faceted search combinations is worse than a smaller, honest list.

Point to the sitemap from robots.txt with an absolute URL. That helps crawlers that never opened a search console account for you.

Submitting to search consoles

Google Search Console, Bing Webmaster Tools, and similar products let you verify the site and paste a sitemap URL. Verification is usually a DNS TXT record or a file. After submission, look for errors: URLs that 404, that redirect, that are blocked by robots.txt, or that return the wrong host.

Submission does not guarantee indexing. It is a hint. Useful pages with real links still matter more than a sitemap alone. If the console says “discovered, not indexed,” the sitemap is doing its job of discovery; the quality or duplication question is elsewhere.

When you change domains, submit sitemaps on the new host and keep 301s from the old URLs. Do not expect a sitemap to replace redirects.

Common mistakes

• Disallow: / left over from staging, shipped to production. The whole site asks not to be crawled.
• Staging and production sharing a sitemap domain, or production listing staging URLs.
• Blocking CSS and JS that the crawler needs to understand the layout, then wondering why the page is considered broken.
• Listing both http and https, or both www and apex, as separate loc values.
• Password pages in the sitemap.
• A robots.txt that disallows /wp-admin/ (often fine) and also disallows /wp-content/ which can hide assets.
• Using robots.txt to “hide” duplicate parameters instead of fixing canonicals and internal links.
• Serving robots.txt as a 404 or as an HTML error page. It should be 200 and text/plain.

Test by fetching the file in a browser and by using the search console robots tester if you have it. After a launch, search for site:example.com as a rough check, knowing that operator is not a complete inventory.

A small working pair

For many small sites:

# robots.txt
User-agent: *
Disallow: /admin/
Disallow: /thank-you/
Sitemap: https://www.example.com/sitemap.xml

Put noindex on thank-you and cart pages if they are crawlable and you do not want them in results. Keep the sitemap to indexable articles and core pages only. Update it when you publish, automatically if the CMS can.

Checklist

• robots.txt is reachable on the canonical host, as text, without blocking the whole site.
• Disallow is for crawl waste, not for secrecy or for de-indexing.
• noindex pages remain crawlable unless they are behind a login.
• Sitemap lists only canonical, 200, indexable HTTPS URLs.
• Search console sitemap report is free of mass redirect/404 errors.

Open robots.txt and two sitemap URLs this week and click them. If you would not want a stranger to treat those URLs as your best pages, fix the list.

Related guides

• Website Speed Settings
• Email Authentication with SPF, DKIM, and DMARC
• Meta Tags, Favicons, and Social Previews

General technical information; test changes carefully and keep backups of your settings.