Robots.txt Sitemap: The Exact Syntax and How to Set It Up

A robots.txt sitemap directive is a single line, Sitemap: https://example.com/sitemap.xml, that tells any crawler reading your robots.txt file exactly where your XML sitemap lives. You add it by opening robots.txt at your site root and appending that line with your sitemap's full, absolute URL, anywhere in the file. It works alongside, not instead of, submitting the sitemap in Search Console.

This is a narrow, mechanical topic, but the details trip people up constantly: relative paths that get silently ignored, a stray Disallow: / that blocks the very file you just pointed crawlers to, and confusion about whether Bing or Yandex even bother reading the line. This guide covers the exact syntax, what Google's own robots.txt specification says about it, and the mistakes that quietly break discovery.

Key takeaways

• The syntax is Sitemap: followed by a full, absolute URL. Relative paths do not work.
• You can list as many Sitemap: lines as you need. Google's specification places no limit on the count.
• The directive is global and not tied to any User-agent block, but a Disallow rule that matches the sitemap's own path can still block crawlers from fetching it.
• Bing and other major crawlers read the same field Google documents, so one line reaches more than just Google.
• Adding the line does not replace submitting your sitemap in Search Console; the two give you different things.

What does the robots.txt sitemap directive do?

The Sitemap directive tells a crawler where to find your XML sitemap the moment it opens robots.txt, before it has crawled a single page of your site. According to Google's How Google Interprets the robots.txt Specification, the sitemap field "isn't tied to any specific user agent and may be followed by all crawlers," which is what makes it different from every other line in the file.

Every other robots.txt directive (User-agent, Disallow, Allow) controls what a crawler is permitted to fetch. The Sitemap line does the opposite job: it hands the crawler a map instead of a rule. Robots.txt is usually the first file a bot requests on a new domain, so putting your sitemap location there means discovery can start on the very first visit, not after the crawler happens to find a link to it.

What is the exact Sitemap directive syntax?

The line is Sitemap: (the field name), a colon, a space, then the fully qualified URL of your sitemap file. Nothing else is required. Here is a minimal, valid example:

```robots.txt
User-agent: *
Disallow: /wp-admin/
Disallow: /cart/

Sitemap: https://example.com/sitemap.xml
```

If you run a sitemap index file instead of a single sitemap, point the directive at the index, not at every child file individually:

```robots.txt
Sitemap: https://example.com/sitemap_index.xml
```

Three rules matter here, straight from Google's specification. First, the URL must be fully qualified, including the protocol and host; /sitemap.xml on its own will be ignored. Second, the field name Sitemap is case-insensitive, so sitemap:, SITEMAP:, and Sitemap: all parse the same way, but the URL value itself is case-sensitive, so /Sitemap.xml and /sitemap.xml are treated as different addresses. Third, the URL does not need to be URL-encoded and does not have to sit on the same domain as the robots.txt file, so referencing a sitemap hosted on a subdomain or CDN is valid.

Where should the Sitemap line go in robots.txt?

Placement does not change how it functions. Because the directive is global rather than scoped to a User-agent block, it works whether you put it at the top, the bottom, or between two rule sets. Convention, not requirement, puts it at the very bottom of the file, after every User-agent and Disallow group, so it reads as a closing pointer rather than an access rule.

There is a technical reason placement is forgiving: Google's specification notes that crawlers ignore sitemap lines when grouping User-agent rules together, so a Sitemap line sitting in the middle of the file will not accidentally split or break a rule group the way a misplaced User-agent line could. Put it wherever is easiest for your team to find and maintain, and keep it consistent across environments.

Can you list multiple sitemaps in robots.txt?

Yes. Google's robots.txt specification is explicit that you "can specify multiple sitemap fields, with no limit to the number of sitemaps you can include," each on its own line. This matters for a few real setups: a site that splits content across separate sitemaps by type (pages, posts, images, video), a business running several subdomains that each maintain their own sitemap, or a migration period where you are pointing crawlers at both an old and a new sitemap file at once.

```robots.txt
Sitemap: https://example.com/sitemap-pages.xml
Sitemap: https://example.com/sitemap-posts.xml
Sitemap: https://blog.example.com/sitemap.xml
```

For most sites this is unnecessary complexity. If your platform already generates a sitemap index file (Yoast, Rank Math, and most modern CMS platforms do this automatically past a certain URL count), one Sitemap: line pointing at that index is simpler to maintain than several individual lines, and our guide on how to make a sitemap walks through building that index file if you do not already have one.

Here is what the Sitemap field is built to do versus what only a Search Console submission gives you back.

Does a Disallow rule block your sitemap from being crawled?

It can. The Sitemap field is not scoped to a User-agent, but Google's specification is specific about the condition: crawlers will follow it "provided it isn't disallowed for crawling." If your sitemap file sits at a path caught by a broad rule, most commonly a blanket Disallow: / left over from a staging environment, or an overly wide pattern like Disallow: /sitemap, the same rule that blocks the rest of the site can block the sitemap file itself.

This is a more common mistake than it sounds. Sites migrating off staging frequently forget to remove a site-wide Disallow, and the Sitemap line keeps working syntactically while the file it points to is quietly unreachable. Check that the exact path in your Sitemap line is not caught by any Disallow rule in the same file, and confirm it in a robots.txt tester rather than assuming.

A related but separate rule: do not list URLs inside your sitemap that are themselves disallowed. Google's sitemap documentation recommends only listing canonical, indexable URLs, so a sitemap full of pages your own robots.txt blocks sends crawlers a contradictory signal about what you actually want indexed.

Do all search engines honor the robots.txt Sitemap line?

The major ones do. Bing's own Webmaster Blog confirms you can "reference your sitemap in your site's robots.txt file to support automatic discovery and consistent crawling," using the identical Sitemap: format Google documents, and that Bing will attempt to fetch it once it is referenced. Yandex and most other standards-compliant crawlers follow the same convention, since the Sitemap field originates from the shared sitemaps.org protocol rather than being a Google-only extension.

That shared support is the real advantage of putting the line in robots.txt instead of relying only on manual submissions: one file, read by every crawler that visits your domain, replaces separate sitemap submissions in each search engine's own webmaster tool. You still get more detail (errors, discovered-URL counts) from submitting directly in Google Search Console and Bing Webmaster Tools, but the robots.txt line is what guarantees baseline discovery regardless of which tools you actively monitor.

How to add the Sitemap directive: step by step

1. Confirm your sitemap's exact URL by opening it directly in a browser. It should load and list URLs, not 404.
2. Open your robots.txt file, usually accessible by FTP, your host's file manager, or your CMS's SEO plugin settings.
3. Add a new line: Sitemap:, a space, then the full URL from step 1. Place it anywhere in the file; the bottom is conventional.
4. Save and re-upload the file if you edited it locally, then load yourdomain.com/robots.txt in a browser to confirm the line appears exactly as written.
5. Check the sitemap's path against every Disallow rule in the same file to make sure nothing blocks it.
6. Test the file with a robots.txt tester, such as our free robots.txt tester, to confirm the Sitemap line parses correctly and nothing disallows it.
7. Still submit the sitemap URL in Google Search Console and Bing Webmaster Tools for the fetch-status and error reporting robots.txt cannot give you.

If you do not have a robots.txt file yet, or want to generate one that already includes a correctly formatted Sitemap line, our free robots.txt generator builds one in under a minute. If you have not built the sitemap file itself, start with our full how to make a sitemap guide, which covers plugins, generators, manual XML, and framework-based options.

Here is a quick summary of what Google's specification actually requires from the field, since most of the mistakes above come from skipping one of these four rules.

Common mistakes with the robots.txt sitemap directive

• Using a relative path. Sitemap: /sitemap.xml is not valid. Crawlers need the full URL with protocol and host.
• Pointing at a URL that 404s or redirects. If the sitemap moved or was renamed and the robots.txt line was never updated, crawlers hit a dead end every time they check.
• Leaving a site-wide Disallow in place after launch. A staging-era Disallow: / silently blocks the sitemap along with everything else, even though the Sitemap line itself still looks correct.
• Assuming the line alone guarantees indexing. It only helps discovery. Google's documentation is explicit that a sitemap, however it is discovered, does not guarantee crawling, indexing, or better rankings.
• Only listing the sitemap in one place. Relying solely on a Search Console submission means non-Google crawlers never learn where your sitemap is. Relying solely on robots.txt means you lose Google's error reporting. Do both.
• Forgetting to re-check after a CMS migration. A new platform often changes the sitemap's default path (from /sitemap.xml to /sitemap_index.xml, for example), and an unmodified robots.txt line will keep pointing at the old, now-missing file.

Most of these are five-minute fixes once you know to look for them. Verify the final result with our free robots.txt tester or the XML sitemap validator before you consider the setup done, and if you want the same discipline applied across every technical signal on your site, our complete SEO site audit checks crawlability, robots.txt, and sitemap health in one pass.

Frequently asked questions

What does the Sitemap directive in robots.txt do? It points crawlers to the location of your XML sitemap so they can find it without guessing. Google's robots.txt specification treats it as a standalone field that any crawler reading the file can follow, separate from the Disallow and Allow rules.

What is the exact syntax for the Sitemap directive? Sitemap: followed by a space and the full absolute URL, for example Sitemap: https://example.com/sitemap.xml. The field name is case-insensitive, but the URL itself is case-sensitive, and it must include the protocol and host, not a relative path.

Can I list more than one sitemap in robots.txt? Yes. Google's specification says you can include multiple Sitemap fields with no limit to how many you list, each on its own line. This is common for large sites that split content into several sitemap files or run multiple subdomains.

Does a Disallow rule stop crawlers from reading my sitemap? It can. The Sitemap field is not tied to a user-agent, but Google notes crawlers will follow it only if it is not disallowed for crawling. If your sitemap sits at a path blocked by Disallow, such as a blanket Disallow: /, that block can prevent the sitemap itself from being fetched.

Where should the Sitemap line go in robots.txt? Anywhere in the file works, since the directive is independent of User-agent blocks and applies globally. Convention places it at the very bottom, after all the User-agent and Disallow rules, so it reads as a final pointer rather than a rule.

Do Bing and other search engines honor the robots.txt Sitemap line? Yes. Bing's Webmaster Blog confirms you can reference your sitemap in robots.txt to support automatic discovery, and Bing will attempt to fetch it. Most major crawlers, including Bing and Yandex, read the same Sitemap field Google documents.

Do I still need to submit my sitemap in Search Console if it is in robots.txt? It is worth doing both. The robots.txt line reaches every crawler automatically, but Search Console submission is the only way to see Google's fetch status, errors, and discovered-versus-indexed counts for that specific sitemap.

Does the Sitemap directive need to point to a file on the same domain? No. Google's specification states the sitemap does not have to reside on the same site as the robots.txt file, as long as the URL is fully qualified. This lets you host a sitemap on a CDN or subdomain and still reference it from your main robots.txt.

Will adding a Sitemap line to robots.txt improve my rankings? No. It helps crawlers discover your sitemap faster, which is a prerequisite for crawling and indexing, but discovery is not a ranking factor on its own. Google's sitemap documentation is explicit that a sitemap does not guarantee indexing or better positions.

What happens if the URL in my Sitemap line is wrong or unreachable? Crawlers that try to fetch it will get an error and move on; they will not retry indefinitely or fall back to guessing a different path. Test the exact URL in a browser and in a robots.txt tester before you rely on it, since a typo here quietly breaks discovery for every crawler that reads the file.

What to do next

Open your robots.txt file, add or check the Sitemap: line against the syntax above, and confirm it is not caught by a Disallow rule. Then run it through a tester and submit the same sitemap URL in Search Console so you get both universal discovery and Google-specific error reporting. If you want someone to check this alongside every other technical signal holding your site back, start with a free SEO audit from Rankite.