This file tells crawlers what they may request. It controls crawling, not indexing. To keep a page out of Google, use a noindex tag or password instead.
Built by Rankite, the SEO team behind Swordfish AI's +400% revenue and Zluri's +45% organic growth. See the case studies
A robots.txt file is a plain text file in the root of your domain that tells crawlers which parts of your site they are allowed to request. When a well-behaved bot like Googlebot visits, it reads this file first and obeys the Allow and Disallow rules it finds. The file always starts with a User-agent line, and the generator above keeps that line in place for you so the output stays valid.
The single most important thing to understand is that robots.txt controls crawling, not indexing. Blocking a URL stops Google from reading the page, but if other sites link to it, Google can still list the URL in search results without a description. To actually keep a page out of the index, let it be crawled and add a noindex meta tag, or protect it with a password. Disallow and noindex solve different problems.
| Mistake | Why it hurts | Better approach |
|---|---|---|
| Blocking CSS and JavaScript | Google renders pages like a browser. Blocking assets makes pages look broken to the crawler and can lower rankings. | Leave /wp-content/ and script folders open. |
| Using Disallow to hide a private page | The URL can still appear in results, and the file is public for anyone to read. | Use noindex or a login wall. |
| Blocking your whole site by accident | Disallow: / on a live site removes you from search entirely. | Only choose Block all on staging sites. |
| Forgetting the sitemap line | Crawlers find new pages slower without a sitemap pointer. | Add your full sitemap URL. |
Keep the file short and deliberate. Block low-value paths like internal search results, cart and checkout pages, and faceted query strings, and leave everything that should rank fully open. If you want a second pair of eyes on what your site is blocking and whether it is costing you traffic, request a free SEO audit and we will check your robots rules against your competitors.
Build valid JSON-LD structured data for FAQ, Article, Organization, Local Business, Product and Breadcrumb rich results.
Turn any title into a clean URL slug live: lowercase, accent-free, hyphenated, with stop-word removal and a full-URL preview.
Paste a page's HTML and see what a crawler extracts: title, meta tags, heading outline, visible text and alt checks.
Get a free, no-obligation SEO audit and a 30-minute strategy session. We'll show you exactly where the growth is hiding.
Fill out the form and we'll get back to you within one business day. Prefer email? Write to us directly at contact@rankite.com.