Robots.txt: What It Does and How to Write One Correctly
One tiny text file tells search engines which parts of your site to crawl — and one wrong line can wipe you off Google. Here is how to get it right.
Of all the files on a website, robots.txt has the highest ratio of power to size. It is a plain text file, often under ten lines, sitting at yourdomain.com/robots.txt — and a single misplaced character in it can remove your entire site from search results. Worth understanding, then.
What robots.txt actually is
When a search engine crawler arrives at your site, the first thing it fetches is /robots.txt. The file contains simple instructions about which parts of the site the crawler may visit. It is a set of house rules for bots.
A minimal, healthy example:
User-agent: *
Allow: /
Disallow: /admin
Sitemap: https://example.com/sitemap.xml
Line by line: rules apply to all bots (*), everything is crawlable, except anything under /admin, and here is the sitemap. That structure covers the vast majority of sites.
What belongs behind Disallow
Good candidates for blocking: admin panels, login pages, internal search result pages, cart and checkout flows, thank-you pages, and staging areas. These pages waste crawler attention and have no business appearing in search results.
The Robots.txt Generator builds a correct file from checkboxes — pick what to block, get clean output, no syntax memorizing.
The catastrophic one-liner
User-agent: *
Disallow: /
That single slash blocks everything. It is a standard setting on staging sites (correctly!) — and the classic disaster is launching the site with the staging robots.txt still in place. Traffic falls off a cliff and nobody knows why. If a site's search traffic ever collapses mysteriously, robots.txt is the first thing to check. It takes ten seconds: visit /robots.txt and read it.
What robots.txt does NOT do
Three widespread misunderstandings:
It is not security. The file is public and advisory. Polite bots obey it; malicious ones read it as a map of interesting places. Protect private areas with authentication, never with robots.txt.
Disallow does not remove pages from Google. A blocked page can still appear in results (as a bare URL) if other sites link to it. To keep a page out of results, use a noindex meta tag — and note the trap: Google must be allowed to crawl the page to see its noindex tag. Blocking it in robots.txt prevents exactly that.
It does not speed up your site. It manages crawler traffic, which on small sites is negligible. Its value is directing crawl attention, not saving bandwidth.
The sitemap line
The Sitemap: line tells crawlers where your sitemap lives — the list of URLs you want crawled. Robots.txt says "don't go there"; the sitemap says "definitely go here". They are partners. Generate one with the Sitemap Generator if your platform does not produce one automatically, and always reference it in robots.txt.
A five-minute health check
- Visit yourdomain.com/robots.txt right now.
- Confirm it exists (a missing file is fine — everything gets crawled — but you lose the sitemap pointer).
- Read every Disallow line and make sure you recognize and intend each one.
- Confirm the Sitemap line exists and its URL actually loads.
- Confirm nothing important — your blog, products, tools — falls under any Disallow path.
While you are auditing, the Redirect Checker and Canonical Checker cover the two neighboring plumbing issues that quietly hurt crawling.
The bottom line
Robots.txt is ten lines of text with outsized consequences. Keep it simple, block only what should never be searched, point to your sitemap, and check it after every site launch or migration. Five minutes a year, catastrophe avoided.
Quick answers
My site has no robots.txt — is that an emergency? No. With no file, crawlers simply crawl everything, which is fine for most small sites. You mainly lose the sitemap pointer and the ability to fence off admin paths.
How fast do changes take effect? Search engines re-fetch robots.txt frequently — typically within a day. The bigger delay is re-crawling the pages the change affects.
Can I block just one bad bot? Yes: give that bot's user-agent its own Disallow rules while leaving * open. Well-behaved bots respect it; abusive ones need server-level blocking instead.