robots.txt

How the directives work

A robots.txt file is a sequence of rule groups. Each group opens with one or more User-agent lines naming the crawler the rules target: a specific name like Googlebot or GPTBot, or the wildcard asterisk for all bots. Under that come Disallow lines listing path prefixes the crawler should not request, and optional Allow lines that re-permit narrower paths inside a blocked directory. Disallow with a single slash blocks the entire site; an empty Disallow value blocks nothing.

For the file to be honored under RFC 9309 and Google's own specification, it must be reachable at exactly the site root, served with a text/plain content type, and returned with an HTTP 200 status. A common non-directive addition is the Sitemap line, which points crawlers to an XML sitemap and is independent of any User-agent group. Crawl-delay, which asks bots to space out requests, is widely used but nonstandard, and Google explicitly does not honor it, so throttling for Google must be managed elsewhere.

robots.txt vs. the meta robots tag

These two controls are frequently confused because both mention robots, but they act at different stages. robots.txt operates before a fetch: it tells a crawler which URLs not to request at all, site-wide. The meta robots tag lives in the HTML head of an individual page and controls what a crawler does after it has already fetched that page: chiefly whether to index it and whether to follow its links. The X-Robots-Tag HTTP header does the same job as the meta tag but is set in the server response, which makes it usable for non-HTML files like PDFs.

The distinction has a practical trap. Because a Disallow rule stops the crawler from ever reading the page, it also stops the crawler from seeing any noindex meta tag on that page. A URL blocked in robots.txt can therefore still surface in search results if other sites link to it. To reliably keep a page out of an index, operators must allow crawling and use a noindex tag, not block it in robots.txt.

How competitive-intelligence tools use robots.txt

For competitive-intelligence tooling that monitors rival websites for changes (new pages, pricing updates, blog posts, job listings), a competitor's robots.txt is usually the first file checked before any crawling begins. It reveals which paths the site considers off-limits, whether the operator broadly blocks bots or AI crawlers, and where the sitemap lives. Reading it first lets a tool scope its collection to permitted paths and pace requests accordingly.

Respecting Disallow rules, and throttling in the spirit of crawl-delay even where it is nonstandard, is the working definition of ethical scraping that separates legitimate CI monitoring from abusive scraping. This matters beyond etiquette. Courts have treated ignoring robots.txt as evidence of bad faith or unauthorized access in disputes such as eBay v. Bidder's Edge and AP v. Meltwater, and robots.txt compliance has featured in more recent AI-crawler litigation. Good-faith collection is easier to defend when the collector demonstrably honored the site's stated preferences.

What robots.txt does not do

robots.txt is not an access-control or security mechanism. It publishes, in plain text, the paths an operator would rather crawlers avoid, which means anyone can read it to learn where those paths are. Listing a sensitive directory in a Disallow rule advertises its existence without protecting it, so the file should never be relied on to guard private data. Real protection requires authentication, not a hint to well-behaved bots.

It is also not self-enforcing. Any HTTP client can request a disallowed URL and receive the content, because the file carries no technical teeth. On its own robots.txt is not legally binding either, though as noted it has been cited in litigation as evidence about intent. Finally, it does not manage indexing directly: its job is to control crawling, and the paradox that a blocked page can still be indexed via external links follows directly from that scope. Treat robots.txt as a signal of preference and a courtesy standard, not a wall.

Frequently Asked Questions

Related terms