How to write a robots.txt that actually blocks AI scrapers

The uncomfortable truth about robots.txt

A robots.txt file is not a lock. It is a sign on the door.

That distinction matters when teams ask whether they can “block AI scrapers” with one small text file. For reputable crawlers that follow the Robots Exclusion Protocol, yes: a correctly written robots.txt can tell them not to crawl your pages. For unknown scrapers, impersonators, browser automation, and bots that simply do not care, it will do nothing by itself.

So the practical goal is not “make scraping impossible.” It is:

• Tell compliant AI crawlers not to use your site.
• Avoid accidentally blocking search engines or useful services.
• Add stronger server-side controls for abuse.
• Keep the policy maintainable as crawler names change.

That is the boring version. It is also the version that works.

What robots.txt can and cannot do

A robots.txt file lives at the root of a site:

https://example.com/robots.txt

Crawlers request it before crawling. The file contains groups of rules. Each group starts with one or more User-agent lines, followed by Allow or Disallow directives.

A simple full-site block looks like this:

User-agent: GPTBot
Disallow: /

That says: if you are GPTBot, do not crawl anything on this site.

But robots.txt has hard limits:

1. It is voluntary. Bad actors can ignore it.
2. It does not prevent a URL from being requested by a normal browser or script.
3. It does not remove content already collected elsewhere.
4. It does not define copyright, licensing, or training rights by itself.
5. It can be misconfigured in ways that block the wrong bots.

If you need genuine access control, use authentication, authorization, rate limiting, IP-based controls, bot management, or legal controls. Robots.txt is still useful, but it belongs in a wider content protection strategy.

This is similar to other web governance problems: the visible control is rarely the whole control. If your organization already has unmanaged AI use internally, the same principle applies; a quick shadow AI audit is often more useful than pretending a single policy document solves the issue.

Start with your policy decision

Before editing the file, decide what you are actually trying to block.

There are at least four different things people mean by “AI scraper”:

• Crawlers used to collect training data.
• AI search or answer-engine crawlers.
• User-triggered fetchers, such as when someone asks an AI product to summarize a URL.
• Generic scrapers pretending to be ordinary browsers.

You may want to block all of them. Or you may want search discovery while opting out of model training. These are not the same policy.

For example, OpenAI documents separate user agents for different purposes, including GPTBot, ChatGPT-User, and OAI-SearchBot. Google uses Google-Extended as a control token for some Gemini and Vertex AI use cases, while normal Google Search crawling is handled by other Googlebot user agents.

That separation is important. If you block broad user agents carelessly, you can harm ordinary search visibility while trying to block AI training.

A reasonable AI-blocking robots.txt template

Here is a conservative starting point for blocking several commonly documented AI-related crawlers while leaving general search crawlers alone:

# AI training and AI product crawlers
User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

# Default rule for other crawlers
User-agent: *
Allow: /

This is not a magic universal list. It is a maintainable pattern.

A few notes:

• Disallow: / means “do not crawl any path.”
• User-agent: * applies to crawlers not matched by a more specific group.
• Allow: / is not strictly required for the default group, but it makes your intent clear.
• Keep comments short. Some parsers are forgiving, but robots.txt should stay boring.
• Do not include private URLs in robots.txt. The file is public, and listing sensitive paths can advertise them.

The last point is worth repeating. Robots.txt is not a secrecy mechanism. If /client-contracts/ should not be public, protect it with authentication. Do not merely disallow it.

Be careful with Google-Extended

Google-Extended is widely misunderstood. It is not the same as blocking Google Search.

According to Google’s documentation, Google-Extended is a standalone product token that publishers can use to manage whether site content may help improve certain Gemini and Vertex AI capabilities. Blocking it should not, by itself, block Googlebot from crawling for Search.

That said, do not replace all Google directives with a broad block like this unless you truly mean it:

User-agent: Googlebot
Disallow: /

That would tell Google Search’s main crawler not to crawl your site. For most public websites, that is not what you want.

The same distinction applies elsewhere. Some vendors separate training crawlers from user-triggered browsing or AI search crawlers. Others do not. You need to read the documentation for the bots you care about and treat your robots.txt as a living file, not a one-time checkbox.

Test the file like production code

Robots.txt looks simple, which is why it is easy to break.

Common mistakes include:

• Uploading it to the wrong place, such as /assets/robots.txt instead of /robots.txt.
• Using smart quotes copied from a document editor.
• Blocking all crawlers with User-agent: * and Disallow: / by accident.
• Assuming one domain’s file applies to another subdomain.
• Forgetting that http://, https://, www, and non-www hosts may be handled differently depending on your setup.

For multi-domain sites, check every canonical host. A robots file at https://www.example.com/robots.txt does not automatically govern https://app.example.com/robots.txt.

When debugging, inspect the actual HTTP response, not just what your CMS preview shows. You want a 200 OK response, text/plain content type if possible, and the exact file you expect. If redirects, caching, or CDN rules are involved, raw header inspection helps. The workflow in debugging redirects and HTTP headers in production applies directly here.

Add server-side controls for bots that ignore the rules

If the crawler is compliant, robots.txt is the cleanest signal. If the crawler is abusive, you need enforcement.

Practical controls include:

Rate limiting

Set thresholds for unusual request patterns: too many pages per minute, deep pagination traversal, repeated 404s, or high request volume from a small set of IPs. Rate limits should be generous enough not to punish real users and strict enough to make bulk extraction expensive.

User-agent filtering

You can block documented AI crawler user agents at the web server, reverse proxy, CDN, or application layer. This is stronger than robots.txt because it returns an actual denial response.

For example, Nginx can block a user agent pattern, though production rules should be tested carefully:

if ($http_user_agent ~* "GPTBot|CCBot|ClaudeBot|Bytespider") {
    return 403;
}

This is not foolproof. User-agent strings are easy to fake. But it stops the honest or lazy traffic and reduces load.

IP and ASN controls

Some operators publish IP ranges, but many scraper ecosystems do not. IP-based blocking can work for obvious abuse, especially from cloud hosting ranges with no normal user traffic, but it can also create false positives. Use logs before rules.

Authentication and paywalls

If content must not be copied at scale, do not put the full content on a public URL. Robots.txt is unsuitable for confidential material, licensed databases, private communities, or paid archives.

Content minimization

Sometimes the best protection is architectural. Do not expose unnecessary APIs, large JSON payloads, hidden metadata, draft endpoints, or full archives if the public page only needs a small subset. Image-heavy sites should also think about what metadata they publish; the privacy logic in stripping EXIF metadata before sharing photos online applies to content operations too.

Use robots meta tags for page-level rules

Robots.txt controls crawling. Robots meta tags and X-Robots-Tag headers control indexing and snippet behavior for compliant search engines and crawlers.

For example:

<meta name="robots" content="noindex, noarchive">

Or as an HTTP header:

X-Robots-Tag: noindex, noarchive

These are not AI-specific shields. They are useful when you want a page accessible but not indexed. However, if you block a crawler from fetching a page in robots.txt, it may never see the page-level meta tag. Do not rely on a noindex tag on a URL that the crawler is forbidden to crawl.

The rough rule:

• Use robots.txt to reduce or prevent crawling.
• Use meta robots or X-Robots-Tag to control indexing behavior.
• Use server-side controls to enforce access.

Monitor logs after publishing

Publishing the file is only step one. After that, check your logs.

Look for:

• Requests to /robots.txt from the user agents you named.
• Continued crawling after disallow rules are served.
• Suspicious user agents with high volume.
• Browser-like user agents requesting thousands of pages in sequence.
• Repeated access to feeds, sitemaps, search pages, and pagination.

If a bot requests robots.txt, sees a full disallow, and then stops, robots.txt did its job. If it continues, move that bot into enforcement: rate limits, blocks, or authentication.

Also review your sitemap exposure. Sitemaps are useful for search engines, but they are also convenient maps for scrapers. That does not mean you should remove them from ordinary sites. It does mean you should not include URLs you do not want public systems to discover.

Keep the file small and reviewed

Robots.txt tends to rot. A marketing team adds a campaign microsite. A developer adds a staging path. A vendor changes its crawler name. Two years later nobody knows why half the rules exist.

Treat it as configuration:

• Store it in version control when possible.
• Add a short comment for each AI crawler group.
• Review it quarterly.
• Check vendor documentation before adding broad rules.
• Test after CDN, CMS, or hosting changes.

If your site publishes AI-assisted content, also separate crawler policy from editorial transparency. Blocking AI scrapers is about access and reuse. Disclosure is about reader trust. They overlap ethically, but they are not the same control. A practical disclosure approach is covered in what honest AI disclosure looks like on a small website.

<!-- tool-cta:start -->

💡 Try this: After adding rules for AI crawlers, confirm the syntax with the Robots.txt Tester so you don't accidentally block legitimate bots too.

<!-- tool-cta:end -->

The bottom line

A good robots.txt file will block compliant AI crawlers. It will not stop determined scraping, copied user-agent strings, compromised browsers, or people pasting your content into AI systems manually.

That does not make it useless. It makes it one layer.

Write explicit rules for documented AI crawlers. Avoid broad blocks that damage search visibility. Test the served file, not the draft. Watch logs. Enforce with server-side controls where behavior crosses from unwanted into abusive.

The web has always run on a mix of protocol, norms, and enforcement. Robots.txt is the norms layer. Use it, but do not mistake it for a wall.