SEO & Discoverability
How to write a robots.txt that actually blocks AI scrapers
robots.txt can block many AI crawlers, but only when written precisely—and only for bots that choose to obey it.
9 min read