by lukasz | Jun 6, 2026 | Essays
For thirty years, the deal was simple. You put a robots.txt at the root of your site, it said allow or disallow, and crawlers obeyed. It was a gentlemen’s agreement: you stated a preference, well-behaved bots respected it, and in...
