Back

The worst part? These companies are ignoring well-established robots.txt standards. They have been caught ignoring the robots.txt file multiple times and are now creating issues for the rest of us. They use residency proxy services even if you block all data center ranges. It is like a war for them to collect all code, images, PDFs, books, videos, and stuff on the Internet to train their LLM without paying for anything. It is wild how they are getting away with copyright violations.

@nixCraft It’s probably time to try something like this https://xeiaso.net/blog/2025/anubis/