As someone who writes a fair bit of content across several sites, I’ve been manually removing the ability for AI bots to scrape my content. This process involves updating my robots.txt file across each site, which isn’t bad by any means, but it does feel like playing a game of whack-a-mole.
Cloudflare just introduced a new feature (on the free plan, too!) that denies these bots from scraping your sites, all with one click.
They have a huge amount of data to work from, and they will continue to update the list as they identify new bots emerging on the scene.
This feature seems like a no-brainer at this point, and it is another huge win for Cloudflare’s already amazing free service.