Cloudflare Tightens AI Bot Blocking

Introduction

Hello, I'm a professional engineer.
Today, I'll be discussing Cloudflare's upcoming change on September 15th: blocking access by AI bots for training purposes.

What You Need to Consider

If you're already using Cloudflare, after selecting your domain and navigating to "Security → Settings," scroll down a bit, and you'll find an option to block access by AI bots for training purposes. The consideration here is to decide and set this option before the September 15th release.

Blocking is the default, as it's safer from Cloudflare's perspective.

Important Notes

As shown in the image above, you need to be aware of "mixed-purpose crawlers."

As of September 15th, mixed-purpose crawlers used for both search indexing and AI training will be included in the AI training blocking.

Judging from this, it seems that even crawling by search engines might potentially be blocked. Although major search crawlers are usually clearly separated to avoid Cloudflare's rules, it's still unknown how much will actually be blocked.

Which Option Should You Choose?

So, I've outlined the options along with their pros and cons.
It seems the general consensus is that it's safer to choose "Allow" below.

Block (Will be blocked on September 15th)

• Pros: Completely prevents your site's data from being used for AI training.
• Cons: There's a risk that search engine crawlers might also be blocked, potentially removing your site from Google search results (killing your SEO).

Allow (Will continue to be allowed)

• Pros: Maintains access from search engines as before.
• Cons: There's a possibility that your site's information could be taken as AI training data.

So, Which Does the Author Choose?

As selected in the image above, I intend to default to blocking. My thinking is that if Google or Bing crawlers are blocked, I can simply configure security rules to allow them through.

Looking at Cloudflare's security logs, there's quite a lot of access from strange bots, and I definitely don't want to receive that directly on my own server. It seems bot access will only increase in the future... Personally, I feel it would be easier to manage with a whitelist.

Therefore, I'll observe the situation with blocking enabled!

In Conclusion

Anyway, I don't get much traffic to begin with lol

This article was translated from the original Japanese version (CloudflareのAI Botに対するブロックが厳しくなるぞ) using Gemini API.