Our application serves a deliberately simple robots.txt. Three lines:
User-Agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlThe file the internet received was about sixty lines long, blocked nine named crawlers, and carried a directive we had never heard of. Nobody on our side had written it.
What Was Actually Being Served
Fetching our own robots.txt over the public domain returned a large block wrapped in these markers:
# BEGIN Cloudflare Managed content
User-agent: *
Content-Signal: search=yes,ai-train=no,use=reference
Allow: /
User-agent: Amazonbot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: meta-externalagent
Disallow: /
# END Cloudflare Managed ContentOur own three lines were appended underneath, where they had no practical effect on anything the block above had already decided.
This is a Cloudflare feature called Managed robots.txt, under AI Crawl Control. It was enabled on the zone without anyone choosing it, which appears to be the default for a lot of zones. If you have never opened that page, it is worth doing so before reading further, because the odds are good that this is running on your domain too.
curl -s https://yourdomain.com/robots.txt | grep -i "cloudflare managed"The Part That Is Not Actually About AI
Two costs here, and the smaller one is the easier sell.
That Content-Signal: line is a Cloudflare proposal, not part of the robots.txt standard. Combined with the duplicate User-agent: * group it creates (Cloudflare's, then yours), automated auditing tools report the file as invalid. In our case Lighthouse had been reporting an SEO score of 92 for weeks, with the only deduction being robots.txt validity. Turning the feature off, so the application's own file was served again, took it to 100 with no other change.
Search Console agreed, moving from flagging the file to reporting robots.txt: All files are valid.
So if you have been hunting a stubborn SEO deduction on an otherwise clean site, check this before you touch anything else.
The Part That Is About AI
The bigger question is the nine blocked crawlers, and it deserves a clearer framing than it usually gets.
Blocking Google-Extended does not affect your normal Google Search ranking. Googlebot is a separate user agent and is not in that block list. Your search rankings are unaffected either way. This surprises people and it is the single most important fact in the decision.
What the block does affect is whether an AI assistant can read your site when a user asks it a question. If someone asks an assistant for a recommendation in your field, a blocked site cannot be part of the answer, no matter how good its content is.
The two things being conflated are worth separating:
- Training on your content. You get nothing from this. Declining is entirely reasonable.
- Reading and citing your content in an answer. This is a referral with your name on it, which is what marketing content is for.
Managed robots.txt bundles both into one switch. For a business whose content exists to attract customers, that bundle is a bad trade.
Read the Dashboard Carefully
Before flipping anything, we looked at Cloudflare's AI Crawl Control report to see what was actually being blocked. It showed 825 AI crawler requests, which initially read as a lot of blocked demand.
It was not. The most-crawled path was one of our own API subdomains, and the traffic was overwhelmingly our own tooling authenticating against our own service, being classified as AI crawler traffic because of its user agent. Once that was excluded, genuine external AI interest in the marketing site was three requests from one crawler and one from another.
That changed the conclusion honestly. Turning the block off removes a self-imposed ceiling. It does not create demand that was not there. Anyone telling you that unblocking AI crawlers will bring traffic is selling something. Unblock it because there is no good reason for the block, not because you expect a wave.
What To Do
- Fetch your own robots.txt over the public domain, not from your repository. They are frequently different files.
- If you see the Cloudflare Managed block, decide deliberately instead of leaving a default in place.
- If you turn it off, check the per-crawler settings on the same page too. Managed robots.txt only writes a request. A block action there is enforced at the network layer and is the one that actually stops a crawler.
- Re-run Lighthouse afterwards. If robots.txt validity was your only SEO deduction, that is a free ten points.
The broader lesson has nothing to do with AI. A CDN that can rewrite files your application serves is a CDN that can change your site's behaviour without a deploy, without a commit, and without anything appearing in your logs. Whatever you decide about crawlers, it is worth knowing which of your files your edge is authoring on your behalf.