Cloudflare Bot Check: Keep AI Assistants Reading

You are on Cloudflare's free tier, with no distribution and a product you can still explain one-on-one. You are after the first 100 users, and the channel you can hand-hold is the one that can explain your product without you doing the explaining. AI assistants ask, compare, and recommend, so a default Cloudflare setting can hide your pricing page from them.

For new domains and free-tier accounts that did not change settings, Cloudflare's default blocks Agent and Training crawlers on ad-supported pages as of September 15, 2026. Cloudflare fronts more than 20% of web domains, so its September 15, 2026 crawler default reaches a large share of internet traffic. That is the exact situation many first-users sites are in: a simple landing page, maybe a pricing page, and no time to audit bot rules.

A default Cloudflare setting can hide your pricing page

This is a manual check, not a growth tactic that needs scale. You can do it while you are still answering user questions yourself. If you cannot separate Search, Agent, and Training behavior, the default blocks mixed-purpose crawlers on ad-bearing pages. The failure is easy to miss: your site still loads for humans, but the machines that feed AI answers may not receive the content.

A Training block can also stop Search-and-Training crawlers, so a legacy Block AI bots setting can block Googlebot, Applebot, and BingBot. A rule that looks narrow can reach the crawlers that keep your site findable while you are still hand-holding users.

Machine-driven traffic crossed the 50% mark a year before Cloudflare expected, and most AI crawl traffic repeatedly requests pages that have not changed. That traffic is real, but the default can still decide whether your product is part of the answer.

Ten minutes is enough to keep the channel open

You do not need a bot strategy. You need only a short list of pages and a few settings checked. The list covers the failure modes that matter for first users: missing pages, blocked crawler access, an old AI block, and no proof the fix worked. Do not skip the verification step, because a setting can look right in the account and still fail from the outside. Work through it once, then re-check after any redesign or ad change. The checklist is short because a missing page is a lost conversation.

• List the pages an assistant should read. Put your pricing page and product page in a note, open each one in a browser, and confirm it loads without login or paywall. If a page is behind either, an assistant cannot use it as a recommendation.
• Check the bot controls in your Cloudflare account and confirm Agent-class crawlers are allowed on public pages. If you see a block, remove it for the pages you want AI assistants to read.
• Keep Search crawlers allowed, because Cloudflare's Content Signals default allows Search-classified crawlers on pages carrying ads while blocking Training and Agent crawlers. Open the page in a normal browser to make sure it still appears.
• Remove any legacy Block AI bots toggle, then confirm your pricing page still loads in a normal browser.
• Verify with a test fetch. Ask someone outside your network to fetch your pricing page from a machine that is not behind your firewall, or run a simple web fetch from a different network. Confirm the response returns the page content, not a block page. If it fails, the setting is still wrong.

The pages you list are the pages you would hand to a stranger. If you would not send someone to a broken page, do not leave it as the page an assistant reads. Legibility is what turns a question about your product into a first user; a clever bot policy does not.