Your technical scan answers one question: can AI crawlers read your site? It fetches your pages as GPTBot, OAI-SearchBot, ClaudeBot and the rest, then reports what came back.
That is one moment in time, from our IP address. It says nothing about the other 29 days of the month, and it cannot see a block aimed at a crawler's real IP ranges.
Connect Cloudflare and Site audit answers the harder question. What did they actually do?
What is new
There is a new section at the bottom of Site audit > Technical, called AI crawler access.
It opens with a verdict, in one line, so you know where you stand before reading a number. There are 4, in order of severity:
| Verdict | What it means |
| Blocked | Your CDN's AI bot block is on. Nothing you publish can be quoted. |
| Not reading | No answer engine fetched a page, whatever other crawlers did. |
| Needs a fix | They are reading, but some pages turned them away. |
| Reading | They are reading and nothing failed. |
Under the verdict are the pages that turned an engine away, each with the crawler, the status it got and the last time it happened. That is the part you can go and fix today, so it is the only detail shown by default. Everything else is folded away until you want it.
Why this matters for AEO
An engine can only quote a page it managed to fetch. Everything else in AEO, the prompts you track, the pages you optimize, the sources you try to get cited by, sits on top of that one assumption. Until now you had no way to check it.
The crawlers are grouped by what their visit means, which is where the useful surprise usually lives. On my own site over 29 days: 1,175 requests from crawlers that only collect training data, against 473 from the ones that feed AI answers. Training outweighs answers 2.5 to 1. And 3 of the 8 answer-engine crawlers never came at all, Claude-SearchBot among them.
A scan cannot tell you any of that. It also cannot see the most expensive failure mode in AEO: a WAF or bot rule that lets our scan through from an ordinary IP and quietly turns away the verified crawler IP ranges that matter. Measured traffic sees it immediately.
How it works
Once connected, we read your zone's bot traffic from Cloudflare's analytics. The first sync pulls the last 30 days in the background, then it keeps up day by day. Cloudflare keeps about 30 days, so our own stored rows become your longer history.
3 things are worth knowing about the numbers.
Only verified crawlers count. Most traffic claiming to be an AI crawler is not one. On a typical month, roughly 4 requests in 5 that name GPTBot or ClaudeBot come from somewhere else. We count only what Cloudflare verified, or what arrived from the IP ranges the AI vendor publishes. The impostors are counted separately, kept out of every figure, and reported in the footer so you can see the scale of it.
Pages mean pages. Robots.txt, sitemaps, favicons, scripts and stylesheets are filtered out. Without that, the most-read page on every site is robots.txt forever.
Findings become actions. A page turning an answer engine away raises an action on your Actions board, with the paths as evidence, and closes itself once a later sync shows the page loading again.
What you need
Cloudflare has to serve your site's traffic, not just host its DNS. In Cloudflare's terms that means proxied records, the orange cloud. If your DNS is there but the traffic goes straight to your host, Cloudflare never sees the crawlers, and the panel tells you so instead of asking you to connect something with no data behind it.
It works on any Cloudflare plan, including the free one.
We ask for read-only access to 4 things: Zone Read, Zone Analytics Read, Zone Settings Read and Bot Management Read. The last 2 are optional and only read your AI bot block setting. Decline them and everything else still works. We never write anything to your account.
How to connect
- Open Site audit > Technical and scroll to AI crawler access.
- Click Connect Cloudflare.
- Approve on Cloudflare's own sign-in screen, where you will see those 4 permissions.
- If your domain does not match a zone name exactly, pick the right one from the list.
- The first sync starts in the background. The panel says how many days are in so far while it runs.
If you would rather not use OAuth, there is a link to connect with a read-only API token instead. Either way you can disconnect from the panel at any time, which removes the stored credential.
Bing and PostHog are next.