This page hosts the open data behind our AI crawler study. It aggregates 65,116,344 verified AI-bot requests across 38 websites, logged from 17 October 2025 to 7 July 2026 in Cloudflare bot logs. Free to use with attribution. No gating.
The full analysis and the 11 findings are in the study itself, What 65 Million AI Crawler Requests Reveal About AI Visibility.
How to cite this dataset
Mostash (2026), AI Crawler Behavior Dataset, mostash.com/ai-crawler-dataset. Aggregated from first-party Cloudflare verified-bot logs across a 38-site portfolio. Licensed CC BY 4.0.
The files
| File | Size | What it contains |
|---|---|---|
| 01-crawler-summary.csv | 7 rows | Per crawler totals, share of traffic, and the percentage of requests that reached a live page (HTTP 2xx), with and without the outlier site. |
| 02-monthly-by-crawler.csv | 58 rows | Monthly request volume per crawler, October 2025 to July 2026, outlier excluded. July is a partial month. |
| 03-content-freshness.csv | 5 rows | Typical (median) age in days of the content each bot fetched, per site, ChatGPT-User vs GPTBot. |
| 04-wordcount-attention.csv | 8 rows | Median AI-bot requests per page by word-count band, on two content-heavy sites. |
| 05-site-level.csv | 22 rows | Per-site totals, GPTBot vs ChatGPT-User split, success rate, and reads-per-crawl ratio for every site above 10k requests. |
| 06-perbot-status.csv | 7 rows | Full HTTP status breakdown (2xx, 3xx, 404, other 4xx, 5xx) per crawler, with and without the outlier site. Backs the per-bot error audit. |
| 07-content-type-attention.csv | 9 rows | AI-bot requests per page by content type (review, guide, comparison, listicle, and more), across both study periods, with page counts. |
| 08-time-to-crawl.csv | 5 rows | How fast each bot first crawls new posts: typical days, the usual spread, and the share of new posts it never crawled. Cohort of 333 posts published after 1 November 2025. |
| 09-page-class.csv | 16 rows | AI-bot attention by page class (free minitool vs product landing vs blog), pooled and per product domain, with the live-fetch share and success rate. |
| 10-crawl-vs-serp.csv | 20 rows | Per-site relationship between how often AI bots crawl a URL and its Google impressions, clicks, and average position, in both study periods. |
Column dictionary
- requests_total / requests_excl_outlier. One site (an app in our portfolio) accounts for about 80% of raw volume and skews every average, so we report both. Prefer the excluding-outlier figures for general claims.
- success_2xx_pct and the status columns. The share of a crawler’s requests ending in each HTTP status class. 2xx means it reached a live page. Heavy 3xx is redirect chasing, heavy 5xx is your server failing the bot.
- reads_per_crawl. ChatGPT-User requests divided by GPTBot requests. Above 1 means the page or class is read into live answers more than it is crawled for training.
- median_age_days. The typical age of the pages a bot fetched, weighted by how often each page was requested. Lower means fresher.
- time-to-crawl columns. For a new post, the typical number of days until a bot first crawls it (p25 and p75 show the usual spread), plus how often the bot never showed up.
- content-type medians and means. Requests per page for each content type. The median is the typical page, the mean is pulled up by a few heavily fetched pages, so we publish both.
- correlation columns. How closely AI crawl frequency tracks a page’s Google performance. Positive for impressions and clicks means more-crawled pages get shown and clicked more. Google position is lower-is-better, so the near-zero, sign-flipping position values mean crawl volume does not track where a page actually ranks.
Method, in brief
The source is Cloudflare’s verified-bot classification; bots that evade it are invisible to us. Crawlers covered are GPTBot, ChatGPT-User, and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended, and Applebot-Extended. Google-Extended was only classified from 14 April 2026, so its near-absence partly reflects that start date.
This is one operator’s portfolio of English-language SaaS and content sites, not a web-wide sample. The full methodology, the hypotheses we locked in before the analysis ran, and the stated limitations are on the study page.
License
CC BY 4.0. Use it, share it, build on it. Credit Mostash and link back to this page. The URL of this page is permanent; every file above stays where it is so your citations keep resolving.