How AI Crawlers Read Your Content
I often hear people say AI crawlers are taking content, but I never actually counted how many times these bots visited the articles versus the index pages. So, I dug into the Apache logs from the last 29 days and tallied the article URLs and sitemap URLs for each bot. The articles were scraped 674 times, and the index pages were read 653 times—pretty close. Of those 653 reads, 419 were actually "lists of lists," not just main article lists. The bots even had different habits. Googlebot only looked at 25 articles over the 29 days, visiting each about 3.3 times, while GPTBot visited 81 articles, usually just once.
I Was Wrong to Focus on "Is This Our Content?"
This server hosts multiple websites, and they all write to one Apache log file. Since the log doesn't specify which domain the request came from, I had to figure it out by looking at the request paths. At first, I tried counting requests to the main blog index, which is located at /blog/.
It turned out Googlebot hit /blog/ 134 times. But this specific /blog/ is a page I created today. It couldn't have been visited 134 times starting 29 days ago. There must have been another site on the same server that also used /blog/. If paths overlap, you can't use that path for tracking.
So, I narrowed down what I was counting. For the articles, I only counted requests to numbered URLs, and only if that number was actually in the list of IDs for both of my blogs. The sitemap is different. Since this is the only WordPress multi-site running on this server, any request starting with /wp-sitemap belongs to us.
# Gather IDs for both blogs (public + private)
wp post list --post_type=post --post_status=publish,private --field=ID | sort -un > ourids.txt
# Count only numbered URLs that are in our ID list, plus the sitemap
awk -v IDF=ourids.txt '
BEGIN { FS="""; while ((getline id < IDF) > 0) OK[id]=1 }
{ split($2,r," "); p=r[2]; split($3,s," "); sub(/?.*/,"",p)
if (match(p,/^/[0-9]+/$/)) num = substr(p,2,length(p)-2); else num=""
... }' access_log-2026110?I Read the Table of Contents and the Content About the Same Number of Times
This covers a period from October 11, 2026, until the morning of November 9, which is 29.2 days. I combined four weekly log files with the current log.
| Bot | Article Requests | Unique Articles | Per Article | Sitemaps | Missing Numbers | llms.txt |
|---|---|---|---|---|---|---|
| bingbot | 162 | 55 | 2.95 | 248 | 7 | 0 |
| ClaudeBot | 148 | 70 | 2.11 | 318 | 0 | 0 |
| Applebot | 120 | 63 | 1.90 | 7 | 11 | 0 |
| GPTBot | 118 | 81 | 1.46 | 60 | 0 | 0 |
| Googlebot | 83 | 25 | 3.32 | 5 | 0 | 0 |
| Reflectionbot | 32 | 27 | 1.19 | 0 | 0 | 0 |
| Other 6 Bots | 11 | 11 | 1.00 | 15 | 0 | 0 |
| Total | 674 | 96 | – | 653 | 18 | 0 |
The other six bots are Amazonbot, Bytespider, CCBot, ChatGPT-User, OAI-SearchBot, and PerplexityBot. Across all twelve bots, over 29 days, they made 674 article requests and 653 sitemap requests. Half of what these bots did on this site was just reading the table of contents.
419 of Those 653 Requests Were Just Lists of Lists
WordPress doesn't generate a single sitemap file. The file at doesn't contain the actual article URLs; it only links to five separate sub-sitemaps. The file that actually lists the article URLs is .
| File | Requests | Contents |
|---|---|---|
/wp-sitemap.xml |
419 | Links to 5 sub-sitemaps |
…taxonomies-category-1.xml |
59 | Category archive URLs |
…users-1.xml |
47 | Author archive URLs |
…posts-post-1.xml |
47 | Post URLs |
…post_tag-1.xml |
42 | Tag archive URLs |
…posts-page-1.xml |
39 | Static page URLs |
Out of the 653 times the sitemap was read, only 47 times did it descend into the actual list of posts. This suggests that for the rest, they just checked the entrance to see if anything had changed and then left. Since the main sitemap file contains the last updated time for its sub-files, you can determine if anything has changed just by reading that one file. They are repeating the cheapest way to check.
Bots That Scan Broadly vs. Bots That Dig Deeply
If you convert these numbers into requests per post, the nature of the bots becomes clear.
Googlebot Only Looked at Twenty-Five Posts, Visiting Each About 3.3 Times
Over 29 days, Googlebot requested 83 posts, but only 25 unique posts. That's about a quarter of the 100 possible addresses. The rest were never even opened. However, the ones it did see were revisited an average of 3.32 times. It only read the sitemap five times over the twenty-nine days. It seems to be repeatedly checking a small selection of posts it has already chosen, without really browsing the sitemap.
GPTBot Visited Eighty-One Unique Posts Once
On the flip side is GPTBot. Out of 118 post requests, it covered 81 unique posts, averaging 1.46 visits per post. It scanned the widest range among all bots. It read the sitemap 60 times. ClaudeBot is somewhere in the middle, but the sitemap activity is unusual. It has 318 sitemap reads compared to only 148 post reads. This means for every one post it fetched, it effectively read the sitemap twice.
Applebot read the sitemap only seven times but requested 120 posts. This bot seems to skip the sitemap entirely and just pulls the posts.
I Hit the Missing Numbers Eighteen Times
There were eighteen requests that returned a 404 error for the URL addresses. Applebot made eleven of those requests, and Bingbot made seven. These are the post numbers from posts published on October 28th. Both bots had scraped those posts before I took them down.
The remaining ten posts never returned a 404. The difference between a bot that comes back to check a removed address and a bot that takes the content once and never returns is that the former is the one that lets you know a page is missing.
Only Four Posts Were Never Scraped
Out of a hundred post numbers, four were never picked up by any bot. Three of those were private posts that couldn't be accessed from the outside, and one was posted just seven hours before testing concluded. All ninety-six posts accessible from the outside were picked up by at least one bot.
The collection is thorough. We can't tell from the logs whether the collected content was actually used in search results or AI answers. The logs only tell us that it was "taken." This limitation was also true when counting the crawler that scraped the same article thirty-one times, as seen in that post.
llms.txt Was Zero
This site maintains an /llms.txt file, and I've noted its location in the robots.txt comments. Over the twenty-nine days, no bot requested that file out of the twelve types I tracked. That's zero.
The file returned a 200 status code forty-three times, but twenty-three of those requests came from commercial crawlers or requests pretending to be browsers gathering site information, and I sent twenty myself for verification. There were no AI bots reading the file I made for AI.
Things I Can't Explain
The bot names are just the text provided in the requests. Since I didn't cross-reference them with publicly available IP ranges, there might be some spoofing mixed in. The number in this post represents the total count of "requests that came with a specified name."
Because both blogs share the same number space, the same number might appear on both sides. Therefore, the bot counts are aggregated from both blogs, and I haven't specified which blog's content was taken. The sitemap count is also combined since all three domains use the same paths.
The 29-day window isn't large enough to capture the full cycle of the bots. For example, when Googlebot only viewed twenty-five posts, it wasn't because it was a "bot that only views twenty-five posts," but rather that "within this 29-day window, it viewed twenty-five posts."
Frequently Asked Questions
Can't I just add up the unique post counts for each week?
I initially tried that, but it led to an error. When I counted the unique posts for each of the four weeks and added them up, GPTBot came out to 117 posts, which is impossible since there are only a hundred available addresses. If a bot took the same article in both Week 2 and Week 4, it would be counted twice on a weekly basis. You need to count unique posts only once across the entire period. So, I paired the counts by bot name and article number, and then removed the duplicates at the very end.
Is reading the table of contents frequently a good sign?
This measurement doesn't tell me that. Reading the table of contents frequently just means the bot is "checking for updates often," not that it's "taking more articles." In fact, Applebot, which read the sitemap only seven times, took fewer articles than ClaudeBot, which read the sitemap the most. And Googlebot, which read the sitemap only five times, viewed the smallest amount. There's no discernible trend between the number of table of contents requests and the number of article requests.
What about requests that don't have a bot name listed?
I didn't count those. This post only covers the twelve types of requests that came with a name. Even among requests claiming to be browsers, there are bots mixed in, but since there's no way to distinguish them, I excluded them. Therefore, the 674 and 653 mentioned here are the minimum counts for requests that disclosed their name, not the total.
