AI Bot Log File Analysis: See Which AI Crawlers Visit a Client's Site

Every AI citation starts with a fetch. Before ChatGPT, Claude or Perplexity can quote a page, one of their crawlers has to request it. Analytics tools cannot see that happen, because crawlers do not run tracking scripts. Server logs can.

Log analysis answers questions no other data source can: which AI systems visit a client's site, which pages they read, how often, and whether they hit errors. This guide covers how to collect the data, how to verify it, what to look for and how to report it.

Why logs matter for AI visibility

Logs tell you whether the first link in the citation chain is working. If OAI-SearchBot never requests a client's service pages, those pages cannot appear in ChatGPT search. If PerplexityBot keeps hitting a firewall challenge, the site cannot be cited in Perplexity. No amount of content work fixes an access problem you cannot see.

Logs also separate the three kinds of AI crawler traffic:

• Training crawlers such as GPTBot and ClaudeBot collect content for model training.
• Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot build AI search indexes.
• User-triggered agents such as ChatGPT-User, Claude-User and Perplexity-User fetch pages because someone asked.

The third group is especially useful. A spike in ChatGPT-User requests for a particular page usually means people are asking ChatGPT about that page or topic. See AI crawler user agents for the full list.

Step 1: Get the logs

Where logs live depends on the hosting setup.

If a CDN sits in front of the site, use CDN logs where possible. The CDN sees every request, including those it blocks or challenges before they reach the server, which is exactly what you need to spot firewall problems.

Ask for at least 30 days of logs with timestamp, IP address, user agent, requested URL, response status code and response size.

Step 2: Filter for AI user agents

Filter the logs for user-agent strings containing the AI crawler names. A simple command-line example for a standard access log:

grep -Ei "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Google-Extended|Applebot|CCBot|meta-externalagent|Bytespider" access.log > ai-bots.log

Then group by agent to see the volume per crawler:

grep -oEi "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User" ai-bots.log | sort | uniq -c | sort -rn

For larger sites, load the filtered data into a spreadsheet, a database or a log analysis tool so you can group by URL, status code and date.

Step 3: Verify the bots are genuine

User-agent strings are easy to fake. Scrapers often pretend to be GPTBot or Googlebot to get past filters. Before drawing conclusions, verify a sample of requests.

• OpenAI and Perplexity publish JSON lists of IP ranges for their crawlers. Check request IPs against them.
• Google supports verification by reverse DNS lookup: the IP should resolve to a googlebot.com or google.com hostname, and a forward lookup should return the same IP.
• Anthropic documents its crawlers and how to identify them in its help centre.

Exclude unverified requests from the analysis, and consider blocking persistent impostors at the firewall.

Step 4: Analyse what the crawlers do

Once you have verified AI crawler requests, five questions cover most of the value.

Which crawlers visit, and how often?

A healthy site open to AI search will usually show regular visits from OAI-SearchBot, PerplexityBot and Claude-SearchBot, plus training crawlers if they are allowed. A complete absence of a search crawler is worth investigating.

Which pages do they request?

Compare the URLs AI crawlers request with the client's priority pages. If they mostly fetch old blog posts and never reach the service pages, internal linking, sitemaps or access rules may be steering them away.

What response codes do they get?

Group requests by status code.

• 200 is healthy.
• 301 and 302 chains waste crawls and can hide final pages.
• 403 and 429 usually mean a firewall, rate limit or bot-protection rule is blocking the crawler, even if robots.txt allows it.
• 404 means crawlers are following broken links or old URLs.
• 5xx errors mean the server failed, which can make engines treat the site as unreliable.

Are user-triggered fetches rising?

Track ChatGPT-User, Claude-User and Perplexity-User requests per page over time. Pages with growing user-triggered fetches are being discussed in AI conversations, and they are good candidates for content investment.

Do crawlers request robots.txt and llms.txt?

Most AI crawlers request robots.txt regularly. Very few request llms.txt; analyses across many sites have found most llms.txt files receive no AI requests at all. Checking the client's own logs is the most convincing way to show this. See does llms.txt actually work?.

Step 5: Turn findings into fixes

For the readability check, see can AI crawlers read JavaScript?. For quotability, see answer-first content.

Step 6: Report it monthly

A short monthly section in the client report keeps access visible:

• Verified requests per AI crawler, compared with last month.
• The share of priority pages fetched by at least one AI search crawler.
• Error rates for AI crawlers, with any fixes made.
• Pages with the most user-triggered fetches.

Pair it with citation tracking, so the client can see access and outcomes side by side. See AI share of voice and our SEO client report template.

What a monthly AI crawler summary looks like

A one-table summary is usually enough for clients. Here is an illustrative example for a mid-sized service business; the numbers are examples, not benchmarks.

Two lines of commentary turn the table into insight: which priority pages are still not being reached and why, and which pages are attracting user-triggered fetches and might deserve more investment.

Patterns by site type

Small service businesses. Expect low but regular volume from search crawlers. The most common problem is a hosting firewall or security plugin that challenges unknown bots, producing 403 responses for AI search agents.

E-commerce. Crawlers often spend most requests on filtered category URLs and parameter variations. Tighten canonical tags, parameter handling and sitemaps so crawls reach product and category pages that matter.

Publishers and content-heavy sites. Training crawlers can generate very high volume. Many publishers block training agents while allowing search agents, and logs confirm whether the split works as intended.

SaaS and documentation sites. Developer documentation often attracts user-triggered fetches from coding assistants. That traffic is a useful signal of which docs pages developers ask AI tools about.

Privacy and data handling

Server logs contain IP addresses, which can be personal data. When you request logs from a client, agree what you need, filter out non-bot traffic early, store the data securely, and delete it when the analysis is done. Many agencies ask clients to export only requests matching bot user agents, which avoids handling visitor data at all.

How Rankwyn helps

Rankwyn connects to each client site and records verified AI crawler activity per page, flags blocked or failing requests, and shows which priority pages AI search crawlers have reached, alongside citation results from the AEO Citation Radar. It works across every site in a portfolio from one view.

Key takeaways

• Server or CDN logs are the only direct record of AI crawler activity; analytics cannot see crawlers.
• Separate training, search and user-triggered agents, and verify bots by IP before trusting the data.
• Look at which pages are fetched, how often, and with what status codes; 403 and 429 responses usually point to firewall rules.
• Rising user-triggered fetches show pages that people are asking AI assistants about.
• Report access monthly next to citation results so clients see cause and effect.

The bottom line

If AI crawlers cannot reach a page, it cannot be cited. Logs show you whether they can. Collect 30 days of logs, filter and verify AI agents, check which priority pages they reach and what errors they hit, and fix access problems before investing in content. It is unglamorous work, and it often explains why a good site is missing from AI answers.