Server Logs Reveal Who's Really Crawling You: 20+ Bot Signatures

Search Console tells you what Google sees. GA4 tells you what humans do. Neither tells you who is actually hitting your server — which bots, which user agents, which paths. When we wanted to know whether AI crawlers were reading our content pages, the answer was sitting in a file most people never open: the nginx access log.

What GA4 and Search Console cannot show you

GA4 is browser-centric and strips most bot traffic. Search Console only reports Google's crawl data. So neither could answer a question we cared about a lot: is GPTBot reading our articles? Is ClaudeBot hitting our blog? Which paths do Perplexity and Gemini actually fetch? The only place with this detail was the raw server log.

Parsing the logs: UA signatures and the multi-space trap

We parse nginx access logs in combined format and identify crawlers by User-Agent signature. The table is bigger than you might think: Googlebot (Desktop and Mobile), GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Gemini, and the Meta/Facebook bots — 20+ signatures in total.

One parsing detail matters: a User-Agent string contains spaces, so a naive space-split breaks. Log lines in combined format must be split carefully, keeping the UA intact, otherwise every AI crawler silently disappears from the report.

Aggregating by path and domain

With signatures recognized, we aggregate requests by URL path and by domain. That gives us, for each crawler, its top paths — for example, which content pages GPTBot fetches most. We also split the output into separate panels: Google crawlers in one, AI crawlers in another, so the two never blur together. And because we run multiple domains, filtering by domain matters — we can inspect one site at a time.

What this gives you

Suddenly the AI crawler behavior is visible: which articles GPTBot read, which product pages ClaudeBot visited, which paths Perplexity hit. That is direct evidence for content and GEO strategy — you can see what AI systems are actually reading before they decide whether to cite you.

The reusable checklist

• Pull the nginx access log. Combined or common format both work.
• Build the UA signature table. Include Google bots, AI crawlers (GPTBot, ClaudeBot, Perplexity, Gemini) and social crawlers.
• Aggregate by path and domain. Handle the multi-space UA problem, and support per-domain filtering.
• Rerun periodically. Trends matter more than one snapshot.

This log-intelligence workflow is exactly what the server log analysis inside our marketing analytics module does for you automatically — with AI crawlers split out from Google bots by default.