Bots & AI crawlers

Because MetrikStack can see traffic beyond the browser (server, edge, log sources), it also sees a lot of what a script-only tool never does: search engine crawlers, uptime monitors, scrapers, and increasingly, AI crawlers and AI assistants fetching pages on a user's behalf. Rather than silently discarding this traffic, MetrikStack classifies it and stores it, so you can see and report on it deliberately.

How classification works

Classification is layered and deterministic:

1. Known-bot user-agent list: a maintained list, grouped by category, covering among others GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-Web, anthropic-ai, PerplexityBot, Perplexity-User, Google-Extended, Googlebot, Bingbot, Applebot, Amazonbot, CCBot, Bytespider, Meta-ExternalAgent, DuckAssistBot, YandexBot, AhrefsBot, SemrushBot, and common uptime monitors. New bots are added to the list without any change on your side.
2. Heuristics for user agents not on the list: missing Accept-Language, HEAD requests, no referrer combined with no user agent, or an unusually high request rate from one visitor hash.
3. Source-aware rule: an event from the script source is almost certainly not a crawler (crawlers rarely execute JavaScript); the same user agent seen from an edge, server or logs source is treated as a bot.

What categories exist

Each event carries is_bot, bot_name and bot_category. Categories include:

• ai_crawler: AI companies' training/answer crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, …)
• search: traditional search engine crawlers (Googlebot, Bingbot, …)
• monitor: uptime and synthetic monitoring services
• scraper: SEO tools and generic scrapers (AhrefsBot, SemrushBot, …)
• unknown: flagged as a bot by heuristics but not matched to a named entry

Storage and default filtering

Bots are stored, never dropped at ingest. Dropping them would make it impossible to report on which AI crawlers read your site and how much referral traffic AI assistants send back.

The dashboard filters bots out by default everywhere except the dedicated Bots & AI crawlers page, which does the opposite: it defaults to showing bot traffic and lets you break it down by name and category. The query API mirrors this: POST /v1/query takes include_bots (default false); set it to true, or filter by bot_category / bot_name as a dimension in POST /v1/query, to see it explicitly.

Getting told when it changes

Watching the page is one thing; being told is another. Two of the alert kinds are built on exactly this data: ai_crawler_spike fires when AI-crawler traffic jumps against the same window a week earlier, and ai_referral_spike does the same for visits arriving from an assistant's answer. Both deliver to a signed webhook or by email.