AI crawlers and bots

Server, edge and log sources see requests a browser script never does: search engine crawlers, uptime monitors, scrapers, AI crawlers and AI assistants fetching a page because someone asked them a question. MetrikStack classifies this traffic and stores it, so you can report on it.

How classification works

Classification runs in layers and gives the same answer for the same input:

1. Known bots by user agent: a maintained list, grouped by category. It covers GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, MistralAI-User, Googlebot, Bingbot, Applebot, Amazonbot, CCBot, Bytespider, Meta-ExternalAgent, DuckAssistBot, YandexBot, AhrefsBot, SemrushBot, common uptime monitors and more. New bots are added without any change on your side. Google-Extended and Applebot-Extended are not on it: they are robots.txt tokens that never appear as a user agent, because Google and Apple collect AI data with their regular search crawlers.
2. The network a request came from: a browser user agent from a cloud or hosting network (Amazon Web Services, Google Cloud, Azure, DigitalOcean, OVHcloud, Hetzner and similar) is counted as a bot. People browse from home, mobile and office networks; scanners and scrapers run on servers. Networks that people browse through, such as iCloud Private Relay, Cloudflare WARP and consumer VPNs, are not on that list. These requests get the provider as bot name, for example Hetzner (datacenter).
3. Probes: a page request that the site refused or couldn't find (a 4xx status) and that has no referrer is a scanner looking for /graphql, /.env or /wp-login.php. People following a broken link arrive with a referrer. The visitor stays flagged for 30 minutes, because scanners mix refused paths with ordinary pages. The bot name is "Probe".
4. Other signals for user agents not on the list: HEAD requests, no user agent at all, a very high request rate from one visitor, and, for sources that see request headers (server SDKs, the Cloudflare Worker), a missing Accept-Language together with no referrer. Access logs never contain Accept-Language, so for them its absence says nothing.
5. The browser script almost never reports a crawler, because crawlers rarely run JavaScript. For events from the script source only the known-bot list and the network check apply.

Traffic classes

Every event gets one traffic_class. This is the split the dashboard leads with:

People also get a channel (direct, search, social, ai, email, paid, campaign, referral) that says how they arrived. ai covers both a referrer from an assistant and links an assistant tagged with utm_source. ChatGPT tags its links this way and often sends no referrer.

Bot categories

Next to the class, each event carries is_bot, bot_name and the more detailed bot_category:

• ai_assistant: fetches made for a person, at the moment they ask
• ai_search: AI search indexers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, …)
• ai_training: crawlers collecting training data (GPTBot, ClaudeBot, CCBot, Bytespider, …)
• search: search engine crawlers (Googlebot, Bingbot, …)
• seo: SEO tools (AhrefsBot, SemrushBot, …)
• social: link previews (facebookexternalhit, LinkedInBot, Slackbot, …)
• monitor: uptime and synthetic monitoring services
• scraper: generic HTTP clients, headless browsers and scrapers
• datacenter: a browser user agent from a cloud or hosting network
• unknown: flagged as a bot by the other signals (including probes) but not matched to a named entry

Verified bots

Anyone can put GPTBot in a user agent. To tell the real crawler from an impostor, MetrikStack checks the IP address of the request against the address list the bot's operator publishes. Each event gets a bot_verification:

• verified: the request came from the operator's published network.
• failed: the operator publishes its network and the request came from somewhere else. The event counts as other bots (automation), whatever its user agent claims, so impostors don't inflate your AI or search numbers.
• empty: not checked. Either the visitor is not a bot, the operator publishes no list, its list isn't available right now, or the request's own address is unknown (a server source that sends no visitor IP).

Verified today:

The lists are refreshed daily. Bots whose operator publishes no list, such as Meta's and Amazon's crawlers or Bytespider, are never marked failed: their events stay unchecked and keep the class their user agent gives them. A bot is only marked failed against a current list from its operator.

The check happens in memory while the request is processed. The address is discarded right after, like every IP (see Privacy).

Requests from MetrikStack itself

For sites with a Bunny or access-log source, MetrikStack requests the home page about once a day, and one of the files it loads, to see whether the site's log will show the files a browser fetches with a page: where those files live and how long browsers keep them. These requests identify themselves as MetrikStack-SiteCheck and show up among your other bots with the bot name "MetrikStack". Nothing from the page is kept apart from three yes-or-no answers.

Where bots show up

Bots are stored at ingest. Without them you couldn't see which AI crawlers read your site or how many people AI assistants send back.

The dashboard shows them on three pages. Overview starts with every request, split into people, AI assistants, AI crawlers, search engines and other bots, and lets you focus on any one of them. People is web analytics for people only. AI visibility breaks AI traffic down by assistant and purpose, from training crawls to the people an answer sends to your site.

The query API returns people only by default: POST /v1/query takes include_bots (default false). Set it to true and break down or filter by traffic_class, bot_category, bot_name or bot_verification in POST /v1/query to see the rest.

Alerts on AI traffic

Two of the alert kinds use this data. ai_crawler_spike fires when pageviews by AI crawlers and AI assistants jump compared with the same window a week earlier. ai_referral_spike does the same for people arriving from an AI assistant. Both deliver to a signed webhook or by email.