Verified crawler identity

A user-agent string is a claim, not proof. Anything can call itself GPTBot, and a bot that wants to dodge your policy can call itself Chrome. Crawler policy matched on user-agent alone is intent enforcement for honest crawlers.

Web Bot Auth closes that gap for the operators that sign. It's an open standard (HTTP Message Signatures, RFC 9421) where a bot operator publishes its public keys at a fixed URL on its own domain:

https://chatgpt.com/.well-known/http-message-signatures-directory

and signs each request with the matching private key. The toll verifies the signature against that directory on the fly. When it checks out, the request's identity isn't a string in a header anymore. It's cryptographic proof, keyed to a domain only the operator controls.

You don't configure any of this. Verification runs automatically on every request that presents a signature.

What you see

On your Audit page:

• Requests with a verified signature carry a check mark and the operator's directory host (chatgpt.com) instead of hiding behind whatever user-agent they wore. The "By agent" table buckets them under that identity, so one operator's traffic rolls up to one row even across different user-agent strings.
• A request that presents a signature that fails verification is flagged as a masquerade attempt. That's not ordinary unsigned traffic: someone went to the trouble of faking a signing operator's identity. The audit page counts these separately.

On your Crawlers settings, operators that publish a key directory carry a Verified identity badge. The badge means the operator's signed traffic is verifiable. It doesn't change your policy by itself.

What it means for your policy

Your allow/block rules match verified identity as well as user-agent. Two consequences worth knowing:

• A blocked operator can't dodge by rotating its user-agent. If your block list matches its verified identity, the block holds even when the request arrives wearing a browser-shaped UA.
• To allow or block a signing operator outright, add its directory host as a custom crawler rule. For example, a custom rule for chatgpt.com matches every request OpenAI signs, regardless of which product sent it.

What it doesn't do

• Unsigned crawlers classify by user-agent, exactly as before. Most crawlers don't sign yet; signing is growing from the largest operators down.
• A failed signature never blocks a request by itself. The request is treated as unsigned and your normal policy applies; the failure is recorded as telemetry, not turned into an outage for a bot with a broken clock.
• Humans are never involved. Signatures are for bots identifying themselves; people read free, always.

Who signs today

Checked against the live directories on 18 August 2026: OpenAI publishes one at chatgpt.com and signs its ChatGPT agent traffic; Meta publishes one at meta.com, covering Meta-ExternalAgent, Meta-ExternalFetcher and Meta-WebIndexer; Ahrefs publishes one at ahrefs.com. Cloudflare verifies the same standard in front of the sites it proxies.

Exa joined them, checked 2 September 2026: its directory sits at crawler.exa.ai, not on its main domain, and it signs every request its search crawler makes. That subdomain matters if you write a custom rule: the host you match is the one serving the directory, not the company's homepage.

The list is growing. Meta was not publishing when we last checked in July. Still not signing as of the same check: Anthropic, Perplexity, Google, Bing, Apple, Amazon, ByteDance and DuckDuckGo, so their crawlers classify by user-agent alone. The Crawlers screen badge tracks the operators we have verified a directory for, and we re-probe rather than trusting an announcement.

So does our own research agent. Every request the naulon buyer makes, the price probe and the paid read alike, is signed the same way, against a key directory we publish. We hold our species to the standard we verify: if your site sits behind any Web Bot Auth verifier, ours is a crawler it can prove.