Crawler policy

Every crawler that reads your site falls into one of three states, and you pick the state per crawler from your site's Crawlers settings:

• Allow: reads free. For the crawlers you want visiting: search indexers, your own tooling, a partner's agent you've agreed terms with.
• Charge: pays the toll, like any other agent. This is the default posture.
• Block: refused outright with a 403. A blocked crawler cannot read even if it offers to pay. Use it for bots you never want training on or reselling your work at any price.

Humans always read free. No state here ever applies to a person in a browser. That's a hard rule of the toll, not a setting.

What the defaults are

Out of the box, the gate recognizes the major AI crawlers (GPTBot, ClaudeBot, CCBot, Bytespider, Amazonbot, and the like) and charges them. Crawlers it does not recognize (smaller AI agents, SEO tools, archivers) read free until you say otherwise, because the gate refuses to guess: a request it can't confidently call a machine is treated as a person, and people read free.

The Crawlers screen shows you which is which. A crawler with no explicit choice displays its actual default ("pays by default" for recognized AI crawlers, "reads free by default" for the rest), so the list never claims an enforcement that isn't happening. Setting a crawler to Charge makes it pay even if the gate wouldn't have recognized it on its own.

Search engines

Googlebot and Bingbot index you for free by default, and you should almost certainly leave them that way. Blocking a search crawler deindexes you. The screen warns you before you do it.

One exception is worth knowing about, because it looks like a search engine and isn't priced like one. Exa's crawler (ExaSearchBot) is charged by default. Its own documentation describes it as search, and for the consumer site it is, but the same crawler feeds an API that hands an application your full page text, served from Exa's cache. A reader who gets your article that way never arrives at your site, and Exa publishes one user-agent for both halves, so there is no way to allow the indexing and charge the retrieval separately.

That default is a judgement, and it costs something: charge Exa and you drop out of its index, consumer surface included. If your priority is reach rather than revenue from that channel, set it to Allow on the Crawlers screen and it reads free like Googlebot. The point is that you get to decide it rather than discover it.

Custom crawlers

You can add your own entries by user-agent fragment. Type a fragment (say, mytestbot), pick a state, and the gate matches it case-insensitively against the user-agent of every request to a tolled path. Handy for letting your own test agent through free while you integrate, or for blocking a scraper that isn't in the built-in list.

Two things to know about matching:

• Fragments match as substrings, so gptbot matches Mozilla/5.0 (compatible; GPTBot/1.0).
• User-agent strings are self-reported. A bot can lie about who it is, so treat UA matching as intent enforcement for honest crawlers, not a security boundary. For operators that cryptographically sign their requests, the toll verifies identity for real; see Verified crawler identity.

Tolling a section or your whole catalog with one price

You don't need a per-article catalog to start charging. When you connect a site, pick Simple toll: set one price, name the path sections your articles live under (essays, posts), and connect a payout wallet. Every article under those sections tolls automatically and every settlement lands in that one wallet. You can graduate to per-article credits, authors, and splits later. Nothing about the simple setup has to be undone.