Crawler policy

Who pays is decided in two rows of your site's Site settings, in two layers.

Rules by use come first. Under Who may read you pick one of three positions:

• Charge AI: agents pay your price to read or cite. Search engines index free.
• Free for AI: agents read and cite at no charge. No 402 is sent and no price is advertised, in your licence or in your payment manifest.
• Block AI: reading, indexing and training are refused with a 403, paid or not. Search engines still index free.

The card names the crawlers a choice would refuse before you confirm it, and you can set each use separately if none of the three fits.

Crawler exceptions come second. In the row below it, every crawler we know is listed with what the rules do to it right now. Make an exception to change one:

• Allow: reads free, whatever the price. A crawler that publishes its addresses only reads free from those addresses; see impostors.
• Charge: pays the toll, even if the gate would not have recognised it on its own.
• Block: refused outright with a 403, even if it offers to pay.

An exception can tighten a rule or let a crawler read free. It cannot undo a refusal: a crawler your terms refuse shows as locked, and Who may read is where you change it.

Humans always read free. Nothing on this page applies to a person in a browser.

What the defaults are

By default the gate recognizes the major AI crawlers (GPTBot, ClaudeBot, CCBot, Bytespider, Amazonbot and others) and charges them. Crawlers it does not recognize (smaller AI agents, SEO tools, archivers) read free until you say otherwise: a request the gate can't confidently call a machine is treated as a person.

The crawler list shows each crawler's state as the gate applies it, rules and exceptions combined. The line above it counts how many are charged, how many read free and how many are turned away.

Search engines

Googlebot and Bingbot index you free by default. Blocking a search crawler deindexes you, and the screen warns you before you do it.

Exa's crawler (ExaSearchBot) is charged by default, although Exa describes it as search. The same crawler feeds an API that hands applications your full page text from Exa's cache, so a reader who gets your article that way never visits your site. Exa uses one user-agent for both, so indexing and retrieval cannot be priced separately.

Charging Exa drops you out of its index, consumer search included. If you want the reach more than the revenue, set it to Allow in the crawler list and it reads free like Googlebot.

Custom crawlers

You can add your own entries by user-agent fragment. Type a fragment (say, mytestbot), pick a state, and the gate matches it case-insensitively against the user-agent of every request to a tolled path. Use it to let your own test agent through free while you integrate, or to block a scraper that isn't in the built-in list.

Matching works like this:

• Fragments match as substrings, so gptbot matches Mozilla/5.0 (compatible; GPTBot/1.0).
• User-agent strings are self-reported, so UA matching only governs honest crawlers. For operators that sign their requests, the toll verifies identity; see Verified crawler identity.

A different price for one company

Allow, charge and block decide whether an agent pays. To decide how much one company pays, open the Price row and fill in Prices for signed agents. The list holds every company that signs its requests, one row per signature: OpenAI's row covers GPTBot, ChatGPT-User and OAI-SearchBot because all three sign as chatgpt.com.

The price applies only to a request that carries a valid signature from that company. An unsigned request, or one that only writes the company's name into its user-agent, pays your normal price; otherwise any scraper could claim the discount by typing a name.

A signed agent's price replaces your section prices for that company, on every path. It never appears in your RSL document or your /.well-known/x402 manifest, since those are read by every agent and would offer that company's price to all of them.

Tolling a section or your whole catalogue with one price

You don't need a per-article catalogue to start charging. When you connect a site and it asks what kind of site it is, pick A whole site: set one price, toll everything or name the path sections that count (essays, posts), and connect a payout wallet. Every article under those sections tolls automatically and every settlement lands in that one wallet. You can move to per-article credits, authors and splits later without undoing anything.

Seeing which companies read you

Your Audit page groups every crawler under the company that runs it, so GPTBot and ChatGPT-User both count toward OpenAI. Pick a company's card to narrow the whole page to its requests, and pick it again to clear the filter.

Each company also has a page of its own. It shows what each of your sites charges that company's crawlers, what it read and what those reads were worth at your prices, every user agent it sent, and whether its reads came from the addresses the company publishes. Open any request there to see what happened step by step. For a paid read that includes the payment and a link anyone can use to check it.

Charging one company on every site

The switch on a company's page, and on its card once you pick it, moves that company's AI crawlers onto your charge list on every site you can edit. For OpenAI that is GPTBot and ChatGPT-User. Its search crawler is never touched, because charging it is a separate choice about appearing in that company's search. Turning the switch off lets those crawlers read free. A crawler you have blocked stays blocked.

Before anything changes, the confirmation lists each site and what will happen there. If a site's terms refuse AI reads, or give them away, the switch cannot change what that site does, and the list says so. The switch reads on once there is nothing left for it to charge, so a site whose terms decide never holds it off. If the company has not paid you for a read yet, the confirmation says so: until it pays, charging it means its crawlers get your price instead of the page. You have ten seconds to undo it afterwards, and every change is recorded in your activity log with the rules before and after.