Skip to content
All posts

Humans read free, AI crawlers get an HTTP 402 and a price

robots.txt can only tell a crawler no. This lets a site tell it how much, and the money goes straight to the person who wrote the page.

Raveen Abeywickrama7 min read

robots.txt can tell a crawler no. It cannot tell it a price. So I built a gate that does. A person who opens an essay on Inner Axiom reads it free. GPTBot asking for the same essay gets an HTTP 402 and a price of a tenth of a cent, payable to the writer.

I built Inner Axiom, a site where other people write long essays on philosophy, myth and spiritual traditions. I did not write those essays. AI assistants read them and answer questions with them, and the people who wrote them get nothing.

Until now a writer had two choices: block the crawlers and disappear from the answers, or let them in and give the work away. Licensing deals exist, but only for publishers large enough to send a lawyer. naulon is a third choice. It puts a price on the page, and the agent that wants the page pays the person who wrote it.

The same essay gets two answers

I fetched one essay, The Journey of the Soul, from a terminal on 19 October 2026, once as a browser and once as GPTBot.

curl -s -o /dev/null -w '%{http_code}\n' -A 'Mozilla/5.0 (X11; Linux x86_64) Chrome/130' https://inneraxiom.com/articles/the-journey-of-the-soul
200

curl -s -o /dev/null -w '%{http_code}\n' -A 'GPTBot/1.2' https://inneraxiom.com/articles/the-journey-of-the-soul
402

The browser gets 200 and the essay. GPTBot gets 402 Payment Required, a status code HTTP has reserved since 1997 and almost nothing has used. You can run both commands yourself.

Drop -o /dev/null and the 402 explains itself. Two headers say why the request was charged and what it costs:

x-naulon-verdict: agent (publisher charge policy matched "gptbot")
crawler-price: USD 0.0011

The body says the same in words an agent can act on:

{
  "error": "payment_required",
  "message": "This request was classified as an automated agent, and this resource is tolled. Humans read free. To read it, pay USD 0.0011 over x402 in USDC: ..."
}

One paid read takes two requests

x402 is an open standard for paying over HTTP. The signed terms sit in a third header, payment-required, so an agent can pay without trusting anything else in the response. USDC is a dollar stablecoin, so the price is a fixed number of dollars.

An agent asks for the page and gets a 402 with a price of $0.0011. It asks again with a signed payment and gets the page with a licence. $0.001 goes to the author, and the $0.0001 fee is a separate transfer.AgentSite + gateAuthor1. GET the page2. 402: $0.00113. GET + payment4. 200 + licence5. $0.001 to the author$0.0001 fee: a separate transfer
One paid read, start to finish. A browser skips straight to step 4.

The agent reads the terms, signs a payment and asks again. The second request gets the essay and a licence: a signed record that this read was paid for. Anyone can check a licence at naulon.app/verify against our published keys, without asking us.

Of the $0.0011, $0.001 (a tenth of a cent) goes to the writer. The quote asks for a separate $0.0001 (a hundredth of a cent) as the naulon fee.

The money goes to the writer and never passes through us

An essay charges agents only once one of its writers has added a wallet. Until then it stays free to everyone, agents included. Nobody at Inner Axiom signed a licensing deal on the writers' behalf. Each of them chose, essay by essay.

The writer's share and the naulon fee are two separate transfers from the agent's own balance. The writer's share lands in the writer's own Gateway balance at Circle, which only their wallet can withdraw. Neither transfer passes through an account we control. Holding other people's money is a regulated business in most countries, and the simplest way to stay out of it is to never hold it.

A tenth of a cent is only worth sending if no read pays its own network fee. Settled one at a time on a blockchain, a payment has to be worth about a cent before the fee stops eating it. Here the agent signs the payment off-chain, Circle's Gateway credits the writer at once, and Circle later writes many payments to its Arc chain in a single transaction and pays that fee itself.

The gate charges agents that say who they are

The gate decides per request, and it is built to fail toward the reader. Charging a person by mistake would break the one promise the toll makes, while letting a crawler through costs a tenth of a cent.

An agent that signs its requests with Web Bot Auth, a standard where a crawler proves its identity with a key its operator publishes, is charged whatever user agent it sends. A request that claims to be a search crawler from an address outside that operator's published ranges loses the free read it was claiming. A user agent naming a known crawler, python-requests or plain curl is charged.

A crawler that sends a browser user agent and signs nothing is read as a person, and it reads free. The Chrome request at the top of this post came from curl. The gate does not try to win that arms race. It charges the agents that identify themselves, because those are the ones that will pay. A person, meanwhile, never sees a banner or a wall, and no publisher setting can change that.

GPTBot does not pay yet, and the price is still worth posting

OpenAI's crawler does not speak x402, so for GPTBot today this 402 is a refusal with a price on it. What can pay is any agent built with an x402 client: it reads the header, signs the payment, asks again and gets the essay.

The price has to be on the door before anyone pays it. A crawler operator deciding whether to pay needs a standard way to learn what a page costs and who to pay. Until a site says it, the only answer a crawler ever gets is yes or no.

Inner Axiom is one site, so this is a working toll, not a measurement across the web. I do not know yet how many agents will pay rather than leave.

Pay per crawl prices a domain, naulon prices a read

Cloudflare's pay per crawl sets one flat price for a whole domain, and Cloudflare is the merchant of record: it charges the crawler and pays the publisher later. Its newer Monetization Gateway pays sellers directly in stablecoins, but it covers sites behind Cloudflare and is a waitlist today. TollBit runs a licensing marketplace that publishers join. naulon prices each read, pays each writer of the page directly, works on any origin, and hands the buyer a licence anyone can verify.

You can run the gate on your own site

The gate is MIT licensed, at github.com/naulonapp/naulon. The classifier described above is in packages/enforce/src/agentDetect.ts, so you can read exactly who it charges and who it lets through. naulon.app is the hosted version for sites that do not want to run it.

If you run a site where people write, your readers keep reading free and the agents that can pay will pay your writers directly. Start with getting tolled.