Crawler route

The crawler route tolls AI crawlers at the CDN you already use. One generated Cloudflare Worker goes into your Cloudflare account, installed by naulon with a token you create or pasted by you. It sends requests from AI crawlers to naulon, which quotes and charges them, and passes everything else to your site as before. Your DNS, your origin, your HTTPS certificate and your human visitors do not change.

Use it when your site is behind Cloudflare and you would rather not install anything on the server. If you can add a package or a WordPress plugin to your site, the in-app setup works just as well; both toll the same crawlers.

Before you start

• Your domain is proven in naulon (a TXT record, a meta tag or a file; the site's Domains card walks you through it).
• Your site charges for at least one path. The Worker only forwards the paths you toll.
• Your origin is served over HTTPS.
• You can create a Worker and add routes in the Cloudflare account that holds your domain's zone.

Let naulon install it

1. In Cloudflare, open My Profile > API Tokens and create a custom token with three permissions: Account · Workers Scripts · Edit, Zone · Workers Routes · Edit and Zone · Zone · Read. Limit the zone resources to your domain's zone. If you set a start date, Cloudflare counts it in UTC, and the token does nothing before then.
2. On your site's Domains card, answer My site is behind Cloudflare, paste the token and press Install. naulon finds your zone, uploads a Worker named naulon-route- followed by your domain, adds its routes and sends a test crawler.
3. Delete the token in Cloudflare. naulon used it for that one request and did not keep it.

If your card shows Connect Cloudflare, you can approve access on Cloudflare's own page instead of creating a token. naulon asks for the same permissions, uses them once, and hands the access back when the install finishes.

naulon never replaces something it did not create. If another Worker already handles one of the route patterns, the install stops before changing anything and names the pattern, so you can decide what to keep.

Paste it yourself

1. On your site's Domains card, answer My site is behind Cloudflare, then press Create the route. naulon generates the Worker file and the route patterns for your domain.
2. In Cloudflare, open Workers & Pages, create a Worker, replace its code with the file and deploy it.
3. In the Worker's settings, add each route pattern shown on the card on your domain's zone. For a site that tolls /essays/, that is example.com/essays/* and example.com/license.xml.
4. Back on the card, press Send a test crawler. You should see "The route works".

The card then waits for the first real AI crawler and shows when the last one came through.

What the Worker does

• It acts only on your own hostname and only on the paths you toll.
• It forwards a request to naulon when the user agent is a known AI crawler, or when the request already carries a payment or a licence. Browsers are never forwarded.
• It skips requests carrying your origin secret in x-naulon-origin-auth. That is naulon fetching your page after a crawler paid, and forwarding it back would loop.
• If naulon does not answer within the timeout, or answers with a server error, the Worker serves your site directly. A problem on our side never takes your site down; crawlers read free until it clears.
• It serves your licence document at /license.xml from naulon, and falls back to your own copy if naulon cannot answer.

The test

Send a test crawler makes three requests to your site and reports one verdict:

Request	Expected
A test crawler (user agent naulon-route-check)	answered by naulon through the route
The same request with your origin secret	answered by your site directly
A browser	answered by your site directly

The test crawler is charged like any unrecognised agent, but it is never counted as a crawler arriving. If Cloudflare challenges the test, the card says so instead of blaming your rule.

Secrets

The file carries two secrets: the route secret that lets the Worker into naulon, and your origin secret. Treat the file like a password and keep it out of public repositories.

To replace the route secret, press Generate a new secret. You get a new file; the old one keeps working until you deploy the new one and press I've deployed the new file. If naulon installs the new file on Cloudflare for you, the old secret is retired as soon as the install finishes. Showing the file again and generating a secret are both recorded in your audit log.

If you rotate your origin secret on the Protect your origin panel, or change your crawler policy or the paths you charge for, install the Worker again (or paste the new file). naulon does not keep access to your Cloudflare account, so it cannot update the Worker on its own. The test reports a stale file as a loop.

Limits

• Cloudflare only for now.
• Workers count against your Cloudflare plan's request allowance. Only forwarded requests (crawlers on tolled paths) run the Worker's forwarding path, but every request on a matched route invokes the Worker. If the daily allowance runs out, Cloudflare stops running the Worker and the test reports it.

Removing it

Press Remove from Cloudflare on the card and give access the same way you installed. naulon deletes its Worker and routes, then removes the route. Crawlers read free from that moment.

Or press Remove the route, then delete the Worker in Cloudflare yourself. Until you do, it keeps running but only serves your site.