Charging an AI bot to crawl your website is a visibility decision, not a revenue decision. Every search engine that you put behind a paywall chooses which agents read, mention, and recommend you, and most website owners are already doing that without realizing that they are the ones to do it. The mechanism is HTTP 402, a status code that was written on the web in 1997 and remained inactive for nearly three decades. In July 2025, Cloudflare removed it to charge AI crawlers. On June 15, 2026, AWS added the same capability to its firewall. Two of the biggest web infrastructure companies are now selling the same thing: a mechanical toll booth. The question they give you is who can turn.
Toll Booth Everyone Reads Like a Line of Revenue
The launch post is independent of each crawl like a new currency, but it is about permission. For years, AI companies have been crawling the open web to train models and basic responses, and website owners have had two blunt choices: leave the door open, or block searches and trust the rules in place. Pay-per-crawl is the first method that allows a website to say something more precise than yes or no. It allows you to set goals. That’s a real revolution where permission and control, not pennies, are the real prize.
The reason any of this exists in the first place is a broken transaction. For 30 years, the deal has been simple: Let the search engine in, index you, send people back. The AI detectives kept the first half and dropped the second. Cloudflare’s breakdown of why those bots crawl puts training at about 80% of an AI bot’s job, with search intent downloads that can return cating, a small slice of what’s left. Another is a release that takes the content and doesn’t give it back to anyone. So, this toll booth is a web that tries to renegotiate the auction that the AI crawlers have stopped respecting.
Here is where the celebration comes from the announcement post and my reading of it separately. The moment you can set goals, goals become a tradeoff for visibility, and that’s where posts are made.
Who Charges AI Crawlers, and How They Work
Cloudflare launched the first major version on July 1, 2025. The publisher sets a low, per-request price for its domain. When an AI crawler requests a page, it provides a payment intent in its request headers and receives content with a 200 response, or receives a 402 Payment Required response that carries a value. Cloudflare describes itself as a Seller of Record and runs a settlement, which is important: because Cloudflare sits between searchers and publishers, it can act as a clearing house for both parties. At the time of launch, the feature was in private beta, and Cloudflare included it as an initial test, not a finished market.
AWS added a similar view to its Web Application Firewall on June 15, 2026. When an AI browser requests a protected article, data feed, or licensed archive, AWS WAF can return an HTTP 402, and the price and payment details go through the x402 protocol as a machine-readable manifest. Payment is in stablecoin via Coinbase, controls are in WAF Bot Control, and the price can be set per content method and bot type without touching the website code. Two of the largest web infrastructure providers now sell tollbooth machines within a year of each other.
They are not alone, although every field is shaped differently. TollBit uses a bot-and-agent paywall and marketplace, charging for AI access on a usage-based basis rather than a low price per request. Akamai brings that monetization approach to its edge by integrating TollBit with Skyfire, announced in September 2025, so its customers can enforce payment at the network layer without building it. Underneath the new tolls sits x402, Coinbase’s open standard for the same HTTP 402 status code. It resides in a stablecoin and does not require an account from a paying agent, which will extend the payment from known, identified bots to unknown ones. To get a full picture of which searchers are reaching you, the user agent’s AI status is a reference, because you can’t price a bot that you can’t say.
In all of them, the toll lives on the edge, on the CDN and the firewall, the same layer that already decides who gets in. Payment and approval are integrated into a single control point.
The Open-Web Bargain Is Unraveling
The open web innovation agreement is broken down and sold line by line. The free crawl-traffic deal that powers search, and the entire practice built on it, assumes that crawling is the cost a website pays to get distribution. Pay-per-crawl is the first crawl invoice that doesn’t return favors in a reliable way.
That unraveling has a certain shape. Access control and billing used to be separate concerns, handled by separate systems. Now they sit on the same edge, on the same WAF rule, on the same Cloudflare dashboard. The firewall that determines whether a request is allowed becomes the meter that determines how much the request costs. When permission and payment are folded into a single layer, the decision to accept a visitor ceases to be a default and becomes a deliberate choice. This is the question of the agent as guest, the one subject to the case of Amazon v. Confused about whether the agent is an authorized visitor, which has been removed from the official termination into the setting you are preparing.
And the crawling grace that came back with it did not end; it’s gone. AI places people back: Adobe’s 2026 data shows AI-directed traffic to US retailers up 393% year-over-year. That number is why the payment is the first impression decision and the payment in one second. The search engine you can charge and the answer that directs the customer to you are usually the same.
What You Should Charge Depends on Who You Are
A publisher with a deep, licensed archive, news store, reference database, or proprietary data set, has two things that make charging reasonable: The content is really worth paying for, and searches it can’t deny because your distribution doesn’t depend on it. For that owner, the toll is powerful, and using it makes sense.
A content or commercial website chasing AI visibility sits on the other side. To that owner, the search is a distribution. Make a wall, and you may collect a few cents, or you may remove it from the place where your customers ask their questions. The toll only works if two things are true at the same time: Your background is worth paying for, and the bot you’re blocking is one you don’t need to be seen by. The website’s strategy is that the response the agent gives describes the search engine that it cannot charge for. This is why the website has become a resource instead of a megaphone: Value is learned, not hedged.
So the move is to look at the door first. Find out which AI bots are actually reaching you and what they are taking, separate the ones that give answers and send referrals from the ones that don’t give anything back, and decide only where, and where, the toll belongs. That’s the test that should be done before anyone charges a cent: measure one bot in one direction, see what it does to both your volume of clarity and your presence in responses, and let the result, not the hype of the launch, inform your policy. You should not invest a lot of money in traffic that you have never looked at.
The Unsettled Question Is Whether Anonymous Bots Will Pay
No public research has yet shown how much share per citation a website loses by charging for a particular bot. The cost perception is palpable: A browser that reads both of you and feeds AI feedback can remove you from that feedback if you refuse. How big the effect is in practice is unknown, and that’s why this is always a decision you make on your own data, not a rule of thumb.
The signal I will watch is if the anonymous payment-bot becomes real. Today, effective models rely on targeted searchers presenting a paying intent. The wish of x402 is payment without prior registration, paid in stablecoin, which can extend the amount charged to an unknown amount of automatic traffic. If that works at scale, the priced web reaches far beyond a handful of named bots. If it’s always business pipelines, pay per share is always a tool for big publishers and everyone’s curiosity. Which of these realizations determines whether a toll booth is a niche or a new road condition.
Either way, the booth is built, and it’s ours to operate. The bill you can see is the one paid by the browser. The bill you won’t see is the answer you’re disappearing from. Decide which one to watch.
Additional resources:
This post was originally published on No Hacks.
Featured image: Roman Samborskii/Shutterstock
