Cloudflare has launched a bold new idea to help website owners earn money from AI bots.
The company announced a new tool called Pay per Crawl — a marketplace where sites can charge AI companies every time they scrape content. The marketplace is launching in private beta and aims to give publishers more control and a possible revenue stream in the AI age.
What Is Pay per Crawl?
For years, AI bots have been scraping the web — often without permission and without paying.
Pay per Crawl changes that. It lets website owners decide how they want to handle AI crawlers:
- Charge a small fee (micropayment) each time a bot crawls their site
- Allow AI bots to scrape content for free
- Block crawlers completely
Site owners can even set different rules for different bots.
Why Cloudflare Built This
Cloudflare provides cloud services to around 20% of the internet. Over the past year, the company rolled out several features to help publishers fight back against AI crawlers:
- A one-click AI bot blocker
- A dashboard to track which bots are visiting
- A focus on AI transparency and control
Now, with Pay per Crawl, they’re taking it a step further — by creating a way for content creators to get paid.
How It Works
To join the marketplace, both the publisher and the AI company need Cloudflare accounts. Then:
- Publishers set a price for each “crawl”
- AI companies choose whether to pay, request access, or skip the site
- Cloudflare handles the payment and sends earnings to the publisher
There are no cryptocurrencies involved — yet. But Cloudflare CEO Matthew Prince says they’re exploring the idea of creating their own stablecoin for future payments.
Default AI Blocking for New Sites
Starting now, every new website that signs up with Cloudflare will block AI bots by default.
If site owners want to allow certain bots, they’ll have to give permission. This change flips the default from open to closed — giving publishers more power.
Big Publishers Support the Move
Several top media brands have already backed Cloudflare’s approach, including:
- TIME
- Condé Nast
- The Atlantic
- The Associated Press
- Fortune
- ADWEEK
These publishers are also using Cloudflare’s AI bot blocker. Many have grown frustrated as their content is used to train AI models — without clear benefits in return.
Publishers Getting the Short End of the Stick
The way the web works is changing fast. For decades, publishers allowed Google to crawl their content because it sent traffic back through search. That traffic led to ad revenue.
Now, AI bots are scraping more and referring less.
Cloudflare shared some numbers from June:
- Google: 14 crawls per one referral
- OpenAI: 1,700 crawls per one referral
- Anthropic: 73,000 crawls per one referral
Those stats paint a clear picture — AI bots take a lot but give little back.
Looking Ahead: The Agentic Web
Cloudflare sees an “agentic” internet on the horizon — where AI agents browse the web for users, gathering info like a personal assistant.
Imagine:
You ask an AI to find you the best restaurant in Soho. That agent visits websites, checks reviews, and pulls together a list — all without you ever visiting those pages.
In this future, Cloudflare says Pay per Crawl could act like an automatic paywall, charging agents in real-time for access to quality content.
No Guarantees — But a Strong Start
There’s still a long way to go. Convincing AI companies to pay for something they currently get for free won’t be easy.
But Cloudflare is one of the few companies with the size, tech, and relationships to actually make this model work.
Key Takeaways
- Pay per Crawl is a new marketplace by Cloudflare for charging AI bots per content scrape.
- Site owners can block bots, allow them for free, or charge per visit.
- The tool is now in private beta and available to Cloudflare users.
- Several big publishers are already blocking AI bots using Cloudflare’s tools.
- Cloudflare plans to explore stablecoin-based payments in the future.
- AI bots are scraping thousands of times more than they send traffic back.
- This model gives small and large publishers a new way to earn in the AI era.