The Web Is Getting an Access Layer: llms.txt, Signed Agents, and Pay-Per-Crawl

The web has spent thirty years running on a single honour-system file. robots.txt asks politely, identifies nobody, and enforces nothing — and for most of that time it was enough, because the thing reading your pages was a search engine that sent traffic back.

That exchange has broken down, and four separate efforts are now trying to replace it. They are not competing versions of the same idea. They answer three different questions — what to read, who is asking, and what it costs — and only one of them is a finished standard.

The Layer That Is Actually Missing Is Identity

Start with what robots.txt can and cannot do. The Robots Exclusion Protocol standard gives publishers a way to express preferences per user-agent string. The weakness is in that last phrase: a user-agent string is a claim, not a credential. Anyone can send any string, and IP allowlists age badly as infrastructure moves.

So a publisher who wants to treat two crawlers differently has no reliable way to tell them apart. Every proposal below is downstream of that gap. The practical mechanics of the file itself — syntax, directives, and how collectors should read it — are covered in the robots.txt guide for web scraping.

llms.txt Answers "What Should I Read?"

The llms.txt proposal puts a Markdown file at /llms.txt containing a curated summary and links to clean text versions of a site's important pages. The llms.txt proposal was published by Jeremy Howard in September 2024, and its stated problem is context windows: models cannot ingest an entire site, and converting HTML to usable text is lossy.

It is worth being precise about its status, because it is often described as a standard. The site itself calls it "a proposal to standardise." There is no RFC, no working group, and no requirement that anyone honour it.

What it is genuinely good at is curation. A publisher who writes one says here is the good version of my content, which helps a well-behaved reader and costs a badly-behaved one nothing. It expresses preference, not permission — the same limitation robots.txt has.

Web Bot Auth Answers "Who Is Asking?"

This is the piece that actually addresses the identity gap. The approach: every request from an automated client carries a cryptographic signature made with a private key belonging to its operator, so the origin server can verify who is calling instead of trusting a header.

The current specification is the Web Bot Auth HTTP message signatures draft, an active individual Internet-Draft built on HTTP Message Signatures. Its own framing is that IP allowlisting and User-Agent strings are not adequate identification.

Two caveats matter. It is an individual submission rather than the output of a working group, and an earlier architecture draft from the same authors has already expired and been replaced — normal for standards work, but a sign this is early. And signing proves who you are, not that you are allowed. It gives publishers something to make decisions about; it does not make the decision.

Pay-Per-Crawl and RSL Answer "What Does It Cost?"

Two efforts address compensation, from opposite directions.

Cloudflare's pay-per-crawl uses HTTP 402, the status code reserved for payment in the HTTP semantics specification and left unused for decades. Publishers set a per-request price and mark each crawler allow, charge, or block; crawlers signal intent through price headers and are identified by Web Bot Auth signatures. It was announced in July 2025 and remains in private beta — a live experiment, not infrastructure you can build against.

RSL takes the licensing route. The Really Simple Licensing standard defines machine-readable licensing terms — attribution, pay per crawl, pay per inference — as XML that can be referenced from robots.txt, HTML, HTTP headers, RSS feeds, or media files. RSL 1.0 shipped in 2025 with backing from Akamai, Cloudflare, Creative Commons, Fastly, Reddit, O'Reilly Media, Vox Media, Yahoo, and Ziff Davis.

Of the four, RSL is the furthest along as a published specification with named industry adoption. That does not make it settled — a licensing grammar still needs someone willing to enforce it, and enforcement returns you to identity.

The Stack They Imply

Read together, these are layers rather than alternatives:

• Preference — robots.txt and llms.txt say what a publisher would like.
• Identity — Web Bot Auth says who is asking, cryptographically.
• Terms — RSL says what the content may be used for and at what price.
• Settlement — pay-per-crawl-style 402 flows say how money actually moves.

The order matters. Terms without identity are unenforceable, and settlement without terms is a toll booth with no posted rate. Identity is the load-bearing layer, and it is the least finished.

Where This Is Likely to Fall Short

The obvious objection to all of it: nothing here binds anyone who chooses not to participate. A client that ignores llms.txt, sends no signature, and never reads an RSL file is in exactly the position clients have always been in. These mechanisms work on operators who want to be identifiable — which is most large, accountable ones, and none of the rest.

The second problem is consolidation. Signature-based identity favours operators large enough to run key infrastructure and get their keys recognised. A researcher, a startup, or a public-interest archive may find the new access layer harder to enter than the honour system it replaces. That is a real cost, and it is worth naming rather than treating as a rounding error on a cleaner web.

Third, none of this addresses the asymmetry that caused the breakdown. Search-era crawling traded access for traffic. A model that reads your page and answers the question itself returns nothing, and a payment rail does not restore the relationship — it prices its absence.

What to Actually Do

If you publish: write an llms.txt — it is cheap and it expresses curation you control. Watch RSL, because a machine-readable licence is the artifact a future dispute will turn on. Treat pay-per-crawl as an experiment to follow rather than a plan.

If you collect data: the durable move is to become the kind of client these systems are designed to accommodate. Identify yourself honestly, stay on public data, respect the directives that exist today, keep volume proportionate, and keep a record of what you collected and from where. Every proposal above rewards operators who can answer "who are you and what did you take" — and that is worth doing now, while the answer is still voluntary.

The honour system is ending. What replaces it is unfinished, and the useful posture is neither to wait for it nor to ignore it, but to behave now the way the finished version will require.

Start on the Scrapeless free plan to see how the Scrapeless AI Agent handles public web data, and review the Scrapeless pricing when you plan a collection programme.

FAQ

Q: Is llms.txt an official standard?

No. It is explicitly presented as a proposal to standardise, published in September 2024. There is no RFC and no working group behind it, and no requirement that any client honour it. Sites adopt it because curation helps well-behaved readers, not because anything compels it.

Q: Does llms.txt replace robots.txt?

No — they answer different questions. robots.txt, standardised in RFC 9309, expresses which paths a client should not fetch. llms.txt points at the content a publisher considers most useful and offers cleaner text versions of it. One restricts, the other curates, and neither authenticates the client.

Q: What problem does Web Bot Auth actually solve?

It makes a crawler's identity verifiable. Today a client identifies itself with a User-Agent string that anyone can copy, so publishers cannot reliably distinguish between crawlers. Web Bot Auth has each request signed with the operator's private key using HTTP Message Signatures, so the origin can verify who is calling. It is an active individual Internet-Draft, not a finished standard.

Q: Can publishers charge AI crawlers today?

Only experimentally. Cloudflare's pay-per-crawl uses HTTP 402 with per-request pricing and was announced in July 2025, but it remains in private beta. RSL, published in 2025 with backing from a group of infrastructure companies and publishers, defines machine-readable licensing terms including pay-per-crawl and pay-per-inference — but a licence still depends on someone identifying and enforcing against the client.

Q: What should a data collection team do while this settles?

Behave as though identity and terms are already enforced. Collect public data only, respect current directives, keep request volume proportionate to what a site can serve, and record what was collected and from where. Every proposal on the table rewards operators who can answer that question, so the work is not wasted regardless of which one wins.