resiliparse Web Scraping: Fast HTML-to-Text Extraction

TL;DR:

• resiliparse turns HTML into clean text fast, using a C++ extraction core built for processing web archives at billion-page scale.
• main_content=True drops the boilerplate. It returns the article text without the navigation, login, and footer chrome that a naive text dump keeps.
• What resiliparse does not do is fetch. It has no HTTP client and runs no JavaScript; hand it markup and it extracts.
• The gap shows up in one script. A plain GET on a JavaScript-rendered demo page yields 23 characters of text; the same URL fetched through the Scrapeless Universal Scraping API with js_render yields the real content.
• The extraction is real in this guide. A live run reduced 10,968 characters of rendered HTML to 1,469 characters of clean main-content text.
• Free to start on the fetch side. Create your Scrapeless API key at app.scrapeless.com.

What resiliparse is, and what it is not

resiliparse is the parsing and extraction half of the ChatNoir Resiliparse toolkit, built by the Web Archive research group to process the Common Crawl and similar corpora. Its HTML-to-text extraction runs in a C++ core, which is why it stays fast when the input is millions of documents rather than one. The function you reach for most is extract_plain_text, and its main_content=True mode is the useful part: it strips the navigation, sidebars, and footers that make a raw text dump noisy, leaving the content a reader actually came for.

It is an extraction library, not a scraper. resiliparse has no HTTP client, holds no session, and runs no JavaScript. Give it a string or bytes and it returns text; ask it to fetch a URL and there is no method for that. So every "resiliparse web scraping" setup is two layers: something that returns faithful HTML, and resiliparse that extracts from it. This guide uses the Scrapeless Universal Scraping API for the first layer, because a plain HTTP client returns the pre-render markup on any page that builds its content with JavaScript — and empty markup extracts to empty text. For structured fields rather than readable text, the Python web scraping tutorial covers the selector-based route.

Install

resiliparse and requests are the whole toolchain. The version this guide was written against is resiliparse 1.0.9:

pip install "resiliparse==1.0.9" requests

Keep your key in the environment, never in source:

export SCRAPELESS_API_KEY="sk_your_scrapeless_key"

Get HTML worth extracting

Extraction quality is bounded by fetch fidelity, so start there. On a JavaScript-rendered page, the markup a plain HTTP client receives holds none of the content — it arrives only after scripts build the DOM, a lifecycle defined by the HTML scripting specification. One script shows what resiliparse gets from each:

# fidelity.py — what resiliparse extracts: plain GET vs server-side rendering
import os

import requests
from resiliparse.extract.html2text import extract_plain_text

URL = "https://quotes.toscrape.com/js/"
MARKER = "The world as we have created it"

plain = requests.get(URL, timeout=60).text
plain_text = extract_plain_text(plain, main_content=True)
print("plain text chars:", len(plain_text), "| has quote:", MARKER in plain_text)

resp = requests.post(
    "https://api.scrapeless.com/api/v2/unlocker/request",
    headers={"Content-Type": "application/json", "x-api-token": os.environ["SCRAPELESS_API_KEY"]},
    json={"actor": "unlocker.webunlocker", "input": {"url": URL, "js_render": True}},
    timeout=120,
)
resp.raise_for_status()
rendered_text = extract_plain_text(resp.json().get("data", ""), main_content=True)
print("rendered text chars:", len(rendered_text), "| has quote:", MARKER in rendered_text)

The plain fetch extracts to almost nothing; the rendered one carries the real content:

plain text chars: 23 | has quote: False
rendered text chars: 1435 | has quote: True

Rendering, unlocking, and proxy routing all happen server-side in that one POST — the Universal Scraping API is the fetch layer, and the rendered bytes are what resiliparse should extract from.

Extract clean text

With real HTML in hand, resiliparse does the extraction. Run it once with defaults and once with main_content=True to see the boilerplate come off:

# extract.py — full text vs main-content extraction
import os

import requests
from resiliparse.extract.html2text import extract_plain_text

resp = requests.post(
    "https://api.scrapeless.com/api/v2/unlocker/request",
    headers={"Content-Type": "application/json", "x-api-token": os.environ["SCRAPELESS_API_KEY"]},
    json={"actor": "unlocker.webunlocker", "input": {"url": "https://quotes.toscrape.com/", "js_render": True}},
    timeout=120,
)
resp.raise_for_status()
html = resp.json().get("data", "")

full = extract_plain_text(html)
main = extract_plain_text(html, main_content=True)
print("rendered html chars:", len(html))
print("full text chars:", len(full))
print("main_content chars:", len(main))
print("has first quote:", "The world as we have created it" in main)

The run turns 10,968 characters of HTML into readable text, and main_content trims the chrome further:

rendered html chars: 10968
full text chars: 1674
main_content chars: 1469
has first quote: True

That is the whole pipeline: one POST to fetch and render, one call to extract. main_content=True is what makes the output feed a language model or a search index cleanly — the navigation and login prompts that survive a full-text dump are gone, and the quote text remains.

Get your API key on the free plan: app.scrapeless.com

Advanced patterns

• Pass bytes, not a decoded string, when the encoding is uncertain. resiliparse detects the encoding from the document; handing it the raw response bytes avoids a double-decode that mangles non-ASCII text.
• Use alt_texts and links flags to keep or drop detail. extract_plain_text accepts flags that preserve image alt text or link targets when you need them and omit them when you do not.
• Reach for resiliparse.parse.html when you need the tree. The toolkit also exposes an HTML tree and selector API, so you can extract structured fields from the same fast core when plain text is not enough.
• Keep the fetch and extract steps separate. Extracting is CPU-bound and fetching is IO-bound; pull the pages first, then run extraction over the batch, so neither waits on the other.

Troubleshooting

• Almost no text from a page you can see in a browser. The content is JavaScript-rendered and your fetch returned the pre-render HTML. The first script's 23-character result is exactly this case; fix it at the fetch layer with js_render.
• Navigation and footer text in the output. You called extract_plain_text without main_content=True. Turn it on to drop the boilerplate.
• Garbled accented characters. You handed resiliparse a mis-decoded string. Pass the response bytes and let it detect the encoding.
• Content you wanted got trimmed. main_content is aggressive by design. For pages with unusual structure, compare against the full-text output and fall back to it when the main-content pass drops too much.

Conclusion

resiliparse earns its place as the extraction layer that scales: a C++ core that turns HTML into clean text fast, with a main_content mode that removes the chrome. The layer that decides whether any of that is possible is the fetch — the first script's 23-versus-1,435-character split settles it — and one server-rendered POST closes the gap. Wire the two together and a rendered page becomes clean text, ready for an index or a model.

Create a free Scrapeless account to get an API key, and the developer docs cover the unlocker.webunlocker parameters. Check Scrapeless pricing when you plan a recurring job.

FAQ

Q: Can resiliparse scrape websites by itself?

No. resiliparse extracts text from HTML you already have; it has no HTTP client and runs no JavaScript. Pair it with a fetch layer — here the Scrapeless Universal Scraping API, which renders the page server-side — and resiliparse handles the text extraction from the returned bytes.

Q: How is resiliparse different from a boilerplate remover like trafilatura?

Both remove boilerplate, but resiliparse comes from the Web Archive research group and is built in C++ for corpus-scale throughput, and it ships as part of a wider toolkit that also reads WARC files and detects encoding and language. Reach for it when speed across many documents is the constraint.

Q: What does main_content=True actually do?

It runs a content-extraction pass that keeps the primary article text and drops navigation, sidebars, headers, and footers. The comparison in this guide shows the trimmed output against the full-text output so you can see what comes off.

Q: Does resiliparse handle JavaScript-rendered pages?

Not on its own — it never executes scripts. If the content loads after the initial HTML, a plain fetch extracts to almost nothing, as the 23-character result shows. Fetch the page through the Scrapeless API with js_render first, then extract.

Q: Is scraping with resiliparse legal?

The extraction library does not change the collection rules. Fetch public pages only, respect site terms and the robots directives standardized by the Robots Exclusion Protocol, keep volumes bounded, and handle any personal data under the laws that apply to you.