selectolax Web Scraping: Fast HTML Parsing in Python

TL;DR:

• selectolax parses HTML with CSS selectors and does it fast, because it wraps the C-based Lexbor engine instead of pure-Python parsing.
• What selectolax does not do is fetch. It has no HTTP client and renders no JavaScript; hand it bytes and it parses them, nothing more.
• The gap shows up in one script. A plain GET on a JavaScript-rendered demo page gives selectolax 0 quote nodes; the same URL fetched through the Scrapeless Universal Scraping API with js_render gives it all 10.
• The parse is real in this guide. A live run pulled all 10 quotes with authors and tag arrays intact from the rendered HTML, using css and css_first.
• The two layers are cleanly split. Scrapeless fetches and renders; selectolax turns the bytes into records. Neither reaches into the other's job.
• Free to start on the fetch side. Create your Scrapeless API key at app.scrapeless.com.

What selectolax is, and what it is not

selectolax is a Python HTML parser built for speed. It binds the Lexbor engine, a C library that implements the HTML standard, so parsing a document and running CSS selectors over it happens in compiled code rather than in the interpreter. For high-volume extraction — thousands of pages, tight loops — that difference is the reason people reach for it over heavier parsers.

It is a parser and only a parser. selectolax has no HTTP client, holds no session, and runs no JavaScript. Give it a string or bytes and it builds a tree you can query; ask it to go get a URL and there is no method for that, by design. So every "selectolax web scraping" setup is two layers: something that fetches faithful HTML, and selectolax that turns it into data. This guide uses the Scrapeless Universal Scraping API for the first layer, because a plain HTTP client returns the wrong bytes on any page that builds its content with JavaScript. For a broader tour of the Python side, the Python web scraping tutorial covers the ecosystem around this.

Install

selectolax and requests are the whole toolchain. The version this guide was written against is selectolax 0.4.11:

pip install "selectolax==0.4.11" requests

Keep your key in the environment, never in source:

export SCRAPELESS_API_KEY="sk_your_scrapeless_key"

Get HTML worth parsing

Parse quality is bounded by fetch fidelity, so start there. On a JavaScript-rendered page, the document a plain HTTP client receives is not the document a reader sees: the content arrives only after scripts build the DOM, a lifecycle defined by the HTML scripting specification. One script counts the difference through selectolax itself:

# fidelity.py — what selectolax sees: plain GET vs server-side rendering
import os

import requests
from selectolax.lexbor import LexborHTMLParser

URL = "https://quotes.toscrape.com/js/"

plain = requests.get(URL, timeout=60).text
print("plain GET chars:", len(plain), "| quote nodes:", len(LexborHTMLParser(plain).css("div.quote")))

resp = requests.post(
    "https://api.scrapeless.com/api/v2/unlocker/request",
    headers={"Content-Type": "application/json", "x-api-token": os.environ["SCRAPELESS_API_KEY"]},
    json={"actor": "unlocker.webunlocker", "input": {"url": URL, "js_render": True}},
    timeout=120,
)
resp.raise_for_status()
rendered = resp.json().get("data", "")
print("rendered chars:", len(rendered), "| quote nodes:", len(LexborHTMLParser(rendered).css("div.quote")))

The run prints 0 quote nodes for the plain fetch and 10 for the rendered one:

plain GET chars: 5806 | quote nodes: 0
rendered chars: 8940 | quote nodes: 10

Rendering, unlocking, and proxy routing all happen server-side in that one POST — the Universal Scraping API is the fetch layer, and the rendered bytes are what selectolax should parse.

Parse it with selectolax

With real HTML in hand, selectolax does the extraction. css returns every node matching a selector; css_first returns the first, so you can pull a field off each record. The selectors follow the W3C Selectors specification, the same syntax you already use in CSS:

# extract.py — fetch the rendered page, then extract records with selectolax
import os

import requests
from selectolax.lexbor import LexborHTMLParser

resp = requests.post(
    "https://api.scrapeless.com/api/v2/unlocker/request",
    headers={"Content-Type": "application/json", "x-api-token": os.environ["SCRAPELESS_API_KEY"]},
    json={"actor": "unlocker.webunlocker", "input": {"url": "https://quotes.toscrape.com/js/", "js_render": True}},
    timeout=120,
)
resp.raise_for_status()
tree = LexborHTMLParser(resp.json().get("data", ""))

records = []
for quote in tree.css("div.quote"):
    records.append({
        "text": quote.css_first("span.text").text(),
        "author": quote.css_first("small.author").text(),
        "tags": [tag.text() for tag in quote.css("a.tag")],
    })

print("records:", len(records))
print("first author:", records[0]["author"])
print("first tags:", records[0]["tags"])

The live run returned all 10 records, with the author and tag array intact on the first one:

records: 10
first author: Albert Einstein
first tags: ['change', 'deep-thoughts', 'thinking', 'world']

That is the whole scraper: one POST to fetch and render, one tree to query. text() returns a node's text content, and building a list of dicts per record is the pattern that scales to any repeated block on a page.

Get your API key on the free plan: app.scrapeless.com

Advanced patterns

• Use LexborHTMLParser for speed, HTMLParser for the Modest backend. selectolax ships both engines behind the same API; Lexbor is the newer, faster one and the right default for volume.
• Prefer css_first with a default. node.css_first("span.text", default=None) returns None instead of raising when a field is missing, which keeps a loop over uneven records from crashing on the one page that differs.
• Read attributes with .attributes. For links and images, node.attributes.get("href") pulls the attribute off the node — selectors find the element, .attributes reads its data.
• Fetch markdown when you do not need the tree. If you only want readable text, the Scrapeless API can return rendered content directly; reach for selectolax when you need selector-level control over structured fields.

Troubleshooting

• Zero nodes from a page you can see in a browser. The content is JavaScript-rendered and your fetch returned the pre-render HTML. Count a known selector the way the first script does; a near-empty tree is a fetch problem, fixed with js_render, not a selector problem.
• AttributeError: 'NoneType' object has no attribute 'text'. A css_first found nothing and you called .text() on None. Pass default=None and check before reading, or confirm the selector matches the rendered markup.
• Text has extra whitespace. text() returns the raw node text; call .strip() or pass deep=True/separator options when a node nests children whose text you want joined.
• Encoding looks wrong. Pass the response bytes rather than a mis-decoded string; selectolax reads the document's declared encoding when you hand it bytes.

Conclusion

selectolax earns its place as the parse layer that never touches the network: hand it faithful HTML and it returns records fast, with CSS selectors you already know. The layer that decides whether any of that is possible is the fetch — the first script's 0-versus-10 count settles it — and one server-rendered POST closes the gap. Wire the two together and the demo page's ten quotes arrive as clean dicts, tags and all.

Create a free Scrapeless account to get an API key, and the developer docs cover the unlocker.webunlocker parameters. Check Scrapeless pricing when you plan a recurring job.

FAQ

Q: Can selectolax scrape websites by itself?

No. selectolax parses HTML you already have; it has no HTTP client and renders no JavaScript. Pair it with a fetch layer — here the Scrapeless Universal Scraping API, which renders the page server-side — and selectolax handles the extraction from the returned bytes.

Q: Why use selectolax instead of BeautifulSoup?

Speed. selectolax wraps the C-based Lexbor engine, so parsing and selector queries run in compiled code, which matters at high page volume. The trade-off is a smaller feature surface; for selector-based extraction over many pages, that is usually the right trade.

Q: What is the difference between css and css_first?

css returns a list of every node matching the selector; css_first returns only the first match (or a default you supply). Use css to loop over repeated blocks like search results, and css_first to pull a single field off each block.

Q: Does selectolax handle JavaScript-rendered pages?

Not on its own — it never executes scripts. If the content loads after the initial HTML, a plain fetch hands selectolax an empty tree. Fetch the page through the Scrapeless API with js_render first, then parse the rendered HTML, as this guide does.

Q: Is scraping with selectolax legal?

The parser does not change the collection rules. Fetch public pages only, respect site terms and the robots directives standardized by the Robots Exclusion Protocol, keep volumes bounded, and handle any personal data under the laws that apply to you.