parsel Web Scraping: CSS and XPath Selectors in Python

TL;DR:

• parsel selects data from HTML with both CSS and XPath, over the same document — it is the selector engine Scrapy uses, available as a standalone library.
• What parsel does not do is fetch. It has no HTTP client and runs no JavaScript; you hand it markup and it builds a queryable tree.
• The gap shows up in one script. A plain GET on a JavaScript-rendered demo page gives parsel 0 quote nodes; the same URL fetched through the Scrapeless Universal Scraping API with js_render gives it all 10.
• CSS and XPath, side by side. A live run pulled the same 10 authors two ways — sel.css("small.author::text") and sel.xpath("//small[@class='author']/text()") — from the rendered HTML.
• The two layers stay separate. Scrapeless fetches and renders; parsel selects. Neither reaches into the other's job.
• Free to start on the fetch side. Create your Scrapeless API key at app.scrapeless.com.

What parsel is, and what it is not

parsel is a Python library for extracting data from HTML and XML using CSS selectors and XPath expressions. It is the selector layer that Scrapy is built on, packaged so you can use it anywhere, and it wraps lxml underneath, so both selector languages run against the same fast parse tree. The reason to reach for it over a CSS-only parser is XPath: when a field is defined by its position, its text, or its relationship to a sibling rather than a class, XPath expresses it and CSS cannot.

It is a selector library and nothing more. parsel has no HTTP client, holds no session, and runs no JavaScript. Give it a string and it builds a Selector you can query; ask it to fetch a URL and there is no method for that. So every "parsel web scraping" setup is two layers: something that returns faithful HTML, and parsel that selects from it. This guide uses the Scrapeless Universal Scraping API for the first layer, because a plain HTTP client returns the pre-render markup on any page that builds its content with JavaScript. The broader Python web scraping tutorial covers the surrounding ecosystem.

Install

parsel and requests are the whole toolchain. The version this guide was written against is parsel 1.11.0:

pip install "parsel==1.11.0" requests

Keep your key in the environment, never in source:

export SCRAPELESS_API_KEY="sk_your_scrapeless_key"

Get HTML worth parsing

Selection quality is bounded by fetch fidelity, so start there. On a JavaScript-rendered page, the markup a plain HTTP client receives is not the markup a reader sees: the content arrives only after scripts build the DOM, a lifecycle defined by the HTML scripting specification. One script counts the difference through parsel itself:

# fidelity.py — what parsel sees: plain GET vs server-side rendering
import os

import requests
from parsel import Selector

URL = "https://quotes.toscrape.com/js/"

plain = requests.get(URL, timeout=60).text
print("plain GET chars:", len(plain), "| quote nodes:", len(Selector(text=plain).css("div.quote")))

resp = requests.post(
    "https://api.scrapeless.com/api/v2/unlocker/request",
    headers={"Content-Type": "application/json", "x-api-token": os.environ["SCRAPELESS_API_KEY"]},
    json={"actor": "unlocker.webunlocker", "input": {"url": URL, "js_render": True}},
    timeout=120,
)
resp.raise_for_status()
rendered = resp.json().get("data", "")
print("rendered chars:", len(rendered), "| quote nodes:", len(Selector(text=rendered).css("div.quote")))

The run prints 0 quote nodes for the plain fetch and 10 for the rendered one:

plain GET chars: 5806 | quote nodes: 0
rendered chars: 8940 | quote nodes: 10

Rendering, unlocking, and proxy routing all happen server-side in that one POST — the Universal Scraping API is the fetch layer, and the rendered markup is what parsel should select from.

Extract with CSS and XPath

With real HTML in hand, parsel does the extraction. css and xpath both return a SelectorList; get returns the first match and getall returns every match. The ::text pseudo-element and the text() node test both pull text, so you can read the same field either way — CSS follows the W3C Selectors specification and XPath follows the W3C XPath specification:

# extract.py — fetch the rendered page, then select with CSS and XPath
import os

import requests
from parsel import Selector

resp = requests.post(
    "https://api.scrapeless.com/api/v2/unlocker/request",
    headers={"Content-Type": "application/json", "x-api-token": os.environ["SCRAPELESS_API_KEY"]},
    json={"actor": "unlocker.webunlocker", "input": {"url": "https://quotes.toscrape.com/js/", "js_render": True}},
    timeout=120,
)
resp.raise_for_status()
sel = Selector(text=resp.json().get("data", ""))

css_nodes = sel.css("div.quote")
xpath_authors = sel.xpath("//div[@class='quote']//small[@class='author']/text()").getall()
print("css quote nodes:", len(css_nodes))
print("xpath authors:", len(xpath_authors))

records = []
for quote in css_nodes:
    records.append({
        "text": quote.css("span.text::text").get(),
        "author": quote.xpath(".//small[@class='author']/text()").get(),
        "tags": quote.css("a.tag::text").getall(),
    })
print("first author:", records[0]["author"])
print("first tags:", records[0]["tags"])

The live run selected all 10 records, with CSS and XPath returning the same authors:

css quote nodes: 10
xpath authors: 10
first author: Albert Einstein
first tags: ['change', 'deep-thoughts', 'thinking', 'world']

That is the whole scraper: one POST to fetch and render, one Selector to query. Mixing CSS for the easy fields and XPath for the positional ones — quote.xpath("...") runs relative to each node — is the pattern that keeps a scraper readable as pages get complicated.

Get your API key on the free plan: app.scrapeless.com

Advanced patterns

• Reach for XPath when CSS runs out. Selecting by text (//a[contains(text(), "Next")]), by position ((//tr)[2]), or by an axis (following-sibling::td) is XPath's territory; CSS has no equivalent.
• Chain selectors relative to a node. quote.css(...) and quote.xpath(".//...") both scope to that node — the leading . in the XPath keeps it relative, which is the difference between "authors inside this quote" and "all authors on the page".
• Use get(default="") to avoid None. sel.css("span.missing::text").get(default="") returns an empty string instead of None, which keeps a loop over uneven records from breaking on the odd page.
• Pull attributes with ::attr() or @. sel.css("a::attr(href)") and sel.xpath("//a/@href") both read an attribute; use whichever language the rest of the selector is already in.

Troubleshooting

• Zero nodes from a page you can see in a browser. The content is JavaScript-rendered and your fetch returned the pre-render HTML. Count a known selector the way the first script does; a near-empty tree is a fetch problem, fixed with js_render, not a selector problem.
• get() returns None. The selector matched nothing. Check the rendered markup, confirm the class or path, and pass a default so downstream code does not crash on the miss.
• XPath returns elements when you wanted text. Add /text() to the path or .get() on a ::text CSS selector; a bare element path returns the node, not its string.
• Relative XPath grabs the whole page. A path starting with // is absolute even when called on a node. Prefix it with . — .//small — to scope it to the current element.

Conclusion

parsel earns its place as the selection layer that speaks both dialects: CSS for the common cases, XPath for the ones CSS cannot reach, over one lxml-backed tree. The layer that decides whether any of that is possible is the fetch — the first script's 0-versus-10 count settles it — and one server-rendered POST closes the gap. Wire the two together and the demo page's ten quotes arrive as clean records, selected whichever way reads best.

Create a free Scrapeless account to get an API key, and the developer docs cover the unlocker.webunlocker parameters. Check Scrapeless pricing when you plan a recurring job.

FAQ

Q: Can parsel scrape websites by itself?

No. parsel selects data from HTML you already have; it has no HTTP client and runs no JavaScript. Pair it with a fetch layer — here the Scrapeless Universal Scraping API, which renders the page server-side — and parsel handles the extraction from the returned markup.

Q: What is the difference between parsel and Scrapy selectors?

None, in practice — parsel is the selector engine Scrapy uses, released as a standalone library. If you have used response.css or response.xpath in a Scrapy spider, that is parsel. Using it directly lets you keep the selector API without adopting the whole framework.

Q: Should I use CSS or XPath with parsel?

Both, as needed. CSS is shorter for class- and tag-based selection; XPath handles selection by text, position, or axis, which CSS cannot express. parsel runs both against the same tree, so mix them per field.

Q: Does parsel handle JavaScript-rendered pages?

Not on its own — it never executes scripts. If the content loads after the initial HTML, a plain fetch hands parsel an empty tree. Fetch the page through the Scrapeless API with js_render first, then select from the rendered HTML, as this guide does.

Q: Is scraping with parsel legal?

The selector library does not change the collection rules. Fetch public pages only, respect site terms and the robots directives standardized by the Robots Exclusion Protocol, keep volumes bounded, and handle any personal data under the laws that apply to you.