pyquery Web Scraping: jQuery-Style HTML Parsing in Python

TL;DR:

• pyquery gives you jQuery's API in Python. If you know $("div.quote").find("small.author").text(), you already know pyquery.
• What pyquery does not do is fetch. It wraps lxml for parsing and selecting; it has no HTTP client and runs no JavaScript.
• The gap shows up in one script. A plain GET on a JavaScript-rendered demo page gives pyquery 0 quote nodes; the same URL through the Scrapeless Universal Scraping API with js_render gives it all 10.
• The extraction is real in this guide. A live run used .items(), .find(), and .text() to pull all 10 quotes with authors and tags from the rendered HTML.
• The two layers stay separate. Scrapeless fetches and renders; pyquery selects. Neither reaches into the other's job.
• Free to start on the fetch side. Create your Scrapeless API key at app.scrapeless.com.

What pyquery is, and what it is not

pyquery is a Python library that puts a jQuery-style API on top of lxml. You wrap markup in a PyQuery object — conventionally named d — and then select and traverse with the same calls a front-end developer already uses: d("selector"), .find(), .eq(), .text(), .attr(), .items(). For anyone coming from the browser, it is the shortest path from "I know how to query the DOM" to "I can extract this in Python", and because it sits on lxml, the selection underneath is fast.

It is a parsing and selection library, nothing more. pyquery has no HTTP client, holds no session, and runs no JavaScript. Give it a string and it builds a queryable document; ask it to fetch a URL and, while it can technically pull one, it uses a plain request with no rendering, which is the wrong tool for any page built by scripts. So every "pyquery web scraping" setup is two layers: something that returns faithful rendered HTML, and pyquery that selects from it. This guide uses the Scrapeless Universal Scraping API for the first layer. The broader Python web scraping tutorial covers the surrounding ecosystem.

Install

pyquery and requests are the whole toolchain. The version this guide was written against is pyquery 2.0.1:

pip install "pyquery==2.0.1" requests

Keep your key in the environment, never in source:

export SCRAPELESS_API_KEY="sk_your_scrapeless_key"

Get HTML worth parsing

Selection quality is bounded by fetch fidelity, so start there. On a JavaScript-rendered page, the markup a plain HTTP client receives is not the markup a reader sees — the content arrives only after scripts build the DOM, a lifecycle defined by the HTML scripting specification. One script counts the difference through pyquery itself:

# fidelity.py — what pyquery sees: plain GET vs server-side rendering
import os

import requests
from pyquery import PyQuery as pq

URL = "https://quotes.toscrape.com/js/"

plain = requests.get(URL, timeout=60).text
print("plain GET chars:", len(plain), "| quote nodes:", pq(plain, parser="html")("div.quote").length)

resp = requests.post(
    "https://api.scrapeless.com/api/v2/unlocker/request",
    headers={"Content-Type": "application/json", "x-api-token": os.environ["SCRAPELESS_API_KEY"]},
    json={"actor": "unlocker.webunlocker", "input": {"url": URL, "js_render": True}},
    timeout=120,
)
resp.raise_for_status()
rendered = resp.json().get("data", "")
print("rendered chars:", len(rendered), "| quote nodes:", pq(rendered, parser="html")("div.quote").length)

The run prints 0 quote nodes for the plain fetch and 10 for the rendered one:

plain GET chars: 5806 | quote nodes: 0
rendered chars: 8940 | quote nodes: 10

Rendering, unlocking, and proxy routing all happen server-side in that one POST — the Universal Scraping API is the fetch layer, and the rendered markup is what pyquery selects from.

Extract with the jQuery API

With real HTML in hand, pyquery does the extraction the way jQuery would. .items() turns a selection into an iterator of PyQuery objects, .find() scopes a sub-selector to each one, and .text() reads the text — the selectors follow the W3C Selectors specification, the same syntax jQuery uses:

# extract.py — fetch the rendered page, then select with the jQuery API
import os

import requests
from pyquery import PyQuery as pq

resp = requests.post(
    "https://api.scrapeless.com/api/v2/unlocker/request",
    headers={"Content-Type": "application/json", "x-api-token": os.environ["SCRAPELESS_API_KEY"]},
    json={"actor": "unlocker.webunlocker", "input": {"url": "https://quotes.toscrape.com/js/", "js_render": True}},
    timeout=120,
)
resp.raise_for_status()
d = pq(resp.json().get("data", ""), parser="html")

records = []
for quote in d("div.quote").items():
    records.append({
        "text": quote.find("span.text").text(),
        "author": quote.find("small.author").text(),
        "tags": [pq(tag).text() for tag in quote.find("a.tag").items()],
    })
print("records:", len(records))
print("first author:", records[0]["author"])
print("first tags:", records[0]["tags"])

The live run selected all 10 records, tags and all:

records: 10
first author: Albert Einstein
first tags: ['change', 'deep-thoughts', 'thinking', 'world']

That is the whole scraper: one POST to fetch and render, one PyQuery object to query. The .items() iterator is the pyquery idiom worth remembering — it is what turns a selection into per-record objects you can .find() into, the direct analogue of jQuery's .each().

Get your API key on the free plan: app.scrapeless.com

Advanced patterns

• Always iterate with .items(). Looping over a PyQuery selection directly yields raw lxml elements, not PyQuery objects, so .find() breaks. .items() gives you wrapped objects with the full API on each.
• Read attributes with .attr(). quote.find("a::attr(href)") is not pyquery syntax; use quote.find("a").attr("href"), exactly as in jQuery.
• Pass parser="html" for real pages. It selects lxml's forgiving HTML parser, which handles the malformed markup real sites ship; the default can be stricter than you want.
• Chain, do not re-query. d("div.quote").eq(0).find("small.author") scopes each step to the last, which is both faster and closer to how the jQuery you already know reads.

Troubleshooting

• Zero nodes from a page you can see in a browser. The content is JavaScript-rendered and your fetch returned the pre-render HTML. Count a known selector the way the first script does; a near-empty document is a fetch problem, fixed with js_render, not a selector problem.
• .find() raises AttributeError. You iterated the selection directly instead of with .items(), so you got a bare lxml element. Switch the loop to for x in sel.items():.
• .text() returns everything concatenated. pyquery joins descendant text. Scope tighter with a more specific selector, or read a single node with .eq(0).text().
• Encoding looks wrong. Pass the response text pyquery-first with parser="html"; lxml reads the document's declared encoding when the parser is the HTML one.

Conclusion

pyquery earns its place as the selection layer that speaks jQuery: the same .find(), .text(), and .items() calls you know from the browser, over a fast lxml tree. The layer that decides whether any of that is possible is the fetch — the first script's 0-versus-10 count settles it — and one server-rendered POST closes the gap. Wire the two together and the demo page's ten quotes arrive as clean records, selected the way you already think.

Create a free Scrapeless account to get an API key, and the developer docs cover the unlocker.webunlocker parameters. Check Scrapeless pricing when you plan a recurring job.

FAQ

Q: Can pyquery scrape websites by itself?

Not really. pyquery can pull a URL, but it does so with a plain request and no JavaScript rendering, so on a modern page it selects from empty markup. Treat it as a parser: pair it with a fetch layer — here the Scrapeless Universal Scraping API, which renders the page server-side — and pyquery handles the selection from the returned HTML.

Q: Is pyquery the same as jQuery?

It mirrors jQuery's API in Python — d("selector"), .find(), .text(), .attr(), .items() — but it runs server-side on lxml, not in a browser, so it selects from static markup and does not execute scripts or handle events. The selection syntax transfers; the runtime does not.

Q: pyquery or BeautifulSoup?

Preference. pyquery reads like jQuery and is a natural fit if you come from front-end work; BeautifulSoup has a more Pythonic API. Both parse the same HTML, and both need a separate fetch layer for JavaScript-rendered pages.

Q: Does pyquery handle JavaScript-rendered pages?

Not on its own — it never executes scripts. If the content loads after the initial HTML, a plain fetch hands pyquery an empty document. Fetch the page through the Scrapeless API with js_render first, then select from the rendered HTML, as this guide does.

Q: Is scraping with pyquery legal?

The selection library does not change the collection rules. Fetch public pages only, respect site terms and the robots directives standardized by the Robots Exclusion Protocol, keep volumes bounded, and handle any personal data under the laws that apply to you.