JMESPath Web Scraping: Query JSON APIs Declaratively

TL;DR:

• jmespath queries JSON declaratively. One expression reshapes a nested API response into flat records — no loops, no manual dictionary walking.
• JSON APIs are the cleanest scrape target. Many sites build their pages from a backend JSON endpoint; fetch that JSON and the data arrives already structured.
• What jmespath does not do is fetch. It has no HTTP client and does not parse HTML; you hand it a decoded JSON object and it queries it.
• Fetch through Scrapeless when the API is protected. A live run pulled a product API through the Scrapeless Universal Scraping API, then used jmespath to select, filter, and sort the results.
• Filters and projections in one line. products[?price < \50`].titlereturned the six products under $50;sort_by(products, &price)[0]` returned the cheapest.
• Free to start on the fetch side. Create your Scrapeless API key at app.scrapeless.com.

What jmespath is, and what it is not

jmespath is a query language for JSON. You write an expression that describes the shape you want, and the library walks the document and returns it — projections pull a field from every element of a list, filters keep only the elements that match a condition, and multiselect hashes rebuild each element into a smaller record. It is the same expression language the AWS CLI uses for its --query flag, standardized by the JMESPath specification, and available as a small Python library.

It is a query language, not a scraper. jmespath has no HTTP client, does not fetch URLs, and does not parse HTML — it operates on a JSON value you have already decoded, defined by the JSON data interchange standard. So a "jmespath web scraping" setup is two layers: something that returns the JSON, and jmespath that reshapes it. This matters because a large share of the data on modern sites is served by a backend JSON API that the page calls in the background; hitting that endpoint directly skips HTML parsing entirely. When the endpoint is geo-restricted or rate-limited, this guide fetches it through the Scrapeless Universal Scraping API. For the HTML side of scraping, the Python web scraping tutorial covers selectors instead.

Install

jmespath and requests are the whole toolchain. The version this guide was written against is jmespath 1.0.1:

pip install "jmespath==1.0.1" requests

Keep your key in the environment, never in source:

export SCRAPELESS_API_KEY="sk_your_scrapeless_key"

Fetch a JSON API through Scrapeless

The fetch layer returns the raw JSON. Because the endpoint serves JSON rather than a rendered page, js_render stays off; the Scrapeless API handles the request, proxy routing, and any access controls in front of the endpoint, and returns the body defined by the HTTP semantics standard. Decode it once and jmespath takes over:

# fetch.py — pull a JSON API through Scrapeless, then query it
import json
import os

import jmespath
import requests

resp = requests.post(
    "https://api.scrapeless.com/api/v2/unlocker/request",
    headers={"Content-Type": "application/json", "x-api-token": os.environ["SCRAPELESS_API_KEY"]},
    json={"actor": "unlocker.webunlocker", "input": {"url": "https://dummyjson.com/products?limit=10", "js_render": False}},
    timeout=120,
)
resp.raise_for_status()
payload = json.loads(resp.json()["data"])

print("products in page:", jmespath.search("length(products)", payload))
print("first title:", jmespath.search("products[0].title", payload))
print("total available:", jmespath.search("total", payload))

The run reads the response's shape without a single loop:

products in page: 10
first title: Essence Mascara Lash Princess
total available: 194

The Scrapeless call is the fetch layer — the Universal Scraping API returns the JSON body, and payload is now a plain Python object jmespath can query.

Reshape and filter with jmespath

The point of jmespath is turning a verbose response into exactly the records you want. A projection with a multiselect hash rebuilds each product; a filter expression keeps only the matches; sort_by orders them — all as expressions, not procedural code:

# query.py — reshape, filter, and sort in three expressions
import json
import os

import jmespath
import requests

resp = requests.post(
    "https://api.scrapeless.com/api/v2/unlocker/request",
    headers={"Content-Type": "application/json", "x-api-token": os.environ["SCRAPELESS_API_KEY"]},
    json={"actor": "unlocker.webunlocker", "input": {"url": "https://dummyjson.com/products?limit=10", "js_render": False}},
    timeout=120,
)
resp.raise_for_status()
payload = json.loads(resp.json()["data"])

records = jmespath.search("products[].{title: title, price: price, rating: rating}", payload)
under_50 = jmespath.search("products[?price < `50`].title", payload)
cheapest = jmespath.search("sort_by(products, &price)[0].{title: title, price: price}", payload)

print("records:", len(records))
print("first record:", json.dumps(records[0], ensure_ascii=False))
print("under $50:", len(under_50))
print("cheapest:", json.dumps(cheapest, ensure_ascii=False))

Each line is one query doing the work of a loop:

records: 10
first record: {"title": "Essence Mascara Lash Princess", "price": 9.99, "rating": 2.56}
under $50: 6
cheapest: {"title": "Red Nail Polish", "price": 8.99}

That is the whole extractor: one POST to fetch the JSON, three expressions to shape it. The multiselect hash {title: title, price: price} is the workhorse — it drops the fields you do not need and renames the ones you keep, so what you store is exactly what you asked for.

Get your API key on the free plan: app.scrapeless.com

Advanced patterns

• Filter before you project. products[?rating > \4.5`].{title: title}` keeps the matches first, then reshapes them; ordering the pipe this way keeps the expression readable and the result small.
• Flatten nested lists with []. When records nest their own lists, products[].reviews[].rating flattens every review rating across every product into one list — the flatten operator does what a double loop would.
• Pipe expressions with |. products | length(@) and sort_by(@, &price) | [0] chain a result into the next expression; @ is the current node, which is how you feed one query's output into another.
• Guard against missing keys. jmespath returns None for a path that is absent rather than raising, so a field that only some records carry will not crash the query — check for None when you store it.

Troubleshooting

• json.loads raises on the response. The endpoint returned HTML, not JSON — often an error or block page. Confirm the URL is the JSON API and not the HTML page that calls it, and that the fetch succeeded before decoding.
• A projection returns an empty list. The path does not match the document's shape. Print the top-level keys and walk down one level at a time; JSON APIs nest their arrays under a key like products or results, not at the root.
• A filter matches nothing. Backtick literals are required for numbers and strings in a filter — price < \50`, not price < 50`. Without the backticks the value is read as a field name.
• The result keeps fields you did not want. You used a bare projection products[] instead of a multiselect hash. Add .{title: title, price: price} to select only the fields to keep.

Conclusion

jmespath earns its place as the layer that turns a JSON response into records without procedural code: projections, filters, and sorts as single expressions. The layer that gets you the JSON is the fetch — a backend API is the cleanest source there is, and one Scrapeless POST returns its body past whatever guards the endpoint. Wire the two together and a verbose product feed becomes the four fields you actually store.

Create a free Scrapeless account to get an API key, and the developer docs cover the unlocker.webunlocker parameters. Check Scrapeless pricing when you plan a recurring job.

FAQ

Q: Can jmespath scrape websites by itself?

No. jmespath queries a JSON value you already have; it has no HTTP client and does not fetch URLs or parse HTML. Pair it with a fetch layer — here the Scrapeless Universal Scraping API, which returns the JSON body — and jmespath reshapes it into records.

Q: Why scrape a JSON API instead of the HTML page?

Because the data arrives already structured. Many pages render from a backend JSON endpoint they call in the background; hitting that endpoint skips HTML parsing and selector maintenance entirely, and jmespath turns the response into exactly the records you want.

Q: How is jmespath different from jsonpath?

Both query JSON, but jmespath has a formal specification and a compact expression language with projections, filters, functions, and multiselect hashes that reshape output. Its multiselect syntax — renaming and dropping fields in the query — is the feature that makes it a good fit for extraction.

Q: What if the site has no JSON API?

Then parse the HTML instead: fetch the rendered page through Scrapeless and use a selector library. jmespath applies only when the source is JSON; the two approaches cover the two shapes data comes in.

Q: Is scraping a JSON API legal?

The query language does not change the collection rules. Fetch public endpoints only, respect site terms and the robots directives standardized by the Robots Exclusion Protocol, keep volumes bounded, and handle any personal data under the laws that apply to you.