Polars Web Scraping: From Live Page to Fast DataFrame

TL;DR:

• Polars is a fast, Arrow-backed DataFrame library, and it pairs with Scrapeless to turn a live page into a typed, queryable frame.
• Polars has no HTTP client and no HTML parser, so the fetch and extract steps come from Scrapeless and a parser; Polars takes over once the records exist.
• Build a frame from a list of dictionaries with pl.DataFrame(records), and the schema is typed from the first row when you coerce values during extraction.
• The expression API replaces pandas-style chained indexing: pl.col("price_gbp").mean() inside group_by(...).agg(...) reads and runs cleanly.
• The lazy path, df.lazy()...collect(), lets Polars plan and optimize the whole query before it touches data, which is where it pulls ahead on larger sets.
• Start on the Scrapeless free plan and point the fetch step at your own catalog.

Polars keeps showing up in data work because it is fast and its API is pleasant, but almost every tutorial hands it a Parquet file that already exists. Scraping is the opposite situation. The data is live, it is HTML, and it is untyped until you make it typed. This guide takes a public books catalog from a URL to a Polars DataFrame you can group and aggregate, and it is honest about the one thing Polars will not do for you: get the page.

The fetch comes from the Scrapeless Universal Scraping API, which returns rendered HTML for a public URL. A parser turns that HTML into records. Polars does the rest, and does it fast.

What Polars Adds to a Scraping Workflow

Polars is a DataFrame library built on the Apache Arrow columnar memory format, which is what makes its typed columns and grouped aggregations quick. For scraped data the payoff is concrete: once your records are in a frame, cleaning and summarizing them is a few expressions rather than a hand-written loop, and the columns keep their types. Polars leans on the Apache Arrow columnar format for its in-memory layout, so a column of prices is a real numeric column, not a list of strings you re-parse on every operation.

Install

Polars handles the frame; a parser handles the HTML. Install both.

pip install polars lxml

Set your Scrapeless API key once in the shell. Use the real key at run time and keep the placeholder out of your source.

export SCRAPELESS_API_KEY="sk_your_key_here"

Fetch, Extract, and Build a Frame

The first stage fetches the page, pulls one record per product, and builds the DataFrame. Because Polars cannot fetch or parse HTML, this stage is where Scrapeless and the parser do their work; Polars takes the finished records.

import json
import os
import urllib.request

import polars as pl
from lxml import html as lxhtml

API_URL = "https://api.scrapeless.com/api/v2/unlocker/request"
RATINGS = {"One": 1, "Two": 2, "Three": 3, "Four": 4, "Five": 5}


def fetch_html(url: str) -> str:
    payload = json.dumps(
        {"actor": "unlocker.webunlocker", "input": {"url": url, "js_render": False, "headless": False}}
    ).encode()
    request = urllib.request.Request(
        API_URL,
        data=payload,
        headers={"x-api-token": os.environ["SCRAPELESS_API_KEY"], "Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(request, timeout=90) as response:
        return json.loads(response.read())["data"]


dom = lxhtml.fromstring(fetch_html("https://books.toscrape.com/"))
records = []
for card in dom.cssselect("article.product_pod"):
    records.append(
        {
            "title": card.cssselect("h3 a")[0].get("title"),
            "price_gbp": float(card.cssselect("p.price_color")[0].text_content().strip().lstrip("£")),
            "rating": RATINGS[card.cssselect("p.star-rating")[0].get("class").split()[-1]],
            "in_stock": "In stock" in card.cssselect("p.instock.availability")[0].text_content(),
        }
    )

df = pl.DataFrame(records)
print(f"records extracted: {df.height}")
print("schema:", dict(df.schema))

The extraction uses the CSS Selectors standard through lxml's cssselect, and it coerces each value as it reads it: the price becomes a float, the star-rating word becomes an integer, and availability becomes a boolean. That coercion is the important habit. Build the records with real Python types and Polars infers a real schema from them.

records extracted: 20
schema: {'title': String, 'price_gbp': Float64, 'rating': Int64, 'in_stock': Boolean}

Twenty product cards on the first page become a frame with a typed schema: String, Float64, Int64, and Boolean. No column is left as text that later has to be re-parsed.

Transform With Expressions

Polars replaces chained indexing with an expression API, and this is the part worth learning properly. An expression like pl.col("price_gbp").mean() describes a computation that Polars runs efficiently, and expressions compose inside filter, group_by, and agg.

summary = (
    df.lazy()
    .filter(pl.col("in_stock"))
    .group_by("rating")
    .agg(pl.len().alias("books"), pl.col("price_gbp").mean().round(2).alias("avg_price"))
    .sort("rating")
    .collect()
)
print(summary)

df.lazy() turns the eager frame into a query plan, the chain describes the work, and .collect() executes it. On this small set the difference is invisible, but the lazy path lets Polars analyze the whole query and skip work before it reads a single value, which is why it scales. The aggregate groups in-stock books by star rating and reports the count and mean price per group.

shape: (5, 3)
┌────────┬───────┬───────────┐
│ rating ┆ books ┆ avg_price │
│ ---    ┆ ---   ┆ ---       │
│ i64    ┆ u32   ┆ f64       │
╞════════╪═══════╪═══════════╡
│ 1      ┆ 6     ┆ 40.02     │
│ 2      ┆ 3     ┆ 36.83     │
│ 3      ┆ 3     ┆ 42.32     │
│ 4      ┆ 4     ┆ 31.1      │
│ 5      ┆ 4     ┆ 39.75     │
└────────┴───────┴───────────┘

The Complete Script

Put the stages together, add one more expression to find the cheapest five-star book, and store the frame as Parquet.

import json
import os
import urllib.request

import polars as pl
from lxml import html as lxhtml

API_URL = "https://api.scrapeless.com/api/v2/unlocker/request"
RATINGS = {"One": 1, "Two": 2, "Three": 3, "Four": 4, "Five": 5}


def fetch_html(url: str) -> str:
    payload = json.dumps(
        {"actor": "unlocker.webunlocker", "input": {"url": url, "js_render": False, "headless": False}}
    ).encode()
    request = urllib.request.Request(
        API_URL,
        data=payload,
        headers={"x-api-token": os.environ["SCRAPELESS_API_KEY"], "Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(request, timeout=90) as response:
        return json.loads(response.read())["data"]


dom = lxhtml.fromstring(fetch_html("https://books.toscrape.com/"))
records = []
for card in dom.cssselect("article.product_pod"):
    records.append(
        {
            "title": card.cssselect("h3 a")[0].get("title"),
            "price_gbp": float(card.cssselect("p.price_color")[0].text_content().strip().lstrip("£")),
            "rating": RATINGS[card.cssselect("p.star-rating")[0].get("class").split()[-1]],
            "in_stock": "In stock" in card.cssselect("p.instock.availability")[0].text_content(),
        }
    )

df = pl.DataFrame(records)
print(f"records extracted: {df.height} | columns: {df.columns}")

summary = (
    df.lazy()
    .filter(pl.col("in_stock"))
    .group_by("rating")
    .agg(pl.len().alias("books"), pl.col("price_gbp").mean().round(2).alias("avg_price"))
    .sort("rating")
    .collect()
)
print(summary)

cheapest = df.filter(pl.col("rating") == 5).sort("price_gbp").select("title", "price_gbp").head(1)
print("cheapest 5-star:", cheapest.to_dicts())

df.write_parquet("books.parquet")
print(f"parquet bytes: {os.path.getsize('books.parquet')}")

The run reports each step, ending with a Parquet file that keeps the schema for the next reader.

records extracted: 20 | columns: ['title', 'price_gbp', 'rating', 'in_stock']
shape: (5, 3)
┌────────┬───────┬───────────┐
│ rating ┆ books ┆ avg_price │
│ ---    ┆ ---   ┆ ---       │
│ i64    ┆ u32   ┆ f64       │
╞════════╪═══════╪═══════════╡
│ 1      ┆ 6     ┆ 40.02     │
│ 2      ┆ 3     ┆ 36.83     │
│ 3      ┆ 3     ┆ 42.32     │
│ 4      ┆ 4     ┆ 31.1      │
│ 5      ┆ 4     ┆ 39.75     │
└────────┴───────┴───────────┘
cheapest 5-star: [{'title': 'Set Me Free', 'price_gbp': 17.46}]
parquet bytes: 2233

write_parquet stores the typed columns in the Apache Parquet file format, so the next process reads price_gbp back as a float with no re-coercion. The Polars user guide documents the full expression and lazy API when you outgrow this example.

What Polars Will Not Do

Polars is a dataframe engine, and that is the whole boundary: it has no HTTP client and no HTML parser. It will not request a URL, it will not render JavaScript, and it will not turn <article> tags into rows. That is by design, and it is why this workflow uses two tools in front of it. Scrapeless retrieves the page, and a parser reads the markup into records. If you need a deeper treatment of the parsing step, the guide on using BeautifulSoup for web scraping covers HTML extraction in Python, and the same records feed straight into pl.DataFrame.

When a target renders its content with JavaScript, the first fetch returns an empty shell and the extraction loop finds no cards. Set js_render to True in the request so Scrapeless returns the page after its scripts run, and leave it False when the markup is already server-rendered, as this catalog is.

Before you widen the crawl past the first page, read the target's robots.txt and terms. The Robots Exclusion Protocol states which paths a site asks automated clients to avoid, and staying inside it keeps the workflow sustainable. Keep the volume bounded and the data public, as this example does.

Ready to try it on your own data? Create a free Scrapeless account and change the URL and selectors in the fetch stage.

Conclusion

Polars turns scraped records into a typed, queryable frame with very little code, provided something else gets the page there first. Scrapeless fetches the HTML, a parser builds the records with real types, and Polars groups, aggregates, and stores them through its expression and lazy APIs. Start from the complete script, swap the URL and selectors for your own target, and reach for the lazy path as your data grows.

Start with the Scrapeless free plan to fetch your own pages, and check Scrapeless pricing when you plan a recurring job.

FAQ

Q: Does Polars scrape web pages on its own?

No. Polars is a DataFrame library with no HTTP client and no HTML parser, so it cannot request a URL or read markup. Pair it with a fetch layer such as Scrapeless and a parser such as lxml or BeautifulSoup; Polars takes over once the records exist.

Q: Why choose Polars over pandas for scraped data?

Polars is built on the Apache Arrow columnar format and offers a lazy query engine, so grouped aggregations and filters on large frames are fast, and the expression API is consistent. One trade-off is relevant to scraping: pandas has read_html for tables, while Polars expects you to build the frame from records yourself, which suits card-style pages better than table-only ones.

Q: What is the difference between eager and lazy in Polars?

An eager DataFrame runs each operation immediately, while df.lazy() builds a query plan that only executes when you call .collect(). The lazy path lets Polars optimize the whole query and skip unnecessary work, which is why it scales better on large datasets even though the result is identical on small ones.

Q: Why do my scraped numbers show up as strings in Polars?

Scraped values arrive as text, and if you pass them to pl.DataFrame unchanged, the column type is String. Coerce each value during extraction, as the example casts price to float and the rating word to an integer, so the schema is typed from the first row and no column needs re-parsing later.

Q: How do I build a Polars DataFrame from a list of scraped records?

Collect each record as a dictionary with consistent keys and pass the list to pl.DataFrame(records). Polars infers the schema from the values, so typed values in the dictionaries produce a typed frame, ready for the expression API.