Rust Web Scraping With reqwest and scraper: A Practical Guide

TL;DR:

• Rust web scraping in 2026 runs on two crates: reqwest for the HTTP request and scraper for CSS-selector parsing, with tokio driving the async runtime.
• The compiler forces you to handle every failure path, so a Rust scraper that builds is usually a scraper that survives malformed markup and non-200 responses.
• tokio::spawn turns a page-by-page crawler into a concurrent one in about five lines, and this guide runs it across five pages for 50 records.
• Neither reqwest nor scraper executes JavaScript, so a client-rendered page returns markup that parses cleanly to zero records.
• The Scrapeless Universal Scraping API renders the page first and hands back HTML that the same unchanged scraper code parses to 10 records.
• Start on the Scrapeless free plan and run the pivot example against your own target.

Rust shows up in scraping work for a specific reason: the crawler is usually the part of a data pipeline that runs unattended for hours against markup nobody controls. A borrow-checked binary with no garbage-collector pauses and explicit Result handling at every boundary is a good fit for that job.

This guide builds a working scraper with the current crate releases, runs it, and then shows exactly where the pure-Rust stack stops — with measured numbers rather than a warning.

What You Need

Rust web scraping needs three crates and a stable toolchain. The versions below are the current published releases on the Rust community crate registry at the time of writing, and every example here was compiled and run against exactly these:

The scraper crate wraps html5ever, the same parsing engine used in Servo, so it follows the HTML parsing specification rather than treating markup as text to regex over. That matters on real pages, where unclosed tags are normal.

Install

Create the project and add the dependencies:

cargo new rust-scraper
cd rust-scraper

Then declare the dependency block in Cargo.toml:

[package]
name = "rust-scraper"
version = "0.1.0"
edition = "2021"

[dependencies]
reqwest = { version = "0.13.4", features = ["json"] }
scraper = "0.27.0"
tokio = { version = "1.53.0", features = ["macros", "rt-multi-thread"] }
serde_json = "1"

The json feature on reqwest pulls in request and response serialization. The macros and rt-multi-thread features on tokio are what make the #[tokio::main] attribute work.

Fetch and Parse a Page

A complete Rust scraper is shorter than its reputation suggests. This one fetches a server-rendered page, selects every quote container, and pulls two fields from each:

use scraper::{Html, Selector};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let body = reqwest::get("https://quotes.toscrape.com/")
        .await?
        .error_for_status()?
        .text()
        .await?;

    let document = Html::parse_document(&body);
    let quote_sel = Selector::parse("div.quote").unwrap();
    let text_sel = Selector::parse("span.text").unwrap();
    let author_sel = Selector::parse("small.author").unwrap();

    let mut count = 0;
    for quote in document.select(&quote_sel) {
        let text = quote.select(&text_sel).next().map(|e| e.inner_html());
        let author = quote.select(&author_sel).next().map(|e| e.inner_html());
        if let (Some(t), Some(a)) = (text, author) {
            if count == 0 {
                println!("first quote: {} — {}", t, a);
            }
            count += 1;
        }
    }
    println!("quotes parsed: {}", count);
    Ok(())
}

Running it prints:

first quote: “The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.” — Albert Einstein
quotes parsed: 10

Three details in that code carry most of the weight.

error_for_status() converts a 4xx or 5xx response into an Err instead of letting you parse an error page as if it were data. Without it, a 403 body becomes zero selector matches and looks identical to an empty result set. The distinction between those status classes is defined in the HTTP semantics specification.

Selector::parse is fallible because a selector string is compiled, not interpreted at match time. Compiling selectors once outside the loop — rather than per element — is why the nested select calls stay cheap. The syntax follows the W3C Selectors Level 4 specification.

The ? operator propagating out of main is what turns Box<dyn std::error::Error> into a practical return type. Every fallible call unwinds to the same place, and the process exits non-zero on failure — useful when the scraper runs from a scheduler.

Scrape Pages Concurrently

Rust's concurrency story is the main argument for using it here, and tokio::spawn is where it shows up. Each page becomes an independent task, all of them share one connection-pooled client, and the results are collected in order:

use scraper::{Html, Selector};

async fn page_quote_count(client: &reqwest::Client, page: u32) -> Result<usize, reqwest::Error> {
    let url = format!("https://quotes.toscrape.com/page/{page}/");
    let body = client.get(&url).send().await?.error_for_status()?.text().await?;
    let quote_sel = Selector::parse("div.quote").unwrap();
    Ok(Html::parse_document(&body).select(&quote_sel).count())
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = reqwest::Client::new();

    let tasks: Vec<_> = (1..=5)
        .map(|page| {
            let client = client.clone();
            tokio::spawn(async move { (page, page_quote_count(&client, page).await) })
        })
        .collect();

    let mut total = 0;
    for task in tasks {
        let (page, result) = task.await?;
        let count = result?;
        println!("page {page}: {count} quotes");
        total += count;
    }
    println!("total quotes across 5 pages: {total}");
    Ok(())
}

The run:

page 1: 10 quotes
page 2: 10 quotes
page 3: 10 quotes
page 4: 10 quotes
page 5: 10 quotes
total quotes across 5 pages: 50

reqwest::Client is cheap to clone because the clone shares the underlying connection pool. Creating a fresh client per task would open a new pool each time and discard the benefit. Keep the page range bounded and sized to what the target can comfortably serve — concurrency is a tool for finishing a known workload, not for issuing as many simultaneous requests as the runtime allows.

Ready to point this at a target that renders server-side? Create a free Scrapeless account and keep the parsing code you already have.

Where reqwest and scraper Stop

Neither crate executes JavaScript. reqwest returns the bytes the server sent, and scraper parses exactly those bytes — so on a page that builds its content in the browser, the selectors match nothing.

This is measurable rather than theoretical. The site used above publishes a client-rendered twin of the same data at /js/. Pointing the identical parsing logic at it:

use scraper::{Html, Selector};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let body = reqwest::get("https://quotes.toscrape.com/js/")
        .await?
        .error_for_status()?
        .text()
        .await?;

    let document = Html::parse_document(&body);
    let quote_sel = Selector::parse("div.quote").unwrap();
    let count = document.select(&quote_sel).count();

    println!("html bytes: {}", body.len());
    println!("quotes parsed: {}", count);
    Ok(())
}

html bytes: 5808
quotes parsed: 0

The request succeeded. The status was 200. The parser worked correctly on 5,808 bytes of valid HTML that simply did not contain any quote elements — they are written into the DOM after a script runs. A scraper that only checks for HTTP failure treats this as an empty page rather than a missing capability, which is why the earlier error_for_status() call is necessary but not sufficient.

Render the Page With the Universal Scraping API

The Scrapeless Universal Scraping API closes that gap by rendering the page in a cloud browser and returning the resulting HTML, which means the Rust side stays a plain HTTP call. Set js_render to true and the response body contains the post-script DOM.

Authenticate with your key from the environment:

export SCRAPELESS_API_KEY="your_api_key_here"

Then swap only the fetch layer — the selector code below is identical to the first example:

use scraper::{Html, Selector};
use serde_json::json;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let api_key = std::env::var("SCRAPELESS_API_KEY")?;

    let payload = json!({
        "actor": "unlocker.webunlocker",
        "input": {
            "url": "https://quotes.toscrape.com/js/",
            "js_render": true,
            "headless": true
        }
    });

    let envelope: serde_json::Value = reqwest::Client::new()
        .post("https://api.scrapeless.com/api/v2/unlocker/request")
        .header("x-api-token", api_key)
        .json(&payload)
        .send()
        .await?
        .error_for_status()?
        .json()
        .await?;

    let body = envelope["data"].as_str().unwrap_or_default();

    let document = Html::parse_document(body);
    let quote_sel = Selector::parse("div.quote").unwrap();
    let text_sel = Selector::parse("span.text").unwrap();
    let author_sel = Selector::parse("small.author").unwrap();

    let mut count = 0;
    for quote in document.select(&quote_sel) {
        let text = quote.select(&text_sel).next().map(|e| e.inner_html());
        let author = quote.select(&author_sel).next().map(|e| e.inner_html());
        if let (Some(t), Some(a)) = (text, author) {
            if count == 0 {
                println!("first quote: {} — {}", t, a);
            }
            count += 1;
        }
    }
    println!("html bytes: {}", body.len());
    println!("quotes parsed: {}", count);
    Ok(())
}

first quote: “The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.” — Albert Einstein
html bytes: 8985
quotes parsed: 10

Same page, same selectors, same crate versions. The only change is which layer fetched the HTML, and the record count moves from 0 to 10 while the payload grows from 5,808 to 8,985 bytes — the difference being the quote markup that the script wrote into the DOM.

The response arrives as a JSON envelope with the rendered document in data as a string, which is why the example parses to serde_json::Value first and hands data to Html::parse_document. Details of the render options live in the Scrapeless documentation, and the same js_render behavior is covered in more depth in the JS rendering guide.

Troubleshooting

Selectors match nothing on a page you can see in a browser. Print the response length first, as the JS example above does. A few thousand bytes with zero matches usually means the content is client-rendered, not that the selector is wrong. View the page source rather than the inspector — the inspector shows the DOM after scripts have run, which is not what reqwest received.

Selector::parse panics at startup. The selector string is invalid CSS. Pseudo-elements and some jQuery-style extensions are not part of the specification and will not compile.

The build fails on the TLS backend. reqwest compiles a TLS stack; on a minimal container, the system needs a C toolchain and CMake available, or you can switch to the pure-Rust rustls backend through feature flags.

inner_html() returns markup instead of text. That method returns everything inside the element, tags included. Use text() and collect the fragments when a field can contain nested elements.

Before pointing any of this at a live target, check the site's terms and its /robots.txt directives, which follow the Robots Exclusion Protocol standard. Keep collection to public data and to a volume the target can serve comfortably.

Conclusion

Rust gives you a scraper that handles failure explicitly and parallelizes without much ceremony — reqwest for transport, scraper for selectors, tokio for concurrency, and a compiler that will not let you ignore a Result. That stack covers server-rendered pages completely.

What it does not do is run JavaScript, and the cost of that gap is a silent zero rather than an error. Measuring it is the useful habit: 10 records on the server-rendered page, 0 on the client-rendered twin, 10 again once something renders the page before Rust parses it.

Start with the Scrapeless free plan to run the render step against your own targets, and review the current Scrapeless pricing when you size a job.

FAQ

Q: Which Rust crate is best for web scraping?

reqwest paired with scraper covers most work. reqwest handles HTTP with an async-first API, and scraper provides CSS-selector querying over an html5ever parse tree. Reach for a headless-browser crate or a rendering API only when the page builds its content client-side.

Q: Is Rust good for web scraping compared with Python?

Rust is a good fit when the scraper runs long, runs concurrently, or runs unattended, because there is no garbage-collector pause and the compiler forces every error path to be handled. Python still wins on ecosystem breadth and iteration speed for one-off extraction. The parsing concepts transfer directly between them.

Q: Can reqwest execute JavaScript?

No. reqwest is an HTTP client and returns the bytes the server sent. On a client-rendered page that means valid HTML with none of the content in it — the example above parses 5,808 bytes to zero records. Rendering has to happen somewhere else, either in a headless browser or through an API that returns the rendered DOM.

Q: Do I need async and tokio for a simple scraper?

Not strictly — reqwest ships a blocking client behind a feature flag, which is fine for a single sequential request. The async client is worth the tokio dependency as soon as you fetch more than one page, since that is where tokio::spawn turns sequential fetches into concurrent ones.

Q: How do I keep a Rust scraper from breaking when the site changes?

Assert on structure rather than trusting it. Check that the container selector matched a plausible number of elements before extracting fields, and treat a sudden zero as a failure rather than an empty result. Compiling selectors once at startup also surfaces invalid CSS immediately instead of mid-crawl.