Playwright + Scrapeless Scraping Browser: Intercept the Hidden JSON API

A page that says "quotes" ships zero quotes. Request https://quotes.toscrape.com/scroll with a plain HTTP client and the response body contains one empty <div class="quotes"></div> — the actual text, authors, and tags arrive afterward, over a GET /api/quotes?page=N call the page's own JavaScript fires on load and on every scroll. Reading that call directly, instead of waiting for a browser to render it into HTML and then parsing the HTML back out, is network request interception: connect a real browser over the Chrome DevTools Protocol, listen to the traffic it makes, and read the JSON the site already produces for itself.

This guide connects Playwright to the Scrapeless Scraping Browser over CDP, then intercepts that hidden endpoint two ways — Playwright's own response events and the raw CDP Network domain underneath them — before replaying the endpoint directly over plain HTTP once its shape is known. Every command below ran against the live target.

Why Read the Wire Instead of the DOM

quotes.toscrape.com/scroll is a public scraping-practice target built specifically to demonstrate infinite scroll, and its markup makes the case for interception on its own. Fetch the page and look for the data:

curl -s https://quotes.toscrape.com/scroll | grep -o 'class="quote"' | wc -l

That returns zero every time, because the quotes never live in the HTML the server sends. A small jQuery block calls $.get('/api/quotes', {page: page}) once on load and again whenever the scroll position nears the bottom, then appends the returned rows into the empty container by hand. A browser that runs the JavaScript and waits long enough will eventually show the same 10 quotes per page in its DOM — but by the time you're back to counting .quote elements and pulling text out of <span class="text"> and <small class="author"> nodes, you've reconstructed, by hand, data the page already had as clean, typed JSON: a quotes array, each with text, an author object, and a tags list, plus a has_next flag that tells you exactly when to stop. Reading the response directly skips the reconstruction step and gives you fields instead of nodes.

Prerequisites

You need Python 3.9 or newer — playwright 1.59.0 declares Requires-Python >=3.9 on PyPI — the playwright package itself, and a Scrapeless API key from the free plan at app.scrapeless.com. The key travels as the token query parameter on the Scraping Browser's CDP endpoint, so keep it in an environment variable rather than a literal in your script. No local Chrome install is required: connect_over_cdp reaches a browser that already exists in the cloud.

Connect Playwright to the Scraping Browser Over CDP

Install the client and set the key:

pip install playwright

export SCRAPELESS_API_KEY="your_scrapeless_api_key"

The Scraping Browser's CDP endpoint is wss://browser.scrapeless.com/api/v2/browser, reached with three query parameters — token, sessionTTL, and proxyCountry — the same builder every Playwright-over-Scraping-Browser script in this series uses:

import os
from urllib.parse import urlencode

API_KEY = os.environ["SCRAPELESS_API_KEY"]

def scraping_browser_url(proxy_country="US", session_ttl=180):
    params = urlencode({
        "token": API_KEY,
        "sessionTTL": session_ttl,
        "proxyCountry": proxy_country,
    })
    return f"wss://browser.scrapeless.com/api/v2/browser?{params}"

chromium.connect_over_cdp(scraping_browser_url()) hands back a standard Playwright Browser object. Nothing about the interception patterns below is Scraping-Browser-specific — they run against any CDP-reachable Chromium — but running them on Scraping Browser means the render happens on infrastructure that already carries residential egress and anti-detection Chromium, so sites that fingerprint aggressively still hydrate normally while you read their traffic.

Capture the Hidden API With a Response Listener

Playwright's page.expect_response() binds the wait to the action that triggers it — no arbitrary sleep, no polling a selector count until it looks stable. Bind it around page.goto() for the first /api/quotes call, then around the scroll that triggers the second:

import os
from urllib.parse import urlencode
from playwright.sync_api import sync_playwright

API_KEY = os.environ["SCRAPELESS_API_KEY"]

def scraping_browser_url(proxy_country="US", session_ttl=180):
    params = urlencode({
        "token": API_KEY,
        "sessionTTL": session_ttl,
        "proxyCountry": proxy_country,
    })
    return f"wss://browser.scrapeless.com/api/v2/browser?{params}"

with sync_playwright() as p:
    browser = p.chromium.connect_over_cdp(scraping_browser_url())
    page = browser.new_page()

    with page.expect_response("**/api/quotes*") as first_page:
        page.goto("https://quotes.toscrape.com/scroll", wait_until="domcontentloaded")
    data = first_page.value.json()
    print("DOM quote count before the first response lands:", page.locator(".quote").count())
    print(f"intercepted page {data['page']} -> {len(data['quotes'])} quotes, has_next={data['has_next']}")

    with page.expect_response("**/api/quotes*") as second_page:
        page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
    data = second_page.value.json()
    print("DOM quote count once the scroll settles:", page.locator(".quote").count())
    print(f"intercepted page {data['page']} -> {len(data['quotes'])} quotes, has_next={data['has_next']}")

    browser.close()

Running it against the live site prints:

DOM quote count before the first response lands: 10
intercepted page 1 -> 10 quotes, has_next=True
DOM quote count once the scroll settles: 20
intercepted page 2 -> 10 quotes, has_next=True

The glob pattern "**/api/quotes*" matches the endpoint by URL shape, which is Playwright's documented pattern for waiting on a specific network response rather than a fixed timeout. Because the wait is tied to the triggering action, data['page'], data['quotes'], and data['has_next'] come back as the same typed Python values the site's own script consumes — no .author.name reconstructed from a <small> tag, no tag list rebuilt from anchor text.

Go Lower: Read Raw Frames With the CDP Network Domain

Playwright's response events sit on top of the Chrome DevTools Protocol Network domain, and you can talk to that domain directly through a CDPSession. This matters when you're not driving Playwright at all — a bare CDP client in another language, a tool that only exposes protocol events, or a case where you want response headers and timing that a specific binding doesn't surface — because Network.responseReceived and Network.getResponseBody work the same way regardless of which client library sits above them:

import os, json
from urllib.parse import urlencode
from playwright.sync_api import sync_playwright

API_KEY = os.environ["SCRAPELESS_API_KEY"]

def scraping_browser_url(proxy_country="US", session_ttl=180):
    params = urlencode({
        "token": API_KEY,
        "sessionTTL": session_ttl,
        "proxyCountry": proxy_country,
    })
    return f"wss://browser.scrapeless.com/api/v2/browser?{params}"

request_ids, bodies = {}, []

with sync_playwright() as p:
    browser = p.chromium.connect_over_cdp(scraping_browser_url())
    page = browser.new_page()
    cdp = page.context.new_cdp_session(page)
    cdp.send("Network.enable")

    def on_response_received(event):
        if "/api/quotes" in event["response"]["url"]:
            request_ids[event["requestId"]] = event["response"]["url"]

    def on_loading_finished(event):
        rid = event["requestId"]
        if rid in request_ids:
            raw = cdp.send("Network.getResponseBody", {"requestId": rid})
            data = json.loads(raw["body"])
            bodies.append((request_ids[rid], data["page"], len(data["quotes"]), data["quotes"][0]["author"]["name"]))

    cdp.on("Network.responseReceived", on_response_received)
    cdp.on("Network.loadingFinished", on_loading_finished)

    page.goto("https://quotes.toscrape.com/scroll", wait_until="domcontentloaded")
    page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
    for _ in range(20):
        if len(bodies) >= 2:
            break
        page.wait_for_timeout(300)

    for url, page_no, count, author in bodies:
        print(f"CDP Network.getResponseBody on {url}: page={page_no}, quotes={count}, first_author={author}")

    browser.close()

CDP Network.getResponseBody on https://quotes.toscrape.com/api/quotes?page=1: page=1, quotes=10, first_author=Albert Einstein
CDP Network.getResponseBody on https://quotes.toscrape.com/api/quotes?page=2: page=2, quotes=10, first_author=Marilyn Monroe

Network.responseReceived fires with headers and a requestId as soon as the response starts; the body itself isn't available until Network.loadingFinished confirms the transfer completed, which is why the handler splits into two events instead of reading the body off the first one. getResponseBody returns the exact bytes the browser received, plus a base64Encoded flag for binary payloads — the layer Playwright's response.json() builds on top of and hides from you.

Get free Scraping Browser runtime by signing up at app.scrapeless.com and running both scripts above against your own target.

Skip the Browser Once You Know the Shape

Both interceptions above proved the same thing: /api/quotes?page=N is a public, unauthenticated GET that returns quotes, page, and has_next. Once you know that shape, a browser is no longer required to walk it — a plain HTTP client can page through the whole collection directly:

import json
import urllib.error
import urllib.request

def fetch_page(n):
    url = f"https://quotes.toscrape.com/api/quotes?page={n}"
    req = urllib.request.Request(url, headers={"User-Agent": "network-interception-demo/1.0"})
    with urllib.request.urlopen(req, timeout=10) as resp:
        if resp.status != 200:
            raise urllib.error.HTTPError(url, resp.status, "unexpected status", resp.headers, None)
        return json.loads(resp.read())

all_quotes, page = [], 1
while True:
    data = fetch_page(page)
    all_quotes.extend(data["quotes"])
    if not data["has_next"]:
        break
    page += 1

print("pages fetched:", page)
print("total quotes:", len(all_quotes))
print("first quote author:", all_quotes[0]["author"]["name"])
print("last quote author:", all_quotes[-1]["author"]["name"])

pages fetched: 10
total quotes: 100
first quote author: Albert Einstein
last quote author: George R.R. Martin

Ten plain HTTP requests pull the entire 100-quote collection over the same JSON contract the browser session already confirmed, with no browser process running at all. This is the actual payoff of interception: the browser's only job here was to reveal the endpoint. Once you have it, the fastest way to page through the collection is usually to stop rendering and call the endpoint directly, following the Fetch standard that both browsers and plain HTTP clients ultimately implement.

When You Still Need the Browser

Not every hidden endpoint is this cooperative. Plenty require a session cookie the server sets during an earlier page load, a CSRF or signed-request token baked into the page's JavaScript bundle, or a request body assembled from state that only exists client-side, and some ship over a WebSocket frame or a GraphQL POST instead of a REST-shaped GET like the XMLHttpRequest standard describes. In those cases the direct-replay step in the previous section doesn't apply — you can't reconstruct an auth header you never captured — but the interception step still does. page.expect_response() and the CDP Network domain read a page's traffic regardless of what authenticates it or what shape the payload takes, because they observe what the browser actually sent and received rather than assuming a particular request format ahead of time. On those sites, keep the browser in the loop for every page: let it mint the session, drive the navigation, and hand you each response as it arrives.

For a worked example of the same render-then-read-the-network pattern applied to a full site rather than a practice target, see the TikTok scraping guide, which captures comment and post XHR the same way while scrolling a real feed.

Interception is a discovery step as much as an extraction one. The first time you touch an unfamiliar site, open its DevTools Network panel, filter by Fetch/XHR, and watch what fires as you interact with the page — that manual pass is what tells you which endpoint to bind page.expect_response() to before you ever write the script. Once the endpoint is known, everything above — the response listener, the raw CDP session, and the direct replay — is that same discovery encoded as code that runs itself.

Sign up at app.scrapeless.com for free Scraping Browser runtime, or see the Scraping Browser product page and pricing for scaled runs.

FAQ

Q: What is network request interception in web scraping?
It's reading the XHR/fetch traffic a page's own JavaScript generates — usually a JSON API call — instead of waiting for that traffic to render into HTML and then parsing the rendered markup back out.

Q: Is intercepting a site's own API calls legal?
Reading responses your browser already receives while visiting a public page carries different considerations than accessing content behind authentication or reaching non-public data. Scope any workflow to public pages, respect the target's terms of service and robots directives, and keep request volume bounded — treat interception as a way to read traffic more precisely, not as license to ignore access rules.

Q: Do you still need a browser once you know the hidden endpoint?
Only if the endpoint requires something the browser supplies — a session cookie, a signed token, JavaScript-computed state. A public, unauthenticated endpoint like the one in this guide can be replayed with a plain HTTP client, as the direct-replay example shows.

Q: What's the difference between page.expect_response() and listening to the raw CDP Network domain?
page.expect_response() is Playwright's higher-level wrapper: bind it to a URL pattern, trigger the action, get a parsed Response object back. The CDP Network domain is the protocol underneath it — Network.responseReceived and Network.getResponseBody — useful when you're not using a Playwright binding at all, or need protocol-level detail a specific client library doesn't expose.

Q: Does this work on endpoints that return something other than JSON?
Both interception patterns read whatever bytes the response carries — response.text() or response.body() in Playwright, the raw body field from Network.getResponseBody in CDP — so HTML fragments, XML, or any other payload come through the same way. JSON is simply the common case for a page's own internal API.

Q: Can page.expect_response() catch requests fired before the page you're watching loads?
No — it has to be listening before the triggering action runs, which is why it wraps the specific page.goto() or interaction that causes the request, rather than being attached after the fact.

Q: Does interception need the Scrapeless Scraping Browser specifically, or does it work with any CDP-reachable Chromium?
The technique itself is generic CDP behavior and works against any Chromium you can reach over connect_over_cdp, local or remote. Running it on Scrapeless Scraping Browser adds anti-detection Chromium and residential egress, which matters when the target you're intercepting fingerprints aggressively enough that a plain local browser wouldn't get past rendering to produce the traffic in the first place.

Q: What happens if the site changes its endpoint or response shape?
The interception code keeps working as long as the URL pattern still matches; a renamed field or restructured payload breaks the code reading data['quotes'], the same way a CSS selector breaks when a class name changes. Neither approach is immune to a redesign — reading the API just means you're tracking a JSON contract instead of a markup structure, which tends to change less often.