How to Scrape Shadow DOM With Playwright and Scrapeless

TL;DR:

• Shadow DOM can display text that does not exist in a host element’s light DOM. JavaScript may create the visible content inside a shadow root after the custom element loads.
• Playwright CSS and text locators can traverse open shadow roots. The same selector fails with document.querySelector because document-level DOM queries stop at the shadow boundary.
• Closed shadow roots and XPath require different expectations. Playwright locators do not pierce closed roots, and XPath selectors do not support automatic Shadow DOM traversal.
• Shadow DOM extraction logic is independent of browser transport. Verify the workflow locally, then reuse the same locators through the Scrapeless CDP connection when a hosted browser or managed egress is required.
• Free to start. New Scrapeless accounts include free Scraping Browser runtime—sign up at app.scrapeless.com.

A Shadow DOM component can display text that is absent from the host element's ordinary light-DOM children. This guide uses Playwright to traverse a real open shadow root, shows why a document-level selector returns nothing, and separates that extraction logic from the Scrapeless Scraping Browser connection that runs the same code in a cloud browser.

What Shadow DOM Changes for a Scraper

Shadow DOM gives a custom element an encapsulated subtree. The page's main document can see the host tag, but ordinary DOM APIs do not automatically cross from that host into its shadow root.

On the public MDN demo used below, the source contains a <popup-info> host with a data-text attribute. A deferred script upgrades that tag, calls attachShadow({ mode: "open" }), and creates an inner .info span. The browser displays the help text from that span. The initial HTML does not contain the span at all.

MDN's Shadow DOM guide defines the boundary and the difference between open and closed roots. The distinction matters: an open root exposes element.shadowRoot; a closed root returns null through that API.

Why Use Playwright With Scrapeless

Playwright locators pierce open shadow roots for normal CSS and text lookup. The Playwright locator documentation states two exceptions: XPath does not pierce shadow roots, and closed-mode roots are not supported.

The traversal code does not depend on where Chromium runs. A local browser is enough for a public demo; the Scrapeless Scraping Browser becomes useful when the target needs a hosted browser, managed fingerprint, residential egress, or a persistent cloud session. Playwright reaches that browser through the connect_over_cdp API.

Prerequisites

• Python 3.10 or later.
• playwright 1.59.0 or a current compatible version.
• A local Chrome/Chromium runtime for the fully executable fallback below.
• A funded Scrapeless account and SCRAPELESS_API_KEY for the cloud connection. The verification account returned code 14500 for new browser sessions, so the cloud connection block is marked as a prerequisite gap rather than shown as successful.

Install

python -m pip install "playwright==1.59.0"

The verification host already has Google Chrome at /usr/bin/google-chrome. If your machine does not, install a Playwright-managed Chromium with the project's normal browser setup command.

Connect to the Scrapeless Scraping Browser

Note: This connection requires funded Scrapeless browser balance. The account used for final verification returned insufficient balance, please recharge first; run it with your own funded key. The traversal blocks after this section ran completely in local Chrome against the same public page.

import os
from urllib.parse import urlencode
from playwright.sync_api import sync_playwright

params = urlencode({
    "token": os.environ["SCRAPELESS_API_KEY"],
    "sessionTTL": 120,
    "proxyCountry": "US",
})
cdp_url = f"wss://browser.scrapeless.com/api/v2/browser?{params}"

with sync_playwright() as p:
    browser = p.chromium.connect_over_cdp(cdp_url)
    page = browser.contexts[0].pages[0]
    # Run the extraction steps below with this page.
    browser.close()

The Scrapeless developer docs are the current source for connection parameters. Once page exists, Shadow DOM lookup is standard Playwright.

Inspect the Host and Its Open Root

The MDN component demo is small and public:

TARGET_URL = (
    "https://mdn.github.io/web-components-examples/"
    "popup-info-box-web-component/"
)

page.goto(TARGET_URL, wait_until="domcontentloaded")
page.wait_for_selector("popup-info .info")

host = page.locator("popup-info")
print("host light-DOM text:", repr(host.inner_text()))
print("shadow mode:", host.evaluate("element => element.shadowRoot.mode"))

host light-DOM text: ''
shadow mode: open

The empty host text is not evidence that the component contains no data. It says the host has no ordinary child text. The actual .info node lives below the open shadow root created after the custom element upgrades.

Prove the Document Selector Stops at the Boundary

Run a plain browser DOM query from the document:

plain_match = page.evaluate(
    "document.querySelector('popup-info .info') !== null"
)
print("document query found inner node:", plain_match)

document query found inner node: False

The selector is valid CSS. The failure comes from scope: document.querySelector searches the document tree and does not descend into the host's shadow tree.

Let a Playwright Locator Pierce the Open Root

Playwright's ordinary locator chain crosses the open root:

info = page.locator("popup-info .info")
record = {
    "host": "popup-info",
    "shadow_mode": host.evaluate("element => element.shadowRoot.mode"),
    "text": info.inner_text(),
    "source_url": page.url,
}

print("inner matches:", info.count())
print("help text:", record["text"])
print("text chars:", len(record["text"]))

inner matches: 1
help text: Your card validation code (CVC) is an extra security feature — it is the last 3 or 4 numbers on the back of your card.
text chars: 118

That text is the component's real data-text value copied into the shadow-root span. The character count includes punctuation and spaces, so it is a useful guard against accidentally extracting an empty element or the icon's alternate text.

Ready to move the same locator to a hosted browser? Create a free Scrapeless account and keep every line after browser creation unchanged.

Complete Local Verification Script

This script runs the whole diagnostic in local Chrome. Replacing launch(...) with the CDP connection above moves it to Scrapeless without changing the extraction logic:

from playwright.sync_api import sync_playwright

TARGET_URL = (
    "https://mdn.github.io/web-components-examples/"
    "popup-info-box-web-component/"
)

with sync_playwright() as p:
    browser = p.chromium.launch(
        executable_path="/usr/bin/google-chrome",
        headless=True,
    )
    page = browser.new_page()
    page.goto(TARGET_URL, wait_until="domcontentloaded")
    page.wait_for_selector("popup-info .info")

    host = page.locator("popup-info")
    info = page.locator("popup-info .info")
    host_text = host.inner_text()
    shadow_mode = host.evaluate("element => element.shadowRoot.mode")
    plain_match = page.evaluate(
        "document.querySelector('popup-info .info') !== null"
    )
    help_text = info.inner_text()

    assert shadow_mode == "open"
    assert not plain_match
    assert "card validation code" in help_text

    print("page title:", page.title())
    print("host light-DOM text:", repr(host_text))
    print("shadow mode:", shadow_mode)
    print("document query found inner node:", plain_match)
    print("inner matches:", info.count())
    print("help text:", help_text)
    print("text chars:", len(help_text))
    browser.close()

The run proves the boundary rather than only showing a successful locator: light-DOM text is empty, the document selector is false, the root reports open, and Playwright returns one 118-character help string.

What Open and Closed Roots Mean

Open root

attachShadow({mode: "open"}) leaves host.shadowRoot available. Playwright locators can traverse it with CSS/text selectors. Nested open roots are handled the same way, though each component may load on a different schedule.

Closed root

attachShadow({mode: "closed"}) makes host.shadowRoot return null. Playwright's normal locators do not pierce it. Do not present script injection or browser instrumentation as a universal workaround: a closed root is an explicit encapsulation boundary, and changing page code can alter behavior you intended to measure.

Slotted content

A <slot> displays light-DOM children inside a shadow-tree layout. The assigned nodes still belong to the host's light DOM, so their ownership and selector behavior differ from elements created inside the shadow root. Inspect the live DOM before deciding which side owns the data.

Common Shadow DOM Failure Modes

The host exists but the inner locator times out

The custom element may not have upgraded yet, its JavaScript may have failed, or the selector may target a closed root. Wait on the inner element as the real readiness marker and inspect console errors before changing selectors.

XPath finds nothing

Use a CSS or text locator. Playwright's documented shadow traversal does not apply to XPath selectors.

A nested component returns partial text

Locate the deepest stable element instead of calling inner_text() on a large host. This avoids mixing slot text, visually hidden labels, and nested component output.

Source HTML has no inner node

That is normal when JavaScript constructs the shadow tree. Use a browser. An HTTP parser only sees the source response and cannot observe a subtree created after load.

Conclusion

Shadow DOM scraping is a scope problem before it is a selector problem. Prove which tree owns the node, wait for the custom element to upgrade, and use Playwright's CSS/text locators for open roots. Keep the closed-root boundary honest. The browser transport is separate: run locally for a public test surface, then switch browser creation to the Scrapeless CDP endpoint when the target needs a hosted anti-detection browser or managed egress.

Start with Scrapeless, review Scrapeless pricing, and read what the Chrome DevTools Protocol controls for the surrounding CDP architecture.

FAQ

Q: Can Playwright scrape an open Shadow DOM root?

Yes. Playwright's CSS and text locators pierce open shadow roots, so a locator such as page.locator("popup-info .info") can reach the inner node.

Q: Why does document.querySelector fail on the same selector?

document.querySelector searches the document tree and stops at the shadow host. You must query through the open root yourself or use a Playwright locator that performs that traversal.

Q: Can Playwright pierce a closed shadow root?

No, not through its normal locator behavior. A closed root returns null from host.shadowRoot and is a documented unsupported boundary.

Q: Does XPath work inside Shadow DOM?

No. Playwright's official locator guidance says XPath selectors do not pierce shadow roots; use CSS or text locators.

Q: How do I handle nested open shadow roots?

Chain locators toward the stable inner element and wait for that element. Playwright can cross multiple open roots, but each custom element may upgrade asynchronously.

Q: Do I need a browser for Shadow DOM scraping?

Yes when JavaScript creates the shadow tree. A plain HTTP response contains the host markup and scripts, not the live subtree produced after execution.

Q: Is Shadow DOM scraping legal?

The DOM mechanism does not decide permission. Review the target's terms and robots directives, use public data, cap request volume, and seek legal advice for sensitive or commercial collection.