Capture Dynamic DOM Changes With MutationObserver and Playwright
Web Data Collection Specialist
TL;DR:
- Dynamic pages can change the DOM after the initial HTML has loaded. A scraper must wait for the usable state instead of treating the first response as the final document.
- MutationObserver records how the DOM changes. Install the observer before the triggering action when you need added nodes, removed nodes, text updates, or attribute transitions.
- Playwright locator waits confirm when the required state is usable. Use the observer for change provenance and locators for final-state readiness.
- A bounded observer should watch only the relevant subtree and mutation types. Serialize small records, filter noisy attributes, and disconnect after the expected state appears.
- Network interception is usually better when structured response data is the real target. MutationObserver is most useful when the DOM transition itself matters.
- The same observation workflow runs locally and through Scrapeless. Moving to Scrapeless Scraping Browser changes browser creation, not the page-side observer.
- Free to start. New Scrapeless accounts include free Scraping Browser runtime—sign up at app.scrapeless.com.
Introduction: The Initial HTML Is Only the Starting State
A dynamic page can insert, reveal, replace, or remove data after its initial HTML has loaded. This guide installs a bounded MutationObserver before triggering a public Selenium fixture, captures the exact DOM changes, and compares event-driven observation with Playwright's locator waits before moving the same workflow to the Scrapeless Scraping Browser.
What MutationObserver Adds to Scraping
MutationObserver reports changes to a DOM subtree. It can watch child insertion/removal, attributes, and text changes without repeatedly scanning the page. MDN's MutationObserver reference defines the callback and observation options.
An observer does not decide when data is ready by itself. It records what changed. Playwright locators remain the clearer way to wait for a specific usable element, while observer records explain how and when that state appeared.
Why Use Playwright With Scrapeless
Playwright can install an observer in the page before the triggering click, perform the action, wait for final UI state, and return a compact event log. The Scrapeless Scraping Browser supplies a hosted Chromium session for the same page-side script.
The cloud browser uses connect_over_cdp. Only browser creation changes.
Prerequisites
- Python 3.10 or later.
playwright1.59.0 or a current compatible version.- Local Chrome/Chromium for the complete live example.
- A funded Scrapeless account and
SCRAPELESS_API_KEYfor the cloud connection. The verification account returned code 14500, so that connection is a labelled prerequisite gap.
Install
bash
python -m pip install "playwright==1.59.0"Connect to the Scrapeless Scraping Browser
Note: This block needs funded Scraping Browser balance. The final verification account returned
insufficient balance, please recharge first; the complete observer workflow below ran in local Chrome.
python
import os
from urllib.parse import urlencode
from playwright.sync_api import sync_playwright
params = urlencode({
"token": os.environ["SCRAPELESS_API_KEY"],
"sessionTTL": 120,
"proxyCountry": "US",
})
cdp_url = f"wss://browser.scrapeless.com/api/v2/browser?{params}"
with sync_playwright() as p:
browser = p.chromium.connect_over_cdp(cdp_url)
page = browser.contexts[0].pages[0]
# Install and consume the observer below.
browser.close()Use the Scrapeless developer docs for current connection parameters.
Step 1 — Inspect the Page’s Initial State
Selenium's public fixture exposes two buttons. One inserts #box0 after one second; the other reveals a hidden #revealed input after one second:
python
TARGET_URL = "https://www.selenium.dev/selenium/web/dynamic.html"
page.goto(TARGET_URL, wait_until="domcontentloaded")
print("box before:", page.locator("#box0").count())
print("input visible before:", page.locator("#revealed").is_visible())
text
box before: 0
input visible before: FalseThe fixture's initial HTML contains the hidden input but not the box. That difference is why the observer must watch both child-list and attribute mutations.
Step 2 — Install the Observer Before the Action
Store plain dictionaries rather than DOM nodes. MutationRecord exposes the mutation type, target, changed attribute, and added/removed nodes.
python
page.evaluate("""
window.capturedMutations = [];
window.scrapeObserver = new MutationObserver(records => {
for (const record of records) {
for (const node of record.addedNodes) {
if (node.nodeType === Node.ELEMENT_NODE) {
capturedMutations.push({
type: 'added',
id: node.id,
display: getComputedStyle(node).display,
});
}
}
if (record.type === 'attributes') {
capturedMutations.push({
type: 'attribute',
id: record.target.id,
name: record.attributeName,
display: getComputedStyle(record.target).display,
});
}
}
});
scrapeObserver.observe(document.body, {
childList: true,
subtree: true,
attributes: true,
attributeFilter: ['style'],
});
""")The attributeFilter keeps unrelated class, ARIA, or framework attributes out of this bounded capture.
Step 3 — Trigger the Changes and Wait for Usable State
python
page.click("#adder")
page.click("#reveal")
page.wait_for_selector("#box0")
page.locator("#revealed").wait_for(state="visible")Playwright's auto-waiting guidance explains why locators are the right readiness check. The observer log is complementary evidence, not a replacement for a concrete final-state assertion.
Step 4 — Disconnect the Observer and Read the Records
python
changes = page.evaluate("""() => {
scrapeObserver.disconnect();
return capturedMutations;
}""")
print("changes:", changes)
print("change count:", len(changes))
print("box matches:", page.locator("#box0").count())
print("input visible:", page.locator("#revealed").is_visible())
text
changes: [{'type': 'added', 'id': 'box0', 'display': 'block'}, {'type': 'attribute', 'id': 'revealed', 'name': 'style', 'display': 'inline-block'}]
change count: 2
box matches: 1
input visible: TrueDisconnecting prevents later unrelated page changes from expanding the event list. A long-lived observer should always have an explicit stop condition.
Ready to observe a hosted browser session? Create a free Scrapeless account and replace only browser creation.
Complete Runnable Capture
python
from playwright.sync_api import sync_playwright
TARGET_URL = "https://www.selenium.dev/selenium/web/dynamic.html"
with sync_playwright() as p:
browser = p.chromium.launch(
executable_path="/usr/bin/google-chrome",
headless=True,
)
page = browser.new_page()
page.goto(TARGET_URL, wait_until="domcontentloaded")
box_before = page.locator("#box0").count()
input_before = page.locator("#revealed").is_visible()
page.evaluate("""
window.capturedMutations = [];
window.scrapeObserver = new MutationObserver(records => {
for (const record of records) {
for (const node of record.addedNodes) {
if (node.nodeType === Node.ELEMENT_NODE) {
capturedMutations.push({
type: 'added',
id: node.id,
display: getComputedStyle(node).display,
});
}
}
if (record.type === 'attributes') {
capturedMutations.push({
type: 'attribute',
id: record.target.id,
name: record.attributeName,
display: getComputedStyle(record.target).display,
});
}
}
});
scrapeObserver.observe(document.body, {
childList: true,
subtree: true,
attributes: true,
attributeFilter: ['style'],
});
""")
page.click("#adder")
page.click("#reveal")
page.wait_for_selector("#box0")
page.locator("#revealed").wait_for(state="visible")
changes = page.evaluate("""() => {
scrapeObserver.disconnect();
return capturedMutations;
}""")
box_after = page.locator("#box0").count()
input_after = page.locator("#revealed").is_visible()
assert box_before == 0 and input_before is False
assert changes == [
{"type": "added", "id": "box0", "display": "block"},
{
"type": "attribute",
"id": "revealed",
"name": "style",
"display": "inline-block",
},
]
assert box_after == 1 and input_after is True
print("box before:", box_before)
print("input visible before:", input_before)
print("changes:", changes)
print("change count:", len(changes))
print("box after:", box_after)
print("input visible after:", input_after)
browser.close()The final run captures exactly one added node and one style-attribute change, then independently confirms one box and one visible input.
What You Get Back
The completed workflow returns a compact mutation log together with independent final-state checks:
json
{
"before": {
"box_matches": 0,
"input_visible": false
},
"mutations": [
{
"type": "added",
"id": "box0",
"display": "block"
},
{
"type": "attribute",
"id": "revealed",
"name": "style",
"display": "inline-block"
}
],
"after": {
"box_matches": 1,
"input_visible": true
}
}The mutation records explain how the page changed, while the locator checks confirm that the final elements are usable. Keeping both prevents an observed event from being mistaken for a completed UI transition.
Choosing the Right Dynamic-Page Signal
| Signal | Use It When | What It Returns | Main Limitation |
|---|---|---|---|
| MutationObserver | The sequence or type of DOM changes matters | Added, removed, text, or attribute mutations | Requires bounded scope and an explicit stop condition |
| Playwright locator wait | Only the final usable state matters | A ready element or confirmed state | Does not explain how the page reached that state |
| Network capture | Structured data arrives through requests | Request and response payloads | Does not describe later DOM transformations |
| Bounded polling | No stable event, request, or locator exists | Periodic state snapshots | Repeated checks add noise and page work |
A locator wait should be the default readiness primitive. Add MutationObserver when the extraction also needs change provenance, and prefer network capture when the structured response is more stable than the rendered DOM.
Common MutationObserver Failure Modes
The First Event Is Missing
The observer was installed after the action. Register it before clicking or navigating to the state transition.
The Capture Grows Too Large
Observe a smaller root, filter attributes, and store only fields you need. Never retain DOM nodes in a long-running capture.
Attribute Changes Repeat
Frameworks may write the same attribute several times. Decide whether you need every transition or only the last value, then deduplicate outside the callback.
The Observer Never Stops
Disconnect it once the expected records arrive or when the task's deadline expires. Open-ended observation creates unbounded memory growth.
Conclusion
MutationObserver gives you provenance for client-side DOM changes, while Playwright locators confirm the usable final state. Install the observer before the action, serialize small records, watch only the relevant mutation classes, and disconnect promptly. Moving the workflow to Scrapeless changes browser creation without changing the event capture.
Ready to Capture Dynamic Page State?
Join developers building browser-based extraction workflows in the Scrapeless community: Discord · Telegram.
Start with Scrapeless, review Scrapeless pricing, and compare this workflow with network request interception when the page exposes structured responses.
FAQ
Q: Is MutationObserver better than Playwright waits?
They answer different questions. The observer records what changed; a locator wait confirms that the element or state you need is ready.
Q: When should I install the observer?
Before the action that can trigger the mutation. Installing it afterward can miss the only event you needed.
Q: Can MutationObserver see network responses?
No. It sees DOM mutations. Use request/response events when you need network payloads.
Q: Why filter observed attributes?
Frameworks change many attributes. An attributeFilter keeps the capture bounded and relevant.
Q: Should I store MutationRecord objects directly?
No. Convert them to small plain objects because records reference DOM nodes and are not convenient serialized output.
Q: How do I stop an observer?
Call disconnect() after the expected records arrive or at a bounded task deadline.
Q: Is observing a public page always permitted?
No. Review the site's terms, collect only necessary public data, cap the session and request volume, and follow robots policy where applicable.
At Scrapeless, we only access publicly available data while strictly complying with applicable laws, regulations, and website privacy policies. The content in this blog is for demonstration purposes only and does not involve any illegal or infringing activities. We make no guarantees and disclaim all liability for the use of information from this blog or third-party links. Before engaging in any scraping activities, consult your legal advisor and review the target website's terms of service or obtain the necessary permissions.
