MechanicalSoup: Automate Form-Driven Scraping in Python

TL;DR:

• MechanicalSoup automates form-driven sites by pairing requests with BeautifulSoup. One StatefulBrowser object opens a page, selects a form, fills its fields, submits it, and parses the response — no browser process and no JavaScript engine.
• The browser keeps cookies and session state across requests for you. After a login or a Set-Cookie, the same StatefulBrowser sends the stored cookie back on every later request, so multi-step flows behave like a real session.
• select_form plus dictionary-style field assignment is the whole API surface. You target a form by CSS selector, set browser["fieldname"] = value, and call submit_selected() — the library handles the encoding and the redirect.
• MechanicalSoup reads only the HTML the server returns — it does not execute JavaScript. A page that builds its content client-side comes back empty, because there is no DOM render step between the HTTP response and BeautifulSoup.
• When a page needs rendering or trips a bot challenge, hand the fetch to Scrapeless Scraping Browser and keep parsing with BeautifulSoup. The Scrapeless SDK mints a cloud session, Playwright drives it over CDP to run the JavaScript, and the rendered HTML flows straight back into the same soup.select(...) selectors.
• Free to start. New Scrapeless accounts include free Scraping Browser runtime — sign up at app.scrapeless.com.

Introduction: forms are still where most scraping work hides

A large share of useful data sits behind a form — a search box, a login, a filter panel, a multi-page wizard. MechanicalSoup exists for exactly that shape of site. It wraps the requests HTTP library and BeautifulSoup into a single stateful browser object: it fetches a page, lets you fill and submit HTML forms on it, follows the redirect, and parses whatever comes back. No Selenium, no Chromium, no driver binary.

The minimalism keeps it fast, and it also draws a hard line around what the library can reach. MechanicalSoup speaks HTTP semantics and parses HTML. It never runs the page's JavaScript, so anything rendered client-side — an infinite-scroll feed, a React result list, a search box that fetches results over XHR — comes back as the empty shell the server first sent. It stays quick on a server-rendered form and goes blind on a client-rendered one.

This guide walks through a real MechanicalSoup workflow end to end — install, open a page, fill and submit a form, carry cookies across a session, and scrape the results — then shows the honest limit. When a target renders in the browser or sits behind an active anti-bot challenge, the fetch moves to Scrapeless Scraping Browser over the Chrome DevTools Protocol, while your BeautifulSoup parsing code stays exactly as it was.

What You Can Do With It

• Submit login forms and stay authenticated. Fill the username and password fields, submit, and the StatefulBrowser keeps the session cookies for every page after it.
• Drive search and filter forms. Set a query field, submit, and parse the result rows the server returns — the classic search-and-scrape loop.
• Walk multi-page flows. Follow links and submit successive forms in one browser object, with the cookie jar and referer carried automatically.
• Read server-rendered tables and lists. Anything present in the raw HTML — pricing tables, listings, directory pages — is one soup.select() away.
• Script repetitive submissions. Re-run the same form with different field values to sweep a catalog of queries without touching a real browser.

Why Scrapeless Scraping Browser

Scrapeless Scraping Browser is a customizable, anti-detection cloud browser designed for web crawlers and AI agents. For the pages MechanicalSoup can't reach on its own, it brings:

• Cloud-side JavaScript rendering — the page's scripts run on the remote browser, so client-built content exists in the HTML by the time you parse it.
• Residential proxies in 195+ countries — pin egress with proxy_country so geo-gated pages and region-locked forms serve the content they would to a local visitor.
• Anti-detection fingerprinting — the session presents as a real browser, so the form or result page renders instead of returning a challenge interstitial.
• Session persistence — cookies and auth state stay warm across navigations inside one session, the same property MechanicalSoup gives you locally.
• A standard CDP endpoint — browser_ws_endpoint is an ordinary WebSocket URL, so Playwright (or any CDP client) attaches with one call and your parsing code stays unchanged.

Get your API key on the free plan at app.scrapeless.com.

Prerequisites

• Python 3.10 or newer
• A Scrapeless account and API key — sign up at app.scrapeless.com (only needed for the cloud-browser section)
• Basic familiarity with CSS selectors and the terminal

Install

MechanicalSoup is a single package that pulls in requests and beautifulsoup4 as dependencies:

pip install mechanicalsoup

Confirm the install and the version:

python -c "import mechanicalsoup; print(mechanicalsoup.__version__)"
# 1.4.0

Configure: open a page with a StatefulBrowser

Everything in MechanicalSoup runs through a StatefulBrowser. It holds the current page, the session, and the cookie jar. Set a user agent at construction so requests carry a sensible identity:

import mechanicalsoup

browser = mechanicalsoup.StatefulBrowser(
    user_agent="Mozilla/5.0 (compatible; data-collector)"
)

browser.open("https://httpbingo.org/forms/post")
# browser.page is now a BeautifulSoup object you can query directly

browser.open() returns the requests response; browser.page is the parsed BeautifulSoup tree for the page you just loaded.

Basic implementation: fill and submit a form

The core loop is three calls — select the form, assign its fields, submit. select_form takes a CSS selector; field assignment is dictionary-style on the browser; submit_selected() posts the form and follows the redirect.

import mechanicalsoup

browser = mechanicalsoup.StatefulBrowser(
    user_agent="Mozilla/5.0 (compatible; data-collector)"
)
browser.open("https://httpbingo.org/forms/post")

# Target the form by its action, then fill fields by name
browser.select_form('form[action="/post"]')
browser["custname"] = "Ada Lovelace"
browser["custtel"] = "555-0100"
browser["custemail"] = "ada@example.com"
browser["size"] = "medium"             # radio button
browser["topping"] = ["bacon", "cheese"]  # multi-value checkboxes
browser["comments"] = "Leave at door"

response = browser.submit_selected()
print(response.status_code)
data = response.json()
print(data["url"])
print(data["form"])

The httpbin endpoint echoes the parsed form body, which confirms exactly what MechanicalSoup sent:

{
  "url": "https://httpbingo.org/post",
  "form": {
    "comments": "Leave at door",
    "custemail": "ada@example.com",
    "custname": "Ada Lovelace",
    "custtel": "555-0100",
    "delivery": "",
    "size": "medium",
    "topping": ["bacon", "cheese"]
  }
}
// Values reflect the real submission; the empty "delivery" is an unset field on the form.

Radio buttons take a single string, checkbox groups take a list, and any field left unset is sent empty — the same encoding a browser would produce.

Advanced patterns

Carry cookies across a session

A StatefulBrowser reuses one requests.Session, so any cookie the server sets persists to later requests automatically. That is what makes logins and multi-step flows work:

import mechanicalsoup

browser = mechanicalsoup.StatefulBrowser()

# The server sets a cookie on this request
browser.open("https://httpbingo.org/cookies/set?session_id=abc123")
print(browser.session.cookies.get_dict())
# {'session_id': 'abc123'}

# A later request on the same browser sends the stored cookie back
echo = browser.open("https://httpbingo.org/cookies")
print(echo.json())
# {'cookies': {'session_id': 'abc123'}}

For a real login, submit the login form first, then keep using the same browser object — the auth cookie rides along on every subsequent page.

Submit a search form and scrape the results

A GET search form is the same pattern: set the query field, submit, parse the result rows out of browser.page.

import mechanicalsoup

browser = mechanicalsoup.StatefulBrowser(
    user_agent="Mozilla/5.0 (compatible; data-collector)"
)
browser.open("https://www.scrapethissite.com/pages/forms/")

browser.select_form('form[action="/pages/forms/"]')
browser["q"] = "boston"
browser.submit_selected()
print(browser.url)  # https://www.scrapethissite.com/pages/forms/?q=boston

rows = browser.page.select("table.table tr.team")
print(f"{len(rows)} rows")
for row in rows[:3]:
    name = row.select_one(".name").get_text(strip=True)
    year = row.select_one(".year").get_text(strip=True)
    wins = row.select_one(".wins").get_text(strip=True)
    print(name, year, "wins:", wins)

Because the result page is server-rendered, the rows are in the HTML MechanicalSoup already holds — no second fetch needed.

Get your API key on the free plan: app.scrapeless.com

Where MechanicalSoup stops: pages that render in the browser

MechanicalSoup hands BeautifulSoup whatever the server returned over HTTP — nothing more. When a page builds its content with client-side JavaScript, that raw HTML is an empty shell, and the selectors find nothing:

import mechanicalsoup

browser = mechanicalsoup.StatefulBrowser()
browser.open("https://quotes.toscrape.com/js/")
quotes = browser.page.select(".quote .text")
print("MechanicalSoup quotes found:", len(quotes))
# MechanicalSoup quotes found: 0

Zero. The /js/ variant of that page injects its quotes with JavaScript after load, so there is nothing in the server's HTML for BeautifulSoup to match. The same wall appears in front of pages that gate on an anti-bot challenge or only serve content to a residential IP — none of which an HTTP-only client can clear.

The fix keeps everything you already wrote. Let Scrapeless Scraping Browser do the rendering: the SDK mints a cloud session, Playwright attaches to it over CDP and runs the page's JavaScript, and you pass the rendered HTML straight into the same BeautifulSoup selectors.

Install the SDK and a Playwright client, then fetch the browser binary Playwright drives:

pip install scrapeless playwright beautifulsoup4
python -m playwright install chromium

Set your key in the environment — never hardcode it:

export SCRAPELESS_API_KEY="your_api_token_here"

Now render the page cloud-side and parse the result locally:

import os
from scrapeless import Scrapeless
from scrapeless.types import ICreateBrowser
from playwright.sync_api import sync_playwright
from bs4 import BeautifulSoup

client = Scrapeless()  # reads SCRAPELESS_API_KEY from the environment

# Mint a cloud session; pin US residential egress for geo-bound pages
session = client.browser.create(ICreateBrowser(
    session_name="mechanicalsoup-guide",
    session_ttl=180,
    proxy_country="US",
))

with sync_playwright() as p:
    # browser_ws_endpoint is a standard wss:// CDP URL
    browser = p.chromium.connect_over_cdp(session.browser_ws_endpoint)
    page = browser.contexts[0].pages[0]
    page.goto("https://quotes.toscrape.com/js/", wait_until="domcontentloaded")
    page.wait_for_selector(".quote .text")
    html = page.content()
    browser.close()

# Same BeautifulSoup parsing you used with MechanicalSoup
soup = BeautifulSoup(html, "html.parser")
quotes = soup.select(".quote .text")
print("Scrapeless + BeautifulSoup quotes found:", len(quotes))
print(quotes[0].get_text())
# Scrapeless + BeautifulSoup quotes found: 10
# “The world as we have created it is a process of our thinking. …”

The page that returned 0 rows to MechanicalSoup returns 10 through the cloud browser, because the JavaScript actually ran before the HTML was read. The rendering and egress move cloud-side; the parsing layer — soup.select(...) — is identical. For a library-native take on the same escalation, the Scrapling cloud-browser guide routes an adaptive-selector fetcher through the same browser_ws_endpoint.

Troubleshooting

Conclusion: keep the parser, swap the fetch

MechanicalSoup is the right tool for the large set of sites that are still plain HTML and forms: open a page, select_form, assign fields, submit_selected(), and read the rows out of BeautifulSoup. The cookie jar makes logins and multi-step flows work without extra code. Its one hard boundary is JavaScript — it reads HTML, it does not render it. When a target builds itself in the browser or gates behind an anti-bot wall, the cleanest fix is to change only the fetch: mint a Scrapeless session, render the page over CDP, and feed the resulting HTML into the same selectors. When the page needs a full headless browser with its own proxy egress, the Puppeteer proxy guide covers the same cloud-side pattern, and the Scraping Browser docs document the full CDP surface. Pin US egress for geo-bound pages, reuse one session across steps, and treat absent fields as nullable.

Ready to Build Your AI-Powered Data Pipeline?

Join our community to claim a free plan and connect with developers building form-automation and rendering pipelines: Discord · Telegram.

Sign up at app.scrapeless.com for free Scraping Browser runtime and adapt the patterns above to the forms, logins, and rendered pages your workflow needs. See pricing for scale.

FAQ

Q: Does MechanicalSoup run JavaScript?
No. MechanicalSoup wraps requests and BeautifulSoup, so it only sees the HTML the server returns. Pages that build their content client-side come back empty; render those through a cloud browser and parse the resulting HTML with the same BeautifulSoup selectors.

Q: How does MechanicalSoup handle logins and sessions?
A single StatefulBrowser reuses one requests.Session, so any cookie the server sets persists to every later request automatically. Submit the login form once, then keep using the same browser object and the auth cookie rides along.

Q: How do I select a specific form on a page?
Pass a CSS selector to select_form, for example browser.select_form('form[action="/post"]'). If no form matches you get a LinkNotFoundError — print browser.page.select("form") to see the real attributes and target one of them.

Q: Is scraping a site with MechanicalSoup legal?
Scraping publicly visible data is generally permissible, but the rules vary by jurisdiction and by the site's terms of service. Review the target's ToS, respect robots directives, avoid personal or restricted data, and consult counsel for anything ambiguous.

Q: Do I need a proxy with MechanicalSoup?
For open, server-rendered pages, often not. For pages that gate by region or only serve content to residential IPs, route the fetch through the Scrapeless Scraping Browser and pin proxy_country so the request egresses from an IP the site trusts.

Q: Can I keep my BeautifulSoup code when I move to the cloud browser?
Yes. The cloud browser only replaces the fetch step — it hands back rendered HTML, which you parse with the same soup.select(...) calls you used with MechanicalSoup. The parsing layer does not change.