Pydantic AI + Scrapeless: Give Your Agent Live Web Tools Over MCP

TL;DR:

• Pydantic AI connects to the Scrapeless MCP Server over streamable HTTP and gives an agent 21 live web tools, from scrape_markdown to a full browser-automation set.
• The connection uses three classes from pydantic_ai.mcp: a StreamableHttpTransport, a FastMCPClient, and an MCPToolset you attach to an Agent.
• The handshake, the tool list, and a real scrape_markdown call all run with no model-provider key; only the final agent.run generation needs one.
• defer_model_check=True lets the Agent construct before a model key exists, so you can wire and inspect the toolset first.
• One scrape_markdown call returns the target page as clean Markdown, ready to hand back to the model as context.
• Start on the Scrapeless free plan and connect your first agent.

Pydantic AI gives an agent structure: typed outputs, validated tool arguments, and a clean way to compose tools. What it does not give the agent is a way to reach the live web. That gap is exactly what the Model Context Protocol closes. Point Pydantic AI at an MCP server and every tool that server exposes becomes a tool your agent can call, with the argument schemas validated the same way the rest of your Pydantic AI code is.

This guide connects Pydantic AI to the Scrapeless MCP Server, lists the tools it serves, calls one for real, and attaches the whole set to an Agent — all verified against the live server. The only step that needs a model-provider key is the generation call at the end, and this post is explicit about where that line falls.

Why Scrapeless MCP

The Scrapeless MCP Server exposes web-scraping and browser tools that an agent can call directly, so you do not build or host the scraping layer yourself. A single connection serves 21 tools: scrape_markdown and scrape_html for page content, google_search and google_trends for search data, scrape_screenshot for captures, and a full browser_* set that drives a cloud browser for click, type, scroll, and navigation. The Scrapeless MCP Server post covers the server itself; this guide is about wiring it into Pydantic AI.

Because the tools run on Scrapeless infrastructure, the agent gets rendered pages and search results without a local browser or proxy pool. The browser_* tools drive the Scrapeless cloud browser, so an agent can navigate an interactive page and read what renders.

Prerequisites

• Python 3.10 or later.
• A Scrapeless API key from the dashboard, exported as SCRAPELESS_API_KEY.
• A model-provider key (such as OPENAI_API_KEY) only for the final generation step. The handshake, tool list, and tool calls do not need one.

Install

Install Pydantic AI with the MCP extra, which pulls in the MCP client classes.

pip install "pydantic-ai-slim[mcp]"

Set your Scrapeless key in the shell. Use the real key at run time and keep the placeholder out of your source.

export SCRAPELESS_API_KEY="sk_your_key_here"

Connect and List the Tools

The connection is three objects. A StreamableHttpTransport names the endpoint and carries the API key in the x-api-token header, a FastMCPClient speaks the protocol over that transport, and an MCPToolset wraps the client so Pydantic AI can use it. Entering the toolset's async context runs the handshake; list_tools returns what the server serves.

import asyncio
import os

from pydantic_ai.mcp import FastMCPClient, MCPToolset, StreamableHttpTransport

transport = StreamableHttpTransport(
    url="https://api.scrapeless.com/mcp",
    headers={"x-api-token": os.environ["SCRAPELESS_API_KEY"]},
)
scrapeless = MCPToolset(FastMCPClient(transport))


async def main() -> None:
    async with scrapeless:
        tools = await scrapeless.list_tools()
        names = sorted(t.name for t in tools)
        print("tool count:", len(names))
        print("tools:", ", ".join(names))


asyncio.run(main())

The live server returns 21 tools, and no model-provider key was set to get here.

tool count: 21
tools: browser_click, browser_close, browser_create, browser_get_html, browser_get_text, browser_go_back, browser_go_forward, browser_goto, browser_press_key, browser_screenshot, browser_scroll, browser_scroll_to, browser_snapshot, browser_type, browser_wait, browser_wait_for, google_search, google_trends, scrape_html, scrape_markdown, scrape_screenshot

The tool names are flat, with no server prefix, so scrape_markdown is addressable by exactly that name. The transport and message layer follow the Model Context Protocol specification, which itself rides on the JSON-RPC 2.0 specification.

Call a Tool Directly

Before handing the tools to an agent, call one yourself to see what it returns. direct_call_tool invokes a tool by name with its arguments, which is the quickest way to confirm a tool works and inspect its output.

import asyncio
import os

from pydantic_ai.mcp import FastMCPClient, MCPToolset, StreamableHttpTransport

transport = StreamableHttpTransport(
    url="https://api.scrapeless.com/mcp",
    headers={"x-api-token": os.environ["SCRAPELESS_API_KEY"]},
)
scrapeless = MCPToolset(FastMCPClient(transport))


async def main() -> None:
    async with scrapeless:
        result = await scrapeless.direct_call_tool("scrape_markdown", {"url": "https://quotes.toscrape.com/"})
        markdown = result if isinstance(result, str) else str(result)
        print("markdown chars:", len(markdown))
        print("contains a quote:", "The world as we have created it" in markdown)


asyncio.run(main())

The call returns the page as Markdown, and the content check confirms a real quote from the target page is present.

markdown chars: 4308
contains a quote: True

This is the shape your agent gets back: clean Markdown it can reason over, rather than raw HTML it has to strip. The Pydantic AI MCP client documentation covers the toolset methods in full.

Attach the Tools to an Agent

Attaching is one argument: pass the toolset to the Agent in toolsets. Because a normal Agent construction validates the model right away, defer_model_check=True lets it build before a model key is set, so you can wire and inspect the toolset first.

import asyncio
import os

from pydantic_ai import Agent
from pydantic_ai.mcp import FastMCPClient, MCPToolset, StreamableHttpTransport

transport = StreamableHttpTransport(
    url="https://api.scrapeless.com/mcp",
    headers={"x-api-token": os.environ["SCRAPELESS_API_KEY"]},
)
scrapeless = MCPToolset(FastMCPClient(transport))

# defer_model_check lets the agent construct before the model key is set,
# so the toolset can be wired and inspected first.
agent = Agent("openai:gpt-4o", toolsets=[scrapeless], defer_model_check=True)


async def main() -> None:
    async with scrapeless:
        names = sorted(t.name for t in await scrapeless.list_tools())
    web = [n for n in names if n.startswith(("scrape_", "google_"))]
    print("agent wired with", len(names), "Scrapeless tools")
    print("web tools:", web)


asyncio.run(main())

The agent now carries every Scrapeless tool, and the web-scraping subset is the part most tutorials reach for first.

agent wired with 21 Scrapeless tools
web tools: ['google_search', 'google_trends', 'scrape_html', 'scrape_markdown', 'scrape_screenshot']

To hand the agent only a few tools instead of all 21, MCPToolset exposes filtered and renamed, so you can scope an agent to scrape_markdown and google_search alone rather than the full browser set.

Run a Prompt

With the toolset attached, the agent decides when to call a tool. This is the one step that needs a model-provider key.

Note: agent.run needs a model-provider key such as OPENAI_API_KEY. Everything above — the handshake, the 21-tool list, the scrape_markdown call, and the attachment — runs without it. Only this generation call is a prerequisite gap; it is shown here with the exact shape it takes, not as a captured result.

async def run_prompt() -> None:
    async with agent:
        result = await agent.run(
            "Use scrape_markdown to fetch https://quotes.toscrape.com/ "
            "and list the first three quotes with their authors."
        )
    print(result.output)


asyncio.run(run_prompt())

At run time the model reads the prompt, calls scrape_markdown with the URL, receives the Markdown the earlier call already demonstrated, and writes the answer. The tool layer is identical whether you call it directly or let the model call it.

Conclusion

Pydantic AI plus the Scrapeless MCP Server is a short path from a bare agent to one that reads the live web. Three classes make the connection, list_tools shows the 21 tools, direct_call_tool proves one works, and one toolsets argument attaches them all. Only the generation step needs a model key, which keeps the whole integration explorable before you commit a provider. Start from the scripts above, scope the toolset to the tools your agent needs, and let the model do the rest.

Create a free Scrapeless account to get an API key, and check Scrapeless pricing when you plan a recurring agent.

FAQ

Q: Does Pydantic AI need a model key to list MCP tools?

No. The handshake, list_tools, and direct_call_tool all run with only the Scrapeless API key. A model-provider key is required solely for agent.run, when the model itself decides which tools to call, so you can explore and test the entire tool surface before committing a provider.

Q: What is the difference between FastMCPClient and MCPToolset?

FastMCPClient speaks the MCP protocol over a transport and exposes low-level operations like list_tools. MCPToolset wraps that client so Pydantic AI can treat the server's tools as agent tools, and it adds toolset features such as filtered and renamed. You attach the MCPToolset, not the client, to an Agent.

Q: How do I connect to a stdio MCP server instead of HTTP?

Swap the transport. Use StdioTransport with the server command instead of StreamableHttpTransport with a URL, then wrap it in the same FastMCPClient and MCPToolset. The Scrapeless MCP Server is a hosted HTTP endpoint, so this guide uses StreamableHttpTransport.

Q: Why use defer_model_check when constructing the Agent?

Constructing an Agent normally validates the model provider immediately, which fails if no key is set. defer_model_check=True postpones that check to run time, so you can build the agent, wire the toolset, and inspect the available tools without a model key present.

Q: How do I give an agent only some of the tools?

Use MCPToolset.filtered to expose a subset, or renamed to change how tools appear to the model. Scoping an agent to scrape_markdown and google_search alone is safer than handing it all 21 tools when the task only needs content and search.

Q: What does scrape_markdown return?

It returns the target page rendered as Markdown, which in the verified call was 4,308 characters for the quotes page and contained the page's real text. Markdown is easier for a model to reason over than raw HTML, so it is a good default for feeding page content back into a prompt.

Q: Is scraping through the tools bound by the target's rules?

Yes. The tools fetch public pages, and you remain responsible for honoring each target's terms and its Robots Exclusion Protocol directives. Keep the volume bounded and the data public, and scope the agent to the tools the task actually needs.