LlamaIndex + Scrapeless: Feed Your Index Live Web Pages

A retrieval index is only as current as the documents you put in it. LlamaIndex handles the chunking, embedding, and retrieval well; the part that quietly breaks is the step before all of that, where live web pages have to become clean text.

Connecting LlamaIndex to the Scrapeless MCP Server covers that step. The MCP tools return rendered pages as markdown, LlamaIndex wraps them as Document objects, and the rest of your ingestion pipeline carries on unchanged. This guide runs the connection, the tool discovery, a real page fetch, and the document-to-node split end to end.

What This Setup Gives Your Index

Your ingestion code gets 21 callable tools from one connection, and they arrive as native LlamaIndex tools rather than something you wrap yourself.

The groups matter for ingestion:

• Page retrieval — scrape_markdown returns a page already converted to markdown, which is the format a splitter and an embedding model both handle best. scrape_html and scrape_screenshot return the other two shapes.
• Search — google_search and google_trends let an ingestion job discover URLs rather than being handed a fixed list.
• Live browser control — sixteen browser_* tools for pages that need interaction before the content exists.

The rendering, proxy routing, and access handling all happen server-side, so the ingestion process stays a plain Python job with no browser to install.

Why the Scrapeless MCP Server

The Model Context Protocol specification defines how a client discovers tools and their argument schemas from a server, which is what makes this different from writing a fetch helper: the tool list and each tool's parameters arrive from the server rather than being hard-coded in your project. Calls travel as JSON-RPC 2.0 messages.

Scrapeless hosts the endpoint, so there is no server process to run alongside your indexer. Authentication is one header. The browser_* group is backed by the Scrapeless Scraping Browser, and per-tool parameters are documented in the Scrapeless documentation.

Prerequisites

• Python 3.10 or later. Both llama-index-core and llama-index-tools-mcp currently declare >=3.10,<4.0.
• A Scrapeless API key from the dashboard.
• For the agent section only: an LLM integration package such as llama-index-llms-openai plus that provider's key.

Note: Everything through the ingestion section below was executed with a Scrapeless key and no model-provider key. The MCP connection, tool discovery, argument schemas, the live tool call, and the Document-to-node split all ran. The agent step at the end is a prerequisite gap — constructing a FunctionAgent raises ImportError: llama-index-llms-openai package not found without an LLM integration installed, so that block is shown as the code you add rather than as captured output.

Install

pip install "llama-index-tools-mcp==0.4.8"

That package brings llama-index-core and the mcp client with it. Set the key in your shell:

export SCRAPELESS_API_KEY="your_api_key_here"

Connect and List the Tools

BasicMCPClient takes the endpoint URL and the headers; McpToolSpec turns the server's tool list into LlamaIndex tools:

import asyncio, os
from llama_index.tools.mcp import BasicMCPClient, McpToolSpec

async def main():
    client = BasicMCPClient(
        "https://api.scrapeless.com/mcp",
        headers={"x-api-token": os.environ["SCRAPELESS_API_KEY"]},
    )
    spec = McpToolSpec(client=client)
    tools = await spec.to_tool_list_async()
    print("tool count:", len(tools))
    print("sample names:", sorted(t.metadata.name for t in tools)[:6])

asyncio.run(main())

tool count: 21
sample names: ['browser_click', 'browser_close', 'browser_create', 'browser_get_html', 'browser_get_text', 'browser_go_back']

The API is async throughout, which is why the example runs inside asyncio.run. Tool names arrive flat, with no server prefix or dotted namespace, so scrape_markdown is the literal name your code and your agent will use.

Take Only the Tools the Job Needs

An ingestion job rarely needs browser session control. McpToolSpec accepts allowed_tools and returns only those, which keeps the surface small and the schemas easy to read:

import asyncio, os
from llama_index.tools.mcp import BasicMCPClient, McpToolSpec

async def main():
    client = BasicMCPClient(
        "https://api.scrapeless.com/mcp",
        headers={"x-api-token": os.environ["SCRAPELESS_API_KEY"]},
    )
    spec = McpToolSpec(client=client, allowed_tools=["scrape_markdown"])
    tools = await spec.to_tool_list_async()

    tool = tools[0]
    print("filtered count:", len(tools))
    print("name:", tool.metadata.name)
    print("fn_schema fields:", list(tool.metadata.fn_schema.model_fields))

asyncio.run(main())

filtered count: 1
name: scrape_markdown
fn_schema fields: ['url']

The schema comes from the server, so it is the real contract rather than an assumption: scrape_markdown takes a single url. LlamaIndex exposes it as fn_schema, the same Pydantic model an agent would use to build its call.

Ready to point this at your own sources? Create a free Scrapeless account and connect with the key from your dashboard.

Turn Live Pages Into Nodes

This is the part that matters for retrieval. Call the tool directly, wrap each result as a Document with its source in the metadata, then split into nodes:

import asyncio, os
from llama_index.tools.mcp import BasicMCPClient, McpToolSpec
from llama_index.core import Document
from llama_index.core.node_parser import SentenceSplitter

async def main():
    client = BasicMCPClient(
        "https://api.scrapeless.com/mcp",
        headers={"x-api-token": os.environ["SCRAPELESS_API_KEY"]},
    )
    spec = McpToolSpec(client=client, allowed_tools=["scrape_markdown"])
    tool = (await spec.to_tool_list_async())[0]

    urls = [
        "https://quotes.toscrape.com/js/",
        "https://quotes.toscrape.com/page/2/",
    ]

    docs = []
    for url in urls:
        markdown = str(await tool.acall(url=url))
        docs.append(Document(text=markdown, metadata={"source": url}))
    print(f"documents: {len(docs)}")

    splitter = SentenceSplitter(chunk_size=256, chunk_overlap=32)
    nodes = splitter.get_nodes_from_documents(docs)
    print(f"nodes after splitting: {len(nodes)}")
    print(f"first node source: {nodes[0].metadata['source']}")
    print(f"first node chars: {len(nodes[0].get_content())}")

asyncio.run(main())

documents: 2
nodes after splitting: 14
first node source: https://quotes.toscrape.com/js/
first node chars: 571

Several things in that output are worth reading carefully.

The first URL is a client-rendered page — its content is written into the DOM by a script — and it still produced usable markdown, because the render happened server-side before conversion. A plain HTTP fetch of that same URL returns markup with none of the content in it.

chunk_size=256 counts tokens, not characters, which is why the first node is 571 characters long. Sizing a splitter in characters is a common way to end up with chunks that overflow an embedding model's context.

The metadata={"source": url} on each Document survives the split and lands on every node derived from it. That is what lets a retrieval result cite where it came from, and it is far easier to attach here than to reconstruct later.

Markdown is the right intermediate format for this: headings and links survive, while scripts, styling, and layout markup do not, so the embedding budget goes to content.

Give the Tools to an Agent

Once the tools are in hand, an agent can decide which to call rather than following a fixed URL list. This step needs an LLM integration package and that provider's key.

Note: This block is a prerequisite gap. Without an LLM integration installed, constructing the agent raises ImportError: llama-index-llms-openai package not found, please run pip install llama-index-llms-openai, so no output is shown for it.

from llama_index.core.agent.workflow import FunctionAgent
from llama_index.llms.openai import OpenAI

agent = FunctionAgent(
    tools=tools,
    llm=OpenAI(model="gpt-4.1-mini"),
    system_prompt="Research public pages and return clean notes with sources.",
)

response = await agent.run("Summarise the authors quoted on quotes.toscrape.com")
print(response)

Conclusion

Connecting LlamaIndex to the Scrapeless MCP Server takes a client, a tool spec, and one header. The server supplies 21 tools with their own argument schemas, allowed_tools narrows them to what an ingestion job actually needs, and scrape_markdown returns pages in the format a splitter and an embedding model both prefer.

The habit worth carrying over is attaching the source URL as Document metadata at fetch time. It costs one dictionary, it survives the node split, and it is what turns a retrieval hit into an answer you can trace back to a page.

Start on the Scrapeless free plan to get a key, check the Scrapeless pricing when you size an ingestion run, and see the Scrapeless MCP Server overview for the full tool reference.

FAQ

Q: What is the Scrapeless MCP Server endpoint for LlamaIndex?

The hosted endpoint is https://api.scrapeless.com/mcp, reached with your key in the x-api-token header via BasicMCPClient. There is no local server process to run, because the tools are served remotely.

Q: How many tools does the Scrapeless MCP Server expose to LlamaIndex?

A live connection returns 21: sixteen browser_* session-control tools, three page-retrieval tools (scrape_markdown, scrape_html, scrape_screenshot), and two search tools (google_search, google_trends). Enumerate them at runtime rather than assuming, since a server can add tools between releases.

Q: Can I load only some MCP tools?

Yes. Pass allowed_tools=["scrape_markdown"] to McpToolSpec and the list comes back with just that tool. For ingestion this is worth doing — it keeps the schemas readable and stops an agent from opening browser sessions it does not need.

Q: Do I need an LLM key to fetch pages through MCP?

No. The connection, tool discovery, schema inspection, and direct tool.acall(...) all work with only the Scrapeless key. A model provider is needed once you hand the tools to an agent, because that is when something has to decide which tool to call.

Q: Why use markdown instead of HTML for retrieval?

Markdown keeps the structure that helps retrieval — headings, lists, links — and drops the scripts, styling, and layout markup that consume embedding context without adding meaning. scrape_html is still the right choice when you intend to run your own selectors instead of embedding the text.

Q: How do I keep track of which page a retrieved chunk came from?

Put the URL in Document(metadata={"source": url}) when you create the document. That metadata is copied onto every node the splitter derives from it, so each retrieved chunk carries its origin without any extra bookkeeping.

Q: What should I check before ingesting a site?

Review the site's terms and its /robots.txt directives, which follow the Robots Exclusion Protocol standard. Keep ingestion to public pages, work from an explicit URL list or a bounded discovery step, and record the source URL on every document so the provenance of anything the index returns stays clear.