Gemini Web Scraping: Schema-First Extraction in Python

TL;DR:

• Gemini's cheap flash tier is built for exactly this job. Feed it a rendered page and a strict schema and it returns validated records — a live run in this guide pulled all 20 products off a 50,449-character catalogue page for 12,904 tokens on a flash-lite model.
• Schema first, prompt second. The current Gemini API takes a Pydantic-derived JSON schema in the request itself, so field names and types are enforced by the API rather than begged for in prose.
• What Gemini does not solve is getting the page. The model cannot render JavaScript, hold sessions, or clear access challenges at your chosen volume — a fetch layer does that, and this guide's fetch is one POST per page through the Scrapeless Universal Scraping API.
• Token cost tracks page size, so fetch quality is cost control. Sending trimmed, rendered content instead of broken or bloated fetches is the difference between cents and real money at scale.
• No Google key yet? The same extraction runs through OpenRouter. This guide executed it live on google/gemini-2.5-flash-lite and shows the captured output.
• Free to start on the fetch side. Create your Scrapeless API key at app.scrapeless.com.

Can Gemini scrape websites?

Gemini extracts; it does not collect. Hand the model page text and a schema and it returns clean records — that part is genuinely excellent, and cheap on the flash tier. But a scraping pipeline also has to fetch specific pages, render their JavaScript, and do so at a volume and cadence you control, and none of that lives inside a language model API.

Google does ship URL-context tooling that lets the API read a link you pass in, and it is worth an honest framing: convenient for one-off reads, but you inherit its fetching — no control over rendering behavior, geography, or what happens when a page challenges automated access. Production extraction keeps the layers separate: a fetch layer you control, then Gemini as the parser.

One disambiguation, because the keyword cuts both ways: this guide points Gemini at the web as an extraction engine. Capturing Gemini's own answers as data is the reverse job — that is the Gemini Scraper API guide, and the LLM scraper explainer covers that category.

Install

Two clients: Google's SDK for the native path and requests for the fetch layer. The runs in this guide used Python 3.12:

pip install google-genai requests

Configure

Both keys come from the environment:

export GEMINI_API_KEY="your_google_ai_studio_key"
export SCRAPELESS_API_KEY="sk_your_scrapeless_key"

Fetch a page worth extracting from

The demo target is a public bookstore catalogue page built for scraping practice — 20 products, each with a title, price, stock flag, and star rating buried in presentational HTML. One POST to the Universal Scraping API returns the page with rendering, unlocking, and proxy routing handled server-side:

# fetch_catalogue.py — one POST returns the rendered catalogue page
import os

import requests

resp = requests.post(
    "https://api.scrapeless.com/api/v1/unlocker/request",
    headers={
        "Content-Type": "application/json",
        "x-api-token": os.environ["SCRAPELESS_API_KEY"],
    },
    json={
        "actor": "unlocker.webunlocker",
        "input": {"url": "https://books.toscrape.com/catalogue/page-1.html", "method": "GET"},
    },
    timeout=120,
)
resp.raise_for_status()
html = resp.json().get("data", "")
print(f"fetched {len(html):,} characters")
print("product cards in the HTML:", html.count('<article class="product_pod">'))

with open("page.html", "w", encoding="utf-8") as f:
    f.write(html)

The run prints 50,449 characters and 20 product cards. That 50k figure matters later: it is what Gemini will read, and what you will pay tokens for.

Basic implementation: schema-first extraction

The current Gemini API accepts a JSON schema in the request, and the cleanest way to produce one is a Pydantic model — the schema language itself is the one defined by the JSON Schema specification. Ratings on this page live in a CSS class rather than text, so the schema and system rule spell out how to read them.

Note: This block needs a GEMINI_API_KEY — the one prerequisite this guide does not assume. The next section runs the identical extraction live through OpenRouter, with captured output. The exact request surface is documented in the Gemini structured-output guide.

# extract_gemini.py — native Gemini extraction (requires GEMINI_API_KEY)
from google import genai
from pydantic import BaseModel


class Book(BaseModel):
    title: str
    price_gbp: float
    in_stock: bool
    rating: int


class Catalogue(BaseModel):
    books: list[Book]


client = genai.Client()  # reads GEMINI_API_KEY from the environment
page_html = open("page.html", encoding="utf-8").read()

interaction = client.interactions.create(
    model="gemini-3.5-flash",
    input="Extract every book. rating is 1-5, read it from the star-rating class name.\n\n" + page_html,
    response_format={
        "type": "text",
        "mime_type": "application/json",
        "schema": Catalogue.model_json_schema(),
    },
)
print(interaction)

The schema does the enforcement: price_gbp comes back as a number, in_stock as a boolean, rating as an integer — no post-hoc string cleanup.

No Google key? Run it through OpenRouter

The extraction also runs over an OpenAI-compatible surface with a Gemini model behind it, which is how this guide executed it end to end — fetch and extraction in one self-contained script:

# extract_openrouter.py — the same extraction, executed via OpenRouter
import json
import os

import requests
from openai import OpenAI

resp = requests.post(
    "https://api.scrapeless.com/api/v1/unlocker/request",
    headers={
        "Content-Type": "application/json",
        "x-api-token": os.environ["SCRAPELESS_API_KEY"],
    },
    json={
        "actor": "unlocker.webunlocker",
        "input": {"url": "https://books.toscrape.com/catalogue/page-1.html", "method": "GET"},
    },
    timeout=120,
)
resp.raise_for_status()
page_html = resp.json().get("data", "")

client = OpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key=os.environ["OPENROUTER_API_KEY"],
)

completion = client.chat.completions.create(
    model="google/gemini-2.5-flash-lite",
    temperature=0,
    max_tokens=4000,
    response_format={"type": "json_object"},
    messages=[
        {
            "role": "system",
            "content": 'Extract every book. Reply ONLY with JSON: '
            '{"books":[{"title":str,"price_gbp":number,"in_stock":bool,"rating":int}]}. '
            "rating is 1-5, read it from the star-rating class name.",
        },
        {"role": "user", "content": page_html},
    ],
)

data = json.loads(completion.choices[0].message.content)
print(f"extracted {len(data['books'])} books")
print(json.dumps(data["books"][0], ensure_ascii=False))

The live run returned all 20 products, first record:

extracted 20 books
{"title": "A Light in the Attic", "price_gbp": 51.77, "in_stock": true, "rating": 3}

Price parsed to a float, stock to a boolean, and the three-star rating read correctly out of a class name — from 50k characters of catalogue HTML, in one call, for 12,904 tokens on a model priced in fractions of a dollar per million.

Get your API key on the free plan: app.scrapeless.com

Advanced patterns: token economics

Extraction cost is input-dominated, so the fetch layer is also the budget layer.

• Measure tokens per page early. The run above cost 12,904 tokens for one catalogue page. Multiply by your page count before committing to a model tier, not after the invoice.
• Trim before you send. Page chrome is a tax. Isolating the product container, or fetching pages as markdown rather than raw HTML, cuts input size — often by more than half — with zero loss of extractable content.
• Stay on the flash tier for schema work. A strict schema plus temperature=0 makes extraction a constrained task; the run above needed nothing larger. Escalate model size only when fields come back wrong on clean input.
• Cache nothing in the prompt. The schema lives in the system message once per call; do not paste examples of previous pages into new calls — it inflates input and teaches the model to echo stale records.

Troubleshooting

• Fields come back as strings. Your schema is not reaching the API — check that the native call passes schema (or the OpenRouter call passes the JSON contract in the system message) rather than describing fields loosely in prose.
• Zero or three products from a 20-product page. Inspect the fetched HTML first: count product cards the way the fetch script does. A near-empty fetch is a rendering or access issue — a fetch-layer fix, not a prompt fix.
• Ratings all come back as 5. The value is encoded in markup, not text. Say where it lives ("read it from the star-rating class name"), as both extraction blocks here do.
• Costs jump between runs. Compare input sizes; a redesign that doubles page weight doubles your token bill. Trimming and markdown fetching are the levers.

Conclusion

Gemini's flash tier plus a strict schema turns a 50k-character catalogue page into 20 typed records for pocket change — that is the extraction half, and it is the easy half now. The half that decides whether the records exist at all is the fetch: rendered, controlled, one POST per page. Keep the layers separate, measure tokens per page, and the same forty lines scale from a demo bookstore to a real catalogue.

Ready to Feed Gemini Real Pages?

The fetch layer here is the Universal Scraping API — plans and request volumes are on the pricing page, and the developer docs cover every unlocker.webunlocker parameter. Create a key on the free plan at app.scrapeless.com and the catalogue fetch above runs as written.

FAQ

Q: Can Gemini access websites directly?

Only through Google's URL-context tooling, which fetches a link server-side for one-off reads. It does not give you rendering control, geographic egress, or volume — the properties a scraping pipeline is built on — which is why production setups pair Gemini with a dedicated fetch layer and pass the model clean page text.

Q: Which Gemini model should I use for extraction?

The smallest flash-tier model that holds your schema. Extraction against a strict schema is a constrained task, not a reasoning benchmark — the live run in this guide used a flash-lite model and typed all 20 records correctly. Move up a tier only when clean input still produces wrong fields.

Q: How much does Gemini web scraping cost at scale?

Input tokens dominate, so cost is roughly pages × tokens-per-page × model rate. The measured run here was 12,904 tokens for a 50,449-character page; trimming chrome or fetching markdown cuts that substantially. Measure one real page before extrapolating — invented averages are how budgets die.

Q: How is this different from a Gemini scraper?

Direction. This guide uses Gemini as the parser pointed at the web. A Gemini scraper points a scraper at Gemini — capturing its answers, citations, and sources as data. Scrapeless ships that as an actor; the Gemini Scraper API guide linked in the introduction covers it.

Q: Is scraping with Gemini legal?

The extraction layer changes nothing about collection rules. Fetch public pages only, respect site terms and the robots directives standardized by the Robots Exclusion Protocol, keep volumes bounded, and handle any personal data lawfully — plus Google's API terms on the model side.