Build a Computer-Use Browser Agent Loop From Scratch

TL;DR:

• A computer-use agent can be reduced to a bounded perceive–decide–act loop. The browser captures the current page, a vision model chooses one action, and Playwright executes it before the cycle begins again.
• The action schema keeps model output executable. Restricting each decision to click, type, scroll, or done turns the model response into a small deterministic dispatcher.
• Fresh screenshots prevent stale decisions. Every model call sees the page after the previous action, so the agent reasons from the browser's current state.
• A step cap controls both autonomy and cost. The example allows at most four iterations and finishes in two: click Next, confirm the new first quote, then stop.
• A custom loop is best when decision logic needs to stay visible. Use a single vision call for one-shot extraction and a packaged agent when the loop no longer needs frequent tuning.
• Free to start. New Scrapeless accounts include free Scraping Browser runtime — sign up at app.scrapeless.com.

Introduction: The Small Loop Behind Computer Use

A computer-use agent does three things on repeat: it looks at the screen, it decides on one action, and it performs that action. Products like OpenAI's Computer Use tool and Google's Gemini computer-use mode package that loop behind a single API call. Agent frameworks like Skyvern package it behind a task runner. Underneath either one, the loop itself is a small amount of code — a screenshot, a vision-model call constrained to a short list of actions, and a browser command that carries out whatever the model picked.

This guide writes that loop directly, with no packaged agent in between. It connects to the Scrapeless Scraping Browser over the Chrome DevTools Protocol, screenshots the page, sends the image to a vision-capable model with four choices — click, type, scroll, or done — and executes whatever the model returns with Playwright, then repeats. Every step below runs against a real page, with the model's actual decisions captured as they happened.

The Loop at a Glance

Each iteration has one source of truth: the page as it appears in the latest screenshot.

The walkthrough uses a public quotes site and a narrow goal: leave the first page only when the first quote is written by Albert Einstein, then stop as soon as a different author appears.

Why Build the Loop Yourself?

Writing the loop directly gives you control over the perception format, the action vocabulary, and the stopping rule.

A structured-snapshot agent and a packaged vision framework solve the same problem at a higher level. The first can hand an LLM a tree of roles, labels, and states built from the WAI-ARIA accessibility tree rather than pixels, which is cheaper and more exact on a page with clean markup. Browser Use running on the Scraping Browser takes that path — it drives the same cloud Chromium over the same CDP connection, but its agent reasons over extracted page text, not an image.

The other shortcut hands the whole loop to a packaged vision framework, which screenshots, reasons, and acts the same way this walkthrough does, except the prompt, the action schema, and the decision code all live inside the library rather than in the calling script.

Every part of a custom decision stays visible and editable: the action vocabulary, the prompt that requests one action, the step cap, and the line that turns the decision into a Playwright call. A different goal, target, or model changes one part of the script instead of a framework-wide configuration surface.

Prerequisites

You need Python 3.9 or newer, the playwright and requests packages, a Scrapeless API key for the browser session, and a key for a vision-capable model. Get the Scrapeless key on the free plan at app.scrapeless.com. The example below routes model calls through OpenRouter, which fronts several model providers behind one key and one endpoint.

Set Up the Project

Install the Dependencies

pip install playwright requests

Configure the API Keys

export SCRAPELESS_API_KEY="your_scrapeless_api_key"
export OPENROUTER_API_KEY="your_model_api_key"

Build the Perceive–Decide–Act Loop

The complete script connects a remote browser, defines a four-action contract, and runs the contract until the model returns done or the step cap is reached.

Connect to the Remote Browser

The Scraping Browser exposes one CDP endpoint over a WebSocket connection — the WebSocket protocol carries Chrome DevTools Protocol traffic in both directions. Playwright's connect_over_cdp attaches to that endpoint the same way it would attach to a local Chromium instance started with a debugging port open; the browser itself runs in Scrapeless's cloud, not on the machine running this script. The Playwright proxy guide covers the same remote-CDP connection pattern in more detail.

Define the Decision Contract

Three functions carry the three parts of the loop: perceive takes the screenshot, decide sends it to the model and parses the reply, and act turns that reply into a Playwright call. The prompt asks the model to fill a small JSON object in a fixed field order — naming one concrete fact from the image before choosing an action keeps the decision grounded in what the screenshot actually shows, rather than in an open-ended guess about whether the goal is met:

Run the Complete Example

import os
import json
import base64
import requests
from urllib.parse import urlencode
from playwright.sync_api import sync_playwright

GOAL = (
    'Goal: get to a page where the FIRST quote card is not written by "Albert Einstein". '
    'Read the author name on the first quote card. If it is "Albert Einstein", click the '
    '"Next" pagination link. Otherwise the goal is already complete.'
)
SCHEMA_PROMPT = (
    GOAL + "\n\nRespond with ONLY a JSON object, filling every field in order:\n"
    '{"first_quote_author": "<the author name on the first quote card, read from the image>", '
    '"action": "click" | "type" | "scroll" | "done", '
    '"target": "<visible text of the element to click or type into, omit for scroll/done>", '
    '"value": "<text to type, only when action is type>", '
    '"reason": "<under 8 words>"}'
)


def browser_url():
    return "wss://browser.scrapeless.com/api/v2/browser?" + urlencode(
        {"token": os.environ["SCRAPELESS_API_KEY"], "sessionTTL": 180, "proxyCountry": "US"})


def perceive(page):
    return page.screenshot(full_page=True)


def decide(png_bytes):
    img = "data:image/png;base64," + base64.b64encode(png_bytes).decode()
    resp = requests.post(
        "https://openrouter.ai/api/v1/chat/completions",
        headers={"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}", "Content-Type": "application/json"},
        json={"model": "google/gemini-2.5-flash-lite",
              "messages": [{"role": "user", "content": [
                  {"type": "text", "text": SCHEMA_PROMPT},
                  {"type": "image_url", "image_url": {"url": img}}]}],
              "temperature": 0, "max_tokens": 300},
        timeout=120,
    )
    resp.raise_for_status()
    return json.loads(resp.json()["choices"][0]["message"]["content"])


def act(page, decision):
    action = decision["action"]
    if action == "click":
        page.get_by_text(decision["target"], exact=False).first.click(timeout=5000)
        page.wait_for_load_state("networkidle")
    elif action == "type":
        page.get_by_text(decision["target"], exact=False).first.fill(decision["value"], timeout=5000)
    elif action == "scroll":
        page.mouse.wheel(0, 900)
        page.wait_for_timeout(300)
    return action


with sync_playwright() as p:
    browser = p.chromium.connect_over_cdp(browser_url())
    page = browser.new_page(viewport={"width": 1280, "height": 900})
    page.goto("https://quotes.toscrape.com/", wait_until="networkidle")

    max_steps = 4
    for step in range(1, max_steps + 1):
        decision = decide(perceive(page))
        print(f"step {step}: author={decision['first_quote_author']!r} "
              f"action={decision['action']} reason={decision['reason']!r}")
        if decision["action"] == "done":
            break
        act(page, decision)

    print("final url:", page.url)
    print("steps taken:", step)
    browser.close()

Get your API key on the free plan: app.scrapeless.com

Verify the Run

Against the live page, the example navigated once and stopped on the next iteration:

step 1: author='Albert Einstein' action=click reason='First quote author is Albert Einstein, need to click next.'
step 2: author='Marilyn Monroe' action=done reason='The first quote is not by Albert Einstein.'
final url: https://quotes.toscrape.com/page/2/
steps taken: 2

Trace the Two-Step Decision

The trace shows why the loop needs a fresh screenshot after every action.

Step 1: Read the First Card and Click Next

Step 1 screenshots the front page, and the model reads "Albert Einstein" off the first quote card — a fact anyone looking at the same image would confirm — and returns {"action": "click", "target": "Next →"}. The act function resolves that target with page.get_by_text("Next →", exact=False), the same locator pattern a script written against the DOM directly would use, and clicks it. The click navigates to /page/2/.

Step 2: Confirm the New State and Stop

Step 2 screenshots the new page. The first quote card now belongs to Marilyn Monroe, the model reports that fact, and the action it returns is done. The loop exits on its own after two of the four allotted steps — it never needed to reach the cap, and it never needed the scroll action either, because perceive captures the full page in one image rather than only the visible viewport. For a page that loads more content as the user scrolls — infinite scroll, lazy-loaded sections — a full-page capture cannot see content that has not loaded yet, which is why scroll stays in the action vocabulary even though this target never triggers it. The type branch works the same way, filling whatever element matches target with value; this walkthrough's goal never needs a form field, so that branch runs the identical dispatch logic without this particular run exercising it.

Keep the Loop Bounded and Current

Two design choices keep a loop like this from running away. The max_steps bound means a model that never returns done still stops on its own, and a screenshot-plus-single-decision cycle means the loop never acts on stale information — every decision is made against the page as it exists at that moment, not against an assumption about what changed since the last look.

Choose the Right Browser-Agent Shape

The right approach depends on whether the task needs one observation or several and whether you need to inspect the decision logic.

A single vision call reads one screenshot and returns structured data without taking an action. What an LLM scraper does covers the broader extraction pattern. Use that shape when the task is simply to read a value.

The custom loop is for goals that take more than one look to satisfy. Each screenshot depends on the previous action, and the exact decision logic remains in code that can be inspected and changed.

A packaged agent framework wraps the same loop — or a text-based variant of it, in Browser Use's case — behind a task runner. That trade is useful once the loop's shape has stabilized and the main job is running many goals against many pages.

Conclusion: Keep the Loop Small and Observable

The mechanics behind a "computer use" agent are a screenshot, a vision-model call constrained to a short list of actions, and a browser command that carries out the result. The cycle repeats until the model reports the goal is met or a step cap is hit.

The run above took two turns to move from a page fronted by Albert Einstein to one fronted by Marilyn Monroe. The model's decision stayed visible at each step, and the Playwright call that executed it remained ordinary application code.

Read what the runtime offers on the Scraping Browser product page and weigh step count against model cost on the pricing page before scaling a loop like this past a handful of goals.

Ready to Build a Browser Agent Loop?

Join our community to claim a free plan and compare notes with other developers building browser agents: Discord · Telegram.

Sign up at app.scrapeless.com for free Scraping Browser runtime and adapt the bounded loop to a public page and goal you can verify step by step.

FAQ

Q: What is a computer-use style browser agent loop?

A computer-use style browser agent loop is a cycle of three steps repeated until a goal is met: perceive the page with a screenshot, decide on one action by sending that screenshot to a vision-capable model, and act by executing that decision with a real browser command. OpenAI's Computer Use tool and Google's Gemini computer-use mode package the same cycle behind a hosted API.

Q: How is this different from running Skyvern or Browser Use on the same browser?

Those frameworks run the same kind of loop internally, but the prompt, the action schema, and the decision code live inside the library. This walkthrough writes those three pieces directly, so every part of the decision is visible and editable in the calling script rather than in a framework's configuration surface.

Q: How is this different from a single screenshot-to-JSON extraction call?

A single vision call reads one image and returns data, with no action taken and no loop. This walkthrough repeats perceive, decide, and act until the model reports the goal is met, and each step's screenshot reflects the outcome of the previous step's action.

Q: Why constrain the model's reply to a fixed action schema instead of free-form instructions?

A small, fixed vocabulary — click, type, scroll, done — turns the model's reply into something the calling code can dispatch deterministically with an if/elif chain. An unconstrained reply would need its own parsing and interpretation layer before it could safely drive a browser.

Q: Why does the loop ask the model to name a concrete fact before choosing an action?

Asking the model to state one checkable detail from the image — the author on the first card, in this example — keeps the action grounded in what the screenshot actually shows. A field that asks the model to reason openly about whether the goal is met is more prone to drifting from the image in front of it.

Q: Why take a full-page screenshot instead of only the visible viewport?

So the model can see elements below the fold, like a pagination link at the bottom of a long list, without a dedicated scroll step first. A page that loads more content only as the user scrolls — infinite scroll or lazy-loaded sections — is the case where a full-page capture falls short and the scroll action earns its place in the loop.

Q: Why cap the number of steps?

max_steps bounds how many perceive-decide-act cycles a single goal can run, so a model that never returns done still stops instead of acting indefinitely. Each step is one screenshot and one model call, so the cap also bounds the run's cost.

Q: Which vision model does the example use, and can another model replace it?

The example routes to google/gemini-2.5-flash-lite through OpenRouter. Any vision-capable model reachable through the same chat-completions shape can replace it by changing the model field.

Q: Is it safe to point a loop like this at a real site?

Yes, when the target is public, the step count is bounded, and the goal excludes logins, payments, and private data. The example above runs two bounded steps against a public quotes site built for scraping practice.