How to Handle Pagination in Web Scraping: Every Type Explained

TL;DR:

• There are four pagination types and each has its own stop condition. Next-button, numbered pages, load-more, and infinite scroll — get the stop condition wrong and you either miss pages or loop forever.
• Follow the site's own "next" link; don't guess URLs. Reading li.next a and following it page by page survives gaps and irregular numbering that a ?page=N loop would silently skip.
• Infinite scroll ends when the page stops growing. Scroll, wait, and compare scrollHeight — when it stops increasing, you've reached the end; that's the only reliable signal.
• Pagination is where you get rate-limited. Walking dozens of pages from one IP is exactly the pattern anti-bot systems watch; residential egress and a real browser keep the run clean.
• It all runs on Scrapeless Scraping Browser with plain Puppeteer. Mint a session, navigate, extract, advance — the cloud browser handles rendering and anti-detection so the pages keep loading.
• Free to start. New Scrapeless accounts include free Scraping Browser runtime — sign up at app.scrapeless.com.

Introduction: pagination is the part that breaks scrapers

Scraping one page is easy. Scraping all of them is where most scrapers quietly fail — they grab page one and stop, or loop past the end fetching empty pages, or get rate-limited by anti-bot defenses halfway through. The reason is that "pagination" isn't one thing. A site uses one of four mechanisms, and each needs a different way to advance and a different way to know it's done.

Get the type right and the rest is mechanical: extract the current page, find the way forward, repeat until the stop condition. Get it wrong — assume ?page=N when the site uses a "Load more" button, or sleep a fixed time instead of waiting for new content — and you lose data without an error to tell you.

This guide covers all four types and runs them on Scrapeless Scraping Browser, an anti-detection cloud browser connected to Puppeteer. The next-button walk below was run against a live paginated site; the other patterns share the same loop shape.

The four pagination types

The single most common mistake is treating a next-button or load-more site as if it had clean numbered URLs. Follow the site's own controls instead of constructing URLs and you stay correct even when numbering has gaps.

Why Scrapeless Scraping Browser

Scrapeless Scraping Browser is a customizable, anti-detection cloud browser designed for web crawlers and AI agents. For multi-page crawls specifically, it brings:

• A standard Puppeteer connection — Puppeteer.connect() returns a normal Browser, so your navigation and extraction code is unchanged.
• Residential proxies in 195+ countries — walk many pages without the IP-reputation hit that triggers rate-limiting.
• Anti-detection fingerprinting — long crawls read as a real browser, so later pages keep rendering.
• Cloud-side JS rendering — load-more and infinite-scroll content (built by JavaScript) actually appears.
• Session persistence — keep cookies warm across the whole crawl.

Get your API key on the free plan at app.scrapeless.com.

Prerequisites

• Node.js 18 or newer
• A Scrapeless account and API key — sign up at app.scrapeless.com
• Basic familiarity with Puppeteer and the page DOM and selectors

Install

npm install @scrapeless-ai/sdk puppeteer-core

export SCRAPELESS_API_KEY="your_api_token_here"

Connect

import { Puppeteer } from '@scrapeless-ai/sdk';

const browser = await Puppeteer.connect({
  apiKey: process.env.SCRAPELESS_API_KEY,
  sessionName: 'pagination',
  proxyCountry: 'US',
  sessionTTL: 300,
});

const page = await browser.newPage();

Type 1 — Next button (the most robust)

Follow the site's own "next" link until it's gone. This survives gaps in numbering because you never construct a URL — you read the one the site gives you:

const BASE = 'https://quotes.toscrape.com';
let url = `${BASE}/page/1/`;
const items = [];
let pages = 0;

while (url) {
  await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 30000 });
  pages++;

  const pageItems = await page.evaluate(() =>
    [...document.querySelectorAll('.quote .text')].map((q) => q.textContent),
  );
  items.push(...pageItems);

  // Read the next link; null when we're on the last page
  const next = await page.evaluate(() => {
    const a = document.querySelector('li.next a');
    return a ? a.getAttribute('href') : null;
  });
  url = next ? new URL(next, BASE).href : null;
}

console.log(pages, 'pages,', items.length, 'items');
// 10 pages, 100 items

The loop ends naturally when li.next a no longer exists — no page-count guess, no off-by-one.

Get your API key on the free plan: app.scrapeless.com

Type 2 — Numbered pages

When the site exposes clean numbered URLs and a visible last page, read the highest page number once, then iterate. Still prefer reading the last-page link over assuming it:

await page.goto(`${BASE}/page/1/`, { waitUntil: 'domcontentloaded' });
// e.g. read the last pager link if present, else advance until a page yields nothing
for (let n = 1; ; n++) {
  await page.goto(`${BASE}/page/${n}/`, { waitUntil: 'domcontentloaded' });
  const count = await page.evaluate(() => document.querySelectorAll('.quote').length);
  if (count === 0) break;   // empty page = past the end
  // ...extract
}

The empty-page check is the safety net: even if you misjudge the last page number, you stop the moment a page returns no items.

Type 3 — Load-more button

The content is one page; a button appends more. Click it until it's gone, waiting for new items between clicks:

await page.goto(TARGET, { waitUntil: 'domcontentloaded' });
while (true) {
  const before = await page.evaluate(() => document.querySelectorAll('.item').length);
  const btn = await page.$('button.load-more');
  if (!btn) break;                       // no more button = done
  await btn.click();
  await page.waitForFunction(
    (n) => document.querySelectorAll('.item').length > n,
    {},
    before,
  );                                      // wait for new items, not a fixed sleep
}

Waiting on the item count to grow — instead of a fixed delay — is what makes this reliable across slow responses.

Type 4 — Infinite scroll

New items load as you scroll. Scroll, wait, and stop when the page height stops increasing:

await page.goto(TARGET, { waitUntil: 'domcontentloaded' });
let prevHeight = 0;
while (true) {
  const height = await page.evaluate(() => document.body.scrollHeight);
  if (height === prevHeight) break;       // height stopped growing = end
  prevHeight = height;
  await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
  await new Promise((r) => setTimeout(r, 1500)); // let the next batch load
}
const items = await page.evaluate(() =>
  [...document.querySelectorAll('.item')].map((el) => el.textContent),
);

What You Get Back

Whatever the type, you end with a flat list assembled across pages — for the verified next-button run, 100 items across 10 pages. A few honest observations:

• Dedupe defensively. Load-more and infinite scroll can re-emit items; key on a stable id.
• Cap the loop. Add a max-pages guard so a misbehaving site can't spin forever.
• Watch for a stable selector. If li.next or the load-more button changes class, the loop ends early — re-check selectors when counts look low.
• Pace the crawl. A short delay between pages and modest concurrency keeps the IP-reputation signal clean.

Conclusion: pick the type, then the loop is mechanical

Every paginated crawl is the same shape — extract, advance, check the stop condition — once you've identified which of the four types the site uses. Follow the site's own next/load-more controls instead of guessing URLs, and stop on the real signal (no next link, empty page, missing button, stable height). Running on Scrapeless Scraping Browser keeps long crawls rendering and unblocked with residential proxy egress. For collecting the page URLs to crawl in the first place, see the Etsy scraper guide; the Scraping Browser product page and docs cover the full SDK surface. Read the site's controls, wait on content not the clock, and guard the loop.

Ready to Build Your AI-Powered Data Pipeline?

Join our community to claim a free plan and connect with developers building multi-page crawlers: Discord · Telegram.

Sign up at app.scrapeless.com for free Scraping Browser runtime and adapt the patterns above to the paginated sites your workflow needs. See pricing for scale.

FAQ

Q: How do I know which pagination type a site uses?
Look at the page: a "Next" link is next-button; numbered links are numbered pages; a "Load more" button is load-more; new content appearing as you scroll with no control is infinite scroll. Many sites mix two — handle the one that loads the data.

Q: Why does my ?page=N loop miss data?
Because the site doesn't use clean numbered URLs, or numbering has gaps. Follow the site's own next link instead of constructing URLs.

Q: How do I stop an infinite-scroll loop?
Compare document.body.scrollHeight before and after each scroll. When it stops increasing, there's nothing left to load.

Q: Do I need a proxy for multi-page crawls?
Often yes — walking many pages from one IP is a classic rate-limit trigger. Pin residential egress with proxyCountry and pace the crawl.

Q: How do I avoid getting blocked partway through?
Add a short delay between pages, keep concurrency modest, and run on a real anti-detection browser so later pages keep rendering.

Q: Can I run this without an AI agent?
Yes. It's plain Puppeteer over the Scrapeless session — no agent required.