Build a Distributed Web Crawler in Node.js: Queues and Deduplication

TL;DR:

• A distributed web crawler needs a durable URL frontier, stateless workers, canonical URL keys, per-host request budgets, and an explicit terminal quarantine path.
• Keep Node.js responsible for discovery, scheduling, state, and storage. Delegate JavaScript rendering and page acquisition to a managed execution layer when the source requires it.
• Use the canonical URL hash as both the queue job ID and the storage idempotency key. This blocks duplicate work before it reaches a worker.
• Global worker concurrency and per-host pacing solve different problems. Scale the first with capacity; set the second from source permission and observed server behavior.
• Start with a small authorized source set, measure accepted pages rather than attempted pages, and scale only after queue depth, freshness lag, and schema rejection are visible.

A single-process crawler fails in predictable ways: its memory queue disappears on restart, duplicate links multiply, one slow host occupies the event loop, and browser execution consumes the same machine that should be scheduling work.

A distributed web crawler separates those responsibilities. Node.js owns the control plane—the URL frontier, job state, deduplication, and storage decisions. Independent workers own the data path. For JavaScript-rendered public pages, a managed service can execute the page and return Markdown or HTML without placing browser processes inside every worker container.

Define the requirements and failure modes

Before choosing a queue, write the crawl contract:

• Which domains and paths are authorized?
• How many pages and levels may each crawl discover?
• What freshness window does each source need?
• Which response formats and required fields define an accepted page?
• What request pace is allowed per host?
• Where do invalid, empty, or unexpected results go?
• How long must job and page versions be retained?

The first version should also have stop conditions: maximum depth, maximum accepted pages, maximum discovered URLs, and a deadline. These limits prevent a calendar archive, faceted navigation, or tracking parameter from turning a small crawl into an open-ended graph walk.

Common failure modes are architectural:

Separate the control plane from page execution

The crawler can be drawn as two planes:

Control plane: seeds → URL normalization → deduplication → Redis queues → job state → storage metadata

Execution plane: worker → page acquisition → content validation → link extraction → accepted record or quarantine

This boundary matters because scheduling and browser execution scale differently. Queue operations are small and stateful. Page execution is network-heavy and may require JavaScript, regional routing, or an isolated browser session.

Scrapeless Crawl supports single-page, batch, and linked-site collection with formats including Markdown, HTML, links, metadata, and screenshots. When the execution path needs an interactive browser, the Scrapeless Scraping Browser keeps that runtime outside the worker container. The Node application can remain the system of record for scope and state while Scrapeless handles acquisition.

For a conceptual introduction to discovery and extraction, read what a web crawler is. The design below begins where a single-machine crawler stops: shared queues, idempotent enqueueing, and multiple workers.

Build the Redis-backed URL frontier

BullMQ implements distributed job execution on Redis. Its official documentation covers queues, workers, events, delayed jobs, rate limiting, and job state.

The frontier should store a small job payload:

Do not place page HTML in Redis. Store large results in an object store or database and keep only identifiers, states, and compact metadata in the queue.

Use a separate queue per approved host when different domains require different request budgets. Workers assigned to docs-example-com can then use one limiter, while another host gets its own pace. A global queue is simpler, but its limiter cannot express independent host policies.

Make enqueueing idempotent

Two pages can be equivalent while their raw URLs differ:

• a fragment points to a location inside the same document;
• tracking parameters change without changing content;
• query parameters appear in a different order;
• default ports or trailing slashes vary;
• relative links resolve to the same absolute page.

Normalize before queue insertion. The WHATWG URL Standard defines the parsing model implemented by the Node.js URL class.

A safe policy is source-specific. Removing every query parameter can merge genuinely different pages. Maintain an allowlist or denylist for known parameters and preserve any parameter that changes the resource.

After normalization, hash the URL with SHA-256. Use that hexadecimal digest as:

• the BullMQ job ID;
• the database upsert key;
• the provenance link between job and stored page.

BullMQ ignores a new job when another job with the same ID already exists in that queue. Job records removed by retention no longer provide that protection, so durable storage must keep the same idempotency key.

Minimal version-pinned Node.js project

This project is intentionally compact:

distributed-crawler/
├── package.json
└── src/
    └── crawler.mjs

The dependency versions were checked against their package registries at drafting time. The example is a prerequisite-gap block: it requires Node.js, Redis, a SCRAPELESS_API_KEY, and an authorized public hostname set in ALLOWED_HOST. It is designed for one host so the queue limiter is truly host-specific.

{
  "name": "distributed-crawler-example",
  "private": true,
  "type": "module",
  "scripts": {
    "start": "node src/crawler.mjs"
  },
  "dependencies": {
    "@scrapeless-ai/sdk": "1.3.1",
    "bullmq": "5.79.3"
  },
  "engines": {
    "node": ">=22"
  }
}

The worker below has four terminal outcomes: accepted, unchanged, rejected, and quarantined. It does not create an automatic failure loop. An operator can inspect the quarantine record, fix its cause, and explicitly requeue the canonical URL.

import { createHash } from "node:crypto";
import { Queue, Worker } from "bullmq";
import { ScrapingCrawl } from "@scrapeless-ai/sdk";

const redis = {
  host: process.env.REDIS_HOST ?? "127.0.0.1",
  port: Number(process.env.REDIS_PORT ?? 6379)
};
const allowedHost = process.env.ALLOWED_HOST ?? "example.com";
const queueName = `crawl-${hostKey(allowedHost)}`;
const frontier = new Queue(queueName, { connection: redis });
const quarantine = new Queue(`${queueName}-quarantine`, { connection: redis });
const crawl = new ScrapingCrawl({
  apiKey: process.env.SCRAPELESS_API_KEY
});

function sha256(value) {
  return createHash("sha256").update(value).digest("hex");
}

function hostKey(host) {
  return host.toLowerCase().replaceAll(".", "-");
}

function canonicalize(input) {
  const url = new URL(input);
  url.hash = "";
  url.hostname = url.hostname.toLowerCase();
  for (const key of ["utm_source", "utm_medium", "utm_campaign"]) {
    url.searchParams.delete(key);
  }
  url.searchParams.sort();
  return url.href;
}

async function enqueue(url, crawlId, depth = 0, parentUrl = null) {
  const canonicalUrl = canonicalize(url);
  const parsed = new URL(canonicalUrl);
  if (parsed.hostname !== allowedHost) {
    throw new Error(`Host outside crawl scope: ${parsed.hostname}`);
  }

  const key = sha256(canonicalUrl);
  await frontier.add(
    "collect-page",
    {
      url: canonicalUrl,
      host: parsed.hostname,
      depth,
      crawlId,
      parentUrl,
      schemaVersion: "crawl-page-v1"
    },
    {
      jobId: key,
      removeOnComplete: 1000,
      removeOnFail: false
    }
  );
  return key;
}

const worker = new Worker(
  queueName,
  async (job) => {
    try {
      const result = await crawl.scrapeUrl(job.data.url, {
        formats: ["markdown", "links"],
        onlyMainContent: true,
        timeout: 15000
      });
      const markdown = result.markdown ?? result.data?.markdown;
      if (typeof markdown !== "string" || markdown.length < 200) {
        return { state: "rejected", reason: "content-contract", url: job.data.url };
      }

      const record = {
        key: job.id,
        url: job.data.url,
        crawlId: job.data.crawlId,
        depth: job.data.depth,
        contentHash: sha256(markdown),
        markdown
      };

      console.log(JSON.stringify({ state: "accepted", ...record }));
      return { state: "accepted", key: record.key, contentHash: record.contentHash };
    } catch (error) {
      await quarantine.add("inspect-page", {
        ...job.data,
        sourceJobId: job.id,
        reason: error instanceof Error ? error.message : "unknown"
      });
      return { state: "quarantined", key: job.id };
    }
  },
  {
    connection: redis,
    concurrency: 4,
    limiter: { max: 2, duration: 1000 }
  }
);

worker.on("completed", (job, result) => {
  console.log(JSON.stringify({ event: "completed", jobId: job.id, result }));
});

worker.on("failed", (job, error) => {
  console.error(JSON.stringify({
    event: "worker-failed",
    jobId: job?.id,
    message: error.message
  }));
});

await enqueue(`https://${allowedHost}/`, "demo-crawl");

The example prints accepted records to make the data contract visible. Replace console.log with a database upsert keyed by record.key. Compare contentHash with the stored value before writing a new page version or rebuilding an index.

Understand the task state machine

A crawler needs a state model that operators can explain:

Keep failed as an infrastructure event reported by the queue, not as the only business state. A job that returns an empty application shell is technically completed but should be rejected. A page outside scope should be cancelled before acquisition. These distinctions make dashboards actionable.

Control concurrency by host

Worker concurrency answers: “How many jobs can this process handle?” A host request budget answers: “How much traffic may this origin receive?” They must be configured independently.

For an authorized domain:

1. Read robots rules and contractual limits.
2. Set a conservative per-host pace.
3. Run multiple workers only when the queue and host budget allow it.
4. Track response status, content acceptance, and server latency.
5. Lower the host budget when the source shows distress or the agreement changes.

BullMQ workers can share a queue across processes and machines. The queue limiter coordinates the selected queue, which is why the example uses a host-specific queue. For many domains, generate queues from an approved registry and cap the number of active worker objects.

The Robots Exclusion Protocol is standardized by RFC 9309. Robots rules are not a grant of permission, and they do not replace site terms, privacy duties, or applicable law.

Delegate page acquisition without losing control

The Node.js control plane should decide what may be collected. The execution layer should decide how to obtain the permitted page representation.

The Scrapeless Crawl quickstart documents asynchronous crawl status and page-level results. For pages requiring broader interaction or JavaScript execution, use Crawl's browser options while the queue retains scope, job identity, and storage decisions.

Validate the returned content rather than assuming the executor made the business decision. Require expected text, fields, language, URL, and minimum content. Store the acquisition route in provenance so a later audit can explain how each page entered the dataset.

Discover links without escaping scope

Link discovery belongs after content acceptance. Parse only pages that met the content contract, then apply these filters before enqueueing:

• allowlisted hostname;
• allowed path prefixes;
• supported HTTP schemes;
• maximum depth and page count;
• canonicalization and job-ID lookup;
• file-type exclusions;
• source-specific query-parameter policy.

Do not let redirects silently expand the allowed host set. Record the final URL, compare it with scope, and reject cross-domain results unless the source registry explicitly authorizes them.

For sitemaps, treat each URL as discovered input rather than trusted output. Canonicalize, filter, and deduplicate it through the same frontier path as an HTML link.

Store page versions and crawl provenance

A useful storage model has three records:

1. Crawl: scope, seed URLs, deadline, policy version, and overall state.
2. Page: canonical key, source URL, latest accepted hash, collection time, and schema version.
3. Page version: content hash, payload location, metadata, and acquisition provenance.

The queue is not the long-term database. Job cleanup policies may remove completed entries, while the page table must keep idempotency keys and version history according to retention policy.

If a page is unchanged, update the freshness timestamp without duplicating the payload. If it changes, write a new immutable page version, point the page record at it, and notify downstream indexing through a separate event.

Observe the crawl

Queue length alone can be misleading. A crawler can empty its frontier while rejecting every page. Track:

OpenTelemetry describes traces, metrics, logs, and baggage in its telemetry signals guide. Use one crawl ID across producer, queue event, acquisition call, storage write, and downstream index event.

Alert on stale accepted data and old quarantine entries, not only on process crashes. A process can be healthy while the dataset quietly stops changing.

Deployment checklist

• Run Redis with persistence, authentication, network controls, and backups appropriate to the workload.
• Keep workers stateless and deploy the same image across instances.
• Store API keys in a secret manager or environment injection, never in job payloads.
• Define queue retention separately from page retention.
• Bound crawl depth, pages, time, and host scope.
• Use one host policy source shared by producers and workers.
• Drain workers during deployment so active leases are not abandoned.
• Test cancellation, quarantine release, storage idempotency, and source-policy changes.
• Record versions of Node.js, BullMQ, the Scrapeless SDK, and the page schema.

Start with one approved host and a small page cap. Add domains only after dashboards show that accepted data, freshness, and request policy remain within their objectives.

Conclusion: scale the frontier, not the uncertainty

A distributed web crawler becomes dependable when every URL has one canonical identity, every host has an explicit budget, and every page ends in a meaningful state. Node.js and BullMQ can own the frontier and worker coordination; Scrapeless can own page execution where managed rendering and network handling are required.

Create a bounded proof of concept with one public, authorized domain, check the Scrapeless pricing page, then create a Scrapeless account. Route acquisition through Crawl and measure accepted pages and freshness before increasing worker count.

Frequently Asked Questions

What makes a web crawler distributed?

Its URL frontier and job state are shared across multiple worker processes or machines. Workers can claim independent jobs, write results to shared storage, and scale without relying on one process's memory.

Why use BullMQ for a Node.js crawler?

BullMQ provides Redis-backed queues, distributed workers, concurrency controls, events, and job identifiers. The crawler still needs its own URL policy, content contract, durable storage, and observability.

How does URL deduplication work across workers?

Normalize the URL before enqueueing, hash the canonical form, and use the digest as the queue job ID and storage key. Redis coordinates job creation, while the database preserves idempotency after queue retention removes old jobs.

Should every worker launch its own browser?

Not necessarily. Local browsers increase container size, memory use, and operational work. A managed acquisition layer can return rendered content while workers remain focused on scheduling, validation, discovery, and storage.

How should failed page jobs be handled?

Separate business outcomes from infrastructure events. Invalid content can be rejected; unresolved execution errors can enter a quarantine queue for inspection and explicit release. Avoid an unbounded automatic loop.