How to Scrape Instagram: Profiles, Posts, Comments, and More

TL;DR:

• Instagram renders from its own internal JSON APIs — so call them directly. The profile page is hydrated by web_profile_info; posts and comments come from the GraphQL endpoint. You fetch that JSON from inside a rendered session instead of scraping the DOM.
• Two things make the API calls work: the x-ig-app-id header and the page's own cookies. Load instagram.com first so the session has cookies, then fetch() the API from inside the page with credentials: 'include' and the app-id header.
• Region and a real browser are not optional. Instagram weighs IP reputation and fingerprints hard; the same call that stalls from one egress returns cleanly from another. Pin proxyCountry and run on an anti-detection browser.
• You get structured objects back, not HTML. web_profile_info returns the full user object — followers, following, post count, verification, bio — exactly as Instagram's own front-end consumes it.
• One pattern covers profiles, posts, and comments. Mint a session, hit the matching internal endpoint, parse the JSON. The surface changes; the technique doesn't.
• Free to start. New Scrapeless accounts include free Scraping Browser runtime — sign up at app.scrapeless.com.

Introduction: scrape Instagram from its own APIs

Instagram's web pages are React shells hydrated from internal JSON endpoints. Scraping the rendered DOM is the hard, brittle way to do it — selectors churn and most data isn't even in the initial markup. The reliable way is to call the same endpoints the front-end calls: web_profile_info for a profile, the GraphQL query for a post and its comments. They return clean, structured JSON.

The catch is that those endpoints only answer a request that looks like it came from the Instagram web app. That means three things: the request carries the x-ig-app-id header, it includes the session's cookies, and it originates from an IP and browser fingerprint Instagram trusts. Miss any one and you get an empty body or a checkpoint.

This guide runs on Scrapeless Scraping Browser — anti-detection Chromium with residential egress — connected to Puppeteer over CDP. You load instagram.com to get a real session, then call its internal APIs from inside the page. The profile extraction below was captured live. Public data only.

What You Can Do With It

• Pull a profile — followers, following, post count, verification, bio, external links.
• Fetch a single post by shortcode with its media, caption, and counts.
• Collect a post's comments with author, text, and like counts.
• Walk a user's posts by paging the timeline endpoint.
• Enrich profiles at scale by looping the profile call over a handle list.

Why Scrapeless Scraping Browser

Scrapeless Scraping Browser is a customizable, anti-detection cloud browser designed for web crawlers and AI agents. For Instagram specifically, it brings:

• Self-developed Chromium — a real browser that boots the session and holds the cookies the API calls need.
• Anti-detection fingerprinting — the request reads as the Instagram web app, so the internal endpoints answer instead of returning a checkpoint.
• Residential proxies in 195+ countries — pin egress by country; the right region is what gets a clean response.
• Configurable session TTL — keep the session alive across multiple endpoint calls.
• A standard Puppeteer connection — mint a session with the SDK, then puppeteer.connect() over CDP; the rest is plain Puppeteer.

Get your API key on the free plan at app.scrapeless.com.

Prerequisites

• Node.js 18 or newer
• A Scrapeless account and API key — sign up at app.scrapeless.com
• Basic familiarity with Puppeteer and JSON

Install

npm install @scrapeless-ai/sdk puppeteer-core

export SCRAPELESS_API_KEY="your_api_token_here"

Step 1 — Mint a session and load Instagram

The SDK creates the cloud session; Puppeteer connects over CDP. Navigating to instagram.com first is what gives the session the cookies the internal API expects:

import { Scrapeless } from '@scrapeless-ai/sdk';
import puppeteer from 'puppeteer-core';

const client = new Scrapeless({ apiKey: process.env.SCRAPELESS_API_KEY });

const { browserWSEndpoint } = await client.browser.create({
  proxyCountry: 'US',   // pin egress — region affects whether the API answers
  sessionTTL: 180,
});

const browser = await puppeteer.connect({ browserWSEndpoint });
const page = await browser.newPage();
await page.goto('https://www.instagram.com/', {
  waitUntil: 'domcontentloaded',
  timeout: 60000,
});

If a call stalls or returns empty, switch proxyCountry before changing anything else — region is the most common cause.

Step 2 — Scrape a profile from web_profile_info

With the session warm, fetch() the profile endpoint from inside the page. The two load-bearing details are the x-ig-app-id header and credentials: 'include' (so the page's cookies ride along):

const user = await page.evaluate(async (username) => {
  const res = await fetch(
    `https://i.instagram.com/api/v1/users/web_profile_info/?username=${username}`,
    {
      headers: { 'x-ig-app-id': '936619743392459' }, // the public web app id
      credentials: 'include',
    },
  );
  const data = JSON.parse(await res.text());
  return data.data.user;
}, 'nasa');

console.log({
  username: user.username,
  full_name: user.full_name,
  verified: user.is_verified,
  followers: user.edge_followed_by.count,
  following: user.edge_follow.count,
  posts: user.edge_owner_to_timeline_media.count,
});
// {
//   username: 'nasa',
//   full_name: 'NASA',
//   verified: true,
//   followers: 104420451,
//   following: 91,
//   posts: 4817
// }

That returns Instagram's full user object verbatim — the same structure its own front-end renders from.

Get your API key on the free plan: app.scrapeless.com

Step 3 — Posts and comments from GraphQL

Individual posts and their comments come from Instagram's GraphQL endpoint. The shape is the same — fetch() from inside the page — but it's a POST with the post's shortcode in the variables:

const shortcode = 'C1234567abc'; // from instagram.com/p/<shortcode>/

const post = await page.evaluate(async (shortcode) => {
  const variables = JSON.stringify({ shortcode });
  // doc_id identifies the persisted GraphQL query the web app uses
  const body = `variables=${encodeURIComponent(variables)}&doc_id=YOUR_DOC_ID`;
  const res = await fetch('https://www.instagram.com/graphql/query', {
    method: 'POST',
    headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
    body,
    credentials: 'include',
  });
  return JSON.parse(await res.text());
}, shortcode);

The doc_id is the persisted-query id Instagram's front-end sends; capture the current one from the network panel of a real post view. From the response you read the media object (xdt_shortcode_media) for the caption, media URLs, and the comment edges.

What You Get Back

The profile call returns Instagram's user object. The shape below is a real capture; counts move over time:

{
  "username": "nasa",
  "full_name": "NASA",
  "is_verified": true,
  "is_private": false,
  "biography": "Making the seemingly impossible, possible...",
  "external_url": "https://...",
  "edge_followed_by": { "count": 104420451 },
  "edge_follow": { "count": 91 },
  "edge_owner_to_timeline_media": { "count": 4817 }
}
// Real capture from web_profile_info (@nasa). The full user object carries dozens more fields; read the ones you need.

A few honest observations:

• The user object is large. It includes business flags, AI-agent fields, highlight counts, and more — read the handful you need and ignore the rest.
• Counts live in edge_*.count. Followers, following, and posts are nested under edge_followed_by, edge_follow, and edge_owner_to_timeline_media.
• Private accounts return the profile shell but not the media. Check is_private before expecting posts.
• Region matters as much as headers. Pin proxyCountry; an unlucky region is the usual cause of an empty body.

Conclusion: call the APIs Instagram already exposes

Scraping Instagram cleanly means calling its internal JSON endpoints from inside a real, warmed session — web_profile_info for profiles, GraphQL for posts and comments — with the x-ig-app-id header and the page's own cookies. Running on Scrapeless Scraping Browser is what makes those calls answer: anti-detection Chromium plus residential egress, so Instagram treats the request as its own web app. For the same SDK-over-CDP pattern on another social surface, see the TikTok scraper guide; the Scraping Browser product page and docs cover the full SDK surface. Pin the region, warm the session, send the app-id header, and read the JSON.

Ready to Build Your AI-Powered Data Pipeline?

Join our community to claim a free plan and connect with developers building social-data pipelines: Discord · Telegram.

Sign up at app.scrapeless.com for free Scraping Browser runtime and adapt the patterns above to the profiles, posts, and hashtags your workflow needs. See pricing for scale.

FAQ

Q: Is scraping Instagram legal?
Collecting publicly visible data is generally permissible, but Instagram's Terms of Service apply and rules vary by jurisdiction. Scrape only public data, review the ToS, and consult counsel for your use case.

Q: Why does the profile API return an empty body or a checkpoint?
Usually one of three things: a missing x-ig-app-id header, no session cookies (you skipped loading instagram.com first), or an egress region Instagram distrusts. Send the header, use credentials: 'include', and pin proxyCountry.

Q: Do I need to log in?
For public profiles and posts, the web-app endpoints answer an anonymous-but-warmed session. Logged-in scraping is a different, higher-risk path; this guide stays on public data.

Q: Where does the doc_id for posts come from?
It's the persisted-query id Instagram's front-end sends to GraphQL. Read the current value from the network panel on a real post view — it changes over time.

Q: Do I need a proxy?
Yes. Instagram weighs IP reputation heavily; pin residential egress with proxyCountry so the internal endpoints respond.

Q: How many requests can I make?
Keep it modest — a handful of sessions per region and a delay between calls — so the IP-reputation signal stays clean.

Q: Can I run this without an AI agent?
Yes. It's the Scrapeless SDK plus plain Puppeteer over CDP — no agent required.