Indeed Scraper

This is a working Indeed scraper walkthrough in Python and Node, built on the ScrapingBee web scraping API. It shows how to pull job title, company, and location off an Indeed results page, how to page through results, and how to skip the boilerplate with a ready-made package.

Indeed is a hard target for a naive script. The listings load through JavaScript, and Indeed screens automated requests, so requests.get("https://www.indeed.com/jobs...") on its own returns a near-empty page. The fix is to render the page in a real browser and route the request through residential proxies. ScrapingBee does both from a single API call.

How to scrape Indeed in Python

Send the Indeed URL through ScrapingBee with render_js on, then read the job cards with BeautifulSoup. The selectors below come from ScrapingBee's Indeed guide and match the current results markup.

import requests
from bs4 import BeautifulSoup

API_KEY = "YOUR_SCRAPINGBEE_API_KEY"
search_url = "https://www.indeed.com/jobs?q=python+developer&l=New+York"

response = requests.get(
    "https://app.scrapingbee.com/api/v1/",
    params={
        "api_key": API_KEY,
        "url": search_url,
        "country_code": "us",
        "render_js": "true",
        "premium_proxy": "true",
    },
)

soup = BeautifulSoup(response.text, "html.parser")
for card in soup.select("div.job_seen_beacon"):
    title = card.select_one("h2.jobTitle span")
    company = card.select_one('span[data-testid="company-name"]')
    location = card.select_one('div[data-testid="text-location"]')
    print(
        title.get_text(strip=True) if title else "",
        "|", company.get_text(strip=True) if company else "",
        "|", location.get_text(strip=True) if location else "",
    )

render_js=true runs the page in a headless browser, and premium_proxy=true sends it over residential IPs so Indeed serves the full listing. Grab a key with 1,000 free credits at ScrapingBee.

Selectors this Indeed web scraper uses

Field	CSS selector
Job card	div.job_seen_beacon
Title	h2.jobTitle span
Company	span[data-testid="company-name"]
Location	div[data-testid="text-location"]

If Indeed changes its markup, update these selectors or switch to natural-language extraction with ScrapingBee's AI extraction.

Paging through results

Indeed paginates with a start parameter that grows by 10 per page. Loop it to collect more than the first page.

for start in range(0, 30, 10):  # first three pages
    url = f"https://www.indeed.com/jobs?q=python+developer&l=New+York&start={start}"
    # send `url` through ScrapingBee exactly as above

Return JSON without parsing HTML

Instead of BeautifulSoup, you can pass ScrapingBee data extraction rules and get structured JSON straight back:

import json, requests

rules = {
    "jobs": {
        "selector": "div.job_seen_beacon",
        "type": "list",
        "output": {
            "title": "h2.jobTitle span",
            "company": 'span[data-testid="company-name"]',
            "location": 'div[data-testid="text-location"]',
        },
    }
}

response = requests.get(
    "https://app.scrapingbee.com/api/v1/",
    params={
        "api_key": "YOUR_SCRAPINGBEE_API_KEY",
        "url": "https://www.indeed.com/jobs?q=python+developer&l=New+York",
        "render_js": "true",
        "premium_proxy": "true",
        "extract_rules": json.dumps(rules),
    },
)
print(response.json())

Ready-made packages

If you would rather not wire this up by hand, the same logic is packaged for both ecosystems:

pip install indeed-scraper-api

from indeed_scraper_api import IndeedScraper

jobs = IndeedScraper(api_key="YOUR_SCRAPINGBEE_API_KEY").search("python developer", location="New York")
print(jobs["jobs"])

npm install indeed-scraper-api

const { IndeedScraper } = require('indeed-scraper-api');
const scraper = new IndeedScraper({ apiKey: 'YOUR_SCRAPINGBEE_API_KEY' });
scraper.search('python developer', { location: 'New York' }).then((r) => console.log(r.jobs));

Credit cost

With premium_proxy and render_js both on, a request runs at 25 ScrapingBee credits (Premium tier plus JavaScript). Where a target does not need residential IPs, drop premium_proxy for a cheaper datacenter call. The ScrapingBee pricing page has the full credit table.

FAQ

How do you scrape Indeed job data?

Send the Indeed results URL through a scraping API that renders JavaScript and uses residential proxies, then read the job cards with a CSS selector like div.job_seen_beacon. The Python example above does this in about fifteen lines.

What is the best Indeed scraper approach?

For a maintainable Indeed job scraper, use a managed API for the fetch and residential proxies, and keep the extraction in CSS selectors or server-side extraction rules. That splits the fragile part (getting the HTML) from the easy part (reading fields) and survives most layout changes.

Is it allowed to scrape Indeed?

Scrape only public, pre-login pages and follow the site's terms and applicable law. ScrapingBee prohibits scraping behind a login, so this Indeed web scraper targets the public results pages only.

How do I get more than one page?

Increment the start query parameter by 10 for each page (0, 10, 20, and so on) and send each URL through the same request.

Links

• How to scrape Indeed (ScrapingBee guide)
• ScrapingBee Indeed scraper
• Data extraction rules
• ScrapingBee pricing

License

MIT