How to Scrape Job Postings With Python
Collect job posting data with Python by choosing permitted sources, extracting stable fields, handling pagination and duplicates, and saving clean records.

To scrape job postings with Python, first choose a source and collection method you are allowed to use. Prefer an official API or partner integration when available. For a permitted server-rendered listing page, request its HTML with a timeout and conservative rate, parse stable fields with Beautiful Soup, follow pagination, deduplicate records, and save them to CSV. Use a browser automation tool only when the source permits it and the page genuinely requires JavaScript rendering.
This guide builds a small, runnable collector for a permitted HTML source. Job boards have different terms and access rules; the code is a template for a source you are authorized to collect from, not a method for bypassing access controls.
1. Choose a source and confirm permitted access
Before writing a scraper, check whether the source offers an API, partner integration, or export designed for your use. Indeed documents APIs for jobs, candidates, employers, and search integrations. Its Job Sync API is a GraphQL API for ATS partners to create, update, expire, and check posting status. LinkedIn documents an approval and vetting process for Job Posting API integrations. These routes have their own eligibility and terms; access to a documented API does not grant permission to use it for unrelated collection.
Review the source’s current terms and robots directives for your intended collection. LinkedIn’s crawling terms prohibit automated crawling and indexing without express permission and require permitted crawling to follow authorized paths and robot-exclusion restrictions. LinkedIn’s recruiter help says third-party software, crawlers, bots, browser plug-ins, and scripts that scrape or automate activity are not permitted on its services. Indeed’s developer agreement restricts copying, redistribution, unauthorized purposes, permanent database creation, algorithmic query generation, and attempts to bypass access limits. Read the applicable rules directly before collecting.
- Indeed developer documentation and Job Sync API.
- LinkedIn Job Posting API terms, crawling terms, and prohibited software guidance.
Do not interpret a page being publicly viewable as blanket permission to automate collection, store its contents, or redistribute them. If the rules are unclear, ask the source or use its approved integration.
2. Pick the right Python approach
| Approach | Use it when | Main tradeoff |
|---|---|---|
| Official API | The source documents an API that fits your use and you meet its access requirements. | Requires following API authentication, quotas, partner terms, and pagination model. |
| Requests + Beautiful Soup | The permitted page returns listing content in its initial HTML. | Simple to operate, but selectors can break when markup changes. |
| Scrapy | You have permission and need to traverse many pages with queues, retries, and item pipelines. | More structure and configuration than a one-page script. |
| Playwright or Selenium | Permitted content only appears after client-side JavaScript renders it. | Browser processes use more resources and add wait, rendering, and failure modes. |
Start by opening a single listing page and inspecting its HTML. Look for stable semantic attributes, embedded structured data such as JSON-LD, or documented API fields. Prefer source IDs and canonical links over positional selectors like “the third div.” The referenced Web Scraping with Python resource covers Beautiful Soup, Scrapy, Selenium, Requests, and related techniques.
3. Install dependencies and set the source
This example expects a permitted HTML page whose listing cards use CSS classes that you will adapt after inspecting that source. It extracts title, employer, location, description, employment type, salary when displayed, and the posting URL. Fields absent from a page remain empty; the script does not infer missing salary or other facts.
python -m venv .venv
# macOS or Linux:
source .venv/bin/activate
# Windows PowerShell:
# .venv\Scripts\Activate.ps1
python -m pip install requests beautifulsoup4
Set a URL you are authorized to collect from. Never point this example at a service whose rules prohibit the automation.
export JOBS_URL='https://example.org/permitted-jobs'
# PowerShell: $env:JOBS_URL = 'https://example.org/permitted-jobs'
4. Fetch, parse, paginate, and export
import csv
import os
import time
from datetime import datetime, timezone
from urllib.parse import urljoin, urlparse, urldefrag
import requests
from bs4 import BeautifulSoup
START_URL = os.environ.get("JOBS_URL", "https://example.org/permitted-jobs")
USER_AGENT = "JobResearchCollector/1.0 (contact: data-team@example.org)"
MAX_PAGES = 5 # Set this to the permitted scope for your task.
REQUEST_DELAY_SECONDS = 2
TIMEOUT_SECONDS = 20
FIELDS = [
"title", "employer", "location", "description", "employment_type",
"salary", "published_at", "posting_url", "source_url", "retrieved_at",
]
session = requests.Session()
session.headers.update({"User-Agent": USER_AGENT, "Accept": "text/html"})
def text_or_empty(node):
return " ".join(node.stripped_strings) if node else ""
def canonicalize(url):
absolute = urljoin(START_URL, url)
return urldefrag(absolute)[0]
def fetch_html(url):
response = session.get(url, timeout=TIMEOUT_SECONDS)
response.raise_for_status()
content_type = response.headers.get("Content-Type", "")
if "html" not in content_type.lower():
raise ValueError(f"Expected HTML at {url}, got {content_type!r}")
return response.text
def parse_page(html, page_url):
soup = BeautifulSoup(html, "html.parser")
records = []
# Replace these example selectors with stable selectors from the permitted source.
for card in soup.select("article.job-card"):
link = card.select_one("a.job-card__title")
if not link or not link.get("href"):
continue
title_node = card.select_one(".job-card__title")
employer_node = card.select_one(".job-card__employer")
location_node = card.select_one(".job-card__location")
description_node = card.select_one(".job-card__description")
type_node = card.select_one(".job-card__type")
salary_node = card.select_one(".job-card__salary")
date_node = card.select_one("time[datetime]")
records.append({
"title": text_or_empty(title_node),
"employer": text_or_empty(employer_node),
"location": text_or_empty(location_node),
"description": text_or_empty(description_node),
"employment_type": text_or_empty(type_node),
"salary": text_or_empty(salary_node),
"published_at": date_node.get("datetime", "") if date_node else "",
"posting_url": canonicalize(link["href"]),
"source_url": page_url,
"retrieved_at": datetime.now(timezone.utc).isoformat(),
})
next_link = soup.select_one("a[rel='next']")
next_url = canonicalize(next_link["href"]) if next_link and next_link.get("href") else ""
return records, next_url
def collect(start_url):
seen_pages = set()
records_by_url = {}
page_url = start_url
for page_number in range(MAX_PAGES):
page_url = canonicalize(page_url)
if page_url in seen_pages:
break
seen_pages.add(page_url)
html = fetch_html(page_url)
records, next_url = parse_page(html, page_url)
for record in records:
key = record["posting_url"]
# Later sightings replace earlier copies, keeping the latest retrieved record.
records_by_url[key] = record
if not next_url:
break
# Stop rather than following pagination onto another host.
if urlparse(next_url).netloc != urlparse(start_url).netloc:
raise ValueError(f"Pagination left the source host: {next_url}")
time.sleep(REQUEST_DELAY_SECONDS)
page_url = next_url
return list(records_by_url.values())
if __name__ == "__main__":
rows = collect(START_URL)
with open("jobs.csv", "w", newline="", encoding="utf-8-sig") as f:
writer = csv.DictWriter(f, fieldnames=FIELDS, extrasaction="ignore")
writer.writeheader()
writer.writerows(rows)
print(f"Saved {len(rows)} unique postings to jobs.csv")
Replace the sample selectors and card structure with those observed on your permitted source. The script follows only a same-host next link, stops on repeated pages, caps page count, and deduplicates by canonical posting URL. Its two-second pause is a conservative example setting, not a universal safe rate; follow the source’s documented limits and reduce collection or stop if it signals blocking or its rules change.

5. Make records useful and auditable
For analysis, a record normally includes job title, employer, location, canonical posting URL or source ID, description, employment type, compensation fields when present, publication or update time when shown, source, and retrieval timestamp. Preserve the raw source value when normalizing. For example, keep the displayed salary string in one column and add a separate normalized minimum, maximum, currency, and period only when the page clearly provides those values. Do not convert an unknown salary to zero or infer whether a number is annual or hourly.
CSV is convenient for a modest one-time export. SQLite is often a better next step when you need repeatable updates, uniqueness constraints, and queries. Store a source ID or canonical URL as a deduplication key, and decide whether a later observation updates the existing row or becomes a history record. Keep retrieval time separate from the source’s publication time. If permitted, retain enough response metadata to investigate parsing changes without storing more page content than your policy allows.
When to move to Scrapy or an API
Use Scrapy when you need a maintained crawl queue, item pipelines, retry handling, or a larger set of permitted pages. Keep the same scope controls: allowed domains, request pacing, page limits, and stop conditions. Choose the official API when it expresses the data and use you need; implement its documented authentication, cursor or page token, quota behavior, and permitted storage rules. Do not assume an API removes contractual limits.
6. JavaScript-rendered pages
If the permitted page’s initial HTML lacks the listing and the data appears only after scripts run, first check for an approved API or structured data source. When the source allows browser automation, Playwright can render the page and pass its HTML to the same parser. Install Playwright and its browser runtime using the project’s documented install command, then use a deliberately narrow page scope and wait for a meaningful selector rather than an arbitrary long sleep.

import asyncio
from playwright.async_api import async_playwright
async def rendered_html(url):
async with async_playwright() as p:
browser = await p.chromium.launch(headless=True)
page = await browser.new_page()
try:
await page.goto(url, wait_until="domcontentloaded", timeout=30000)
await page.locator("article.job-card").first.wait_for(timeout=10000)
return await page.content()
finally:
await browser.close()
async def main():
html = await rendered_html(START_URL)
records, _ = parse_page(html, START_URL)
print(f"Parsed {len(records)} postings")
if __name__ == "__main__":
asyncio.run(main())
The selector in the wait call is still an example and must match the permitted page. A timeout can mean the selector changed, the page did not load, or the content is not available to your session. Do not try to evade a CAPTCHA, bot check, login boundary, or other access control; stop and use an authorized route.
7. Pagination, duplicates, and changing listings
Pagination can use a next link, numbered pages, cursor tokens, or an API response field. Follow only the documented or clearly exposed mechanism within the authorized scope. Set a maximum page or result count, detect repeated URLs and cursors, and stop when there is no next value. A query that changes on each request can produce duplicates; canonicalize URLs by removing fragments and use a source posting ID when available. Avoid stripping query parameters unless the source’s URL semantics show they are tracking-only.
Job listings change: salary may be added, a role may expire, or a description may be edited. Decide whether your dataset is a current snapshot or a history. For a snapshot, update a matching key and track when it was last seen. For history, record observations with retrieval timestamps and avoid treating a missing listing as proof of deletion until the source’s pagination and update behavior are understood.
8. Reliability, performance, and cost
For a small collector, network waits and page rendering usually dominate parser work. Fetch one page at a time unless the source explicitly permits concurrency; concurrency raises request load and can make failures harder to diagnose. Reuse a Requests session, set timeouts, and keep output incremental for long runs. If you add retries, use a bounded retry count with backoff for transient server failures, and honor any retry guidance and rate limits. Do not retry access-denied or blocking responses in a loop.
Watch status codes, content type, page count, records per page, duplicate rate, and missing-field rate. A successful HTTP response can still contain an error page or changed markup, so validate that expected listing structures are present before treating an empty result as a valid zero. Pause collection when the source signals blocking or changes its rules. Browser automation costs more memory and setup than parsing already-rendered HTML, so use it only when permitted and needed.
There is no universal cost estimate: it depends on volume, infrastructure, browser use, and the source’s access model. Budget for storage, scheduled runs, parser maintenance, and review of terms. The lowest operational cost is often an authorized API or export that avoids fragile page parsing; compare its access requirements and permitted uses before choosing.
9. Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| HTTP 403 or access denied | The source disallows the request, requires an approved route, or has restricted access. | Stop automated requests. Check the source’s terms and API or partner path; do not attempt to bypass the restriction. |
| HTTP 429 | Request limit reached or the request rate is too high. | Stop or slow down, honor documented retry guidance, and reduce scope. Do not immediately repeat requests. |
| CSV has headers but no rows | Selectors no longer match, content is rendered by JavaScript, or the page returned a non-listing response. | Inspect one permitted response, check status and content type, then update selectors or use an allowed rendering/API method. |
| Missing salary or location | The source does not display the field on every posting or uses a different card/detail layout. | Represent absence as empty, and inspect permitted detail pages only if your scope allows it. |
| Repeated pages or duplicate jobs | Pagination loops, tracking parameters differ, or multiple pages include the same posting. | Track visited page URLs and cursors; deduplicate on canonical URL or source ID. |
| Playwright selector timeout | Selector changed, rendering failed, or the listing is unavailable to the session. | Inspect the page structure and load state. If access is blocked, stop and use an authorized route. |
| CSV text looks garbled in a spreadsheet | Encoding detection differs by spreadsheet application. | Try UTF-8 import explicitly; the example writes a UTF-8 BOM for common spreadsheet compatibility. |
10. ScreenshotNeo: capture permitted listing pages without browser setup
If your job research workflow also needs page snapshots for review or archiving, ScreenshotNeo is a website screenshot API and MCP server. It does not scrape or grant permission to collect job data; use the source’s authorized data route for records. For a permitted page where a visual capture is useful, one GET request returns an image or PDF. The API uses a single URL parameter and the parameter names used by other screenshot APIs also work. See the ScreenshotNeo API documentation.
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://example.org/permitted-jobs -o shot.webp
import requests
r = requests.get(
"https://api.screenshotneo.com/v1/shot",
params={"access_key": "YOUR_API_KEY", "url": "https://example.org/permitted-jobs"},
timeout=90,
)
r.raise_for_status()
open("shot.webp", "wb").write(r.content)
const q = new URLSearchParams({
access_key: 'YOUR_API_KEY',
url: 'https://example.org/permitted-jobs'
});
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);
if (!res.ok) throw new Error(`Screenshot request failed: ${res.status}`);
await Bun.write('shot.webp', new Uint8Array(await res.arrayBuffer()));
Cookie banners, newsletter popups, and chat widgets are removed before capture, and each cleanup step can be turned off. Bot checks or CAPTCHAs, blank pages, timeouts, failed loads, and cache hits are not billed; response headers identify the page verdict and billing status. An MCP server lets AI agents use screenshot, page information, and PDF capture tools. Plans include 1,000 screenshots per month free without a card; paid plans start at $5 for 3,000. Screenshot output is a visual record, not structured job data, and it must still be used within the page source’s rules. Sign up for 1,000 free screenshots a month, with no card required.
11. FAQ
Can I scrape Indeed or LinkedIn with Beautiful Soup?
Beautiful Soup can parse HTML technically, but that does not establish permission. Check the source’s current terms and use documented APIs or approved integrations where applicable. LinkedIn’s guidance restricts unauthorized crawling and scraping software.
Should I store job descriptions forever?
Only if the source’s terms and your purpose permit it. Indeed’s developer agreement specifically restricts permanent database creation for covered use. Review terms for the source and intended storage.
Can this code find every job on a board?
No. It follows a bounded set of exposed next links and extracts fields matching the page structure. Availability and scope depend on the source’s permissions, pagination, and data presentation.
What if a field is not shown?
Keep it blank or explicitly mark it unknown. Do not infer compensation, posting date, or employment type from unrelated fields.


