How to Scrape Bürklin Product Pages
Learn to discover Bürklin product URLs, extract product data from JSON-LD with Python or cURL, and build a polite, auditable scraper.
To scrape Bürklin product pages, start with ordinary HTTP requests, discover candidate URLs from the site’s sitemap, and parse the page’s application/ld+json Product data before relying on visual selectors. Store the raw response, retrieval time, source and canonical URLs, and parser version alongside each result. Refresh price and availability because they change, and keep requests paced and retries bounded.
This guide shows a small Python scraper, a cURL inspection workflow, and a path to a reliable incremental catalogue job. Bürklin’s catalogue spans electronics categories with different attributes and units, so the extraction model should preserve category-specific data instead of forcing every product into one schema. Bürklin’s own FAQ describes an assortment of more than 500,000 articles; that is a catalogue-scale statement, not a count of URLs you can or should crawl.
1. Define a narrow, responsible crawl
Before collecting pages, define what fields you need and why. For price and availability monitoring, for example, you usually need product identifiers, displayed price and currency, stock text, and retrieval time—not account, checkout, cookie, or analytics data. Bürklin’s privacy policy describes analytics such as page views, referrers, visit duration, visit frequency, and subpages. Keep your collection limited to product metadata relevant to your task.
Also distinguish collection from reuse. Bürklin’s imprint says that website text, images, and graphics are protected by copyright and may not be copied, modified, or used on other websites without express written permission. Scraping a page does not grant permission to republish its text or images. Get the necessary permission before reusing protected material.
Technical data and illustrations can change as manufacturers update products, and the terms warn that photos may be symbolic. Preserve source and retrieval dates; treat captured data as a record of what the page displayed at that time, not as a guaranteed specification or suitability assessment.
2. Discover product URLs from sitemaps
Use the sitemap index and its child sitemap files as your initial discovery source where they are available. A Bürklin-specific Crawlbase recipe reported 13,017 sitemap URLs across 19 files and documented /de/{slug}/{slug} as a successful product-page path shape. Those are measurements from that recipe, not a promise that the current sitemap contains the same URLs or that every matching path is a product.
- Fetch the sitemap index, then fetch each referenced sitemap.
- Keep candidate URLs matching the observed product-page shape, but verify them with canonical metadata and page contents.
- Retain locale and country variants explicitly. Do not silently merge German and English pages.
- Record each URL’s first-seen and last-seen times, and any sitemap last-modified value.
- Deduplicate tracking URLs and language variants using normalized URLs and product identifiers.
A small sitemap parser can provide a seed list. This example reads sitemap XML from a local file; download the sitemap index and child files with your normal HTTP client, then run it on each XML file.
from pathlib import Path
import xml.etree.ElementTree as ET
NS = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
def sitemap_entries(path):
root = ET.parse(path).getroot()
entries = []
for node in root.findall("sm:url", NS):
loc = node.findtext("sm:loc", namespaces=NS)
lastmod = node.findtext("sm:lastmod", namespaces=NS)
if loc:
entries.append({"url": loc, "lastmod": lastmod})
return entries
for entry in sitemap_entries(Path("sitemap.xml")):
if "/de/" in entry["url"]:
print(entry)A sitemap index uses sitemap entries pointing to other XML files rather than url entries. Detect that structure and fetch child files separately. Avoid assuming every URL under a locale path is a product page: categories, search pages, and other content may share a locale prefix.
3. Inspect a page and extract JSON-LD
Fetch a single known product page before building a large job. Inspect its status, headers, canonical link, and JSON-LD blocks. A documented Bürklin recipe says Product JSON-LD usually carries product name, price, currency, and availability, with those values also visible in markup. “Usually” matters: validate missing values and unexpected JSON-LD shapes rather than assuming every page is identical.
Use cURL to inspect a page response:
curl -L --max-time 30 \
-A "Mozilla/5.0 (compatible; ProductMetadataResearch/1.0)" \
"https://www.buerklin.com/de/example-section/example-page" \
-o product.htmlReplace the example path with a URL you have discovered. A descriptive user agent helps identify your client; it does not bypass access controls. The Python example below extracts Product JSON-LD and returns the fields without depending on Bürklin’s visual layout.
import json
import time
from datetime import datetime, timezone
from urllib.parse import urljoin
import requests
from bs4 import BeautifulSoup
HEADERS = {
"User-Agent": "ProductMetadataResearch/1.0 (contact: data@example.org)"
}
def walk_json(value):
if isinstance(value, dict):
yield value
graph = value.get("@graph")
if graph:
yield from walk_json(graph)
elif isinstance(value, list):
for item in value:
yield from walk_json(item)
def extract_product(page_url):
response = requests.get(page_url, headers=HEADERS, timeout=(10, 30))
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")
canonical_tag = soup.find("link", rel="canonical")
canonical_url = urljoin(page_url, canonical_tag["href"]) if canonical_tag and canonical_tag.get("href") else page_url
found = None
for tag in soup.find_all("script", type="application/ld+json"):
try:
document = json.loads(tag.string or tag.get_text())
except (json.JSONDecodeError, TypeError):
continue
for item in walk_json(document):
kind = item.get("@type", [])
kinds = [kind] if isinstance(kind, str) else kind
if "Product" in kinds:
found = item
break
if found:
break
if not found:
return {
"source_url": page_url,
"canonical_url": canonical_url,
"retrieved_at": datetime.now(timezone.utc).isoformat(),
"http_status": response.status_code,
"product_jsonld": None,
}
offers = found.get("offers", {})
if isinstance(offers, list):
offers = offers[0] if offers else {}
return {
"source_url": page_url,
"canonical_url": canonical_url,
"retrieved_at": datetime.now(timezone.utc).isoformat(),
"http_status": response.status_code,
"name": found.get("name"),
"sku": found.get("sku"),
"mpn": found.get("mpn"),
"brand": found.get("brand"),
"price": offers.get("price"),
"currency": offers.get("priceCurrency"),
"availability": offers.get("availability"),
}
if __name__ == "__main__":
product_url = "https://www.buerklin.com/de/example-section/example-page"
result = extract_product(product_url)
print(json.dumps(result, ensure_ascii=False, indent=2))Install the two dependencies with python -m pip install requests beautifulsoup4. The example uses a 10-second connection timeout and a 30-second read timeout, raises on unsuccessful HTTP status, and tolerates malformed JSON-LD blocks. Set a real contact address in the user agent for an operational crawler. Do not treat an empty Product result as proof the page has no product data; inspect a saved response and decide whether the page needs a separate, stable extraction rule.
4. Normalize fields without losing meaning
Use a core product record plus category-specific attributes. Bürklin’s official taxonomy includes semiconductors, passive components, electromechanics, connectors, cables and wires, power supplies, tools and workshop equipment, measurement, automation, and PC accessories. A resistor and a power supply do not share one useful set of technical attributes.
| Field group | Store | Why it matters |
|---|---|---|
| Identity | Source URL, canonical URL, locale, manufacturer, manufacturer part number, Bürklin article number | Lets you match variants and audit the source page. |
| Listing | Product name, category path, price, currency, packaging quantity or unit | Prevents unit and package prices from being compared as if equivalent. |
| Availability | Displayed availability text and lead-time statement | Stock wording and timing may change between captures. |
| Technical attributes | Typed value, unit, original display string, original label | Preserves meaning across languages and category schemas. |
| Provenance | Retrieval timestamp, HTTP status, parser version, raw-response reference | Makes corrections and parser changes auditable. |
Keep original display strings even when also storing normalized values. For example, normalize a numeric value and unit for filtering, but retain the page’s original label and text so a later parser can be checked against what was actually captured. If you store image or datasheet URLs, treat reuse permissions separately from the act of recording a URL.
5. Make the crawler incremental and reliable
Do not repeatedly crawl the entire catalogue to refresh a handful of changing fields. Keep a URL queue and schedule incremental checks for prices and availability, with a slower full-catalogue reconciliation to discover additions and removals. Use sitemap last-modified values as a hint where present, not as a guarantee that every page change is represented.
- Start small: validate your parser on a small sample across multiple categories and locales.
- Pace requests: cap concurrency and avoid bursts. Cache unchanged pages and avoid duplicate fetches.
- Retry deliberately: retry transient failures with backoff; set a small retry limit.
- Keep failures visible: record status, attempt count, and final outcome, then send persistent failures to a review queue.
- Version your parser: save a parser version with each record and retain raw responses for debugging.
The Bürklin-specific recipe reports that 93% of its recorded failures were HTTP 403 responses. It recommends one bounded retry with a country setting used by successful requests, while noting that most successful requests left country unset. That is provider-reported operational data, not an independent audit or a guarantee of future behavior. Start without a country parameter; if a request receives 403, make at most one considered retry and then review the URL instead of looping. Respect access controls and stop if the site denies access.
The same recipe reports a 2.8-second median response for its measured path and says a browser was unnecessary. These are measurements from that recipe, not a service-level promise. Ordinary HTTP is a sensible first attempt for this workflow. A browser adds time and complexity when pages require client-side rendering, but use one only after verifying that ordinary HTML does not contain the data you need.
6. Troubleshooting common failures
| Symptom | Likely cause | Practical response |
|---|---|---|
| 403 Forbidden | The request was denied or the page is protected. | Check that the URL is correct, use a bounded retry if appropriate, then stop and review persistent denials. Do not build an infinite retry loop. |
| 429 Too Many Requests | Your request rate is too high or a limit was reached. | Reduce concurrency, slow the queue, honor any retry guidance, and resume gradually. |
| 5xx or connection timeout | Temporary server or network failure, or an overly short timeout. | Use a bounded backoff retry, retain the failed attempt, and move on if the limit is reached. |
| HTTP 200 but no Product JSON-LD | The page may not contain the expected block, the data shape may have changed, or the URL may not be a product page. | Save and inspect the HTML, check canonical and page type, then add a category or template-specific parser only if justified. |
| JSON parse error | A JSON-LD block may be malformed or contain non-JSON content. | Skip the bad block, log the parse issue, and continue through other blocks. Keep the raw response for review. |
| Unexpected or missing price | Offer data may be nested, absent, or represented differently on a page. | Handle offers as either an object or list; preserve missing values as missing instead of inventing a default. |
| Duplicate products | Locale variants, tracking URLs, or aliases point to the same item. | Retain canonical URL and locale, then deduplicate with normalized product identifiers without discarding language distinctions. |
7. Performance, reliability, and cost
Your main costs are request volume, storage, engineering time, and any paid infrastructure you choose. Estimate the number of pages, refresh frequency, and retries before selecting a provider. Cache responses, skip unchanged URLs when possible, and separate frequent price and stock checks from slower technical-attribute reconciliation. Avoid using browser rendering on every page if ordinary HTTP returns the required data.
Reliability comes from recoverable work: a durable queue, bounded retries, rate limits, raw-response retention, status logging, and idempotent upserts. A failed page should not halt the entire catalogue job. Keep records of the retrieval time and original units so downstream users can distinguish a current observation from an older one.
A managed scraping API can be useful when its retry and scheduling support saves operational effort, but compare the cost per page, geographic controls, throughput, and ability to preserve raw responses. The Crawlbase Bürklin recipe documents a request shape and reports provider-measured success rates; those figures should be read as that provider’s measurements, not a guarantee. For a sitemap-scale job, run a small representative trial and calculate the actual request count and retry rate before committing.
8. Or skip the browser setup
For a screenshot of a product page, use ScreenshotNeo, a website screenshot API and MCP server from Yorker Media. It is a visual capture tool, not a replacement for structured product-field extraction. One GET request returns an image or PDF. See the API documentation for request options.
curl -G "https://api.screenshotneo.com/v1/shot" \
-d access_key=YOUR_API_KEY \
--data-urlencode url=https://www.buerklin.com/de/example-section/example-page \
-o shot.webpimport requests
r = requests.get(
"https://api.screenshotneo.com/v1/shot",
params={
"access_key": "YOUR_API_KEY",
"url": "https://www.buerklin.com/de/example-section/example-page",
},
timeout=90,
)
r.raise_for_status()
open("shot.webp", "wb").write(r.content)const q = new URLSearchParams({
access_key: 'YOUR_API_KEY',
url: 'https://www.buerklin.com/de/example-section/example-page'
});
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);
if (!res.ok) throw new Error(`Screenshot request failed: ${res.status}`);
await Bun.write('shot.webp', res);ScreenshotNeo removes cookie banners, newsletter popups, and chat widgets before capture; each cleanup step can be turned off. Bot checks and CAPTCHAs, blank pages, timeouts, failed loads, and cache hits are not billed, and response headers identify the page verdict and billing status. Its MCP server gives AI agents tools for taking screenshots, getting page information, and capturing PDFs. The free plan includes 1,000 screenshots per month with no card; paid plans start at $5 for 3,000, and every feature is available on every plan. Sign up for 1,000 free screenshots a month, with no card.
FAQ
Can I scrape the whole Bürklin catalogue from one sitemap?
A sitemap can seed URL discovery, but it does not establish that every listed URL is a product page or that the set is complete. Validate page type and reconcile periodically.
Should I store the HTML?
Keeping raw responses or a controlled reference to them makes parser changes auditable. Apply retention and access controls appropriate to your project, and do not collect unrelated account or analytics data.
Can I republish product photos and descriptions?
Bürklin’s imprint says written permission is required to copy, modify, or use its website text, images, and graphics on other websites. Obtain permission before republishing them.
Does a screenshot give me price and stock as structured fields?
No. A screenshot is a visual record. For structured extraction, parse JSON-LD or another page data source and retain the observed text and retrieval time.
