How to Scrape Product Offer Tables Across Multiple Countries

To scrape product offer tables across multiple countries, model each offer as a market-specific record. Keep the country, language, currency, original price text, numeric price, availability, condition, shipping information and source URL together. Do not treat a price as globally valid just because the product name matches.

Use this collection order:

1. Documented feed or commerce API.
2. Product and Offer structured data such as JSON-LD.
3. Rendered HTML table or product page.
4. Browser automation only when the data appears after JavaScript, consent handling or interaction.

This order usually produces more stable inputs and less localization parsing. Google documents offer price, currency, availability and country-scoped shipping fields. eBay product feeds expose price, marketplace currency, availability, condition and item URL. Shopify and WooCommerce document market-specific feeds or prices. Treat those schemas as your target shape even when your source is a rendered page.

1. Define the market matrix and offer identity

Start with a matrix of every country and storefront you intend to collect. Record the target country, language, currency, URL, retrieval time and the method used to select the market. A localized site may use a country subdomain, a path, a cookie, an account setting, a shipping destination or a combination of these.

Field	Purpose
product_key	Stable observed SKU, GTIN, product ID or canonical identifier.
offer_key	Identifier for a seller, variant or market-specific offer.
country	ISO country code selected for the offer.
language	Locale used to render labels and formatted numbers.
source_url	Exact localized URL collected.
price_raw	Original text or value, preserved for auditing.
price_value	Parsed decimal value, without silently converting currencies.
currency	Three-letter ISO 4217 code such as EUR, GBP or USD.
availability	Original status and, optionally, a controlled value such as InStock.
condition	New, used, refurbished or the source’s original label.
shipping	Destination, cost, delivery estimate and tax wording when exposed.
retrieved_at	Timestamp in UTC.
parser_version	Version of the extraction code used for the row.

Keep product identity separate from offers. One product can have several country URLs, variants and merchants. Do not merge rows solely because translated titles look similar. A comparison price can be derived later, but the original market values must remain unchanged.

2. Prefer feeds, APIs and structured data

Inspect the source’s documentation and page source before writing CSS selectors. Product data feeds and APIs can provide update mechanisms and explicit fields that a visual table does not. JSON-LD commonly appears in a <script type="application/ld+json"> element containing Product, Offer or an AggregateOffer.

For each country, compare:

• Coverage: Does the source include every product, variant and seller?
• Localization: Is country, language, currency and delivery destination explicit?
• Completeness: Are condition, availability, shipping and tax terms present?
• Freshness: How are updates delivered, and can you identify stale records?
• URL specificity: Is the URL tied to a country and currency?
• Access conditions: Are credentials, rate limits or permission requirements documented?

Google’s product guidance recommends distinct URLs when a product is offered in multiple currencies and focuses product markup on a single product or its variants rather than a general category listing. If you need a category table, use the listing only to discover product URLs, then collect each product page or feed.

3. A Python scraper for JSON-LD and offer tables

The following script first extracts JSON-LD, then falls back to a table with explicit column mappings. It preserves raw values and applies a locale-aware decimal parser for common comma and period conventions. It does not perform currency conversion.

import json
import re
from datetime import datetime, timezone
from decimal import Decimal
from urllib.parse import urljoin

import requests
from bs4 import BeautifulSoup

MARKETS = [
    {"country": "US", "language": "en", "currency": "USD", "url": "https://example.com/us/products"},
    {"country": "DE", "language": "de", "currency": "EUR", "url": "https://example.com/de/produkte"},
]

HEADERS = {"User-Agent": "OfferCollector/1.0 (contact: data@example.com)"}

def parse_decimal(raw):
    if raw is None:
        return None
    value = re.sub(r"[^0-9,.-]", "", str(raw)).strip()
    if not value:
        return None
    # Use the last separator as the decimal separator when both appear.
    if "," in value and "." in value:
        if value.rfind(",") > value.rfind("."):
            value = value.replace(".", "").replace(",", ".")
        else:
            value = value.replace(",", "")
    elif "," in value:
        value = value.replace(",", ".")
    try:
        return str(Decimal(value))
    except Exception:
        return None

def jsonld_objects(soup):
    for node in soup.select('script[type="application/ld+json"]'):
        try:
            data = json.loads(node.string or node.get_text())
        except json.JSONDecodeError:
            continue
        values = data if isinstance(data, list) else [data]
        for value in values:
            if isinstance(value, dict) and "@graph" in value:
                values.extend(value["@graph"])
            if isinstance(value, dict):
                yield value

def offers_from_jsonld(soup, market):
    rows = []
    for obj in jsonld_objects(soup):
        if obj.get("@type") not in ("Product", "Offer", "AggregateOffer"):
            continue
        product = obj if obj.get("@type") == "Product" else {}
        offers = obj.get("offers") if product else obj
        if isinstance(offers, list):
            offer_list = offers
        elif isinstance(offers, dict):
            offer_list = [offers]
        else:
            offer_list = []
        for offer in offer_list:
            if not isinstance(offer, dict):
                continue
            rows.append({
                "product_key": product.get("sku") or product.get("gtin") or product.get("name"),
                "offer_key": offer.get("sku") or offer.get("url"),
                "country": market["country"],
                "language": market["language"],
                "source_url": market["url"],
                "price_raw": offer.get("price"),
                "price_value": parse_decimal(offer.get("price")),
                "currency": offer.get("priceCurrency") or market["currency"],
                "availability": offer.get("availability"),
                "condition": offer.get("itemCondition"),
                "shipping": offer.get("shippingDetails"),
                "retrieved_at": datetime.now(timezone.utc).isoformat(),
                "parser_version": "jsonld-1",
            })
    return rows

def offers_from_table(soup, market):
    rows = []
    for tr in soup.select("table tbody tr"):
        cells = [c.get_text(" ", strip=True) for c in tr.select("th, td")]
        if len(cells) < 3:
            continue
        # Adapt these indexes to the documented table for the target site.
        rows.append({
            "product_key": cells[0],
            "offer_key": cells[1],
            "country": market["country"],
            "language": market["language"],
            "source_url": market["url"],
            "price_raw": cells[2],
            "price_value": parse_decimal(cells[2]),
            "currency": market["currency"],
            "availability": cells[3] if len(cells) > 3 else None,
            "condition": cells[4] if len(cells) > 4 else None,
            "shipping": cells[5] if len(cells) > 5 else None,
            "retrieved_at": datetime.now(timezone.utc).isoformat(),
            "parser_version": "table-1",
        })
    return rows

all_rows = []
for market in MARKETS:
    response = requests.get(market["url"], headers=HEADERS, timeout=30)
    response.raise_for_status()
    soup = BeautifulSoup(response.text, "html.parser")
    rows = offers_from_jsonld(soup, market)
    if not rows:
        rows = offers_from_table(soup, market)
    all_rows.extend(rows)

print(json.dumps(all_rows, indent=2, ensure_ascii=False))

For production, replace the example table indexes with selectors validated against the target site. Store the HTML or feed response when policy permits so a parser change can be investigated later.

4. Browser rendering for JavaScript tables

Use a browser when rows are inserted after JavaScript, when a country selector changes cookies or local storage, or when the page requires a consent interaction before the table becomes visible. Playwright is a practical choice for a Node.js implementation.

import { chromium } from 'playwright';

const markets = [
  { country: 'US', locale: 'en-US', url: 'https://example.com/us/products' },
  { country: 'DE', locale: 'de-DE', url: 'https://example.com/de/produkte' }
];

const browser = await chromium.launch();
const context = await browser.newContext({ locale: 'en-US' });
const page = await context.newPage();
const result = [];

for (const market of markets) {
  await page.goto(market.url, { waitUntil: 'networkidle', timeout: 60000 });
  await page.waitForSelector('table tbody tr', { timeout: 30000 });
  const rows = await page.locator('table tbody tr').evaluateAll((trs) =>
    trs.map((tr) => [...tr.querySelectorAll('th,td')].map((cell) => cell.textContent.trim()))
  );
  for (const cells of rows) {
    result.push({ market: market.country, source_url: market.url, cells });
  }
}

await browser.close();
console.log(JSON.stringify(result, null, 2));

Set the country through the documented URL or site control. If a selector changes a cookie, create a separate browser context per country so state cannot leak between markets. Save a screenshot and the final URL for failed samples, but avoid collecting personal data that is not needed for the offer record.

5. cURL checks and lightweight extraction

cURL is useful for checking whether a localized URL returns the expected language, redirect chain and structured data before you schedule a scraper.

curl -L --compressed \
  -H 'Accept-Language: de-DE,de;q=0.9' \
  -A 'OfferCollector/1.0' \
  'https://example.com/de/produkte' \
  -o de-products.html

rg -n 'application/ld\+json|priceCurrency|availability|shipping' de-products.html

Do not infer currency from a symbol alone. A dollar sign can represent different currencies, and a localized page may show a converted display price while the underlying offer uses another currency. Require an ISO currency field from markup, a feed, a documented market setting or a controlled mapping that you record in your audit trail.

6. Normalize without losing the source value

Keep price_raw and price_value side by side. Parse according to the page locale: 1.234,56 and 1,234.56 represent the same number in different conventions. When both separators appear, use the last separator as the decimal marker only when that rule has been validated for the market.

If you convert prices, put the result in a derived field such as comparison_value. Record the conversion date, source and rate. Never overwrite the original amount or currency. Keep tax-inclusive and tax-exclusive amounts distinct, and do the same for shipping. Google’s localization guidance ties target countries to language, price, currency, delivery and tax information.

7. Validation and deduplication checklist

• Verify at least one known product in every country and language.
• Check that the source URL, selected market and returned currency agree.
• Confirm that availability and shipping belong to the same destination as the price.
• Compare a sample with the visible page or source feed after every markup change.
• Flag missing currencies, impossible decimal values and negative prices.
• Deduplicate by a stable product or offer identifier plus country and seller.
• Keep multiple variants separate when size, color or condition changes the offer.
• Record parser version and retrieval time for every row.

8. Troubleshooting common failures

Symptom	Likely cause	Fix
Every country has the same price	Country cookie or redirect state was reused.	Use a separate context or session per market and verify the final URL.
Price parses as 123456	Locale separators were removed incorrectly.	Preserve raw text and apply a market-specific decimal parser.
JSON-LD has no offers	The page is a category listing, incomplete markup or JavaScript-only.	Discover product URLs, use a feed, or render the page.
Availability is stale	Cached HTML or a feed with delayed updates.	Check update timestamps, lower cache duration and compare with a live sample.
Shipping is missing	Shipping appears only after a destination is selected.	Set the documented destination before extraction and store the destination with the value.
Rows are duplicated	Variants, sellers or repeated mobile and desktop markup were merged.	Use product, variant, seller, country and source URL in the deduplication key.
Browser waits forever	A never-ending request prevents network-idle detection.	Wait for the table selector with a bounded timeout and continue after known analytics requests.
403 or CAPTCHA response	The site requires permission, authentication or a browser challenge.	Use an approved feed or API, review the site’s rules, and do not attempt to bypass access controls.

9. Performance, reliability and cost

Prefer feeds and APIs for large catalogs because one response can contain many offers and avoids launching a browser for every product. For rendered pages, reuse a browser process, limit concurrency per domain, cache unchanged URLs and retry only transient failures with exponential backoff. Keep a per-country queue so one slow market cannot block all others.

Cache carefully. A cached response can reduce load and cost, but it can also make availability stale. Store the retrieval timestamp and cache age, and make the freshness requirement explicit for each use case. For price monitoring, a scheduled sample may be enough; checkout or inventory decisions need a shorter freshness window and a verification step.

Estimate cost from requests, browser minutes, proxy or feed charges, storage and currency conversion. Measure parse failure rate by market and alert when a field suddenly becomes empty. Do not claim accuracy from a small sample: validate against the source and report coverage and freshness separately.

10. Or skip the browser setup with ScreenshotNeo

ScreenshotNeo can capture a localized page when your workflow needs a visual record of each offer table. Its API accepts one GET request and returns PNG, JPEG, WebP or PDF. See the ScreenshotNeo documentation for all parameters.

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp

import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)

const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);

Adapt the target URL for each country and pass the relevant headers, cookies, user agent, timezone or geolocation when the site documents those controls. ScreenshotNeo supports full-page capture with lazy images loaded, CSS selector element capture, custom CSS and JavaScript, click actions, selector or network-idle waits, request blocking, caching with a chosen TTL, bulk capture of up to 100 URLs per call, PDFs and signed webhooks for asynchronous jobs.

Cookie banners, newsletter popups and chat widgets are removed before the shot. Bot checks, blank pages and failed loads are never billed, and response headers report the page verdict and whether it was billed. An MCP server lets Claude, Cursor and other MCP clients call take_screenshot, get_page_info and capture_pdf. The Free plan includes 1,000 screenshots each month with no card; paid plans start at $5 for 3,000. Create a free ScreenshotNeo account.

11. Short FAQ

How do I compare product prices in different countries?

Keep each amount with its ISO currency and country, then create a separate converted comparison field using a recorded rate and date. Include tax and shipping assumptions in the comparison.

Should I scrape a category table or each product page?

Use category pages for discovery when necessary. Collect the final offer from a product page, feed or API when you need variant, condition, availability or shipping details.

How do I scrape prices in different currencies?

Capture the localized URL and market selection, preserve the displayed string, parse it with the locale’s rules and require an explicit currency code. Never infer currency from the symbol alone.

When is browser automation necessary?

Use it when JavaScript inserts the rows, a destination selector changes the offer, or consent and interaction are required. Otherwise, a documented feed, API or structured data is usually easier to maintain.

Is cross-country scraping always allowed?

Check the target site’s terms, robots directives, API agreement and applicable law before collecting data. The permission status depends on the specific site and jurisdiction.