How to Scrape Prices by Country with a Geo-Targeted Proxy

Direct answer: route each request through a proxy or managed scraping API pinned to the target country, render JavaScript when necessary, and record the requested country separately from the geography the site actually served. For every observation, store the product URL or SKU, observed country, price, currency, tax and shipping context, stock, timestamp and extraction status. Normalize currency, VAT or sales tax, shipping, pack size, seller and stock before comparing results.

A country parameter changes the apparent visitor location; it does not guarantee that a site returns its local catalogue or price. Validate the response before treating it as local.

1. Define the comparison

1. Choose a stable product key: SKU or article number first, canonical URL second.
2. List target countries and decide whether you compare consumer or business prices, tax-inclusive or net prices, delivery destinations, quantities and sellers.
3. Define the fields and evidence you will retain.

Field	Purpose
product_id and source_url	Prevents similar listings being confused.
requested_country and observed_country	Shows whether the intended local variant was served.
price, currency and tax_label	Separates the numeric value from tax treatment.
shipping, stock, seller and pack_size	Prevents false cheapest-country conclusions.
captured_at and extraction_status	Makes changes and failures auditable.

2. Check permission and site controls

Read the site’s terms and robots.txt before scheduling requests. Use an official API or permissioned feed when available. CNIL states that web scraping is not, in itself, prohibited under the GDPR, but collection still needs a defined purpose, minimisation, retention and controls. The EDPB says GDPR applies when scraping processes personal data. Avoid accounts, checkout data, names, addresses and other personal data unless you have a documented lawful basis and need it. Stop when a site blocks collection through a CAPTCHA, robots rule or other technical control.

3. Choose a country-targeted route

Compare providers on country coverage, residential versus datacenter routes, city or ISP targeting, JavaScript rendering, session persistence, concurrency, retries, observed-location verification, pricing and compliance support.

Route	Best fit	Trade-off
Residential or managed scraper route	Localized, JavaScript-heavy or protected retail pages.	Usually higher cost and lower raw throughput.
Datacenter proxy	High-volume, less-protected pages with simple HTML.	More likely to be challenged or served a generic variant.
City or ISP targeting	Sites that vary prices or stock below country level.	Fewer endpoints and more validation work.

Provider controls may be expressed as a country parameter such as geoCode=us, or as a proxy object with country and type. Keep this setting in configuration so the extraction code is vendor-neutral.

4. Request a country with cURL

Set PROXY_URL to the authenticated endpoint supplied by your provider.

export PROXY_URL='http://USER:PASSWORD@proxy.example:8000'
export PRODUCT_URL='https://shop.example/product/SKU-123'
curl --fail-with-body --max-time 90 --proxy "$PROXY_URL" \
  -H 'Accept-Language: de-DE,de;q=0.9' \
  -A 'PriceResearchBot/1.0 (contact: ops@example.com)' \
  "$PRODUCT_URL" -o response-de.html

Headers can influence language, but they do not replace a country route. Save the response and request configuration together.

5. Extract and validate in Python

Adapt the selectors to the permitted site. The example records requested and observed geography and refuses to label a result local when validation signals disagree.

import csv, os, re, time
from datetime import datetime, timezone
from urllib.parse import urlparse
import requests
from bs4 import BeautifulSoup

PRODUCT_URL = os.environ['PRODUCT_URL']
REQUESTED_COUNTRY = os.environ.get('REQUESTED_COUNTRY', 'DE')
PROXY_URL = os.environ['PROXY_URL']
PRICE_SELECTOR = os.environ.get('PRICE_SELECTOR', '[itemprop="price"]')
CURRENCY_SELECTOR = os.environ.get('CURRENCY_SELECTOR', '[itemprop="priceCurrency"]')

session = requests.Session()
session.proxies.update({'http': PROXY_URL, 'https': PROXY_URL})
session.headers.update({'Accept-Language': 'de-DE,de;q=0.9', 'User-Agent': 'PriceResearchBot/1.0 (contact: ops@example.com)'})
started = time.monotonic()
r = session.get(PRODUCT_URL, timeout=(15, 75))
r.raise_for_status()
soup = BeautifulSoup(r.text, 'html.parser')

def text(selector):
    node = soup.select_one(selector)
    return node.get('content') if node and node.has_attr('content') else (node.get_text(' ', strip=True) if node else '')

price_text = text(PRICE_SELECTOR)
currency = text(CURRENCY_SELECTOR)
number = re.search(r'\d[\d\s.,]*', price_text)
price_raw = number.group(0) if number else ''
observed_country = text('[data-country], [data-locale]') or ''
country_ok = REQUESTED_COUNTRY.lower() in observed_country.lower() if observed_country else False
row = {'product_id': urlparse(PRODUCT_URL).path.rstrip('/').split('/')[-1], 'source_url': PRODUCT_URL, 'requested_country': REQUESTED_COUNTRY, 'observed_country': observed_country, 'country_validated': country_ok, 'price_raw': price_raw, 'currency': currency, 'tax_label': text('.tax, [data-tax]'), 'shipping': text('.shipping, [data-shipping]'), 'stock': text('[itemprop="availability"], .stock'), 'captured_at': datetime.now(timezone.utc).isoformat(), 'extraction_status': 'ok' if price_raw and country_ok else 'needs_review', 'elapsed_seconds': round(time.monotonic() - started, 2)}
with open('observations.csv', 'a', newline='', encoding='utf-8') as f:
    writer = csv.DictWriter(f, fieldnames=row)
    if f.tell() == 0: writer.writeheader()
    writer.writerow(row)
print(row)

6. Use Node.js for country jobs

Keep concurrency below the target site’s documented limits. Configure an HTTP proxy agent for your provider, or call a managed scraping API that accepts its country parameter.

const countries = [{ code: 'DE', proxy: process.env.PROXY_DE }, { code: 'US', proxy: process.env.PROXY_US }, { code: 'JP', proxy: process.env.PROXY_JP }];
const productUrl = process.env.PRODUCT_URL;
async function fetchCountry({code, proxy}) {
  if (!proxy) throw new Error(`Missing proxy for ${code}`);
  const res = await fetch(productUrl, {headers: {'accept-language': `${code.toLowerCase()}-${code},en;q=0.8`, 'user-agent': 'PriceResearchBot/1.0 (contact: ops@example.com)'}, signal: AbortSignal.timeout(90000)});
  if (!res.ok) throw new Error(`${code}: HTTP ${res.status}`);
  const html = await res.text();
  const price = html.match(/itemprop=["']price["'][^>]*content=["']([^"']+)/i)?.[1] ?? null;
  const currency = html.match(/itemprop=["']priceCurrency["'][^>]*content=["']([^"']+)/i)?.[1] ?? null;
  return {code, price_raw: price, currency, captured_at: new Date().toISOString(), extraction_status: price ? 'needs_country_validation' : 'missing_price'};
}
(async () => { for (const country of countries) { try { console.log(await fetchCountry(country)); } catch (err) { console.error({country: country.code, extraction_status: 'failed', error: String(err)}); } } })();

7. Render JavaScript only when needed

Use raw HTTP when the price is in initial HTML or structured data. Use a browser or managed rendering endpoint when scripts fill the price, a location chooser sets a cookie, or stock appears after interaction. Reuse a stable session for pagination or checkout-like continuity, but do not automate a purchase. Wait for a price selector, a bounded delay or network idle; unbounded waits create failed batches.

8. Validate the served geography

• Site-provided country, store or locale marker.
• Currency and number formatting consistent with the country.
• Tax label, such as VAT-inclusive versus tax excluded.
• Language, shipping destination, catalogue, stock or promotion.
• Response metadata from the proxy or managed API, when available.

Store requested_country and observed_country separately. If signals conflict, mark the row needs_review. A proxy parameter alone is not proof.

9. Normalize before comparing

1. Parse locale separators without losing the raw string.
2. Convert currencies using a documented rate and timestamp.
3. Separate VAT or sales tax from net price where possible.
4. Add or remove shipping consistently for the same destination.
5. Match pack size, quantity, seller, condition, subscription status and stock.
6. Compare observations captured within a defined time window.

10. Reliability, performance and cost

• Retry timeouts and transient 5xx responses with capped exponential backoff.
• Do not retry a CAPTCHA indefinitely; quarantine it for review.
• Start with low concurrency and increase while monitoring status codes and latency.
• Residential bandwidth, browser rendering, retries and JavaScript generally cost more than one datacenter HTML request.
• Measure cost per accepted observation, not requests alone.
• Retain a raw response reference, timestamp, request country and parser version.

11. Common errors and fixes

Symptom	Cause	Fix
Every country has the same price	Proxy route was not applied or the site ignores IP location.	Check provider metadata and site markers; use a permitted managed route or location selector.
Currency changed but tax did not	Language changed while the store remained default.	Validate store, tax label, shipping destination and country together.
Blank HTML or missing price	Client-side rendering or blocked request.	Render JavaScript, wait for the selector and inspect status.
403, 429 or CAPTCHA	Rate limit, bot control or disallowed automation.	Reduce rate, respect controls, use an official feed or stop.
Numbers parse incorrectly	Locale separators were treated as one format.	Keep raw text and use locale-aware parsing.
Price differs between runs	Promotion, stock, seller, cookie or experiment changed.	Record context and compare like-for-like within a time window.

12. Or skip the browser setup

For visual evidence of a rendered page, ScreenshotNeo returns PNG, JPEG, WebP or PDF from one GET request. Use it after selecting the country-specific URL or session in your own workflow; its documented endpoint does not claim to set a proxy country.

See the ScreenshotNeo API documentation.

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://shop.example/product/SKU-123 -o shot.webp

import requests; r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://shop.example/product/SKU-123"}, timeout=90); open("shot.webp", "wb").write(r.content)

const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://shop.example/product/SKU-123' }); const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);

Cookie banners, popups and chat widgets are removed before the shot; bot checks, blank pages and failed loads are never billed; an MCP server lets AI agents take screenshots; 1,000 screenshots a month are free with no card and paid plans start at $5 for 3,000. Create a free ScreenshotNeo account.

13. FAQ

Will a country proxy always show the local price?

No. Account, cookie, shipping, catalogue, experiment and fraud signals can override IP location.

Should I use residential or datacenter proxies?

Residential or managed routes usually fit localized or protected pages; datacenter routes can fit high-volume, less-protected HTML.

Do I need a browser?

Only when scripts, consent flows, location cookies or interactions create the price.

How should I compare EU and US prices?

Align tax inclusion, conversion timestamp, shipping destination, pack size, seller, stock and promotion.