How to Prevent Duplicate Hotel Prices When Scraping with Puppeteer

To prevent duplicate hotel prices with Puppeteer, make the search context deterministic, wait until complete result cards are visible, extract each offer from one card scope, and deduplicate using a key that preserves the details that make rates different. Do not deduplicate on hotel name and amount alone: occupancy, room, taxes, cancellation terms, meal plan, provider, and itinerary can all distinguish valid offers.

This guide shows a complete Puppeteer pattern, explains why duplicates occur, and covers normalization, pagination, validation, troubleshooting, and production rechecking. The selectors below are examples; inspect the target site and replace them with stable attributes or structured data that it actually exposes.

1. Fix the search context before scraping

Two rates are comparable only when the search inputs match. Set and record check-in, check-out or nights, occupancy, currency, locale, and provider context. Also record the URL and retrieval time. A site may return different prices when the country, device, occupancy, or request time changes, and cached rates can differ from a newly repriced offer.

Google defines a hotel price in the context of a double-occupancy room for a particular check-in date and number of nights. Its rate model supports multiple room, rate-plan, occupancy, and refundability combinations. Those are meaningful dimensions, not noise to erase. See Google Hotel Prices pricing overview and Google price accuracy guidance.

1. Choose fixed dates and occupancy for each run.
2. Set locale, currency, and provider parameters explicitly where the site supports them.
3. Use a stable search URL or build the URL from recorded inputs.
4. Attach a run or navigation identifier to each extraction batch so a delayed response from an earlier search cannot contaminate the current one.

For integrations spanning dates and stays, the search space grows quickly. Google’s general pricing guidance describes 330 advance-booking days and stays up to 30 nights, or 9,900 itinerary entries. Keep your own crawl scope intentional and authorized.

2. Wait for result cards and price completeness

DOMContentLoaded only indicates that the initial document was parsed. Hotel rates are often fetched afterward. Wait first for a visible result container, then for network quiet if appropriate, then for a site-specific condition that means cards are complete. Puppeteer documents page.waitForSelector() visibility and timeout options, and page.waitForNetworkIdle() waits for the network to be idle for at least the configured idle time. Network quiet alone does not prove prices are final.

await page.goto(url, { waitUntil: 'domcontentloaded' });
await page.waitForSelector('[data-hotel-card]', {
  visible: true,
  timeout: 30_000
});
await page.waitForNetworkIdle({ idleTime: 500, timeout: 15_000 });
await page.waitForFunction(() => {
  const cards = [...document.querySelectorAll('[data-hotel-card]')]
    .filter(el => el.offsetParent !== null);
  return cards.length > 0 && cards.every(card =>
    /\d/.test(card.querySelector('[data-total]')?.textContent ?? '')
  );
}, { timeout: 20_000 });

Adjust timeouts to the site and your own service-level needs. If cards can legitimately load without a price, the predicate should check the subset of cards the page marks complete, rather than wait forever for every card. Puppeteer references: waitForSelector and waitForNetworkIdle.

3. Extract one offer from one card

A common source of apparent duplicates is querying a price selector across the whole document. The same displayed text may appear in a desktop card, a mobile clone, a sticky summary, a hidden template, or a modal. Select repeated result cards first, filter to visible cards, and read fields within each card. Return plain objects in a single page-context evaluation.

const raw = await page.$$eval('[data-hotel-card]', cards => cards
  .filter(card => card.offsetParent !== null)
  .map(card => ({
    provider: card.dataset.provider ?? '',
    hotelId: card.dataset.hotelId ?? '',
    hotelName: card.querySelector('[data-hotel-name]')?.textContent ?? '',
    roomId: card.dataset.roomId ?? '',
    roomName: card.querySelector('[data-room-name]')?.textContent ?? '',
    ratePlanId: card.dataset.ratePlanId ?? '',
    checkIn: card.dataset.checkIn ?? '',
    nights: card.dataset.nights ?? '',
    occupancy: card.dataset.occupancy ?? '',
    currency: card.dataset.currency ?? '',
    totalText: card.querySelector('[data-total]')?.textContent ?? '',
    taxesText: card.querySelector('[data-taxes]')?.textContent ?? '',
    feesText: card.querySelector('[data-fees]')?.textContent ?? '',
    mealPlan: card.querySelector('[data-meal-plan]')?.textContent ?? '',
    cancellationText: card.querySelector('[data-cancellation]')?.textContent ?? '',
    sourceUrl: location.href,
    scrapedAt: new Date().toISOString()
  })));

$$eval applies a function to all matches and returns serializable data; $eval addresses only the first match. Puppeteer’s page.evaluate() runs in the page context and awaits returned promises. See the page interaction guide and evaluate API. Avoid returning DOM nodes: extract strings and primitive values while still in the page context.

4. Normalize fields, then deduplicate semantic offers

Preserve raw values for audit and retain normalized values for comparison. Normalize Unicode and whitespace; parse numbers according to the page locale rather than removing punctuation blindly; canonicalize dates and ISO currency codes; and map cancellation text to structured policy categories while keeping its original wording. For example, 1.299,00 and 1,299.00 can mean the same amount in different locales.

A useful canonical record includes:

• provider, hotel ID and name
• room ID and name, rate-plan ID
• check-in date, nights, occupancy
• currency, total, nightly amount, tax and fee inclusion
• meal plan, refundability, cancellation text and normalized policy
• source URL and retrieval timestamp

Construct a stable key from provider and hotel identity, room identity, rate plan, itinerary, occupancy, currency, total, tax/fee inclusion, cancellation or refundability, and meal plan. Use a serialization with an explicit field order or a stable hash of that serialization.

function clean(value = '') {
  return value.normalize('NFKC').replace(/\s+/g, ' ').trim();
}

function makeSemanticOfferKey(o) {
  // totalMinor must be parsed with the page's locale and currency rules.
  return JSON.stringify([
    clean(o.provider), clean(o.hotelId), clean(o.roomId), clean(o.ratePlanId),
    o.checkIn, Number(o.nights), clean(o.occupancy), o.currency,
    o.totalMinor, o.taxIncluded, o.feesIncluded,
    o.refundable, o.cancellationPolicy, clean(o.mealPlan)
  ]);
}

const byKey = new Map();
for (const offer of normalizedOffers) {
  const key = makeSemanticOfferKey(offer);
  const previous = byKey.get(key);
  if (!previous || offer.totalMinor < previous.totalMinor) {
    byKey.set(key, offer);
  }
}
const offers = [...byKey.values()];

The example assumes normalizedOffers contains validated objects and totalMinor is a correctly parsed integer in the currency’s minor unit. If all keyed fields match, retaining the lower total can be useful when duplicated representations disagree slightly. Keep both raw records in an audit trail and record a dedupe_reason and merged_from list. If occupancy, room, taxes, provider, refundability, or any other key field differs, keep both offers.

5. Handle pagination, lazy loading, and retries

Infinite-scroll pages often leave existing cards in the DOM while adding more. Do not append extracted arrays blindly. Maintain a run-level map keyed by semantic identity, and merge each newly extracted batch through the same validation and deduplication path. If an extraction is retried, associate it with the same navigation token; discard results from a stale token after a new search starts.

1. Scroll or activate “load more” using the site’s intended interaction.
2. Wait for the result count to increase or for a known loading indicator to disappear.
3. Extract the visible cards again, or extract only newly added cards if the site exposes reliable markers.
4. Merge through the semantic-key map and log duplicate counts by reason.

Do not select the cheapest offer across different occupancy or policy variants. Choose the cheapest only within an identical semantic key, and present the differences when multiple valid rates exist.

6. Validate before using or publishing prices

Reject incomplete records that lack provider or hotel identity, itinerary, occupancy, currency, total, or room/rate identity. Treat a zero or unparsable total as invalid unless the target explicitly represents a free stay. Keep original text alongside parsed values so parsing mistakes can be diagnosed.

For a booking or publication decision, recheck the selected rate with an authorized endpoint or the site’s final booking step. Expedia’s Rapid Shopping API describes verifying a previously selected rate and returning a booking link when the price matches; use it only where its access terms and coverage fit your use case. See Expedia Rapid Shopping documentation.

Google’s price-accuracy guidance discusses automated navigation through booking funnels and Schema.org microdata on the final visible stay price. Structured data can be more resilient than fragile layout selectors when the site implements it correctly; verify that it represents the visible final price and the same itinerary.

7. Choose the right extraction source

Approach	Good fit	Tradeoff
DOM cards	Page-specific fields and visible offer details	Flexible, but selectors and responsive layouts can change
Structured data	Pages that expose complete, accurate offer data	Coverage varies; verify it matches the visible final price
Authorized lodging API	Production rate verification and booking workflows	Access, terms, latency, and field coverage depend on the provider

Compare sources on identity stability, freshness and revalidation, tax and fee coverage, occupancy and policy fidelity, authorization, latency, and operational cost. For any method, preserve enough context to explain why two offers were merged or kept separate.

8. Troubleshooting

Symptom	Likely cause	Fix
The same amount appears several times	Global selector catches cards, summaries, hidden templates, or mobile clones	Extract within each visible result card and inspect its hotel/room identifiers
Duplicate rows appear after scrolling	Each pass re-reads cards already collected	Merge batches into a run-level map using the semantic offer key
Prices are missing or intermittently blank	Extraction ran before asynchronous pricing completed	Wait for the card and a site-specific complete-price predicate; do not rely only on navigation completion
The wait times out	Selector is wrong, cards are hidden, or some card never receives a price	Inspect the live DOM, wait for a smaller stable container, and define completion for eligible cards rather than every placeholder
Distinct rates collapse into one	Key omits occupancy, room, rate plan, taxes, meal plan, or policy	Add the missing dimension; review audit records and restore distinct offers
The same offer gets two keys	Locale-formatted amount, whitespace, date, or cancellation wording differs	Normalize with locale-aware parsing and structured policy mapping while retaining raw values
Old results leak into a new search	A retry or delayed extraction completes after navigation changed	Tag work with a search token and discard stale batches
Scraped price differs at booking	Price changed, cached data was served, or taxes/fees/context differ	Record retrieval context and revalidate through an authorized endpoint or final booking step

9. Performance, reliability, and cost

One $$eval pass that returns compact plain objects reduces repeated page-to-process round trips. Avoid polling the whole document at very short intervals; wait on a stable selector and a meaningful predicate. Reuse the browser process where appropriate, but isolate page state between independent searches. Bound navigation, idle, predicate, and retry timeouts, and log which stage failed.

Retries improve resilience to transient loading failures, but a retry can also create duplicate batches. Make merging idempotent and preserve attempt IDs. For large crawls, limit concurrency to what the target and your infrastructure can handle, and follow the site’s terms and applicable access rules. Track request volume, browser runtime, storage, and revalidation calls: these are the main operating-cost drivers. Scraping alone cannot guarantee freshness; rate rechecks add latency and may incur API or infrastructure cost.

10. Or skip the browser setup

If your immediate task is capturing a page for review, documentation, or an AI workflow, ScreenshotNeo takes a screenshot or PDF with one GET request. It is a screenshot API, not a hotel-rate extraction or booking API, so it does not replace the structured offer extraction and rate verification above.

See the ScreenshotNeo API documentation. Example using the Stripe page as the target:

curl -G "https://api.screenshotneo.com/v1/shot" \
  -d access_key=YOUR_API_KEY \
  --data-urlencode url=https://stripe.com \
  -o shot.webp

ScreenshotNeo accepts cookie and consent banners and removes 60+ known consent platforms, newsletter popups, and chat widgets before capture; each step can be turned off. Bot checks, blank pages, timeouts, failed loads, and cache hits are not billed, and response headers report the page verdict and billing status. Its MCP server offers take_screenshot, get_page_info, and capture_pdf for Claude, Cursor, and other MCP clients. The free plan includes 1,000 screenshots a month with no card; paid plans start at $5 for 3,000.

Create a free ScreenshotNeo account for 1,000 screenshots a month with no card.

11. FAQ

Should I deduplicate by hotel name and price?

No. Those fields omit rate identity. Keep offers separate when room, occupancy, itinerary, currency, taxes, provider, meal plan, or cancellation terms differ.

Is network idle enough to know rates are final?

No. It means requests were quiet for the configured period. Check the page’s actual result state and revalidate the chosen price before booking or publishing.

Can I use CSS classes as selectors?

You can, but generated classes are often fragile. Prefer stable data attributes or correctly implemented structured data, and revisit selectors when the page changes.

Should I keep the original cancellation text?

Yes. Store it beside the normalized policy so the original evidence remains available when the wording cannot be mapped confidently.