Duplicate hotel prices usually come from duplicate page representations or from collapsing genuinely different offers. Make the itinerary and page state deterministic, extract each visible result card once, normalize every price dimension, and deduplicate with a semantic key that includes occupancy, room, rate plan, taxes, cancellation policy and provider. Never deduplicate on hotel name and amount alone.
What a duplicate hotel price really is
A hotel result is an offer for a specific context, not just a number. Google’s hotel-pricing model defines a price as the lowest price for a double-occupancy room for a particular check-in date and number of nights. Rate feeds can contain multiple rooms, rate plans, occupancies and refundability combinations. Two cards showing “$180” may therefore be different bookable offers.
Keep both records when any of these differ:
- provider or hotel identity;
- room type or room identifier;
- rate-plan identifier;
- check-in date or stay length;
- occupancy;
- currency, taxes or fees included;
- meal plan;
- refundable status or cancellation deadline.
Only merge records after those values have been normalized and shown to be equivalent. If you need the cheapest offer, select it only inside an identical semantic key.
Make the page state deterministic before extracting
Fix the itinerary and request context
Set check-in, check-out (or nights), occupants, currency, locale, country, device context and provider before navigation. Record those values with every extraction batch. Google documents that cached and live prices can vary with country, device and occupancy, so a second request is not necessarily a duplicate of the first; it may be a repriced or differently contextualized result.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →#1 Best Overall
Wait for both structure and content
Use a visible result-container wait, a network-idle wait when rates are fetched asynchronously, and a site-specific readiness predicate. Puppeteer’s waitForSelector can require visibility and has a default 30-second timeout. waitForNetworkIdle waits for the network to be idle and still waits at least the configured idle time; network idle alone does not prove that every price is final.
await page.goto(url, { waitUntil: 'domcontentloaded' });
await page.waitForSelector('[data-hotel-card]', {
visible: true,
timeout: 30000
});
await page.waitForNetworkIdle({ idleTime: 500 });
await page.waitForFunction(() => {
const cards = [...document.querySelectorAll('[data-hotel-card]')]
.filter(el => el.offsetParent !== null);
return cards.length > 0 && cards.every(card =>
/d/.test(card.querySelector('[data-total]')?.textContent || '')
);
});Replace the data attributes with selectors inspected on the target site. Prefer stable data attributes or embedded structured data over CSS classes that change during redesigns. A stronger predicate can require a known minimum number of cards, a “results loaded” flag, or a specific loading element to disappear.
Extract one object per visible result card
Query the repeated component, not a price selector across the entire document. A global price query can match desktop and mobile clones, sticky summaries, hidden templates and modal layers. Puppeteer’s $$eval evaluates all matching elements in one page-context pass; $eval evaluates only the first match.
const raw = await page.$$eval('[data-hotel-card]', cards => cards
.filter(card => card.offsetParent !== null)
.map(card => ({
provider: card.dataset.provider || null,
hotelId: card.dataset.hotelId || null,
hotelName: card.querySelector('[data-hotel-name]')?.textContent || null,
roomId: card.dataset.roomId || null,
roomName: card.querySelector('[data-room-name]')?.textContent || null,
ratePlanId: card.dataset.ratePlanId || null,
checkIn: card.dataset.checkIn || null,
nights: card.dataset.nights || null,
occupancy: card.dataset.occupancy || null,
currency: card.dataset.currency || null,
totalText: card.querySelector('[data-total]')?.textContent || null,
nightlyText: card.querySelector('[data-nightly]')?.textContent || null,
taxesText: card.querySelector('[data-taxes]')?.textContent || null,
feesText: card.querySelector('[data-fees]')?.textContent || null,
mealPlan: card.querySelector('[data-meal-plan]')?.textContent || null,
cancellationText: card.querySelector('[data-cancellation]')?.textContent || null,
refundable: card.dataset.refundable || null
})));Return plain objects from the page context. Do not pass DOM nodes to Node.js and then attempt to read them after navigation; they are tied to the document that created them.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallCrashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteNormalize values before building a deduplication key
Text, dates and currency
Trim whitespace, collapse repeated spaces and normalize Unicode. Parse locale-specific separators deliberately: “1.234,56” and “1,234.56” can represent the same amount in different locales. Convert dates to an unambiguous form such as ISO calendar dates, and use ISO currency codes rather than symbols. Preserve the original display text for auditing.
Cancellation and inclusion rules
Cancellation prose should become structured fields such as refundable, deadline and penalty, while the original text remains available. Likewise, represent whether taxes and fees are included instead of assuming that two totals are comparable.
function cleanText(value) {
return String(value ?? '')
.normalize('NFKC')
.replace(/s+/g, ' ')
.trim();
}
function numberFromLocale(value) {
const text = cleanText(value).replace(/[^0-9,.-]/g, '');
if (!text) return null;
const comma = text.lastIndexOf(',');
const dot = text.lastIndexOf('.');
let normalized = text;
if (comma > dot) {
normalized = text.replace(/./g, '').replace(',', '.');
} else {
normalized = text.replace(/,/g, '');
}
const result = Number(normalized);
return Number.isFinite(result) ? result : null;
}
function normalizeOffer(raw) {
return {
provider: cleanText(raw.provider),
hotelId: cleanText(raw.hotelId),
hotelName: cleanText(raw.hotelName),
roomId: cleanText(raw.roomId),
roomName: cleanText(raw.roomName),
ratePlanId: cleanText(raw.ratePlanId),
checkIn: cleanText(raw.checkIn),
nights: Number(raw.nights),
occupancy: cleanText(raw.occupancy),
currency: cleanText(raw.currency).toUpperCase(),
total: numberFromLocale(raw.totalText),
nightly: numberFromLocale(raw.nightlyText),
taxes: numberFromLocale(raw.taxesText),
fees: numberFromLocale(raw.feesText),
mealPlan: cleanText(raw.mealPlan),
refundable: cleanText(raw.refundable),
cancellationText: cleanText(raw.cancellationText)
};
}
function makeSemanticOfferKey(offer) {
return JSON.stringify([
offer.provider, offer.hotelId, offer.roomId, offer.ratePlanId,
offer.checkIn, offer.nights, offer.occupancy, offer.currency,
offer.total, offer.taxes, offer.fees, offer.mealPlan,
offer.refundable, offer.cancellationText
]);
}If the site exposes a machine-readable cancellation deadline or policy code, use that in the key and retain the prose as a display field. Do not use hotel name and total as a shortcut.
Deduplicate safely, including pagination and retries
A stable key handles repeated DOM nodes, but a scraper also needs a stable seen-set across pages and scroll operations. “Load more” interfaces often return existing cards again. Retries after a partial response can append the same batch. Tie each extraction to a navigation or request token and discard stale batches.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →const byKey = new Map();
const seenBatch = new Set();
function mergeBatch(raw, navigationToken) {
const normalized = raw.map(normalizeOffer);
for (const offer of normalized) {
if (!offer.provider || !offer.hotelId || !offer.checkIn ||
!Number.isFinite(offer.nights) || !offer.occupancy ||
!offer.currency || !Number.isFinite(offer.total) ||
(!offer.roomId && !offer.ratePlanId)) {
continue;
}
const key = makeSemanticOfferKey(offer);
const batchKey = `${navigationToken}:${key}`;
if (seenBatch.has(batchKey)) continue;
seenBatch.add(batchKey);
const previous = byKey.get(key);
if (!previous || offer.total < previous.total) {
byKey.set(key, {
...offer,
dedupe_reason: previous ? 'same semantic offer; lower total retained' : 'first occurrence',
merged_from: previous ? [...(previous.merged_from || []), previous] : []
});
}
}
}
// After all pages, the unique offers are:
const offers = [...byKey.values()];For auditability, store sourceUrl, scrapedAt, the request context and the raw record. A replacement should be possible to explain: identical semantic identity with a lower parsed total, not an unexplained overwrite.
Why duplicate prices appear
| Cause | What you see | Correction |
|---|---|---|
| Repeated DOM representations | Desktop/mobile cards, hidden templates, sticky summaries or modal details repeat the amount. | Scope $$eval to the result-card component and keep visible, complete cards only. |
| Pagination or lazy loading | A scroll or “load more” pass returns cards already processed. | Maintain a stable semantic seen-set across batches. |
| Retries and navigation races | A partial response is followed by the same page being appended. | Attach a navigation token and discard stale batches. |
| Occupancy and rate-plan variants | The amount matches but room, guest count, refundability or meal plan differs. | Keep every differing dimension in the key and output. |
| Cache and freshness | The same request context returns cached or newly repriced data. | Record country, device, occupancy and retrieval time; revalidate before booking. |
Validate records before showing or booking them
- Reject records missing provider or hotel identity, dates, nights, occupancy, currency, total, or a room/rate identifier.
- Check that the parsed total is non-negative and that the currency is present; retain the unparsed text for review.
- Group only by the complete semantic key. Choose the cheapest record only within that group.
- Before publishing or booking, verify the selected rate through an authorized endpoint or the site’s final booking step.
- Record the verification result and the time it occurred; a displayed search price can change before checkout.
Expedia’s Rapid Shopping API documentation describes verifying a previously selected rate and returning a booking link when the price matches. For sites that expose Schema.org microdata on the final visible stay price, automated navigation through the booking funnel can provide another validation path. Google’s documentation notes that legacy landing-page-only structured data is planned for deprecation in early 2027, so a scraper should not treat an old landing-page markup pattern as a permanent contract.
Choosing DOM, structured data or an authorized API
| Approach | Strength | Risk or limitation |
|---|---|---|
| DOM scraping | Flexible and available on pages without a usable feed. | Selectors are fragile; hidden clones and layout changes can create duplicates. |
| Structured data | Can expose the final visible price without depending on visual CSS classes. | Coverage and freshness depend on correct implementation; markup may be incomplete. |
| Authorized lodging API | Usually best for production revalidation, stable identifiers and explicit policy fields. | Coverage, latency, cost and program terms vary; authorization is required. |
Use the least fragile source that meets your coverage and terms. Do not scrape a site in a way that violates its authorization, terms or access controls.
Performance, reliability and scale considerations
Bound waiting time
Set navigation, selector and function timeouts appropriate to the site, then fail a batch rather than emitting partial prices as complete. Network idle is useful for asynchronous requests, but a site-specific predicate is the completion signal.
Control memory and duplicate work
Process cards in batches when a page can contain thousands of results. Keep the key and a compact audit record instead of retaining every duplicate DOM object. For scroll pagination, wait for the card count or a stable cursor to change before extracting again.
Plan for itinerary volume
Google’s Hotel Prices “Pricing overview” guidance describes 9,900 itinerary entries (330 advance-booking days multiplied by 30-night stays) as a general planning figure. Treat that as a coverage example, not a promise that every property, market or provider supports the same range. At this scale, an authorized rate API and explicit revalidation workflow are generally easier to operate than thousands of layout-dependent browser sessions.
Troubleshooting duplicate or missing prices
Every price appears twice
Inspect matching elements in DevTools and check offsetParent, computed visibility and ancestor containers. You are likely matching a mobile clone, hidden template or sticky summary. Narrow the selector to the repeated card and filter visible elements.
The scraper returns zero cards
The selector may be wrong, the results may be inside an iframe, or the page may still be loading. Confirm the selector after the itinerary is submitted, wait for a visible card, and add a readiness predicate that checks for a numeric total.
Prices are present but incomplete
Do not publish after network idle alone. Wait for each visible card to contain its total, and reject records without occupancy, currency or room/rate identity. If details appear only after clicking, perform that click before extraction and wait for the detail selector.
Totals differ between runs
Compare the recorded country, device, occupancy, currency, cache state and retrieval time. These contexts can change cached or live pricing. Re-run the final selected offer through an authorized verification step immediately before booking.
A retry creates more duplicates
Generate a navigation/request token for every attempt. Discard records from an old token when a newer navigation succeeds, and keep a semantic key across “load more” batches.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.FAQ
Should I keep duplicate records for auditing?
Keep the raw records and a merged_from trail, but expose only one record for an identical semantic offer. This preserves evidence without confusing downstream users.
Recommended Free Tools
Can a screenshot prove that two prices are the same offer?
No. A screenshot shows presentation, not stable provider, room, occupancy, tax and policy identifiers. Use structured extraction and retain a screenshot only as visual evidence.
When should the cheapest result be selected?
Only after required fields validate and the semantic key confirms that room, occupancy, policy, taxes, currency and provider all match. Then recheck the rate at the authorized booking step.
Or skip the browser setup
If your goal is a visual record of a hotel-results page rather than structured rate extraction, ScreenshotNeo provides a website screenshot API and MCP server. It can accept the consent banner like a visitor and remove more than 60 known consent platforms, newsletter popups and chat widgets before capture; each cleanup step can be disabled. Bot checks or CAPTCHAs, blank pages, timeouts, failed loads and cache hits are not billed, and the response reports the page verdict and billing status in X-Page-Verdict and X-Billed headers.
One GET request returns PNG, JPEG, WebP or PDF. The API also supports full-page lazy-image loading, CSS-selector element capture, dark mode, device presets and custom viewports, retina scale, PDF page controls, custom CSS and JavaScript, clicks, hidden selectors, selector/delay/network-idle waits, request and resource blocking, headers, cookies, user agents, authorization, timezone and geolocation, transparent backgrounds, resizing, chosen cache TTLs, signed image links, asynchronous jobs with signed webhooks, bulk capture of up to 100 URLs per call, a usage API and an OpenAPI specification. An MCP server exposes take_screenshot, get_page_info and capture_pdf to Claude, Cursor and other MCP clients.
See the ScreenshotNeo API documentation for parameters. Example:
curl -G "https://api.screenshotneo.com/v1/shot"
-d access_key=YOUR_API_KEY
--data-urlencode url=https://example.com/hotels
-o hotel-results.webpEquivalent calls:
import requests
r = requests.get(
"https://api.screenshotneo.com/v1/shot",
params={"access_key": "YOUR_API_KEY", "url": "https://example.com/hotels"},
timeout=90
)
r.raise_for_status()
open("hotel-results.webp", "wb").write(r.content)const q = new URLSearchParams({
access_key: 'YOUR_API_KEY',
url: 'https://example.com/hotels'
});
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);
if (!res.ok) throw new Error(`Screenshot failed: ${res.status}`);
require('fs').writeFileSync('hotel-results.webp', Buffer.from(await res.arrayBuffer()));The Free plan includes 1,000 screenshots per month with no card. Paid plans start at $5 for 3,000 shots; every feature is available on every plan, and yearly billing gives two months free. Create a free ScreenshotNeo account to start.
Frequently Asked Questions
Should I keep duplicate records for auditing?
Keep raw records and a merged_from trail, but expose one record for an identical semantic offer.
Can a screenshot prove that two prices are the same offer?
No. It shows presentation, not stable room, occupancy, tax and policy identifiers.
Free tools Windows power users keep installed
One-click scans. No signup required.
When should the cheapest result be selected?
After required fields validate, the semantic key matches, and the rate is rechecked at the authorized booking step.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
