iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
To scrape product prices, stock and GTINs from any online store, use an adaptable workflow—not one universal selector. Start with a merchant feed or documented API when available, inspect structured product data in the page’s initial HTML, and use site-specific parsing or browser rendering only when necessary and permitted. Then validate every result against the shopper-facing offer, including its currency, variant and relevant location or fulfillment context.
What you need to capture—and keep separate
A product identity and a purchasable offer are related, but they are not the same record. Product details can include a name, brand, SKU and GTIN. An offer describes the terms under which a particular product or variant is sold, including its price, currency and availability.
Keep the offer attached to the exact product or variant it describes. A product page can contain multiple variants or offers; pairing a price with the wrong GTIN can make an otherwise well-formed record misleading.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →- Product identity: product or variant identifiers, such as SKU and GTIN, plus descriptive details such as name and brand.
- Offer observation: price, currency, availability, the source page and the time observed.
- Context: the selected variant and, where relevant, locale, location, account state or fulfillment option.
Choose an extraction route
Use the least brittle supported route that provides the fields and context you need. A feed or API may offer a cleaner interface; page markup can expose useful data without rendering a browser; and site-specific parsing or browser rendering may be necessary when those options do not work. None guarantees complete or current data for every store.
#1 Best Overall
| Route | When to try it | What to validate |
|---|---|---|
| Merchant feed or documented API | First, when the store expressly offers one. | Field coverage for price, currency, availability, GTIN and variants; update timing; and the feed’s or API’s access conditions. |
| Structured data in initial HTML | When no suitable supported interface is available, or as a way to inspect the page’s product and offer data. | Whether the relevant fields are present, map to the correct variant and match what shoppers see. |
| Site-specific HTML parsing | When needed fields are absent from structured data and collection is permitted. | Whether the page structure consistently identifies the correct product, selected offer and displayed values. |
| Browser rendering | When relevant content is inserted after the initial page response and access is permitted. | Whether the rendered offer matches the shopper-facing selection, and whether the extra rendering step provides data that the initial HTML does not. |
This is a decision aid, not a published performance ranking. Compare routes by field coverage, freshness, accuracy, context dependence and operating constraints for the stores you actually need to handle.
Check access before collecting data
Look for a merchant feed, documented API or other expressly supported route before building a crawler. Check the store’s applicable terms and its robots.txt instructions before making requests. RFC 9309 describes the Robots Exclusion Protocol as crawler guidance; it explicitly does not make robots.txt an access authorization. A path not disallowed there is not, by itself, permission to collect data.
Do not treat a technical ability to fetch or render a page as proof that collection is allowed. If the available terms or supported interfaces do not permit the intended collection, do not proceed on the assumption that changing the scraper will resolve the issue.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Inspect structured data in the initial HTML
Before writing selectors for page-specific markup, inspect the HTML returned by the server for Schema.org product data. Look for a Product and its nested Offer, commonly represented as JSON-LD, while recognizing that stores may use other encodings or omit suitable markup.
Google recommends JSON-LD and recommends putting Product structured data in the initial HTML for best results. For the matching conditions described in its Merchant Center guidance, the markup must be in the HTML returned by the server. That makes the initial response a useful first inspection point, but not a guarantee that every store exposes all required values there.
Read product and offer fields as distinct data:
- Product: name, SKU, brand and identifier properties.
- Offer: price, price currency, availability and condition.
- Relationship: which exact product or variant the offer belongs to.
Google Merchant Center maps its GTIN attribute to Schema.org’s gtin, gtin8, gtin12, gtin13, gtin14 or isbn properties. When a page has several offers, do not assume an identifier found somewhere on the page belongs to every offer; resolve the relationship before storing the record.
Capture prices as observations, not bare numbers
For every observed offer, save the numeric price with its currency, observation time, source page and associated product or variant. A price without its currency cannot be compared safely with another price. Keep sale and regular prices distinct when the page distinguishes them; do not silently substitute one for the other.
Recommended Free Tools
Google identifies price, priceCurrency, availability and condition among the fields used for automatic item updates, and says structured markup should match the values shown to users. Its Merchant Center guidance specifies a period as the decimal separator in JSON-LD price values. Treat that representation as source data, and separately retain enough context to check that it matches the visible offer.
A practical observation record should make those relationships explicit:
product_or_variant_id: the product or variant this offer belongs to
source_page: the page where it was observed
observed_at: the fetch time
price: the observed numeric price
currency: the currency shown for that price
availability: the source value and any normalized value
context: locale, selected variant, location or fulfillment state, when relevantThis is an illustrative record shape, not a store-specific schema. Preserve the source value as well as any normalized value so that a later mapping can be checked rather than mistaken for what the page actually said.
Map stock labels without overstating what they mean
Availability can be expressed with values such as InStock, OutOfStock, BackOrder, PreOrder, PreSale, LimitedAvailability and SoldOut. Map a store’s wording to a controlled value only when the page supports that interpretation. Keep the original label when the meaning is uncertain rather than forcing it into a more definite category.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Clear out junk files and repair common Windows errors3Fix the driver behind crashes, sound loss and screen glitches“Available” may depend on the selected variant, a delivery location, an account or a fulfillment method. Record the relevant state when it is exposed, and avoid treating one observation as a universal stock claim. A page-level label alone may not establish availability for every variant or location.
Validate GTIN format and identity
Store a GTIN as text, not as a number: it is an identifier, not a quantity to calculate with. Check that its form is appropriate and that its check digit is valid, then verify identity with GS1 when the consequence of a mismatch warrants it.
Verified by GS1 can check whether a number is issued and return assigning-company information; product data is returned when available. GS1 stated in 2024 that the public service permits 30 free queries per day, with more advanced options available through a local GS1 Member Organisation. Because that limit is time-sensitive, check the service’s current terms before planning a verification workload.
A syntactically valid GTIN is not proof that it identifies the product shown on the page. Treat format validation and identity verification as separate checks, and do not claim a product match merely because the check digit passes.
Handle JavaScript-rendered offers selectively
Distinguish data in the server’s initial response from content inserted after the page loads. If the initial HTML has the needed offer data, a browser-rendering step may add complexity without improving the result. If relevant price or availability appears only after rendering, and collection is permitted, browser automation may be needed to inspect it.
Best Value
Google warns that JavaScript-generated Product markup can make crawling less frequent and less reliable for rapidly changing prices and availability. Rendering is therefore neither always necessary nor sufficient: after using it, compare the captured values with the visible offer and the selected variant rather than assuming the rendered page proves correctness.
Validate every observation against the shopper-facing page
Structured data and parsed markup are inputs, not a substitute for checking what a shopper is offered. Google says structured data should match values shown to users; its Merchant Center guidance also warns against landing-page content that changes dynamically based on user information for the matching requirements it describes.
- Record the observation context. Save fetch time, source page, locale and currency, plus the selected variant and any relevant location, account or fulfillment state.
- Compare identity and offer. Check that the GTIN or other product identifier belongs to the same product or variant as the price and availability you captured.
- Compare against the visible page. Confirm the displayed price, currency and availability for that same selection. Investigate a mismatch instead of silently choosing whichever value is easier to parse.
- Recheck changes when possible. A later observation can reveal whether a difference reflects a changing offer or a parsing error; retain observation times so those cases can be distinguished.
Build per-store handling into the design
“Any online store” is a goal for an adaptable system, not a promise that one selector or extraction method will work universally. Stores differ in which interfaces they support, what appears in initial HTML, how offers map to variants, and whether price or availability depends on shopper context.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Keep source-specific extraction separate from a shared record format. For each store, document the route used, the fields it exposes, the context needed to interpret them and the checks that confirm the result matches the displayed offer. If a field is missing or the relationship between product and offer cannot be established, mark it as unavailable or unresolved rather than inventing a value.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
