Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Start by asking suppliers for an approved product feed, API, portal export, or relevant GS1 data-pool connection. These structured routes are usually easier to keep consistent than scraping web pages. Scrape a supplier site only when no suitable approved source is available and the site permits your intended access. Then preserve source identifiers and retrieval times, validate records, monitor changes, and export a documented catalog.
Choose an approved source before scraping
Supplier product information may be available through synchronized data pools, APIs, portal exports, or product pages. Compare the options against your required fields, supplier and item coverage, update latency, identifier quality, permitted use, integration work, and total cost. No route is a universal winner; access, fields, and terms vary by supplier and service.
| Route | Useful when | Check before implementation |
|---|---|---|
| Supplier feed or GS1 GDSN data pool | The supplier participates and recurring synchronization matters. | Supplier and item coverage, schema, attributes, update behavior, subscription, and use terms. GDSN synchronizes data between participating trading partners; it does not mean every supplier or item is included. GS1 GDSN |
| Supplier or registry API | You need structured queries, system integration, or repeatable ingestion. | Authentication, rate limits, fields, bulk support, price, licensing, geography, and storage or redistribution rights. GS1 US documents API workflows and subscription-dependent capabilities. GS1 US Data Hub |
| Portal export | A one-time or periodic catalog download is enough. | Export format, selectable fields, record limits, refresh procedure, and terms. GS1 US documents filtered exports and subscription-dependent options. GS1 US Data Hub |
| Website scraping | No suitable structured source is available and the intended access is permitted. | Current terms, robots.txt, technical restrictions, request volume, content rights, and applicable law. Robots.txt is a crawler protocol, not authorization. IETF RFC 9309 |
Ask the supplier or service for its schema, coverage, update cadence, use conditions, and change-notification behavior before building around it. GS1 Data Link is described by GS1 Netherlands as an API connection to data-pool label information, subject to conditions that include keeping data current. GS1 Netherlands: GS1 Data Link
Define the catalog fields and identity rules
Write down the fields your downstream users actually need. A starting schema might include a source product identifier, GTIN where present, supplier SKU, brand, title, description, dimensions, images, availability, and source update time. Treat these as requested fields, not a promise that a feed, API, or page will expose them.
#1 Best Overall
- Choose a canonical product key and decide how variants and packaging levels remain distinct.
- Preserve supplier SKU, GTIN, or another stable source identifier where available. Do not match records by product name alone: similar names can refer to different variants or package sizes.
- Record which source is authoritative for each field if values can come from more than one supplier or service.
- Document required formats, units, allowed empty values, and validation rules.
Verified by GS1 can help check whether an identifier is properly structured and which company is associated with it. It is an identity check, not a complete or necessarily current product catalog. GS1 describes the question its service addresses as: “Is this the product that I think it is?” Verified by GS1
GS1 describes its Global Data Model as a globally consistent set of foundational product attributes needed to list, store, move, and sell products. Use an applicable standard or supplier schema where it meets your needs rather than assuming a lookup service supplies every field. GS1 Global Data Model
Request access and prepare the ingestion workflow
- Contact the supplier. Ask for an approved feed, API, portal export, or relevant data-pool connection, and state your intended use and destination systems.
- Confirm the contract and technical details. Get the schema, authentication method, field coverage, limits, refresh cadence, cost, storage and redistribution permissions, and any notice of schema changes.
- Test representative records. Include variants, different package levels, missing fields, and discontinued or unavailable items if those exist in the source.
- Map into a versioned internal schema. Keep the source value and normalized value distinguishable so a transformation does not erase what the supplier provided.
- Run validation before accepting a batch. Flag missing required identifiers, malformed values, unexpected units, duplicate keys, and incompatible schema changes.
GS1 US describes API-based automated ingestion and product export workflows, with capabilities dependent on the relevant subscription and selected service. Confirm current access and terms directly with the provider. GS1 US Data Hub
Scrape supplier pages only when the route is permitted
When you have no suitable approved structured source, first review the target site’s current terms, robots.txt instructions, technical access controls, and the rules applicable to your location and intended use. If access requires credentials you do not have, is blocked, or is unclear, stop and ask the supplier or seek legal review. Do not bypass access controls.
- Check the site’s robots.txt instructions for the paths and crawler behavior relevant to your planned collection.
- Use a clear crawler identity and a conservative request rate. Avoid unnecessary repeat requests and stop if the site blocks access or returns signs of excessive load.
- Extract only the fields needed for the permitted purpose. Keep a source URL and retrieval timestamp with every record.
- Handle missing, changed, or ambiguous page fields as validation cases, not as permission to infer data.
- Retain raw responses or snapshots only where your permissions allow, and define retention accordingly.
RFC 9309, published in September 2022, standardizes robots.txt rules such as Allow and Disallow matching. It says, “These rules are not a form of access authorization,” and notes robots.txt is not a substitute for valid content security measures. A robots.txt check therefore does not settle contractual, legal, or other access questions. IETF RFC 9309
The same standard says crawlers should not use a cached robots.txt version for more than 24 hours unless the file is unreachable. That is guidance about crawler handling of robots.txt, not a supplier-data refresh interval. IETF RFC 9309
Normalize records and keep provenance
Maintain a record that lets your team trace each catalog value back to its origin. A practical internal record can include the canonical product key, supplier, original source identifier, source URL or feed name, retrieval time, source schema or version, raw and normalized values where permitted, and field-level validation outcome. This is a workflow design recommendation, not a universal GS1-required format.
- Keep identifiers as identifiers; do not silently reformat or discard leading zeros.
- Normalize units and text consistently, but retain enough source evidence to audit transformations.
- Distinguish a supplier’s own update timestamp from the time your system retrieved the record.
- Do not treat a successful identifier lookup as proof that descriptions, dimensions, images, or availability are complete or current.
Monitor changes and download reliable exports
Compare each incoming record with the last accepted version. Separate source changes from changes caused by your own normalization rules, and route ambiguous or high-impact differences for review. Examples include a changed GTIN, package dimensions, brand, or availability value; decide which deltas require review according to your business rules.
Best Value
- Set refresh frequency from the supplier’s stated update cadence and the business cost of stale data. The available standards and service descriptions do not establish one universal polling schedule.
- Validate a new batch before replacing accepted values, and retain a status for rejected or review-needed records.
- Alert the appropriate team to material changes, missing updates, feed failures, or schema changes.
- Export a documented schema with retrieval timestamps so downstream users can distinguish current values from stale ones.
- Keep a change history appropriate to your audit and retention needs, subject to source terms.
GS1 GDSN is designed for current product-data synchronization among participating trading partners, while GS1 US describes API and export workflows. The exact fields, access, and update behavior depend on the supplier and service selected. GS1 GDSN · GS1 US Data Hub
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Troubleshoot common catalog problems
| Symptom | Likely cause | What to check |
|---|---|---|
| Products are missing from a data-pool search | The supplier or item may not participate, or the selected service may not cover it. | Confirm supplier/item coverage and access with the supplier or data-pool provider; do not assume network participation is universal. |
| Fields are empty or inconsistent between sources | Sources expose different attributes, use different schemas, or update at different times. | Compare schemas and timestamps, confirm the authoritative source per field, and avoid filling gaps by guesswork. |
| Two catalog rows appear to be the same product | Name-based matching may have merged variants or packaging levels, or source identifiers differ. | Review GTIN and supplier SKU, packaging level, and variant attributes; keep uncertain matches separate until checked. |
| A scraper receives blocked or restricted responses | The site may disallow the access, require authorized credentials, or apply technical restrictions. | Stop rather than bypassing the restriction; request an approved feed, API, export, or permission. |
| A crawler continues using old robots.txt rules | The crawler may retain a cached file too long. | Follow RFC 9309’s recommendation not to use a cached robots.txt for more than 24 hours unless it is unreachable. |
| An export cannot be reconciled with the previous one | Schema or normalization changes may be mistaken for supplier data changes. | Version the schema and transformation rules, compare source values separately from normalized values, and route uncertain deltas for review. |
Or skip the browser setup
If a supplier page is an appropriate and permitted source, ScreenshotNeo can capture a page as an image or PDF with one GET request. It is a screenshot API, not a structured product feed: use it to preserve a visual page capture, then extract or verify catalog fields with a separate process. Its clean-shot options accept cookie or consent banners and remove more than 60 known consent platforms, newsletter popups, and chat widgets before capture; each step can be turned off. Bot checks or CAPTCHAs, blank pages, timeouts, failed loads, and cache hits are not billed, and response headers report the page verdict and billing status. An MCP server provides take_screenshot, get_page_info, and capture_pdf tools for AI agents using Claude, Cursor, or another MCP client.
Example cURL request (replace the target URL and use your API key):
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpSee the ScreenshotNeo documentation for request options and configuration. Free includes 1,000 shots per month with no card; paid plans start at $5 for 3,000 shots. Every feature is available on every plan. Sign up for 1,000 free screenshots a month with no card.
Frequently Asked Questions
Does Verified by GS1 provide a complete supplier catalog?
No. It helps check identifier structure and the company associated with a key; product-field availability is not the same as a complete catalog.
Does robots.txt permission mean a scrape is legally allowed?
No. RFC 9309 defines crawler instructions, not access authorization. Check the applicable site terms, access controls, contracts, and law.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
