To extract a specific website field reliably, define the output you need, locate its source in the page, apply a CSS selector, XPath expression, or pattern, and then validate the result on representative URLs. If the value is inserted by JavaScript, use rendered HTML rather than the initial response. Finally, decide how repeated matches and missing values should be represented.
What a custom extraction rule does
A custom rule tells a crawler or scraping service what to read and where to store it. The source may be an HTML element, an attribute, the element’s text or HTML, or a value derived from the URL. For example, a rule can place a page heading in an article_title field, a displayed price in price, or a date captured from a URL in separate year, month, and day fields.
Keep the field name and its expected shape explicit before writing a selector. Decide whether the result should be one value, a list, joined text, an attribute such as href, or raw HTML.
A practical workflow for extracting one field
- Define the value and destination field. Write down exactly what counts as a match and name the output field, such as
author,article_title, orprice. These are illustrative names; the target site’s markup determines the actual rule. - Inspect a representative page. Use browser developer tools or Screaming Frog SEO Spider’s built-in browser and visual selector helper to find the element. Confirm whether the value is present in the original HTML or appears only after scripts run.
- Choose the narrowest suitable expression. Use a CSS selector or XPath for an HTML element. Use a regular expression when the value is a pattern, such as a date embedded in a URL. A distinctive class, element, or attribute is safer than a broad selector that happens to match today.
- Select the returned form. Configure the extractor to return text, an attribute, inner HTML, the selected element, or a function-derived value, depending on what the consuming system needs.
- Handle repeated matches deliberately. Decide whether to keep every match as an array or join values into one field. Elastic Open Web Crawler documents configurable multi-value joining through
join_as. - Test several URLs. Include pages with the normal layout, optional content, repeated items, and a different template. Compare the extracted value with what a user actually sees.
- Switch to rendering when necessary. If the initial response lacks the value, run the extractor against rendered HTML or an enabled browser-rendering path. A page can be considered loaded before JavaScript finishes inserting content.
- Check authorization. Confirm that the site’s terms and applicable rules permit the collection you intend. Technical access does not by itself establish permission.
Selectors, XPath, and regular expressions
CSS selectors
CSS is usually the quickest choice when the value sits in a recognizable element. Target a stable identifier or a distinctive combination of element and attribute, and avoid selectors that depend only on visual position.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
#1 Best Overall
- Read Before You Buy — No Video Output: These adapters support charging and USB 2.0 data transfer, but cannot transmit video signals. Except for standard USB webcams (which use USB data only), they are not compatible with HDMI/DisplayPort cables, video-capable USB-C hubs, or docking stations with video output.
- Convert USB-A Ports to USB-C: Designed to connect USB-C earphones, cables, flash drives, card readers, and other USB-C accessories to standard USB-A ports. Plug-and-play with no drivers or software required.
- Aluminum Alloy Housing: Built with a sturdy aluminum alloy shell that aids in heat dissipation and protects against daily wear and scratches. Designed to maintain a stable and secure connection.
- Compact & Travel-Friendly: The ultra-compact design allows the adapter to stay plugged into your device without blocking adjacent ports or adding bulk, reducing wear and tear on your original USB ports.
- 12-Month Warranty: Backed by a 12-month manufacturer warranty for peace of mind. Designed to meet strict quality control standards for reliable everyday performance.
XPath
XPath is useful when the relationship between elements matters, such as selecting a heading beneath a particular article container. It can also express conditions that are awkward in a simple CSS selector.
Regular expressions
Regex is best for pattern-shaped data rather than arbitrary page structure. Use capture groups to return only the desired substring. Elastic’s extraction examples use URL regex groups to split year, month, and day into separate values; the same principle prevents a rule from returning an entire URL when only one component is needed.
Rank #2
- 5-in-1 USB-C Hub: Experience comprehensive connectivity featuring a Power Delivery input, two USB-A 2.0 ports, a USB-A 3.0 port, and an HDMI port. (Note: The USB-C power delivery input port is only for connecting an external wall charger to power your laptop and cannot power peripheral devices.)
- 90W Pass-Through Charging: Achieve optimal charging with 90W pass-through power to your laptop, supported by a total input of 100W, with the hub reserving 10W for operational efficiency. (Note: Wall charger not included.)
- Quick Data Transfers: Accelerate your productivity with rapid data transfers using a high-speed 5Gbps USB 3.0 port and two 480Mbps USB 2.0 ports.
- 4K HDMI Display: Enhance your visual experience with a hub capable of delivering 4K resolution at 30Hz in both mirror and extend modes. Please note that this hub is compatible with MacBook (macOS 12 and newer), Windows 10 and 11, ChromeOS, and laptops equipped with DP Alt Mode and Power Delivery. Note: This device is not compatible with Linux.
- What You Get: Anker USB-C Hub (5-in-1, 4K HDMI), welcome guide, 18-month warranty, and our friendly customer service.
Choose an extraction approach
| Approach | Documented capabilities | Best fit | Important consideration |
|---|---|---|---|
Cloudflare Browser Rendering /scrape |
Accepts a URL or HTML and selected elements identified with CSS selectors; examples include headings, links, prices, and repeated content. | A hosted endpoint for selected elements from a page. | Its page-loaded state may occur before JavaScript rendering has finished, so timing and rendered-content checks matter. |
| Screaming Frog SEO Spider | Site crawling with custom extraction using XPath, CSS Path, or regex; visual extraction assistance; static or JavaScript-rendered HTML. | Crawl-wide audits and configurable extractors. | The custom extraction feature requires a licence, and rendered crawling should be enabled for client-side-only values. |
| Elastic Open Web Crawler | Rulesets scoped to domain entries, URL filters, HTML CSS/XPath extraction, URL regex capture groups, named fields, and configurable handling of multiple values. | Configuration-driven crawling with structured output fields. | Rules apply only where URL filters match, so scope and field mapping must be reviewed together. |
These documented capabilities do not establish that one approach is more accurate, faster, cheaper, or easier overall. Choose based on whether you need a hosted request, a desktop crawl, or configuration-managed fields.
How to diagnose an empty or incorrect result
The selected content is missing
Inspect the raw response and the browser-rendered DOM separately. If the raw HTML does not contain the value, enable JavaScript rendering or use a rendering-capable path, then account for the page’s actual timing.
Free tools Windows power users keep installed
One-click scans. No signup required.
Rank #3
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
The rule selects the wrong element
Inspect surrounding markup and narrow the selector to a distinctive element or attribute. Visual selector suggestions are a starting point, not proof that the expression will remain correct on other templates.
The rule works on one URL only
Compare the page structures and check URL filters. Elastic documents filters such as begins, ends, contains, and regex; an otherwise correct rule will not run when the URL is outside its configured scope.
Rank #4
- Dual Converters, Infinite Potential：Includes 2× USB C male to USB A female adapters and 2× USB A male to USB C female adapters. Perfect for a wide range of uses—tablets with Bluetooth keyboards, expand USB ports on macbook, and more. Two different converters for all your daily needs
- Next-Level 10Gbps & 3A Charging: No more slow 480Mbps, this usb to usb c adapter has a transfer speed of up to 10Gbps, allowing you to do more transferring in less time. This usb adapter fits both USB A and USB C charger, supporting up to 3A fast charging
- Upgraded Exquisite Craftsmanship: With an aluminum alloy housing and metal connector, the usbc to usb adapter is extremely durable and sturdy. Rigorously tested to withstand more than 10,000 times of plugging and unplugging, ensuring long-lasting performance
- Broad Compatible: The usb c to usb adapter widely supports all USB C/ USB A devices like laptops, tablets, cellphones, car chargers, and phone chargers. Such as compatible with MacBook Pro/Air 2023/2022, Thunderbolt 4/3 Devices,Apple MagSafe Watch 9/8/7/SE/Ultra, iPad Pro 2022/2021, Samsung Galaxy S23/S20/S10, and iPhone 17/16/15 Pro. Plug and play
- Please Note: To reach 10Gbps speed, keep the cable under 3.3 ft. For USB A Male to USB C adapters, try flipping the USB C connector. USB C Male to USB A adapters support bidirectional 10Gbps transfer within 3.3 ft
Several values are returned
Decide whether multiple matches are meaningful. Preserve them as separate values when each item matters, or join them with an explicit separator when the destination field is intentionally scalar.
A pattern captures too much
Add capture groups and map the group you need rather than returning the full match. This is especially important for dates or identifiers embedded in longer URLs or strings.
Recommended Free Tools
Best Value
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
Use screenshots to verify what a rendered page shows
When a field is populated after scripts run, a screenshot can provide a quick visual check against the value your extractor returns. ScreenshotNeo is a website screenshot API and MCP server; it is not a replacement for your selector or field-mapping rule, but it can capture the rendered state you are validating.
Or skip the browser setup
For a rendered-page snapshot, make one request to ScreenshotNeo (see the API documentation):
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpimport requests; r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90); open("shot.webp", "wb").write(r.content)const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' }); const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);- Cookie and consent banners, newsletter popups, and chat widgets are removed before the shot; each cleanup step can be turned off.
- Bot checks or CAPTCHAs, blank pages, timeouts, failed loads, and cache hits are not billed; response headers identify the page verdict and billing status.
- An MCP server provides
take_screenshot,get_page_info, andcapture_pdftools for Claude, Cursor, and other MCP clients. - The Free plan includes 1,000 screenshots per month with no card. Paid plans start at $5 for 3,000 shots; every feature is available on every plan.
Create a free ScreenshotNeo account to get started.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
