How to Extract Structured Data from Search Results with an API

iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more

To extract structured data from search results, call a search API, inspect its JSON response, and map each result into a schema your application controls. A Google Custom Search JSON API request uses an API key (key), a Programmable Search Engine ID (cx), and a query (q). The response normally gives you result metadata such as a title, URL, and snippet; richer values such as authors, dates, ratings, or prices are conditional on the source page markup and on what the provider exposes.

The important implementation decision comes first: choose an API whose search scope and result types match your application. Google says its Custom Search JSON API is closed to new customers and that existing customers must transition by January 1, 2027, so a new integration should evaluate alternatives before production work begins.

#	Preview	Product	Price
1		Compressed Air Duster,130000RPM Super Power Cordless Air Duster,Rechargeable Brushless Blower,3 Gear...	$21.59	Buy on Amazon
2		EVEO® Screen Cleaner Spray Kit (Streak-Free) TV Screen Cleaner for Smart TV, Lcd, LEd, Oled |...	$15.97	Buy on Amazon
3		BladeHawks RGB Gaming Mouse Pad Extra Large Extended 31.5x12 Inch-Black | LED mouse pad with 14...	$13.99	Buy on Amazon
4		Vaydeer Wrist Rest for Keyboard and Mouse, Computer Ergonomic Wrist Support Pad, Soft Memory Foam...	$15.99	Buy on Amazon
5		Under Desk Cable Management Tray No Drill, Metal Mesh Cable Management Under Desk with Clamp Mount,...	$13.89	Buy on Amazon

What structured data means in a search API

Search results arrive as machine-readable records rather than as a page of visual links. The basic record usually contains a result title, destination URL, and text snippet. A provider may also expose metadata about the query, the search engine configuration, estimated result counts, and additional result-specific objects.

That basic result record is different from structured data extracted from the destination page. A page can publish machine-readable fields such as an author, publication date, rating, price, or product identifier in JSON-LD, Microformats, RDFa, Microdata, PageMap data, or selected meta tags. The search provider may crawl and interpret some of those fields, but their presence is not guaranteed. A result with no rating field means “not supplied or not exposed,” not a rating of zero.

#1 Best Overall

Sale

Compressed Air Duster,130000RPM Super Power Cordless Air Duster,Rechargeable Brushless Blower,3 Gear Adjustable Blower with LED Light,Dust Remover for Computer/House/Outdoor/Car

• Powerful Turbo Fan:CAGIWIRU Electric Air Duster have reached speed up to 130000RPM,Can efficient remove dust and debris.With LED light and Three Adjustable Speed to meet with different cleaning tasks.
• With Attached USB C Fasting Charging cable.Charing 3 hours,Enjoy up to 240 minutes of use on the lowest setting, For third speed can use for 25mins.with four charging options to meet with your needs.
• Wide Application: attached 5 different nozzles,making it suitable for a variety of scenes, such as car,pet, pc, keyboards, and other electronic devices. It also serves for office and home clean duster.
• Handy Design & Portable:Cordless air duster electric nozzle is Light weight and small size, design and comfortable to hold and space saving, convenient to carry around. You can put it in your room or inside the car without taking up space.
• Economical &Reusable: Built durable materials with a long-lasting battery,our CAGIWIRU air duster it's a sustainable alternative to disposable air cans.

$21.59

Buy on Amazon

Design your own data model around that distinction:

• Search context: the query, provider, search-engine identifier, locale or location settings, and retrieval time.
• Stable result fields: title, canonical result URL, and snippet when supplied.
• Optional extracted fields: page metadata such as dates, authors, ratings, prices, or provider-specific objects.
• Traceability: the original response, source URL, and mapping version when your application needs auditing or reprocessing.

Do not assume that an API returning JSON has converted every page into a complete structured record. Treat optional properties as nullable and preserve the difference between an absent field, an empty string, and a real numeric zero.

Choose the right search API before writing a parser

“Search API” can mean a restricted search over selected domains, a full-web-style search-results page, or a service focused on one result type such as news or shopping. The same parser will not fit all three. Define the scope and fields your application actually needs.

Google describes Vertex AI Search as an alternative for searching up to 50 domains. That is a defined-domain option, not documentation of an equivalent full-web SERP feed. Google directs full-web use cases to contact it about a separate full-web search solution. A provider such as SerpApi documents Google and Bing search API endpoints with categories including organic, local, ad, knowledge graph, direct-answer, image, news, shopping, and video results. Those documented categories establish what its endpoints can represent; they do not prove equivalent coverage, reliability, or price for your workload.

Google Custom Search JSON API request anatomy

Google documents the Custom Search JSON API at https://www.googleapis.com/customsearch/v1. The documented request is an HTTP GET with three essential inputs:

• key: your Google API key.
• cx: the identifier of your configured Programmable Search Engine.
• q: the search expression.

A minimal request looks like this:

curl 'https://www.googleapis.com/customsearch/v1?key=YOUR_API_KEY&cx=YOUR_SEARCH_ENGINE_ID&q=structured+data+API'

Keep the key out of browser code, public repositories, logs, and error messages. Store it in a secret manager or environment variable on the server that makes the request. URL-encode the query rather than concatenating untrusted text into a URL manually.

Rank #2

EVEO® Screen Cleaner Spray Kit (Streak-Free) TV Screen Cleaner for Smart TV, Lcd, LEd, Oled | Computer Screen Cleaner Spray and Wipe Microfiber Cloth - TV, Monitor, Laptop, iPad, Car - 6.7oz 1 Pack

• Screen Cleaner is Your Screen’s New Bestfriend! - EVEO is proud to present a screen cleaner that’s perfectly suited for your TV. Meet your new screen cleaner, computer screen cleaner, laptop cleaner, iPad cleaner, Macbook,tv cleaner or any other electronic device. Our screen cleaner spray and wipe along with the included plush microfiber cloth, easily removes dust, fingerprints & other smudges on the screen’s surface. Clean effectively for a streak-free screen
• Gentle on your Screens - Our screen cleaner spray and wipes is compatible to be used even for the most sensitive LCD, LED, CRT, and OLED screens in proper use. Be at ease knowing that this product is compatible with all the major brands: LG, SONY, Samsung, Sharp, iPad, iPhone, Android screens, Kindle, PC monitors - we’ve got you covered! Can be used as LCD screen cleaner, tv cleaner, smart tv screen cleaner, and more
• Premium Super Soft Plush Microfiber Cloth - Included in this efficient screen cleaning kit. This screen cleaner spray and wipe has our signature plush microfiber cloth specially designed to comply with screens and monitors and leave them streak-free. Has been shown to effectively absorbs stubborn stains, zero streaks, and no fiber shedding. Always use this together with our screen cleaner for the best results
• Easy-to-Use Screen Cleaner Kit + microfiber cleaning cloth - The EVEO Screen Cleaning Kit will easily be your new favorite tool when it comes to taking care of the things you love most, quick and easy solution for a streak-free screen. Simply screw on the sprayer gun on the screen cleaner bottle. Spray the cleaning liquid onto the microfiber cloth, then wipe the screen until desired cleanliness & shine is achieved.
• Quality Screen Care Kit Worth Purchasing - Enjoy a convenient Screen Cleaner anytime, anywhere you need it. Your satisfaction is our top priority. We stand behind the quality of our products and that’s why, you shouldn’t be worrying at all. We manufacture the Best Screen Cleaner. and can be used for laptop cleaning kit, laptop screen cleaner, tv screen cleaner for smart tv. MacBook screen cleaner, monitor cleaner or screen cleaner spray for electronic devices with microfiber cleaning cloth

$15.97

Buy on Amazon

In Python, a request and basic status check can be written as:

import os
import requests

endpoint = 'https://www.googleapis.com/customsearch/v1'
params = {
    'key': os.environ['GOOGLE_API_KEY'],
    'cx': os.environ['GOOGLE_SEARCH_ENGINE_ID'],
    'q': 'structured data API',
}

response = requests.get(endpoint, params=params, timeout=20)
response.raise_for_status()
payload = response.json()

The timeout, secret handling, and exception policy are application choices. In production, record the provider’s HTTP status and an internal request identifier, but avoid logging the API key or complete responses if they could contain sensitive query content.

Inspect the JSON response before mapping fields

The response reference describes query metadata, search information, engine metadata, and an items collection. A result item can include fields such as title, link, snippet, and pagemap. Optional properties can be absent, so inspect real responses from the exact search engine configuration you will deploy.

This abbreviated payload illustrates the shape without implying that every response contains every property:

{
  "queries": {
    "request": [{ "searchTerms": "structured data API" }]
  },
  "searchInformation": {
    "totalResults": "estimated value",
    "formattedSearchTime": "provider-formatted value"
  },
  "items": [
    {
      "title": "Example result",
      "link": "https://example.com/page",
      "snippet": "Text shown for the result...",
      "pagemap": {
        "metatags": [{ "og:title": "Example result" }]
      }
    }
  ]
}

totalResults is an estimate, not an exact inventory of all matching pages. Use it for display or rough reporting, not for asserting that a precise number of pages exists.

Map only the fields your application needs and retain the original URL and query context:

Rank #3

Sale

BladeHawks RGB Gaming Mouse Pad Extra Large Extended 31.5x12 Inch-Black | LED mouse pad with 14 lighting modes, covers keyboard & mouse, anti-slip rubber base, waterproof surface, USB powered

• 【Large size】：Measuring 800 x 300 mm/ 31. 5×12 inches, which is wider and taller than others. Large mouse pad can perfectly fit your keyboard and mouse of any size, and there is also ample space for peripherals such as phones, tablets, etc.
• 【LED backlight mouse pad】：Blade Hawks RGB mouse pad provides 7 static modes and 7 dynamic modes will give you the options you are looking for. Power-off memory function remembers the last lighting mode you selected. The super glow fiber Chroma will give you a colorful gaming setup that you want.
• 【Anti-slip rubber base】：Made from Natural rubber, this gaming mousepad will firmly grip your desktop, allowing you to enjoy the ultimate precision in the games you are most passionate about. It also comes with a texture that enhances this ability even further.
• 【Smooth and waterproof surface】: Micro-textured cloth surface, for extremely precise mouse control and a smooth movement. When liquid splashes on the gaming mouse pad, it forms water droplets and slides down. Will not delay your work or gameplay.
• 【Plug and play】: This mouse and keyboard pad is powered by USB, plug and play, no driver required. One button control light modes. Press one time to change the lighting mode, press twice to change the brightness, press and hold about 3 seconds turn ON/OFF.

$13.99

Buy on Amazon

def map_result(item, query):
    page_map = item.get('pagemap') or {}
    return {
        'query': query,
        'title': item.get('title'),
        'url': item.get('link'),
        'snippet': item.get('snippet'),
        'pagemap': page_map,
    }

Using .get() and an explicit empty object for pagemap prevents a missing optional property from crashing the parser. Validate that a URL is present before enqueueing a record for downstream processing; a malformed or absent URL should be rejected or quarantined rather than silently converted into a usable-looking record.

Separate ordinary result data from page metadata

Store the ordinary result fields and extracted page metadata in separate namespaces. This avoids overwriting a provider’s snippet with a page’s description or treating a search-engine title as the authoritative page title.

• Result title: the title supplied in the search result object.
• Result URL: the link supplied by the provider, retained for traceability.
• Snippet: provider-generated or provider-selected text; it may be truncated or query-dependent.
• PageMap and metadata: structured values exposed from the crawled page, when available.
• Application fields: normalized values such as published_at, author, or rating_value, each with a documented source and conversion rule.

When normalizing a value, keep the raw representation alongside the normalized one if later review matters. For example, a page date can be stored as the original string plus a parsed timestamp and a flag indicating that parsing succeeded. Do not infer a missing date from the order of search results.

Use structured operators only when the markup supports them

Programmable Search Engine can filter, sort, bias, and restrict results by range using supported structured attributes. The relevant source formats include PageMap, selected metadata tags, estimated page dates, JSON-LD, Microformats, RDFa, and Microdata. Operator support differs by format.

For example, a filter that works against a PageMap or metadata attribute should not be copied unchanged to a JSON-LD-only site. First identify the source format, then select the operator documented for that format. If a field is not exposed in the response, do not claim that the filter proved the field’s value; it may only have influenced retrieval.

Build a resilient extraction pipeline

A dependable integration is more than one HTTP request. Use a staged pipeline so that provider changes and incomplete pages do not corrupt your application data.

Rank #4

Sale

Vaydeer Wrist Rest for Keyboard and Mouse, Computer Ergonomic Wrist Support Pad, Soft Memory Foam Arm Cushion for Desk, Palm Hand Office Laptop Typing

• 【Softer and More Comfortable】Vaydeer wrist rest has unique diamond pattern, which is the combination of softness and aesthetics. The materials of wrist rest are improved into higher quality memory foam and covered with silky smooth lycra. The computer wrist rest makes you as comfortable and cushiony as like rest your wrists on clouds.
• 【Ergonomic Wrist Saver】The wrist rests for keyboard and mouse comes with a 17.32×3.15×0.83 inch keyboard wrist pad and a 5.94×3.15×0.83 inch mouse wrist support. Based on ergonomic design, the unique concave shape is the perfect fit for your wrist joints. The wrist rest pad fits most computer keyboards and laptops, improve hand and wrist posture, release your wrist and arm stress.
• 【Non-Slip Rubber Bottom】Featuring an anti-skid silicone base on the bottom, this wrist keyboard support stays firmly in place on your desk, preventing the padding from sliding around, ensuring stable and consistent wrist support during extended computer sessions.
• 【Better Experience & Pain Relief】Our keyboard arm rest is beneficial to alleviate the soreness caused by direct contact and friction between your arm and a hard desk surface, reducing the risk of wrist fatigue or carpal tunnel. The soft texture of memory foam can evenly distribute the pressure around your wrists and provide good support with just enough give.
• 【Helpful in Multiple Scenarios】Whether you're working, studying, writing, typing, gaming, this keyboard and mouse rest combo is an essential accessory to add comfort and support to your hands and wrists. It’s also a great gift for men, women, family, friend, coworker, gamer, teacher, etc.

$15.99

Buy on Amazon

1. Define the contract. Write down required fields, nullable fields, accepted URL schemes, date and number formats, and whether snippets are for display or analysis.
2. Capture request context. Store the query, provider name, search-engine identifier, locale or location settings, retrieval timestamp, and an internal correlation ID.
3. Call the endpoint. Apply a bounded timeout, authenticate with a server-side key, and classify HTTP failures separately from valid responses containing zero items.
4. Validate the top level. Confirm that the decoded body is an object and that items, when present, is an array. Preserve unknown top-level fields rather than discarding them blindly.
5. Normalize each item. Read title, link, and snippet defensively. Keep pagemap or equivalent provider objects separate from normalized application fields.
6. Handle optional metadata. Treat absent authors, dates, ratings, and prices as null. Do not turn missing values into zero, “unknown,” or a fabricated default that could affect ranking.
7. Validate and deduplicate URLs. Normalize only transformations you can justify, preserve the provider’s original link, and define how redirects, tracking parameters, fragments, and duplicate URLs are handled.
8. Persist provenance. Record which provider field produced each normalized value and which parser version performed the mapping.
9. Monitor drift. Alert on sudden changes in item counts, missing required fields, unknown result shapes, authentication failures, and quota responses.
10. Test with fixtures. Keep representative responses containing a normal item, an item without pagemap, an empty result set, a malformed optional field, and a provider error. Re-run them whenever the mapping code changes.

Separating retrieval from mapping lets you replay a stored response when your schema evolves. It also makes a provider migration less disruptive: the application consumes your internal record format while provider-specific adapters handle each external schema.

Example: a defensive Python extractor

The following example returns normalized records and keeps optional metadata intact. It is a mapping pattern, not a guarantee that every response contains the illustrated fields.

import os
from typing import Any
import requests

ENDPOINT = 'https://www.googleapis.com/customsearch/v1'

def search_records(query: str) -> dict[str, Any]:
    params = {
        'key': os.environ['GOOGLE_API_KEY'],
        'cx': os.environ['GOOGLE_SEARCH_ENGINE_ID'],
        'q': query,
    }
    response = requests.get(ENDPOINT, params=params, timeout=20)
    response.raise_for_status()
    payload = response.json()

    raw_items = payload.get('items') or []
    if not isinstance(raw_items, list):
        raise ValueError('items is not an array')

    records = []
    for item in raw_items:
        if not isinstance(item, dict):
            continue
        url = item.get('link')
        if not isinstance(url, str) or not url.startswith(('http://', 'https://')):
            continue
        records.append({
            'query': query,
            'title': item.get('title'),
            'url': url,
            'snippet': item.get('snippet'),
            'pagemap': item.get('pagemap'),
        })

    return {
        'query': query,
        'estimated_total': (payload.get('searchInformation') or {}).get('totalResults'),
        'records': records,
        'raw': payload,
    }

The function deliberately does not invent a value when items is absent, and it does not treat the estimated total as the number of records returned. A production service should add provider-specific error handling, bounded retries for transient failures, quota monitoring, and a retention policy for raw responses.

Missing data, schema changes, and failure handling

Missing optional fields

Use nullable columns or explicit option types for metadata. A missing pagemap should not prevent you from storing a valid title, URL, and snippet. If your business rule requires an author or date, mark the record incomplete and route it to a separate workflow rather than guessing.

Empty results

An empty items array is a valid outcome and should be distinct from an authentication error, a quota response, a timeout, or invalid query parameters. Record the provider status and response body classification so operators can tell “no matches” from “the request failed.”

Unexpected result shapes

Preserve unknown fields in quarantined raw payloads and alert on structural changes. Avoid strict deserializers that reject every new provider field, but remain strict about the fields your application cannot safely operate without, such as a destination URL.

Best Value

Sale

Under Desk Cable Management Tray No Drill, Metal Mesh Cable Management Under Desk with Clamp Mount, 2 Hole Cable Tray with Wire Management and Cord Organizer for Home Office Standing Desk

• 【No Drill Mounted】Cable management tray can be assembled in as little as 3 minutes or less-simply. Side clips easily clampe the edge of desk, pass through the holes on the sides, and secure excess cables with baskets. Desk cable management supports inward or outward installation to meet your needs in different scenarios, and provides you with great convenience when collecting and organizing wires.
• 【Rubber Pad Protects Desk】Wire organizer under desk built-in soft rubber pads on the clamp protect your desk from scratches, so you will not damage your furniture or office.
• 【Sturdy and Beautiful】Cord management under desk is made of high-quality carbon steel and holds about 15 lbs. The wire basket has 2 holes on each side of the tray to enable your cables to pass freely. Cable baskets are suitable for desk thickness use from 0.4" to 2.4".
• 【Space Saving & Keep Organized】Desk cable management tray holds all your power cords, outlet strips, USB hubs and computer transformers hidden from your feet, and the data cables will not enter the vacuum cleaner to keep your desk clean and tidy. Desk wire organizer is also suitable for use in the kitchen and outdoors.
• 【Reduce Safety Risks】Cable tray under desk keeps all plugs away from your kids, no more worry about tripping. Using 2 holes wire tray saves extra space, helps you easily cabling and wire protection. The mesh design is not easy to accumulate dust, but avoids safety accidents caused by messy wires.

$13.89

Buy on Amazon

Incorrect structured values

Structured markup is supplied by source pages and can be incomplete, stale, or inconsistent. Validate dates, currencies, numeric ranges, and URLs before using them in calculations or user-visible labels. Keep the provider’s value and your validation result so a correction does not erase the original evidence.

Retries and quotas

Retry only failures that are plausibly transient, with exponential backoff and a maximum attempt count. Do not retry malformed requests or invalid credentials indefinitely. Expose quota consumption and failure rates as metrics, and set an operational budget before enabling high-volume jobs.

Google’s lifecycle and migration decision

Google’s official overview states that the Custom Search JSON API is closed to new customers. Existing customers have until January 1, 2027 to transition. The same overview identifies Vertex AI Search for searches covering up to 50 domains and asks full-web users to contact Google about its full-web search solution. Treat these statements as current vendor guidance, not a permanent guarantee; verify availability when you start implementation and again before launch.

Build a provider adapter rather than spreading Google-specific field names throughout your application. Keep an internal record such as provider, query, title, url, snippet, result_type, structured, and retrieved_at. Then map each provider’s response into that contract and retain the original provider payload for troubleshooting where your retention policy allows.

Published Google pricing terms for existing customers

Google’s overview lists the following terms for existing customers until the January 1, 2027 discontinuation: 100 search queries per day free, then $5 per 1,000 additional queries, with a limit of 10,000 queries per day. These are Google’s published terms as accessed in 2026; they are not a new-customer offer. Confirm the current account-specific terms before budgeting.

Estimate usage from actual jobs, retries, staging traffic, and cache misses rather than from the number of users alone. A parser that retries every failed request or executes several queries per page can consume a quota much faster than a simple one-query workflow.

Quick Recap

SaleBestseller No. 1

Compressed Air Duster,130000RPM Super Power Cordless Air Duster,Rechargeable Brushless Blower,3 Gear Adjustable Blower with LED Light,Dust Remover for Computer/House/Outdoor/Car

$21.59

Bestseller No. 2

EVEO® Screen Cleaner Spray Kit (Streak-Free) TV Screen Cleaner for Smart TV, Lcd, LEd, Oled | Computer Screen Cleaner Spray and Wipe Microfiber Cloth - TV, Monitor, Laptop, iPad, Car - 6.7oz 1 Pack

$15.97

SaleBestseller No. 3

BladeHawks RGB Gaming Mouse Pad Extra Large Extended 31.5x12 Inch-Black | LED mouse pad with 14 lighting modes, covers keyboard & mouse, anti-slip rubber base, waterproof surface, USB powered

$13.99

SaleBestseller No. 4

Vaydeer Wrist Rest for Keyboard and Mouse, Computer Ergonomic Wrist Support Pad, Soft Memory Foam Arm Cushion for Desk, Palm Hand Office Laptop Typing

$15.99

SaleBestseller No. 5

Under Desk Cable Management Tray No Drill, Metal Mesh Cable Management Under Desk with Clamp Mount, 2 Hole Cable Tray with Wire Management and Cord Organizer for Home Office Standing Desk

$13.89

Security, privacy, and compliance checks

• Keep API credentials on a server and restrict them to the APIs and environments that need them.
• Redact keys, authorization headers, and sensitive query strings from logs.
• Decide how long raw responses, snippets, and query histories may be retained.
• Review whether a query can contain personal, confidential, or regulated information before sending it to a provider.
• Check provider rules for caching, display, transformation, and redistribution. The existence of a JSON response is not universal permission to republish it.
• Document the source and timestamp of normalized page metadata so users can distinguish provider data from your own calculations.

Implementation checklist

1. Choose a provider based on search scope, result types, structured fields, geography, reliability, terms, and cost.
2. Configure the search engine context and credentials; for Google’s documented API, that means an API key and cx identifier.
3. Send the query as an encoded GET request and protect the key.
4. Inspect real JSON responses before finalizing your schema.
5. Map title, URL, and snippet separately from optional PageMap or metadata fields.
6. Use structured operators only where the provider documents support for the page’s markup format.
7. Represent absent values as absent, not as zeros or guesses.
8. Validate URLs and typed values, preserve provenance, and retain raw payloads according to policy.
9. Instrument errors, quotas, latency, schema drift, and retry counts.
10. Recheck provider eligibility, lifecycle, pricing, and terms before production deployment.

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.