If you need search results in a program, first check the search engine’s rules and use an authorized API or other approved route. For Google, its published policy says automated queries—including scraping results for rank checking or other automated access without express permission—violate its spam policies and Terms of Service. Google documents a JSON API for results from a configured Programmable Search Engine, but it is closed to new customers and existing customers are told to transition by January 1, 2027. This guide explains how to choose an access route without treating a technical workaround as permission.
What scraping search results means—and why the target matters
Scraping search engine results means collecting results pages or result data automatically, rather than reading them manually. Before choosing a method, specify the engine and the fields you need: organic links, titles, snippets, ads, local results, or another feature. An interface that returns links in JSON may not provide every element shown on a rendered results page; confirm coverage rather than assuming parity.
This differs from crawling ordinary websites. A crawler visits pages on those sites, while a search-results collector requests data from the search engine itself. The rules and technical interfaces are not interchangeable: check the search engine’s access policy for results, and consult each destination website’s crawler guidance separately.
Check the provider’s rules before writing a collector
Google’s Search spam policy specifically identifies automated queries to Google Search, including scraping results for rank checking and other automated access without express permission. Google states: “Such activities violate our spam policies and the Google Terms of Service.” Read Google’s machine-generated traffic policy and Google’s Terms of Service before designing a Google-related workflow.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →#1 Best Overall
That is Google’s stated policy and contractual position, not a universal legal ruling about every search engine or every circumstance. This guide does not establish legal permission to automate access. For another provider, locate its current terms and automated-access policy rather than assuming Google’s rules—or an API’s existence—settle the question.
Use an official or authorized data route where available
Google Custom Search JSON API
Google documents the Custom Search JSON API as a way to retrieve results in JSON from a configured Programmable Search Engine. It requires an API key and a configured engine. See Google’s API introduction for the interface and setup details.
Availability is a significant constraint. Google’s overview says the API is closed to new customers; existing customers have until January 1, 2027 to transition. The same overview states an allowance of 100 free queries per day, with additional queries available for a fee. These are changeable product details, so confirm current eligibility, limits, pricing, and transition instructions directly with Google before building around them. Google lists Vertex AI Search for searching up to 50 domains and says it is gathering interest for a full-web-search solution; do not assume these are equivalent to the Custom Search JSON API or provide access you need. Check the current API overview.
Because access requires an eligible account, an API key, and a configured Programmable Search Engine, an example request should not be mistaken for a universally available route. If you already have authorized access, follow the current API documentation for request parameters, response fields, error handling, and billing. Do not build a new production dependency until you have confirmed that your account is eligible and that the transition schedule works for your use case.
Evaluating a third-party SERP API
A third-party service may offer structured result data, but the name “API” alone does not establish permission, coverage, or suitability. This research does not verify any particular vendor or program. Before adopting one, get clear answers to the following:
- Authorization and eligibility: What access model does the provider describe, and what uses does its agreement allow?
- Fields and result types: Does the response include the exact organic, ad, local, or other data your application needs?
- Geography and language: Can you request the markets and languages you need, and how are location settings represented?
- Limits and cost: What are the quotas, rate limits, overage charges, and total cost at your expected volume?
- Storage and reuse: How long may results be retained, displayed, or redistributed?
- Operations: What are the provider’s documented error responses, update timing, and reliability commitments, if any?
Compare providers using their current documentation and contract terms. Do not infer data rights or regional accuracy from a successful sample response.
How to build a compliant, maintainable collection workflow
- Write down the requirement. Specify the engine, result types and fields, target geographies and languages, expected volume, and whether the data must be stored, displayed, or reused.
- Read the current policy and terms. If the provider does not permit your proposed automated access, do not treat a different client, browser, or request pattern as authorization.
- Check approved interfaces and eligibility. Confirm that the API or other route permits your intended use, returns the necessary fields, and is actually available to your account. Record quotas, pricing, and any transition dates.
- Estimate cost and operational load. Model normal volume and likely retries against documented limits and fees. Add headroom without exceeding the provider’s rules.
- Implement conservative handling. Use the documented interface, modest request volume, clear identification where permitted, and limited data retention. Handle errors explicitly and provide a stop path for blocks or access challenges.
- Recheck the contract and product status. Policies, API eligibility, pricing, and response formats can change. Review them before launch and periodically while the integration is in use.
Why robots.txt does not authorize search-result scraping
robots.txt is a mechanism for communicating crawler access preferences and managing crawler traffic on a website. Google explicitly says it is not a way to ensure a URL is absent from Search. It is not a substitute for a search engine’s terms, and a robots.txt file does not establish legal authorization to collect search results. For the scope of the file, see Google’s robots.txt documentation.
When crawling an ordinary website, consult that site’s crawler guidance separately. A rule for a site’s pages does not grant permission to automate queries to a search engine, and the search engine’s policy does not define every destination site’s rules.
Reliability, performance, and cost considerations
Do not optimize an unauthorized collection method for speed or scale. For an approved API, design around its documented quota and rate limits rather than assumptions about how quickly requests will succeed. At minimum, distinguish successful responses from errors, avoid retry loops that multiply traffic, keep only data you are entitled to retain, and stop when the provider blocks requests or returns an access challenge.
Estimate total cost from expected query volume, the provider’s current pricing, and its retry and overage rules. Include maintenance work: API changes, availability changes, eligibility limits, and a migration path. Google’s stated transition date for existing Custom Search JSON API customers is a concrete example of why product status belongs in the design, not just the initial setup.
For reliability, define what your application should do when results are unavailable: defer the job, surface an error, or use a separately authorized fallback. Do not silently switch to a different access method whose terms or data coverage have not been checked.
Common problems and practical fixes
- You cannot obtain a Google API key or enable the API. The Custom Search JSON API is closed to new customers according to Google’s overview. Verify your account’s eligibility and current options; do not assume a copied API example grants access.
- Your existing integration is approaching a deadline. Google’s overview gives existing customers until January 1, 2027 to transition. Check the latest migration information and identify an authorized replacement that actually supports your required data.
- The API response lacks a result feature you expected. Confirm the API’s documented output and the configuration of your Programmable Search Engine. Do not assume its JSON mirrors every element of a general web results page.
- Requests are blocked or challenged. Stop automated requests and review the provider’s rules and your authorization. Do not respond by increasing request volume or disguising the client.
- Your cost estimate is wrong. Recalculate with current quotas, paid-query pricing, retries, and expected volume. Google’s published 100-free-queries-per-day figure is a changeable service allowance, not a guaranteed permanent limit.
- You are relying on robots.txt to decide whether search queries are allowed. It addresses crawler access to website URLs; it does not determine the search engine’s terms or grant authorization for automated result queries.
What the current litigation does—and does not—establish
Publicly reported procedural developments in Google LLC v. SerpApi do not provide a universal yes-or-no answer about scraping. A reproduced opinion reports a July 20, 2026 dismissal; an August 25, 2026 post by SerpApi says Google filed an amended complaint and SerpApi moved to dismiss it. The current docket state and the precise implications of the ruling are not established here. These developments should not be read as blanket permission to scrape Google or any other engine. For the reported materials, see the case opinion reproduction and SerpApi’s August 25 update.
Free tools Windows power users keep installed
One-click scans. No signup required.
Best Value
Or skip the browser setup
If your actual task is to capture a website page—not collect a search engine’s result data—a screenshot API is a different tool for a different job. ScreenshotNeo is a website screenshot API and MCP server from ScreenshotNeo; it captures a supplied URL as an image or PDF, not a SERP data feed. For search-result collection, use an authorized search-data route instead.
One GET request can capture a page. The following cURL example uses Stripe as the target; replace the URL with the page you are authorized to capture. See the ScreenshotNeo documentation for request options.
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp
ScreenshotNeo removes cookie/consent banners, newsletter popups, and chat widgets before capture; each step can be turned off. Bot checks, blank pages, timeouts, failed loads, and cache hits cost nothing, with response headers identifying the page verdict and billing status. Its MCP server provides take_screenshot, get_page_info, and capture_pdf for AI agents. The free plan includes 1,000 screenshots per month with no card; paid plans start at $5 for 3,000. Sign up free for 1,000 screenshots a month, with no card required.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallOutdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchFrequently Asked Questions
Does this guide establish whether scraping search results is legal?
No. It explains provider rules and access routes, not a legal determination. Seek qualified legal advice for a legal assessment of your circumstances.
Can I use a website screenshot API to collect structured SERP data?
No. A screenshot API captures a rendered page as an image or PDF; it is not a substitute for an authorized search-results data interface.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
