The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Start with permission, not code. A public listing page is not automatically yours to copy, store, republish or resell. For a production dataset, first look for a licensed API, MLS feed or broker agreement. If an authorized HTML source is your only option, collect conservatively, preserve raw evidence, normalize the fields, and track listing history.
What “scraping listings” should mean in practice
A useful listing pipeline does more than download HTML. It produces records that can be compared over time and whose permitted use is documented. Define these items before selecting a source:
- Geography, property types, and sale, rental or both.
- Fields: listing ID, canonical URL, address, price and currency, beds, baths, area and units, property type, status, broker or agent data, and permitted image references.
- Refresh interval, retention period, and whether the data is for internal analysis, a customer-facing search page, resale or advertising.
- Quality requirements such as acceptable missing fields, geocoding confidence and how quickly status changes must appear.
Treat every field as licensed content until the source agreement says otherwise. A price, address or bedroom count may be factual, while the surrounding description, photographs, logos, agent details and video can have separate copyright, trademark, privacy or contractual restrictions.
Check permission before sending automated requests
Terms and licenses control the use
Realtor.com’s Move Network Terms of Use prohibit scraping, screen scraping, database scraping and other automated collection without express written permission. Zillow’s terms restrict reproducing or publicly displaying listing data and images on another service unless the applicable terms expressly allow it. “I could see it in a browser” is not a license to copy it.
#1 Best Overall
National Association of Realtors Policy Statement 7.85 says listing brokers should own, or have authority to license, photographs, images, graphics, audio/video, descriptions, remarks, pricing and other listing details submitted to an MLS. That policy is a warning that rights can exist at several layers, even when a page is publicly reachable.
Inspect the allowed access paths
- Read the target site’s current Terms of Use and privacy notice.
- Read its API documentation, partner program and any MLS or broker agreement.
- Check
robots.txtfor the publisher’s crawler preferences. It communicates instructions but does not replace contractual permission. - Ask for written authorization when the terms are unclear. Record the account, API key, allowed fields, rate limits, attribution wording, storage duration and redistribution rules.
Recheck those documents when your geography, audience or product changes. A license for internal analytics may not cover a public search site, resale, model training or displaying photographs.
Choose an API, feed or HTML collector
Compare sources against the same operational and legal criteria rather than choosing the easiest page to parse.
| Criterion | Licensed API or MLS/broker feed | Authorized HTML collection |
|---|---|---|
| Permission scope | Usually stated in a contract, API terms or feed agreement; verify display and redistribution limits. | Must be established separately; page visibility alone is insufficient. |
| Schema and field completeness | Documented fields and identifiers are easier to validate. | Depends on markup and can change without notice. |
| Freshness | May provide webhooks, incremental updates or a documented polling interval. | Requires your own polling schedule and change detection. |
| Reliability and limits | Published quotas, authentication and support may be available. | You must honor site limits and handle layout, denial and timeout failures. |
| Rights and attribution | Contract usually specifies branding, storage and permitted display. | Every copied field and asset needs a defensible permission basis. |
| Maintenance cost | Integration work is front-loaded; schema changes are normally announced. | Parser maintenance, monitoring and policy review are ongoing. |
For a commercial or public product, a licensed API or MLS/broker feed is generally the safer starting point. An HTML collector is appropriate only when the source permits it and you can stop quickly if its policy or technical behavior changes.
Build a respectful collector for an authorized HTML source
Request behavior
- Identify your application with a clear user agent and contact address.
- Use conservative concurrency, a delay between requests, caching and conditional requests such as
If-None-MatchorIf-Modified-Sincewhen supported. - Use exponential backoff for transient 429 and 5xx responses. Stop on repeated denials, CAPTCHAs, authentication challenges or a changed policy.
- Never bypass a login, paywall, CAPTCHA, bot check or other technical control.
- Keep an audit record of request time, response status, source URL and parser version.
Example Python collector
The following example is a starting point for a source that has authorized HTML access. It prefers JSON-LD, keeps the original response, and writes a normalized record. Replace the URL and selectors only after confirming that your agreement permits the collection.
Rank #2
import json
import time
from datetime import datetime, timezone
from decimal import Decimal, InvalidOperation
import requests
from bs4 import BeautifulSoup
URL = "https://authorized.example/listing/123"
HEADERS = {"User-Agent": "MyListingResearchBot/1.0 (contact: data@example.com)"}
def money(value):
if value is None:
return None
text = str(value).replace(",", "")
digits = "".join(ch for ch in text if ch.isdigit() or ch == ".")
try:
return str(Decimal(digits)) if digits else None
except InvalidOperation:
return None
def first_json_ld(soup):
for tag in soup.select('script[type="application/ld+json"]'):
try:
value = json.loads(tag.string or tag.get_text())
except json.JSONDecodeError:
continue
items = value if isinstance(value, list) else [value]
for item in items:
if isinstance(item, dict) and (item.get("@type") in ("Product", "Residence", "House")):
return item
return {}
session = requests.Session()
response = session.get(URL, headers=HEADERS, timeout=30)
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")
ld = first_json_ld(soup)
address = ld.get("address") if isinstance(ld.get("address"), dict) else {}
record = {
"source_url": response.url,
"observed_at": datetime.now(timezone.utc).isoformat(),
"source_listing_id": soup.select_one("[data-listing-id]").get("data-listing-id")
if soup.select_one("[data-listing-id]") else None,
"address": {
"street": address.get("streetAddress"),
"locality": address.get("addressLocality"),
"region": address.get("addressRegion"),
"postal_code": address.get("postalCode"),
"country": address.get("addressCountry"),
},
"price": money(ld.get("offers", {}).get("price") if isinstance(ld.get("offers"), dict) else None),
"currency": (ld.get("offers", {}).get("priceCurrency")
if isinstance(ld.get("offers"), dict) else None),
"beds": ld.get("numberOfRooms"),
"baths": None, # Map this only when the source documents a bath field.
"area": None, # Preserve the source unit alongside any normalized value.
"property_type": ld.get("@type"),
"raw_html": response.text,
"parser_version": "2026-01",
}
with open("listing.json", "w", encoding="utf-8") as file:
json.dump(record, file, ensure_ascii=False, indent=2)
time.sleep(2.0) # Example delay; follow the source's stated limit instead.Real sites often expose different names or no JSON-LD at all. Add selectors for documented semantic elements, but retain the original value and unit next to every normalized value. Do not silently turn an unknown or locale-specific price into a number.
Model, normalize and deduplicate the data
Fields worth retaining
- Source listing ID and canonical URL.
- Original address components plus a normalized address used for matching.
- Original price text, normalized amount, currency and any “from”, auction or monthly qualifier.
- Beds, baths, floor area and its unit; do not assume square feet or square metres.
- Property type, availability status, broker or agent fields only when licensed.
- Permitted image URLs or hashes, not copied image files by default.
first_seen,last_seen, observation timestamps and parser version.
Deduplication and history
Use the source listing ID when available. Without one, combine the canonical URL and a cautiously normalized address, but do not treat that combination as permanent identity: a property can be relisted, subdivided or merged. Store snapshots or field-level history so a price, status or availability change can be explained. Keep the raw response only for the period allowed by the source agreement, then delete it on schedule.
Validate records and detect parser breakage
- Reject or quarantine records missing required fields rather than filling them with zero.
- Check numeric ranges, currency codes, units and geocoding confidence.
- Flag impossible transitions, such as a listing moving directly from an unknown state to sold without an observation, for review.
- Compare a sample of normalized records with the live source page on each deployment.
- Monitor HTTP errors, empty-result rates, field-null rates and parser exceptions.
- Alert when a schema or layout change causes a sudden drop in listing IDs or prices.
Validation is also a permission control: if the source changes its terms, requires authentication or begins serving a bot challenge, pause the collector instead of trying to work around it.
Free tools Windows power users keep installed
One-click scans. No signup required.
Publish only what your agreement permits
Before displaying a record, map each field to the permission that covers it. Preserve required source attribution, broker or listing-agent language and correction or takedown procedures. Do not republish photos, descriptions, logos, contact details or video merely because they appeared in a browser. If your product lets users search or export listings, confirm that redistribution, caching duration and commercial use are explicitly allowed.
Performance, reliability and cost decisions
Polling every page at the same interval is wasteful and can violate limits. Prefer an official incremental feed or webhook when available. Otherwise, prioritize recently changed or high-value pages, use conditional requests, cache immutable assets only when permitted, and spread work over time. Bound retries and use jittered exponential backoff so a failing source does not create a request storm.
Budget for more than HTTP traffic: licensed feeds may charge by record or call, while an HTML collector consumes engineering time for parser changes, monitoring, legal review and incident response. Track request counts, successful records, rejected records and source errors separately so a low bill does not hide poor coverage.
Troubleshooting common failures
403, 429 or a CAPTCHA appears
Cause: the source denied automation, your rate exceeded a limit, or the request lacks required authorization. Fix: stop the job, review the agreement and API path, lower concurrency only when permitted, and contact the source. Do not rotate identities or attempt to defeat the control.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Clear out junk files and repair common Windows errors3Scan for outdated or missing drivers - takes under a minuteEvery record is empty after a redesign
Cause: the page moved data into a new schema, client-side request or renamed attributes. Fix: inspect the documented response or authorized page, add a versioned parser, replay saved raw responses, and quarantine new records until validation passes.
Prices or addresses are wrong
Cause: locale formatting, currency ambiguity, ranges, hidden text or an incorrect selector. Fix: preserve the source string, parse with an explicit locale and currency, retain units, and require a review when normalization confidence is low.
Duplicates appear after relisting
Cause: URL or address matching was treated as a permanent identifier. Fix: prefer the source listing ID, keep a history table, and allow one property to have multiple listing episodes.
The collector times out
Cause: slow rendering, an overloaded source or a network failure. Fix: use the source’s documented API if possible, set bounded timeouts, retry only transient failures, and record the failed observation without deleting the previous valid state.
Or skip the browser setup
If your immediate need is a clean visual capture of an authorized listing page—for QA, an audit trail or a human review—ScreenshotNeo provides a screenshot API rather than a listing-data feed. It accepts a URL and can capture PNG, JPEG, WebP or PDF. Cookie and consent banners are accepted before capture, and more than 60 known consent platforms, newsletter popups and chat widgets are removed. Bot checks or CAPTCHAs, blank pages, timeouts, failed loads and cache hits are not billed; the response identifies the result with X-Page-Verdict and X-Billed headers.
Use the documented options at https://screenshotneo.com/docs/ for full-page captures with lazy images, a CSS-selected element, device or custom viewport, dark mode, retina scale, custom CSS or JavaScript, clicks, selector waits, network-idle waits, blocked resources, headers, cookies, authorization, timezone, geolocation, transparent backgrounds, resizing, chosen cache TTL, signed image links, asynchronous jobs, webhooks and bulk capture of up to 100 URLs per call. An MCP server exposes take_screenshot, get_page_info and capture_pdf to Claude, Cursor and other MCP clients.
One-call examples
cURL
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://screenshotneo.com -o shot.webpPython
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://screenshotneo.com"}, timeout=90)
r.raise_for_status()
open("shot.webp", "wb").write(r.content)Node.js
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://screenshotneo.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);
if (!res.ok) throw new Error(`${res.status} ${res.statusText}`);
const fs = await import('node:fs/promises');
await fs.writeFile('shot.webp', Buffer.from(await res.arrayBuffer()));ScreenshotNeo is not a substitute for a licensed listing API: it captures what an authorized page renders. It is useful when you need a clean visual record without writing browser-consent and popup handling yourself. The Free plan includes 1,000 screenshots per month with no card; paid plans start at $5 for 3,000, and every feature is included on every plan. Create a free ScreenshotNeo account.
FAQ
Can a source revoke permission after data has been collected?
Yes. Agreements can change or be terminated. Design a disable switch, record the governing version and stop new collection when the source withdraws access; follow the contract’s deletion and correction terms for stored records.
How should sale and rental listings be separated?
Use distinct scopes and status vocabularies unless the source explicitly defines a shared schema. A monthly rent, sale price and “for lease” status should never be normalized into the same field without a type marker.
What should an audit record prove?
It should connect each normalized value to its source URL, observation time, raw response or permitted evidence, parser version and authorization context. That makes later corrections explainable without retaining material longer than the agreement allows.
Frequently Asked Questions
Can a source revoke permission after data has been collected?
Yes. Agreements can change or be terminated. Design a disable switch, record the governing version and stop new collection when the source withdraws access; follow the contract’s deletion and correction terms for stored records.
How should sale and rental listings be separated?
Use distinct scopes and status vocabularies unless the source explicitly defines a shared schema. A monthly rent, sale price and “for lease” status should never be normalized into the same field without a type marker.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →What should an audit record prove?
It should connect each normalized value to its source URL, observation time, raw response or permitted evidence, parser version and authorization context. That makes later corrections explainable without retaining material longer than the agreement allows.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
