Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more

A web monitor is a scheduled pipeline: fetch a page, extract the part that matters, compare it with the last successful observation, save the result, and notify someone only when a meaningful change occurs. Start with ordinary HTTP requests and conditional GET; use a browser worker only when the content you need is rendered by JavaScript. Add robots.txt and site-policy checks, bounded retries, durable state, and alert deduplication before running checks continuously.

What a web monitor service needs to do

A monitor does more than fetch a URL on a timer. It must distinguish a real content change from a failed request, a transient layout variation, or a page that has not finished rendering. Treat each check as an observation with a known outcome, then make comparison and notification separate steps.

  1. Schedule: decide which monitors are due and enqueue checks durably.
  2. Fetch: request the page over HTTP or render it in a browser when necessary.
  3. Extract and normalize: isolate the requested text or field and discard irrelevant variation.
  4. Compare: detect a changed value or structured difference against the last successful observation.
  5. Persist: save the observation, validators, status, and event identity.
  6. Notify: deliver a deduplicated alert through an outbox or equivalent retryable mechanism.

Keep the monitor definition separate from check history. A useful definition includes URL, cadence and timezone, extraction rule, request headers, a reference to any credentials, and alert policy. History should record the last successful check, HTTP status, ETag and Last-Modified validators, extracted value or its hash, errors, and emitted event IDs.

Free tools Windows power users keep installed

One-click scans. No signup required.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Choose HTTP polling or browser rendering

Use an HTTP client for static HTML and data endpoints

A normal HTTP client is simpler and lighter than a browser for a static page or JSON endpoint. If a site exposes the information as JSON, prefer that stable field over parsing presentation markup, provided the endpoint is intended for your use and you respect its access rules.

#1 Best Overall
Tecmojo 12U Open Frame Network Rack for IT & AV Gear, AV Rack Floor Standing or Wall Mounted,with 2 PCS 1U Rack Shelves & Mounting Hardware,Network Rack for 19" Networking,Audio and Video Device
  • 【Powerful Load-bearing】12U Network Rack Open Frame is constructed from durable cold rolled steel; Rack shelf supports enhance stability, wall-mounted capacity of 130lbs, the ground-mounted up to 260lbs
  • 【Considerate Designs】Open-frame layout, including a top panel adding space, anti-slip shelf stops fixing devices and compatible racks for stack and expansion to meet requirements of home server rack
  • 【Complete Accessories】A 12U open frame server rack, two ventilated shelves, four shelf stops, four velcro straps and a set of equipment mounting screws
  • 【Versatile Application】Ideal for space-efficient multi-device setups in warehouses, retail, classrooms, offices and more; Excellent choices as AV Rack/IT Rack
  • 【Effortless Setup】 Network Rack includes hardware, a comprehensive manual, mounting hole drilling template and an online assembly video to simplify setup

Save response validators from successful responses. On the next check, send them as If-None-Match and If-Modified-Since. A 304 Not Modified means the representation has not changed under that validator; do not parse a body or create a change alert. RFC 9110 describes conditional GET as the most efficient mechanism for HTTP cache updates, and MDN documents ETag and Last-Modified as validators usable for conditional requests.

Escalate to a browser only when the content requires it

For a client-rendered page, an initial HTTP response may not contain the text visible to a visitor. Use a browser context such as Playwright to wait for the relevant selector, then extract that element rather than hashing the entire DOM. Playwright can observe page requests and responses and inspect WebSockets; its APIRequestContext can also make HTTP calls with headers and isolated cookies. If request interception seems to miss traffic, service workers may be handling requests; Playwright documents disabling them when predictable interception events are needed.

Before introducing a browser, check whether the rendered page calls a JSON endpoint that can supply the same data directly. Browser rendering costs more resources and has additional failure modes: navigation timeouts, consent dialogs, authentication state, animations, and content that changes after initial load. Keep browser work in a separate worker pool so one slow page cannot consume all your lightweight HTTP capacity.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Build the first monitor in Python

This small example shows a single HTTP check with SQLite persistence, conditional requests, a CSS selector, normalized text, and a basic change event printed to standard output. Install requests and beautifulsoup4 with python -m pip install requests beautifulsoup4. Before running it against a host, fetch and parse that host’s /robots.txt, check site terms and published limits, and confirm your intended path and user-agent are allowed. The example deliberately leaves notification-provider integration and production scheduling to the service layer.

import hashlib
import sqlite3
import time
from urllib.parse import urlparse

import requests
from bs4 import BeautifulSoup

URL = "https://example.com/pricing"
SELECTOR = ".price"
MONITOR_ID = "pricing-example"
DB = "monitor.db"
USER_AGENT = "ExampleWebMonitor/1.0 (contact: ops@example.com)"


def init_db():
    with sqlite3.connect(DB) as db:
        db.execute("""CREATE TABLE IF NOT EXISTS monitors (
            id TEXT PRIMARY KEY, url TEXT NOT NULL, etag TEXT,
            last_modified TEXT, content_hash TEXT, value TEXT,
            checked_at TEXT, last_status INTEGER
        )""")
        db.execute("""CREATE TABLE IF NOT EXISTS events (
            event_key TEXT PRIMARY KEY, monitor_id TEXT NOT NULL,
            old_value TEXT, new_value TEXT, created_at TEXT DEFAULT CURRENT_TIMESTAMP
        )""")
        db.execute("INSERT OR IGNORE INTO monitors (id, url) VALUES (?, ?)",
                   (MONITOR_ID, URL))


def check_once():
    init_db()
    with sqlite3.connect(DB) as db:
        row = db.execute(
            "SELECT etag, last_modified, content_hash, value FROM monitors WHERE id=?",
            (MONITOR_ID,)
        ).fetchone()
    etag, last_modified, old_hash, old_value = row
    headers = {"User-Agent": USER_AGENT, "Accept": "text/html,application/xhtml+xml"}
    if etag:
        headers["If-None-Match"] = etag
    if last_modified:
        headers["If-Modified-Since"] = last_modified

    try:
        response = requests.get(URL, headers=headers, timeout=(5, 20),
                                allow_redirects=True)
        if response.status_code == 304:
            with sqlite3.connect(DB) as db:
                db.execute("UPDATE monitors SET checked_at=CURRENT_TIMESTAMP,
                            last_status=304 WHERE id=?", (MONITOR_ID,))
            print("Unchanged (304)")
            return
        response.raise_for_status()
    except requests.RequestException as exc:
        # Production code should persist this failure and schedule a capped retry.
        print(f"Fetch failed: {exc}")
        return

    soup = BeautifulSoup(response.text, "html.parser")
    node = soup.select_one(SELECTOR)
    if node is None:
        print(f"Extraction failed: selector {SELECTOR!r} was not found")
        return
    value = " ".join(node.get_text(" ", strip=True).split())
    digest = hashlib.sha256(value.encode("utf-8")).hexdigest()

    with sqlite3.connect(DB) as db:
        if old_hash and digest != old_hash:
            event_key = hashlib.sha256(
                f"{MONITOR_ID}:{old_hash}:{digest}".encode("utf-8")
            ).hexdigest()
            inserted = db.execute(
                "INSERT OR IGNORE INTO events (event_key, monitor_id, old_value, new_value) "
                "VALUES (?, ?, ?, ?)",
                (event_key, MONITOR_ID, old_value, value)
            ).rowcount
            if inserted:
                print(f"CHANGE for {URL}nOld: {old_value}nNew: {value}")
        db.execute("""UPDATE monitors SET etag=?, last_modified=?, content_hash=?,
                    value=?, checked_at=CURRENT_TIMESTAMP, last_status=? WHERE id=?""",
                   (response.headers.get("ETag"),
                    response.headers.get("Last-Modified"), digest, value,
                    response.status_code, MONITOR_ID))


if __name__ == "__main__":
    check_once()

The script assumes one monitor and one process. For a service, load each monitor’s URL and selector from configuration storage, validate allowed schemes and hosts, and ensure that a redirect does not silently move a check into a disallowed scope. Remove unused imports such as time and urlparse if keeping the example minimal; they are useful when extending it with scheduling and URL policy checks.

Rank #2
Sale
StarTech 42U 4-Post Open Frame Rack, 19in, 22-40in, 1323lb/600kg
  • ADJUSTABLE DEPTH: 4-Post 42U open frame server rack with 4 vertical rails and adjustable mounting depth 22" to 40" (56,0cm to 101,7cm); Compatible with various servers / switches / data / AV and other IT equipment; EIA/ECA-310-E Compliant
  • EASY ASSEMBLY: Mobile network rack with easy-to-follow assembly instructions and online video; Compact flat-pack shipping to avoid damage and facilitate installation; Total product height of 80.3in (204 cm) with casters, 78in (198cm) without casters
  • COLD ROLLED STEEL: Durable 4 Post 19in open frame rack designed for ventilation with 42U mounting height and 1320lb (600kg) weight capacity (stationary); 3 install options included: casters, levelling feet, or base-plate to secure rack to the floor
  • HARDWARE INCLUDED: Rolling computer/data rack includes cage nuts and screws to mount equipment, easy to read Units (U) and depth adjustment markings, cable management hooks for organization, and required assembly tools
  • THE IT PRO'S CHOICE: Designed and built for IT Professionals, this 42U rack is backed for 2-years, including free lifetime 24/5 multi-lingual technical assistance

A first successful observation becomes the baseline and does not trigger an alert. A missing selector is an extraction failure, not evidence that the monitored content disappeared; persist the error and alert on repeated extraction failures separately. Store raw content only if there is a reason to audit it, and restrict retention because pages may contain personal or sensitive information.

Normalize carefully and avoid noisy alerts

Whole-page comparisons are fragile. Pages commonly include clocks, rotating recommendations, randomized IDs, ad slots, and whitespace changes that are unrelated to the thing the reader cares about. Prefer a stable selector or structured field. Normalize whitespace and encoding consistently, and remove only known irrelevant fields: aggressive cleanup can hide a real change.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
  • Use a hash of normalized text for a cheap unchanged/changed decision.
  • Keep the extracted value as well as the hash when alerts or debugging need to show what changed.
  • Use a structured diff when the monitored value has meaningful fields, such as a product’s price and availability.
  • For noisy pages, require the same candidate change to appear in two consecutive successful checks before notifying.
  • Persist a unique event key based on monitor and observation fingerprint so retries cannot send the same alert repeatedly.

When displaying extracted HTML in a dashboard or message, sanitize it. Treat page content as untrusted input; do not insert raw HTML into an administrative interface.

Respect robots.txt, site rules, and request budgets

Fetch /robots.txt for each host, parse the group that applies to your monitor’s user-agent, and apply the most specific matching Allow or Disallow rule. RFC 9309 says these rules are requested for crawlers to honor and also makes clear that they are not access authorization. A robots.txt allowance does not grant permission to bypass authentication, terms, rate limits, or other access controls.

RFC 9309 says crawlers should not use a cached robots.txt copy for more than 24 hours unless the file is unreachable; honor its response caching information and record when the policy was last checked. Use a descriptive user-agent with a contact route, per-host concurrency limits, conservative polling intervals, and exponential backoff for transient failures. Do not try to work around CAPTCHAs or access restrictions.

Rank #3
Sale
TECMOJO 12U Open Frame Network Rack for IT & AV Gear, 4-Post With Casters, Mobile With 2 PCS 1U Server Shelf & Mounting Hardware, for 19" Network, Audio and Video Device
  • 【Powerful load-bearing】12U Network Rack Open Frame is constructed from durable Cold Rolled Steel; Rack Shelf Back Support enhances stability; load-bearing capacity of 260lbs
  • 【Sliding&Considerate】Open-frame layout, including four wheels easy to move, a top panel adding space, anti-slip shelf stops fixing devices and compatible racks for stack and expansion to meet requirements of home server rack
  • 【Complete Accessories】A 12U open frame server rack, two ventilated shelves, four shelf stops, four casters, four velcro straps and a set of equipment mounting screws
  • 【Versatile Application】Ideal for space-efficient multi-device setups in warehouses, retail, classrooms, offices and more; Excellent choices as AV Rack/IT Rack
  • 【Effortless Setup】Server rack with wheels includes hardware, a comprehensive manual, mounting hole drilling template and an online assembly video to simplify setup

Make the service reliable under failure

Bound every kind of wait

Set connect and read timeouts for HTTP requests, navigation and selector timeouts for browser work, and a total deadline for each job. Without a total deadline, a page can hold a worker indefinitely even if individual network waits are bounded. Record a timeout as a failed observation and retain the last successful value; never replace good state with an empty value simply because a fetch failed.

What’s actually slowing this PC down?

Pick the symptom - the matching free tool is one click away.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Retry selectively with backoff

Retry 429 responses and transient 5xx failures with capped exponential backoff and jitter, respecting any server-provided retry guidance. Avoid blind retries for authentication failures and permanent 4xx responses. Set a maximum attempt count and move repeatedly failing work to a visible failure state instead of retrying forever. A check failure should be observable separately from a content-change alert.

Use durable scheduling and an outbox

Enqueue due checks into a durable queue rather than relying on an in-memory timer. Persist an observation before dispatching its alert. Write notification jobs to an outbox in the same transaction as the detected event; a separate sender can retry provider timeouts without losing the event or repeating the comparison. Use idempotency keys when the notification destination supports them.

Keep credentials and browser sessions isolated

  • Store credentials in a secret manager or encrypted configuration, not in monitor URLs or logs.
  • Use separate browser contexts and cookies for each monitor or tenant.
  • Redact Authorization headers, cookies, and query parameters that contain secrets from diagnostics.
  • Validate user-supplied URLs to prevent checks from reaching internal network addresses or cloud metadata endpoints.
  • Restrict stored page data and define a retention period appropriate to its sensitivity.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

Scale polling without hammering websites

Do not schedule every monitor at the same instant. Spread checks across the cadence window, limit simultaneous requests per host, and keep global limits for HTTP and browser workers. Conditional GET reduces unnecessary body transfer but does not eliminate the request itself, so cadence and concurrency still matter. Cache robots.txt within its applicable period, and do not make repeated checks faster just because the latest response was unchanged.

Measure your own service rather than assuming a universal performance figure. Track check latency, response-status distribution, bandwidth and body sizes, extraction success, queue age, browser timeouts, notification delivery delay, and cost per completed observation. Playwright’s webServer configuration is one documented example of waiting for a URL, reusing an existing server, and setting startup timeouts; the same discipline helps in integration tests for your worker.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
Rank #4
VEVOR 12U Open Frame Server Rack, 23-40 in Adjustable Depth, Free Standing or Wall Mount Network Server Rack, 4 Post AV Rack with Casters, Holds All Your Networking IT Equipment AV Gear Router Modem
  • Adjustable Depth: 23-40'' adjustable depth is used for servers and network equipment, ensuring enough space for AV equipment, components, and cabling, while allowing you to access ports and equipment from multiple sides.
  • Strong Load Capacity: Ground-Mounted Load Capacity: 500 lbs, Wall-Mounted Load Capacity: 150 lbs. The av rack is made of carbon steel for better weldability performance and can help save space while meeting your need to place multiple devices.
  • User-friendly Design: Ergonomic design makes the open frame av rack easier to use. The additional top panel is able to place other items with more available space. Roller design moves anywhere and anytime, is convenient, and is more energy-saving.
  • Complete Accessories: We provide the accessories you need, including 2 x Pallets, 145 x M5*10 Cross Head Screws, 4 x Casters, 4 x M10*50 Expansion Screws,10 x M6*12 Cage Nuts, 1 x Grounding Wire, 1 x User Manual.
  • Wide Application: The server rack wall mount maximizes the use of available space, suitable for retail venues, classrooms, offices, and other places where space is limited.

Or skip the browser setup

If your monitor needs a clean rendered capture as part of inspection, ScreenshotNeo is a website screenshot API and MCP server for developers. Its browser capture can return PNG, JPEG, WebP, or PDF; it is an option to use alongside the monitor pipeline, not a replacement for scheduling, comparison, and notification.

Example cURL request for the page you want to inspect:

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://example.com/pricing -o shot.webp

See the ScreenshotNeo API documentation for request options and response headers. ScreenshotNeo removes supported cookie/consent banners, newsletter popups, and chat widgets before capture; those cleanup steps can be turned off. Bot checks/CAPTCHAs, blank pages, timeouts, failed loads, and cache hits are not billed, and responses identify page verdict and billing status in headers. Its MCP server provides take_screenshot, get_page_info, and capture_pdf for Claude, Cursor, and other MCP clients. The free plan includes 1,000 shots per month without a card; paid plans start at $5 for 3,000 shots.

Sign up for ScreenshotNeo to start with 1,000 screenshots a month free, with no card required.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Troubleshooting common monitor failures

Symptom Likely cause What to do
Every check reports a change The selector includes dynamic text, timestamps, ads, or rotating content. Narrow the selector, normalize known noise, and inspect successive extracted values before changing the alert rule.
HTTP check sees an empty or old value The page fills the content with JavaScript after the initial response. Use a browser worker that waits for the target selector, or identify an appropriate underlying data endpoint.
Browser interception misses requests A service worker may be handling them. Review Playwright’s service-worker guidance and disable service workers when predictable interception events are necessary.
Checks stop after a 304 The client may be treating 304 as an error or attempting to parse a nonexistent body. Handle 304 before generic error handling; update check time and status while retaining the previous extracted value.
Duplicate notifications arrive A retry or concurrent worker is processing the same detected change. Use a unique event key and transactional insert, then deliver from a retryable outbox with idempotency where available.
Worker backlog keeps growing Cadence exceeds capacity, jobs are synchronized, or slow browser jobs occupy all workers. Spread due times, set per-host and global concurrency bounds, isolate browser capacity, and inspect queue age and job latency.
A page suddenly becomes inaccessible The site may have changed policy, rate limits, authentication, or structure. Stop aggressive retries, check the site’s rules and response status, update credentials or extraction only when permitted, and surface the failure separately from content changes.

Frequently Asked Questions

Should each monitor run in a separate process?

Not necessarily. A durable queue and isolated per-job state usually provide a clearer starting point; split worker pools when browser jobs, host-specific limits, or security boundaries require it.

How can I test alert logic without repeatedly contacting a live site?

Save representative HTML or JSON fixtures and test extraction, normalization, comparison, and deduplication against those fixtures. Keep a smaller integration check for the fetch layer and run it within the site’s allowed limits.

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.