Short answer: Don’t build a government-contract tracker by scraping SAM.gov’s pages. SAM.gov’s Terms of Use prohibit automated data gathering and web-scraping tools. Use authorized APIs, public downloads, or extracts instead: SAM.gov for current opportunities and contract-award data, and USAspending.gov for award and spending history. Schedule retrievals, retain source timestamps, and detect changes so your alerts remain useful and auditable.
Why a compliant tracker should not scrape SAM.gov
The title asks about web scraping, but scraping SAM.gov’s HTML pages is not the appropriate way to automate federal procurement monitoring. SAM.gov’s Terms of Use say: “Automated data gathering, web scraping tools are prohibited and, if detected, will result in the associated account(s) being denied access to SAM.gov via Login.gov.” In practical terms, do not run a bot against the site’s pages or mine data through a logged-in account. Use authorized machine-readable access instead, and check the terms and access requirements that apply to each interface.
The official sources serve different purposes. SAM.gov’s Contract Opportunities is for current procurement notices. Its contracting information also points users to public API and data-file access. SAM.gov’s Contract Award Data supports searching awards; GSA documents a Contract Awards API that returns JSON records. USAspending.gov provides public access to award and spending history, including recipients and agency or geographic breakdowns. Use each source for the question it answers rather than treating opportunity notices and completed awards as one record type.
Choose the right source for each tracking question
| Question | Source to start with | What to track |
|---|---|---|
| What federal opportunities are currently posted? | SAM.gov Contract Opportunities | Notice identifier, title, contracting agency, available set-aside or eligibility details, posted date, response deadline, and source URL. |
| What awards have been made, and to whom? | SAM.gov Contract Award Data and its documented API | Awardee, agency, award type, dates, obligations, and related identifiers where present. |
| How does federal spending break down by recipient, agency, or geography? | USAspending API | Recipient and spending-history information available through its machine-readable endpoints. |
For API-specific routes and request formats, use the official USAspending endpoint documentation and GSA Contract Awards API documentation. API details can change; follow the current documentation rather than relying on a copied endpoint or parameter list. GSA documents a Contract Awards API response in JSON. The documented page behavior for that API is 10 records by default, with a limit parameter allowing up to 100 records per page. That limit is an API behavior, not a promise about completeness or update speed.
Build a tracker in six layers
1. Discover notices through an authorized interface
Query SAM.gov Contract Opportunities using the keywords and filters relevant to your work, through an authorized API or public data access method. Capture the notice identifier, title, agency, any eligibility or set-aside information provided, posted date, response deadline, and original notice URL. Preserve the exact source value; don’t infer an eligibility category from a title or keyword.
2. Keep opportunities and awards separate
An opportunity is a notice about potential work; an award is a record of a contract or other award. They answer different questions and need not map one-to-one. Query SAM.gov Contract Awards and USAspending when you need awardee, agency, award type, obligations, dates, or related identifiers. Store these as award records and link them to opportunities only when a reliable identifier or other evidence supports the relationship.
3. Refresh on a schedule suited to the deadline
Choose retrieval frequency according to how quickly your team must react, the interface’s permitted access pattern, and the response deadlines you monitor. Record the retrieval time for every response. More frequent polling can reduce the interval before your system sees a change, but it cannot guarantee that the source has published an update or that a later award record will already be available.
Rank #2
4. Normalize without losing the original
Map agency names, NAICS or PSC codes, identifiers, dates, and dollar fields into stable internal fields so searches and alerts work consistently. Preserve the original field value, source, and record alongside the normalized value. This makes it possible to audit a transformation or correct a mapping without losing the source record.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →5. Detect changes and deduplicate
Use a stable source identifier to deduplicate repeated retrievals. Store a hash or version marker for the content you monitor and compare it with the previous version. A change should create a history entry rather than overwrite the old record: retain what changed, when your system retrieved it, and the source link. This is particularly important for response deadlines and amendments.
6. Alert on events, not just new rows
Useful alerts include newly posted notices, deadline changes, amendments, sources-sought notices, and awards relevant to tracked agencies, vendors, or NAICS/PSC codes. Set separate rules for opportunities and awards; otherwise a search for a new solicitation can be confused with an award-history update.
Decide what your records and alerts must contain
A useful tracker needs more than a title and a link. At minimum, plan for the following fields and system metadata:
- Identity: the source’s notice or award identifier and the source name.
- Meaning: record type (opportunity or award), title or description, agency, and relevant vendor or awardee details when available.
- Dates: posted date, response deadline for opportunities, award dates where available, and your own retrieval timestamp. Keep source dates distinct from retrieval time.
- Classification and value: set-aside or eligibility fields when present, NAICS/PSC codes, award type, and dollar or obligation fields where supplied.
- Provenance: source URL, endpoint or download used, response status, and a retained raw response or source record.
- Change history: previous and current values for tracked fields, plus the time your system detected the difference.
Do not treat a missing field as proof that the underlying contract lacks that property. Preserve it as missing or unavailable from that record, and avoid filling gaps with guesses. Likewise, keep an award-to-opportunity relationship unconfirmed when the available identifiers do not establish a match.
Free tools Windows power users keep installed
One-click scans. No signup required.
Use APIs and downloads carefully
Public API access and public data files can reduce the need to parse page markup, but access is not automatically unrestricted. Some interfaces require an API key or impose rules on how data may be accessed or used. Follow each service’s current instructions for credentials, rate limits, and restricted information. Do not use SAM.gov login credentials as a way to authorize automated mining. Use public versions of APIs for public displays, and do not collect sensitive or For Official Use Only (FOUO) information unless you are authorized to handle it.
When using USAspending’s documented Contract Awards API, account for pagination: the docs describe 10 records per page by default and a maximum page size of 100 through the limit parameter. A successful first page is not a complete result set. Follow the documented pagination behavior, retain the request and retrieval details, and verify that your collector reaches all pages needed for the query. For SAM.gov or other interfaces, use their current documentation for the exact request format rather than guessing route names or parameters.
Measure coverage and freshness honestly
There is no authoritative universal accuracy percentage for a contract tracker built by scraping—or for a tracker assembled from multiple sources. A single score can conceal whether a source covers opportunities, awards, particular agencies, or only a slice of spending data. Report what each source covers and how stale your last successful retrieval is instead.
Useful operational measures include the time of the latest successful retrieval per source, the time between a source record’s posted or changed date and your system’s detection (when both are available), the proportion of expected retrievals that completed, and the number of records with unresolved identifiers or missing fields. These measures describe your pipeline; they do not prove that the source itself is complete or that every change was published immediately.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallCrashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteBest Value
Troubleshoot common tracking failures
- No results for a search: Recheck the exact filters and keyword choices in the official interface. Separate an empty result from an access or request error, and retain the response status and query details.
- Repeated duplicate notices: Deduplicate on the stable identifier supplied by the source, not title alone. Titles can be similar or change; retain the source record and version history.
- A deadline appears to move backward or forward: Keep the previous value, new value, source timestamp, and retrieval time. Confirm the change against the current notice and its amendment history instead of silently replacing the deadline.
- Awards do not match an opportunity: Keep the records separate until a reliable identifier or documented relationship supports a link. Similar names or keywords are not enough to claim that a particular opportunity resulted in an award.
- Only part of a result set is stored: Check pagination against the API’s documentation. For the documented GSA Contract Awards API behavior, the default page size is 10 and the documented maximum via
limitis 100; a single page may not contain all matching records. - Access fails after adding automation: Stop automated page gathering and review the source’s terms and API access requirements. Do not try to work around a restriction with login credentials or page scraping.
- Your alert arrives after the response deadline: Review your permitted polling schedule, retrieval failures, and alert delivery time. A tracker cannot alert on a change it has not yet retrieved, so show the last successful refresh time with the alert or dashboard.
Or skip the browser setup
ScreenshotNeo is a website screenshot API and MCP server by Yorker Media; it can capture a page, but it is not a government-contract data API and does not replace the official procurement data sources above. For a visual snapshot of a public notice page, one GET request can return an image or PDF. See the ScreenshotNeo documentation for options and response details.
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://sam.gov/opportunities -o shot.webpScreenshotNeo accepts cookie or consent banners like a visitor and removes more than 60 known consent platforms, newsletter popups, and chat widgets before capture; each of those steps can be turned off. Bot checks/CAPTCHAs, blank pages, timeouts, failed loads, and cache hits cost nothing, and responses identify the page verdict and billing status. Its MCP server gives AI agents using Claude, Cursor, or another MCP client the tools take_screenshot, get_page_info, and capture_pdf. The Free plan includes 1,000 shots a month without a card; paid plans start at $5 for 3,000 shots. ScreenshotNeo is at screenshotneo.com. Sign up for 1,000 free screenshots a month with no card.
Can web scraping ever be the right choice here?
For SAM.gov, the published prohibition makes automated page scraping a poor choice for a compliant tracker. Prefer the official API, authorized extracts, or downloads. If a different procurement source offers no machine-readable option, check that source’s terms and obtain permission before automating access; a technique that is allowed on one site is not automatically allowed on another.
Frequently Asked Questions
How can I monitor a federal contract recompete?
Track the relevant prior award and agency in award data, then monitor new opportunity notices for the agency, scope, and identifiers connected to that work. Treat a possible recompete as a lead for review, not a confirmed relationship unless the source records establish it.
Can I use a commercial procurement-intelligence platform instead of building this pipeline?
Yes. Such platforms may combine source normalization, historical analysis, and alerts. Evaluate their stated source coverage, refresh timing, provenance, and whether they distinguish opportunity notices from awards.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
