Robots.txt and a website’s Terms of Service are not the same kind of restriction. Robots.txt communicates crawler instructions; the Internet Engineering Task Force’s RFC 9309 says those rules are not access authorization. Terms of Service may create contractual obligations if an applicable agreement was formed, but whether it can be enforced depends on the facts and governing law. Neither point, by itself, settles whether a particular scraping operation is lawful.
What is the difference between robots.txt and Terms of Service?
| Restriction | What it does | What it does not establish by itself |
|---|---|---|
| Robots.txt | Communicates crawler instructions under the Robots Exclusion Protocol, standardized in IETF RFC 9309, published in September 2022. | It is not a password, technical access-control gateway, or agreement with every crawler operator. RFC 9309 expressly says, “These rules are not a form of access authorization.” |
| Terms of Service | Sets conditions a website says apply to its users or visitors. If an applicable agreement was formed, its restrictions may be contractual obligations. | The terms’ existence alone does not establish that a particular person assented to them, that they apply to the conduct, or that a court will enforce them. |
RFC 9309 is an Internet Standards Track protocol specification, not legislation. Its crawler instructions are requested to be honored, but the protocol itself does not turn a disallowed path into legally restricted access. Contract questions about Terms of Service are separate and depend on applicable law and circumstances such as notice, assent, and the agreement’s scope.
Is robots.txt legally binding?
Not as access authorization under RFC 9309. The standard distinguishes crawler guidance from permission to enter or use a site. A robots.txt directive alone should not be treated as a contract, login requirement, or technical barrier.
That does not mean robots.txt can never matter in a broader dispute. Its presence may be part of the circumstances considered alongside other evidence, but the protocol instruction alone does not answer whether a contract or another law was violated. The Ninth Circuit’s 2019 hiQ opinion described robots.txt compliance as voluntary in the context of that case; it did not resolve every possible claim or jurisdiction.
#1 Best Overall
- Used Book in Good Condition
Can a website enforce its Terms of Service against scraping?
Potentially, if the terms formed an applicable agreement and the restriction covers the conduct at issue. Whether there was assent, adequate notice, and an enforceable provision depends on the jurisdiction and facts; the mere presence of terms on a website does not settle those questions.
Contract is only one possible legal issue. Depending on the activity, other claims or laws involving computer access, state-law torts, copyright, privacy, or protected content may also matter. A conclusion about one claim—for example, a federal Computer Fraud and Abuse Act (CFAA) theory—is not a ruling on all of them.
Rank #2
- Compliant Inspection Records: Meets federal requirements for driver vehicle inspection report books, ensuring your fleet stays audit-ready.
- Complete Checklist: Covers tractor, trailer, and essential parts for CDL pre trip inspection and daily truck inspection forms.
- Quick Reference: Includes required inspection steps inside for quick driver reference during pre-trip and post-trip inspections.
- Durable, Convenient Size: 2-ply carbonless vehicle inspection form (white/yellow copies) resist wear in tough trucking environments. Compact 5.5" x 8.5" size fits easily in cabs and clipboards.
- Perfect for Commercial Fleets: Whether you manage a single vehicle or a large commercial fleet, our pretrip inspection book is an essential tool for ensuring the safety and compliance of your operations.
Does ignoring robots.txt make web scraping illegal?
Ignoring a robots.txt instruction does not, by itself, establish that scraping is illegal. It means the crawler has not followed the site’s communicated crawler rules. The legal analysis can change when other facts are present, such as applicable contractual terms, authenticated or permissioned access, technical restrictions, the use of copied material, privacy interests, or demonstrable harm.
Publicly viewable pages and restricted areas should not be treated as interchangeable. Whether a person can access a page without logging in is relevant to some legal questions, but it does not automatically answer every contract, copyright, privacy, or state-law question.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Rank #3
- Large area for complete description of work proposed
- Includes space for customer to sign his/her acceptance of proposal.
- 1-part form includes carbons to create 2 part forms if necessary.
- Space at top for company stamp.
Can a cease-and-desist letter make access to a public website unauthorized?
A letter is not automatically a technical access control or a universal legal switch. It can, however, put a recipient on notice that the site objects to the activity, and it may be relevant to the facts, including whether permission was revoked. Its legal effect depends on the claim, the access circumstances, and applicable law.
The distinction was important in hiQ Labs, Inc. v. LinkedIn Corp. LinkedIn objected to hiQ’s collection of publicly viewable profile information, including through a letter and its terms. The Ninth Circuit’s 2022 opinion considered whether access to generally available information after a targeted refusal raised serious questions under the CFAA’s “without authorization” language. That analysis did not decide whether the parties had a contract or resolve every other possible claim.
What did the Ninth Circuit decide in hiQ?
The 2019 appeal
In its September 9, 2019 opinion, the Ninth Circuit reviewed a preliminary injunction involving hiQ’s scraping of publicly viewable LinkedIn profiles after LinkedIn demanded that it stop. The court considered the likelihood of success and balance of hardships at that preliminary stage; it did not issue a final ruling that all public scraping is lawful. It also observed that claims such as trespass to chattels might be available in some circumstances, including where scraping exceeded consent and caused demonstrable harm, but it did not decide that claim.
The 2022 opinion on remand
After the U.S. Supreme Court’s 2021 decision in Van Buren, the Ninth Circuit reconsidered hiQ. In its April 18, 2022 opinion, it again affirmed preliminary relief, finding that hiQ had raised serious questions about whether the CFAA covered access to generally accessible information after a targeted refusal. The panel explained that Van Buren reinforced its reasoning, while not directly addressing the CFAA’s “without authorization” clause because it concerned “exceeds authorized access.”
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Best Value
- Durable limited power of attorney legal forms kit. Do-it-yourself legal forms.
- The package contains 2 blank liberty legal forms. 1 Full-size worksheet with step-by-step instructions handy quick reference guide.
- A limited power of attorney allows you to extend specific powers granted to your agent in case you are unable to act on your behalf, which limits your named representative's specific powers over what they can manage on your behalf as opposed to a blank power of attorney over all matters.
- It is effective immediately and continues in the event you become incapacitated. It is easy to understand and it Is written in plain language
- The liberty legal forms series is intended to assist persons who wish to create specific legal documents. The laws of a particular, state, province, or territory may vary as to what legal forms are permitted in that jurisdiction
These were preliminary-injunction rulings about particular public profile data and the CFAA, not a final merits judgment establishing a nationwide right to scrape. They do not resolve whether Terms of Service were contractually binding in another situation, nor do they settle claims under state law or other legal regimes.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Which facts matter most in a scraping dispute?
- Where the content sits: pages available to anyone differ from authenticated, permissioned, or technically restricted areas.
- Whether terms apply: consider notice, assent, governing law, and whether the relevant provision covers the activity.
- What happened after an objection: a cease-and-desist, account restriction, authentication gate, IP block, or other measure may change the factual and legal analysis.
- How access occurred: bypassing technical controls is different from requesting generally available pages, though the full legal result depends on the claim and jurisdiction.
- What the scraper did with the material: the content, use, scale, privacy interests, server burden, and evidence of harm can matter to claims beyond the CFAA.
- Where the dispute is heard: the Ninth Circuit’s hiQ decisions do not resolve every U.S. jurisdiction or non-U.S. law.
For a real operation, assess these facts under the law that governs it rather than treating either robots.txt or a court decision about a different site as a complete answer. The key sources are IETF RFC 9309, published September 2022, and the Ninth Circuit’s hiQ opinions filed September 9, 2019, and April 18, 2022.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
