Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchPC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Publishers are increasingly telling Internet Archive crawlers not to copy their sites, largely because they fear unauthorized AI training, lost licensing leverage, and weak attribution. Nieman Journalism Lab’s May 20, 2026 analysis found 382 news websites with at least one Internet Archive-associated crawler disallowed in robots.txt. That is a warning about access to the historical record, but it is not a census of publishers or proof that every directive successfully stopped a crawler.
How widespread are the restrictions?
Nieman Lab’s updated sample identified 382 news websites disallowing at least one Internet Archive-associated bot. Of those, 342 were local news sites, and 93% were based in the United States. The sample covered ten countries. Many local outlets belonged to large chains, so the headline “major publishers” captures some corporate ownership but not the makeup of the sites counted.
The January analysis found 241 sites. The May update added 141 sites, bringing the sample to 382. These figures describe the sites Nieman Lab examined, not every news outlet online and not necessarily the number of sites that an Internet Archive crawler could no longer reach.
Why the local-outlet majority matters
Local reporting is especially vulnerable to disappearing when a publication changes its content system, is sold, or closes. A restriction at a chain-owned local site can therefore affect both a large company’s policy and the only durable public copy of a community’s reporting.
#1 Best Overall
- Easily store and access 2TB to content on the go with the Seagate Portable Drive, a USB external hard drive
- Designed to work with Windows or Mac computers, this external hard drive makes backup a snap just drag and drop
- To get set up, connect the portable hard drive to a computer for automatic recognition no software required
- This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable
- The available storage capacity may vary.
Why are publishers blocking the Wayback Machine?
Concern about AI training and other reuse
Publishers told Nieman Lab that AI companies might use archived pages for model training or other third-party products without permission or payment. This was presented as a concern and a reason for restricting access; as of May 20, 2026, Nieman Lab reported that no news publisher had confirmed to it that an AI company had already scraped its content from the Wayback Machine.
Protecting commercial and licensing leverage
Archived copies can extend the life of journalism beyond a publisher’s own site. Some organizations view unrestricted copying as reducing the value of their work or weakening future licensing negotiations. Advance Local spokesperson Christine deWit described its policy this way: “This is part of a broader effort to protect the value of our published work from unfair third‑party use. This decision is not specific to the Wayback Machine.”
Keeping attribution tied to the original newsroom
The Baltimore Banner emphasized whether AI products would link readers back to the original reporting. Its chief technology officer and AI strategist, Biswajit Ganguly, said, “The threat is definitely not the Internet Archive,” pointing instead to the broader problem of products reusing journalism without reliable links or credit.
The Atlantic’s stated policy is broader still. Senior vice president of communications Anna Bross said: “Our default is to block: No one should be scraping The Atlantic’s journalism without permission, regardless of the use.”
Rank #2
- Easily store and access 5TB of content on the go with the Seagate portable drive, a USB external hard Drive
- Designed to work with Windows or Mac computers, this external hard drive makes backup a snap just drag and drop
- To get set up, connect the portable hard drive to a computer for automatic recognition software required
- This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable
- The available storage capacity may vary.
What did the Nieman Lab analysis actually count?
The analysis used journalist Ben Welsh’s database of 1,167 news-site robots.txt files for the January review, then checked additional files for the May update. It counted a site when its file disallowed at least one of seven Internet Archive-associated crawler names.
A robots.txt rule is a request to cooperating crawlers. It shows the behavior a site is asking for; it does not demonstrate that every bot obeyed the rule, that a network firewall blocked traffic, or that a page has no archived copy elsewhere.
There is also an attribution caveat. Wayback Machine founder Mark Graham said Wayback does not use the names “ia_archiver,” “ia_archiverbot” or “ia_archiver-web.archive.org.” Nieman Lab nevertheless included “ia_archiver-web.archive.org” because publishers were disallowing it under the assumption that the Internet Archive used it. The count should therefore be read as evidence of requested crawler restrictions, not a verified list of technically effective Wayback blocks.
Recommended Free Tools
Rank #3
- Easily store and access 1TB to content on the go with the Seagate Portable Drive, a USB external hard drive.Specific uses: Personal
- Designed to work with Windows or Mac computers, this external hard drive makes backup a snap just drag and drop. Reformatting may be required for Mac
- To get set up, connect the portable hard drive to a computer for automatic recognition no software required
- This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable
- The available storage capacity may vary.
What is lost when news archives become harder to reach?
Evidence for reporting and accountability
Archived pages let journalists, researchers and readers compare a story with earlier versions, check what a public body or company said at a particular moment, and recover pages removed during a redesign. Nieman Lab described working journalists using local-news archives, including cases in which articles disappeared during site migrations and a defunct publication’s archive went offline.
Edward McCain, journalism librarian at the University of Missouri, told Nieman Lab: “Blocking the Internet Archive’s web crawlers threatens one of the most effective ways that we capture and store news content for the long term,”
Permanent links and the public record
Internet Archive Europe argued in a June 9, 2026 commentary that blocking crawlers reduces access to a public historical record. The organization says the Wayback Machine holds more than one trillion archived web pages and preserves permanent citations for nearly 5 million news articles referenced on Wikipedia. Those figures are claims from an interested institution, not an independent audit presented in the commentary.
Rank #4
- Easily store and access 4TB of content on the go with the Seagate Portable Drive, a USB external hard drive.Specific uses: Personal
- Designed to work with Windows or Mac computers, this external hard drive makes backup a snap just drag and drop
- To get set up, connect the portable hard drive to a computer for automatic recognition no software required
- This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable
- The available storage capacity may vary.
Internet Archive Europe also said more than 250 journalists had signed an open letter by that date. The dispute is therefore not only about convenience: it concerns whether future readers can inspect the sources behind today’s reporting.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →How can you find an old news article when a site is blocked?
- Try the Wayback Machine first. Search the article’s original address and nearby dates. A publisher’s current restriction may prevent new captures while leaving older snapshots available. Do not assume that a failed lookup proves the page never existed.
- Search the publisher’s own archive. Use the site’s date, author and topic filters, and look for a digital archive or library page. Publisher-controlled archives can preserve material that an external crawler never captured, but access rules and retention vary.
- Check library and institutional databases. ProQuest and LexisNexis are paid services that may be available through a public library, university, newsroom or individual subscription. They can provide article text or indexing even when the publisher’s page has changed.
- Ask a newsroom or local library. Reporters, editors and librarians may retain copies, print editions or internal archives, especially for a closed local publication. Explain the headline, author, approximate date and original address.
- Document the version you used. Record the publisher, headline, author, publication date, original address, archive date and the service that supplied the copy. That makes a later citation understandable if the live page changes again.
How the main preservation choices compare
| Option | Access | Capture breadth and continuity | Who controls retention | Cost and newsroom capacity | Links and earlier versions |
|---|---|---|---|---|---|
| Internet Archive Wayback Machine | Generally public when a capture is available | Broad, cross-site web captures; continuity depends on crawling and publisher restrictions | Internet Archive | Not a publisher subscription; the newsroom need not operate the archive | Can preserve dated copies and stable archive citations, but coverage is not guaranteed |
| Publisher’s own archive | May be open, metered or subscriber-only | Potentially strong for that publisher’s content; depends on migration and retention practices | Publisher | Requires the publisher to maintain storage, search and access systems | Can keep original URLs or internal links, but older versions may not be exposed |
| ProQuest or LexisNexis | Paid or institutionally licensed | Coverage depends on the database’s agreements and title history | Commercial database and its licensors | Subscription or library access; no archive operation by the newsroom using it | Useful for retrieval and citation, though the database copy may not reproduce every web element or revision |
| Newsroom archiving program | Set by the participating newsroom or project | Can target the outlet’s most important work; scale depends on staff and technical resources | Participating newsroom or preservation partner | Requires planning, training and ongoing technical capacity | Can preserve source files and metadata if the program designs for them; results vary by implementation |
No option described by these sources is shown to replace the Wayback Machine completely. Public web capture, publisher control and paid databases solve different parts of the preservation problem.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What are newsrooms doing in response?
Newsrooms can combine an internal archive with external preservation rather than relying on a single service. Nieman Lab reported a December partnership among the Internet Archive, the Poynter Institute and Investigative Reporters and Editors to train newsrooms in archiving strategies. The initial cohort included 33 local and national news outlets, with a goal of training 300 newsrooms by the end of 2027.
Best Value
- [Upgraded Version] - This external hard drive features a mirrored logo stripe combined with a striped anti-slip design, and the rounded corners of the casing make it easier to grip. The stripes also have a heat dissipation function, ensuring stable and fast data transfer.
- 【Ultra-thin and quiet】 - The motherboard adopts JMicron 578 noise-free solution, giving you a quiet working environment. Lightweight and portable size designed to fit in your pocket for easy portability.
- 【Ultra-Fast Data Transfers】 - Pairing this external hard drive with JMicron 578 solution USB 3.0 and USB 2.0 interfaces enables blazing-fast data transfer. It boasts theoretical read speeds of up to 125MB/s and write speeds of up to 103MB/s.
- 【Plug and Play】 - With no software to install, just plug it in and the drive is ready to use.The hard disk chip is wrapped with an aluminum anti-interference layer to increase heat dissipation and protect data.
- 【What You Get】 - 1 x Portable Hard Drive, 1 x USB 3.0 Cable, 1 x User Manual, Gift-type shell packaging ,Three-year manufacturer's warranty and free technical support services.
That approach treats preservation as an operational task: deciding what to save, recording metadata, assigning responsibility and checking that readers can retrieve the material later. Commercial databases and a publisher’s own archive can complement public captures, but each brings different access and retention limits.
What remains uncertain?
- The 382-site figure is a sample-based count from Nieman Lab’s methodology, not a global census.
- A disallow rule in
robots.txtindicates requested crawler behavior, not confirmed technical blocking. - Publishers’ AI-scraping explanations describe feared or possible reuse. The May 20, 2026 reporting did not include a publisher confirmation that an AI company had already taken its material from Wayback captures.
- Internet Archive Europe’s preservation totals and open-letter count are statements from that organization and should be attributed accordingly.
The practical consequence is straightforward: when a publisher withdraws permission for external crawling, future access to that outlet’s reporting becomes more dependent on the publisher’s own archive, licensed databases and whatever captures already exist.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
