Recommended Free Tools
iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Web scraping can help investigators find and preserve publicly available social-media material, but a downloaded post is not automatically authentic, admissible, or proof of a war crime. A defensible investigation defines a narrow purpose, collects only what is necessary, preserves the original content and associated information, records every step, verifies source and context, and protects the people represented in the data.
What scraping can—and cannot—establish
Scraping is a collection technique inside a broader digital open-source investigation. It can locate public posts, copy page content, save media, and create a reproducible record of when and how material was obtained. It cannot, by itself, prove who created a post, whether a video is genuine, where or when it was recorded, or whether the conduct shown satisfies the legal elements of a war crime.
The Berkeley Protocol on Digital Open Source Investigations (Human Rights Center at UC Berkeley and the Office of the UN High Commissioner for Human Rights, 2020) treats collection as an investigative decision governed by purpose, necessity, and proportionality. The ICC Office of the Prosecutor and Eurojust guidance (2022) adds a practical distinction: an ordinary screenshot or PDF may be adequate work product, while potentially probative material should be downloaded or recorded with relevant associated information and assigned a hash. A 2024 guide for judges and fact-finders stresses authenticity, source, metadata, location, and time.
The protocol was developed through six workshops and consultations involving more than 150 experts. Alexa Koenig, then executive director of UC Berkeley Law’s Human Rights Center, called it “a living document that will help strengthen war crimes investigations using 21st century methods, with the goal of improving justice and accountability worldwide.” That figure describes the protocol’s development, not the accuracy or success rate of scraping.
#1 Best Overall
- Easily store and access 2TB to content on the go with the Seagate Portable Drive, a USB external hard drive
- Designed to work with Windows or Mac computers, this external hard drive makes backup a snap just drag and drop
- To get set up, connect the portable hard drive to a computer for automatic recognition no software required
- This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable
- The available storage capacity may vary.
1. Define the investigation before collecting anything
Write a short collection plan that another investigator could understand without asking you questions. State:
- The factual question, such as whether a particular unit was present at a location during a specified attack.
- The intended use: lead generation, investigative analysis, reporting, or material that may be submitted to a court or truth-seeking body.
- The geographic and time boundaries.
- The types of posts, accounts, media, comments, and metadata that could answer the question.
- What you will deliberately exclude, such as unrelated bystanders, private accounts, or personal data that cannot advance the inquiry.
- Who may access the collected material and how long it will be retained.
Record changes to the plan and the reason for each change. Narrow scope reduces irrelevant personal information, review workload, security exposure, and the risk that investigators lose track of why an item was collected.
2. Identify public sources and assess collection risk
List the public URLs, account names, hashtags, channels, dates, and search terms that may be relevant. Confirm that each item is accessible without logging in, bypassing a technical restriction, defeating a bot check, or inducing someone to disclose restricted information under false pretenses. Public visibility does not settle whether a method is lawful or ethical. Applicable rules depend on jurisdiction, platform terms, access controls, privacy law, and the investigation’s purpose.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallCrashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteBefore running automation, ask whether bulk collection is actually necessary. Public pages can contain names, faces, phone numbers, location traces, medical information, and details about children or survivors. Establish secure devices, storage, account separation, malware scanning, and an incident process before visiting hostile or compromised pages. The Berkeley Protocol emphasizes security by design and the digital, physical, and psychosocial safety of witnesses, survivors, investigators, and others.
Manual capture or automation?
| Approach | Best use | Main strengths | Main risks |
|---|---|---|---|
| Itemized manual capture | A small number of high-value posts | High selectivity; easier to review the exact context and document judgment | Slow; content can disappear while an investigator works |
| Automated collection | A defined, repeatable set of public pages or a short-lived event requiring timely coverage | Consistent timestamps and processing; handles larger, pre-scoped sets | May gather unnecessary personal data, trigger defenses, miss JavaScript-rendered content, or create a large disclosure and security burden |
The Berkeley Protocol does not label automation universally good or bad. Choose the least intrusive method that can answer the defined question. If a handful of posts matter, itemize them. If automation is justified, use a rate limit, a fixed URL list, a stop condition, and logging. Do not use a general-purpose crawler to collect everything from a platform.
3. Capture public pages and preserve the original response
For low-stakes work product, a screenshot or PDF can show what a reviewer saw. For material that may have probative value, preserve the underlying file or recording and its associated information. Capture at least the source URL, final URL after redirects, collection time in UTC, HTTP status, content type, response headers, account or page identifier, visible text, and the original bytes. Save the investigator, tool version, command, and any transformations in a separate log.
A SHA-256 hash is a compact fingerprint of a captured file. Recomputing it later can show whether that file changed. A hash does not prove who uploaded the content, that the content is true, or that the file was obtained from the claimed source.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Rank #2
- Easily store and access 1TB to content on the go with the Seagate Portable Drive, a USB external hard drive.Specific uses: Personal
- Designed to work with Windows or Mac computers, this external hard drive makes backup a snap just drag and drop. Reformatting may be required for Mac
- To get set up, connect the portable hard drive to a computer for automatic recognition no software required
- This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable
- The available storage capacity may vary.
Minimal Python collector for a public, static page
Use this only on pages you are authorized to access without bypassing controls. It saves the raw HTTP response and a JSON record; it does not attempt a login, defeat a CAPTCHA, or crawl links.
Install the dependency with python -m pip install requests, then save the following as collect_public_page.py:
import datetime as dt
import hashlib
import json
from pathlib import Path
import requests
url = 'https://example.org/public-post'
out_dir = Path('case_capture')
out_dir.mkdir(exist_ok=True)
headers = {
'User-Agent': 'EvidenceCollector/1.0 (contact: investigator@example.org)'
}
response = requests.get(url, headers=headers, timeout=30, allow_redirects=True)
response.raise_for_status()
raw = response.content
retrieved = dt.datetime.now(dt.timezone.utc).isoformat()
sha256 = hashlib.sha256(raw).hexdigest()
html_path = out_dir / 'page.html'
html_path.write_bytes(raw)
record = {
'requested_url': url,
'final_url': response.url,
'retrieved_utc': retrieved,
'status_code': response.status_code,
'content_type': response.headers.get('content-type'),
'content_length': len(raw),
'sha256': sha256,
'response_headers': dict(response.headers),
'file': str(html_path)
}
(out_dir / 'collection.json').write_text(
json.dumps(record, indent=2, ensure_ascii=False), encoding='utf-8'
)
print(json.dumps(record, indent=2, ensure_ascii=False))Keep the original bytes unchanged. If you create a redacted copy for sharing, give it a new filename and hash, and retain the relationship between the original and derivative. A static request may return only a shell when a site renders posts with JavaScript; record that limitation instead of treating an empty response as proof that no content existed.
4. Preserve screenshots, PDFs, media, and associated information
For each important item, make the capture method match the expected evidentiary value:
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
| Capture | What it preserves | When it is insufficient |
|---|---|---|
| Screenshot | Visible layout, text, account label, and what a viewer saw at a moment | When original media bytes, hidden metadata, redirects, or page history may matter |
| PDF conversion | A paginated rendering useful for review and annotation | When dynamic content, video, original resolution, or machine-readable metadata is important |
| Original download or recording plus associated information and hash | File bytes, collection details, and a way to detect later alteration | Still does not establish authorship, truth, location, or legal responsibility by itself |
Record whether the platform displayed an edit marker, repost label, warning, view count, timestamp precision, or deleted-content notice. Preserve the surrounding conversation when it is necessary to interpret the item, but do not copy unrelated personal data merely because it is available.
5. Verify before drawing an investigative conclusion
Authenticate the file and the source
Compare the captured media with other copies, inspect compression and editing artifacts, and preserve platform-provided metadata where available. Check the account’s history, creation date, naming changes, repost patterns, and whether an alleged original exists. Treat an unverified account attribution as a lead, not an established identity.
Establish location and time
Use visible landmarks, signs, terrain, shadows, weather, satellite imagery, corroborating eyewitness material, and reliable event timelines. Metadata can be missing, stripped, or altered, so explain how much confidence each indicator deserves. Distinguish the time a post was uploaded from the time an event occurred.
Rank #3
- High capacity in a small enclosure – The small, lightweight design offers up to 6TB* capacity, making WD Elements portable hard drives the ideal companion for consumers on the go.
- Plug-and-play expandability
- Vast capacities up to 6TB[1] to store your photos, videos, music, important documents and more
- SuperSpeed USB 3.2 Gen 1 (5Gbps)
Separate observation from assertion
Write separately what the image or video visibly shows, what the accompanying text claims, and what independent sources support. Seek corroboration and test alternative explanations, including recycled footage, staged scenes, mislabeled locations, satire, and translation errors. Do not convert a collection result into a conclusion about a commander or organization without evidence addressing attribution and intent.
6. Handle deletion, edits, and platform moderation
Online material can be removed by a platform, deleted or edited by its author, corrupted, or separated from its original context. Lindsay Freeman’s 2022 report Digitally Disappeared describes how moderation, retention, and disclosure practices affect preservation. Capture promptly when an item may matter, but label the record with the exact collection time and the fact that later versions may differ.
If a post disappears, preserve your prior capture and log what you observed. Do not present a screenshot of an earlier version as the current page. A later mirror or archive can corroborate availability, but it does not automatically establish provenance.
7. Store, review, and share safely
- Use encrypted storage with role-based access and a separate backup.
- Keep an append-only activity log showing downloads, transformations, reviews, and exports.
- Scan files in an isolated environment; images, documents, and archives can carry malicious payloads.
- Minimize names, faces, addresses, and contact details that are not relevant.
- Protect sources and survivors from retaliation, doxxing, and renewed exposure to graphic material.
- Share the smallest useful set with a clear description of provenance, uncertainty, and handling restrictions.
Over-collection creates privacy, security, review, and disclosure burdens. A smaller, well-documented set is often more useful than an enormous unstructured scrape.
Legal and ethical limits
There is no universal answer to whether scraping is allowed. Review the law applicable to the investigator, the people represented, and the place where systems are hosted. Check platform terms and access controls. Avoid false pretenses, credential use, circumvention, and collection of restricted material unless a qualified lawyer and the relevant investigative mandate specifically authorize the method. Public accessibility alone does not eliminate privacy, copyright, data-protection, or safety obligations.
The Berkeley Protocol calls for documented investigative processes and attention to privacy and terms-of-service issues. If a proposed collection could expose a witness or survivor, pause and obtain specialist advice before proceeding.
Common failures and fixes
403, 429, or a bot-check page
Cause: The platform is restricting automated traffic or enforcing a rate limit. Fix: Stop rather than rotate identities or bypass the control. Narrow the scope, wait, use an authorized export, or collect manually if lawful. Record the failed attempt.
Rank #4
- Plug-and-play expandability
- SuperSpeed USB 3.2 Gen 1 (5Gbps)
The saved HTML contains no post
Cause: The page is rendered in JavaScript, requires a session, or returned a consent/interstitial page. Fix: Verify the response URL and content type, save the interstitial as part of the record, and use an authorized browser capture for the public view. Do not treat missing HTML as missing content.
The page changes between captures
Cause: Edits, moderation, personalization, or time-sensitive feeds. Fix: Capture the full context promptly, record UTC time and account state, and hash each version separately.
Free tools Windows power users keep installed
One-click scans. No signup required.
The hash does not match later
Cause: The file was transformed, re-encoded, truncated, or replaced. Fix: Recompute the hash from the original bytes, preserve both files, and document every transformation. Do not overwrite the first capture.
The dataset is too large to review
Cause: Collection was broader than the question required. Fix: Stop the crawler, apply documented inclusion criteria, delete or quarantine irrelevant personal data according to policy, and retain a record of the filtering decision.
Performance, reliability, and cost decisions
Use a single-threaded or low-concurrency collector unless the plan justifies more. Add timeouts, retries only for transient server errors, exponential backoff, and a maximum page count. Log failures instead of silently dropping URLs. Keep raw captures separate from parsed text so a parser bug does not destroy the source record. Test the process on a small, non-sensitive set before a live collection.
Scraping itself may be inexpensive; the major costs are secure storage, analyst review, translation, verification, legal advice, and safe handling of sensitive material. No validated statistic establishes a universal accuracy, admissibility, or success rate for web scraping in war-crimes investigations.
Or skip the browser setup
For a public page where you need a clean rendering, ScreenshotNeo is the first screenshot API to try because it removes common consent banners, newsletter popups, and chat widgets before capture, bills only clean shots, and has the lowest paid plan. It is a rendering and capture service, not an authentication or legal-verification system; you still need to preserve provenance, verify the material, and follow your investigation plan.
The API accepts one GET request. See the ScreenshotNeo documentation for parameters and response headers.
Best Value
- 【Upgraded version】 - The mirror logo strip is combined with the striped non-slip design. The rounded corners of the shell are more suitable for holding. The strips play a heat dissipation function to ensure a stable and fast transmission process.
- 【Ultra-thin and quiet】 - The motherboard adopts JMicron 578 noise-free solution, giving you a quiet working environment. Lightweight and portable size designed to fit in your pocket for easy portability.
- 【Ultra-Fast Data Transfers】 - Pairing this external hard drive with JMicron 578 solution USB 3.0 and USB 2.0 interfaces enables blazing-fast data transfer. It boasts theoretical read speeds of up to 125MB/s and write speeds of up to 103MB/s.
- 【Plug and Play】 - With no software to install, just plug it in and the drive is ready to use.The hard disk chip is wrapped with an aluminum anti-interference layer to increase heat dissipation and protect data.
- 【What You Get】 - 1 x Portable Hard Drive, 1 x USB 3.0 Cable, 1 x User Manual, Gift-type shell packaging ,Three-year manufacturer's warranty and free technical support services.
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://example.org/public-post -o shot.webpimport requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://example.org/public-post"}, timeout=90)
open("shot.webp", "wb").write(r.content)const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://example.org/public-post' }); const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);ScreenshotNeo can capture full pages, selected elements, dark mode, device and retina settings, PDFs, custom CSS or JavaScript, click-and-wait flows, blocked resource types, custom headers and cookies, geolocation and timezone, transparent backgrounds, resizing, chosen cache TTLs, signed links, asynchronous jobs with signed webhooks, bulk capture of up to 100 URLs per call, usage data, and OpenAPI integration. Every response identifies whether the page was clean, a bot check, blank, timed out, failed, or a cache hit through X-Page-Verdict and X-Billed headers. Bot checks, blank pages, timeouts, failed loads, and cache hits cost nothing.
An MCP server provides take_screenshot, get_page_info, and capture_pdf tools for Claude, Cursor, and other MCP clients. The service offers 1,000 screenshots per month free with no card. Paid plans are:
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →| Plan | Included shots | Price |
|---|---|---|
| Free | 1,000 per month | $0, no card |
| Starter | 3,000 | $5 |
| Growth | 15,000 | $15 |
| Pro | 60,000 | $39 |
| Scale | 250,000 | $99 |
| Business | 1,000,000 | $249 |
Yearly billing gives two months free, and every feature is available on every plan. Cookie banners, popups, and chat widgets are removed before the shot; bot checks, blank pages, and failed loads are never billed; AI agents can capture through MCP; and the free tier requires no card. Create a free ScreenshotNeo account to start.
A defensible evidence record
A strong collection file lets an independent reviewer answer five questions: why this item was collected, how it was obtained, what exactly was preserved, what checks support its authenticity and context, and who handled it afterward. Scraping can supply the first physical copy of a public post. The investigative value comes from disciplined scope, preservation, verification, corroboration, and protection of people—not from automation alone.
Frequently Asked Questions
Can a screenshot alone be submitted as war-crimes evidence?
It may be useful supporting material, but its weight depends on provenance, authenticity, context, and the rules of the forum considering it. Preserve the underlying content and collection record when the item may be probative.
Does a SHA-256 hash prove that a social-media post is authentic?
No. It shows whether the captured file changed after hashing. It does not prove authorship, upload history, truth, location, or timing of the depicted event.
Is scraping a public social-media page automatically legal?
No. Legality varies with jurisdiction, platform rules, access controls, privacy obligations, and the investigation’s purpose. Public visibility is only one fact to assess.
What should investigators do when a platform removes a post?
Retain the earlier capture and its metadata, record when the item disappeared, and label mirrors or later copies as separate sources. Do not claim that a screenshot represents the page’s current state.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
