Use the official X API to collect public posts with a documented query, fixed time window, and complete pagination. Then clean and validate the resulting sample before assigning sentiment. Recent Search covers the previous seven days; Full-archive Search can reach back to March 2006 but requires eligible Self-serve or Enterprise access. Check your account’s current access, prices, quotas, and policy terms before you design the study.
1. Define exactly what you want to measure
Sentiment results are only as defensible as the population you define. Write a short collection specification before requesting data.
Specify the population
- Topic: the product, event, brand, or phrase of interest.
- Language: one language or a multilingual sample that will be modeled separately.
- Dates: UTC start and end times, including whether the end is inclusive.
- Post type: decide whether replies and reposts are included.
- Unit of analysis: normally one original post or one post including replies.
- Account scope: all matching public posts or posts from selected accounts.
A keyword query is a query-defined sample, not a census of opinion. It can miss people who use different words and include posts where a term has another meaning. Keep the query and inclusion rules unchanged when comparing periods or groups; record every revision.
Build a reproducible query
X search syntax supports exact phrases, hashtags, mentions, account filters such as from: and to:, language filters such as lang:en, and exclusions including -is:retweet and -is:reply. For example:
Free tools Windows power users keep installed
One-click scans. No signup required.
#1 Best Overall
- Ask, Measure, Learn: Using Social Media Analytics to Understand and Influence Customer Behavior
- O'Reilly Media
- ABIS BOOK
("electric vehicle" OR #EV) lang:en -is:retweet -is:replySave the literal query, date boundaries, language, exclusions, collection timestamp, API access level, and code version alongside the data. Operator availability and access requirements can change, so verify the current Search Posts documentation before production use.
2. Choose Recent or Full-archive Search
| Route | Date coverage | When it fits | Important qualification |
|---|---|---|---|
| Recent Search | Previous seven days | Monitoring, rapid studies, and current events | Older posts are outside this window. |
| Full-archive Search | Back to March 2006 | Historical studies and long-term comparisons | The documented quickstart requires Self-serve or Enterprise access; confirm your account’s present eligibility. |
Do not copy historical tier prices or quotas into a current plan. X access, pricing, regional eligibility, and usage limits are volatile. Confirm them directly in your developer account before promising a sample size or archive range.
3. Obtain credentials and protect them
Register for X developer access, create a project and app, and obtain a Bearer Token with permission to read the public data your study requires. X makes public posts and replies available through its API, but it can suspend or terminate access for policy violations. Read the current developer policy, especially rules for storage, redistribution, deletion handling, and research use.
- Store the token in an environment variable, never in source control.
- Restrict file and log permissions on machines that hold raw data.
- Do not publish raw post text or personal data unless your policy and consent basis permit it.
- Keep post IDs and collection metadata so you can honor deletion or withholding requirements.
4. Collect every page with Python
The following client uses the standard v2 search endpoint, requests up to 100 posts per page, follows next_token, and writes newline-delimited JSON plus a collection manifest. Replace the endpoint or fields if your account’s current API documentation specifies a different route.
import json
import os
import time
from datetime import datetime, timezone
import requests
TOKEN = os.environ["X_BEARER_TOKEN"]
QUERY = '("electric vehicle" OR #EV) lang:en -is:retweet -is:reply'
START = "2026-09-01T00:00:00Z"
END = "2026-09-08T00:00:00Z"
OUT = "x_posts.ndjson"
url = "https://api.x.com/2/tweets/search/recent"
params = {
"query": QUERY,
"start_time": START,
"end_time": END,
"max_results": 100,
"tweet.fields": "id,text,created_at,lang,author_id,public_metrics,conversation_id",
"expansions": "author_id",
"user.fields": "id,username,protected,public_metrics"
}
headers = {"Authorization": f"Bearer {TOKEN}"}
count = 0
next_token = None
errors = []
with open(OUT, "w", encoding="utf-8") as f:
while True:
request_params = dict(params)
if next_token:
request_params["next_token"] = next_token
for attempt in range(6):
response = requests.get(url, headers=headers, params=request_params, timeout=60)
if response.status_code != 429:
break
time.sleep(min(60, 2 ** attempt))
if response.status_code != 200:
errors.append({"status": response.status_code, "body": response.text})
response.raise_for_status()
payload = response.json()
for post in payload.get("data", []):
f.write(json.dumps(post, ensure_ascii=False) + "n")
count += 1
next_token = payload.get("meta", {}).get("next_token")
if not next_token:
break
manifest = {
"query": QUERY,
"start_time": START,
"end_time": END,
"collected_at": datetime.now(timezone.utc).isoformat(),
"count": count,
"errors": errors
}
with open("collection_manifest.json", "w", encoding="utf-8") as f:
json.dump(manifest, f, indent=2)
print(f"Saved {count} posts")Set the token before running:
export X_BEARER_TOKEN='YOUR_BEARER_TOKEN'
python collect_x.pyThe API may return fewer than 100 posts even when more are available. Continue until no next_token remains. Store response metadata and errors, not just post text.
5. Make pagination and rate limits reliable
Pagination
A successful response can include a next_token. Pass it unchanged into the next request. The XDK for Python can iterate pages for you, but you still need to save the query, boundaries, response metadata, and failures.
HTTP 429 responses
HTTP 429 indicates a rate limit or usage-cap response. Pause and retry with exponential backoff, as the example does. For large jobs, divide work into non-overlapping UTC windows, persist progress after each page, and make reruns idempotent by deduplicating on post ID.
Partial coverage
Protected-account posts, deleted posts, and posts withheld in some regions may not be returned. A successful request therefore does not prove complete platform coverage. Report the accessible, query-matching sample and the collection period rather than claiming to represent all X users or public opinion.
6. Prepare posts for sentiment analysis
Preserve an audit trail
Keep the post ID, timestamp, language, author ID (subject to your privacy policy), conversation ID, query, and collection time. Keep raw responses access-controlled and create an analysis table derived from them. Deduplicate according to your design; reposts and quoted material can otherwise dominate counts.
Decide how text is interpreted
- Separate languages or use a classifier evaluated for every language in the sample.
- Document treatment of URLs, mentions, hashtags, emojis, punctuation, and repeated characters.
- Decide whether replies require conversational context.
- Flag sarcasm, slang, negation, coded language, and posts whose meaning depends on an image or video.
Validate labels
Model output is not ground truth. Define what positive, negative, neutral, or other labels mean; evaluate the classifier on manually reviewed examples from your topic and language; and report uncertainty and known failure cases. Do not claim that a model is accurate merely because it produced a label. If comparing models, use the same held-out, human-labeled set and report class-specific results rather than only an overall score.
7. Analyze without overstating the sample
Use counts or proportions with the denominator and date window visible. If comparing groups, apply the same query, language rules, inclusion criteria, and deduplication method. A rise in negative posts may reflect a change in vocabulary, news attention, account activity, or access—not a change in the population’s underlying attitude.
Describe your result as “posts matching this query and available through the X API during this period.” Do not generalize it to all users, residents of a country, or public opinion unless you have an independent sampling design that supports that inference. Earlier studies of the former Twitter Academic API, including a 2022 study that found evidence of nearly complete samples for many search terms, do not establish current X API completeness or representativeness.
Recommended Free Tools
8. Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| 401 or 403 | Missing, expired, or insufficiently privileged token | Check the app’s credentials and read permissions; regenerate the token and confirm your account’s product access. |
| 400 invalid query | Unsupported operator, malformed parentheses, or unavailable field | Reduce the query to a known-valid term, then add operators one at a time using the current operator reference. |
| 429 | Rate or usage cap | Honor backoff, lower concurrency, split time windows, and verify the account quota. |
| Fewer posts than expected | Narrow query, seven-day Recent window, protected/deleted/withheld posts, or exhausted pages | Check dates and exclusions, inspect pagination metadata, and report coverage limits. |
| Duplicate posts | Overlapping windows or rerun after interruption | Use post ID as the primary key and remove duplicates before modeling. |
| Sentiment looks wrong | Sarcasm, negation, slang, multilingual text, or missing context | Review a stratified sample, refine preprocessing, and validate or replace the model for the target domain. |
Or skip the browser setup
If your workflow also needs visual snapshots of pages, ScreenshotNeo provides a one-call screenshot API and MCP server. It removes cookie banners, newsletter popups, and chat widgets before capture; bot checks, blank pages, failed loads, and cache hits are not billed; and AI agents can call its take_screenshot, get_page_info, and capture_pdf tools through MCP.
cURL:
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpPython:
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)Node.js:
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);See the ScreenshotNeo documentation for the 63 capture options, response headers, PDFs, signed links, bulk jobs, and webhooks. The Free plan includes 1,000 screenshots per month with no card; paid plans start at $5 for 3,000. Create a free ScreenshotNeo account.
Frequently Asked Questions
Can I use tweets deleted after collection?
Retain and use data only as permitted by the current X developer terms and your approved research or privacy process; maintain deletion handling for stored identifiers and content.
How far back does Recent Search go?
The documented Recent Search window is the previous seven days. Older dates require Full-archive access, subject to your account’s current eligibility.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchDoes a larger post count make sentiment representative?
No. Size does not correct query bias, missing protected or deleted posts, language imbalance, or unequal user activity.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
