Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11To make a pandas DataFrame use less memory, first measure its columns with df.memory_usage(deep=True), then selectively convert repeated text to categorical data, downcast numeric columns only when their ranges and precision allow it, or use sparse types for genuinely sparse data. If you mean a smaller saved file, optimize Parquet separately: compression reduces file bytes, not necessarily the memory required when the data is loaded.
How to find which columns use the most memory
Start by measuring the current frame. The result is an estimate of bytes per column; the index is included by default. deep=True inspects values stored in object-dtype columns, which ordinary accounting may undercount. The deeper inspection can take more time, and its total is not a measurement of the entire Python process’s resident memory.
usage = df.memory_usage(deep=True).sort_values(ascending=False)
print(usage)
print(f"Total: {usage.sum():,} bytes")To exclude the index from the report, pass index=False. Pandas’ FAQ explains that the “+” shown in some memory reports means actual use could be higher because values in object columns are not counted by ordinary accounting. See the memory_usage API documentation and the pandas memory FAQ.
Which dtype changes can reduce DataFrame memory?
Convert repeated, low-cardinality text to category
A categorical stores the distinct categories separately and represents rows with codes. It is often a good candidate when a long column repeats a relatively small set of labels, such as status, region, or group. The benefit depends on both the number of rows and the number of distinct categories: a near-unique text column may not shrink and can use more memory.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →#1 Best Overall
- Get NVMe solid state performance with up to 1050MB/s read and 1000MB/s write speeds in a portable, high-capacity drive(1) (Based on internal testing; performance may be lower depending on host device & other factors. 1MB=1,000,000 bytes.)
- Up to 3-meter drop protection and IP65 water and dust resistance mean this tough drive can take a beating(3) (Previously rated for 2-meter drop protection and IP55 rating. Now qualified for the higher, stated specs.)
- Use the handy carabiner loop to secure it to your belt loop or backpack for extra peace of mind.
- Help keep private content private with the included password protection featuring 256‐bit AES hardware encryption.(3)
- Easily manage files and automatically free up space with the SanDisk Memory Zone app.(5). Non-Operating Temperature -20°C to 85°C
before = df.memory_usage(deep=True).sum()
df["group"] = df["group"].astype("category")
after = df.memory_usage(deep=True).sum()
print(f"Before: {before:,} bytes")
print(f"After: {after:,} bytes")Compare the measured result on your own data, and retain the conversion only if categorical semantics suit the column and the full workload. See pandas’ categorical data guide.
Downcast numeric columns only after checking their requirements
Smaller integer and floating-point dtypes can reduce memory, but a narrower dtype may not represent every value or the precision your calculations require. Check each column’s minimum and maximum, missing-value behavior, and required numerical precision before converting. Pandas provides pd.to_numeric(..., downcast=...) to select a smaller compatible numeric dtype; measure again after the conversion.
# Example only: choose downcast options after validating each column.
df["id"] = pd.to_numeric(df["id"], downcast="unsigned")
df["amount"] = pd.to_numeric(df["amount"], downcast="float")
print(df.memory_usage(deep=True).sort_values(ascending=False))The exact safe type depends on the data and calculations, so do not treat a downcast choice as universally suitable. Pandas’ scaling guide demonstrates downcasting on a particular generated dataset; its result is an illustration, not a general memory or speed guarantee.
Rank #2
- Solid state performance with up to 800MB/s read speeds in a portable drive. (Based on internal testing; performance may be lower depending on host device, interface, usage conditions and other factors. 1MB=1,000,000 bytes.)
- Back up your content and memories on a storage solution that fits seamlessly into your mobile lifestyle.
- Take it with you on your adventures—up to two-meter drop protection means this durable drive can take a beating. (Based on internal testing.)
- Secure it to your belt loop or backpack for extra peace of mind thanks to the tough rubber hook.
- From Sandisk, a brand professional photographers trust to take on assignments.
Use sparse types when most values are the fill value
Sparse storage is intended for columns or matrices in which most entries are the fill value, commonly zero. For dense data, it may not save memory, and storage savings do not guarantee that every operation will be faster or supported in the same way.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problemsprint(df.sparse.density)Check that the data are sparse, measure memory after using a suitable SparseDtype, and test the operations your application actually needs. Pandas documents the sparse accessor and density in its sparse API.
What does pandas’ scaling example show?
The pandas 3.0.6 scaling guide uses a generated frame with 1,051,201 rows. After converting a repeated name field to category and downcasting numeric columns, it reports a new-to-original deep-memory ratio of 0.42—about 42% of the original memory for that example. The same passage describes the result as “1/5” of the original, which does not agree with the printed ratio. The ratio is the consistent figure to use; neither number predicts the result for another dataset.
Rank #3
- Easily store and access 2TB to content on the go with the Seagate Portable Drive, a USB external hard drive
- Designed to work with Windows or Mac computers, this external hard drive makes backup a snap just drag and drop
- To get set up, connect the portable hard drive to a computer for automatic recognition no software required
- This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable
- The available storage capacity may vary.
How to make a saved Parquet file smaller
File size and in-memory DataFrame size are different measurements. Parquet is a columnar binary format, and compression can reduce bytes on disk without producing an equivalent reduction in memory after loading. Pandas’ to_parquet requires either pyarrow or fastparquet; compare the resulting file size, load time, and loaded dtypes for your actual use case.
-
Choose Parquet and an available engine, then try an appropriate compression option with
DataFrame.to_parquet.Recommended: PC Feels Slow? A Free Scan Shows What's Dragging Windows Down →Recommended: Update Every Outdated Driver on Your PC in One Scan - Free →Recommended: Fix Windows Errors and Clear Junk Files in Minutes - Free Scan →Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy. -
Decide explicitly whether to write the index; index serialization changes the saved representation.
Rank #4
Sale
Sandisk 1TB Extreme Portable SSD, Up to 2000MB/s Transfer Speeds-New Model- NEARLY 2X FASTER THAN OUR PREVIOUS GENERATION(8) – move 1,000 high-res photos in under 60 seconds(6) with up to 2000MB/s transfer speeds(2).
- IP65 RATING AND UP TO 3M DROP PROTECTION(3) – protects against spills and drops.
- POCKET-SIZED – fits easily in pockets and small bags.
- SPACE TO OWN YOUR AI CONTENT – speed and capacity to download your high-res clips and photo edits.
- 256-BIT AES ENCRYPTION(4) – helps keep private files secure with password protection.
-
If categorical columns contain unused categories, consider removing them before writing. The extra category metadata can enlarge the output.
-
Read the file back and check its size, dtypes, and behavior against the application’s needs.
See the to_parquet API documentation and pandas’ Parquet I/O guide.
Best Value
- Easily store and access 5TB of content on the go with the Seagate portable drive, a USB external hard Drive
- Designed to work with Windows or Mac computers, this external hard drive makes backup a snap just drag and drop
- To get set up, connect the portable hard drive to a computer for automatic recognition software required
- This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable
- The available storage capacity may vary.
A practical order of operations
-
Record per-column and total deep memory usage, including or excluding the index deliberately.
-
Identify repeated text, numeric columns with safely reducible ranges or precision, and genuinely sparse data.
-
Make one dtype change at a time and measure again, checking missing values and value fidelity.
-
Run representative operations, such as grouping or serialization, before keeping a conversion.
Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy. -
If disk space is the problem, optimize and validate the Parquet representation separately.
Chunked processing can help with some large-data workflows, but pandas notes that operations such as DataFrame.groupby() are harder to perform chunkwise; chunking is not a universal fix for memory limits.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
