For a large JSON dataset, use newline-delimited JSON (JSON Lines), read it with pd.read_json(..., lines=True, chunksize=...), and process each chunk without retaining the whole file. Reduce columns and choose compact dtypes before doing any work. Pandas remains an in-memory tool, so global joins, sorts, and groupby operations may still exceed RAM; those workloads need an out-of-core or distributed engine.
What pandas can and cannot do with data larger than RAM
Pandas stores DataFrames in memory. A file that appears to fit in available RAM can still fail when parsing creates temporary objects, type conversions make copies, or an operation builds a second DataFrame. The practical goal is to lower peak memory, not merely the size of the source file.
- Load only the fields needed for the analysis.
- Declare or convert dtypes deliberately, especially for identifiers and low-cardinality text.
- Process independent or associative work one chunk at a time.
- Avoid concatenating every chunk unless the combined result is known to fit in memory.
The pandas 3.0.6 scaling guide illustrates the potential, not a universal benchmark: selecting four Parquet columns used about one-tenth of the memory in its example. In another example, converting a low-cardinality text field to category and downcasting numeric columns produced a displayed memory ratio of 0.42; the guide says the in-memory footprint fell to one-fifth of its original size. Actual savings depend on cardinality, nulls, values, and whether later operations create copies.
Apply memory controls before parsing
Read fewer fields
For JSON, select fields during normalization when possible. For CSV or other staging files, usecols prevents unneeded columns from becoming part of the DataFrame:
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
#1 Best Overall
- Read Before You Buy — No Video Output: These adapters support charging and USB 2.0 data transfer, but cannot transmit video signals. Except for standard USB webcams (which use USB data only), they are not compatible with HDMI/DisplayPort cables, video-capable USB-C hubs, or docking stations with video output.
- Convert USB-A Ports to USB-C: Designed to connect USB-C earphones, cables, flash drives, card readers, and other USB-C accessories to standard USB-A ports. Plug-and-play with no drivers or software required.
- Aluminum Alloy Housing: Built with a sturdy aluminum alloy shell that aids in heat dissipation and protects against daily wear and scratches. Designed to maintain a stable and secure connection.
- Compact & Travel-Friendly: The ultra-compact design allows the adapter to stay plugged into your device without blocking adjacent ports or adding bulk, reducing wear and tear on your original USB ports.
- 12-Month Warranty: Backed by a 12-month manufacturer warranty for peace of mind. Designed to meet strict quality control standards for reliable everyday performance.
import pandas as pd
sales = pd.read_csv(
'sales.csv',
usecols=['account_id', 'region', 'amount'],
dtype={'account_id': 'string', 'region': 'category', 'amount': 'float32'}
)low_memory=True changes CSV parser internals; it does not make the final DataFrame out-of-core. Without chunksize or iterator, pandas still returns one complete DataFrame.
Protect identifiers and choose numeric widths
Keep ZIP codes, account numbers, and other identifiers as strings when leading zeros or nonnumeric characters matter. Downcast only after checking the valid range and missing-value behavior. Nullable pandas or PyArrow dtypes can preserve missing values without forcing an inappropriate Python object column.
Stream newline-delimited JSON with chunks
JSON Lines (often called JSONL) stores one complete JSON object per line. It is the practical JSON format for pandas iteration:
import pandas as pd
reader = pd.read_json('events.jsonl', lines=True, chunksize=100_000)
for chunk in reader:
# Work on this chunk, then release it before reading the next one.
print(len(chunk))With lines=True and chunksize, pandas returns a JsonReader iterator. The chunk size is a working parameter, not a guaranteed safe number: reduce it if parsing or downstream operations peak too high, and increase it only after measuring memory and throughput on representative data.
Recommended Free Tools
Rank #2
- 5-in-1 USB-C Hub: Experience comprehensive connectivity featuring a Power Delivery input, two USB-A 2.0 ports, a USB-A 3.0 port, and an HDMI port. (Note: The USB-C power delivery input port is only for connecting an external wall charger to power your laptop and cannot power peripheral devices.)
- 90W Pass-Through Charging: Achieve optimal charging with 90W pass-through power to your laptop, supported by a total input of 100W, with the hub reserving 10W for operational efficiency. (Note: Wall charger not included.)
- Quick Data Transfers: Accelerate your productivity with rapid data transfers using a high-speed 5Gbps USB 3.0 port and two 480Mbps USB 2.0 ports.
- 4K HDMI Display: Enhance your visual experience with a hub capable of delivering 4K resolution at 30Hz in both mirror and extend modes. Please note that this hub is compatible with MacBook (macOS 12 and newer), Windows 10 and 11, ChromeOS, and laptops equipped with DP Alt Mode and Power Delivery. Note: This device is not compatible with Linux.
- What You Get: Anker USB-C Hub (5-in-1, 4K HDMI), welcome guide, 18-month warranty, and our friendly customer service.
Aggregate without rebuilding the full table
Chunking is strongest when each chunk can be processed independently and partial results can be combined with an associative operation. This example counts event types while parsing timestamps and retaining missing event types:
import pandas as pd
counts = None
for chunk in pd.read_json('events.jsonl', lines=True, chunksize=100_000):
chunk['event_time'] = pd.to_datetime(
chunk['event_time'], errors='coerce', utc=True
)
part = chunk.groupby('event_type', dropna=False).size()
counts = part if counts is None else counts.add(part, fill_value=0)
counts = counts.astype('int64').rename('count').reset_index()Define the aggregation’s missing-value policy and verify that combining partial results gives the same answer as a single pass. Keep only the aggregate, not every processed chunk.
When the source is a regular JSON array
A conventional JSON document commonly contains one large array surrounded by a single pair of brackets. The line-oriented iterator pattern does not apply directly to that layout; loading and decoding the document can require the whole object in memory. If you control the export, produce JSON Lines. Otherwise, convert the array with a streaming-aware preprocessing step or use a parser designed for incremental JSON before handing manageable pieces to pandas.
Flatten nested JSON deliberately
pd.read_json handles file-level JSON parsing and iteration. pd.json_normalize turns nested records into tabular columns. Use the latter when objects contain nested dictionaries or arrays:
Rank #3
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
import json
import pandas as pd
with open('orders.json', encoding='utf-8') as file:
payload = json.load(file)
orders = pd.json_normalize(
payload,
record_path='orders',
meta=['customer_id', 'created_at'],
sep='.'
)record_pathidentifies the list whose members become rows.metacopies parent-level fields onto those rows.sep='.'creates predictable names such asshipping.city.- Missing keys become missing values; validate required fields after normalization.
Account for list expansion
Flattening a nested list changes the table’s grain. One order with five line items becomes five rows; exploding a second list can create a multiplicative combination. State whether a row represents an order, an order item, or another entity before joining the result back to a parent table:
items = pd.json_normalize(payload, record_path='orders', sep='.')
items = items.explode('tags', ignore_index=True)Do not silently aggregate away the extra rows. If arrays represent independent entities, normalize them into separate tables keyed by the parent identifier.
Choose the JSON orientation that matches the producer
| Orientation | Layout | Use and limitation |
|---|---|---|
records |
List of row objects | Natural for records and JSON Lines; index labels are not preserved. |
split |
Separate columns, index, and data arrays | Preserves index and column structure explicitly. |
index |
Object keyed by index | Useful when row labels are the outer keys. |
columns |
Object keyed by column | Column-oriented representation. |
values |
Nested value arrays only | Compact, but carries no labels. |
table |
Schema plus data section | Suitable when the producer supplies a table schema and data. |
Pass the matching orient when reading or writing a non-default document. Do not infer semantic meaning from automatic date conversion: parse units, time zones, and invalid values explicitly when the data contract requires it.
Use PyArrow where its support fits
Pandas can expose nullable columns backed by Apache Arrow with dtype_backend='pyarrow':
Rank #4
- Dual Converters, Infinite Potential：Includes 2× USB C male to USB A female adapters and 2× USB A male to USB C female adapters. Perfect for a wide range of uses—tablets with Bluetooth keyboards, expand USB ports on macbook, and more. Two different converters for all your daily needs
- Next-Level 10Gbps & 3A Charging: No more slow 480Mbps, this usb to usb c adapter has a transfer speed of up to 10Gbps, allowing you to do more transferring in less time. This usb adapter fits both USB A and USB C charger, supporting up to 3A fast charging
- Upgraded Exquisite Craftsmanship: With an aluminum alloy housing and metal connector, the usbc to usb adapter is extremely durable and sturdy. Rigorously tested to withstand more than 10,000 times of plugging and unplugging, ensuring long-lasting performance
- Broad Compatible: The usb c to usb adapter widely supports all USB C/ USB A devices like laptops, tablets, cellphones, car chargers, and phone chargers. Such as compatible with MacBook Pro/Air 2023/2022, Thunderbolt 4/3 Devices,Apple MagSafe Watch 9/8/7/SE/Ultra, iPad Pro 2022/2021, Samsung Galaxy S23/S20/S10, and iPhone 17/16/15 Pro. Plug and play
- Please Note: To reach 10Gbps speed, keep the cable under 3.3 ft. For USB A Male to USB C adapters, try flipping the USB C connector. USB C Male to USB A adapters support bidirectional 10Gbps transfer within 3.3 ft
import pandas as pd
events = pd.read_json(
'events.jsonl',
lines=True,
chunksize=100_000,
dtype_backend='pyarrow'
)Arrow-backed columns can improve interoperability and sometimes reduce object-heavy memory use, but they do not remove pandas’ in-memory execution model. PyArrow is also an IO engine for supported pandas readers, yet engine coverage is reader- and option-specific. The pandas IO documentation notes that some features are unsupported in the PyArrow engine and that chunking behavior can differ by engine. Check the exact reader and pandas version before depending on an engine-specific option, and test dtypes after parsing.
Know when chunking is the right algorithm
| Workload | Chunking fit | Recommended approach |
|---|---|---|
| Per-record validation or file conversion | Strong | Process and write each chunk, keeping only counters and error records. |
| Value counts, sums, and other associative reductions | Strong | Compute a partial result and merge it with add, sums, or another proven associative combine. |
| Global groupby with manageable key state | Conditional | Aggregate partial groups, then combine; monitor the number of unique keys. |
| Joins requiring all keys | Weak | Use a keyed, out-of-core strategy or an engine that can coordinate partitions. |
| Global sorting | Weak | Use an external-sort or query engine rather than concatenating all chunks. |
| Algorithms requiring repeated passes | Weak | Choose a system designed for out-of-core or distributed execution. |
Pandas describes chunking as effective when an operation requires zero or minimal coordination between chunks. A chunked read alone cannot make a globally coordinated operation memory-safe.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.A practical workflow for a large JSON analysis
- Confirm the format. Identify whether the input is JSON Lines, a regular array, or a nested document, and record the expected row grain.
- Inspect a small sample. Check key names, nulls, identifier formats, timestamp units, and nested-list lengths before selecting dtypes.
- Reduce the schema. Normalize only required fields; keep identifiers as strings and choose compact numeric or categorical types where valid.
- Set a conservative chunk size. Start small enough that parsing plus the planned operation fits comfortably in RAM, then measure.
- Process and release. Perform the per-chunk transformation, merge only the necessary partial result, and avoid a list of retained DataFrames.
- Validate totals. Compare row counts, null counts, key coverage, and representative values against the source or a smaller full-load sample.
- Escalate deliberately. If the required operation needs global coordination or repeated passes, move to an out-of-core, parallel, or distributed engine instead of endlessly tuning
chunksize.
Troubleshoot common failures
MemoryError or the process is killed
- Lower
chunksizeand remove unused fields before parsing. - Inspect
df.dtypesand convert object-heavy columns to appropriate string, categorical, nullable, or Arrow-backed types. - Look for accidental copies from chained transformations, concatenation, sorting, or joins.
- Write intermediate aggregates rather than retaining every chunk.
Numbers or identifiers are wrong
Leading zeros disappear when an identifier is parsed as an integer. Supply an explicit string dtype where supported and validate a sample after reading. Mixed numeric and text values should be handled as a documented schema decision, not left to inference.
Dates parse inconsistently
Date inference is a convenience, not proof that a timestamp has the intended timezone or unit. Use pd.to_datetime with explicit error handling and timezone treatment, then check invalid and ambiguous values.
Free tools Windows power users keep installed
One-click scans. No signup required.
Best Value
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
Row counts increase unexpectedly
Inspect every nested array passed through record_path or explode. A list expansion legitimately creates multiple rows, but a second expansion or an incorrect metadata join can multiply them again. Validate the key and grain after each normalization step.
Chunking does not reduce memory
Confirm that the code is iterating over a JsonReader and not calling pd.concat on all chunks. Also check whether a later global operation recreates the entire dataset; the read phase may be bounded while the analysis phase is not.
When to leave pandas
Stay with pandas when the selected columns and required intermediate state fit in memory and the computation can be expressed as independent chunk work or a small associative reduction. Consider an out-of-core or distributed dataframe/query system when the job needs global joins, global sorting, repeated scans, or parallel execution across partitions. PyArrow-backed pandas columns improve type interoperability, but they are not by themselves a distributed execution engine.
Bottom line
For large JSON, prefer JSON Lines, read with lines=True and chunksize, normalize only the fields you need, and combine partial results instead of concatenating raw chunks. Treat dtypes, nested-list grain, timestamp semantics, and parser-engine support as correctness concerns as well as memory concerns. If the algorithm requires coordination across the entire dataset, use a system designed for that workload rather than forcing pandas to hold it all.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
