Free tools Windows power users keep installed
One-click scans. No signup required.
To make a slow pandas workflow faster, first profile it, then replace Python-level row loops and row-wise UDFs with built-in pandas or NumPy operations wherever the same result can be expressed over whole columns. Next, reduce data loaded and memory used. Use eval, Numba, Cython, or another execution engine only when the workload fits and measurements show the added complexity is worthwhile.
How do I find what is making pandas slow?
Time the actual workflow before rewriting it. Separate the time spent reading data, transforming columns, joining or grouping, and writing results. A slow read or output step will not be fixed by vectorizing a calculation, and optimizing one small transformation may not matter if another stage dominates.
Record a local baseline and compare it with each change on representative data. Performance varies with the operation, dtypes, data shape, hardware, memory pressure, and whether setup or compilation time is included. There is no universal row-count threshold at which a particular optimization becomes worthwhile.
How do I vectorize pandas code?
Vectorization means expressing a calculation over whole Series or arrays instead of calling Python code once per row. Prefer pandas and NumPy built-ins for common arithmetic, comparisons, string and datetime operations, and aggregations. These operations can avoid the overhead of repeatedly invoking Python for each record.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minute#1 Best Overall
Replace row-wise calculations with column operations
For example, if a row function calculates a percentage from two columns, use column arithmetic:
df["percentage"] = 100 * (df["one"] / df["two"])This is the pattern pandas uses to illustrate replacing a user-defined row operation with a vectorized expression. Its pandas 3.0.6 UDF documentation reports 5.6435 seconds for the user-defined function and 0.0043 seconds for the vectorized operation in that example. Those are illustrative timings from the documentation, not a general benchmark or a promised speedup for other data and machines. Pandas: User-Defined Functions (UDFs)
Look for Python-level iteration and UDFs
Review iterrows, itertuples used in transformation loops, and DataFrame.apply(..., axis=1). They are not automatically wrong, but they are candidates when each row performs work that can instead be expressed with column arithmetic, boolean masks, vectorized accessors, or built-in groupby and aggregation operations. Keep a row-wise implementation when the logic genuinely depends on Python behavior that cannot be replaced cleanly; measure before investing in a more specialized rewrite.
How can I reduce pandas memory use and unnecessary work?
Less data to read and process often means less work for the whole pipeline. Apply these changes where they preserve the result:
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errors- Read only needed columns. When the input format and reader support column selection, do not load fields the workflow never uses.
- Filter early. Apply valid filters before expensive later transformations when doing so does not change the semantics.
- Inspect dtypes and memory use. Choose suitable, efficient types; lower-cardinality text columns may use more efficient representations than general-purpose strings.
- Use chunks when the task can be split. Chunking can help process data incrementally when results can be accumulated or each chunk handled with little coordination. It is not a guarantee of lower total memory or simpler code if the operation needs data from across chunks.
Pandas’ scaling guidance recommends loading less, using efficient data types, and considering chunking; it points to other libraries when the workload requires more coordination across data than chunk processing handles well. Pandas: Scaling to large datasets
When should I use eval, query, or numexpr?
Consider DataFrame.eval, DataFrame.query, or the numexpr engine for large frames with sufficiently complex arithmetic or boolean expressions. They may help in suitable cases, but parsing and temporary overhead can outweigh the benefit for a simple expression. Test against the ordinary pandas expression on your own representative workload rather than adopting them as a default.
Do not build these expression strings from untrusted input. Pandas warns that query can execute arbitrary code, so interpolating user-controlled text can expose an application to injection. Pandas: DataFrame.query API
When are Numba or Cython worth considering?
Numba
Numba can be useful for supported numerical functions and for selected pandas methods that accept a Numba engine. The code must be compatible with compilation; unsupported Python or NumPy features can prevent effective use. Include JIT compilation in first-run timing, and separately measure warmed-up execution if repeated calls are representative of production.
Cython
Cython is an option for a measured, computationally heavy hot path when compiled lower-level code is justified. It can require more implementation and maintenance than a built-in pandas expression, so reserve it for a bottleneck that simpler changes have not adequately addressed.
Pandas’ performance guide covers vectorization, eval/numexpr, Numba, and Cython, including the trade-offs and overheads that make local measurement important. Pandas: Enhancing performance
When should I consider an alternative to pandas?
Choose based on the shape of the work, not a blanket claim that one library is faster. If the data does not fit comfortably in memory, the task requires coordination across chunks, or the workflow needs a different execution model, evaluate an engine designed for that workload. For SQL-oriented analysis, DuckDB documents querying pandas DataFrames as well as supported file formats through its Python API. DuckDB: Python API
Before switching, compare the options that matter to the actual project:
- Does the dataset fit in memory, and how much data must be coordinated at once?
- Is the hot operation simple arithmetic, a complex expression, a custom numerical kernel, a SQL query, or a cross-partition workflow?
- Will first-run compilation or input loading affect the latency that matters?
- What dependency and code-maintenance cost is acceptable?
- Must outputs remain compatible with downstream pandas code?
- Are expression inputs controlled and safe?
The available documentation does not establish a head-to-head speed ranking among pandas, Polars, Dask, and DuckDB on a common benchmark. Test a representative workload, including the stages and setup costs relevant to your use case. Pandas’ user guide maps its performance and scaling material for further exploration. Pandas: User Guide
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
