iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Compacting Apache Iceberg data files can reduce metadata overhead and the cost of opening many small files, but a drop from 1,000 files to 6 does not by itself prove queries became faster. The result depends on which files were rewritten and which queries you measure: one 2026 benchmark found gains for metadata-heavy queries but slower full-scan aggregations when compaction reduced task parallelism.
What compaction changes—and what it does not prove
Iceberg compaction rewrites selected small data files into larger files. Apache Iceberg describes its Spark rewriteDataFiles action as a way to combine small files, reducing manifest metadata overhead and runtime file-open costs (Apache Iceberg maintenance documentation).
Those mechanisms can help, especially when a query must plan around or open many small files. But “1,000 files became 6” is a file-count result, not a query-performance measurement. It does not establish that all files in a table were rewritten, that only six files remain in the table, or that a representative query ran faster. The count might refer to one partition or just the files selected by a rewrite.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallWhy performance can improve for one query and regress for another
A 2026 study by Jannic Cutura and Subash Prakash reports that compaction benefited metadata-heavy queries in its benchmark, while full-scan aggregations could become slower when compaction reduced task parallelism (Cutura and Prakash, 2026). This is a result from the authors’ benchmark setup, not a universal prediction for every Iceberg table or engine.
#1 Best Overall
- Metadata- or planning-heavy queries: Fewer files can mean less file metadata to process during planning, potentially reducing overhead.
- Selective queries: Results depend on whether the rewritten layout preserves useful pruning and whether the query still reads a similar amount of data.
- Broad scans and aggregations: Larger files may reduce the number of independent tasks available to process data in parallel. Whether that matters depends on the engine, cluster, and workload.
The study analyzed 2,376 simulated Iceberg tables and extracted 17 metadata features from manifest files. Its XGBoost model reported R² = 0.998 for predicting the continuous file-reduction ratio in the described simulation; cross-schema validation used 96 TPC-H tables and reported R² = 0.976. These figures describe the paper’s simulation and validation, not a measurement of a 1,000-to-6 rewrite or proof of a particular latency improvement.
How to verify a 1,000-to-6 result
To connect a file-count change to query performance, first establish exactly what the count represents. Iceberg’s Spark procedure reports rewritten and added data-file counts as well as rewritten bytes, which are distinct from the total number of files in a table (Spark procedures reference).
- Define the file count. Say whether 1,000 refers to all table files, a particular partition, or the files selected for rewriting. Identify whether 6 is the rewrite’s added-file count or the resulting table-wide count.
- Record the environment. Report the table size and layout, file-size distribution, query engine and exact version, storage environment, compaction strategy and options, and any relevant table properties.
- Use the same queries before and after. Include query text or a precise description of each query pattern. Test metadata-heavy and selective queries as well as broad scans or aggregations if those workloads matter to users.
- Control and disclose run conditions. State cache state, concurrency, and the number of repeated runs. Separate planning time from execution time where possible.
- Show what the engine did. Include bytes scanned, files read, or task counts alongside elapsed times so readers can see whether the physical work or parallelism changed.
Without those measurements, the defensible conclusion is limited: the rewrite reduced a reported file count, while its effect on query performance remains unknown.
Choosing a rewrite strategy and target size
The current Iceberg Spark procedure reference documents binpack as the default rewrite strategy and also supports sort with specified orderings, including Z-order and Hilbert. It documents a 512 MB default target output size when that value is inherited from the table property write.target-file-size-bytes. The same reference lists a minimum rewrite size of 75% of the target and a maximum of 180%.
Rank #3
These are version- and configuration-sensitive procedure details, not a universal prescription to target 512 MB. Check the deployed Iceberg version and table properties before relying on a default. The maintenance page also illustrates a Spark action using a 500 MB target and recommends inspecting the files metadata table to review data-file sizes and decide whether partitions need compaction (Apache Iceberg maintenance documentation).
When manifest maintenance is the more relevant fix
Data-file compaction and manifest rewriting address related but different costs. Data-file compaction changes the layout of data files. Iceberg’s separate rewrite_manifests maintenance path reorganizes manifest files, which help query planning and pruning. If write patterns do not align with query filters, manifest organization may be relevant to planning performance; compacting data files alone does not guarantee that every planning bottleneck is resolved (maintenance documentation; Spark procedures reference).
Other documented engines and version checks
Spark is not the only documented route. Iceberg’s Flink maintenance documentation describes a RewriteDataFiles action and shows an example with a 256 MiB target and 32 MiB minimum (Flink maintenance documentation). Confirm that the API and options are available in the Iceberg release you have deployed; nightly documentation may not match an older production version.
Free tools Windows power users keep installed
One-click scans. No signup required.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
