What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
To control when a Kafka consumer records progress, set enable.auto.commit to false, finish processing the records you intend to acknowledge, then commit the next offset the application should consume. Use Java’s commitSync when the caller should wait for the result; use commitAsync when it should return without waiting and your application can handle failures through a callback.
What a committed offset means
A committed offset is the consumer group’s stored restart position. When a consumer starts or recovers after a rebalance, Kafka uses that position to determine where fetching resumes. A commit records progress in Kafka; it does not make a database write, HTTP request, or other external side effect atomic with that progress.
If offset n is the last record in a partition whose work is complete, commit n + 1. The committed value identifies the next record to consume, not the last one already processed. The Kafka 4.1 Java consumer API also recommends including leader-epoch metadata when available.
How to commit offsets manually
- Disable automatic commits. Set
enable.auto.commit=falsein the consumer configuration so the application, rather than a periodic background commit, controls the processing boundary. See the Kafka 4.2 consumer configuration reference for the setting. - Poll for records. Use the Java
KafkaConsumerto retrieve records and process them according to your application’s completion rules. - Track completed work by partition. For each partition, identify the next offset after the last record whose work is complete.
- Commit that position. Use
commitSyncorcommitAsync, choosing based on whether the caller should wait for the result.
With concurrent processing, track progress separately for each partition. Do not commit past an earlier unfinished record in a partition just because a later record finished first; doing so could move the recovery position beyond work that has not completed.
Choose between commitSync and commitAsync
| Method | Java API behavior | Practical tradeoff |
|---|---|---|
commitSync |
Waits until the commit succeeds, an unrecoverable error occurs, or the call times out. | The calling flow can make later decisions based on the result, but it waits for the commit. |
commitAsync |
Returns without waiting. Errors are delivered to a supplied callback; if no callback is supplied, errors are discarded. | Avoids waiting in the caller, but the application must decide how to observe and respond to failures. |
Use commitSync when proceeding depends on knowing whether the commit completed. Use commitAsync when avoiding the wait matters and you have a deliberate way to handle callback failures. The Kafka 4.1 API documentation describes ordering for successive asynchronous commits and specifies that earlier asynchronous commits complete before a subsequent synchronous commit returns. That ordering does not remove the need to handle an asynchronous failure when correctness depends on the commit.
Manual commits versus periodic auto commits
When enable.auto.commit is enabled, Kafka commits offsets periodically in the background. Kafka 4.2 documents auto.commit.interval.ms with a default of 5,000 milliseconds (5 seconds); that is a configuration default, not a recommendation or a guarantee that external processing has finished before an offset is committed.
Manual commits are useful when the recovery point should follow a specific completion event, such as finishing a batch or waiting for a downstream operation. Periodic auto commits require less application code, but the interval alone does not express that exact processing boundary.
Recommended Free Tools
What can go wrong if the commit and processing diverge?
- The commit moves ahead of unfinished work: after a restart or rebalance, Kafka may resume beyond that work, so the application can skip it.
- Work finishes but the commit does not take effect: the stored position may remain behind the completed work, so the application can process those records again after recovery.
These outcomes follow from the relationship between completed work and the stored restart position. A Kafka offset commit does not provide exactly-once effects for external systems; design downstream operations to account for the possibility of repeated work where necessary.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Check the API version before using examples
The Java commit behavior and next-offset guidance here are documented in Apache Kafka 4.1, while the automatic-commit settings and 5-second default are documented in Kafka 4.2. Method signatures and configuration defaults can differ between client releases, so verify the API and configuration for the Kafka client version your application actually uses. The Kafka 3.5 consumer configuration reference is available for comparison with that earlier release.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
