To prevent a Spring Boot Kafka listener from losing messages, make sure its consumer offset advances only after the message has been processed successfully—or after a failure has been deliberately recovered, such as by publishing the record to a dead-letter topic (DLT). Keep auto-commit disabled, choose an acknowledgment mode that matches your listener, and let processing failures reach Spring Kafka’s error handler instead of swallowing them.
How Kafka offsets affect message loss
A consumer-group offset records the group’s position in a partition. If the group commits past a record before its business work is complete, a crash can leave that record unprocessed and prevent it from being delivered again to that group. If processing finishes but the offset is not committed before a crash, Kafka can deliver the record again. That favors avoiding loss, but means a handler or downstream operation may run more than once.
For most applications, duplicate delivery is safer than losing an unprocessed record. Design handlers and external writes to tolerate retries, and advance offsets only after successful processing or an intentional, durable recovery action.
Choose an acknowledgment mode that matches the listener
Spring Kafka disables the Kafka client’s auto-commit by default unless it is explicitly enabled. The listener container manages commits according to its acknowledgment mode; the documented default mode is BATCH. Set enable.auto.commit=false explicitly in production configuration so an inherited client setting cannot commit offsets independently of listener processing.
#1 Best Overall
| AckMode | When offsets are committed | When it fits |
|---|---|---|
RECORD |
After each record listener call completes successfully. | Use when each successful record should advance independently. |
BATCH |
After the records from a poll batch have been processed. | Use when replaying the batch after a failure is acceptable. |
MANUAL |
The listener acknowledges records; commits follow batch semantics. | Use when the listener must decide when to acknowledge, and batch-style commit behavior is acceptable. |
MANUAL_IMMEDIATE |
The container commits when the listener calls Acknowledgment.acknowledge() on the listener thread. |
Use when the listener intentionally controls acknowledgment and needs the commit at that point. |
Manual modes move responsibility into application code: acknowledge only after the work that justifies advancing the offset has succeeded. With MANUAL_IMMEDIATE, call acknowledge() on the listener thread. Do not use a manual mode simply to make failures disappear.
Let failures reach Spring Kafka’s error handler
Do not catch an exception and return normally just to keep the consumer moving. A normally returning listener can be treated as successful, depending on the acknowledgment mode and handler configuration, allowing the offset to advance even though the business operation failed. If the record has not been handled, let the exception propagate so the container’s CommonErrorHandler can apply the configured failure policy.
Bound retries and decide what happens next
DefaultErrorHandler supports a BackOff and a recoverer. Use a finite retry policy or another deliberate backoff policy rather than allowing a poison record to fail indefinitely. Classify exceptions that should not be retried, and decide whether exhausted failures should be recovered, sent to a DLT, or left for another operational response. Inline retries can hold up progress on the affected partition; retry topics or a DLT can isolate a failing record, but require a plan for ordering and later replay.
Use a DLT when failed records must be retained
A DeadLetterPublishingRecoverer can publish a record after retries are exhausted. With its default destination resolver, it sends the record to <originalTopic>-dlt on the original partition. The DLT therefore needs at least as many partitions as the source topic. Treat a DLT publish failure as a recovery failure, not a successful handling: monitor recoverer failures and ensure the failed record is not silently committed as though it were safely retained.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errorsRank #3
For manual acknowledgment configurations, DefaultErrorHandler.setCommitRecovered(true) is appropriate only when the recovered record has actually been dealt with—for example, intentionally published to a durable DLT. Committing a recovered record without durable recovery can turn a processing failure into permanent loss.
Use Kafka transactions for Kafka-only read-process-write flows
For a flow that consumes from Kafka, processes a record, and produces results back to Kafka, configure a KafkaAwareTransactionManager. Spring Kafka sends the consumed offsets to the Kafka transaction before commit. If the listener throws, the transaction rolls back and the consumer is repositioned so the rolled-back records can be fetched again. Spring describes this transactional read-process-write flow as exactly-once semantics; the individual read and process operations retain at-least-once characteristics.
Let exceptions that require rollback propagate. A custom error handler that returns normally can make the listener appear successful; when rollback is required, a transactional error handler must throw. Kafka transactions coordinate Kafka records and consumer positions, not arbitrary external work. A database write, HTTP request, or email can still succeed or fail independently of the Kafka transaction. For those boundaries, use idempotency keys, an outbox/inbox pattern, or a transaction manager that truly coordinates the external resource.
Match the design to failure isolation, ordering, and recovery
| Design choice | Trade-off to plan for |
|---|---|
| Commit after each record or batch | Later commits favor redelivery over loss, but a crash can cause duplicates. Choose whether independent record progress or replaying a poll batch is acceptable. |
| Inline retries | Keep retry behavior close to the listener, but a repeatedly failing record can delay later records on its partition. |
| Retry topics or a DLT | Can isolate a poison record from the original processing path, but require monitoring, inspection, and a safe way to re-drive corrected records. Recovery paths can also affect ordering. |
| Kafka transaction | Coordinates Kafka output and consumed offsets for a Kafka-only flow; it does not make non-Kafka side effects atomic. |
| External side effect | Use idempotency or an outbox/inbox or genuinely coordinated transaction so a retry does not duplicate or omit the external operation. |
If strict partition ordering matters, do not assume that retrying or moving a failed record elsewhere preserves the original order automatically. Choose whether to pause progress behind the failed record or isolate it, and make the recovery and replay process respect the ordering requirement.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Test the paths that can change delivery behavior
Exercise failure and recovery behavior in the application environment, not only the listener’s happy path. The relevant behavior is defined by Spring Kafka and Kafka configuration, but no single setup guarantees the same result across every external dependency.
Quick Recap
- Restart the application during processing and verify whether the record is replayed or its offset has advanced.
- Cause a rebalance and confirm that unfinished work is not treated as complete.
- Simulate a broker outage and verify that errors remain visible and recovery does not silently commit.
- Test deserialization failures, which can occur before the listener handles a record, against the configured error policy.
- Make DLT publication fail and verify the original record is not treated as durably recovered.
- Deliver the same record more than once and confirm that handlers and downstream writes are safe to repeat.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
