Free tools Windows power users keep installed
One-click scans. No signup required.
To preserve order, keep every record for a session or entity on the same Kafka partition and do not commit the consumer offset past a failed record until it has been processed. This blocks later records in that partition while the failure is unresolved. A retry topic can let the source partition continue, but without per-key coordination, later records may overtake the retried message.
What “preserve order” means in Kafka
Kafka guarantees record order within a partition, not across every partition in a topic. If a session’s records must be handled in sequence, route them to the same partition—commonly by using a stable key such as a session or entity ID. The key must remain consistent for the records whose order matters. See Apache Kafka’s Kafka 4.0 design documentation.
This gives you a clear ordering boundary: a consumer can process one partition’s records in offset order, but Kafka does not impose a shared sequence across different partitions. If the requirement is broader than one partition or key, the application needs additional coordination.
Choose how a failure affects later records
The core trade-off is whether later records for the affected partition are allowed to proceed while an earlier record is failing. The right choice depends on how strict the ordering requirement is and how long the failure may last.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →| Approach | Ordering effect | Progress and trade-off |
|---|---|---|
| Retry in place without advancing the committed offset | Preserves partition order because later records do not pass the failed record. | The affected partition waits; other partitions can continue independently. |
| Send the failure to a retry topic and continue the source partition | Does not by itself preserve the original per-key sequence; later source records may be processed first. | Allows the source partition to move on, but requires per-key coordination if strict order remains necessary. |
The retry-topic consequence follows from Kafka’s partition and offset model; a retry topic is a separate scheduling path, not an ordering guarantee. Kafka’s documentation explains consumer positions and replay in its distribution documentation.
Retry in place when strict sequence takes priority
- Keep the consumer position behind the failure. Do not commit an offset beyond the failed record while later records in that partition must wait.
- Retry the record according to your application’s attempt and delay policy. Kafka provides the partition ordering and consumer-position mechanics; your processing logic determines the retry schedule.
- Advance only after successful processing. Once the failed record succeeds, process subsequent records in the partition in offset order and commit an appropriate position.
Because committed offsets determine where a group resumes after a restart, committing past an unprocessed failure can cause the consumer to resume after it rather than replay it. Consumers can also rewind to re-consume records. These behaviors are described in the Kafka 4.0 design documentation.
This approach can stall all later records in the same partition for as long as the failure remains unresolved. It does not stop consumers of other partitions from making progress.
Use a retry topic only with ordering coordination
A retry topic is useful when you want to separate delayed work from the source partition, but moving a failed record there changes the scheduling path. If the source consumer commits past that record and processes later records for the same key, those later records can finish first.
For strict per-key order with retry topics, coordinate processing so a key cannot advance beyond its outstanding failure. The exact coordination design depends on the application; Kafka’s partition and offset guarantees alone do not provide it. If later records may safely proceed, a retry topic can improve progress, but that is an explicit relaxation of the sequence requirement.
Prevent producer retries from reordering records
Consumer retry handling is only one part of the problem. On the producer side, retries can reorder records when idempotence is disabled and more than one request is in flight. Kafka 4.0 documents producer idempotence as enabled by default when configuration does not conflict with it. Idempotence requires acks=all, retries greater than zero, and max.in.flight.requests.per.connection of at most 5. Consult the exact Kafka 4.0 producer configuration for your client and version.
If idempotence is disabled, limiting max.in.flight.requests.per.connection to 1 removes the concurrent-request ordering risk associated with retries, though it can reduce throughput. Idempotence protects the producer-to-broker retry path; it does not make consumer business logic execute once, or coordinate the order of records spread across source and retry topics.
Use transactions for Kafka-to-Kafka processing
For consume-transform-produce workflows that read and write Kafka, transactions can atomically commit output records together with the consumed offsets. This avoids committing the input position separately from the Kafka output in a way that leaves those Kafka-side actions inconsistent. Downstream consumers that must not see aborted transactional output should use read_committed. Details are in Kafka’s design documentation and consumer configuration reference.
Transactions do not automatically include writes to an external database or API. Those side effects need their own coordination or duplicate-handling strategy; Kafka’s transaction does not make the outside system part of the same atomic operation.
In read_committed mode, the consumer returns only committed transactional records. It can also hold later records until an earlier open transaction reaches a decision, stopping at the last stable offset, as described in the Kafka 4.0 consumer configuration reference.
Quick Recap
Checklist for choosing a retry design
- Define the ordering scope: partition, session/key, or a broader application-level sequence.
- Decide whether a failure may block progress: in-place retry preserves sequence but stalls that partition; independent retry scheduling needs coordination if later records must not overtake.
- Set a retry delay and attempt policy: determine how long a record may hold up the partition and what happens when attempts are exhausted.
- Account for duplicate effects: producer idempotence addresses producer retries, not whether business processing or external side effects occur once.
- Identify the systems involved: Kafka transactions cover Kafka output and consumed offsets, not external database or API writes.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
