To preserve event order within a Kafka session, produce every event for that session with the same stable key so Kafka assigns the records to the same partition. Then process that partition sequentially wherever application-side effects must follow the log order. Kafka guarantees order within a partition—not a single total order across a topic’s partitions.
What “ordered per session” means in Kafka
Kafka topics are divided into partitions. Apache Kafka’s protocol documentation describes topic partitions as ordered commit logs. A consumer reads records in partition order, so the partition—not the topic as a whole—is the ordering boundary. Apache Kafka protocol documentation
For session ordering, define a stable identifier that groups all records belonging to the same session, such as a session ID. Use it as the record key consistently across producers. Kafka’s producer assigns records to partitions; semantic partitioning routes records with a key together so they can be processed with local state while retaining partition order.
Sessions routed to different partitions can be processed independently. Records in separate partitions have no guaranteed relative order. If the requirement is a single order across every event, keyed partitioning alone is insufficient; the design needs a single serialization point or another mechanism that establishes that global order.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
#1 Best Overall
Choose a partitioning design
Use a stable session key
Set each Kafka record’s key to the session identifier, and make sure every producer uses the same identifier format and partitioning scheme. Avoid keys that change during a session or are inconsistently normalized: records that should share an ordering lane may otherwise be routed separately.
Balance ordering scope with parallelism
A partition is a sequential lane. A consumer group can distribute partitions among its members, allowing distinct sessions on different partitions to be processed in parallel. Work within one partition still needs sequential handling when database writes or other side effects must preserve record order. Adding partitions can provide more lanes, but it does not make records within a single lane parallel without risking reordered effects.
Rank #2
Plan partitioning changes as migrations
Kafka’s within-partition ordering guarantee does not establish that a session’s records remain in one uninterrupted ordered stream if the topic’s partition count or partitioning scheme changes. Plan such changes explicitly: account for records already in the old assignment, how producers switch, and how consumers avoid applying newer events before older ones. A key is not a guarantee of global ordering across a reassignment.
Produce keyed records with the Go client
Confluent’s confluent-kafka-go is a Go wrapper around librdkafka. Its repository documents a producer workflow using Produce and asynchronous delivery reports. Check the API and configuration against the version pinned in your project; repository documentation on the mutable master branch may not match that version. confluent-kafka-go repository
Rank #3
- Choose the key. Use the session ID as the record key for every event in that session.
- Produce the event. Call
Producewith the key and payload. Production is asynchronous, so a successful call means the request was accepted for processing by the client, not that the broker has confirmed delivery. - Handle delivery reports. Check each message’s delivery result and error. Track outstanding messages if the application needs to know which records succeeded or failed.
- Drain on shutdown. Before closing the producer, wait for outstanding delivery reports or call
Flushwith a timeout. Handle any messages that remain undelivered according to the application’s retry or recovery policy.
Consume and apply effects sequentially
A consumer group assigns partitions among its members; it does not impose a total order across those partitions. If downstream effects must reflect Kafka’s order, ensure that a given partition’s records are completed sequentially. Avoid dispatching records from one partition to concurrent workers unless you also enforce completion and commit behavior that prevents later records from taking effect first.
During shutdown, finish in-flight work or safely abandon it before committing offsets. Committing past work that has not completed can cause the application to resume after records whose effects were never applied. The precise shutdown policy depends on the application and should be explicit in its implementation.
Choose delivery semantics for the failure you need to handle
| Approach | What it addresses | Important boundary |
|---|---|---|
| Keyed production and sequential partition processing | Preserves the order in which a session’s records appear in its partition when the application also applies their effects sequentially. | Does not establish a total order across partitions or prevent every duplicate during retries. |
| Idempotent producer | Addresses duplicate log entries caused by producer retries within Kafka’s producer semantics. | Does not make external side effects atomic with Kafka. |
| Kafka transactions | Can atomically write Kafka output records and advance consumed offsets for a consume-transform-produce workflow. | Requires transaction lifecycle and error handling; does not replace choosing the right key or make arbitrary external-system writes atomic. |
Kafka’s design documentation describes transactions for coordinating consumed offsets and produced records in a consume-transform-produce workflow. Apache Kafka design documentation: message delivery semantics
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Use transactions when processing Kafka input into Kafka output
For a transactional consume-transform-produce flow with confluent-kafka-go, the producer uses a transactional.id. The consumer must disable automatic offset commits. The application initializes and begins a transaction, produces output, includes the next offsets to consume along with consumer group metadata, then commits. If processing fails, abort the transaction and retry or stop according to the error class and application policy. Consumers that should not see aborted output need transaction-aware isolation using read_committed. Verify exact API calls and settings against the client version in use. confluent-kafka-go repository
Best Value
Do not describe ordinary keyed processing as “exactly once.” Kafka transactions can atomically couple Kafka output with Kafka input offsets within Kafka. A database write or other external side effect needs its own coordination or idempotency design; the Kafka transaction mechanism does not establish atomicity with arbitrary external systems.
Quick Recap
Decide whether this design fits
- Choose keyed partitions when ordering is required per session or entity and separate sessions should be able to progress independently.
- Choose a single serialization point or another global-order mechanism when every record must have a defined order relative to every other record.
- Add Kafka transactions when Kafka input offsets and Kafka output records must commit atomically; do not add them as a substitute for session keys or sequential side-effect processing.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
