Free tools Windows power users keep installed
One-click scans. No signup required.
iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
No: “exactly-once delivery” does not mean your handler can only run once or that an external side effect can never happen twice. A broker can provide real duplicate-suppression or transactional guarantees inside a defined boundary. If your handler changes a separate database, charges a card, or calls an API, that effect is not automatically part of the broker’s transaction. Design for retries by making the business operation idempotent.
Does exactly-once delivery mean my handler runs only once?
No. Delivery, processing, and business effect are separate claims. A broker may suppress certain repeated sends or atomically coordinate consumed offsets with records produced back into that broker. Neither claim, by itself, proves that arbitrary application code ran once or that an external effect happened once.
For example, a consumer can charge a customer successfully, then crash before its acknowledgment or offset is durably committed. The broker sees unfinished work and delivers it again. If the handler charges without checking whether that business event was already applied, the second attempt can charge again.
The reverse order is risky too: if a consumer marks work complete before carrying out the business operation, a crash between those steps can leave the operation undone while the broker treats the message as finished. The core problem is coordinating two durable facts: “the business effect happened” and “the broker can advance past this message.”
Why can a consumer process the same message twice?
With separate systems, the consumer usually cannot commit the business change and broker acknowledgment as one atomic operation. If the business change succeeds but the acknowledgment is lost—or the consumer crashes before committing progress—the broker may retry. This is a normal failure mode, not proof that the broker malfunctioned.
Kafka’s design documentation describes the trade-off: committing progress before processing risks losing work, while processing before committing progress can lead to repeat processing. For external destinations, an atomic outcome generally requires cooperation from the destination system. Where both systems do not share a transaction, the handler needs a way to recognize and safely absorb a repeat.
What does “exactly once” mean in Kafka?
Kafka can provide exactly-once processing within a Kafka-to-Kafka flow: a consumer reads Kafka records, transforms them, and writes results back to Kafka while coordinating the consumed offsets and output records in a Kafka transaction. A downstream consumer should use read_committed isolation so it does not read records from aborted transactions.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problemsRank #2
That guarantee has a boundary. If a Kafka handler calls a payment API or updates a separate relational database, that external effect is not included in Kafka’s transaction merely because the Kafka offsets and output records are transactional. The destination must participate in a compatible atomic transaction, or the application must use an idempotent design.
Transactional consumer configuration
For the direct producer/consumer pattern described in Kafka’s design documentation, the documented settings include isolation.level=read_committed and enable.auto.commit=false. The pattern coordinates produced records and consumed offsets in a transaction; Kafka describes one producer instance per consumer instance as the simpler recommended approach. A producer’s transactional.id supports recovery and fencing of a previous instance.
These are version-sensitive implementation details: Kafka’s design documentation is published on a moving branch, so verify configuration and APIs against the release actually deployed. Do not treat the configuration as a way to include a separate database or HTTP service in Kafka’s transaction.
What the guarantee does not promise
Kafka’s idempotent producer uses sequence numbers to prevent duplicates within a producer session. Transactional IDs support transactional recovery across application sessions, but neither turns every external effect into a once-only operation. Kafka’s transaction proposal also notes that a consumer’s view is not necessarily an indivisible bundle of every record in a committed transaction: compaction, log-segment deletion, seeking, or reading only some participating partitions can affect what is observed.
Recommended Free Tools
How does Amazon SQS handle duplicates?
The delivery and deduplication behavior depends on the queue type and the operation being considered. These are bounded product guarantees, not universal guarantees about a consumer’s business effects.
| Queue or flow | Documented behavior | What remains outside that behavior |
|---|---|---|
| SQS standard queue | At-least-once delivery; a message may be delivered more than once. | A consumer’s database update or API call must tolerate a repeat. |
| SQS FIFO send deduplication | Repeated sends with the same deduplication ID are suppressed within a five-minute deduplication interval. | The interval does not prevent a repeated external effect after delivery, or after the interval has elapsed. |
| SNS FIFO to subscribed SQS FIFO | AWS describes an end-to-end guarantee subject to conditions including an existing queue that permits delivery, processing and deletion before the visibility timeout expires, no message filtering, and no network disruption that prevents acknowledgment; the deduplication interval is five minutes. | The stated conditions and scope matter; this is not an unconditional guarantee for arbitrary handlers or destinations. |
AWS explains that a standard queue stores copies on multiple servers. If a server is unavailable during a receive/delete sequence, a copy may remain and later be delivered again. That is why AWS advises making standard-queue consumers idempotent.
FIFO content-based deduplication has a subtle limit
SQS FIFO can use an explicit message deduplication ID or content-based deduplication. Content-based deduplication hashes the message body with SHA-256, but does not include message attributes. If two business messages have the same body while an attribute carries the distinction, content-based deduplication can treat them as duplicates. Use an explicit, stable deduplication ID when the business identity is not fully represented by the body.
The five-minute interval is specifically a broker-side send-deduplication window. It is not a recommended retention period for an application’s idempotency records, and it does not cover arbitrary replays or redrives outside that period.
How do I stop a retried message from charging a customer twice?
Give each business operation a stable identity, then make the durable business update conditional on that identity. A delivery-attempt ID is not enough: retries of the same business event must use the same key, while distinct business events must have distinct keys.
- Put a stable business event ID in the message. Choose an ID that survives redelivery and replay; do not derive it from a transient delivery attempt.
- In one database transaction, claim the ID and apply the business change. Insert the ID into an inbox or idempotency table protected by a uniqueness constraint, and perform the corresponding update in that same transaction.
- Handle a uniqueness conflict as an already-applied event. If the key already exists, do not repeat the business change. Return or record a result that is safe for the caller to retry as well.
- Commit the database transaction before acknowledging the broker. If the process crashes after the database commit but before acknowledgment, a redelivery encounters the existing key and becomes a no-op.
- Use an outbox for changes that must publish another event. Write the outgoing event to an outbox table in the same database transaction as the business update, then relay it asynchronously. Give the outgoing event its own stable ID and make its consumers retry-safe.
- Retain idempotency records for the real replay horizon. Keep them at least as long as messages can be replayed, redriven, or retried by clients; a broker’s short deduplication interval may be much shorter.
This pattern does not make delivery happen only once. It makes repeated delivery converge on one durable business outcome. It also keeps the business update and the idempotency record from disagreeing, because both commit or roll back together.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Which boundary should I name when I claim exactly-once behavior?
State the scope instead of using “exactly once” as an unqualified system-wide property. A useful description identifies the broker flow, what is being deduplicated or committed atomically, the destination, the duplicate identity, the retention window, and the failure assumptions.
- Scope: producer retry suppression, broker delivery, consume-transform-produce, or an external business effect.
- Atomic boundary: broker-only transaction, a transaction shared with the destination, or separate operations protected by application idempotency.
- Duplicate identity: producer sequence number, explicit deduplication ID, body hash, or stable business event key.
- Retention: the broker’s duplicate-suppression period and the application’s idempotency-record lifetime.
- Failure assumptions: process crash, lost acknowledgment, rebalance, visibility-timeout expiry, replay or redrive, and network disruption.
- Ordering: whether ordering is global, per partition or group, or limited to a product-specific message group.
A precise claim might say that a Kafka consumer transactionally writes output records and offsets within Kafka, with downstream readers using read_committed. A separate claim is needed for a database or API effect: for example, that the application applies a stable event key and business update in one database transaction. Those statements describe what is protected without implying that the entire end-to-end workflow is magically atomic.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
