iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Services pass work through a message broker so a sender can hand off a task without waiting for the receiver to be online. A broker can buffer messages, but it cannot by itself guarantee that every business effect happens once: services must also handle retries, duplicates, failures, and ordering deliberately.
The food-delivery flow below is a teaching example, not a description of Swiggy’s exact payment or order-processing implementation. Swiggy Engineering has documented using a Kafka-based cloud messaging platform for event streaming and service orchestration and choreography; its incident account does not verify every step in this example.
How does a message broker let services work independently?
Without a broker, Service A might call Service B directly and wait for a response. If B is unavailable, A must decide whether to fail the customer’s request, keep retrying, or hold work in its own system. A broker introduces a durable handoff point: A publishes a message, and B can process it later.
For a food-delivery order, imagine an order service recording an order and publishing an event. Separate services could use that event to begin payment, restaurant, delivery, or customer-notification work. These are illustrative responsibilities, not claims about Swiggy’s internal implementation.
#1 Best Overall
Queue: one task for a worker group
A queue is useful when a task should be handled by one worker from a competing-consumer group. The broker holds work while workers are busy or temporarily offline. Under normal operation, one worker handles a given delivery of the task; a failure and retry can still cause the work to be delivered again.
Pub-sub: one event, independent subscribers
With publish-subscribe, a service publishes an event to a topic and independent subscribers can react at their own pace. For example, payment and restaurant workflows could each subscribe to an order-created event. Each subscriber tracks its own progress, so a slow subscriber need not hold up every other subscriber. Which subscribers receive an event depends on topic, subscription, and filtering configuration; fan-out is not automatic to every conceivable service.
| Shape | Who receives the work? | Progress and coupling |
|---|---|---|
| Queue with competing consumers | One worker in the group handles a task at a time under normal operation; redelivery is possible after failure. | Workers share the group’s work. The sender can hand off without waiting for a particular worker to be available. |
| Pub-sub topic | Configured independent subscribers can each receive and process an event. | Each subscriber advances at its own pace and maintains its own progress, subject to the system’s subscription and retention settings. |
What does “without losing a message” actually guarantee?
Message delivery semantics describe what can happen between a producer, broker, and consumer. They do not automatically promise that an external business action—such as charging a payment method or updating a separate database—will happen exactly once.
Free tools Windows power users keep installed
One-click scans. No signup required.
| Delivery mode | Loss and duplicate risk | What the guarantee covers |
|---|---|---|
| At-most-once | A message may be lost; the chosen path avoids redelivery. | Delivery attempt semantics, not successful completion of every downstream business effect. |
| At-least-once | Redelivery is possible, so duplicates must be expected. | Under the system’s stated durability assumptions, messages are not lost, but a consumer may receive a message more than once. See Apache Kafka’s delivery-semantics documentation and Google Cloud’s Pub/Sub guidance. |
| Exactly-once processing | Duplicate processing can be prevented within a defined scope, but the scope matters. | Kafka documents transactional exactly-once processing for Kafka input/output flows. An external destination generally must cooperate to extend that guarantee to its own writes or side effects. See Apache Kafka’s design documentation. |
An acknowledgement is the consumer’s signal that it has advanced or completed work. If a consumer acknowledges before its business effect is safely recorded, then crashes, the broker may consider the message finished even though the application did not complete the work. Conversely, if the effect succeeds but the acknowledgement does not reach the broker, redelivery can happen. This is why reliable systems plan for both lost progress and repeated work rather than assuming the broker can eliminate either.
Why can a database update and a message publish get out of sync?
A common failure is a dual write: a service commits a business change to its database and separately publishes an event. If the database commit succeeds but the publish fails, the business state changes without the event reaching downstream services. Reversing the order creates the opposite risk: an event may be published for a database change that later fails.
Use an outbox to make the handoff recoverable
In a transactional outbox pattern, the service writes the business change and an event record to an outbox table in the same database transaction. A separate relay reads pending rows and publishes them. If the transaction rolls back, neither the business update nor its event record is committed. The relay can resume after an outage, but a crash around publication can still lead it to publish an event twice. AWS describes the pattern and its duplicate and ordering considerations in its transactional outbox guidance.
Rank #3
Consumers should therefore make repeated handling safe. A typical design uses a stable event or operation ID, records that ID as processed, and commits that record atomically with the business effect where possible. If the same ID arrives again, the consumer can recognize that the effect has already been applied. This prevents a retry from repeating an operation such as creating a second shipment or applying the same order update twice. The exact mechanism depends on the destination system and operation; an outbox alone does not remove duplicates.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
What happens when a receiver is down or a message keeps failing?
If a receiver is temporarily unavailable, a broker can retain work for later delivery when its persistence and retention settings support it. When the receiver returns, it can resume from its saved progress. If a consumer fails after doing the work but before acknowledging it, the message may be delivered again; if it fails before doing the work, the retry can allow the task to complete.
Retries are not free. They consume capacity, can delay healthy work, and may multiply traffic during an outage. A failure response can also be ambiguous: the receiver may have completed an operation even though the sender never received confirmation. Retrying that operation without duplicate protection can repeat its effect.
Move repeatedly failing messages out of the normal path
A dead-letter or unprocessed-message queue gives operators a place to inspect messages that have exhausted their retry policy or cannot be processed. It keeps a malformed or otherwise problematic message from cycling indefinitely through ordinary work, but it does not fix the underlying problem. The message may need diagnosis, correction, and a controlled replay. Google Cloud describes routing failed messages to another topic for review or reprocessing in its event-driven architecture guidance.
Does a broker preserve message order?
Not necessarily at a global level. Parallel workers, topic partitions, multiple producers, retries, and independent subscribers can all affect the order in which work is observed or completed. Some systems preserve order only within a defined partition, key, or subscription scope; a topic should not be assumed to impose one universal order across all messages.
Choose the ordering scope from the business rule. If events for a particular order must be applied sequentially, use a stable key or partitioning strategy that keeps that order’s events together, and ensure consumers do not apply them out of sequence. Unrelated orders can often proceed in parallel. Stronger ordering constraints can reduce concurrency, so the design should protect only the order the domain requires. Kafka documents partitions and consumer offsets in its design documentation; Google Cloud also covers ordering and failed-message handling in its Pub/Sub guidance.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What does a real messaging outage reveal?
Swiggy Engineering’s 2022 incident account, “#BehindTheBug — Kafka Under The Water,” describes an outage in its Kafka-based cloud messaging platform. The post says a few partitions of one cluster were offline for four minutes and that the event affected 18 internal services. The cluster served more than 50 services in the order-fulfilment journey. These are figures from that company’s specific 2022 account, not general Kafka behavior or a statement about every current Swiggy service.
Swiggy reported that insufficient in-sync replicas led to NOT_ENOUGH_REPLICAS errors. Some Java producer clients then repeatedly received NOT_LEADER_FOR_PARTITION because a library bug involving a NullPointerException prevented metadata from updating. The post says retry behavior coincided with a 100X increase in overall cluster ingress and that clients were throttled. Swiggy’s reported response included reducing ordering-service coverage in the top 20 cities for nearly an hour and restarting impacted services one by one. Read the incident account at Swiggy Bytes.
The practical lesson is that a broker’s durability and retry settings are only part of resilience. Client behavior, capacity limits, visibility into lag and errors, and a workable incident response all matter. A retry loop can amplify an outage when clients cannot recover their view of broker state.
Quick Recap
Reliability checklist for service-to-service messaging
- Persist the business change and the event together before publishing, using an outbox when the database and broker cannot share a transaction.
- Acknowledge only after the consumer has safely recorded the effect it is responsible for.
- Retry transient failures with backoff and a cap rather than retrying indefinitely at full speed.
- Make handlers idempotent, using a stable event or operation ID to detect repeated work.
- Route exhausted failures to a dead-letter or unprocessed-message path with an inspection and controlled-replay process.
- Monitor consumer lag, retry volume, and dead-letter depth so stalled or repeatedly failing work is visible.
- Define the required ordering key and scope explicitly; allow unrelated work to proceed in parallel when the business rules permit it.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
