Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more

A clean restart only tells you that the process came back up; it does not prove the logical job ran once. A caller may have submitted the job twice, a queue may have redelivered it, or a workflow may have replayed a step whose external effect happened before its completion was saved. Trace the job across those boundaries, then make the side effect idempotent so a retry can safely return the result of the first attempt.

What does “ran twice” actually mean?

Separate the process from the work it performs. A restarted worker is a new process instance. The logical job is the unit of work the user or system intended to happen once. That job may produce several attempts, and an attempt may reach an external tool or API even if the worker never records or acknowledges success.

There are several distinct duplicate patterns: a producer submits the same request twice; a queue delivers one message more than once; a workflow starts two executions; an activity or step is retried; or one request reaches an external service, but its response is lost and the caller sends it again. These cases look similar in an agent’s logs but require different fixes. Start by finding the first boundary where one logical job becomes two attempts.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Why can a clean restart still lead to duplicate work?

A queue can redeliver a message

Queue consumers often need to tolerate duplicate delivery. With Amazon SQS and Lambda, a function error or timeout, a failed acknowledgement or deletion before the visibility timeout expires, or a transient duplicate from the queue can lead to the same message being processed again. AWS recommends checking the message ID and receive count; if identical content arrives under different message IDs, investigate whether the producer sent it more than once. See AWS’s SQS/Lambda guidance.

#1 Best Overall
Sale
CyberPower ST425 Standby UPS Battery Backup and Surge Protector
  • 425VA/260W Standby Uninterruptible Power Supply (UPS): Uses simulated sine wave output to provide battery backup power and to safeguard home office, home entertainment including computers, gaming consoles, and broadband routers
  • 8 NEMA 5-15R OUTLETS: Four battery backup & surge protected outlets; Four surge protected outlets; INPUT: NEMA 5-15P right angle, 45 degree offset plug with five foot power cord
  • ADDITIONAL FEATURES: LED status light indicates Power-On and Wiring Fault, transformer-spaced outlets
  • GREENPOWER UPS HIGH EFFICIENCY DESIGN: Reduces power consumption by utilizing a compact charger and power inverter to create an ultra-efficient backup power system for home and office use
  • 3-YEAR WARRANTY – INCLUDING THE BATTERY; 75K USD Connected Equipment Guarantee; UL SAFETY CERTIFIED: Product has been tested in a UL certified lab and listed with UL as meeting or exceeding safety standards

For the SQS/Lambda configuration covered by that guidance, AWS says the queue visibility timeout should be at least six times the function timeout plus the batch-window timeout. Treat that as integration-specific guidance, not a universal queue setting: check the current configuration and requirements for your actual queue and consumer.

An asynchronous function may retry

For Lambda asynchronous invocations, function errors are retried by default, and an event can be delivered more than once even if the function did not return an error because the underlying queue is eventually consistent. The retry behavior belongs to the service configuration, not to the restarted process alone. Check the current retry and event-handling settings for your deployment in AWS’s asynchronous invocation documentation.

Rank #2
Sale
CyberPower CP1500PFCLCD PFC Sinewave UPS Battery Backup and Surge Protector
  • 1500VA/1000W PFC Sinewave Uninterruptible Power Supply (UPS): Uses sine wave output to provide battery backup power for Active PFC & conventional power supplies; Safeguards computers, workstations, network devices, and telecom equipment
  • 12 NEMA 5-15R OUTLETS: 6 battery backup & surge protected outlets, 6 surge protected outlets; INPUT: NEMA 5-15P right angle, 45 degree offset plug with 5 foot power cord; 2 USB charge ports (1 Type-A, 1 Type-C) quickly charge phones and tablets
  • MULTIFUNCTION, COLOR LCD PANEL: Displays immediate, detailed information on battery and power conditions; Color display alerts users to potential issues before they can affect critical equipment and cause downtime; Screen tilts up to 22 degrees
  • AUTOMATIC VOLTAGE REGULATION (AVR): Corrects minor power fluctuations without switching to battery power; UL SAFETY CERTIFIED: Product has been tested in a UL certified lab and listed with UL as meeting or exceeding safety standards
  • 3-YEAR WARRANTY – INCLUDING THE BATTERY; $500,000 Connected Equipment Guarantee; FREE PowerPanel Management Software (Download)

A retry may start a new execution

If the caller starts a new workflow execution for every attempt and generates a different execution identifier each time, the workflow service may have no stable way to recognize that both requests represent the same logical job. AWS Durable Functions supports using an execution name as an idempotency key for starts; retries must reuse the name, and payload and state constraints apply. See AWS Lambda’s idempotency guidance.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

A side effect may succeed before its completion is recorded

Suppose an agent asks a service to create a record or make a payment. The service completes the action, but the worker crashes or loses the response before saving the result in its durable history. On replay, the workflow cannot safely assume that the action happened, so it may call the service again. AWS documents at-least-once step semantics by default and notes that “Replay and retry can each run the same operation more than once.” See AWS Durable Execution SDK: Idempotency and retries.

Rank #3
Sale
APC BX1500M UPS Battery Backup & Surge Protector for Computers, Electronics
  • 1500VA / 900W RELIABLE BACKUP POWER: The highest VA capacity available for home use; delivers short-term battery power to keep essential devices powered during blackouts, surges, and unexpected power interruptions
  • TEN PROTECTED OUTLETS: Power your entire setup with 5 battery backup outlets for essential devices, and 5 surge-only outlets for peripherals. Plus built-in coaxial and Ethernet surge protection for added peace of mind
  • AUTOMATIC VOLTAGE REGULATION (AVR): Corrects low voltage brownouts (88V+) and surges (+/-13%) without draining battery. Boosts or trims to stable 120V. Extends runtime for blackouts; Active PFC compatible for gaming PCs
  • REPLACEABLE BATTERY & ENERGY STAR UPS: User-replaceable battery (APCRBC124, sold separately) for zero-downtime swaps. ENERGY STAR certified for 92%+ efficiency, cutting energy costs vs standard UPS units
  • LCD DISPLAY PANEL: Features an intuitive LCD screen that displays real-time status information including battery charge level, estimated runtime, load capacity, and input voltage for easy monitoring of your power protection system

“Retry” can refer to different workflow failures

Workflow systems distinguish failures at different levels. Temporal automatically retries Workflow Tasks, while a failed Workflow Execution does not automatically retry unless a retry policy is configured. When an execution appears to have restarted, check which task or execution failed and what policy applied; the word “retry” alone does not identify the mechanism. See Temporal’s task documentation.

How do you find where the duplicate began?

  1. Assign a stable logical job ID. Record it at request intake, when publishing to a queue, on every worker attempt, at checkpoint reads and writes, and for each external tool call. Alongside it, log an attempt number and the relevant message ID, workflow ID and run ID, and idempotency key.
  2. Find the first boundary with two records. Check whether there were two incoming submissions, two queue deliveries, two workflow starts, two activity attempts, or just one attempt whose external effect occurred twice after an uncertain response. Message IDs and receive counts help distinguish a redelivery from separate producer sends.
  3. Compare processing and acknowledgement timing. For a queue, compare processing duration and any batch window with the visibility timeout. Confirm that successful messages were acknowledged or deleted, and that partial-batch failure reporting is configured as intended.
  4. Check what was durably saved before the restart. Determine whether the checkpoint or workflow-history write completed before the process stopped, and whether recovery resumed that execution or created a fresh one. A restart is not evidence that progress was persisted.
  5. Trace the external call and its response. Match the job ID and attempt to the tool or API request. Establish whether the service completed the action, whether the worker received the response, and whether local completion was recorded. This separates a repeated request from a repeated effect.
  6. Test the uncertain-outcome case safely. In a non-production environment, simulate interruption after an external action succeeds but before local completion is saved. The retry should return a stored result or reconcile safely, rather than repeat an irreversible effect.

How do you prevent the same side effect from happening twice?

Put the protection at the boundary that performs the consequential action, such as the tool or API call—not only in the agent’s prompt or in a process-local checkpoint. Use a stable idempotency key for the logical action, persist the key with the outcome, and make a repeated request with that key return the original result instead of performing the action again. AWS describes idempotency as an operation whose “effect remains the same regardless of how many times they run” in its Durable Execution SDK guidance.

Rank #4
Sale
CyberPower CP1500AVRLCD3 Intelligent LCD UPS Battery Backup
  • 1500VA/900W Intelligent LCD Uninterruptible Power Supply (UPS): Uses simulated sine wave technology to provide battery backup power to safeguard workstations, networking devices, and home entertainment equipment
  • 12 NEMA 5-15R OUTLETS: Six battery backup & surge protected outlets; six surge protected outlets; INPUT: NEMA 5-15P plug with 6-foot power cord; USB charge ports (1 Type-A, 1 Type-C) quickly charge mobile phones and tablets
  • MULTIFUNCTION, COLOR LCD PANEL: Displays immediate, detailed information on battery and power conditions; Color display alerts users to potential issues before they can affect critical equipment and cause downtime
  • AUTOMATIC VOLTAGE REGULATION (AVR): Corrects minor power fluctuations without switching to battery power; UL SAFETY CERTIFIED: Product has been tested in a UL certified lab and listed with UL as meeting or exceeding safety standards
  • 3-YEAR WARRANTY – INCLUDING THE BATTERY; 500,000 Connected Equipment Guarantee; FREE PowerPanel Personal Software (Download)

The key must represent the same intended action across retries. If each attempt generates a fresh key, the service will see each request as new. If a job has distinct side effects, give each logical action its own stable key so that deduplicating one action does not suppress another.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

When a third-party service does not accept idempotency keys or offer deduplication, a local checkpoint cannot guarantee exactly-once effects across that external boundary. Reconcile the service’s state before retrying, or define a compensating action for duplicates where that is safe. Checkpoints still help avoid repeating completed internal work, but they cannot establish whether a separate service completed an action whose response was lost. AWS’s Agentic AI Lens guidance on state management and checkpoint-based recovery addresses recovery through persisted state and checkpoints.

Best Value
Sale
CyberPower EC850LCD Ecologic UPS Battery Backup and Surge Protector
  • 12 NEMA 5-15R OUTLETS: Six battery backup & surge protected outlets; Six surge protected outlets (Three ECO controlled); INPUT: NEMA 5-15P right angle, 45 degree offset plug with five foot power cord
  • MULTIFUNCTION LCD PANEL: Displays immediate, detailed information on battery and power conditions
  • ECO MODE: When the UPS detects a computer is off or in sleep mode, it will automatically turn off power to computer peripherals connected to ECO mode outlets, reducing power usage and lowering energy costs
  • 3-YEAR WARRANTY – INCLUDING THE BATTERY; $100,000 Connected Equipment Guarantee and FREE PowerPanel Personal Edition Management Software (Download)
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

Which recovery design fits the workflow?

No single pattern removes the need to design side effects for retries. Choose based on workflow duration, failure modes, existing infrastructure, language and runtime, operational capacity, and how much recovery behavior you need to control.

Approach Often fits What to evaluate
Queue plus idempotent handler Event-driven jobs using existing queue infrastructure. Delivery guarantees, visibility and acknowledgement behavior, deduplication-store consistency, dead-letter handling, and the operational work of maintaining state storage.
Cloud state machine or orchestrator Workflows where explicit retries, branching, and state transitions are useful. Service-specific retry behavior, execution history and limits, integrations, and whether each external operation is independently idempotent.
Durable code-first workflow engine Long-running workflows that must recover across process or infrastructure failures. Activity idempotency, history and replay constraints, operational model, language support, and managed-service costs. Temporal documents its task and retry behavior; the OpenAI Agents SDK documentation describes durable runs spanning waits, retries, and process restarts.
Manual checkpointing Simpler jobs with a small number of clear progress boundaries. Define what each checkpoint means and persist completion with results atomically where practical. A checkpoint does not make an independent external service exactly-once.

What should you log for the next incident?

Make one logical job traceable from intake through its final side effect. Include the logical job ID, attempt number, message ID and receive count where relevant, workflow ID and run ID, checkpoint version or state transition, idempotency key, external request identifier, outcome, and timestamps for submission, processing, acknowledgement, and persistence. Avoid logging secrets or sensitive payloads; identifiers and transition outcomes are usually enough to establish where a duplicate began.

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.