Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
A task stopping while an Amazon ECS GPU instance is repaired is often expected: ECS can drain the impaired host, stop its service tasks, and place replacements on new capacity. The stop itself does not prove that the replacement GPU is defective. To find the actual cause, correlate three records in order: the task’s stoppedReason and stopCode, recent ECS service events, and the container instance’s health details. Then verify GPU capacity, task-definition settings, agent connectivity, and deployment limits on the replacement host.
First decide which failure you are investigating
GPU incidents look similar from the console, but the remedy depends on where the task failed in its lifecycle. Classify the event before changing a deployment setting or replacing hardware.
| Observed stage | Most useful evidence | What it usually means |
|---|---|---|
| Task stopped while the original instance was being drained | Container instance status changed to DRAINING; service events show replacement activity; task has a normal stop record |
Expected consequence of host repair. The service should place a replacement if eligible capacity and deployment limits permit it. |
Task remains PENDING or never gets placed |
Service events and placement failures such as RESOURCE:GPU, AGENT, or a constraint/attribute error |
The scheduler cannot find an eligible, connected host with the requested resources and attributes. |
Task reaches RUNNING and then exits |
Container exit details, application logs, task timestamps, and GPU restart/XID metrics | A container or application failure, a transient GPU fault, or a compatibility problem; not automatically a failed physical GPU. |
| Replacement host is registered but unhealthy | overallStatus, individual health checks, and statusReason from container-instance health details |
The replacement may still be initializing, may lack a usable GPU, or may have another host or agent problem. |
Understand what ECS GPU auto-repair actually does
On ECS Managed Instances with GPU hardware, Amazon ECS uses NVIDIA Data Center GPU Manager (DCGM) to monitor GPU health. AWS states that a critical DCGM failure can mark the container instance impaired. When GPU auto-repair is enabled, ECS follows a start-before-stop sequence:
- The impaired instance is marked
DRAINING, preventing new task placement. - ECS provisions replacement capacity.
- Running tasks are allowed to stop gracefully, subject to the task stop timeout and the service’s deployment behavior.
- After the drain period, ECS terminates the impaired host.
Repair is deliberately rate-limited. No more than 20% of a capacity provider’s instances can be drained by GPU auto-repair at one time. If the capacity provider has fewer than nine instances, ECS drains at most one instance at a time. Therefore, a service task stopping during this sequence can be normal, while a replacement that remains pending requires separate diagnosis.
#1 Best Overall
- Powerful Turbo Fan:CAGIWIRU Electric Air Duster have reached speed up to 130000RPM,Can efficient remove dust and debris.With LED light and Three Adjustable Speed to meet with different cleaning tasks.
- With Attached USB C Fasting Charging cable.Charing 3 hours,Enjoy up to 240 minutes of use on the lowest setting, For third speed can use for 25mins.with four charging options to meet with your needs.
- Wide Application: attached 5 different nozzles,making it suitable for a variety of scenes, such as car,pet, pc, keyboards, and other electronic devices. It also serves for office and home clean duster.
- Handy Design & Portable:Cordless air duster electric nozzle is Light weight and small size, design and comfortable to hold and space saving, convenient to carry around. You can put it in your room or inside the car without taking up space.
- Economical &Reusable: Built durable materials with a long-lasting battery,our CAGIWIRU air duster it's a sustainable alternative to disposable air cans.
This behavior applies to ECS Managed Instances. An ECS cluster backed by self-managed EC2 capacity does not automatically gain the same managed repair sequence merely because its hosts have NVIDIA GPUs. For self-managed hosts, inspect your own instance, Auto Scaling, health-check, and replacement workflow.
Preserve an incident timeline before changing anything
Record enough identifiers to correlate control-plane events with host and application logs. Keep these values in the incident record:
- Region and cluster name
- Service name, or the fact that the workload was a standalone
RunTasktask - Exact task ARN or task ID and task-definition revision
- Original and replacement container-instance ARNs and EC2 instance IDs
- Task start and stop timestamps
stoppedReason,stopCode, container exit codes, and container reasons- ECS service events around the repair window
- Original and replacement instance status, health-check details, and
statusReason - ECS agent version, GPU-optimized AMI version, instance family, requested GPU count, CPU, memory, and placement constraints
Do not overwrite this evidence by repeatedly stopping and relaunching tasks. A second launch can hide the first placement error and make the timeline harder to interpret.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Step 1: inspect the stopped task
Start with the task that stopped, not with the replacement host. The stopped-task record tells you whether ECS stopped it as part of lifecycle management or whether a container failed.
aws ecs describe-tasks
--region <region>
--cluster <cluster-name>
--tasks <task-arn-or-id>
--query 'tasks[0].{lastStatus:lastStatus,desiredStatus:desiredStatus,stoppedReason:stoppedReason,stopCode:stopCode,stoppedAt:stoppedAt,startedAt:startedAt,containerInstanceArn:containerInstanceArn,containers:containers[*].{name:name,lastStatus:lastStatus,exitCode:exitCode,reason:reason}}'Save the complete JSON as well as the summarized output. The query is only a convenient view; fields omitted from it may matter later.
How to read the result
- A task that stopped during a documented drain window, with no container-level failure, is consistent with host replacement.
- A task with a container exit code or container
reasonneeds application and runtime investigation in addition to ECS events. - A task that never reached
RUNNINGshould be investigated as a placement or task-start problem, even if the prior host had a GPU impairment. - Do not treat a generic repeated-start message as the root cause. Use the individual failed task records to find the concrete reason.
Step 2: read ECS service events around the repair
Service events reveal whether the scheduler is draining, repeatedly failing placement, or repeatedly stopping newly started tasks.
aws ecs describe-services
--region <region>
--cluster <cluster-name>
--services <service-name>
--query 'services[0].events[0:30]'Capture the event timestamps and messages before and after the original instance entered DRAINING. A placement warning is a direction to inspect task and instance evidence, not proof of a particular cause. For example, a message about insufficient resources should be matched to the requested GPU count and the replacement hosts’ actual resources.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Repair Windows errors before they cause bigger problems3Scan for outdated or missing drivers - takes under a minuteRank #2
- Screen Cleaner is Your Screen’s New Bestfriend! - EVEO is proud to present a screen cleaner that’s perfectly suited for your TV. Meet your new screen cleaner, computer screen cleaner, laptop cleaner, iPad cleaner, Macbook,tv cleaner or any other electronic device. Our screen cleaner spray and wipe along with the included plush microfiber cloth, easily removes dust, fingerprints & other smudges on the screen’s surface. Clean effectively for a streak-free screen
- Gentle on your Screens - Our screen cleaner spray and wipes is compatible to be used even for the most sensitive LCD, LED, CRT, and OLED screens in proper use. Be at ease knowing that this product is compatible with all the major brands: LG, SONY, Samsung, Sharp, iPad, iPhone, Android screens, Kindle, PC monitors - we’ve got you covered! Can be used as LCD screen cleaner, tv cleaner, smart tv screen cleaner, and more
- Premium Super Soft Plush Microfiber Cloth - Included in this efficient screen cleaning kit. This screen cleaner spray and wipe has our signature plush microfiber cloth specially designed to comply with screens and monitors and leave them streak-free. Has been shown to effectively absorbs stubborn stains, zero streaks, and no fiber shedding. Always use this together with our screen cleaner for the best results
- Easy-to-Use Screen Cleaner Kit + microfiber cleaning cloth - The EVEO Screen Cleaning Kit will easily be your new favorite tool when it comes to taking care of the things you love most, quick and easy solution for a streak-free screen. Simply screw on the sprayer gun on the screen cleaner bottle. Spray the cleaning liquid onto the microfiber cloth, then wipe the screen until desired cleanliness & shine is achieved.
- Quality Screen Care Kit Worth Purchasing - Enjoy a convenient Screen Cleaner anytime, anywhere you need it. Your satisfaction is our top priority. We stand behind the quality of our products and that’s why, you shouldn’t be worrying at all. We manufacture the Best Screen Cleaner. and can be used for laptop cleaning kit, laptop screen cleaner, tv screen cleaner for smart tv. MacBook screen cleaner, monitor cleaner or screen cleaner spray for electronic devices with microfiber cleaning cloth
Patterns that matter
- Drain followed by successful replacement: the stopped task was probably an expected repair consequence.
- Repeated placement failures: inspect GPU, CPU, memory, attributes, constraints, and agent connectivity on every eligible host.
- Tasks start and repeatedly stop: compare container exit details and application logs, then inspect GPU restart/XID metrics.
- No replacement activity for a service: check desired count, deployment state, capacity-provider capacity, and deployment percentage limits.
Step 3: verify original and replacement instance health
Use container-instance health details for both the impaired host and its replacement. ECS health monitoring applies automated checks to container instances running agent version 1.57.0 or later. AWS documents the CLI workflow with AWS CLI version 1.22.3 or later, or AWS CLI version 2.3.6 or later.
aws ecs describe-container-instances
--region <region>
--cluster <cluster-name>
--container-instances <container-instance-arn>
--include CONTAINER_INSTANCE_HEALTHInspect the returned overall status, each health-check type, and statusReason. The overall status represents the most severe individual check. Documented values include OK, IMPAIRED, INSUFFICIENT_DATA, and INITIALIZING.
Interpret GPU-specific health evidence
For ECS Managed Instances, the ACCELERATED_COMPUTE check covers GPU health. When ECS reports an impaired GPU, its statusReason contains an NVIDIA XID identifier in the form XID_<number>. Preserve the exact code and timestamp; do not generalize one XID into a diagnosis that the evidence does not support.
INITIALIZING and INSUFFICIENT_DATA are not the same as a confirmed GPU failure. Allow health data to populate, then correlate the status with task placement and service events. A newly registered replacement can be temporarily unsuitable for scheduling while checks are still initializing.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Confirm the lifecycle transition
- Was the original instance actually marked
DRAINING? - Did a replacement container instance register in the same cluster and capacity provider?
- Is the replacement
ACTIVE, agent-connected, and health-checked? - Does its health output show the required accelerated-compute status and no blocking reason?
Step 4: separate placement failures from runtime failures
RESOURCE:GPU: no eligible GPU capacity
A documented RESOURCE:GPU placement failure means the number of requested GPUs is unavailable on eligible container instances. Check the task definition’s GPU count, the replacement hosts’ free GPU resources, instance-family eligibility, and any placement constraints. A cluster can have spare CPU and memory while still having no schedulable GPU.
If the replacement capacity has not registered, is still initializing, or has fewer free GPUs than the task requests, the service remains pending until another eligible host is available. Do not “fix” this by reducing the task’s GPU request unless the workload genuinely supports that configuration.
AGENT: the target agent is disconnected
An AGENT placement failure indicates that the target container instance has a disconnected ECS agent. Verify the replacement’s agent connectivity and status before investigating application code. A disconnected agent cannot reliably accept a GPU task even when the underlying EC2 instance is running.
Rank #3
- 【Large size】：Measuring 800 x 300 mm/ 31. 5×12 inches, which is wider and taller than others. Large mouse pad can perfectly fit your keyboard and mouse of any size, and there is also ample space for peripherals such as phones, tablets, etc.
- 【LED backlight mouse pad】：Blade Hawks RGB mouse pad provides 7 static modes and 7 dynamic modes will give you the options you are looking for. Power-off memory function remembers the last lighting mode you selected. The super glow fiber Chroma will give you a colorful gaming setup that you want.
- 【Anti-slip rubber base】：Made from Natural rubber, this gaming mousepad will firmly grip your desktop, allowing you to enjoy the ultimate precision in the games you are most passionate about. It also comes with a texture that enhances this ability even further.
- 【Smooth and waterproof surface】: Micro-textured cloth surface, for extremely precise mouse control and a smooth movement. When liquid splashes on the gaming mouse pad, it forms water droplets and slides down. Will not delay your work or gameplay.
- 【Plug and play】: This mouse and keyboard pad is powered by USB, plug and play, no driver required. One button control light modes. Press one time to change the lighting mode, press twice to change the brightness, press and hold about 3 seconds turn ON/OFF.
Missing attributes or constraints
Inspect required instance attributes, capacity-provider selection, Availability Zone or host constraints, and any custom placement rules. A replacement instance may be healthy but ineligible because it is the wrong instance family, lacks a required attribute, or is excluded by a constraint.
Step 5: verify GPU task configuration and compatibility
Agent GPU support and task resources
Confirm that the ECS agent configuration includes ECS_ENABLE_GPU_SUPPORT=true. Check the task-definition revision to ensure its GPU resource requirement matches the workload’s intended count. ECS assigns the NVIDIA container runtime to containers that request GPU resources and sets NVIDIA_VISIBLE_DEVICES to the device IDs assigned to each container.
Compare the task definition used before and after repair. A new revision, capacity-provider change, or launch-type change can create a placement failure that merely coincides with the host incident.
AMI and instance-family compatibility
Verify that the replacement uses a GPU-optimized AMI and an instance family supported by the ECS GPU documentation. AWS documents support requirements for specific families and identifies the older g2 family as deprecated. Also check whether someone performed an in-place NVIDIA or CUDA driver update on an older supported family; AWS warns that such updates can cause GPU workload failures.
Record the AMI ID, image release, kernel, NVIDIA driver, CUDA compatibility, ECS agent version, and instance type for both hosts. “Same instance count” is not evidence of equivalent GPU capability.
Custom container images
If the image does not use an NVIDIA/CUDA base image, verify the documented NVIDIA_DRIVER_CAPABILITIES requirements for the libraries the application needs. A container can be placed successfully and still fail at startup when its runtime libraries or driver capabilities are incompatible.
Step 6: check draining and deployment limits
When an ECS container instance is DRAINING, ECS places no new tasks there. For a service, ECS stops tasks on the draining instance and attempts replacements according to the service deployment configuration.
Rank #4
- 【Softer and More Comfortable】Vaydeer wrist rest has unique diamond pattern, which is the combination of softness and aesthetics. The materials of wrist rest are improved into higher quality memory foam and covered with silky smooth lycra. The computer wrist rest makes you as comfortable and cushiony as like rest your wrists on clouds.
- 【Ergonomic Wrist Saver】The wrist rests for keyboard and mouse comes with a 17.32×3.15×0.83 inch keyboard wrist pad and a 5.94×3.15×0.83 inch mouse wrist support. Based on ergonomic design, the unique concave shape is the perfect fit for your wrist joints. The wrist rest pad fits most computer keyboards and laptops, improve hand and wrist posture, release your wrist and arm stress.
- 【Non-Slip Rubber Bottom】Featuring an anti-skid silicone base on the bottom, this wrist keyboard support stays firmly in place on your desk, preventing the padding from sliding around, ensuring stable and consistent wrist support during extended computer sessions.
- 【Better Experience & Pain Relief】Our keyboard arm rest is beneficial to alleviate the soreness caused by direct contact and friction between your arm and a hard desk surface, reducing the risk of wrist fatigue or carpal tunnel. The soft texture of memory foam can evenly distribute the pressure around your wrists and provide good support with just enough give.
- 【Helpful in Multiple Scenarios】Whether you're working, studying, writing, typing, gaming, this keyboard and mouse rest combo is an essential accessory to add comfort and support to your hands and wrists. It’s also a great gift for men, women, family, friend, coworker, gamer, teacher, etc.
| Setting | Effect during replacement | Diagnostic question |
|---|---|---|
minimumHealthyPercent |
Controls how far below the desired count ECS may go while replacing tasks. | Can ECS stop a task and still remain within the permitted healthy percentage? |
maximumPercent |
Caps the number of running tasks during deployment. | Is there room to start a replacement before stopping an old task? |
| Both set to 100% | ECS cannot stop old tasks or start additional replacements, so draining and deployment can stall. | Are the percentages preventing the scheduler from making progress? |
Check that the cluster has enough eligible GPU, CPU, memory, networking, and other required resources. A nominal instance count can be misleading when constraints, health checks, or GPU allocation make those instances unusable.
Service tasks versus standalone tasks
Service scheduling replaces service tasks according to the service’s desired count and deployment rules. Standalone tasks launched with RunTask are different: pending or running standalone tasks are not automatically replaced by service scheduling during a drain. They may finish, remain pending, or need to be stopped and launched again by your own automation.
Step 7: investigate tasks that start and then exit
For a task that reached RUNNING, collect container exit details, application logs, and timestamps. Compare those with the host’s XID records and ECS GPU restart metrics.
ECS Managed Instance workloads expose TaskGPURestartAppXidCount and ContainerGPURestartAppXidCount. These count XID errors in NVIDIA’s RESTART_APP resolution category. AWS describes them as useful for detecting transient GPU faults affecting a task or container without implying underlying hardware failure.
- Increasing application-restart XID metrics with a healthy
ACCELERATED_COMPUTEcheck points first toward a transient or application-level fault. - A critical host health impairment with an XID in
statusReasonsupports investigating the managed instance and its replacement. - Neither metric, alone, proves that every subsequent task failure is hardware-related.
Check whether the container was killed by its own process, failed a health check, exhausted memory, or could not initialize CUDA. Keep application-level evidence separate from ECS scheduler evidence.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Failure playbooks
The task stopped, and the replacement is healthy
Confirm the stop occurred during the documented drain window, then verify that the replacement task reaches RUNNING and passes its load-balancer health checks if applicable. No hardware action is justified solely by the original task stop.
Recommended Free Tools
The replacement remains pending with RESOURCE:GPU
Compare requested GPU count with free GPUs on every eligible replacement host. Check that the replacement is registered to the intended capacity provider, uses a supported GPU instance family, and is not excluded by attributes or constraints. Add or restore eligible capacity only when the evidence shows a real capacity shortage.
Best Value
- 【No Drill Mounted】Cable management tray can be assembled in as little as 3 minutes or less-simply. Side clips easily clampe the edge of desk, pass through the holes on the sides, and secure excess cables with baskets. Desk cable management supports inward or outward installation to meet your needs in different scenarios, and provides you with great convenience when collecting and organizing wires.
- 【Rubber Pad Protects Desk】Wire organizer under desk built-in soft rubber pads on the clamp protect your desk from scratches, so you will not damage your furniture or office.
- 【Sturdy and Beautiful】Cord management under desk is made of high-quality carbon steel and holds about 15 lbs. The wire basket has 2 holes on each side of the tray to enable your cables to pass freely. Cable baskets are suitable for desk thickness use from 0.4" to 2.4".
- 【Space Saving & Keep Organized】Desk cable management tray holds all your power cords, outlet strips, USB hubs and computer transformers hidden from your feet, and the data cables will not enter the vacuum cleaner to keep your desk clean and tidy. Desk wire organizer is also suitable for use in the kitchen and outdoors.
- 【Reduce Safety Risks】Cable tray under desk keeps all plugs away from your kids, no more worry about tripping. Using 2 holes wire tray saves extra space, helps you easily cabling and wire protection. The mesh design is not easy to accumulate dust, but avoids safety accidents caused by messy wires.
The replacement remains pending with AGENT
Inspect agent connectivity and container-instance status. Until the agent is connected and the instance is ACTIVE, ECS cannot reliably place the task. Preserve the event and health records before taking host-level corrective action.
The replacement is INITIALIZING or INSUFFICIENT_DATA
Do not label the GPU failed immediately. Allow health checks to populate, then compare the eventual overall status with placement events. If the state progresses to IMPAIRED and includes an accelerated-compute XID reason, treat it as a host-health incident; otherwise continue with capacity, agent, and configuration checks.
The service repeatedly starts and stops tasks
Inspect each failed task individually. Confirm the task definition, image, runtime libraries, GPU visibility, container health checks, and application logs. Use the GPU restart metrics to identify transient application-level XID faults, but do not infer a hardware replacement requirement without corresponding host-health evidence.
The service cannot replace tasks while draining
Check desired count, available eligible capacity, and deployment percentages. A service with both deployment percentages at 100% can be unable to stop an old task or start an additional replacement. Correct the deployment configuration only after confirming that it is the reason the scheduler is blocked, and account for the availability impact of any change.
Validate the replacement before returning traffic
- Confirm the replacement container instance is
ACTIVEand its ECS agent is connected. - Confirm health details are populated and the overall status is acceptable, including the accelerated-compute check where applicable.
- Verify the instance has the requested number of free GPUs and all required attributes.
- Confirm the task definition revision, GPU resource request, image, and runtime settings.
- Watch the service events until the replacement task reaches
RUNNING. - If a load balancer is used, wait for its target health checks to pass before declaring recovery.
- Keep the original host’s XID, task stop, and service-event records attached to the incident.
A replacement that is registered but cannot pass health checks is not ready for production traffic. Conversely, a task that starts normally on a healthy replacement is evidence against assuming that the original stop was an application failure.
Build a repeatable GPU-repair runbook
- Alert on service placement failures and repeated task-start failures, not only on instance termination.
- Capture task stop records, service events, and container-instance health output automatically during the repair window.
- Record the ECS agent version, AMI, NVIDIA driver, CUDA compatibility, instance type, task-definition revision, and requested GPU count.
- Track capacity-provider size because the 20% drain limit and the one-at-a-time rule for providers with fewer than nine instances affect recovery timing.
- Keep separate procedures for ECS Managed Instances and self-managed EC2 capacity.
- Test deployment percentages with GPU workloads so the service can replace a drained task without violating availability or capacity limits.
- Document which tasks are services and which are standalone launches; only services receive automatic service-scheduler replacement.
Final diagnosis checklist
Before closing the incident, you should be able to answer each question with a timestamped record:
- Why did the original task stop: drain, placement failure, container exit, or another stop reason?
- Was the original instance impaired, and did ECS place it in
DRAINING? - Did the replacement register in the correct cluster and capacity provider?
- Was the replacement agent connected and its health status beyond initialization?
- Did it have enough eligible GPU resources and the required attributes?
- Were task-definition GPU settings, agent support, AMI, drivers, and image capabilities compatible?
- Did service deployment percentages or constraints prevent replacement?
- Did the replacement task reach
RUNNINGand pass downstream health checks?
The practical verdict is straightforward: a task stop during managed GPU-host draining is expected; a pending replacement is usually a capacity, eligibility, agent, or deployment problem; and a running task that exits requires application and GPU-runtime evidence. Match the corrective action to those records instead of treating every post-repair restart as proof of another failed GPU.
Free tools Windows power users keep installed
One-click scans. No signup required.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
