Gradient descent algorithms update model parameters in the direction that reduces an objective function. Their differences come down to which examples supply each gradient, how much past gradient information is retained, and whether step sizes adapt for individual parameters. There is no optimizer that is best for every task; use this guide to understand the trade-offs and choose candidates to validate on your data.
How gradient descent updates model parameters
Let the objective be a loss function over model parameters. Gradient descent calculates the gradient—the direction of steepest increase—and moves the parameters in the opposite direction. The learning rate controls the size of that move. A rate that is too large can make training unstable; one that is too small can make progress slow.
The phrase “gradient descent” also describes how much data is used to estimate each update. Those data-use variants are separate from optimizers that add momentum or adapt step sizes.
Batch, stochastic, and mini-batch updates
| Variant | Data used for each gradient | What changes in practice |
|---|---|---|
| Batch gradient descent | The full training dataset | Each update uses information from all training examples, but computing an update can be expensive on large datasets. |
| Stochastic gradient descent (SGD) | One training example | Updates are based on a noisier estimate and can be made frequently, but individual steps may fluctuate. |
| Mini-batch SGD | A subset of the training dataset | Balances the amount of information in an update with the cost of computing it; this is a common formulation in neural-network training. |
These names describe the gradient sample, not necessarily a different rule for handling gradient history or per-parameter step sizes. For example, mini-batch SGD can also be used with momentum or an adaptive optimizer. See Sebastian Ruder’s overview of gradient descent optimization algorithms.
Recommended Free Tools
#1 Best Overall
- Read Before You Buy — No Video Output: These adapters support charging and USB 2.0 data transfer, but cannot transmit video signals. Except for standard USB webcams (which use USB data only), they are not compatible with HDMI/DisplayPort cables, video-capable USB-C hubs, or docking stations with video output.
- Convert USB-A Ports to USB-C: Designed to connect USB-C earphones, cables, flash drives, card readers, and other USB-C accessories to standard USB-A ports. Plug-and-play with no drivers or software required.
- Aluminum Alloy Housing: Built with a sturdy aluminum alloy shell that aids in heat dissipation and protects against daily wear and scratches. Designed to maintain a stable and secure connection.
- Compact & Travel-Friendly: The ultra-compact design allows the adapter to stay plugged into your device without blocking adjacent ports or adding bulk, reducing wear and tear on your original USB ports.
- 12-Month Warranty: Backed by a 12-month manufacturer warranty for peace of mind. Designed to meet strict quality control standards for reliable everyday performance.
Cheat sheet: 10 gradient descent algorithms
The first three entries vary the amount of data used in an update. The remaining entries change how updates use gradient history or scale step sizes. This is a practical selection, not an exhaustive list of optimizers.
| Algorithm | Gradient sample | Update memory | Step-size handling | Main practical caveat |
|---|---|---|---|---|
| Batch gradient descent | Full dataset | None by default | Global learning rate | Each update requires processing the full dataset. |
| Stochastic gradient descent | Single example | None by default | Global learning rate | Single-example gradients can make updates noisy. |
| Mini-batch SGD | Mini-batch | None by default | Global learning rate | Results depend on choices such as batch size and learning rate. |
| SGD with momentum | Usually mini-batch in neural-network training | Velocity from current and prior gradients | Global learning rate | Adds a momentum coefficient to tune. |
| Nesterov accelerated gradient | Usually mini-batch | Momentum with a look-ahead formulation | Global learning rate | Uses a distinct update formulation; it is not just ordinary momentum with a different name. |
| AdaGrad | Usually mini-batch | Accumulated squared gradients | Adaptive per-parameter scaling | Accumulated history can make effective learning rates shrink too much. |
| AdaDelta | Usually mini-batch | Decaying history of squared gradients | Adaptive scaling | Has additional state and hyperparameters; compare its behavior on the target task. |
| RMSProp | Usually mini-batch | Exponential moving average of squared gradients | Adaptive per-parameter scaling | Requires a decay choice and other learning-rate tuning. |
| Adam | Usually mini-batch | Exponential estimates of first and second moments, with bias correction | Adaptive per-parameter scaling | Maintains extra state for each parameter; its popularity does not guarantee best validation results. |
| Nadam | Usually mini-batch | Adam-style moment estimates with a Nesterov-style formulation | Adaptive per-parameter scaling | Combines design choices but still needs evaluation and tuning. |
The table summarizes the core distinctions, not a guarantee about speed or accuracy on a particular workload. The optimization chapter of Deep Learning discusses adaptive methods and notes that there is no consensus on a single best algorithm. Google’s Deep Learning Tuning Playbook provides update rules and tuning guidance for several of these methods.
Rank #2
- 5-in-1 USB-C Hub: Experience comprehensive connectivity featuring a Power Delivery input, two USB-A 2.0 ports, a USB-A 3.0 port, and an HDMI port. (Note: The USB-C power delivery input port is only for connecting an external wall charger to power your laptop and cannot power peripheral devices.)
- 90W Pass-Through Charging: Achieve optimal charging with 90W pass-through power to your laptop, supported by a total input of 100W, with the hub reserving 10W for operational efficiency. (Note: Wall charger not included.)
- Quick Data Transfers: Accelerate your productivity with rapid data transfers using a high-speed 5Gbps USB 3.0 port and two 480Mbps USB 2.0 ports.
- 4K HDMI Display: Enhance your visual experience with a hub capable of delivering 4K resolution at 30Hz in both mirror and extend modes. Please note that this hub is compatible with MacBook (macOS 12 and newer), Windows 10 and 11, ChromeOS, and laptops equipped with DP Alt Mode and Power Delivery. Note: This device is not compatible with Linux.
- What You Get: Anker USB-C Hub (5-in-1, 4K HDMI), welcome guide, 18-month warranty, and our friendly customer service.
How momentum and adaptive optimizers differ
Momentum and Nesterov: use the direction of recent gradients
Momentum combines gradient information over time into a velocity, rather than letting each update depend only on the latest gradient. This can smooth out changes in direction, but adds a coefficient that affects the update trajectory. Nesterov momentum uses a look-ahead contribution in its formulation, so its gradient-based update differs from ordinary momentum. Neither change removes the need to choose a learning rate.
AdaGrad: accumulate squared gradients
AdaGrad tracks the sum of past squared gradients for each parameter and scales that parameter’s step accordingly. This can be useful when gradients vary in scale or are sparse. Its limitation is built into the accumulation: as the sum grows, effective learning rates can become so small that learning slows prematurely, a concern for deep neural-network training. The textbook’s optimization chapter explains both AdaGrad’s theoretical properties in convex optimization and this practical limitation.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Rank #3
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
AdaDelta and RMSProp: let old squared gradients fade
RMSProp replaces AdaGrad’s unbounded sum with an exponentially weighted moving average of squared gradients. Older observations therefore lose influence over time, and the method introduces a decay hyperparameter. AdaDelta is also included among adaptive methods in the textbook overview; its inclusion here signals a related adaptive approach, not a promise that it will outperform RMSProp or SGD.
Adam and Nadam: combine moment estimates
Adam estimates both the first moment (the mean) and second moment (the uncentered variance) of gradients using exponential averages. It applies bias corrections to those estimates, particularly relevant early in training. Nadam combines Adam-style moment estimates with a Nesterov-style momentum formulation; Google documents its update rule in the Tuning Playbook.
Rank #4
- Dual Converters, Infinite Potential：Includes 2× USB C male to USB A female adapters and 2× USB A male to USB C female adapters. Perfect for a wide range of uses—tablets with Bluetooth keyboards, expand USB ports on macbook, and more. Two different converters for all your daily needs
- Next-Level 10Gbps & 3A Charging: No more slow 480Mbps, this usb to usb c adapter has a transfer speed of up to 10Gbps, allowing you to do more transferring in less time. This usb adapter fits both USB A and USB C charger, supporting up to 3A fast charging
- Upgraded Exquisite Craftsmanship: With an aluminum alloy housing and metal connector, the usbc to usb adapter is extremely durable and sturdy. Rigorously tested to withstand more than 10,000 times of plugging and unplugging, ensuring long-lasting performance
- Broad Compatible: The usb c to usb adapter widely supports all USB C/ USB A devices like laptops, tablets, cellphones, car chargers, and phone chargers. Such as compatible with MacBook Pro/Air 2023/2022, Thunderbolt 4/3 Devices,Apple MagSafe Watch 9/8/7/SE/Ultra, iPad Pro 2022/2021, Samsung Galaxy S23/S20/S10, and iPhone 17/16/15 Pro. Plug and play
- Please Note: To reach 10Gbps speed, keep the cable under 3.3 ft. For USB A Male to USB C adapters, try flipping the USB C connector. USB C Male to USB A adapters support bidirectional 10Gbps transfer within 3.3 ft
In their 2014 paper, Adam’s authors, Diederik P. Kingma and Jimmy Ba, describe it this way: “The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters.” That is the authors’ characterization in the paper’s abstract, not a comparative benchmark showing Adam wins across tasks. Read the original Adam paper.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Adam versus AdamW: a practical implementation distinction
AdamW is not one of the ten core algorithms in the cheat sheet, but its weight-decay behavior is a useful distinction when selecting an implementation. PyTorch documents decoupled weight decay for AdamW: the decay does not accumulate in the momentum or variance. This is an implementation property, not evidence that AdamW will produce the best result for every model or dataset. Consult the current PyTorch optimizer documentation for the documented behavior and available options.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errorsBest Value
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
How to choose an optimizer for a real task
Do not choose by name alone. Compare a small set of plausible methods under the same evaluation setup, and base the decision on validation performance and training behavior for the task at hand. These factors help narrow the candidates:
- Tuning effort: SGD and momentum require attention to the global learning rate, and momentum adds a coefficient. Adaptive methods change per-parameter scaling, but still have choices such as the learning rate and decay settings.
- Gradient sparsity and scale: Adaptive scaling may be worth evaluating when gradients are sparse or vary substantially across parameters. The original Adam paper discusses stochastic objectives, including noisy or sparse gradients; that is motivation to test it, not a universal recommendation.
- Noise and stability: A single-example update is noisier than one based on a larger sample. Momentum smooths updates using history, while mini-batch size changes the gradient sample. Evaluate whether the training curve and validation behavior are useful for your task.
- Memory and computation: Momentum and adaptive methods keep state in addition to model parameters. Consider the cost of that state and the update computation for your model and hardware.
- Validation results: Compare held-out performance and the training behavior you care about, using consistent data splits and comparable training conditions. A method’s theoretical property or reputation cannot substitute for this task-specific result.
Batch size and optimizer choice can interact, so a comparison that changes both at once can obscure the reason one run differs from another. Google’s Tuning Playbook discusses this interaction and provides broader tuning guidance.
Further reading
For a deeper treatment of adaptive methods and optimization for deep models, see Chapter 8, “Optimization for Training Deep Models,” in Deep Learning by Ian Goodfellow, Yoshua Bengio, and Aaron Courville.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →
