iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Error feedback does not simply make compressed gradients exact: it carries the part a compressor omits into later steps. The standard argument works most directly for SGD-like updates. Adam’s changing, coordinate-wise scaling means that preserving gradient information does not automatically preserve the parameter updates, so directly adding standard error feedback to Adam does not inherit the same reasoning. That is the specific sense in which “Adam breaks it”—not a claim that every compressed or error-feedback version of Adam fails.
What error feedback does to a compressed gradient
A compressor may reduce communication by approximating a gradient, for example by retaining a coarse representation rather than every coordinate’s full value. That saves bandwidth, but the approximation can introduce error at each step. With error feedback, also called error compensation, the algorithm stores the difference between the vector it meant to send and the compressed vector it actually sent.
In a simple SGD-style description, let g_t be the current gradient and r_t the residual carried from the previous step. The algorithm forms u_t = g_t + r_t, sends or applies a compressed vector c_t = C(u_t), then stores r_{t+1} = u_t - c_t. The residual is not thrown away: it is added to a later input, giving the compressor another chance to transmit that information.
Recommended Free Tools
Across steps, this bookkeeping telescopes: the accumulated compressed vectors differ from the accumulated original gradients by a boundary term involving the residual. This is the intuition for why error feedback can counteract persistent compression bias. It is not itself a convergence proof; the optimizer, compressor, and assumptions in a particular analysis still matter.
#1 Best Overall
Why the basic argument fits SGD-like updates
In plain SGD, a gradient (or compressed gradient) is multiplied by a learning rate and subtracted from the parameters. Momentum also has a linear accumulation structure. In these settings, error carried forward in gradient space can be related relatively directly to the accumulated update: the algorithm is repeatedly adding vectors of the same kind.
Karimireddy, Rebjock, Stich, and Jaggi’s 2019 paper, Error Feedback Fixes SignSGD and other Gradient Compression Schemes, gives a formal example. It reports convex counterexamples for signSGD and proves a convergence-rate result for EF-SGD in the settings it analyzes. The authors write that EF-SGD with an arbitrary compression operator “achieves the same rate of convergence as SGD without any additional assumptions.” Read that statement in its paper-specific scope: it describes their EF-SGD result and analyzed settings, not every optimizer, training problem, or later error-feedback design.
Rank #2
- Use scikit-learn to track an example ML project end to end
- Explore several models, including support vector machines, decision trees, random forests, and ensemble methods
- Exploit unsupervised learning techniques such as dimensionality reduction, clustering, and anomaly detection
- Dive into neural net architectures, including convolutional nets, recurrent nets, generative adversarial networks, autoencoders, diffusion models, and transformers
- Use TensorFlow and Keras to build and train neural nets for computer vision, natural language processing, generative models, and deep reinforcement learning
What changes when the optimizer is Adam
Adam does not turn a gradient directly into a parameter step using one fixed scalar. It maintains an exponential moving average of gradients (the first moment) and an exponential moving average of squared gradients (the second moment), applies bias corrections, then divides the first estimate coordinate by coordinate by the square root of the second estimate plus a small epsilon. In the notation of Kingma and Ba’s 2014 paper, the update has the form θ_t = θ_{t-1} - α m̂_t / (√v̂_t + ε), with the division applied element-wise.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problemsThat denominator is a history-dependent preconditioner: each coordinate’s effective step depends on the gradients seen so far, and the scaling can change from one step to the next. A residual stored as a gradient-space vector and added to a later gradient therefore does not necessarily produce the same parameter-space motion as applying that residual at the earlier step. Even if a sequence of compressed vectors preserves a useful sum in gradient space, Adam’s adaptive transformations mean that this fact alone does not show that its parameter updates match those from uncompressed gradients.
Rank #3
Tang and co-authors make the practical scope explicit in their 2021 paper: earlier error-compensation approaches designed for linearly dependent SGD or momentum updates did not directly work with Adam’s nonlinear adaptive update. So “Adam breaks it” is shorthand for a mismatch between the standard error-feedback argument and a direct application to Adam—not a theorem that all compression or residual methods fail with Adam.
How 1-bit Adam handles the mismatch
Tang and co-authors’ 2021 method does not rely on applying the plain SGD error-feedback argument unchanged throughout Adam training. It first uses a warmup phase, then treats Adam’s variance estimate as a fixed preconditioner during the compression phase. Their stated key finding is that “Adam’s variance becomes stable (after a warmup phase) and can be used as a fixed precondition for the rest of the training (compression phase).” Fixing that scaling changes the problem the compression method has to handle.
Rank #4
The authors report results from their experiments, not expected gains for arbitrary workloads:
| Reported result | Experimental context |
|---|---|
| Up to 5× reduction in communication volume | Maximum reported by Tang and co-authors for their 1-bit Adam experiments |
| Up to 3.3× higher throughput | BERT-Large pretraining experiments |
| Up to 2.9× higher throughput | SQuAD fine-tuning experiments |
| Experiments on up to 256 GPUs | Scale reported in the 2021 paper |
Communication-volume reduction and throughput are different measures: transmitting fewer bits does not, by itself, establish a proportional end-to-end speedup. The reported throughput figures are task-specific maxima from that paper’s experiments, not universal estimates for another model, cluster, or implementation.
Best Value
How to compare error-feedback results fairly
The 2019 EF-SGD result and 1-bit Adam address related communication problems, but they do not establish interchangeable guarantees. One is a convergence analysis for EF-SGD under its stated settings; the other is an Adam-specific distributed training method with a warmup and fixed-preconditioner strategy, plus experimental results for particular tasks.
| Approach | Update structure | Compression or residual idea | What the cited result supports |
|---|---|---|---|
| EF-SGD (Karimireddy et al., 2019) | SGD-style updates | Error feedback retains the discrepancy introduced by compression; the paper analyzes arbitrary compression operators | A convergence-rate result in the paper’s analyzed settings, alongside convex counterexamples for signSGD |
| 1-bit Adam (Tang et al., 2021) | Adam, with a warmup followed by a fixed variance-based preconditioner during compression | A method for compressed distributed Adam training | Reported communication and task-throughput maxima in experiments up to 256 GPUs |
| Quantized Adam with Error Feedback (Chen et al., 2021) | Adam in the paper’s stated stochastic nonconvex setting | Gradient and weight quantization with error feedback | Convergence analyses for that specific algorithm and setting |
For any comparison, check the optimizer update, the location and form of the residual, what is quantized or compressed, the assumptions behind the guarantee, and whether the reported metric is communication volume or end-to-end throughput. A guarantee or speedup for one row should not be transferred to another simply because both methods use compression or error feedback.
What the theory does—and does not—establish
Error feedback is a family of methods, not one universal convergence guarantee. A 2025 analysis by Fatkhullin and co-authors notes that classical EF theory can depend on strong assumptions, including bounded gradients, and can give pessimistic rates in some regimes. Their EF21 work presents extensions aimed at deficiencies identified in those analyses. This is a refinement of the theoretical picture, not evidence that classical error feedback always fails in practice.
Free tools Windows power users keep installed
One-click scans. No signup required.
Taken together, these results support a careful conclusion: error feedback can address compression error, and Adam-specific compression methods exist, but neither the SGD intuition nor a theorem for one algorithm settles the behavior of every Adam-family optimizer, compressor, and residual convention. The cited work does not establish a convergence guarantee for every such combination.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
