Recommended Free Tools
iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Yes—Dust demonstrates that transformers can be pretrained without backpropagation, but it is an experimental method, not a practical replacement for standard training today. It estimates updates from forward-pass loss feedback by perturbing activations independently at each token. In Q Labs’ October 2026 experiments, Dust’s test-loss comparison with backprop changes with the training-token budget and model size; the authors say it is not yet compute-efficient enough to replace backprop.
What Dust does instead of backpropagation
Backpropagation sends error information backward through a network to calculate weight gradients. Dust is a zeroth-order method: it uses changes in forward-pass loss to estimate updates without computing a backward pass.
For most model weights, Dust adds Gaussian noise to each linear layer’s output independently at every token position. The model produces token-level losses, and the change in loss rewards or penalizes the perturbation associated with each token. Averaging reward-weighted noise estimates the error at the layer output; combining that estimate with the layer input produces an estimate of the weight gradient. Attention internals use a related method that estimates attention-output error across current and future tokens. Q Labs’ paper describes the method.
Why the authors call it a “virtual population”
Evolution strategies typically evaluate multiple perturbed candidate models. Dust instead treats token positions as parallel population members within a forward pass, avoiding the need to materialize and evaluate a separate perturbed model for every candidate. Most weights use this token-level approach; residual mixing scalars are trained with ordinary weight-space evolution strategies.
#1 Best Overall
What the experiments tested
Q Labs trained GPT-style decoder-only transformers on FineWeb. The main experiment used a 4096-token BPE tokenizer, batches of 16,000 tokens—eight sequences of 2048 tokens—and one epoch of training with SGD, momentum, and a constant learning rate. Settings were tuned separately for each token budget and population. Results were reported across three seeds per setting, using held-out validation and test sets of 544 sequences each. Test loss is reported at the checkpoint selected by validation loss. The paper and its official appendix provide the experimental details.
The base model had 37.7 million parameters: eight pre-normalized blocks, width 512, eight attention heads, RMS normalization, rotary position embeddings, and a squared-ReLU MLP of width 2048. The separate model-size study ranged from 2.0 million to 243 million parameters.
How Dust compared with backprop
The following are Q Labs’ test losses, averaged over three seeds at validation-selected checkpoints. Lower is better. The base-model comparison used Dust with 16,000 draws.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →| Training tokens | Dust test loss | Backprop test loss | Lower loss |
|---|---|---|---|
| 1 million | 5.916 | 5.959 | Dust |
| 10 million | 5.049 | 4.989 | Backprop |
So the result is not a consistent win for either method: Dust’s loss was lower at 1 million tokens, while backprop’s was lower at 10 million. The study does not establish that Dust remains competitive at larger training budgets; the reported main comparison extends to 20 million tokens.
Results across model sizes at 10 million tokens
At 10 million training tokens and 16,000 draws, the comparison also varies by model size:
| Model parameters | Dust test loss | Backprop test loss | Lower loss |
|---|---|---|---|
| 2.0 million | 5.171 | 5.180 | Dust |
| 7.3 million | 5.065 | 5.066 | Dust |
| 38 million | 5.036 | 5.015 | Backprop |
| 243 million | 5.086 | 5.048 | Backprop |
Dust had slightly lower loss for the two smaller models; backprop had lower loss for the two larger ones. These values describe this experiment, not a general scaling law or proof that either method will lead at other budgets.
What the efficiency claims do—and do not—show
Q Labs estimates that from 1 million tokens upward, Dust may be roughly 1,000 to 10,000 times more efficient than its transformer implementation of EGGROLL, a weight-space evolution-strategy method. That is the authors’ estimate, not an independently established efficiency ratio for zeroth-order training in general. The appendix derives population comparisons by fitting measured curves and extrapolating them, so the estimate should not be read as a direct, universal benchmark.
Q Labs also reports that Dust’s gradient estimates align more closely with backprop as population grows, with diagnostics measured at scales up to a 1-billion-token checkpoint. This is a gradient-alignment result, not evidence that Dust itself completed a 1-billion-token pretraining run.
Best Value
What remains uncertain
- The reported findings come from Q Labs’ experiments; the paper and appendix do not establish independent replication.
- Results beyond the reported main-comparison budget, on substantially larger training runs, or with other architectures remain unresolved.
- The authors frame Dust as a research direction for search-based credit assignment, not as a ready-to-use training substitute. Samip Dahal, Bishwas Mandal, Serdar Gülbahar, and Akshay Vegesna write: “We do not attempt to make it compute-efficient enough to replace backprop today.”
- The authors also leave open training new kinds of neural nets that backprop handles poorly, writing that both that goal and compute efficiency “are left to future work.”
The paper’s implementation is linked from Q Labs’ research page. The published results support treating Dust as a promising experimental approach to credit assignment—not as a reason for practitioners to switch from backprop.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
