Dust: Pretraining Transformers Without Backpropagation
Dust is a new zeroth-order optimization method that successfully pretrains transformer language models without backpropagation by perturbing activations at the token level.
read it The paper provides a rigorous mathematical and empirical foundation for replacing backpropagation with activation-space search in transformers.
summary
The authors present Dust, a zeroth-order optimization algorithm that pretrains transformer language models without backpropagation. By independently perturbing activations at every token, the method evaluates a massive virtual population in parallel during a single forward pass. Experiments demonstrate that Dust is orders of magnitude more efficient than weight-space evolution strategies and closely matches backprop performance at scale. Crucially, the findings challenge the assumption that zeroth-order methods cannot scale, showing that larger models are actually more population-efficient.
key points
- Dust uses per-token activation perturbations to construct a virtual population, avoiding the prohibitive cost of materializing weight-space modifications.
- The method achieves 10^3 to 10^4 greater efficiency compared to transformer implementations of state-of-the-art evolution strategies like EGGROLL.
- Empirical results indicate that larger model sizes improve population efficiency rather than hindering it, contrary to conventional wisdom.
who should read it
Machine learning researchers and deep learning engineers investigating gradient-free optimization and alternative training paradigms.
discussion positive
consensus
Commenters debate whether the trade-offs in computational efficiency and global coordination make non-backprop methods viable on current hardware architecture.
pushback
Critics argue that backprop's rigid global coordination is extremely cheap on modern GPU clusters, meaning alternative methods require fundamentally different hardware to win.
Not necessarily, backprop is highly parallelizable since it is just a bunch of matrix mults.
Something like Dust skips the backward pass on backprop. But other techniques like Neural Predictive Coding can be completely asynchronous, each "weight" can fire independent of those far away from it. Innocenti, et. al have shown that NPC gradients converge to backprop within a certain "regime".
The win with asynchronous techniques like NPC is that you do not need the extreme co-ordination that backprop requires and hence should be computationally much easier given the right device.
Although at …
It analyzes the trade-off between backprop's heavy matrix multiplication parallelizability and the coordination-free benefits of asynchronous training methods.
Generated by an LLM from the article and thread. Check the sources before relying on it.