Compute-efficient pretraining and scaling to trillion-parameter models

summary

Magic has developed a pretraining recipe that is over 10 times more compute-efficient than leading open-weight base models, allowing them to match or outperform high-end models with significantly fewer FLOPs. This efficiency is achieved through a combination of architectural improvements, optimizer tweaks, and rigorous data curation rather than raw hardware scale. By reducing the cost of training trillion-parameter models, they aim to build superhuman coding agents and automate AI research and development.

key points

1. The team achieved a 50x reduction in FLOPs compared to DeepSeek V4 Pro Base while maintaining comparable performance, reducing training costs from over $100M to roughly $0.5M for certain scales.
2. Efficiency gains were not the result of a single breakthrough but the multiplicative effect of dozens of small changes across model architecture, training objectives, and data curation strategies.
3. The model demonstrates strong generalization and knowledge across diverse domains, particularly in coding and math, and shows rapid improvement during reinforcement learning without needing supervised fine-tuning.

who should read it

AI researchers and machine learning engineers interested in compute-efficient scaling laws and the development of high-capability base models with limited hardware resources.

discussion skeptical