Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
summary
Deltafin allows the full, unpruned 2.8-trillion-parameter Kimi K3 model to run on consumer hardware by streaming expert weights from SSDs rather than relying on VRAM. This approach prioritizes raw model quality and exact reproduction of the original weights over the speed gains achieved through pruning or quantization. By pushing the limits of Apple Silicon and external storage, the project demonstrates that frontier-scale models can be hosted locally without compromising the model's integrity.
key points
- The system achieves steady decode speeds of around 1 token per second on an M5 Max MacBook Pro by utilizing a multi-SSD setup to overcome memory constraints.
- Performance is heavily influenced by storage read speeds, where using four drives provides significantly higher throughput than a single drive, though prefill time remains a bottleneck.
- Unlike other projects that reduce model size through 3-bit encoding, Deltafin maintains the original BF16 precision to ensure the output is exactly what the model creators intended.
who should read it
AI researchers and enthusiasts interested in local LLM hosting, hardware optimization, and pushing the boundaries of consumer-grade computing for large-scale models.
discussion positive
consensus
Commenters generally agree that while frontier models provide superior capabilities, the true utility lies in achieving high-speed inference and local execution. There is a debate regarding the role of ASICs versus GPUs, with some arguing that memory bandwidth remains the primary bottleneck regardless of the hardware architecture. The community values the ability to run models locally for privacy and cost reasons, even if it means using smaller, less capable models.
top insight
The discussion on running large models via SSDs reveals that throughput is the primary bottleneck rather than random access latency. Using multiple Thunderbolt 5 enclosures to parallelize reads of expert files can nearly double performance compared to a single internal NVMe drive, demonstrating a practical workaround for RAM limitations in closed architectures like Apple's.
Generated by an LLM from the article and thread. Check the sources before relying on it.