From the creator of Redis; run LLM locally with ds4
Salvatore Sanfilippo (antirez) released ds4, a C-based inference engine optimized for running large MoE models like DeepSeek V4 and Qwen3.8 locally.
summary is enough The tool is highly specialized for specific high-end hardware and a few model families, making it a niche utility rather than a general-purpose breakthrough.
summary
ds4 is a narrow C inference engine designed for high-memory Mac, CUDA, and ROCm machines. It uses asymmetric 2-bit quantization to compress routed experts while preserving critical shared paths, allowing large Mixture-of-Experts (MoE) models to fit on consumer hardware. The project includes a CLI, HTTP APIs, and a native agent that share model state and cache to optimize performance and context handling.
key points
- Supports DeepSeek V4/V4.1, GLM 5.x, and Qwen3.8 Flash Next with specialized GGUF builds.
- Implements asymmetric quantization and SSD streaming to enable large models on machines with 64GB-128GB RAM.
- Features a dedicated agent (ds4-agent) that uses append-only history to maintain a reusable KV cache prefix.
- Achieves high performance on M5 Max 128GB, reaching up to 39.4 tokens/sec generation for Q2 quants.
who should read it
Engineers with high-end Apple Silicon (M5 Max) or high-VRAM GPUs wanting to run state-of-the-art MoE models locally.
discussion positive
consensus
Users generally appreciate the focus on a small set of optimized models over generic support, noting that it feels more stable and performant for those specific targets.
pushback
Some commenters criticize the promotional website's aesthetic as 'slop' and debate whether C was the right choice over Rust for this implementation.
It's more likely to work. Most LLM runners are meant to work with any model, which means there are all kinds of ways you might misconfigure them in a way that causes function tooling not to work, or performance to be less than you would like.
DwarfStar's selling point is that it only supports a small set of carefully chosen models, but it supports them really well.
Explains the core value proposition: by limiting supported models, the engine avoids the misconfigurations and performance pitfalls of generic runners.
Generated by an LLM from the article and thread. Check the sources before relying on it.