Launch HN: Magnitude (YC S25) – Self-optimizing inference engine for agents
Magnitude is an open-source, self-optimizing inference engine for local AI agents that compiles and tunes custom kernels directly on user hardware.
summary is enough Useful launch announcement for local LLM users, though technical specifics and benchmarks are better checked directly on the repository.
summary
Magnitude is a new open-source desktop app and inference engine designed for local AI agents that claims up to 2x faster decode speeds than llama.cpp by compiling and tuning custom kernels directly on user hardware. It features quantized KV caches, shared prefix caches for concurrent sessions, and one-click integration with popular coding agents. The author actively engaged in the discussion, addressing early bug reports, chip-specific optimization gaps, and benchmarking methodology.
key points
- Claims up to 92 percent faster decode on Metal and 19 percent on CUDA compared to llama.cpp via on-device kernel tuning.
- Reduces memory overhead per agent and supports shared prefix caches for fast concurrent sessions.
- Integrates with popular agents like Claude Code, Pi, and Cline through a desktop app and OpenAI-compatible API.
who should read it
Developers and power users running local LLM agents who want to optimize inference speed and memory footprint.
discussion positive
consensus
Commenters find the self-optimizing engine concept compelling, though several noted that performance currently matches or trails specialized baselines like llama.cpp or MLX depending on the chip version.
pushback
Initial friction points include mandatory pre-download model assessment UI locks, lack of multi-GPU support, and missing chip-specific optimizations.
Running an inference network across 11 Macs (Teale.com), so I pointed my orchestrator at the repo.
The autotuner is the real thing - kernel-level search, config budget split by measured time share, winners cached per device/toolchain. Rotating resident weights across layers to dodge the hot-cache trap is a nice touch.
One question on model ranking: the fit scores look like predictions built from cost constants measured on a single M4 Max, not per-device measurements. How do you rank models across genuinely mixed hardware? Does the estimator improve from actual runs over time? That gap …
It offers a firsthand technical validation of the autotuner's kernel-level search and config budgeting from a developer running a distributed machine fleet.
Generated by an LLM from the article and thread. Check the sources before relying on it.