Run Qwen 3.8 Flash Next (125B) on consumer hardware (RTX 4090) at 100T/s
Strata is an open-source inference engine that runs the 125B parameter Qwen 3.8-Flash-Next model on consumer hardware using aggressive quantization.
summary is enough The repository readme provides quick setup steps and speed benchmarks, but deeper technical details require exploring the linked documentation.
summary
Strata enables running a massive 125-billion-parameter Qwen model locally on gaming PCs with 12GB+ VRAM and 32GB+ RAM by leveraging low-bit quantization, expert caching, and hybrid offloading. Users report generation speeds exceeding 100 tokens per second on hardware like an RTX 4090. While some commenters debate the accuracy trade-offs of 2-bit and 3-bit quants, many find the performance practical for coding workflows.
key points
- Strata runs the 125B parameter Qwen 3.8-Flash-Next model on consumer GPUs with 12GB+ VRAM and 32GB+ system RAM.
- It achieves high token generation speeds of 50 to 120+ tokens per second using low-bit quantizations and hybrid offloading.
- Commenters debate whether extreme quantization significantly degrades model capability, though many confirm strong real-world coding performance.
who should read it
Developers and hardware enthusiasts interested in running large open-source language models locally on consumer rigs.
discussion positive
consensus
Commenters are impressed by the raw generation speeds on consumer GPUs, though opinions are split on whether low-bit quants sacrifice too much accuracy.
pushback
Some users note that specialized engines like Strata can underperform standard llama.cpp implementations on specific tasks like vision benchmarks.
I tried it and it worked surprisingly well. On my machine (Nvidia 4090, 128GB DDR5, Ryzen 7950x3d) I'm getting 124 tokens per sec, thought to share it here.
https://huggingface.co/Qwen/Qwen3.8-Flash-Next
It offers practical, firsthand confirmation of high token generation speeds on a high-end consumer GPU setup.
Generated by an LLM from the article and thread. Check the sources before relying on it.