Shapelearn Qwen 3.8 27B (13.1 GB VRAM)
summary
The ByteShape team released full ShapeLearn quantized models for Qwen 3.8 27B, which outperform their previous Lite versions and several competing quantization methods. By optimizing for the quality-speed frontier, these models maintain high accuracy relative to the BF16 baseline while maximizing throughput across various GPU configurations. The findings suggest that traditional fidelity metrics like KL divergence do not always correlate with actual task performance, making comprehensive benchmarking essential.
key points
- Full ShapeLearn models consistently sit on the quality-speed frontier, with the GPU-5 variant achieving 99.63% of BF16 performance and serving as the primary recommendation for most users.
- Speculative decoding using MTP and DFlash2 significantly increases throughput across all GPUs, with DFlash2 offering the highest speed gains but requiring more VRAM and lacking multimodal support.
- Benchmark results demonstrate that lower KL divergence does not automatically translate to better task performance, as seen when ShapeLearn-Lite outperformed some Unsloth models despite higher KLD scores.
who should read it
Developers and AI researchers interested in optimizing large language model deployment on consumer and professional GPUs using llama.cpp.
discussion positive
consensus
AMD users generally find that the performance gains from the new model are negligible or nonexistent compared to Unsloth, likely due to memory bandwidth bottlenecks. There is a shared belief that Nvidia cards, particularly the RTX 5090, will see significantly more benefit due to superior bandwidth and software optimization. The community suggests that performance varies based on whether a system is bandwidth or compute limited.
top insight
The use of speculative decoding with specific draft models and parameter tuning (such as setting n-max=7) can significantly boost tokens per second on bandwidth-limited cards. One user demonstrates that a Qwen fine-tune can improve acceptance rates, potentially pushing performance from 30tps to 80tps on llama.cpp. This suggests that software-level optimization and model selection are more critical than raw hardware specs for local LLM performance.
Generated by an LLM from the article and thread. Check the sources before relying on it.