Show HN: Jevstiller – Distill Jev into a local model, with a disagreement bound
Jevstiller is an open-source proxy that caches LLM classification calls into a tiny local model while mathematically guaranteeing a strict disagreement bound.
read it Essential reading for engineers dealing with LLM classification costs and latency, detailing how to implement statistical guarantees for distillation cascades.
summary
Jevstiller sits in front of classification APIs, learning a tiny multinomial logistic regression model on frozen sentence embeddings from teacher outputs. Standard confidence thresholds often break accuracy budgets due to finite-sample noise, so Jevstiller uses fixed-sequence testing and exact Clopper-Pearson bounds to rigorously enforce a maximum disagreement rate. The system continually retrains on accumulated traffic while a permanent audit loop guards against drift and teacher changes. This approach slashes latency to about 15 milliseconds on a CPU for predictable tasks without sacrificing alignment with the teacher model.
key points
- Naive confidence thresholding breaks predefined error budgets about half the time due to finite-sample noise.
- Jevstiller uses exact Clopper-Pearson bounds and fixed-sequence testing to reliably enforce a strict agreement guarantee with the teacher model.
- The local model consists of a frozen sentence embedding encoder paired with a simple multinomial logistic regression head, running in about 15ms on a CPU.
- A permanent background audit loop monitors live agreement and automatically triggers fallback and retraining if alignment degrades or the teacher shifts.
who should read it
Software engineers and ML practitioners looking to reduce LLM classification latency and inference costs with formal guarantees.
discussion warm
consensus
Commenters find the statistical rigor and practical utility for high-volume text classification compelling, though some note it is limited to simpler classification tasks.
pushback
Critics point out that this only works for straightforward classification and does not solve broader, more complex LLM use cases.
Not really
they distill different things. model2vec distills a sentence transformer into static embeddings, so the output is a faster general-purpose encoder.
Jevstiller keeps the encoder frozen (bge-small by default) and distills Jev's decisions on one specific question into a small head on top of it
Clarifies the architectural distinction between Jevstiller and model2vec, noting it distills specific task decisions on top of a frozen encoder rather than general-purpose embeddings.
Generated by an LLM from the article and thread. Check the sources before relying on it.