Jeff – Jev-compatible 0.8B decision models, trained at home, ~30 ms
Jeff provides open-source, local 0.8B to 2B decision models that deliver Jev-compatible zero-shot text classification and decision-making in under 30 milliseconds.
read it The post provides a comprehensive open-source alternative to proprietary decision APIs, detailing performance benchmarks, local training pipelines, and serving setups.
summary
Jeff is an independent, open-source project offering small 0.8B and 2B models fine-tuned to accept natural language situations and output calibrated probabilities across options in a single forward pass. Operating without text generation or cloud dependencies, it achieves inference times around 22 to 30 milliseconds on local hardware. While zero-shot performance lags behind proprietary options like Jev on complex reasoning tasks, the models can be rapidly fine-tuned on local hardware for domain-specific accuracy.
key points
- Models like Jeff-Qwen3.5-0.8B achieve ~22-30ms inference per decision in a single forward pass without text generation.
- Supports up to 254 options in v1.1, providing choice, binary, and score question types through a Jev-compatible API format.
- Can be fine-tuned locally in under half an hour on a single GPU to dramatically boost domain-specific accuracy.
who should read it
Developers looking to integrate fast, local, low-latency decision-making and classification into their applications without relying on cloud APIs.
discussion positive
consensus
Commenters are impressed by the speed and local training achievement, but many debate whether these small generative models outperform traditional embedding models paired with simple classifiers.
pushback
Some users report that zero-shot accuracy is significantly worse than proprietary Jev on real-world tasks, making fine-tuning a necessity rather than an optional bonus.
Its funny - I was going to leave a similar comment - and I have, earlier, on a different thread. If people need fast classification, on a fairly scoped problem, it is very fruitful to start with an off-the-shelf embedding model like ModernBERT (which Laya uses) and stick a classifier in front - like a Support Vector Machine (SVM). For starters just tune the SVM, you don't even have to fine-tune the embedder - often it works very well, esp. given the compute needed. Plus you can get reliable confidence scores and generate explanations if you want them (using something like SHAP).
It highlights a simpler, battle-tested alternative using embedding models and support vector machines for text classification that avoids the overhead of LLMs.
Generated by an LLM from the article and thread. Check the sources before relying on it.