Ollaya – Ollama for open-source, Jev-style decision models
Ollaya is an open-source local inference engine designed to run single-forward-pass decision models and text classifiers in milliseconds.
summary is enough Useful if you want to deploy local Jev-style classifiers, though the core concept is essentially fast text classification.
summary
Ollaya is a new open-source tool modeled after Ollama that runs local decision models—single-forward-pass classifiers like TypeSafe's Jev—entirely on local hardware. By bypassing token-by-token generation, it delivers calibrated answers and probabilities in milliseconds while maintaining compatibility with TypeSafe's API. The project packages various open model families like Winnow and Laya for structured text classification. However, commenters note that these models are essentially traditional classifiers, and similar capabilities are already arriving in tools like vLLM.
key points
- Ollaya runs single-forward-pass decision models locally in milliseconds without token-by-token generation.
- It provides drop-in compatibility with TypeSafe's Jev API endpoints and supports open weights like Winnow and Laya.
- Commenters debate whether decision models offer meaningful advantages over traditional fine-tuned classifiers.
who should read it
Backend and AI engineers looking for low-latency, cost-effective local classification models.
discussion mixed
consensus
Commenters agree that low-latency local classifiers are useful workflow alternatives to slow LLMs, but debate whether decision models are anything more than rebranded text classifiers.
pushback
Some users report that open alternatives like Laya perform significantly worse on complex queries compared to proprietary services.
> difference between an instruct based re-ranker and laya/jev I just don't see it
Main difference is that laya/jev/et-al give you a zero-shot classifier that requires no training. You can prompt engineer your way to a quick fairly reliable cheap enough decision engine that you can use to iterate quickly (by prompt engineering).
Right now a lot of people are doing this with LLMs and it's too slow and expensive.
Imo the right iterative approach to productionizing these systems is something like:
1. Build it with an LLM. Iterate on the prompt 2. Start building a real-world dataset 3. When …
Outlines a practical production pipeline for moving from prompt-engineered LLMs to cheap local classifiers.
Generated by an LLM from the article and thread. Check the sources before relying on it.