A single function Jev-like wrapper for LLMs, including vision models
A developer implemented a single-function Python wrapper mimicking Jev for LLMs and vision models by reading token log probabilities.
summary is enough The post provides a neat Python script and concept demonstration, but the underlying logprob trick is straightforward.
summary
The author demonstrates how to replicate Jev-like structured classification using standard chat completion APIs by setting max completion tokens to one and reading top log probabilities. They extend this approach to vision models by passing image attachments, enabling real-time webcam frame scoring. Commenters debate how this compares to proprietary Jev in terms of calibration, cost, and native image support.
key points
- Uses single-token generation and top_logprobs to extract relative probabilities for discrete choices or scores.
- Extends the request format to support image attachments, allowing vision models to score video or webcam frames.
- Includes a standalone Python script compatible with both local llama.cpp endpoints and OpenAI APIs.
who should read it
Python developers and AI engineers interested in lightweight structured outputs and vision model inference.
discussion positive
consensus
Commenters note that while the approach is intuitive and matches some Jev functionality, it lacks specialized reinforcement learning for probability calibration.
pushback
Some argue that open implementations rival Jev on accuracy and speed while adding features like image support that Jev lacks.
Empirically this approach is more accurate, faster and about the same price as Jev.
https://github.com/Mushroom-Systems/lichen
Highlights an empirical comparison showing that similar open-source approaches can match Jev's accuracy and speed while supporting images.
Generated by an LLM from the article and thread. Check the sources before relying on it.