Show HN: JevBench, a reproducible benchmark for typed decision models
JevBench v1.5.4 is a new reproducible benchmark evaluating 106 decision models on typed classification tasks across intelligence, calibration, speed, and cost.
summary is enough The leaderboard structure and scoring methodology are clear from a summary, though the comment thread offers sharp technical skepticism worth scanning.
summary
JevBench v1.5.4 measures 1,624 decisions per system using a mix of open and sealed test sets to score decision-making language models. The benchmark ranks 106 systems using a harmonic mean of intelligence, calibration, latency, and cost, placing open alternatives like Cygnet and Winnow near the top. Commenters push back strongly with skepticism over the commercial hype and funding surrounding Jev, comparing its performance against basic fine-tunes of open models like Qwen.
key points
- Version 1.5.4 evaluates 106 systems across 904 open and 720 sealed decisions, incorporating latency and cost constraints into the final score.
- Open-weights alternatives such as Cygnet and Winnow-12B Q8 score competitively against proprietary models at the top of the leaderboard.
- Hacker News commenters express widespread skepticism about Jev's heavy funding given its performance parity with simple fine-tunes of open models.
who should read it
Machine learning engineers and developers evaluating small, fast models for classification and decision tasks.
discussion skeptical
consensus
Commenters are deeply skeptical of Jev's commercial valuation and marketing claims, pointing out that its performance matches basic, unoptimized fine-tunes of open models like Qwen.
pushback
A minority argues that specialized multi-environment training or architectural work could justify the platform's development, while the benchmark author actively engages to address integration questions.
$40m in funding, 2 years in stealth.
Performs on-par with SemIf which was built in a couple days and apparently uses raw Qwen, with no fine-tuning. SemIf runs in your freaking browser. Oh and Jev is twice as expensive?
Is it surprising that Jev consistently thinks it's Qwen?
I'm almost convinced that Jev is a scam. Take Qwen, fine tune it a little, tell investors it cost $10m, spend $1m on advertising, profit.
It encapsulates the thread's primary skepticism by questioning the project's high funding relative to its performance parity with quick open-model alternatives.
Generated by an LLM from the article and thread. Check the sources before relying on it.