Kolibri: A Sovereign Open-Weight Model
Aleph Alpha has released Kolibri, a 78B parameter open-weight Mixture-of-Experts model featuring a 3B active footprint, 1M context length, and a heavy focus on German language and European data sovereignty.
read it The release details a transparent, reproducible training pipeline and mixture-of-experts architecture that will deeply interest practitioners building custom language models.
summary
Aleph Alpha released Kolibri, an Apache 2.0-licensed, bilingual English-German Mixture-of-Experts model with 78B total parameters and 3B active parameters. Designed for regulated enterprise and public sector use, the model emphasizes data sovereignty, strong reasoning capabilities, and an anti-hallucination abstention protocol. The accompanying technical report outlines an automated training pipeline built at high velocity over three months. Commenters praised the exceptional openness of the technical documentation while discussing the implications of European sovereignty and a planned merger with Cohere.
key points
- Kolibri features 78B total parameters with 3.4B active per token, supporting a context window up to 1M tokens under an Apache 2.0 license.
- The pre-training mix includes roughly 21% organic and rephrased German data to ensure authentic cultural context without over-relying on machine translation.
- An automated training pipeline enabled rapid iteration between versions, utilizing exact quantile balancing and the Muon optimizer for stability.
who should read it
Machine learning engineers and enterprise architects interested in open-weight model architectures, training pipelines, and European data sovereignty.
discussion positive
consensus
Commenters overwhelmingly praise the unprecedented level of openness in the technical report, treating it almost as a blueprint for training modern LLMs. However, many note that dense models like Qwen compete closely or outperform it on certain benchmarks.
pushback
Some users question the sovereign European claim given Aleph Alpha's planned merger with Canadian company Cohere, while others report that the model's reasoning mode can lead to excessive token usage.
The paper explains absolutely everything as if it was a tutorial "how to made your own modern agentic LLM". They even tell how they made their dataset. https://aleph-alpha.com/downloads/tech-report.pdf ; It's the first time I see this level of openness.
It highlights that the accompanying technical report offers rare, comprehensive insight into modern LLM training pipelines.
Generated by an LLM from the article and thread. Check the sources before relying on it.