Kolibri – Tech Report [pdf]

summary

Aleph Alpha's technical report introduces Kolibri, an English-German Mixture-of-Experts transformer with 78.1B total parameters and 3.46B active parameters per token. Trained on 24 trillion tokens with over 20 percent German data, the model supports contexts up to 1 million tokens and features configurable reasoning effort. The report details the complete development pipeline, including custom data pipelines, hybrid sliding-window attention architecture, supervised fine-tuning, and reinforcement learning. Released under an Apache 2.0 license, it targets sovereign deployments in regulated European sectors like public administration and aerospace.

key points

1. Kolibri has 78.1B total parameters with only 3.46B active per token using a 6-of-384 routed MoE architecture.
2. Trained on 24 trillion tokens across pre-training, mid-training, and long-context extension, with German comprising over 20 percent of the mix.
3. Features a hybrid attention pattern mixing 40 sliding-window attention blocks and 10 full-attention blocks, scaling context length up to 1 million tokens.
4. Developed end-to-end using the internal Model Factory framework, incorporating extensive synthetic German data generation and rigorous PII filtering.

who should read it

Machine learning engineers, researchers, and technical decision-makers interested in large language model architecture, training infrastructure, and European AI sovereignty.

discussion mixed