Generate fonts where every LLM token is the same width
A web tool compiles custom fonts embedding LLM tokenizers directly into font shaping rules, rendering every token with a uniform visual width.
summary is enough The technical mechanism of baking tokenizers into font shaping rules is clever, but the page is primarily a utility and setup guide.
summary
This web tool compiles custom TrueType fonts that embed LLM tokenizers directly into font shaping rules to equalize token widths or display per-token colors. It generates companion Discord themes and Slack userscripts to let users visually inspect tokenization and spacing in real-time. The accompanying documentation covers extensive edge cases, including whitespace bugs and browser rendering quirks across various model tokenizers. It offers a tangible way to observe how different models chunk text, though formatting and shaping limits remain.
key points
- Bakes tokenizer rules into TTF files using Pyodide and fontTools, enabling token-width visualization without runtime scripts.
- Supports multiple popular model tokenizers including DeepSeek, OpenAI, Llama 3, Qwen, and Claude approximations.
- Packages output into Discord themes and Slack userscripts to target specific users or agents in chat interfaces.
who should read it
AI developers and frontend engineers interested in tokenizer mechanics and visual debugging tools.
discussion positive
consensus
Commenters enjoy the playful utility of the tool and use it to reflect on how non-English languages and typos severely impact token efficiency.
I typed some German and it was breaking up words so much more than English. Not really surprising given tokenizers are optimized for most commonly used text.
Here's the token efficiency of a corpus translated into various languages and tokenized with the latest OpenAI one:
Language Relative tokens -------------------------------------- English 1.00x Portuguese 1.23x Chinese (Simplified) 1.25x German 1.31x Spanish 1.32x French 1.37x Arabic 1.38x Chinese (Traditional) 1.42x Korean 1.47x Swahili 1.49x Hindi 1.57x Japanese 1.66x Burmese 3.16x Amharic 5.78x Santali 13.70x
Source: "Tokenizer …
Shares a detailed cross-language token efficiency table demonstrating the heavy token penalty non-English languages pay compared to English.
Generated by an LLM from the article and thread. Check the sources before relying on it.