Faster prompt lookup drafting in llama.cpp
A series of C++ data structure optimizations makes prompt lookup drafting in llama.cpp up to 140x faster and significantly reduces memory usage.
read it A detailed build log providing valuable C++ optimization strategies and data structure choices for LLM inference.
summary
The author details a series of performance improvements for prompt lookup decoding in llama.cpp, targeting nested hash maps. By replacing redundant copies, adopting segmented dense maps, optimizing inner token searches with sorted vectors, and using binary-fuse-filter-based constmaps for static caches, drafting speeds increased dramatically. Daniel Lemire contributed further algorithmic shortcuts to skip unnecessary score computations, compounding the total speedup to 140x.
key points
- Replacing nested standard unordered maps with segmented dense hash maps and sorted vectors eliminates cache-unfriendly linked list lookups and redundant memory allocations.
- Implementing Daniel Lemire's constmap structure based on binary fuse filters for static n-gram caches reduced loading time by up to 16x and cut peak memory usage.
- Short-circuiting the scoring loop when the most frequent follower fails acceptance thresholds avoids redundant mathematical computations.
who should read it
C++ developers and systems engineers working on LLM inference optimization.
discussion positive
consensus
Commenters praise the massive performance gains and Daniel Lemire's collaborative contributions, though a significant portion of the discussion addresses an interpersonal repository ban involving the author.
pushback
Some commenters criticize the author for airing governance grievances publicly rather than resolving them privately.
Generated by an LLM from the article and thread. Check the sources before relying on it.