ParadeDB Search Performance Improvements
ParadeDB closed a performance gap with PlanetScale's TIN search extension by optimizing fieldnorm locality and implementing MAXSCORE pruning in Tantivy.
read it A masterclass in competitive engineering, offering deep technical details on search engine internals, fieldnorm layout, and Blockmax algorithms.
summary
After PlanetScale released its TIN search extension and benchmarked it as 8x faster than ParadeDB, ParadeDB investigated and closed the performance gap. Rather than adopting TIN's use of Postgres ctids as document IDs, ParadeDB optimized its underlying Tantivy search library. They fixed a major locality bottleneck by storing fieldnorms per term instead of globally, and added a MAXSCORE pruning algorithm for multi-term disjunctions. The post also highlights benchmark discrepancies caused by TIN's common-term elision shortcut, which sacrifices ranking accuracy.
key points
- Moving fieldnorms from a global array to being stored alongside each postings list reduced disk buffer page accesses from 1,500 down to 30 for single-term queries.
- Implementing a MAXSCORE algorithm alongside WAND sped up multi-term disjunction queries by up to 8x, outperforming TIN on the benchmark.
- TIN's performance advantage partly relies on dense-term elision which skips scoring for common words, causing nearly half of query results to diverge from exact BM25 rankings.
who should read it
Engineers and database practitioners interested in search engine internals, Postgres extensions, and performance optimization.
discussion warm
consensus
Commenters widely praise ParadeDB's constructive, highly technical response to a competitor's benchmark rather than dismissing it.
pushback
Some discussion touches on the business defensibility of a narrow search focus and questions around the AGPL-3.0 license.
Massive respect for ParadeDB for seeing a benchmark where they were 8x slower, taking it as an engineering challenge instead of making excuses, profiling the hell out of it, fixing the actual bottlenecks, and publishing everything they learned along the way.
This is what healthy engineering competition looks like.
It highlights why this response represents healthy engineering competition by treating a competitor's benchmark as an actionable profiling challenge.
Generated by an LLM from the article and thread. Check the sources before relying on it.