WalShadow: Sub-second Postgres replication to ClickHouse from physical WAL

summary

WalShadow is a new open-source engine that replicates Postgres data to ClickHouse using physical Write-Ahead Logs (WAL) instead of traditional logical replication. This approach significantly reduces latency and resource overhead on the source database while maintaining high throughput. By decoding physical logs outside the source database and writing native blocks to ClickHouse, it enables near real-time analytical capabilities. The system aims to create a unified data stack where transactions and analytics coexist with minimal complexity.

key points

1. WalShadow processes physical WAL streams through a four-stage pipeline involving schema tracking, parallel decoding, block building, and parallel insertion. This architecture avoids the need for logical replication slots and eliminates intermediate formats like JSON or Kafka.
2. Benchmarks show a dramatic performance increase over logical replication tools, achieving 200ms commit-to-visible latency and sustaining throughput of 289,000 rows per second. This allows the replication pipeline to keep pace with the source database without becoming a bottleneck.
3. The system supports the full replication lifecycle, including initial loads and complex schema evolutions like adding or renaming columns. It ensures data correctness by using source WAL positions to manage row versions in ClickHouse.

who should read it

Database administrators and data engineers who need to synchronize Postgres transactional data with ClickHouse for real-time analytics. It is particularly useful for those facing performance bottlenecks or operational overhead with standard logical replication.

discussion positive