Writing Parquet files using Haskell

summary

The DataHaskell project added a native Parquet file writer to its dataframe library to enable better ecosystem interoperability. The implementation uses pinned mutable byte arrays to dynamically grow buffers without FFI overhead and employs batched loops to hit target page and row group sizes. It also addresses low-level IO considerations like avoiding OS page cache throttling through controlled chunk writing. Future roadmap items include multi-threading, broader compression codec support, and on-disk buffering for memory-constrained environments.

key points

1. Utilizes pinned MutableByteArray in GHC to dynamically grow memory buffers for column chunks and pages without FFI overhead.
2. Employs a batched and sub-batched core loop to ensure row groups and pages hit target byte sizes while minimizing bookkeeping.
3. Writes output in 256 KiB chunks to prevent OS page cache writeback throttling during synchronous file generation.

who should read it

Haskell developers and data engineers interested in low-level binary serialization and dataframe library internals.

discussion mixed