Packing Binary Is Fun
An engineer builds a custom binary schema language and C++ compiler from scratch to shrink JSON payloads by 80%.
summary is enough A fun weekend project exploring binary serialization and schema compilation, but covers ground well-trodden by existing tools.
summary
The author builds a custom binary schema language and dynamic packer in C++ to shrink JSON payloads after wondering how binary serialization works. The implementation uses LEB128 varints, bitwise packing for field numbers and types, and a custom lexer and parser for schema definitions. While achieving significant size reductions over raw JSON, commenters point out that standard alternatives like gzip, Protocol Buffers, or self-describing binary formats already solve this problem.
key points
- The custom binary format uses LEB128 varints and bitwise packing to store field numbers and wire types efficiently.
- The project includes a custom lexer, parser, semantic analyzer, and dynamic JSON packer written in C++.
- Commenters note that existing tools like Protocol Buffers, CBOR, or simple gzip compression achieve similar goals with less custom code.
who should read it
Software engineers interested in serialization formats, binary protocols, and parser implementation.
discussion positive
consensus
Commenters view it as a fun yak-shaving project, noting that existing formats like Protocol Buffers, CBOR, or simply gzipping JSON make reinventing the wheel unnecessary for production.
pushback
A few commenters point out that the custom format lacks crucial production features like magic headers, versioning, and data corruption checks.
I'm missing some important parts that I would argue any (binary)format needs.
- a magic header - a version number - a crc or data corruption check
Additionally I would argue one would be better off writing a custom text parser instead of parsing a binary format in this case, this is similar to the debate about unixlike config vs windows regedit. I would prefer something other then json but sill readable as txt.
Even the json example provided at the bottom can be minified from 418 characters to 166 by replacing the field names with single characters and removing the spaces. Almost all of …
It outlines essential production features missing from the toy format, such as magic headers, version numbers, and CRCs.
Generated by an LLM from the article and thread. Check the sources before relying on it.