You can run Git on object storage if you re-make packfiles
summary
Standard Git packfiles are optimized for local disk access and memory mapping, making them prohibitively slow when used with object storage due to excessive network round-trips. To solve this, the author developed a new columnar packfile format inspired by CD-ROM cue sheets that allows for precise HTTP Range requests. This approach drastically reduces the number of S3 calls and significantly improves push and clone speeds for large repositories.
key points
- Traditional Git packfiles hide the compressed size of objects, preventing the use of HTTP Range requests to fetch specific data without downloading the entire bundle.
- The new format separates metadata into a fixed-width binary .cue file and data into a .bin file, enabling the server to pinpoint and retrieve individual objects instantly.
- Performance benchmarks demonstrate that the new architecture reduces S3 request counts by orders of magnitude and speeds up push operations by up to 14 times in some cases.
who should read it
Software engineers and distributed systems architects interested in Git internals, cloud storage optimization, or building scalable version control infrastructure.
discussion positive
consensus
Commenters generally agree that hosting Git repositories on object storage is a recurring idea with several existing implementations, but it faces significant performance hurdles. The primary technical challenge is Git's reliance on packfiles and delta-chains, which require frequent random-access reads that are inefficient and costly on object storage. While simple static serving via HTTP is possible, true efficiency for large-scale projects remains elusive.
top insight
The most critical technical limitation is that Git packfiles are not contiguous blocks of data, causing a single fetch to trigger numerous small range requests. This inefficiency mirrors challenges in data lake architectures, suggesting that a columnar storage model for packfiles is the only viable way to solve the 'fan out' problem at scale.
Generated by an LLM from the article and thread. Check the sources before relying on it.