A monorepo with 20 million lines of code grew from a manageable 2GB to an unworkable 150GB, prompting an investigation into its exponential growth.
The versioning tool Beach Ball generated thousands of changelog files, causing a separate versioning branch to swell to an enormous 130GB.
A Git expert found that an old hashing algorithm only used the last 16 characters of a filename, causing collisions that prevented proper diffing of changelog files.
The solution was a new "Path Walk" algorithm for `git push` and `git repack` that uses the full file path to avoid hash collisions and ensure correct diffing.
Developers can enable the new algorithm for pushes via a `git config` setting and shrink local clones using the `git repack --use-path-walk` command.
A new built-in command, `git survey`, was created to help developers identify large files, blobs, and binaries in their repository history.
Beyond the specific fix, general best practices like not checking in binaries and avoiding thousands of files in a single folder are crucial for repository health.
The new algorithm has shown significant size reductions for other large monorepos like Chromium, and the fix is being upstreamed to benefit the entire Git community.
42:13Kyle Daigle
37:05Scott Chacon
43:39Adrian Kosmaczewski
24:19Johannes Haux
42:29Lars Hesel Christensen & Basil Brunner
26:46Gerrit Stapper
32:29Martin Beránek
26:35Yishai Beeri
Jobs that call for the skills explored in this talk.
Rocken AG
Vesterling Consulting GmbH
Vesterling Consulting GmbH