Inside GitHub's new Git storage architecture
30 points by ndr
30 points by ndr
maybe GitHub can manage 2 9s of uptime now...
https://isgithubcooked.com/ has become a valuable resource since Github's status page is often inaccurate
GitHub has been continuously writing about scaling up their infrastructure to accommodate the demand from agents. I think Cursor's report on how they're scaling their version control infrastructure is also pretty insightful; particularly the differences in design decisions they've chosen.
Part of what I've been wondering is how attached we should be to Git. Indeed, Facebook and Google had their own version control systems or forks (Mercurial/Sapling and Piper/Jujutsu respectively) to accommodate their monorepo style of development. OpenAI and Anthropic have both developed their own version control systems, along with infrastructure for hosting code.
New startups have also been thinking about this problem: Pierre Computer Company (recently acquired by OpenAI) has code.storage and I recently saw a YC company borrow ideas from OpenAI and Cursor in implementing their own Git server.
I'm pretty optimistic on Jujutsu being a potential next step in VCS. It's more intuitive for humans and reasonable for agents with parallelising work (through workspaces) and merging work (first-class conflicts mean you can defer handling them until you've rebased the workspaces onto your mainline). ESRC seems to be building around Jujutsu, and Tangled accommodates Jujutsu.
I haven't been able to find a ton of information about jj's storage formats, but I don't get the feeling they were trying to solve git scalability, but rather usability problems.
(Edit: and, sadly, the docs on how it stores and uses data are quite lacking -- they vaguely handwave about the internal implementation details of the rust code. They don't mention what the data used is, how it's structured, or give nearly enough information for me to understand what's going on, let alone implement it for myself.)
sadly, the docs on how it stores and uses data are quite lacking -- they vaguely handwave about the internal implementation details of the rust code
The docs quite clearly say that the git backend is the only production-ready one and stores stuff the way git does, and the only other in-tree backend is a demo backend.
They also mention that Google has their own.
They also store additional data alongside git, with their own format. This also seems unspecified.
Agreed on the Jujutsu optimism. I've had excellent luck telling agents to use jj if they detect that a repo is jj-enabled; all the major agents seem pretty adept at using it.
It's especially great when I'm telling the agent to make a stack of PRs. With plain git, all the coding tools I've tried seem to get themselves twisted in a knot with rebases and conflict resolution.
During the presentation,CEO of M$ talked about slapping 4 million cores, 150 petabytes of storage, and 50TB/s networking… in order to make it somewhat usable…