The August 17 outage, and the work ahead
4 points by dhruvp
4 points by dhruvp
It's an interesting read from the perspective of like system administration work. Volume went up and when that happens it is not uncommon to have systems fail in unexpected ways. At a previous job I worked at a credit card gateway and we would regularly find that with every exponential step up in volume, you would find brand new issues with stuff that had been rock solid up to that volume.
I am however no longer confident in GitHub as an engineering lead organization. I know a lot of great people who have worked there over the years, but most didn't leave on great terms with the company and its increasingly more Microsoft-centered approach to work. I (perhaps unfairly) don't have any confidence in Microsoft leadership. I've never seen such catastrophic mistakes that haven't resulted in the termination of some C-level staff. GitHub has started to get Boeing vibes about it.
Despite pouring all of the companies money and resources into it, Microsoft has failed at being a leader in AI. It mangled its hero product with Windows 11 and jamming Copilot down peoples throats. Then Copilot didn't move the needle at all, either culturally or financially for the company. Nobody likes it and they seem in a mad rush to rip it out. Then effectively they purchased all the gaming companies on the market to attempt to sell gaming as a monthly subscription, but seemed surprised that this subscription obviously meant nobody was buying these games at MSRP and so financially those studios suffered. Then they laid everyone off.
So I'm confident the people at GitHub are working as hard as they can. I'm confident they are good people attempting to fix these problems. I don't think they can survive in an organization as dysfunctional and poorly lead as Microsoft and I think any organization not starting the work of migrating off is deluding itself. This problem is going to continue to get worse, Microsoft will allocate additional resources to fix it temporarily and then a new quarter will start and there will be a reorg and that progress will be lost until the next major outage.