There's No Limit to How Bad Code Can Get
14 points by rau
14 points by rau
Perhaps I'm just a naive youngster, but I feel like sometimes it's necessary to burn it all to the ground and start from scratch.
(In business contexts, the starting from would scratch have to happen before ditching the old codebase, I'm assuming.)
In my experience it's so rare for that to work.
You announce the old thing is irrecoverably drowning in tech debt. You spin up a team to rewrite it from scratch. Work begins.
Meanwhile the old thing remains a moving target: it's running the core business, so changes are still necessary. The developers working on it know that it's going to be made obsolete by the new thing soon, so they don't have any incentive to go beyond the smallest effort possible to add the new features. Technical debt continues to mount.
Meanwhile, the team working on the new thing are ambitious and probably a little naive. They start out at a great pace - it's greenfield after all - but as time progresses it becomes apparent that nobody fully understands the behavior and scope of the thing they are replacing. If it was well documented and tested it wouldn't need to be replaced, after all...
After months (or even years) without delivering value, the pressure is on to "ship it", so the new system is launched to handle a subset of what the old system handled - or often for some new feature that was too hard to build with the now mostly unmaintained old system.
... so now you have TWO systems in production - the janky old system that nobody wants to touch, and a new system which handles just a few production features and is 80% inactive code that is meant to replace the old system, eventually.
If you're really lucky the company won't have lost patience with the new system and will allow that work to continue. The longer this all takes, and the longer the old system stays in production and stubbornly continues to work, the higher the risk that "priorities have changed" and the new system total replacement work is abandoned, leaving you with two systems where you used to have one.
The best article I've read about completing this process responsibly is Migrations: the sole scalable fix to tech debt by Will Larson.
If I run into a situation like this in the future, my strong recommendation is going to be shoring up the old system with as much automated testing as possible and then seeing if targeted refactors can get it to the desired shape. My hunch is that in many cases that will have a much higher chance of success than the siren call of a greenfield replacement.
Hah, I just realized the linked article covers the same ground:
The closest option that a mega-corp such as Amazon has is what I call a side-channel, where they split off a team that builds a new, completely disconnected system with only the minimal set of features needed for some new use case. Moving forward they then have the option to direct more new use-cases at this simplified, separate system. Importantly, the old system must remain and be maintained (it's not an 'end'), because all the old use-cases still exist, and new organizational-level pain is felt whenever deciding which to use in the future
Everything that you’ve said was true, but only because you did the “ burn it all to the ground” step second. If you destroy the old system before the rewrite, then people can’t say that your system is wrong for doing it differently.
Obviously, this is not a serious suggestion, but I’ve seen too many green field replacements scuttled because they didn’t replicate behaviour that was wrong the first time. For example, heard a talk by a fellow who dealt with a bug where the sales tax charged to customers was higher than the actual sales tax and the company pocketed the difference. The company lawyers and accountants both confirmed that the new code was correct and the old code was committing tax fraud, but his manager insisted on replicating the old calculations because of Chesterton’s fence. Since no one in the company would explain why they had been engaged in a felony for decades, the coder didn’t know why the system acted the way it did and should not change a system that he didn’t understand.
Anecdotally I've seen cases of this unfolding multiple times, leading to a layer cake of partial replacement systems, reflecting different decades of the latest hot technology, dating back to a primordial system builtin COBOL. They were underway a third time replacing everything with microservices in k8s. I didn't see it through to the end but I have no doubt the powers will be will bail half way through this time as well as the billable hours keep mounting.
I’m always a bit baffled how someone can see a system with business logic they don’t understand and think “rearchitecting it will make it clear”.
To rearchitect it, you have to understand it. The difficulty is that rearchitecting something sounds sophisticated, while understanding the function of an existing system sounds like grunt work.
Once you do understand it, you’ll hopefully start to have ideas for rearchitecting things, but most of the rewriting crowd never reaches that point.
What needs to happen at the org/management level in order to properly encourage a real fix for this, or make sure it never happens in the first place?
I've parachuted or swan dived into a few "haunted graveyards". Once you've seen a few you can tell which systems have been worked by many hands without a definite owner.
Fundamentally, domain experience and continuity result in simpler, tighter systems. People want to contribute, usually by adding features. Each feature adds constraints and invariants, which dilutes a cohesive vision of the system. Accretion is easy, cutting things out is much harder to justify, so washing a system through multiple people reduces conceptual integrity.
A "fix everything now" mentality doesn't work. Climbing into a local maximum by simple changes does work: rename variables, comment discoveries in the code itself, document and automate workflows so the knowledge survives, and write tests to discover behavior and lock it in when you make changes. Once you start understanding things and have improved automated testing, you can make larger, safer changes.
The two biggest references to read about this are "Working Effectively with Legacy Code" by Feathers, and the much less known Object-Oriented Reengineering Patterns.
The problem is as soon as you start putting out the fires, people hear about progress and want to jump into this "safer" codebase and start adding features again.
"Where does it end? How does it end?"
...development velocity approaches zero because nothing can be shipped without a breakage.
The systems usually don't "collapse". The cost of code changes grows, and the speed and correctness of changes erode until people avoid it. Behavior gets put outside the system, and the problem slowly infects the code that interfaces with it, and it compounds as turnover affects those teams.