AI Has No Wisdom and Neither Will You
24 points by cgrinds
24 points by cgrinds
In the future we will see more and more companies proudly boasting their “NO-AI” policy as a competitive advantage. And they will be right.
I appreciate the prediction, but I think it’s definitely wrong, even though I’m not someone who advocates against reading the code.
I don’t know how to make it testable, but I’m confident that the use of AI in companies is going to continue to grow.
That doesn’t mean we won’t see stories about “we tried to replace humans with AI and it failed, here’s why we’re begging to hire them back.” I expect a lot of those stories, as corporations overreach and get burned. But I do not anticipate them going to zero AI, either for their average office worker or for their developers.
I'm extremely bullish about AI-assisted development in general, so take that as a grain of salt.
I do agree with the foundational premise that AI is not very good at writing human-maintainable code. I put human in italics because the author does not state this but it is heavily implied that is what they mean. But writing code that both AI and humans use and modify is a transitional state: we will eventually move to a place where it's only AI reading and writing code. The important thing is that it writes code that is AI-maintainable, which is a different thing [1].
In the author's case of human-maintainable as well, I would probably look to a near-term future where a separate agent is given time overnight to refactor code on an ongoing basis, looking for refactoring opportunities to improve code holistically rather than solving the immediate problem that the AI was trying to solve in the first place.
I think we are still figuring out a lot about what an AI-assisted workflow actually looks like from end-to-end (thought to deployment). Automated overnight refactoring for maintainability/readability would be something that might be part of it.
[1]: And it perhaps is not even known what that means: do LLMs function better with long files and repeated code over short files and functions? What about writing Go with less "spooky action at a distance" even though Java has more training data? Do subclasses never get used again? etc etc
From what I understand, the things that empirically make code accessible to LLMs are the same things that make it accessible to humans - good documentation, consistent structure and style, robust test suites, use of well-known idioms, and so on. From the theoretical perspective, it also makes sense that many code maintenance benchmarks and training tasks are going to be derived from human activities, and that's likely what the models are going to learn to mimic. Between those two perspectives, it seems like the sensible default expectation would be that making code AI-maintainable would require the same kinds of decisions as making code human-maintainable.
There are also objective measures, such as computational complexity of algorithms or lines of code, under which chatbots tend to emit bad code. We have a good explanation for this: chatbot output is definitionally mediocre and the median hunk of wild code has bad computational complexity.
Didn't we spent the last twenty years trying to explain to managers who wanted to quantify performance that these metrics don't matter?
I think we were telling them that these metrics don't measure the performance of individual developers. I don't know anyone who would disagree that there is an optimal region for verbosity and complexity.
If the output is definitionally mediocre due at least in part to the fact that the wild code has bad complexity, what does that mean for the future when more of that same kind of code is out in the wild?
https://discourse.haskell.org/t/how-to-keep-enjoying-programming-in-a-world-of-llms/14705 was an interesting post on trying to get the good parts of llm assisted coding without losing that wisdom. Seems like it requires discipline though (so hopefully the slot machine theory of llms is wrong)