AI Coding will Prevent Expertise

35 points by anex9d

Very good read. Finally a balanced and eloquently write post on the subject.

A reality checklist:

If AI can just replace all engineering, then why not just ask it to "create a business that will make me rich"? Whats stopping AI maximalists to do so? Why stopping just short of that?
What is the point of an AI re-iterating on run-debug-fix loops? Human error arise from our biological incapability of having perfect memory. A computer doesn't need to be impaired by that. Why not getting it right the first time? Why are there so many advances in codding models yet we seem to be going back when it comes to getting it right the first time?

nickmonad

If AI can just replace all engineering, then why not just ask it to "create a business that will make me rich"? Whats stopping AI maximalists to do so? Why stopping just short of that?

I am no fan of AI, especially in software, but this doesn't land very well. The response will be "the models are good, but not that good", or "we still have to steer the output". Some might even say if the models were that good, nobody would have a job.

The second point you made is much stronger. Anybody who uses LLMs regularly has certainly experienced "You're absolutely right, that API doesn't actually exist."
- valenterry
  
  No, the first one is the current reply actually. If someone says "we still have to steer the output" then my reply would be "yeah, that's part of what we do in Engineering" and that's that.
- pm
  
  I am no fan of AI, especially in software, but this doesn't land very well. The response will be "the models are good, but not that good", or "we still have to steer the output". Some might even say if the models were that good, nobody would have a job.
  
  The point was to provide a reality check frame of reference. The expectation that you can now just do away with human technical expertise is [still] absurd. Because if you could, you could do away with everyone's job. All you see out there is companies bragging how they now don't need people and have sorted out all their problems with the AI silver bullet. But ironically they are still a a company with humans in it. Why bothering with just go half way? What would make anyone thinking that replacing an engineer is easier than replacing the CEO? It is in fact the oposite.
chrismorgan

I think it's a science experiment.

Then where’s the control? I have heard of attempts to study these things properly at any meaningful scale failing because they couldn’t muster a control group. And the slop ends up affecting those that want none of it, as their tools adopt it.

It sure looks to me like society in general is just diving in head first and hoping for the best.
- jesseb34r
  
  fwiw not all scientific study requires a control group. It sure makes it easier to draw verifiable conclusions, but whole fields of science such as Environmental Science and Sociology do much of their study without meaningful controls because there is no control earth or control society. Even in the absence of a control, you can still develop theories and test how well they describe the past/present and how well they predict the future.
codekobold

We do not know whether the premise is true that expert-level skills in designing software systems can only be reached by coding those systems directly, by hand. I very much hope the premise is wrong and novices can learn what makes a good system and what common pitfalls lead to one that's fragile, hard to maintain and further evolve by building these systems with coding agents. Because otherwise we are all doomed.
- pgn
  I guess the analogy here is something along the lines of :
  
  Consider a grocery store, the route to which is covered in potholes, snake pits, coyote urine and an assortment of related unpleasantries.
  
  You pay the low low cost of 20$ a month to have Jim Instacartman get your groceries for you from said store.
  
  There is little reason that you would every think "this road needs to be fixed!" unless you experience the displeasure of going through this road, since you yourself are not going down it. If you see Jim suffer, some empathy or sense of "lost efficiency" may motivate you to slightly improve this road, however.
  
  IMO the really dangerous bit is when the road is just bad enough that it's still usable by Jim, but any sane person would realize that it's crap. That's how you end up with cruft accumulating, and the fact that people so readily just offload looking at bad code to their AI of choice is going to make it easier to shit it out (the cost of reading it is paid less and less frequently).
  
  So TL;DR we're probably by and large screwed, at least while discount token rates persist.
cole-k
It's a weird time to be alive. Not more than two years ago, most of us here were saying GenAI was no more than "spicy autocomplete" at best --- and fundamentally could not develop further. Now I read pieces like this and this one about math that have shifted all the way to discussing the human effects from them being good at intellectual work.

Now the future outlook seems more like either
1. GenAI does not advance significantly further, and we're stuck with barely-passable intelligence on tap. In this world, the gap between skilled and unskilled developers widens unless society demands the unskilled developers cultivate the discipline necessary to learn in spite of GenAI. We never reach the promised land and software is just as bad as it ever was. I made similar predictions before, although I think the value of GenAI has increased. One can hope that open --- and especially --- energy-efficient models win in this world, and the resulting industry collapse does not wreck the global economy.
2. GenAI does advance significantly further, which brings about the world Altman et al. predict and perhaps even desire (the quotes in this article are, frankly, horrifying). I shudder to imagine the consequences. Frankly I hope the techno-optimists are right.