Why I remain a skeptic
72 points by jbauer
72 points by jbauer
The not-so-quiet goal of the LLM push by big tech was to homogenise intellectual labour such that it can be made fungible. When labour is fungible you can swap it out whenever you like for an alternative, decreasing the bargaining power of workers.
This is the most important reason that managers want "AI". It's the same reason they wanted offshoring to work circa 2005. It's the same reason they push "nobody is irreplaceable". The real question is why software developers go along with using and installing "AI".
Years of a very strong bargaining position and high wages leads to poor class consciousness, that only becomes clear when the balance shifts. We're a couple years into that at this point.
Lately our meetups have half saying they're part of the laid off club, while the rest grieve their state of employ. I see them commiserate together (that's not all we do, the bulk is still nerding out about programming.)
Point is I'm kind of astonished. My latest memo would've been quite taboo just two years ago.
That's literally why Java exists, and large companies have already ironed this process out a long time ago.
Go is another language which comes to mind.
Tech Workers Coalition has been saying this since forever. AI is not a technology; it's a social project to destroy cognitive labor power.
The real question is why software developers go along with using and installing "AI".
they go along to keep their job, for fear of lagging behind in the rat race, or because they do not understand what's their individual and collective interest. If workers always acted in their collective self-interest, the world would be very different.
AI has been a technology and a research project since the 1950s. The fact the mainstream is only now paying attention to it doesn't change history.
the social project of automation cognitive labor goes back much earlier than that. The technological part has been evolving indepedently and they are arguably completely different technologies bundled together for narrative and marketing purposes, as much as a fan and a plane are different technologies even though they both function because of aerodynamics
Except AI has included a lot of different technologies for a lot longer than LLMs or the current companies and hype surrounding AI have existed. In fact, AI has had boom and bust cycles before, and those were built around different technologies that were still all AI, such as expert systems.
The real question is why software developers go along with using and installing "AI".
I think that one's quite simple. Software developers are more like "ordinary people" than we like to think. Most ordinary people just do what they're told. It's no surprise that that applies to software developers too.
Then among the rest there's a significant portion who think AI tools do help them.
(I'm not intending to pass judgement on any of these groups by stating this.)
If everyone refused to use productivity improving technology then we’d all be a lot poorer. The answer to the “rich people get most of the benefits” problem is a recurring problem of how society is organized and how the majority of people are not organized except by the powerful.
the solution is indeed not to refuse productivity improvements, but allocate the surplus generate by that improvement to the benefit of workers and society, rather than further accumulation by the few. This economic model would collapse if that happened, and therefore must be replaced. Programmers, in such a scenario, wouldn't be destroyers of good life, but creators of wellbeing.
I heartily agree with most of the points made in this post, but I think it's unwise to place a lack of efficacy as the foundation of your objection to LLMs. There are many applications where buggy, crude, unmaintainable piles of plagiarized slop can be widely viewed as acceptable (mockups, one-off scripts) or even valuable because they don't work properly but offer a "moral crumple zone" by redirecting blame from human decision-makers (insurance claim processing, resume filtering). LLMs are inherently unreliable, but if you're willing to burn enough tokens and take enough pulls at the slot machine (or cloak the LLM in enough layers of automated slot-machine pulling), a broken clock can be correct several times a day, for a price. Efficacy can- to an extent- be brute-forced, and that's why LLMs are such a dangerous temptation.
I think it's better to keep the ethical objections first. Using slop is fundamentally irresponsible and harmful to our society and information environment because none of the LLMs that are available today are constructed without plagiarism at a massive scale- not just copyright infringement, but more broadly the total erasure of attribution and authorship. Extruding prose-shaped slop infinitely faster and more cheaply than it can be read and understood by humans undermines the social contracts of written language and produces a chilling effect on all genuine communication. Deferring to opaque boxes of matrix arithmetic is a total surrender of the essence of programming as a process of understanding and codifying precisely how to do things into merely "getting outputs you want, sometimes, somehow." I could go on.
These objections will not change if the LLM industry spends another hundred trillion dollars to distill every recorded word in human history into their models, or to bake present-day frontier models into ubiquitous ten-cent chips that run on a coin cell. Boiling the oceans and shredding all our libraries might let an LLM get the right answer, but it will not make them right.
You can't convince a corporation with ethics. But if LLMs are straight up not worth the money, that's gonna be what gets the c-suites to ditch them. And it seems that at the moment the customers are neither paying the full inference cost, nor are they seeing the promised productivity benifit.
I don't disagree with any of this fundamentally, with the caveat that to influence people you do have to speak their language. In the common scenario where a company wants to lean in on AI, they have already decided they aren't bothered by legal or ethical concerns. You can try to poke at that but you're probably unlikely to succeed. But challenging whether a specific approach makes sense, or whether using it for this particular project makes sense, might be arguments you're able to win.
I think it is still worth it to talk about ethics, because that still can influence the collective bargaining, and union demands would be as hard to ignore as the lack of efficacy of LLM use.
A better way to look at it is to view the process as a genetic algorithm. It's not that you're pulling a lever in a slot machine and getting a random output each time. Agentic harnesses are effective because you have an evolutionary process. The model outputs something that's roughly correct, then it gets tested, model gets feedback and iterates. Through this process it converges on a solution that fits the parameters being tested. And that's not actually all that different from how humans write code either. You almost never solve a non-trivial problem in one shot. You write your first approximation and then iterate on it. The difference is that the LLM can do this process a lot faster.
I fully agree regarding the accountability argument however. LLM is a tool a human uses, and the user should be the one accountable for what they do with this tool.
I heartily agree with most of the points made in this post, but I think it's unwise to place a lack of efficacy as the foundation of your objection to LLMs.
Your entire comment is well-reasoned and well-written. I think it's useful to observe that the (likely) main reason many of us tend to lead with an objection about the poor quality of the output is because that was the most striking thing when the hype machine spun up. Art, writing and programming output from the machinery built around these LLMs was so laughably bad that many of us jumped to point it out. And still do.
"It would be bad to use this slop in production" seemed like a very strong objection. You're likely correct that pushing ethical objections to the fore is stronger case. But I understand why it's more appealing to argue about first order badness (crappy output) than the second or third order consequences that you first need to educate people about before you can make your ethical case.
Art, writing and programming output from the machinery built around these LLMs was so laughably bad that many of us jumped to point it out. And still do.
I think it's a mistake to argue this way, unless you're exceptionally clear about what "laughably bad" means and you believe your audience agrees with or understands you. LLM outputs nowadays were the stuff of science fiction a decade ago, no matter how bad we may agree they are. Even Google in its heyday before enshittification could not answer my query with bespoke code that compiles --- or an image of my design that resembles it in the slightest. Even in the circumstances where you are right to say the code/image sucks, from the lay perspective it can come across as griping about the lack of cup holders on the Model T.
I'm less skeptical myself, however many years in we are, that LLMs can't be used for my work. My skepticism remains in whether they should be. I'm waiting to see what happens when the bill comes due for the wonder machine myself, and I'm hoping that the rest of us aren't expected to pay for it if it's bad.
We are 4 years into this ‘revolution’ and the industry has almost nothing to show for it.
This argument is a little "no true Scotsman". There are things to show, but I'm sure the author has heard of them and dismissed each for a different reason.
2 was a marketing stunt, it's not meaningful in the least.
3 is too fuzzy; the author already mentioned that LLM-directed review seems to produce garbage when it comes to his projects, so they've already brought that particular case up. This leaves 1, which I think is a fair argument. Not good enough to excuse several entire industries completely falling apart while "AI" has trillions put into it, but it's a fair argument for something.
Yes agents will produce trivialities on review but they will often find bugs that are hard to find except on the slowest review, and even more so the more you tell them to use various testing techniques.
2 was a marketing stunt, it's not meaningful in the least
The fact that they ended up going with the Rust version and are now going to maintain that version forward is just a "marketing stunt"? Marketing for what, exactly? That seems a bit dismissive. I think it's a major statement, actually, that pure AI transcribed a famously-anti-AI Zig codebase into a Rust one, for (let's admit it) a hundred thousand dollars or so, and are now going with that codebase.
Note: As can be plainly seen from my Github repo, I am quite pro-Zig, so it would be unusual for me to take this stance unless I was moved by the evidence.
seems to produce garbage when it comes to his projects
Sample size of 1 problem. Also, which AI? Opus 5 produces garbage in my experience, but Fable 5, Sol XHigh and Grok (yes... unfortunate for the Musk-haters) do not.
Not good enough to excuse several entire industries completely falling apart while "AI" has trillions put into it
Um, which are the "several entire industries that are falling apart," again?
LLM-directed review seems to produce garbage when it comes to his projects
There is a very weird effect where people get massively different tiers of performance out of AIs.
I use a Satisfactory mod (Ficsit Networks) that has yet to be updated to the current major version of the game. There's a fork somebody is running where he just has Codex read the issues every few hours. (Codex also wrote the port.) I have not even talked to the guy in days. Every issue and crash that comes up, we just slap it into Codex and have a fix the next day. There's issues where the bot is stumped (there's a weird rendering lag issue with the code ui for instance) but it's otherwise batting nine out of ten so far. As far as I can tell, it's just comfortable with this codebase and doesn't have any difficulty making changes. And (to my awareness) it's not being prompted, it's not being managed or guided. It just reads issues, responds, and makes releases with fixes, autonomously.
At the same time, people report that they can't get the LLMs to do anything in their codebase without causing further issues. I believe their experience, but something seriously strange is going on here, right? It's like we're bifurcating into people who have the magic aura that makes AIs work for them and people who don't.
It is very odd and I’m disappointed that we haven’t seen any studies on it yet, possibly because the big labs don’t want to admit that people who don’t get value from AI are out there.
My hunch is that people who feel that way are trying to one shot things that are obvious to them without the agent having access to the right tooling and context to figure out if what it’s done is correct. Possibly in codebases with minimal indicators of intent. But this really is just my hunch.
What is it to show? Most software got worse, we have proof of that.
The new software that is successful. Aren't they AI tools to create AI tools?
2 even proves my point as it made Bun lose adoption from important projects, not gain it. So it made the software worse for most people.
I took it as nothing "positive" or "groundbreaking" show for it, as I have had the same feeling. It has only introduced more problems, or at least its the problems that have become more prominent over advancements in the products being built.
I think that's the thing. Aside from the myriad of "tools for developers" being built on VC money in the hope that something will stick, most of the startup products, and a lot of the enterprise, are just as sloppy as before. Perhaps a little less formatting issues since LLMs are good at that, and more architecture problems that never get realized because these types of products never reach the scale where the cost is obvious and visible.
We just have all of these products faster now, and the problems of having so much slop, that we now call AI slop, is making the problems more prominent.
People fail to realize the dichotomy here. It's not
It's actually a trichotomy:
Sure, some of b is going into a as AI gets better, but mostly it's c => a. A lot of that is "slop" that deservedly goes away soon after its creation, but not all of it.
As an example of c=>a, i.e. something that probably wouldn't have existed without vibecoding: Qobuz multiplatform client https://github.com/vicrodh/qbz -- sure, it's not making anyone any money (except perhaps to Qobuz itself in a small way), but it sure provides a lot of value. Yogthos's Jolt seems to be another one from this thread. I'm sure there are many others all over the place, many not so public.
“The AI PRs I receive are rubbish” is severe selection bias.
Maybe, but it matches a broad trend in open source. How many projects are locking down or banning LLMs due to contribution quality?
Please tell me why this is being blamed on the tool instead of the person, again?
Who or what is responsible, again? This is literally the first thing about this that people need to understand and the open-source projects you refer to are fucking it up from the get-go. And yes, that includes my fave language of late, Zig. Because as it turns out, no one person understands everything, and Andrew is simply wrong on this.
IT IS ALWAYS THE PERSON WHO IS RESPONSIBLE. If someone's PR makes more work for you than not, give them 1 (final) warning and then if they do it again, BLOCK THEM. If their work contribution is actually fine, but you know it was done with LLM assistance and then you block them due just to that reason, THEN YOU ARE THE PROBLEM, NOT THE AI, BECAUSE YOU ARE IN A CULT. (A system of belief or bias that is not empirical.)
If this means establishing some sort of trust mechanism (which GitHub doesn't make easy, but could), then that is vastly superior to just outright "banning due to possible LLM assistance". Because 1) that assistance is rapidly heading to a point where it will be indispensable, which will make this nearsighted effort look really fucking stupid, and 2) the people who learn to work WITH LLM's instead of against them are going to absolutely end up having superior work output quantity AND quality.
Honestly, I welcome it. The more people reject LLM assistance, the wider the market is for people like me who are doing the complete opposite. DO IT! >..< The line is drawn and the die is cast! Let us see how it plays out.
Please tell me why this is being blamed on the tool instead of the person, again?
because the tool is what is allowing people that don't know about software development to create the PRs and push them posing as professional software engineers.
So, like I’ve been saying, this is the fault of those people, then, and not the tool. If people oppose this, GitHub should’ve created some kind of reputation or trust system by now, and they’ve done essentially nothing. Which is Github’s fault, not the tool’s.
this is like saying the kevlar armor's manufacturers are responsible for wars, not arm manufacturers. Also in this specific example, GitHub is both. Crazy logic.
I don't follow your reasoning here. I think you're framing this as a defensive argument, like, maybe "AI companies should have also provided the defenses against the ways they can corrupt software" or something? Like arms manufacturers should also provide the Kevlar? This is what sounds like crazy logic to me.
All I'm saying is that a trust or karma system could solve this problem, and I don't see one coming from Github. People shouldn't vibe-code PR's they don't understand, end of story. A simple filter on new Github accounts for example (as dumb as it is) would prevent most of the crap, I bet.
no, the point is that AI companies are the aggressors against the common goods and against labor and you put the burden on who should "arm" the victims. Also karma systems never worked, they are very easy to game and the internet collectively moved on from that quite some time ago.
Honestly you're coming across as unhinged here so I'm not sure if there's any discussion to be had. Can we dial it back?
Please tell me why this is being blamed on the tool instead of the person, again?
Because the problem very clearly got much worse when LLMs were introduced, and they lower the bar to make the bad contributions. A project has to deal with the reality they face. If they now get 100x the PRs and 99% are slop that wastes their time, they have to enact policy to keep the project sustainable, even if there are 1% of responsible users.
Honestly, I welcome it. The more people reject LLM assistance, the wider the market is for people like me who are doing the complete opposite.
I think this is what skeptics including the OP have been gesturing at, too. If LLMs are such a competitive advantage, it should become apparent pretty quickly. It has not yet. There isn't a sudden wave of high quality vibe coded projects that people prefer. If you think you can outcompete anyone, demonstrate it.
Can we come at this via the gun debate as a comparison? The idea being that maybe arming literally everyone with a gun, and then seeing a surge in gun violence, is not necessarily entirely the fault of the people wielding the guns. Is that a line of argument/analogy that you think might be more amenable here? That giving too many people the ability to make code "that seems to work but which is actually deeply flawed" is the problem?
I wouldn't disagree btw. I just think that if I got a single PR from someone and it was a mess, I would blame them, and not the tool they used. I'd warn them, and then if they didn't stop, I'd ban them. I wouldn't blame the tool. This is actually EASIER to solve than the gun problem, because any open source project can stop the "bullets" simply by not accepting code from people who either 1) fail to demonstrate aptitude in the language (regardless of source), 2) have nothing to vet for them.
Sorry about the unhinged-ness.
The tension is automatically dialed very high in these discussions and I do appreciate you trying to engage.
I think it's less about who should be blamed for the phenomenon and more about what to do about it. To stick with your analogy, if gun ownership increases have led to an increase in gun violence, some people will advocate for gun control - that does not mean that they blame every gun owner individually or think there is no proper use of a gun. They are trying to solve a problem.
If you read the posts outlining rationale behind LLM policies (e.g., Rust's) you'll find very little blame but also description of a real problem that is leading to widespread burnout and resource challenges, and needs to be addressed.
For my open source projects (3K-5K stars, so not huge but not small) I'm actually seeing the vast majority of AI PRs are quite good, small, targeted fixes. I see the occasional large AI slop PR but I just reject it and move on.
This is of course, also anecdotal.
Your own biases are pretty clear from your post history here and on HN, you are obviously all-in; there can be almost no sensible discussion at that point.
That's ridiculous; I am actually welcome to all sensible discussion; the problem is that I encounter not just skeptics but ignorants who are both skeptical AND who have no remotely significant experience with it AND who think their opinion somehow still counts. Can you at least see how that is frustrating?
You would be right if this was an unfounded belief; it's not. I've hand-coded for 25+ years and agent-coded for about a year now. I have an informed opinion, by definition. (And on Twitter and other platforms, I am OFTEN bitching about the LLM-lying, etc. I just retweeted this! https://x.com/gergelyorosz/status/2088567020623151475?s=46&t=qWag1P6krA-5zkoLCeptNw )
This reminds me of the old "Mac vs. PC" arguments where 99% of the Mac people had used both Macs and PCs extensively and still preferred Macs, but 99% of the PC people had only used PC's (or, sorry, "that one time I used a Mac and couldn't find the Start menu" types), and the latter somehow believed their opinion was informed and thus mattered. You can guess which side I was on, on that debate, as well; it fits perfectly.
(aside: I HATE that there are managers essentially forcing devs to use AI. Absolutely the 100% wrong way to go about that, it’s like the Nazis controlling France and assuming it will just stick and that there won’t be a revolt…)
Anyway, human DNA is vibecoded. >..<
That's ridiculous; I am actually welcome to all sensible discussion; the problem is that I encounter not just skeptics but ignorants who are both skeptical AND who have no remotely significant experience with it AND who think their opinion somehow still counts. Can you at least see how that is frustrating?
Can you see that you sound like someone under a heavy AI psychosis to lots of knowledgeable experienced people - that you are calling skeptics and ignorants?
i’m also a knowledgeable experienced person, unless you are literally claiming that experiencing firsthand the thing you’re discussing, to be the opposite of “experience”, which would be a fantastical gaslighting attempt
Anyway, human DNA is vibecoded. >..<
I assume this was meant as a joke, but it made me think.
I can't come up with a world view where this holds true. For those who believe in evolutionary theory, there isn't a prior-existing set of open weights that you feed into an agent and leave alone to get to human DNA. And for those who believe in intelligent design, although that preexisting that model does exist, the outcome isn't left up to chance and is analogous to hand-crafted code.
Was there a different world view you had in mind?
When I write code with agents, I need to focus on the constraints. I call them controls, because it's more than just "tests". It's literally... all the possible tests. Upfront, as TDD (TDD is not optional). Because the AI lies, can make terrible design decisions when not monitored, and... let's just say that I'm occasionally still cleaning up the stuff I vibecoded 8 months ago. And yet... I still end up vastly more productive at the end of the day. By "productive", I mean "a piece of software that did not exist before which is now able to do work and spark joy". Is there "one step back" to pay for the "two steps forward"? Absolutely and I would never lie about that (unlike the AI). Do I occasionally see bugs, or bug classes, I would have never written myself? Yes. Have I learned to prompt or write controls around many of them to reduce their occurrence? Also yes.
Let's just say I've seen some shit.
It occurred to me one day that NATURE is the constraint on the evolution of DNA. DNA is thus coded by controls ("react too slow to danger, die")... and an unknown force that used to be called "vitalism" before that fell out of favor, but it strives to live and thrive. (Neither AI nor code exhibit that force. I have to be that force for my agents.)
Belief disclaimer: I believe I have a soul and that there is an afterlife because of NDE accounts. I do not subscribe to any religion anymore because religions are corrupt, but I do think Jesus said some wise things. I do not believe we were "designed by hand"; I do think that whoever or whatever cast the original dice that resulted in us, perhaps did it intelligently, but I wouldn't call that "intelligent design", I think we ourselves had a hand in our own development, and continue to.
I still do not use LLMs for anything I actually care about.
I do and it works uncannily well. Give it a try. Seriously.
To be fair, it's hard for anyone not using LLMs to trust anything that someone using LLMs says about using LLMs, because those people not using LLMs have seen the garbage slop that people using LLMs have dumped on them to read / accept / review / clean up.
I'm on both sides of the fence, and it's a painful position to be in, as you can imagine if you can visualize that idiom. LLMs do some things amazingly well, and I do use LLMs for those things. And I try to use them more and more for those things, without losing my ability to figure out those things myself. But I also get patches that were obviously made by LLMs, and where the person submitting the patches clearly has no idea what the LLM did and why it worked to pass the tests and why there's no way in Hades that I can accept that pile of slop that will inevitably lead to the collapse of human society as we know it.
I'm still firmly in the camp of: Never let the LLM spit out anything that you can't fully understand, and furthermore, make sure that you do fully understand it, even if that increases your own time cost by 10x or whatever. If the LLM output is good, and your total time cost represents a time savings, then that may be a good use case for an LLM.
What I genuinely don’t understand is how articles like this still have such widespread support in 2026 despite all the evidence of tremendous change in our industry. And this change is coming from everywhere.
Because most people do not attribute the changes to the intrinsic quality of technologies and approaches labeled as AI. There's a technical discussion, and a social discussion, for example about the war waged against tech workers by American VC funds to cut salaries. The changes are real, but a lot of people can believe that and at the same time believe that software can't reliably be produced with AI assistance.
about the war waged against tech workers by American VC funds to cut salaries
I think that war ist mostly waged in people's heads right now. That's quite far away from what people are actually discussing in that space.
In what spaces? VCs? They seem to be talking about it every day in the news and flaunting how well they are destroying society.
In what spaces? VCs?
Generally in the space of people raising money or working on AI. Beyond the folks of Anthropic which generally believe their creation is going to destroy jobs, the general vibe is not that there will be destruction of jobs, but that "growth will go through the roof".
I'm not sure what news you are reading that suggests that people out there are determined to wage a war against tech workers.
This was fifteen years ago, but it definitely made the headlines: High-Tech Employee Antitrust Litigation.
Also, "growth will go through the roof, but wages will fall towards the floor" is probably more accurate.
100% agree on the problems with using it strictly for code generation; as well as the lack of good arguments and proofs that it helps with creating non-trivial software.
But to be honest, I find LLMs to be a massive boost when used as a thinking partner/research assistant. I would say that in many contexts, my productivity is at least 2x if not 3x or 4x thanks for this additional, enhanced cognitive ability they provide me with ;) Of course you need to understand and validate their output, but man, it helps so much; it feels like having a smart assistant available all the time; never tired and always eager to help
Article:
poor-quality demoware that no reputable organisation would touch with a barge pole
I cannot avoid remembering a related quote:
My job was to lead a team of experts in building something that reduced the degree to which Cliff had lied by just enough that, when the person who signed the check pressed the Power button, we wouldn't all be sent to jail.
(Snowden, about defense contracts)
So, maybe no reputable organisation, but some well-funded ones…
The number of blog posts celebrating LLMs or conversely explaining away LLMs are legion. The next batch of trained LLMs will now know what they need to know to defend or dismiss LLMs. Mission accomplished.
Assuming, for a moment, that we could set aside the context in which these tools exist and have been developed, and consider only the tool itself, I am very curious about the ethics of a certain style of use.
I have been trying to decide whether and how to fit LLMs into my work. I recently put some thoughts together on a very constrained approach to the use of LLMs, following a set of very broadly stated principles:
The last point hints at it, I think, but the idea is constrain use to only that which improves human capability. In other words, the tool must be a tool for human growth.
For example, I recently read some news discussing challenges schools in the United States are facing in developing early childhood literacy. It seems that one ongoing debate involves the preferred methodology, and there are proponents of the “phonetics” approach and of the “whole word” approach. My curiosity was piqued by considering how these approaches might compare for early language learners of a language organised primarily around non-alphabetic writing systems.
It turns out that I can speak one such language to a certain degree, and I can read its non-alphabetic writing system (though I use a typing-used orientated around a non-Latin semi-syllabary.)
Of course, in English I can generally engage with academic research far outside of my own expertise. This isn't to claim that I need not acknowledge my own limitations, but rather than a general university education has prepared me to perform basic research tasks and to analyse and assess basic research results. So I feel fairly confident that I could, in fairly short order, dig around to find the textbooks and studies and research papers that form the basis of these two teaching approaches. It may take me a while to be able to fully understand the research and to truly draw my own opinions (and, to be honest, I strongly believe that if you're outside of a field, specifically if you don't have the ability to conduct your own counter-studies, you are very limited in what you can draw from a research study. This is why when I watch fitness videos on YouTube, the moment someone utter the word “ghrelin,” I close the tab.) In any case, it may take me a while, but I can get started today.
In the other language, I am at a total loss. I can read the newspaper, sure, but academic vocabulary in fields that outside of my direct experience is just too difficult. (I read an English-language book on a topic in ethnomusicology the other week and every time the author wrote the word “imbricate†,” I had to go for a walk around the block to calm my nerves. Just imagine if you weren't a native speaker…) Furthermore, anyone who has had to do basic research in a university undergraduate setting very broadly knows where to find and acquire research artefacts (and, for the English speaking world, they're usually online at some .edu or in some journal at a .org or maybe published via an FTP site on a .gov…)
So, in this particular circumstance, I feel completely unable to satisfy my curiosity. I utterly lack the skill to do basic research on this topic, and I don't really even know how to go about developing that skill, other than, say, going back to college… in another language… in another country. (Also, if you want to spend a couple of hours indulging your curiosity on a topic like this, you need very patient friends, family, neighbours, or work colleagues…)
So I spent some time asking an LLM. I am, in general, wholly disinterested in entertaining its outputs that are related to matters of judgment. I don't really even care that much for its summaries. But it's so much better at searching than I am (especially across a terminology barrier!) that it's just ridiculous. I can spend an hour roughly translating terms like “phonological awareness” or “orthographic approaches,” and I can corroborate my translations with source language materials, and yet still not find anything in my searches or miss whole swathes of documents by not accounting for common variations.
Now, I don't care enough about this topic to go to school for it, but I care enough about it to spend a few hours reading about it and getting some background knowledge, especially if I develop other knowledge or capabilities along the way. In this particular case, I would at least like to be better at searching for other (often non-academic) topics in the non-English language, and I would at least like to expand my vocabulary a bit.
In this case, it seems not only possible but uniquely beneficial to rely on the support of an LLM, and it seems like there is a way to use the LLM to build human knowledge and self-sufficiency. The long-sequence of prompting is pretty easy to guess. Mostly, the form is just slightly more indirect: “find DOIs for research papers on …” → then I go read or skim the paper → “this paper mentions … but doesn't give a citation; find other papers with the same reference, and find the citation” → then I go review the results (or do a few refinement cycles until I get the desired result, or hit a dead-end that I'd have to confer with a human about) → “where else is this paper published or mentioned” → then I review and save these sources for later.
The last step is key: if I'm going to use the LLM, I want to use in a way which allows me to not have to keep using it. Thus, I want my prompts to be unique, otherwise it means I haven't learnt anything. I want it to support me in doing something, but develop the capability myself, so if I keep asking the LLM questions, they are always at (and pushing forward!) the frontier of my knowledge, and I want to eventually develop capability so that I can outperform the LLM.
(In fact, through this approach, I have, in some narrow areas, become faster than the LLM.)
This seems like a very tool-based used of the LLM, and I don't know that it comes in conflict with the complaints the author of this article brings up. What are its ethical ramifications?
† “It is, on the contrary, fundamentally relational, conferred on individuals by communities that recognize them as partners in ongoing conversations, as imbricated in common ‘webs of interlocution’ (Drews Lucas 2017,) and, potentially, as committed to collective political action.”—and that's the English version…
I welcome any LLM skeptic to build something equivalent to Jolt in three months.
FWIW I designed the Kit programming language and built its compiler during my 12 week parental leave back in 2018. Not only was it within three months, I was completely sleep deprived the whole time (hence why I thought designing a programming language was a good idea.) It included a website with a WASM-powered playground, and an NES emulator implemented in the language itself.
I'm not sure there's a ton of value in these "compiler-measuring contests" though.
It's pretty easy to build a toy language in a few weeks. That's not in any way comparable to having feature parity with a mature existing language and being able to run a wide set of official libraries. It's pretty disingenuous for you try and equate these things.
And I get direct value from Jolt because it allows me to write native apps using Clojure without needing the JVM. So, try to at least having a bit of intellectual integrity when making your counterpoints.
I'm really not sure why yours is real and mine is a toy and I'd love it if you would enlighten me.
feature parity with a mature existing language
Kit has seamless interop with C. It parses C headers and can use C interchangeably, similar to Zig. It also worked with third party libraries. The NES emulator I linked was using SDL - and parsing and understanding all of its headers to do so. That even includes making constants implemented as C preprocessor defines fit into Kit's type system which was not trivial.
I'm glad you're getting value from your thing too but I don't think one or the other is obviously a toy here, and I don't appreciate being accused of dishonesty.
Edit: Maybe we can come up with a more objective way to measure which one is a "toy" and which one is "real?"
Any other suggestions? Or do you want to retract the point that no one can do something as amazing as what your bot did in 3 months?
I'm not talking about having interop here. I'm talking about rebuilding an existing language that's been evolving for around 20 years now with all its quirks and subtelties. Not only does Jolt match Clojure semantics, it also implements a large subset of Java semantics and API which existing JVM Clojure libraries use. All of this is done on a completely different runtime, and it performs comparably to the JVM in a lot of cases.
I'm not sure how number of stars or lobse.rs discussion is a measure of complexity in your mind. Not to mention that your language has been around for nearly a decade while not getting significantly more attention than Jolt. So, not sure that's the comparison you want to be making.
An obvious suggestion is to look at the amount of code and actual functionality implemented. The fiber implementation alone in Jolt is more complex than most of your toy. The fact that you keep doubling down on pretending that these things are in any way comparable is frankly incredible. So, no, there's nothing to retract here. You clearly haven't spent even a few minutes to look at what Jolt actually is and what it does.
If we want to measure "complexity" and especially lines of code, I'm quite sure your agent-produced code is going to win, hands down. Whether that's desirable is a different question.
Again, if you think you can write something comparable with less code go ahead and put your money where your mouth is. Also, feel free to look at the amount of code in the official Clojure implementation and the Java standard library. The complexity here is inherent in the problem being solved, and doesn't come from code being agent produced as you're trying to insinuate. Either you genuinely don't understand what you're debating here or you're intentionally trolling. Which is it?
There's a lot of goalpost moving here which doesn't make for a productive discussion, but it's a bit silly to issue a challenge like this - it only takes one counterexample. I think people are capable of a lot more than you imagine they are.
For what it's worth I think your compiler with interop with C is a relatively comparable project. Clojure and Java do certainly have their complexities after 20 years. But so does the C abi.
I don't feel like contributing to what is obviously an unproductive conversation. But I did want to ask, how did you motivate yourself to work on a project like kit? How did you avoid short/long term burnout?
It was a little circuitous. I'm really intrinsically motivated to make games, which is how I got started programming as a kid. Kit was an extreme case of yak shaving where I had been using Haxe which I enjoyed and contributed to, had some ideas on how my workflow could be even better which the Haxe team didn't feel fit into that language, and decided to experiment myself. Eventually I ran into enough pain points that I decided to remove language from my complexity budget and moved to C++ where things are more stable than either Haxe or Kit. Putting the effort into the language and compiler wasn't moving me toward my real goal of making games, but it was a great experience nonetheless, and I'm still using the engine I built during that time, now ported to C++.
There aren't any goal posts moving here. What I said originally was that I was able to make complex project in under three months, and I challenged people to try building something on the same scale by hand.
You sealioned into the thread and tried to make an equivalence with your toy compiler. Then when I pointed out that what I built is far more complex, you started saying that I'm moving some unspecified goalposts. It should be obvious to anybody with even a modicum of integrity that you have to compare apples to apples to have a meaningful discussion.
I've been doing software development for over two decades now, and I know there's no way I could write something like this by hand in that amount of time. In fact, there have been plenty of abandoned attempts to port Clojure to run on Scheme over the years, and I'll leave it as an exercise for the reader to figure out why nobody managed to complete this herculean task manually before.
This is the entire problem with making claims like this.
No one can verify if you're correct or not. You are an anonymous poster on the internet making a claim of valuable complexity that indicates a category distinction. If you are correct, and your project is in fact, significantly more complex, then the level of effort to verify that claim is also significantly higher, to the degree that no reasonable person is going to do it.
Therefore, it follows that you can't really claim complexity as any sort of argument when posting anonymously on a public forum without anything else, success, reputation, etc, to back it up. You may know for a fact that you are right, but no one else can.
Anybody can download Jolt and run the benchmarks in the repo. Anybody can go to the Jolt page and try running existing Clojure libraries listed there https://jolt-lang.net/docs/libraries.html Anybody can go to the examples repo and run example apps there https://github.com/jolt-lang/examples
Anybody can go to the #jolt channel on Clojurians slack and see what people are currently building with it and what their experience is like. I'm also not posting anonymously. I've been around Clojure community for over a decade now, and I have books published, I maintain a web framework, and have lots of open source libraries with literally millions of users.
So, don't pretend that this is some inscrutable thing that nobody can verify.
I'm not pretending that. I'm just trying to get across, and it's a difficult concept to convey so I will be inaccurate, that the sort of verification you're gesturing at is not necessary for someone who already sees why you're right, and is unlikely to be performed in this context by someone who cannot. Hence, I believe claims presented towards the latter group in this unequivocal way are more or less pointless.
You are right, I was wrong to call you anonymous. The concept I was gesturing at was effectively anonymous for the purpose of how folks are going to consume a claim like this.
Again, this isn't some abstract debate. We are talking about a concrete compiler implementation for a well known language. Seems like the problem isn't with demonstrating what I said factually, but that there is a group of people who just want to argue without spending the time to learn the facts. And that's entirely their problem as far as I'm concerned.
I've presented a claim that I can defend on factual basis. If people want to honestly debate this claim they have to spend the time to investigate the facts. As the saying goes, no investigation, no right to speak.
I've presented a claim that I can defend on factual basis.
You can trivially defend achievements and capabilities of your product, you can easily defend its current complexity.
Good luck however defending essential complexity, without putting the burden of proof on the naysayers...
As the saying goes, no investigation, no right to speak.
...There you are. But that's not how it works. You can't change people's minds without presenting evidence, and the kind of evidence you have access to with respect to essential complexity is likely very hard to convey.
I literally did present a concrete project as evidence to discuss, but it's very clear that not a single person in this thread actually wants to engage with it.
We can also see the essential complexity by looking at the size of the official Clojure itself, which is actually bigger than Jolt. So, that gives us a pretty good idea of what the essential complexity here is.
It's kind of interesting how skeptics react when presented with a concrete project, and an opportunity to discuss the development process with the author. Strange how nobody here is able to provide any constructive criticism or discuss it intelligibly.
What's very clear from this thread is that people don't actually care about evidence one bit. If you've already made up your mind, it's much easier to just make personal attacks and jibes.
We can also see the essential complexity by looking at the size of the official Clojure itself, which is actually bigger than Jolt. So, that gives us a pretty good idea of what the essential complexity here is.
This. That's the kind of outside view argument I was looking for (and have possibly missed).
As the saying goes, no investigation, no right to speak.
Okay fair enough. I merely worry that this inevitably means that the camp that has the most power to publish in volume will inevitably be the only camp capable of performing speech, regardless of quality distribution, due to the asymmetric effort implied in the verification effort.
I don't really follow what you're even trying to argue here. We are discussing two compiler implementations here, and whether they're comparable in terms of the scale of the problem they solve. What does any of this have to do with publishing in volumes?
Seems to me that actually understanding the subject people are attempting to debate is the bare minimum requirement. I'm not sure how you propose having any intelligible discussion where the participants don't even bother having a minimal knowledge of the subject they're discussing.
I think where you’re losing your audience is that you’re calling this other person’s language a toy, then failing to substantiate why it’s a toy. No-one is doing Jolt down.
If you look at the repo the author linked, it's very clear that it is a small hobby language. It's precisely the kind of thing you could hope to write by hand in 12 weeks or so.
The entire compiler is under 13k loc of Haskell. The type checker looks like the single biggest subsystem in the whole thing at about 3,500 lines. The README is upfront about calling the whole thing pre-alpha with features not fully implemented.
Kit has no persistent data structures, no lazy sequences, no concurrency model, and its metaprogramming is a term rewriting system over the typed AST.
By contrast, Jolt's compiler front end alone in Clojure weighs in at almost exactly the size of Kit's entire compiler. And that's the smallest part of jolt most of which is written in Scheme with over 50k loc. Overall source is literally an order of magnitude that of Kit while Jolt's tests alone are bigger than Kit's whole project.
And it's not just about lines of code, the projects are genuinely in different categories in terms of what each has to implement. Kit is a statically typed systems language that lowers to C through one backend, a unifier, and a small prelude. Jolt is reimplementing the semantics of a mature dynamic language, persistent data structures, dynamic dispatch, a reader, a loader, a namespace system, an nrepl server, FFI, and it has to get all of the semantics right for existing Clojure programs to actually run on it. For perspective, Clojure proper is similarly large with clojure.lang runtime alone being around 42k lines. So, yes, Kit is a toy in comparison.
It's a sophisticated compiler for a hobby project that implements things like type inference and algebraic data types. But it's simply not in the same league as a language like Clojure. And I personally find it insulting that the author tried to make an equivalnce between the two, then continued to double down talking about absurd metrics like github stars as a measure of complexity.
I think it would have behoved you to say this all much earlier and to be less thin skinned. Yes Jolt is impressive but I think you might be overly emotionally invested in people perceiving how great it is.
I'm not emotionally invested in people perceiving how great it is at all. I'm simply offended that Kit author would make an obvious false equivalence with my project and then refuse to even acknowledge doing that.
What's most disappointing here is how the thread predictably devolved into people just piling on instead of even trying to have an honest discussion on the subject.
The original point I was making is that I was able to make a complex piece of software much faster than I could have done previously. I presented this as something for the skeptics to consider.
And the skeptics predictably started doing mental gymnastics along with making personal attacks and false equivalences. Nobody has so far shown anything even remotely comparable being done by hand on the same time scale.
All this shows to me that people have already made up their minds on the subject, and no amount of evidence will convince them of anything. People made the conclusion that LLM development doesn't work and now it's basically no different from a cult. Rational discussion on the subject is clearly not possible here.
Anybody can download Jolt and run the benchmarks in the repo.
And this is the fundamental problem: You just suggested that someone else invest (possibly waste) their time to validate some giant black box of likely-slop code sprawl that you had a team of GPU-monkeys-with-keyboards spit out.
You're asking for the impossible, like asking someone to review all of the 30,000 minutes of video that is being uploaded to Youtube ... in just the last minute! So if you speed up the video by 3x, and work 24 hours a day with no sleep, you can only fall behind by 43,000,000+ minutes of video every day. This is not a winnable war.
So much of our industry, our relationships, and our society itself is based on trust. The careless use of LLMs has degraded that currency, badly. Perhaps you, for example, carefully instructed the LLM, and personally reviewed, understood, and tested each line of code. Your project isn't slop! It's pristine. It's perfect.
But it's hidden inside 43,000,000 minutes of slop.
No, the fundamental problem is that people want to have opinions on things they haven't spent time investigating. Nobody is asking anyone to write vapid comments on subjects they don't want to spend the time to understand.
Also, we're not talking about some random thing I linked to without any context. We're discussing a project I built. You are talking to the author. There is no reason for me to try to trick you into anything. This project also reflects on my reputation as a developer and I'm putting that on the line by showing it discussing it. The fact that you don't even have the basic decency to acknowledge this says volumes.
My original point was that I made something that solves a non-trivial problem as an example for people who are skeptical about LLMs. Instead of having a rational discussion about the project, this thread turned into personal attacks.
It's pretty clear that there's just an echo chamber of people claiming that LLMs don't work, and when they're presented with counter examples they just glibly dismiss them because apparently it's jut too onerous to actually engage with other developers in good faith for you lot.
My original point was that I made something that solves a non-trivial problem as an example for people who are skeptical about LLMs.
You may know that about your problem. I don't. As you stated in another thread about a challenge I made:
I don't see how I'd be able to use LLMs to solve something that's outside my domain of expertise.
How can I judge your project that is out of my domain expertise? (in this case a Clojure compiler. In what I do know, all I see is a LISP written in a LISP which doesn't seem all that hard, but I don't have domain expertise in that area) I'm skeptical about LLMs but this does little to change my opinion about them. There are studies out there that show people who use LLMs think they were faster (by some percentage) but were actually slower (by that same percentage). There are studies out there that show people who use LLMs have cognitive decline (and yet, all the pro-LLM adherents I've read swear they avoid the cognitive decline by using LLMs responsibly).
No, the fundamental problem is that people want to have opinions on things they haven't spent time investigating.
How much investigation do I need to have a "proper" opinion on a thing? I've tried using various LLMs a few times and found them largely a waste of my time. How much time must I put in before I can safely say they're a waste of my time?
If you can't say anything meaningful on the subject, nobody is forcing you to engage with it. People are skeptical of whether LLMs can be used to build large projects effectively. I feel that Jolt is such a project, I'm sharing it as an example. People who are versed in the domain and qualified to understand what it does can feel free to provide constructive criticism.
Like what they would've done differently had they been writing it by hand. What classes of bugs they think they might've avoided, what different architectural decisions they would've made.
But not a single person in this thread is doing anything of the sort. Instead, you're all just slinging mud.
So, basically there's no project that you can present that would satisfy you lot. Whenever somebody shows you a project you'll just screech that you shouldn't have to spend the time to understand what it is or what problem it solves, but you also insist that people have to take you seriously.
It's up to you how much time you want to spend. Simply moving on and not making a vapid comment is a perfectly rational thing to do. However, if you choose to engage then have the basic decency to spend a bit of time to understand what it is you are discussing.
Why do you feel that you have the right to waste my time with your inane comments?
I've mostly disengaged here but this is completely backwards. You threw down the gauntlet - you said, "I welcome any LLM skeptic to build something equivalent to Jolt in three months." You're implying that no one can do similar work by hand. You then insist that nothing meets your bar because your project is very special and complex and no one understands. So it seems there's no project that would satisfy you.
You're also taking personal offense at the fact that this isn't obvious to everyone else. My stance here was not even antagonistic toward your work in the slightest - "yes, you created something in three months. Cool! So did I." Even that is an insult to you, and you immediately dismissed my work, which at that point you probably hadn't even looked at.
It must be exhausting to think this way and see everything as an attack.
Why do you feel that you have the right to waste my time with your inane comments?
Finally, this is a totally unacceptable type of discourse for a discussion forum, and frankly you need to check your behavior in this thread.
There is nothing backwards here and you are making a straw man because nowhere did I say anything of the sort. I simply asked for an example that is actually similar in scope. What you shared is not in the same league and I have explained why in detail here. If you believe my characterization is unfair, please point to the specific part.
It is very clear to me that you did not spend any time to actually look at Jolt or see how it compares to Kit. If you did, then you clearly understand enough about writing compilers to see how these are very different animals.
I also never claimed the project is too complex for anyone to understand. What I said here repeatedly was that it is strange to see no constructive criticism at all.
I am not sure why you feel no project would satisfy me. I gave clear reasons why your particular project does not meet the bar. If you look at the size and scope, it is simply an order of magnitude smaller effort. That is not an insult of your work. that is just the fact of the situation. You could acknowledge that instead of doubling down.
What I find unacceptable is putting words in my mouth and turning the discussion into a personal attack. That is engagement that's unbecoming of this forum.
I would still appreciate it if you acknowledged that your project solves a much simpler problem with much more limited scope. If you can do that, we can actually have a discussion. If not, I think we are done here.
Who's forcing you to read my inane comments? Just skip them and move on.
Also, if many people didn't respond as you expected, perhaps it's a failure on your part to make yourself understood.
But again, just ignore what I say.
There aren't any goal posts moving here. What I said originally was that I was able to make complex project in under three months, and I challenged people to try building something on the same scale by hand.
Genuinely, this shows your ignorance of programming as a field. There are many, many examples over the last sixty years of programmers doing the same, or more, in lesser time scales independently. Maybe you should ask your LLM about it.
As someone who has taken the time to learn about the history of programming, your pet project is not particularly impressive, and the fact that you have been very visibly bad-mouthing a programmer who accomplished similar without the use of LLMs reflects very poorly on you, as a programmer, as a professional, and as a person.
Genuinely, this shows your ignorance of programming as a field. Feel free to provide examples of people writing a project with the scope of Clojure in a couple of months. As someone who has taken the time to learn about the history of programming, and having done development for decades, I can spot a troll when I see one.
I tried a challenge the other way and it did not go well at all. The person who tried to LLM the code I failed to write basically didn't understand the problem and didn't even bother to do all that much. Feel free to try either issue I mentioned; it should be even easier a year later with an LLM, right?
Also, it took me two weeks to implement by hand, ANS Forth for the Motorola 6809 [1]. An LLM should be able to chunk that out in a day, right? Feel free to try it. Oh, and my ANS Forth system passes the ANS Forth test suite for the wordsets I did implement. Or is this too trivial a project?
[1] I did not implement BLOCK, FACILITY, FILE, FLOATING, MEMORY or XCHAR (or the extensions) as those were more OS-dependent, took too much memory, or don't make sense on an 8-bit system.
I don't really have any context for what the problem being solved there is. I don't see how I'd be able to use LLMs to solve something that's outside my domain of expertise. LLMs aren't magic, they let you work faster, but they don't do thinking for you.
An LLM on its own is not able able to do much of anything, it requires the user to have domain expertise to apply it effectively. The reason I can build a Clojure compiler using these tools is because I have nearly two decades of experience working with the language. I know how it works internally, I know what the end solution needs to look like, and I know pitfalls to avoid.
Trying to solve a problem I have no familiarity with would just be me throwing darts at the board. Maybe the LLM will produce the right solution and maybe it will now. I'm not equipped to evaluate that.
And your challenge is different how? To take it anyone would need domain knowledge, right? Anyone trying to replicate your results by hand would need it. So is it a valid challenge?
My challenge is to build something of comparable scale by hand in the amount of time. It doesn't have to be the same thing I built, but something with a comparable level of complexity in the domain you're comfortable with. Or even better, simply show me a prior example of somebody doing that by hand.
My challenge is to build something of comparable scale [...]
How would either of you know it is of comparable scale, if (and that is most probable) your respective domains are effectively disjoined?
Anybody who's done software development for a while develops a bit of an intuition for size and complexity of projects. You don't have to be a domain expert to tell whether a project is likely to be challenging or not. This is quite different from the question of whether you can implement a project in a particular domain yourself. I can look at the Linux kernel and tell that it's a massive project that would take years to do. I don't have to be a kernel dev to appreciate it. However, if you asked me to build an equivalent of the Linux kernel, I'm obviously not equipped to do that.
Similarly, we know how long it took to build original Clojure implementation, its feature set, what aspects of it are trivial and what aren't. This is all generally known and can be understood, at least at a high level, by most serious developers regardless of their domain expertise. So, it's strange to me to argue that it's impossible to have a general idea of the scale of a software project and the effort that would be involved in building it.
Imllementing an existing language is obviously the easier task? You have the reference materials, all the PLD decisions have already been made for you.
I encourage you to try actually doing a task on this scale sometime to see whether you find it easy or not. While having lots of reference materials is very helpful, actually doing the work is a very labor intensive process, and putting all these things together in a way that actually works is not easy in any sense. In fact, there have been numerous previous attempts to do this, such as clj-scheme that never got to the point of being generally usable. If this was easy to do as you suggest, we would've seen this done a long time ago.
Making your own compiler with an order of magnitude less code where you can control the scope and make all the decisions is a far easier task.
I have found over an immense amount of cases that people who don't know how to do something tend to overestimate how long they take to make as well as how complex they are. Vibecoders and LLM overusers tend to never actually learn how anything works, so they tend to stay in that zone.
Leaving aside the “how clever a human you have to be” aspect, there’s a shovelware aspect to Jolt that indubitably would have been very difficult to do in three months along with everything else. A lot of a product being mature is having a bunch of stuff and features. The frontier of what size of standard library a new language or implementation can ship with has definitely shifted.
So, would you refer to Clojure itself as shovelware as well, or are you just admitting that I was able to get Jolt up to parity in a backhanded way here? The fact that LLMs make it possible to get a project to a usable state faster is sort of the whole point here. What I find particularly telling is that there hasn't been a single constructive criticism of Jolt in this whole thread. There are plenty of people slinging mud at me, denigrating me, attacking my character and my work. But not a single comment actually engaging with the quality of the project or the complexity of the problem it solves. That says a lot about the whole skeptic position if you ask me.
I don’t think anyone is denigrating your work. I think it looks to a lot of people like you were unintentionally (if accurately) denigrating someone else’s work. It looks to me like you’re being very defensive.
Clojure isn’t shovelware because it was painstakingly written by human hands over the course of years. I don’t think there’s anything wrong with shovelware.
You're literally denigrating my work by calling it shovelware which is a term to describe low quality software. If you're claiming Jolt is low quality then I'd ask you to back that claim up or retract it.
And here are just a few examples people denigrating my work a rather toxic way I might add
Manwhile, gonz here started personally harassing me by sending me toxic private messages even after being told repeatedly to stop. So, this is the the state of lobste.rs now I guess.
And there's not a single constructive criticism of the project in the entire thread. I'm really not sure what you think I'm being defensive about here either to be honest.
I gave an example of a concrete project for people to discuss. Instead of having a discussion about the workflow, it's architecture, the types of bugs compared to hand written projects, we have a whole thread of people making personal attacks and claiming that it's impossible to evaluate software project quality.
Finally, I'm curious about your last comment. Why do you feel it is important that the code was written painstakingly by hand over the course of years. Surely you're aware that there are plenty of projects that have been written by hand that are of very low quality. The reason Clojure is a high quality project is because it was built by somebody who is a domain expert and with a clear vision. I see little to suggest that how the code is produced plays a major role here. Clojure was produced at a time when LLMs weren't available, now we have a new tool that shortens the development cycle, I see no basis to claim that it inherently reduces code quality.
Maybe we have different understandings of the term, but I take it to mean software that would require a lot of manual effort not very creative in each line, that has been automated through llm generation of that code.
I've only seen it used in a derogatory way, but I agree with applying your definition here. A lot of the code is indeed just mechanical work, like mapping out Java class hierarchy. And that's precisely what makes LLMs a good fit for doing it. I can focus on interesting things like fiber mechanics, or how to do live program image serialization, while leaving the boring things to the agent.
As much as the progenitor of this thread appears to be missing some of the point, I have some doubt that this is the case here, they look to be credentialed in their field, AI usage aside.
We cant be sure of this though, because I at least lack domain knowledge, and the traditional markers of trust have been almost completely eroded by LLMs when it comes to communicating with an audience even slightly outside of any specific domain. This is the point. Claims like this suck now, there's no path in which they accomplish anything.
Must be incredibly frustrating to be on the other side of it as well, assuming good faith. (Which is also much harder to do, given the tooling and the tone.)
The philosophy exuded by AI boosters still defies the last half-century of theory about what software development even is.
It is near the bottom but this one is so often overlooked. There are skill plugins like superpowers that attempt to let these models/harnesses make use of good practices like TDD, DRY, etc and they actually do improve the quality output in a remarkable way (there are still plenty of sneaky micro hallucinations, but that is besides the point).
But, none of the frontier labs includes those practices themselves in their harnesses or model training to any significant degree as far as I can tell. They all seem to focus on the sort of quick idea to code type development that works for prototypes but falls apart for larger projects.
Most people I know who use AI successfully in their workflow have had to spend A LOT of time putting up a ton of skill scaffolding and other instructions before they got to that point. Some of that comes down to project specific idiosyncrasies but a majority from what I have seen is the same sort of stuff as the superpowers skill set, trying to get these things to follow basic fundamental software development practices.
Then there is the fact that writing the code is just one aspect of the entire process to begin with. Something I have written about before to the point that the quote I am about to include quotes a previous comment as well, going full matryoshka doll.
I agree with a lot of the article but I also do think it is being overly optimistic about rigor in software engineering. Specifically this line
Plenty would kill to get the same kind of automated testing we treat as a given.
I have worked in a multitude of organizations and automated testing is still in this day and age not a given. Specifically robust automated tests at various levels and not part of the process. Software engineers all too often see themselves as the only checkpoint that is really needed and organizations try to eliminate parts of the processes they see as roadblocks.
I have written a a comment about this before so rather than repeating myself entirely I am going to include it below. In addition to that we as a software industry seem to collective suffer from short term memory loss where we are simply refusing to actually codify learned best practices and insist on reinventing the wheel every few years. I can already hear people scuff and point out that a lot of things have standardized over time and a lot of practices are shared between companies. But in my experience what has been standardized is mostly a shared vocabulary on loosely defined things that have wildly different meanings between companies and testing. Terminology like agile should be familiar to anyone. But also when we are talking about "testing" in my experience people think about wildly different highly specific things, not the holistic testing approach as a whole (see my comment below). Most developers will think about unit tests, some might think about a slightly expanded scope with component/integration unit tests. In some companies that is all the testing that will be done before things are yeeted to production even.
Anyway, no we certainly aren't special. But, we aren't the same either even though we probably should be a lot more similar in my view.
It also annoyed me but for a different reason as you are saying. The point of doing a code review, in a healthy organization, should be that it seen as one of many processes that together are responsible for good quality process. Not just a singular gateway that is responsible for something going to production. In fact, in many cases it shouldn't even be the last gateway depending on the complexity of the landscape, integration with other applications, etc. But, unfortunately for many teams and companies the QA process after code is merged is something that might as well not exist anymore.
Anyway, I am a firm believer of everyone being responsible who is involved in the process. However not to diffuse responsibility but because quality should be a integral part of the process before code is even written. Basic boring stuff like three amigo sessions (or whatever method used to get various disciplines involved in talking about spec and requirements), test driven development methods, basic static code analysis tooling integrated in IDEs and various checkpoints, QA/Test automation expertise other than developers themselves integrated in teams and much more.
I know that for many people this is not a reality they ever encounter. But if we are making the comparison to civil engineering. The developer shouldn't be both the engineer and construction worker for the bridge in many cases. Even if they are, in civil engineering there also never is one person who is responsible as the practices I mentioned are very much part of the process there as well. An engineer needs to document everything, make all the necessary calculations for load bearing, have others re-check those calculations, then often get approvals from relevant government bodies who will double check various things. Then during construction there are also many more points where things are audited, checked, approved, etc. Even for simpler construction like that of a single family house this is still the case. Plans need to be drawn up, someone needs to sign of based on calculations, permits need to be given based on those plans and at various points in the process inspections need to be done.
Yes, there are people who have more responsibility than other but overall there never is a single person who has the only responsibility. In fact whenever there is a engineering disaster you often see that it has been a failure in the overall process where at multiple points things should have been stopped but were not.
We like to compare ourselves in software development with civil engineering. But if we do that we need to realize that our processes very often are just not on the same level at all. At least not as far as constructing bridges goes. In the best case scenario we often find ourselves at the level of housing developers who build the cheapest possible houses and try to work around a lot of the legally requirements.