I Don’t Want to Interact With Stochastic Parrots
88 points by cgrinds
88 points by cgrinds
I’m convinced that accepting generated code will lead most projects to become broken beyond all hope of repair in only a few years. "Carefully reviewing" will soon not be enough. Unfathomable code will eventually slip in and gradually destroy the whole project. I predict that the projects too important to be left to rot and die will take decades to recover from those couple years of "assisted productivity."
I always appreciate when people make concrete predictions. I'll take the other side of this one: I expect the use of LLMs in, e.g., Linux, Chrome, Firefox, node.js, z3 etc will increase over the next decades, and that if you could somehow assess the quality of these projects over time you would not notice a negative break in trend with the introduction of LLMs (not to say every project is getting better, just that, to the extent they're getting worse, that is the continuation of an existing trend rather than something caused by LLMs). Oh, and more concretely I predict that the number of long-lived vulnerabilities in high-profile projects will be markedly lower as time goes on.
It's very reasonable to choose not to interact with LLM-generated code or image generator outputs for moral or aesthetic reasons. None of that has to be grounded in whether LLMs can be used for maintainable, high-quality code. It's convenient when the stance one is morally or aesthetically inclined to take is also the technically superior one, but it doesn't always work out that way.
I agree. The fact is that we, humans, have not made a great case for being stewards of great software quality. Most software is shit and will probably continue to be shit. It's got to a point where if an LLM makes a mistake it's something I'll look at and think: "I know people who would do the same". A lot of times those "people" are me.
I agree. The fact is that we, humans, have not made a great case for being stewards of great software quality.
This is an important point. Autonomous agents would not be a serious vulnerability threat if software quality was not already brittle, compromised, or imperfect. I think it’s important to be humble and admit that one doesn’t know everything, and your memory will be imperfect. Getting help from an agent to check your work, or provide coding assistance should be part of good practice. Peer programming and reviews were meant to provide exactly that alternative perspective and knowledge, but have pitfalls when a senior mentor or same-level peer has a finite amount of attention or hours in a given day.
There was an interview recently of a engineer from Mozilla (in French) who said that since they've started using Mythos they went from 20 vulnerabilities corrected in a month to more than 400, to the point that in a recent competition to detect new vulnerabilities in Firefox, most of the teams decided to give up since their findings had already been corrected. His stance was that with tools like Mythos, softwares would improve dramatically in security.
It's convenient when the stance one is morally or aesthetically inclined to take is also the technically superior one, but it doesn't always work out that way.
I take that to mean that you don't see the moral and aesthetic arguments against LLMs to be technically grounded? I certainly see the appeal of that stance, though I'm not yet entirely convinced. I wonder what your take is on the problem of LLMs training on their own output? Of course, I don't imagine there's anybody of significance that argues all code should be written by agents. So it's not like we're rapidly descending into the demonic shrill of an AI echo chamber. We can all agree that humans will always need to be somewhere in the loop, the question is: where?
My university training is actually in Religious Studies, which seems a million miles from Computer Science. But we had a module called Religion and Media in which we studied the great communication paradigm shifts of; orality, literacy, printing and digitisation. One book that stood out is Orality and Literacy: The Technologizing of the Word, in which it is argued that even poetry and song are technologies. Melody, rhythm and narrative allow information to be more easily remembered (disk drives?) and therefore also allow it to be shared with more people across time and space (the internet?). As we all know from the more recent digital paradigm shift of the last few decades, such technologies can have a profound effect on human culture.
LLMs are crucially, by technical definition, not part of this story. They are not rhythm, song, narrative, a system of physical marks that represent spoken language, a machine that faithfully duplicates, nor a protocol for sharing information electronically. They are essentially, as the parent article describes, parrots. Now of course we can also argue that there exist clearly useful parrot-like technologies, for example search engines. So I accept that LLMs are more useful than actual parrots. But the point is that, though they may superficially appear to have a voice, they don't. They do not contribute to any meaningful conversation.
Which makes me think our relationship to AI is in fact fundamentally a matter of morals and aesthetics. We all know, or at least should know, that the big AI companies are going to greater, even destructive, lengths to find non-LLM tainted training data. If that is a technical reality, then should it not also be a technical reality that we have a moral and aesthetic responsibility to identify and avoid creating and promoting LLM-tainted content? And I don't mean that from the capitalist perspective of the current economic bubble. I mean it purely from the cold facts of how LLMs need non-LLM training data. Somebody, somewhere has to draw lines and take a stand.
The morally or aesthetically superior stance is the technically superior stance.
Of course right now there's abundant locally consistent examples of LLM utility, but they underfit the thesis of LLMs' ultimately beneficial contribution to humanity. So let me make a concrete prediction. Any project that does not sincerely reflect on its use of LLMs, though it may well still have maintainable, high-quality code, will eventually fall by the wayside for the simple fact that it is no longer participating in any meaningful conversation with the world in which it claims to support.
I take that to mean that you don't see the moral and aesthetic arguments against LLMs to be technically grounded?
... No? I just mean that it is possible to have moral or aesthetic objections to interacting with LLM outputs entirely independently of the question of whether they work (in the sense of producing maintainable code) or will continue working.
I wonder what your take is on the problem of LLMs training on their own output?
My understanding is that it turned out that so-called model collapse ended up being pretty easy to avert: you just need some other source of feedback during training, which is part of what has lead to the current regime of Reinforcement Learning from Verifiable Rewards (RLVR). That is to say, my take is that this problem appears to have been basically solved. Labs are seeking out old books because they care about ingesting the factual content of those books, not because they are worried they're going to stop being able to improve the models in the absence of ever more sources of human-produced data to feed into the pretrain.
This is an empirical claim about the world. Now, maybe I'm wrong about it. But I don't think it really changes much here, which is my point: it is possible to have moral or aesthetic objections to the technology which hold regardless of the empirical question of whether LLMs training on their own output is ~ a solved problem.
If that is a technical reality, then should it not also be a technical reality that we have a moral and aesthetic responsibility to identify and avoid creating and promoting LLM-tainted content?
You are using "technical reality" in some way I'm not really understanding, because this sounds like is-ought conflation to me.
So let me make a concrete prediction. Any project that does not sincerely reflect on its use of LLMs, though it may well still have maintainable, high-quality code, will eventually fall by the wayside for the simple fact that it is no longer participating in any meaningful conversation with the world in which it claims to support.
I do love a concrete prediction! Let's nail it down a bit more: would you consider Linux to have sincerely reflected on its use of LLMs? If so, I agree with this prediction but only because I think it's largely vacuous; most big projects sincerely reflect (to some extent) on pretty much any significant change in how the project develops. If not, does that mean you are predicting that Linux will eventually fall by the wayside, and not in the way that all endeavors eventually fall by the wayside? For example, do you think that in, say, 40 years - two generations - we will be able to look around and identify that Linux and other projects which today are accepting use of LLMs will have perished in advance of similar projects that never adopted LLMs?
So your point is more that, no matter the moral or aesthetic objection, the fact will remain that LLMs are demonstrably useful? Then yes, I did misread your comment. My reply latched on to the idea that there is no technical basis to moral or aesthetic objections. I don't feel my line of argument replies to your main point closely enough. I'm sorry.
I am still interested in exploring this idea of model collapse though. I don't think RLVR is enough. Is the analogy of the International Space System instructive here? Let's assume that LLM training data, like the ISS, is bound within a closed system. Whilst the ISS is incredibly efficient at recycling air and water, there are fundamental facts of physics and biology that mean regular resupplies are still needed. I'm equating LLMs to the ISS's filtering systems here. They are both undeniably useful. But neither of them create new valuable material, they simple move existing material around.
Now of course LLMs aren't actually in a closed system. And I assume most of us would agree that if LLMs were in a closed system then yes, we would arrive at model collapse. So what's interesting is, what exactly defines something as being outside this closed system? In a dystopian world were humans do literally everything with AI, then the main source of new material comes from prompts. Though perhaps not explicitly prompts themselves, but rather the subtle effects they have on the unique outputs they lead to. Again, I think this is a reality even the most pessimistic of us don't believe will come to pass. But it's sketching out my main question: what exactly are the boundaries of this system? A system does not have to be hermetically sealed to generate closed-system-like behaviours. Even air itself, like in a moving lorry's slipstream, with its eddies and currents, can create a tangible albeit temporary localised system.
Perhaps even the mere assertion, LLMs are of benefit to humanity, is enough to create a metaphorical slipstream? Whilst not enough to create a truly closed system, could the cultural effects of such a sentiment create unpredictable eddies that trap traditional human progress in microclimates? Of course RLVR might alleviate such degeneracies, but I'm not sure they directly address the underlying issue.
This is where I see the role of morals and aesthetics. They are the tools we have to disrupt and dissipate boundary creating eddies and whirlpools. For the overall health of human knowledge, and by extension LLM training data, it is a moral responsibility to not create large predictable "airflows" of regurgitated output. Aesthetics are the tips of a birds wings, gracefully injecting the perfect amount of turbulence to maintain stable flight.
Linux is a great example to pick. It's weathered so many storms and consistently delivered software foundational to modern life. But Linus has started using LLMs for debugging. Well I certainly don't think they're going to do a Bun and get an agent to rewrite everything in Rust. As you say, I think, like most big projects, they sincerely reflect on any big change. So yes, perhaps the point is moot. But I would like to predict that Linux, again like most big projects, will come to have an LLM policy as serious and articulate as its security policy. Becuase ecosystem health is one and the same as software health.
But the point is that, though they may superficially appear to have a voice, they don't. They do not contribute to any meaningful conversation.
How do you jump to that conclusion? Because it is a parrot, it intrinsically does not have a voice? But what about the people voices that it regurgitates?
If the parrot is better than me at finding and bringing to me the meaningful voices of actual people, doesn't it contribute to a meaningful conversation at the end?
If a rubber duck can contribute to a meaningful conversation, what about a rubber duck that can answer back?
I've the feeling that people with strong anti-LLM positions are trying to force the factually false (or at least unfalsifiable) narrative that LLMs "are not useful", "cannot be useful", or some gymnastic like "okay they can be useful but only locally and not as net positive to humanity so that it disqualifies them". When the facts are that people find them useful, people solve real problems with them. You cannot just swipe the reality because you don't like it.
There are some grounds on which LLMs can be argued against, like environmental impacts, power and capital concentration, cognitive atrophy, etc. And I would agree on those. But usefulness is not one of them. When I read people having the latter argument, I tend to think that they don't know what they are talking about and are trying to justify a posteriori something they were already convinced of while not being really honest.
When i read the post, a few times my reaction has been "does he actually think this or is he just trying to be provocative to push reader to think about it?".
Examples:
If you don’t have the resources to pay for an artist, you probably don’t need a picture at all.
Wut?
[...] proprietary software is simply badly coded. Each line is written assuming "I will not be there anymore to deal with it in a few years."
People that write proprietary software and open-source ones are often the same people. Their code does not become intrinsically bad when it is proprietary. What about a proprietary software that open-sourced, does it magically become good?
Which means that, as soon as generated code is added to a project, the whole project becomes unmaintainable by humans. You can’t understand the code because there’s nothing to understand in the first place!
Like that? Magically? The whole project become unmaintainable? Come on... Why would there be nothing to understand? This just read as bad faith to me. And again, that is contradicted by the reallity, there are projects that contain LLM generated code (code that is understandable and doing meaningful things) that are still maintainable and maintained.
The main issue I have with current GenAI solutions is they are gated behind either a monthly payment to an unaccountable US company, or require significant hardware purchases, coupled with extensive labor to work.
It is unwise for open source projects to surrender autonomy in this way for gains in "productivity", terms which are borrowed wholesale from organizations with other goals than FLOSS.
I mostly agree. I think the quality and reliability of software will depend even more than before on the goals of the developers. If AI tools are used in order to add more features, more code or more whatever, then quality will suffer. If it's more targeted, then quality can at least not worsen. AI tools make the first case easier than the second one so I'm not very hopeful for the future but there's a big dependency on humans too.
Fully agree that you should not outsource your thinking, nor your passions to AI. You need to make the call on where that line is, but it's still your responsibility to not hurt the commons or your own well-being by being a meat proxy.
On the other side, I'm not particularly impressed by this article.
AI capabilities are on a pretty steady (though spiky and arguably poor taste) improvement trajectory, which will cause all sorts of economic mayhem and raise all sorts of problems and questions. "AI slop art six fingers am I right" isn't remotely cutting at the heart of the problems here.
What's newsworthy about it? Just occasional soothing post for claker haters?
I don’t mind mistakes, typos, or badly formed sentences. In fact, I subtly appreciate them when we interact because they are part of you, they tell a lot about yourself.
typos are wabi-sabi
But that’s what we are. Just squishy random generators pushing out muscle movements through a giant filter…
/s ? Surely you can’t be that reductive…
Too late. I've already emailed the soft sciences and told them to pack it all up. Us software types can rediscove-- uh, I mean, we can take it from here.
What else is there? A soul? Nobody could come up with any evidence that there's more to it.
A tornado and fog bank are both "just" water and air molecules, but if you get them confused, it's on you.
I do personally believe in souls. For a variety of reasons, but let me at least try to justify/defend the idea that there is more than purely the material (atoms/quarks/photons/etc). I don't believe (A) The material universe has always existed and (B) The universe caused itself to exist.
For (A) if the universe always existed, then the second law of thermodynamics would be violated. Also the expansion of the universe would be more confusing than it already is.
For (B) something that doesn't yet exist can't cause anything (A chicken can't create it's own egg to create itself, just another different chicken), and something that started the universe needs to be immaterial. Otherwise it's just part of an eternal material universe (see A).
Also the heavy bias towards matter in the universe also makes me think that something immaterial created the universe (First law of thermodynamics). In any case we do sometimes see in the vacuum of space (and particularly near black holes) spontaneous creation of matter and anti matter. But that's always a net neutral change in energy (-1 and +1 is created to "equal" 0).
There is a school of belief that we are. I believe Sam Harris is of this mindset. This is what he means wrt not having true free will. I could be wrong as I'm going off memory. But anyway I do think there is plausibility in this.
Free will is the randomness, a combination of randomness in the stimuli and randomness in the neurons and other internal state.
Someone contact the universities, we've got ourselves a real philosophical zombie here! We can open a Department of Experimental Philosophy, and the psych department can run experiments without board review!
the pessimistic view I hold is that it's fair for any software that's in an excellent sandbox, probably not facing the open internet [0]. But it's untenable for anything else. Accessibility of cheap enough LLMs means that it's so easy to mine any codebase for remotely exploitable bugs. Especially if it doesn't employ rigorous sanitisers, fuzzers, and analysers.
I did run a three evening pass over something that I use and found dozens of such bugs. Couple of them which probably could be chained into something worse? It also prompted me to put much tighter sandboxing around a fairly classical daemon, using RestrictSUIDSGID, NoNewPrivileges, etc.
[0] Yes, corporate networks that don't tag each flow are equivalent to open internet.
I appreciate these kind of statements, but at some point it's incredibly difficult to hold the line. Think about the lengths RMS and others go through to only use copyleft software, hardware, etc. The surface area of usable systems under those rules is diminishing and outright inconvenient at times.
The difference I can see is that at least copyleft software clearly identifies itself. There's is a known ecosystem. But it's next to impossible to determine by the human eye if code, prose, or even conversation was authored by AI. So I appreciate this post in its sentiment and that OP is making a stand. I wonder, though, if it's actually realistic or feasible to practice what we're preaching.
I'll always cite It's rude to show AI output to people as the best incarnation of this argument
I appreciate these kind of statements, but at some point it's incredibly difficult to hold the line. Think about the lengths RMS and others go through to only use copyleft software, hardware, etc. The surface area of usable systems under those rules is diminishing and outright inconvenient at times.
Yeah that's a problem indeed. I try to avoid proprietary software as much as I can and do not use LLMs myself, but if I then also have to further restrict my usage of software to AI-free FLOSS, I don't think I'd stay sane.
Holy crap, I don't care. I don't care at all. Anyone voting up this inane use of bytes and bandwidth is contributing to the vibecoding tagged spamming of lobste.rs. Can you guys please just limit your boosting of vibecoding diatribes & thought vomit to maybe once every other week rather than every other day?
I feel like I'm losing it. Why do you people feel that this is worthy of being at the top of the feed?