I don't care how well your "AI" works
115 points by helene
115 points by helene
I've added my first filtered tag, namely "vibecoding". I use LLMs a lot, and would love to read about people making good use of them, but the signal to noise ratio skews too heavily to "I don't like" posts.
If this post and others like it were merely saying "I don't like it", then I would agree that they're noise. What I see instead is a bunch of people desperately trying to convince all of us that there are ethical problems with LLMs and principled reasons not to use them. I think that's a discussion worth having.
That's fair. You make a good point. I might alter my argument on two dimensions:
(1) The signal to noise ratio of people who are arguing from first principles vs. fitting an argument to their position is low. This criticism applies to all-or-nothing arguments at either extreme.
(2) New insights/arguments on this topic are few and far between. Not necessarily in this case, but in general, the upvotes don't reflect novelty, they reflect people's feelings. So, it's hard for me to find curated new views on this topic from the "vibecoding" tag.
For myself, and you can see some of how I'm trying to contribute positively in my other comment in this thread, I look at some of the ethical issues as places to invest my own effort to try to make things better. For me, the concentration of power in state actors and tech bros is concerning, as is the energy/water cost.
I think there should just be a separate "ai-ethics" tag.
I think users would object to every conversation on, say computing devices in general, being overrun by a subsegment of the userbase who object to all computers used for any purpose. I'm not beyond any discussion, but every discussion should have its' place.
As I've mentioned in other comments, there is also the further concern that these ethical concerns amount to intellectual laziness. "AI" is being lumped together as some homogenous pile of everything wrong with tech. If it's so evil, is "I don't engage with it" sufficient? Shouldn't we be pushing for regulation? Talking about informing users on the dangers of misuse? Building in better safeguards into the UIs/models themselves? Explore ways to segment model use to be more power and resource efficient?
Just an “ethics” tag would be nice since these things rarely have anything to do with the building of software with LLMs and are mostly commentary on the companies / process behind their creation and the social impacts of them. Ideally we would recognize ethics posts like this are just off topic for lobsters, but many crustaceans seem to disagree.
Which is precisely what folks who want to discuss LLMs would like to filter out. "Stop telling me about the orphan-grinding, I don't want to hear about it, I just want to talk about how nifty this stuff is!"
To be fully honest, I'm also still drawn to "how nifty this stuff is". I'm fascinated by the idea that these models can manipulate language so fluently and seemingly understand it. But I don't want to be so fascinated. I want revulsion at the ethical problems to win out, such that I don't have any desire to use these products unless those problems are somehow solved (if they even can be).
I’m unfamiliar with the term “orphan grinding” but I would not characterize my position as “I just want to talk about how nifty this stuff is!" See my other comments.
SEEN. I have been thinking about this a lot, too. There really should be an anti-vibecoding tag or something more clever with a bit more moderation. Otherwise, it feels like a waste of everyone's time and a bit of a misslabel.
(1) For my sci-fi novel, I wanted a way for people to give me feedback by annotating the PDF files, without resorting to shared docs, passwords, or third-party servers. Enter PDF.js, Mozilla's incredible in-browser PDF toolkit with a vast and complex API for rendering and interacting with PDF files. Developing a simple, annotation-only editor would have taken, at a guess, upwards of 60 hours (or more) to implement because I would have had to learn a wholly new API. Using a combination of Claude, Gemini, Blackbox, and Copilot got me there in less than six hours. The source code is available.
(2) My repository was using GitList until it came crashing down, nearly taking the shared host down with it. Within about 15 minutes I had a replacement that has been stable for two months and counting. The software isn't as fully featured as GitList, but it's read-only, shows the files, allows downloading individual files, has a last updated timestamp, doesn't go wrangy on resources, and is usable with desktop browsers.
(3) For a safety-critical radio system, I was able to hash out some 20 boilerplate event definitions leveraging a complex C++ macro in the time it takes to list the event names. What's the point in typing the macro name, ifdefs, comments, parens, and other template-specific code 20 times over when a machine can churn them out in five seconds? I also let Copilot hunt for bugs in some C code that a team member wrote and it found an edge case the human code reviewers had missed. LLMs are still a long way from implementing complex requirements specifications, so use them to their strengths.
I've been developing software for over 40 years. LLMs, when they don't hallucinate, feel like a force multiplier, and they are improving on a monthly basis. This isn't a fad. This is a wagon train to nirvana, and people better board before they get left behind. By next year, I hope they'll have a more rigorous understanding of encapsulation, how to write maintainable (readable) code, and useful documentation (i.e., the "why" a function exists, rather than the "what" it does).
doesn't go wrangy on resources,
Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 106618880 bytes) in /home/autonoma/public_html/repo.autonoma.ca/index.php on line 711
seems to crash when viewing a file. highly recommend cgit with perhaps some styling for desktop browsers, or legit: https://tangled.org/anirudh.fi/legit
seems to crash when viewing a file.
Fixed, thank you. Can't use cgit on my shared host, but thank you for the suggestion.
I mean you don't disagree with the person you are replying to? They're saying they are useful, just the signal to noise ratio on posts about using LLM's is poor, as there is far more bad slop out there than skilled engineers writing about how they use it as a force multiplier.
I hope they'll have a more rigorous understanding of encapsulation, how to write maintainable
Why? If the LLMs are writing code, why bother with being readable? Compilers these days don't output easily read assembly code.
Why? If the LLMs are writing code, why bother with being readable? Compilers these days don't output easily read assembly code.
Compiled assembly code is typically correct and needs no verification; AI-generated code currently requires ongoing human involvement (debugged, reviewed, modified, and maintained by humans)—often for years. Even AI tools work better with clean, well-structured code. Until Holodeck-esque machine-to-machine coding exists (which may be several years away), readability remains essential to verify correctness. Other factors:
This is kind of off-topic given the actual content of TFA. But readability might matter because the data on which the LLM was trained was (to a first approximation) readable. So it will have a harder time maintaining a codebase that’s unreadable.
Typically when you're working with an LLM, you're still reading all the code it's generating, and you're adding it to the codebase that you're working on as a human. It needs to be readable for the same reason your colleague's code needs to be readable - at some point, ideally sooner rather than later, you're going to want to read and understand it yourself.
That said, even if you had a codebase entirely written by LLMs, my understanding is that readability is still really important, if anything more important than for humans. When I read some code and figure out what it does, that information is now retained in my brain (to some degree at least), so that next time I come to do something with that code, I don't need to figure out everything from scratch. The nature of context windows with LLMs is that every time they "read" code, they're seeing it for the first time. As a result, it's even more important to optimise for that first-time readability than it would be for humans.
The other side of that is the way that LLMs "read" code. They're not analysing the AST like a compiler and understanding the code at an abstract level. Rather, they're taking in a string and pattern-matching it with a bunch if previously-trained examples, more like a human would. That works well with code that is very clear and explicit, where the wrong thing is clearly impossible and the right thing falls naturally out of the API you've written - essentially the sort of code that humans are good at reading. But if you've got subtle code, or code that looks like it's doing one thing but is actually doing another, then the pattern-matching process can't work as effectively, because there aren't enough existing patterns or the patterns are pointing in the wrong direction. Like I said, this isn't like a compiler which "sees" the code as an abstract data structure that can be processed and manipulated arbitrarily, so how the code looks is very important.
As I said, even if readability wasn't important to the LLM itself, it's still important to humans working in the codebase and needing to understand what's happening. But beyond that, the easier some code is for a human to understand, the easier it is for an LLM to work with it.
Readable code means high-level code, enough for a human to understand it.
If a human can't understand it, the probability is high that the LLM won't understand it either. And next time you'll want modifications, you won't have the original context around. LLMs are currently pretty bad at keeping context. Even if you keep the dialogue you had with it, context can get poisoned. The source code will remain the source of truth, for both humans and LLMs.
Code should be seen as mathematics. Math language is very reasonable, precise, and used for communicating complex ideas. I understand that high-level code will end up being just English for many, but in terms of societal evolution, that's a regression, actually; before modern symbolic notation, maths was rhetorical as well.
As I'm reading the replies to my question (and comment), the less I want to work with LLMs. They all seem to run counter to human nature and in my opinion, seem to be the strawman argument that we can still use C, we just need more programmer discipline when using it.
I also get the vibe from the pro-AI camp that no one in their right mind will use prompts as the source code---am I the only one who can see that happening?
I don't think LLMs work the way anyone wants them to work. It'd be great if we had a magical tool that let us use english the same way we use python, we don't.
The thing is that LLMs not working the way we want them to doesn't mean they don't work in a way that can be useful. But to do so you have to use LLMs for what they are good for, not for what you really wish they were good for.
We can stop using C because we have the better tools that let us do the same thing with less errors. If we didn't have those better tools C would be better than nothing. In the world of LLM-like-tools its LLMs or nothing so far. Personally I'm hopeful that integrating things like formal proof systems into our tooling downstream of LLMs can help here, but we aren't there yet.
As an aside I feel like a lot of the companies pushing LLMs are falling into the same trap of "pushing them where the company really wishes LLMs were useful" instead of "pushing them where the LLMs are actually useful"... it's a very easy trap to fall into.
Prompts will definitely be seen as the source code by many. I, for one, think that this can be good if it empowers normies to do simple programming.
But I don't see prompts being used by professionals as the source code because it's very confusing. English is confusing on its own, and then LLMs make mistakes, so you need a dialog for clarifications. And the context ends up not only with your prompts, but also with its decisions, some of which were wrong. This is why "context poisoning" is a phenomenon, because previous interactions can change its future actions. And for instance sometimes you can witness it doing the same mistake repeatedly, e.g., you tell it to fix A, which breaks B, you then tell it to fix B as well, and it breaks A again.
Note that something similar can happen with the understanding of humans, especially when we are superficial. Which is why, after building our understanding, we resort to unambiguous specs, and math, and high level source code, and tests, to solidify the ideas thrown in a meeting.
If AI advances, say to AGI or close, even if it uses internal language more optimized for it, do you really think that it won't communicate with humans using math, or in source code that's unambiguous in expressing "if this then do that"?
Even if that language is going to be English, it's not going to be plain English, but rather something similar to legalese.
But I don't see prompts being used by professionals as the source code because it's very confusing.
The real question is whether prompts-as-source are ever used for any system I would be sad to see crash and burn, professionalism of people involved is secondary for practical considerations.
I personally don’t touch LLMs with a stick. I don’t let them near my brain.
Two top-level discussions on this quote already! Here's a third opinion. For what it's worth, I suppose I'm a sort of bertologist; in order to do some of my security work, I've had to study the mechanics of LLMs, and this has given me some insight into what's going on. LLMs learn to simulate flowing text by memorizing associations between words. (Between tokens, etc.) If nothing else, this is an excellent point:
When we use a tool, it becomes part of us. That’s not just the case for hammers, pens, or cars, but also for a notebook used to organize thoughts. It becomes part of our cognitive process.
More generally, following Hofstadter's refusal of the caged-bird metaphor, it isn't the case that a mind is confined to a single brain. Instead, a mind can spread across multiple substrates in certain specific ways, extending and distributing cognition.
Our minds are susceptible to outside cues.
Our brains are susceptible to outside cues. We associate calls with responses. We are provoked by symbols. We transmit memes through text, treating it as a medium through which we propagate culture. The mind is merely along for the ride, partially here and partially there. So, what happens when we put the brain in front of a system which can only talk in memes?
I think that LLMs are memetic hazards: they harm humans by exposure to viral cultural artifacts which alter minds by changing their memetic substrate. I'm concerned about two specific effects. First, a recent love bombing effect seems to have emerged; I think that this is because our RLHF data has started to collapse, as if a customer-service agent had started to crack under the strain of too many company policies.
Second, in general, there's something wrong, giving me literary goosebumps, with the output of GPTs and it might be a problem with autoregressive Transformers in general. Reading too much GPT output, especially simulated-chatbot output, especially interactive sessions, does something to flatten out the analytic skills of heavy users. There seems to be a pattern where bertologists are surprised and interested in some emergent feature of a Transformer, then they spend a few sleepless nights chatting with it, then their blogposts start to have every fifth paragraph written by autoregression, and finally they start to believe that there is some sort of spirit, soul, or life within the model. Like, how does the author of Simulators (2022), which is already partway down that pattern, get to the point of seriously saying earlier this year (on Twitter, further context), "the model's soul is intact"? No, that's not a result of a healthy interaction with a non-anomalous object.
I buy all this, and I buy the extended-mind thesis. There's something different this time with AI compared to technologies that have come before.
The motor that powers a forklift is labor-saving, and it can lift more weight than anyone who has ever lived. That sort of power is dangerous, and we have created an entire engineering discipline around industrial control systems. Really precise stuff that keeps workers and bystanders safe.
Pen and paper are also labor-saving, and also force-amplifying if your goal is to communicate, or influence, or harass. That sort of power is dangerous, but we tolerate pen and paper. We've got social systems to steer and cohere, and that's a good thing. The Church, the Party, the editors, the schools, the Father, the Mother, the shame or admiration of our peers.
Somehow AI chatbots combine the mechanics of a slot machine with knowlege work. Yea, typing a prompt is more symbolically rich than pulling a lever, but it is an order of magnitude less complex than googling, finding the stack overflow link related to my error message, smell-checking the upvoted answer, and copy-pasting it into my project. It's making me dumber. I can feel it. It's seductive and easy. I have to fight it.
I am not worried about losing my job as much as I'm worried about becoming one of the humans from Wall-E. That's speaking on a personal level. At a societal level, it is really difficult to see any upside to this stuff.
“This time is different” is a red flag in conversations for me.
Software engineers barely use the pen and paper anymore, and this has happened ever since we started typing on our keyboards with 50+ WPM. And on one hand, it's regrettable, because we may lose much by being bad at penmanship, like our ability to do maths. But on the other hand, I'm pretty sure the same pathways are activated while I'm typing this message. Similarly, normies barely use computer keyboards anymore, since they have portable devices, depending on autocorrect or dictation. Obviously there's a worrying trend, especially for children, but it has happened with every generation.
I remember my parents, in the late 1990s, being upset that I would spend so much time in front of computers. For them, it was unhealthy, I would have problems with my eyesight they said, and also a huge waste of time.
So firstly, we should acknowledge that algorithms breaking how we think is nothing new, and has been happening at least ever since the smartphone became a thing: https://www.youtube.com/watch?v=QEJpZjg8GuA
We also tend to forget that, the printing press being invented was undoubtedly a good thing, but it did lead to false news, propaganda, social upheaval, and contributing to a couple of wars.
“This time is different” is a red flag in conversations for me.
It is for me, too! I think it's a provocative thing to say, especially here. I am a technological optimist. I think that we should all be working 15 hours a week like Keynes predicted 95 years ago.
Full disclosure: I use Claude and I use Kagi Assistant. If you made me Emperor of Earth, I would not ban them, nor prohibit their use. Claude is paid for by my employer, and I pay for Kagi on my own (mostly for search, though).
I want to be a rational person, so I want to have good reasons for why I feel so uneasy. I'll be the first to admit that my thoughts aren't super polished on this.
I am confident in the assertion that - every now and then - there's a technology that is different. Nuclear weapons are one example. They haven't been around that long, but nuclear-armed states don't do direct war on each other. If it ever happens, it will be unlike anything we have ever seen. And, conversely, the situation that has obtained since 1949 is unlike anything seen before.
Do I think an LLM is "as dangerous as" a nuclear bomb? No. But I do this tech is unprecedented. Nothing has ever generated convincing text before. Not like this. You can talk to it. (Are you talking to it?)
You still have to want to talk to it, sure. But desires can be shaped.
If you made me Emperor of Earth, I would not ban them, nor prohibit their use.
Would you at least ban the forced use of them?
I genuinely have no idea how it's possible for a rational person to believe this. I have interacted with LLMs a lot. I still do 100% of my own writing, because why wouldn't I want my own voice to come through as strongly as possible? I also think it's pretty obvious that autoregressive transformers are not "awake" in the same sense as human beings, because why would they be? But to assert that LLMs exert some kind of anomalous corrupting influence on their users, above and beyond the usual influences of media, feels like exactly the same kind of bizarre left-field claim to me as suggesting that the models are ensouled. Like, this is on par with demonic possession in terms of how little empirical support it's got. LLM output is just text! If it was possible to blunt an arbitrary person's reasoning abilities through properly formatted text, the CIA would've been doing it left right and center since the 1950s!
TEL is a neurotoxic cognitohazardous substance which was deliberately added to fuel and polluted the air from the 1920s to the 1980s; by "cognitohazardous" I mean that exposure can negatively impact cognition. Public relations is a renaming of propaganda which supports both government and corporate opinions, founded sometime between the 1900s to the 1920s. The OSS, the predecessor to the CIA, published the Simple Sabotage Field Manual in the 1940s, including tips on how to deliberately derail collaborations with specific patterns of speech and text. We have over half a century of research on TTCs, memes which can be spoken or printed and which tend to arrest flowing cognition.
To be crisp, I don't assert demons. My full argument is lengthy but can be summarized as follows. Modern psychology asserts that we have two systems of cognition: everything else, and economic rationality. I claim instead three systems of cognition: hormones and nerves, rumination, and memetic association. Transformers can't possibly have hormones and nerves, and they structurally cannot ruminate (chain-of-thought tokens mathematically can't substitute for indefinite recurrences!), so to the extent that they resemble our cognition, they only resemble our ability to manipulate and emit memes. Surprisingly, people appear to prefer piles of memes to actual humans, so they prefer LLM chatbots to actual humans too.
You might like Baudrillard, who has written extensively on semiotics and constructed a whole theory, quite close to memetics, around what he calls "simulacra". Hyperreality occurs when signs have entirely supplanted what they used to signify, and no longer have any connection to anything "real".
Written before the advent of the internet or social media, the terminology rings dated but the ideas are quite prescient.
In La Transparence du Mal, one of his later books, he even ruminates a bit on AI:
If men create intelligent machines, or fantasize about them, it is either because they secretly despair of their own intelligence or because they are in danger of succumbing to the weight of a monstrous and useless intelligence which they seek to exorcize by transferring it to machines, where they can play with it and make fun of it. By entrusting this burdensome intelligence to machines we are released from any responsibility to knowledge, much as entrusting power to politicians allows us to disdain any aspiration of our own to power.
Maybe a last nit to pick here:
they structurally cannot ruminate (chain-of-thought tokens mathematically can't substitute for indefinite recurrences
You can trivially train a transformer to "count parentheses", and, with input integer n, output a string with balanced n parentheses. Your token alphabet can also encode a state machine with transition probabilities in the transformer weights, where the recurrence need not be definite. Or did you mean something else?
I personally don’t touch LLMs with a stick. I don’t let them near my brain.
So I'll take whatever this person has to say about LLMs with a heavy grain of salt.
Hmm, interesting. I realize that i don't myself have a consistent criteria for these kinds of things. E.g., if a celibate priest gives a sermon about how to raise a family, or about the immorality of fornication, my immediate reaction is to, similarly, disregard such opinion: "what do they know about this?".
But then, if a person who's fit and healthy, and has avoided junk food for all their life, shares their experience and opinion about how to do that, i probably wouldn't have the same knee-jerk reaction. I might even think that their experience and advice is worth hearing, even though they probably don't know much about how to deal with unhealthy habits.
So there seems to be a preconceived bias towards how i react to such absolutist attitudes. If i sort of already agree with the conclusion, then i'm more prone to ignore the absolutism. Whereas if i disagree, i might point out the absolutism as problematic in itself. And i wonder if this is not a general thing we tend to do.
I think you can learn a lot of something by simply observing it. A priest for example will encounter a lot of families and actually interact with them on various levels. So they might have some useful insights based on those observations. The other subject, well that's a can of worms I will not be touching in this reply :P
I even think an outside observer might have keener insights on how something impacts people. Many parents simply don't really have the time to truly stop and think about a lot of the finer details. It is why advice from parents often boils down to "we did it like this and it turned out fine". Sure, things might have turned out fine but there is of course a whole host of circumstances and context that also influences everything.
In that same way I do think you can have a valid base opinion about LLMs, how they impact people who work with them, etc. Purely by observing those people, reading up on articles written by other people, etc.
People who are deeply involved with all things AI might lack objectivity due to that proximity. While someone observing everything from the outside can have a clearer vision on the impact.
So, I think it is worth at least listening to what an outside observer has to say.
Oh, i think that perspective is something i was missing. I hope i remember this next time i prejudiciously think badly of someone who gives their opinion from these kinds of "observer" POV instead of from direct experience. Thanks! :)
About the topic at hand of LLMs, i personally don' see anything bad or weird about the author "not touching LLMs with a stick". I didn't want to mention that in the previous message because i was wondering if i was falling into this sort of confirmation bias i mentioned.
Tangentially, i feel a sort of echo of what happened with social networks when they first became popular. I remember people saying similar things around 2008-2010, like social networks being the future, being game-changers, and that anyone in tech with a modicum of self-worth should use them, at the very least to not be left behind. Now, with more than 15 years of hindsight, can we really say that the people who preferred to stay out of Facebook or Twitter and observe these things from the side were making such a bad choice? I don't think it's so clear-cut, but we definitely know more about the negative consequences of so-called "social" media now. And also that their effects on both societies and individuals have been at the very least a mixed bag, if not mostly negative.
I can't know what we'll think of of LLMs in 10 or 20 years, but it's easier for me to imagine overall negative consequences than positive ones. So, while not being as adamant as the author, i too prefer not to embrace these things so readily, and especially not to depend on them in my day-to-day activities.
Funny you should say about celibate priest.
Most saints in my region were family men and women. But there was one lifetime celibate dude [0] who wrote a bunch of little poems on advice about living a healthy married life. Those poems are still revered for how true they are.
Point being, its easy to dismiss these things, but you might just be missing out on something very real.
But then, if a person who's fit and healthy, and has avoided junk food for all their life, shares their experience and opinion about how to do that, i probably wouldn't have the same knee-jerk reaction.
I feel like you kind of should. If somebody is fit and healthy and has avoided junk food all their life there's three possibilities: they have developed good self-control and you can learn something, they naturally have good self-control and probably won't have anything useful to say about how to develop this from scratch, or they naturally dislike junk food and their experience is totally alien to yours. From the inside, these are hard to differentiate! People tend to underestimate how much natural variation there is in human minds.
Take the opposite case: someone who spends all day vibe coding, never write code themselves. Should we take anything they say about writing code seriously?
One doesn't need to use LLMs to grasp the ecological and social costs associated to them, or to be aware of the policies of the people and companies who develop and push them. For many, me included, this is a sufficient reason to never use LLMs. So the problem is solved whether LLMs would be "working for me" or not, whether they would save me time or not, whether I would enjoy using them or not, and the answer is "I don't want to use them", in all cases.
But let's imagine a parallel universe where training and using LLMs is harmless. Then I still wouldn't need to use them to know that they would steal from me what I enjoy the most in my work and hobbies: building, programming, thinking, debugging, learning. In that universe, whether they are working or not is also irrelevant: using them would be a net negative for me in all cases.
Agreed, I don't see how this can be a virtuous position to take regardless how valuable you think LLMs are.
Much like the world wide web, LLM's are potentially life changing for everyone in tech and we should at least touch them enough to build our own opinions.
I know I'm building systems in classification and data cleaning I couldn't build without them. But I'm not sold on how much net benefit I get in writing code.
I don't see how this can be a virtuous position to take
What if someone truly believes that an LLM is like an addictive and mind-altering drug? In that case, it would be reasonable to study them from a distance, and observe their effects on others, but not partake.
Well you could say that about assembler or Haskell. Not every honest belief is defensible, and it doesn’t change that if you hold that belief you don’t have sufficient experience to form an opinion.
I would never expect someone commenting on the psychological or social effects of something to be a technical expert on the thing, and vice versa. (Or am I misreading you?)
How much experience is necessary? I hate IDEs even though over the past 40 years I've used them for, maybe collectively, 10 days. And yet, I can defend my position.
Tbh the ides of today are not the weird clunky garbage of yesteryear. For starters vs code is an ok editor.
So I was using IDEs older than 20 minutes. Gotcha. But what about my question? How much experience is necessary?
Some? Experience with something completely outmoded is often irrelevant. If the last car you drove was a model t you probably need at least one lesson on a modern car. Similarly, LLMs from last year and IDEs from 20 years ago aren’t especially relevant to today.
The last time I "talked" to an LLM was last week, and I got an incomplete answer to my first question. Then it made up an answer that wouldn't work. And my question wasn't a hard question at all---it was basically a search-type question. I stopped with the confabulating Markov chain and finally found my answer, in less time, using a normal web search.
But that was last week. Apparently, that LLM is now obsolete and I should try again? How many times do I need to do this before I conclude I'm insane for expecting different results?
The answer is always, always that you are holding it wrong. Now put the glue on the pizza damn it.
I don't think I agree with you. If you have fundamental issues with something, it's likely not a slightly different thing that will solve them. You wouldn't ask a vegan person "but have you really tried all of the animal meats?" for example. Likewise for people subject to severe motion sickness: the brand, model and features of the car are irrelevant.
I’m pretty sure that the smoothness of the ride and sight lines do affect motion sickness. Some things can be in the same category but be wildly different.
You're relying on the implicit stigmatization of psychoactivity at large, but even the most "evil" drugs are often (sometimes widely) used in medical practice. Fentanyl is the most widely used synthetic opioid, methamphetamine is prescribed as Desoxyn to ADHD patients, even MDMA is now being explored for PTSD/other therapeutic uses.
So if they believe caution is warranted, that's fine, drugs can (and do) cause harm.
I feel that the current zeitgeist goes above and beyond "let's regulate opoids, we have a crisis on our hands" (really stretching the analogy here, but OK) and is veering into "opioids are the devil incarnate and we should use the power of prayer instead" territory.
I feel the opposite. The current zeitgeist to me seems to be more like "you have to use a lot of fentanyl before you are allowed to have an opinion on the risks and benefits of fentanyl".
"You can't claim fentanyl should be banned from society because you have seen a lot of junkies"
Well observing that people being junkies is bad is NOT a sufficient justification for prohibition. Further observation shows that prohibitive regimes are far worse at reducing the number of addicts.
"I don't understand how anybody with even a cursory knowledge of pharmacology can be so opposed to fentanyl!"
I'm not sure that's quite what's happening (even in the US, hotbed of AI froth).
Feels a lot more to me like smoking/Opium wars (large corporations/governments pushing ahead and actively working against regulation). So if AI and LLMs are actually bad, they will only do anything about it once the costs are abundantly clear (and after quite a bit of harm has actually occurred).
But it's not obvious to me AI is as harmful as smoking, I suspect it falls closer to video games/internet/social media (which have yet to be regulated, fwiw).
I 100% agree with the points about control and power ... But this is limiting, because the tools are very flexible:
I personally don’t touch LLMs with a stick
You can use them in the "recommended" way, which may atrophy your thinking
Or you can use them to learn, and increase your control / power / agency
What I realized is that BEFORE LLMs, we were already in kind of a bad situation ... A large amount of critical open source is written by large corporations, and many of them write bad code because they are hiring so many people
e.g. literally throwing bodies at Chromium and WebKit and so forth. Kubernetes YAML, etc.
The average programmer or Linux distro has little agency when it comes to those things ... e.g. I wrote here about the new pattern being "your job is to LLM the YAML"
https://news.ycombinator.com/item?id=41737740
I generally agree with this article in that PROGRAMMABILITY is the core of Unix
So basically I'm saying it is possible to use LLMs to learn ideas with wide applicability. They are not limited to the treadmill of tech that's common in industry jobs (e.g. I recall people chafing against the front end framework of the year, as well as the devops tech, etc.)
Another example is that I wrote a comment yesterday about using Claude to explain the 2024 National Association of Realtors settlement: https://news.ycombinator.com/item?id=46040967
It was intended to stop buyer's and seller's agents from colluding against their clients' interests (i.e. your interests)
Although it's not clear if it's working now -- it's a recent development, and agents are working around it
If you're buying a house, then saving say 2% of the purchase price can fund other things in your life (increasing your agency)
Or it can let you retire earlier. So that is relevant and actionable information, and I found it's not understood well by some financially literate people. I don't think it's well covered well in the media -- I point out the counter-spin by the NAR in the thread
Claude actually does a better job (with the right prompts)
One of the biggest problems with LLMs is how intimate the relationship between the user and the model are. A lot of the companies hosting the best models have a lot of data retention and probably more personal details than you would find in a email or instant message.
These types of posts are increasingly common, is this some type of concerted trolling effort?
Just last week I was replyng to someone who, as best I can tell, earnestly believes LLMs are entirely useless and will remain so indefinitely. Their architecture would inherently prevent them from performing any meaningful task efficiently.
Luddites are nothing new, and AI is definitely in a bubble (which might be concerning to American onlookers, I don't know), but what I don't get here is why technical people, who arguably should have better understanding of the internal workings and limitations of these models, are so opposed to them?
Do LLMs represent by proxy capitalism's excesses to Americans, so the modern Luddites reject them much like the high-skilled weavers protested the power looms? Not by intrinsic objection to the technology, but how it redistributes wealth?
but how it redistributes wealth?
YES! Absolutely! The article literally points at that:
Enormous physical infrastructure designed to convert capital into power, and back into capital. Those who control the infrastructure, control the people subject to it. [\n] AI systems being egregiously resource intensive is not a side effect — it’s the point. [\n] Craft, expression and skilled labor is what produces value, and that gives us control over ourselves. In order to further centralize power, craft and expression need to be destroyed. And they sure are trying.
why technical people, who arguably should have better understanding of the internal workings and limitations of these models, are so opposed to them?
Because "being technical" doesn't preclude also "being philosophical and political" and so on.
I know the limitations of these models and I see the way the megacorporations have recklessly deployed them at massive scale and speed —with user interfaces that don't focus on the limitations, but rather explicitly advertise themselves as chats with an "intelligence"— and I perceive all the consequences as very negative for humanity at large.
Even outside of the psychological effects of the chat interface specifically, the whole idea of the capability to rapidly generate massive amounts of very convincing statistically-likely text (and images and other "content") seems to obviously benefit bad actors way, way more than good ones. It's the capability to spam better. To pollute the information space with "something" more convincingly and efficiently than ever before.
And I firmly believe that "trying to filter/shape/curate a sea of crap into something decent" is a fundamentally way worse way of doing anything, a worse life, than having to engage in the intentional craft of creating anything at all with normal, reliable, deterministic tools and the power of the human brain.
Because "being technical" doesn't preclude also "being philosophical and political" and so on.
I'm fine with philosophizing and politicking, but I feel we're wandering off into "capitalism bad, technology use capitalism, make company richer, technology bad". Not every discussion on ActivityPub devolves into the perils and evils of social media. Not every discussion on IPv6 devolves into a discussion into what an informationally connected planet entails.
I know the limitations of these models and I see the way the megacorporations have recklessly deployed them at massive scale and speed —with user interfaces that don't focus on the limitations, but rather explicitly advertise themselves as chats with an "intelligence"— and I perceive all the consequences as very negative for humanity at large.
So is the objection to OpenAI and Anthropic and Google and Microsoft? Why has "AI" become a catch-all metonymy for the villain of the day, whether corporate monolith or abstract economic mechanisms?
I'm fine with discussing how lasers are used for evil in military contexts, but that doesn't preclude a constructive discussion around their physics. We can't bring up Hiroshima every time someone is talking about nuclear energy.
Even outside of the psychological effects of the chat interface specifically, the whole idea of the capability to rapidly generate massive amounts of very convincing statistically-likely text (and images and other "content") seems to obviously benefit bad actors way, way more than good ones. It's the capability to spam better. To pollute the information space with "something" more convincingly and efficiently than ever before.
People have advanced similar arguments about the internet, and I'm not sure I bought them then, and not sure I buy them here. It's easier to send spam emails than spam mail. Bytes are cheaper than paper. Does that, on its own, make snail mail superior to SMTP? Whenever means of disseminating information are democratized, people fear-monger about their potential misuse (internet, printing press (see Samizdat, etc). People will be able to say things I don't agree with! They might even be able to easily make a video!
This argument reduces to an argument against free speech, and I don't find it particularly compelling. The best defense against propaganda is a well-educated, critical populace, not centralized censorship.
Is it really so offensive that now your neighbor can generate misinformation, instead of the carefully crafted, well-financed geopolitical adversaries (CIA, FSB, etc)? If anything, it would seem to me we have devalued misinformation by making more of it. Have self-published smut novels on Amazon killed literature?
And I firmly believe that "trying to filter/shape/curate a sea of crap into something decent" is a fundamentally way worse way of doing anything, a worse life, than having to engage in the intentional craft of creating anything at all with normal, reliable, deterministic tools and the power of the human brain.
Again, similar arguments have long been posed against the internet. "If anyone can contribute to Wikipedia, how will we know anything is true?"
What if LLMs save people from the "intentional craft" of writing boilerplate compliance emails so they can engage in actual aesthetic pursuits? I'm not worried about LLMs replacing authors. But if your white color middle management job was wrangling org state and sending emails all day, that might be under threat sooner rather than later.
I'm fine with philosophizing and politicking, but I feel we're wandering off into "capitalism bad, technology use capitalism, make company richer, technology bad". Not every discussion on ActivityPub devolves into the perils and evils of social media. Not every discussion on IPv6 devolves into a discussion into what an informationally connected planet entails.
But that isn't the question you asked. You asked why technical people aren't protected by their expertise from being opposed to LLMs, and you were given a direct answer - because they're unrelated. If your real concern is why conversations keep going in this direction, you should look elsewhere than at the framing you started with. You will not find the answer in the difference between experts and non-experts.
You misread me, so let me reiterate more carefully.
I did not ask why are people opposed, I am asking why are people so opposed.
My frustration with these discussions is I think there are constructive conversations to be had about regulation, about harm, about safety, about wealth, and especially about the technology itself, especially on a tech forum.
The smash the GPUs segment of blog posts get 100x more upvotes and comments than any AI research/discussion of actual LLM use or practice.
So your frustration comes from the popularity of these discussions or their existence in general?
I mean, I get AI enthousiasts fall on the minority side in lobsters, but lets be honest there are enough vehicles that are thirsty for anything genAI related nowadays. Hacker News, LinkedIn, tech conferences, any big city is hosting weekly meetups to talk about it, I could go on giving examples but I hope you get my point.
What’s the problem if this community likes to bash on it and have a less enthousiastic view of it?
So your frustration comes from the popularity of these discussions or their existence in general?
Neither, actually.
I would just want these discussions under something like "ai-ethics" instead of vibecoding. I think there's a place for both here but they should be separated.
What’s the problem if this community likes to bash on it and have a less enthousiastic view of it?
My issue isn't so much with threads like this, as much as I'm not interested in it. My issue is when threads about using LLMs get completely railroaded by people who want nothing to do with them. Imagine if every php thread had comments of people just dunking on php, often in a way that's completely irrelevant to the article under discussion, and then imagine if those comments always go right to the top.
The fact is that lobsters just isn't a pleasant space to talk about this particular topic.
i think it being one of the only pieces of software you can't build yourself contributes to it. you actually need capital in order to train a model, so any anti-capitalist sentiment is stronger
most software, you can build on an old laptop in a basement. the capital is so you can build a business around it and scale faster
Yeah but you also can't build a Google-like search engine yourself. It took Microsoft at least 10 years to do that, and there are probably only 2 others of the same tier -- from China (Baidu) and Russia (Yandex).
I'm pretty sure that some countries literally tried and failed to build these systems
I mean honestly that does bug me a bit. I can see somebody abstaining from using search engines for that reason, but it's probably quite rare
The funny thing is that there is WAY more competition in the LLM space than there was for search engines! And LLMs are already more open source than search ever was.
Both Microsoft and Google have lots of capital, and yet both of their searches are really poor at surfacing information. 10 years ago Google search was good for finding information, but the lure of ad revenue changed their perspective on providing results.
What does "not build" mean here? You have excellent open-weight models, by any reasonable standard, that run on consumer hardware. You can fine-tune and tweak them with RL on that same hardware.
I've been around technology long enough to comfortably say that this era of computing is vastly more accessible, than say, mainframes before the PC boom. Even the Commodore, a mass-marketed "accessible" PC, still cost ~$4k USD adjusted for inflation. For $4k today, I can build a machine with 2x3090s and 48GB of VRAM.
And we're early. There's no reason to not expect smaller models to become more capable, hardware to become more specialized & software infrastructure to be more standardized (CUDA being an "AI moat" is the most ridiculous thing, same with NVIDIA being 10x the market cap of AMD).
You have excellent open-weight models
Most of which are trained through processes that are not open and hardware not available to most consumers. At least not for the periods of time that you need that hardware. So while those models are available right now they are not like OSS software where you can fork them and keep them updated for future use.
As far as hardware requirements go to actually go ahead and create models, we are right back in the mainframe era. Where only big companies and universities with the funding have the funds available to create these models.
Because that $4k machine you are describing, puts you on the consumer end of the tool chain. Not the producing side. And yes, right now you have a choice out of a plethora of models. But that is also because massive amounts of VC money is being thrown around so a lot of companies don't even have to worry about the costs of training all these models.
Once the hype slows down and profit actually needs to be made I don't think you will have the same amount of models available anymore.
But is that necessarily a concern though?
I'm fine with models as an analogy to hardware, where the open/consumer fabs/tech to actually manufacture hardware lags a few decades behind state-of-the-art. I'm OK buying a laptop, with Intel Management Engine and all the other BIOS restrictions, even though fully open alternatives exist. I expect in 10 years the open laptops will be better than the ones I have today.
The angle under which the hardware analogy is weaker is depreciation: hardware physically deteriorates over time whereas an "open weights" model will be just as powerful in 10 years as now, it's just everything around it might be light-years ahead.
Once the hype slows down and profit actually needs to be made I don't think you will have the same amount of models available anymore.
If you mean released instead of available, sure. But all the models that have already been released, that honestly work pretty well at a bunch of things, will still be around. You can't unpublish the weights that are already out there in the wild.
If you mean released instead of available, sure. But all the models that have already been released, that honestly work pretty well at a bunch of things, will still be around. You can't unpublish the weights that are already out there in the wild.
At first that will be fine yeah. Until the knowledge cutoff becomes too big.
Most of which are trained through processes that are not open and hardware not available to most consumers.
For me, the ladder is this:
Open weight fine tuned model, which relies on Open pre-trained model, which relies on Open training data and process
While there are a good number of open weight fine tuned models, there are fewer that also have an open pre-trained model, and fewer still that provide open training data and process.
The Allen Institute for Artificial Intelligence provides Olmo, the latest found here, that meets my requirements.
None of this addresses "hardware not available to most consumers". That's a fair criticism. There, I delineate between training and inference.
For inference, I'm personally working on a BitNet model, see here, that - because of its architecture - should run 2x as fast as other architectures with less memory.
For training, well I'm not that far yet, but I take some inspiration from Deepthink's success training without access to the Nvidia hardware US companies do. I acknowledge that they have effective state sponsorship. That said, more than anything else I've seen, the papers on this are freely available on arXiv, making more possible.
I just want to point out that TFA makes a direct objection to the technology. There are many, many criticisms to make of LLMs before turning this into a debate on capitalism and tech. One is not a "Luddite" for making them.
Honestly, I'd welcome those discussions under vibecoding 1000x more than the rehashed "I reject AI, no matter how good it gets" blog posts.
Criticize hallucinations (quantify it!), show how RAG doesn't solve it, show how RL is some domains is fraught.
One is not a "Luddite" for saying mechanical weaving is inefficient or should be improved, one is a "Luddite" if you go out saying we should smash the looms.
why technical people, who arguably should have better understanding of the internal workings and limitations of these models, are so opposed to them?
the instruments of a thing are not the thing! Is a piano music? Is a great luthier necessarily a great guitarist?
The attempt is being made to automate the musicians. And the writers, poets, scientists , and politicians along with. These models only remain tools if the AI builders fail.
What will the world look like when intelligence can be outsourced to a machine, and learning no longer has value beyond self amusement?
"What will Chess look like once machines beat humans?" I guess different cheating scandals, but looks pretty similar to me.
"What will the world look like when memory can be outsourced to books, and people no longer need to remember everything through oral history?"
"What will the world look like when walking can be outsourced to a car, a train or a plane, and we only jog and walk with friends for our own enjoyment and health?"
That's an oddly defensive sounding response. If you think that it's a good thing for human brains to be outperformed, why not own it?
If you mistake my three, specific, sarcastic examples that illustrate how technological advancements don't necessarily lead to human obsolescence, for defensiveness, I don't know what else to say.
You sound like a Butlerian Jihad priest.
Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them.
Making AI that can do a better job at thinking than humans is an explicit goal of many large AI labs; if you ask them whether they were aiming to do for all thought what books did to oral history, the answer is a clear yes.
It's possible to think it's a good thing. It's possible to think it's a bad thing. It's possible to think we're going to fail at that goal. It's possible to think that humans will manage to structure society such that we'll chose to avoid using AI, even if it would be more efficient. There are other potential futures.
But comprehensively outdoing human thought is an active, explciit goal of AI research orgs.
One thing is what they say they want to do, another thing is what they actually want to do, and a third thing is what they are actually able to do. I think that these things are very different. I don't think (or more precisely, I can't believe) that anyone serious actually believes that LLMs will one day be considered intelligent. For one, I am personally sleeping tight not expecting to see any artificial intelligence within my lifetime or in the following couple of centuries. We have far more pressing problems caused by people (this "AI" bubble being one of them, for many reasons but none of them being that LLMs might ever become more intelligent than anything with a nervous system (they won't)).
Let's set aside sarcasm and comparisons and analogies and metaphors for a second. I'm genuinely curious what you think will happen if the AI companies can indeed manage to build AI models that are genuinely smarter than people, including true multi-modality like the human brain + capable robotic bodies that give them as much agency in the real world as a human.
Would you care to take a guess about how this development will affect the lives of the 99% given the current political and geopolitical trajectory?
No-one is discussing superintelligence here.
Again, the "AI" metonymy, as I've alluded in other comments, is problematic. It allows people to simultaneously, in one fell stroke, apply all possible criticisms to all possible instantiations of probabilistic models. So they're saying at once that LLMs are useless, produce "slop", are unreliable and untrustworthy (all valid criticisms, for what it's worth, and active areas of research how to make these models better), and also the precursor to an intelligence that will vastly surpass all existing humans.
My objections, and frustration with these discussions, are the people engaging in bad faith: wholesale rejection of "AI" (and see the linked comment at the top of the tree, it really is trolling), and when you prod them to elaborate, it's a smattering of vague circumlocutions about inaccuracy and resource use and capitalism.
There can be coherent discussions about how to make LLMs use less resources. The large AI labs care a lot about this. It's their bottom line.
There is also ways to make LLMs cite sources, which turn them into a conversational search engine (RAG). Sure, they still hallucinate and make mistakes sometimes, but they tend to perform a lot better.
I think discussions about regulation and wealth redistribution are important. Retraining workers (and lack thereof) during the first industrial revolution caused significant political upheaval. But these can be engaged with constructively, instead of "AI is useless and evil no matter how good it gets".
I understand your frustration about the multi-front criticism of the fuzzy notion of "AI", but I think this particular thread became specifically about superintelligence.
And I don't think you can really separate the superintelligence discussion from the discourse around the current generation of LLMs, because the AI companies are committing to trillions of spending, which can end well for them if and only if they end up with (something like) superintelligence (as they keep promising) before the investor and creditor patience runs out. So the superintelligence discussion is not some skeptics reaching for far-fetched arguments. There is a valid argument that by participating in the generative AI revolution (which I personally also do), you are indirectly supporting their explicitly declared forced march towards superintelligence. Then isn't it fair to ask, what might happen to (the vast majority of) humanity if they manage to get there under our current (geo)political context?
Superintelligence is also a fuzzy notion, and the goalposts have long been moving.
If a machine is expected to be infallible, it cannot also be intelligent.
At many reasonable tasks, various (even simplistic) models will outperform humans on in-distribution performance. We have "superintelligent" chess models, vision models, etc. Language models could arguably have superintelligent recall vis-à-vis to their training data. They hallucinate, but I also misremember things from books I read 15 years ago.
Now, if we want to take superintelligence to be the (as of yet, purely hypothetical) pie-in-the-sky task perpetually improving generalized learning machine (continuously getting smarter on every reasonable information task as it slurps all the data it can), I would speculate perhaps similar to nuclear weapons? Is the know-and-learn-everything machine truly more powerful than the destroy planet button? It would feel that it would be more useful for good (research, discovery) but not much more powerful in its ability to cause harm.
What will the world look like when walking can be outsourced to a car, a train or a plane, and we only jog and walk with friends for our own enjoyment and health
Fatter and less healthy. Obviously suburban design that basically requires cars is also a required cause of this problem. Obviously it doesn’t follow that trains were a mistake but it does mean we shouldn’t Pooh Pooh that basically beneficial things and basically harmful things can look alike
Really? Again, with the American-centric perspective. Many of my European friends don't own a car, their cities are not designed for it. Same in Japan and much of East Asia.
The point is not that a car-centric urban design is superior, it's that motorized vehicles aren't inherently good or evil.
I'd be worried if we were designing society and education around LLMs. But not worried about the actual technology underlying it.
If I understood correctly, the two fundamental problems the author identifies with "AI" are control over your thinking process when writing, and the issue of concentrated power of tech companies that happen to also serve the interests of the current U.S. administration.
I'd wager that a small local LLM from a Chinese lab addresses both problems, though only partially. They are so dumb there's no temptation leave thinking to the robot even in the darkest hours of a writer's block. Also you run them on your own hardware and the interests of its creators directly oppose those making huge, closed models.
I share author's sentiment, however I try not to be too reactive to AI tools.
Most AI agents replaced google for me, and 15 years ago I was amazed how people pay me to know what exactly I have to google. Now that magic is gone and they pay me for knowing what to do and what not.
I've trying various agents recently and I like the ideas they suggest, even though I remove half of the code. I like it as a tool to explore ideas and learn. I doubt it will get better, but if it does - good.