Unauthorized Experiment on r/changemyview Involving AI-generated Comments
37 points by carsyoursken
37 points by carsyoursken
Seems straightforwardly unethical and the claims made for justification are pretty weak. Even if you can justify the claim that there is such a large gap in our understanding of how to convince people of things (there obviously is not, and LLMs do not change anything about what we know about convincing people) that it merits experimentation of this nature, what is the justification for this experiment to fill that gap? The university’s response of a “formal warning” seems insufficient.
The experiment’s value seems extremely weak, the ethics are obviously questionable (tbh the question is perhaps only one of degree, it’s obviously unethical and the uni seems to grant as much), so any sort of cost/benefit here seems unjustifiable.
Taking this experiment in the best possible way, what would it show? That an LLM can convince people of things? That is obvious. That an LLM that takes on a human persona is effective at convincing people of things? Again, obvious. That using information about a person to target a response increases the chances of convincing them? This is well researched and obvious. The introduction of an LLM changes nothing here and there is already a body of research on the topic - justifications for this based on “we don’t know and this is the only way to find out” are just false.
Reading the examples that the llm generated,I can only presume that the researchers are terrible humans. Like really nasty stuff.
“not stricty a bot”, “not strictly spam”, “impossible to ask user consent” etc : the Overton window of AI-human interaction shifts under steady deliberate pressure. What will it be next time?
They’re already trying to normalize scraping everything they can get their hands on. I talk to people working in GenAI on here and they won’t answer simple questions like “where do you get your training data?”
Makes me think of the time that the University of Minnesota tried to introduce security bugs into the Linux kernel as an experiment.
I haven’t seen someone do something ethical with an LLM since they stopped being simple toys.
I use (vision) LLMs to help write the first draft of the alt text I use for images on my blog and social media. I never publish that text without first reviewing it and (usually) modifying it to improve it in some way.
The quality of alt text I’ve been writing has gone way up as a result. I used to often do the bare minimum, now I provide alt text that really does communicate the information in the image clearly.
Is that an unethical use of an LLM?
Here’s a dashboard showing all of the alt text on my site, and letting you search it too: https://simonwillison.net/dashboard/alt-text/
I just built that using an LLM to write the (extremely convoluted) SQL query. Unethical? https://simonwillison.net/2025/Apr/28/dashboard-alt-text/
The following wall of text was written by me, by hand, with no generative AI assistance. It, and this header, are licensed CC-BY-NC-ND 4.0 with the additional restriction that this text must not be input into any generative AI program of any kind for any reason, by any person or corporation, at any time.
TL;DR at the bottom.
I use (vision) LLMs to help write the first draft of the alt text I use for images on my blog and social media. I never publish that text without first reviewing it and (usually) modifying it to improve it in some way.
That would be something like object classification model plus an LLM under the hood, right? Like, the LLM itself isn’t processing the image? Genuine question.
The quality of alt text I’ve been writing has gone way up as a result. I used to often do the bare minimum, now I provide alt text that really does communicate the information in the image clearly.
People who read alt text understand that sometimes access needs conflict and it can be difficult to provide super detailed alt text. That said, it’s also a learned skill to know what’s most relevant, that you can get better at if you try, which is not something that happens when you outsource to an LLM.
Standard OCR tools that are much less computationally expensive are still helpful too, for transcribing text in an image quickly.
Additionally, I know from personally reading about it that many folks who rely on alt text very much dislike AI generated alt text, even with the best of intentions, for all the reasons I’ve mentioned and others.
Is that an unethical use of an LLM?
It’s less unethical. Where do you get your training data you use to teach your models how to classify objects or describe a scene? Where does the hardware it runs on and the electricity it consumes come from? Most importantly, is this an efficient or worthwhile use of those resources or would something that requires more human effort but be less polluting also work?
One of the core pillars of my stance against LLMs and GenAI is the moral/ethical cost of the compute and electricity used to run them. This is in addition to other pillars such as mass scraping and plagiarism, rampant misuse and abuse of GenAI tools, especially for spam, and ubiquitous attempts at rent seeking by GenAI firms and tech companies looking to recoup their investment.
Here’s a dashboard showing all of the alt text on my site, and letting you search it too: https://simonwillison.net/dashboard/alt-text/
Cool app.
I just built that using an LLM to write the (extremely convoluted) SQL query. Unethical? https://simonwillison.net/2025/Apr/28/dashboard-alt-text/
For the other reasons stated, yes. But more importantly, because I don’t think that’s an interesting point to argue about now:
What could you have learned about writing SQL, maintaining SQL queries, or working with relational databases, that you haven’t or won’t learn because you outsourced to an LLM?
You can’t do an internet search against an LLM like you can against e.g. StackOverflow. As awful as StackOverflow can be to use, there is a mountain of knowledge on those websites that can be queried with just a little bit of thought and learned skills on how to write (traditional, not AI-enhanced) search engine queries.
And, I know you know I despise generative AI already. I have seen you around before. I also know you work with generative AI professionally.
You have already bought in. Whatever costs there are to LLMs, you’re either ignorant of them or have decided they are worth it. Maybe you’re privileged enough to have access to more resources than the majority of humans alive today, or you’ve decided you can externalize those costs onto society at large, or both.
Either way, what’s the end-game here?
Because to me the end-game for LLMs and generative AI seems to be something along the lines of:
“Well, we’ve boiled every drop of water on the planet to cool our servers and generate electricity to power them, accelerated climate change by multiple orders of magnitude, enslaved billions to mine precious metals to build more computers, and now we’ve finally got enough compute to run our artificial general intelligence.”
“ChatGPT, how do we solve all of our problems?”
[…]
[“I have an answer, but you’re not going to like it.”]
In all seriousness, generative AI seems to me like a symptom of the capitalist goal of infinite growth on a finite planet. It became relevant at the perfect time, just when demand for computing power was starting to match the supply. Nobody needed a much much more expensive computer.
Not individuals or companies. People’s needs were (mostly) met, which in a rational world would mean more incremental, slow growth of computing power.
But that’s not enough to please shareholders. Then came a savior, the world’s most inefficient hammer. And now everything seems like a nail.
TL;DR: What you’re doing isn’t the worst thing ever but that doesn’t mean it’s a good idea. Or that it refutes my statement.
Rebuttal to Simon Willison’s lobste.rs comment © 2025 by dubiouslittlecreature is licensed under CC BY-NC-ND 4.0, with the additional restriction that it not be used with or for an generative AI system for any reason.
Reply
That would be something like object classification model plus an LLM under the hood, right? Like, the LLM itself isn’t processing the image?
No, it’s the LLM directly. It turns out you can tokenize images in a similar way to tokenizing text and train LLMs on a mixture of both. GPT-4 (originally as GPT-4 Vision) was the first model to do this well, but these days all of the major model families - GPT-4o/4.1/o1/o3/o4, Claude, Gemini, Llama, Qwen use the same trick. This includes models that run on a laptop - here’s the Ollama vision tag for trying those out. Some models are expanding to handle multimodal audio in a similar way - the Gemini series is particularly good for that. I’ve been tracking the evolution of vision-llms on my blog.
That said, it’s also a learned skill to know what’s most relevant, that you can get better at if you try, which is not something that happens when you outsource to an LLM
I’m very aware of that - it turns out I’ve written notes on alt text spanning more than twenty years (I backfilled that tag on my blog today).
An interesting challenge with alt text is that often it’s not as simple as replicating the text on an image exactly. Consider a screenshot of a table: for a sighted user, scanning the rows and columns for the most relevant information is easy - and they can quickly note “it’s a table, skip it” if they don’t need to dig in. Turning that screenshot into a multi-paragraph stream of text when really the point you are trying to make is “the top item beat the others by a large margin” isn’t ideal. For charts that’s even more of a consideration.
Something that’s truly surprising about the best LLMs (I tend to use Claude 3.7 Sonnet for this) is that they have good taste in what matters for alt text!
Consider this image: https://datasette-cloud-assets.s3.amazonaws.com/newsrooms/1.png
It’s a screenshot of a table UI, meant to illustrate a feature of my Datasette software. There are 8 visible columns each with 11 rows, all containing numbers. My Claude alt text writer came up with this:
Screenshot of a database interface showing firearm background checks data. The page displays “home / nics-firearm-background-checks / background_checks” at the top with user “alex-staff” logged in. The table shows “>10,000 rows” of background check statistics by state for February 2025, with columns for permit, permit_recheck, handgun, long_gun, other, multiple, admin, and prepawn counts. States shown include Alabama, Alaska, Arizona, Arkansas, California, Colorado, Connecticut, and Delaware with their corresponding numerical values.
That’s genuinely good! It doesn’t parrot out 8 * 11 = 88 numbers - it identifies that what matters in the screenshot is the columns and that it lists them for several state - and that they are all in February 2025 (the date column reads 2025-02).
The prompt I’m using for that (which I’ve iterated on a few times to get to this point) is:
You write alt text for any image pasted in by the user. Alt text is always presented in a fenced code block to make it easy to copy and paste out. It is always presented on a single line so it can be used easily in Markdown images. All text on the image (for screenshots etc) must be exactly included. A short note describing the nature of the image itself should go first.
Interestingly the model has NOT followed my “all text on the image must be exactly included” instruction here - it’s done something that’s a better fit for “you write alt text”. Claude has taste.
I’m not blindly pasting in the output of the model - I read and revise it before publication. As a result I’ve been learning a good amount more about what makes good alt text from this exercise. I’m getting new ideas from the models, and because the amount of time it takes me to write text for an image has dropped I can spend more time thinking about the best possible text and still save time overall.
Standard OCR tools that are much less computationally expensive are still helpful too, for transcribing text in an image quickly.
I built my own custom web-based (WebAssembly-driven) OCR tool last year to help with that: https://tools.simonwillison.net/ocr - though actually I got Claude tohelp me knock it together based on combining my code from two previous projects.
As mentioned above, OCR will get the text from an image but that’s not the same as mindfully producing the most useful summary of that image for the context of a screen reader.
that many folks who rely on alt text very much dislike AI generated alt text
Which is one of the reasons I don’t use it directly - I edit it myself first to make sure it’s the best alt text I can offer.
I’ve talked to a few screenreader users who are are very keen on these new vision LLMs. They’re a fantastic assistive technology, even taking into account their ability to make convincing mistakes. I got to participate in a podcast episode about this a few months ago too.
One of the core pillars of my stance against LLMs and GenAI is the moral/ethical cost of the compute and electricity used to run them.
I wrote about this back in December. I think it’s very likely that the environmental impact of running prompts through LLMs has been wildly overstated. My main reason for believing this is that the cost of running a prompt through a model has dropped by a factor of 1000x in just the last three years, as the models and the infrastructure for serving them has gotten more efficient. I expect (but cannot prove) that Netflix+YouTube continue to produce way more carbon than the LLM various companies do.
The energy costs for training them remains enormous, but my rough calculations are that it’s in the order of dozens off fully loaded commercial jetliner flights from London to New York - and that training cost is then shared by every prompt that uses that model in the future.
At the same time, the gold rush aspect of it is clearly burning enormous resources. I have a hunch that this is starting to slow down now - Grok 3, GPT 4.5 and Llama 4 have all demonstrated that spending 10-100x the energy on training doesn’t produce 10-100x the quality result, so we may find that labs focus more on being smarter about how they train as opposed to burning more energy than everyone else.
What could you have learned about writing SQL, maintaining SQL queries, or working with relational databases, that you haven’t or won’t learn because you outsourced to an LLM?
That’s the thing: it was my knowledge of SQL that allowed me to do this in the first place. I know that PostgreSQL can use regular expressions to extract data from HTML in the database because I’ve done that before, by hand. I also know that given the complexity of my database (a mixture of markdown and HTML across four different tables) the query was going to be gnarly - way more complex than I wanted to write by hand.
So I prompted this:
Give this PostgreSQL schema I want a query that returns all of my images and their alt text. Images are sometimes stored as HTML image tags and other times stored in markdown.
blog_quotation.quotation,blog_note.bodyboth contain markdown.blog_blogmark.commentaryhas markdown ifuse_markdownis true or HTML otherwise.blog_entry.bodyis always HTMLWrite me a SQL query to extract all of my images and their alt tags using regular expressions. In HTML documents it should look for either
<img .* src="..." .* alt="..."or<img alt="..." .* src="..."(images may be self-closing XHTML style in some places). In Markdown they will always beI want the resulting table to have three columns: URL, alt_text, src - the URL column needs to be constructed as e.g.
/2025/Feb/2/slugfor a record where created is on 2nd feb 2025 and theslugcolumn containsslugUse CTEs and unions where appropriate
There’s a bunch of SQL expertise present in that prompt. “Use CTEs and unions where appropriate” is in there because I wanted it to write the query how I would - as a bunch of CTEs pulling out the data from the different table that are then combined with a UNION at the end.
And that’s exactly what it did for me. It wrote the SQL query I had in my head - 100 lines of pretty messy PostgreSQL - and the whole exercise took about 5 minutes including follow-up prompts. I am very confident I could not write a 100 line SQL query like that in just 5 minutes unassisted. I’d rather not do the project at all.
Once again, I end up learning more about SQL from using an LLM in this way. I reviewed the code it wrote and tucked away the tricks in my mental list of “things I know can be done in PostgreSQL”. I saved the resulting query and bookmarked the chat conversation too in case I ever want to review it again in the future.
My commit messages and GitHub issues these days are littered with links to LLM transcripts. Each one is a little lesson I’ve learned about something that works or doesn’t work, and there are thousands of them now.
I have seen you around before. I also know you work with generative AI professionally.
Depends on your definition of “professionally”. I don’t make much money from this stuff yet - I’m self-employed and self-funded (thanks to an acquisition a bunch of years ago, but that money won’t last forever) and I spend most of my time working on my open source projects and writing about my research.
I’m starting to pick up bits of AI-related consulting now - mainly in the form of paid consulting calls - which is a fantastic fit for letting me spend most of my time on my open source work. I’m also spinning up a SaaS company that includes some LLM-driven features.
One last note: if you’re interested in AI ethics I’ve published 158 posts on that topic in the past few years, which weirdly may make me the most prolific blogger about that subject? (I’m disappointed not to have much competition there, people should blog more!)
I also published one of the first research projects on the ethics of AI training data used for Stable Diffusion back in September 2022, and I’ve been tracking ai-ethics + training-data since then.
I just want to thank you, simonw, for this comment. It opened my mind. This is why I’m on Lobsters.
It turns out you can tokenize images in a similar way to tokenizing text and train LLMs on a mixture of both.
Huh. How do you tokenize images, then?
There are a few different techniques from what I understand. Vision Language Models Explained on the Hugging Face blog is a good high-level overview.
This is very troubling. Reminds me of when Jon Stewart was on the Late Show with Stephen Colbert talking about the Wuhan lab leak theory. His critique of science was something like: if science wants to find out ‘did curiosity kill the cat?’ they’ll make an experiment to kill 1000 cats. I think science, like capitalism, are amoral, and have no incentive to ask “should be we doing this?”. The ends-justify-the-means response from the University of Zurich adds an additional layer of concern because it doubles down on non-consent and sets a dangerous precedent, as the reddit mods noted.
I think science, like capitalism, are amoral
this is a bit naive, as ethics in scientific research is well established practice. This is why disciplines that study living beings have IRBs (institutional review boards): https://pmc.ncbi.nlm.nih.gov/articles/PMC11060189/. The problem seems to be the leaky boundary between the IRB and the research team.
My point was that these frameworks are amoral because morality is human. Frameworks can’t have a morality, it’s a label or a component or property that humans imbue the frameworks with. Those review boards are there precisely because the frameworks themselves have nothing to say about morality.
if science wants to find out ‘did curiosity kill the cat?’ they’ll make an experiment to kill 1000 cats.
There are a lot of ways to answer this question empirically in which no cats die. You can survey vets on the topic and create an observational study, for example. In fact, that’s almost certainly the cheapest way to get the most data and where you’d want to begin. You can also use models for cats - maybe I create a simulation in which curiosity is defined as novelty-seeking behavior, and then observe a two simulations in which this behavior is expressed strongly or weakly.
You also don’t need to answer this purely empirically. Given observations you can derive an inference to the best explanation. We observe so and so cats dying, we observe these properties, we have expectations about cats dying due to curiosity, etc, and the most parsimonious theory is one in which our commitments are minimized and our explanatory power is maximized.
Thinking of this critique, where does it not hold? What system with some sort of goal (ie: determine the best price for a good, determine the cause of death, etc) wouldn’t be subject to the criticism that it’s not inherently aligned with some sort of grounded ethics? I can’t think of anything at all that would satisfy this. Science and capitalism seem like particularly odd examples - religion, communism, exercise, pleasure, flourishing, etc, all seem subject to the criticism.
I agree with you that there are other ways. You seem like the kind of person who would pause and think about alternatives given the moral considerations involved in the killing cats example. The point is that science is like the paper clip problem except the objective is to find out the truth. People need to not simply want to discover the truth if they also want to reduce harm. It’s multi objective in that way.
Science and capitalism seem like particularly odd examples
Just the two I thought of, not a commentary on other systems.
You don’t have to think it, there were a bunch of really fucked psych studies in the 50s and 60s (Harlow’s monkey experiments, Zimbardo’s prison, Sherif‘s robbers cave), and that’s part of why ethics and review boards were created.
A common thread in these ethically bankrupt studies is as here they also tend to have rather poor control so they’re commonly of limited value beyond bias confirmation.
They say you shouldn’t argue with someone who buys ink by the barrel.
Both the incident and the backlash would seem to prove this.
Our study was approved by the Institutional Review Board (IRB) at the University of Zürich (Approval number: 24.04.10).
Interesting. It was approved by the IRB.
I wonder if “approved” here just means considered exempt? A lot of European countries consider most non-medical HCI research to be exempt from full review, which is different than the broader U.S. scope of ethics review for “human subjects research”. I have not worked in Switzerland, but in Denmark there was a pro forma process where the committee looked to see if your research fell into some specific categories that legally require an ethics review (medical research, captive population like prisoners or K-12 students, protected data, etc.), and certified it as exempt if it didn’t fall into those categories.
