AI didn’t kill Stack Overflow

For Stack Overflow, the new model, along with highly subjective ideas of “quality” opened the gates to a kind of Stanford Prison Experiment. Rather than encouraging a wide range of interactions and behaviors, moderators earned reputation by culling interactions they deemed irrelevant. Suddenly, Stack Overflow wasn’t a place to go and feel like you were part of a long-lived developer culture. Instead, it became an arena where you had to prove yourself over and over again.

Last year a Stack Overflow moderator decided to schedule a ~10 year old question of mine for deletion as ‘irrelevant’. The question was twice as old as his account and had already been answered. When I asked in the comments if it was a mistake because the question was so old, he seemed embarrassed and reacted by hiding the question and the comment thread.

Yeah, matches my experience. Often the first reaction to the rare time I would ask a question was an antagonistic “why would you want to do that” in the comments. The only thing comparable I’ve encountered IRL was visiting a friend at their private-sector lab for lunch and getting thrown into a pit of postdocs trying to figure out whether I was smart or not by conducting an impromptu oral examination (this is a weird way that academics seem to introduce themselves to each other, you are expected to be able to justify your entire life on the spot). I will say that this phenomenon isn’t unique to StackOverflow. I’ve also encountered it in zulip chats for some projects, and the Arch Linux forums. You have to pass some weird shit-test before people will resign themselves to helping you. And to their credit they do eventually help you, sure, often matching their effort to the amount of effort you demonstrate asking the question.

There is something to the points raised in the article, particularly about making people feel welcome, and how SO’s moderation works against this. Moderation of some sort is absolutely necessary, but many moderators (and high-rep non-moderators with enough privileges to close questions) ended up over-policing the site. For a while I was answering questions on a daily basis. Several times an interesting question would pop up and I’d start to answer, only to have the question closed as a duplicate or “off topic” before I could post my answer. Sometimes it was possible to get the question reopened, and sometimes it wasn’t. It was a real disincentive to answering questions. And it was probably a disincentive to asking them as well.

There were a couple other things about SO that I also think contributed to its decline. The essence of “gamification” is getting points for contributing value. But the gamification needs to be respectful. One of the things SO asks frequent contributors to do is to review edits to others’ contributions. I did this a little bit, on and off. I didn’t find it particularly rewarding, but I did think it was helpful to reject bad edits, e.g., attempts to “clarify” something that changed its meaning. Well one day I picked up a review task for an edit that seemed really weird. It kind of didn’t make sense, and I spent a fair bit more time on it than usual. But I decided something in the end. The result was “Surprise! That was an artificially generated ‘review audit’ designed to test your reviewing skills. You chose incorrectly, so your review privileges have been restricted.” Well FU Stack Overflow. You just wasted my time, so maybe I’ll just never review anything again.

Another thing that may have contributed to SO’s demise is some kind of moderator rebellion drama. Some moderators were banned for and other moderators protested in solidarity. I don’t know what the issues were though or how they were resolved or even if they were resolved. It had to have hurt though.

There have been several articles in the past months about the decline of Stackoverflow. And I have seen the graphs that point downwards, but personally my experience with SO is still mostly good:

• I have asked several questions, most of which have received several helpful answers.
• there was very little moderator interaction.
• if I search for a programming question with Google or Duckduckgo, often there is a good SO answer among the first results.

I had slightly negative experiences as well in the past few months:

• one of my questions from ~15 years ago was closed, despite having several answers. But by today’s rules it really is not an appropriate question for the site, so I can understand that it was closed.
• the number of popups has grown: with a fresh browser profile I now get three different popups (cookies, login with Google, chat-rooms). It feels a bit like the SO people are desperately trying and failing to reach first stage of Enshittification, and I feel a small amount of pity for them. More seriously, I would appreciate if there was a sustainable business model behind SO; I have no idea where they get money from.

So overall, somehow my own experience is still positive and does not match the recent reports. Why is that? Am I such an unusual user?

For the declining number of new questions I would posit the explanation that “most questions” about programming have been asked, and therefore people can use the site in a more read-only way rather than adding new questions. That’s my experience at least: while I have asked only a few questions in the past months, I have read hundreds or thousands of existing answers.

Is there some kind of inverse to the eternal september / CADT going on? Where sites like Wikipedia and SO get dragged down by reputation-gamers and nit-picking moderators?

OTOH, Wikipedia and SO really are amazing resources that you wouldn’t expect to flourish at all, if you didn’t have the fact of their existence to point to. I was helping my kid look up something the other day, and we found this thorough stackexchange answer to a question she had, seemingly written by someone with experience who just wanted to help, but quite obviously written by a human, and I found myself thinking how wonderful it is that she gets to see this tiny corner of real people helping real people before it’s all turned into slop.

I honestly never really got into Stack Overflow much personally; when I was looking at it early on it never really had much coverage for the languages I used; early on it seemed to be biased towards Windows and C# due to the background of its founders.

I do remember four or five years ago going out of my way to delete my account. This was well before the popularization of LLMs, but I recall that I was prompted to do this when the owners of the site decided to change the licensing terms and allow user data to be sold to third parties. This was back in the good old days when you could assume that someone training a model would only use sources they had a legal right to use.

Seems a bit strange that wasn’t mentioned as part of the timeline in the article.

The other weird thing about the article is that it doesn’t mention anything about the decisions of the site leadership. Did they not know that their policy changes had such an adverse effect on the experience of people using the site? Did they know and not care? Did the deletion-happy moderators convince them not to listen to the rest of the users? It does not feel like a mistake of that type should have been fatal to the site if it had leadership that was paying attention and capable of course-correcting or acting democratically.

Yeah, SO became search response spam years and years ago, far before whatever LLM spam it gets now.

Nut also I was answering questions back when it was originally started (this isn’t a brag, this is setting the scene) - the last time I actually logged in I had a bunch of comments on my answers with people berating for posting incorrect answers and similar, because the answer today is different than it was 15 years ago or whenever (e.g. answering questions about JS in the ES3.1 era, and things like that). Not actually updating the answer with “in modern the answer is actually different” or adding new answers, just shitting on an answer that was correct at the time.

\o/

This gets a lot wrong about the origin of the incentive structure of stack overflow - it was all set up by site developers, with the goal of creating a repository of knowledge for consumption by Google. High reputation users, and elected moderators, get the privilege of helping the company with that (not reputation points!)

The first half of this article was essentially paraphrasing an article previously posted here, but the second half goes more into why the site ever worked in the past and how they could still possibly save it.

a bizarre twist where a working democracy sprouted an aristocracy, and that aristocracy killed democracy

I agree that this is fundamentally how stack overflow failed, but it’s extremely common throughout history, so it’s less a bizarre twist and more an expected default outcome unless you set up guard rails to prevent it.

Stack overflow became useless at least 10 years ago, it’s got nothing at all to do with “AI”.