Agent Psychosis: Are We Going Insane?
110 points by mitsuhiko
110 points by mitsuhiko
I usually try to avoid submitting my own stories, but in this case I wanted to make an exception. My motivation here is quite frankly, that I'm struggling with some of the stuff that is happening in the AI space and I'm with some projects completely confused about what is going on. And my suspicion is, I'm not alone in this.
TBH, the scene has always had these weird vibes to it: the quasi-religious ecstasy of giving into the vibes, cheering the demise of craft at the hand of slop, parasocial relationships with AI, and the overall uncritical nature of all the boosters.
Taken together it comes across as deeply unserious, where AI is almost more an identity marker for affix to show that “I’m with it,” and something to revel in for novelty alone.
There are definitely thoughtful people in its midst. But, man, they really can be drowned out by the cult-y mob.
I think it is actually more symptomatic of we don’t really talk about software construction very seriously anymore. Everything bottoms out at “it depends” and talking past each other. This leads people to think that there is no depth to software dev.
I think it is actually more symptomatic of we don’t really talk about software construction very seriously anymore. Everything bottoms out at “it depends” and talking past each other. This leads people to think that there is no depth to software dev.
This feels right to me, but there’s also something to be said about the way many boosters describe their usage through the lens of addiction. That can create a huge incentive for the “uncritical nature” you’re describing. It might be easier for superuser devs to justify their patterns of behavior, since Claude is spitting out JS instead of “that’s right — you’ve got your finger on the pulse, Steven!” but it feels like there’s maybe less psycho-chemical distance between those two outputs than you’d expect.
It’s probably not helped that many professional managers are disdainful of software engineering and in my cynical yet repeated experience, succeed due to cargo-cult-like success markers rather than skilful application of knowledge and experience.
It feels like maybe we are seeing a repeat of capital-A agile, but in high speed, the similarity being that many devs are forced to work a certain specific way (which may not work well), the difference being that the latest shiny thing has a psychological impact that a merely stupid process could never have.
I am yet to see a scrum process that did what it said on the tin, and I’m yet to be convinced that AI is anything more than a really good search engine with autocomplete. But it’s an autocomplete that tells you how awesome you are while making stupid mistakes. It’s so easy to hit its limits and watching it work is soul destroying.
It’s all a huge mind fuck.
I’m yet to be convinced that AI is anything more than a really good search engine with autocomplete.
It is also an impressive achievement in machine translation. Which sometimes can be applied in an unexpected way.
I am pleased to read this. Especially from the same author who was seen as a herald of LLMs becoming legitimate at the end of last year. Thank you for sharing your ongoing journey and being willing to share your shifting opinions.
It takes you a minute of prompting and waiting a few minutes for code to come out of it. But actually honestly reviewing a pull request takes many times longer than that. The asymmetry is completely brutal. Shooting up bad code is rude because you completely disregard the time of the maintainer.
This is why I find these tools so objectionable. The quality is generally very low, producing code which will be quite difficult to maintain. My experience so far is that it takes far longer to review and correct for slop than it would to produce the changes myself.
Aside: I don't think the analogy to Pullman's daemons is accurate. In Pullman's work the daemons are a manifestation of intrinsic parts of ourselves (our soul). I don't think the chat interface with an LLM, an external entity, is the same as talking with your Daemon, essentially externalising ones inner dialogue.
Sorry to nitpick.
Aside: I don't think the analogy to Pullman's daemons is accurate. In Pullman's work the daemons are a manifestation of intrinsic parts of ourselves (our soul). I don't think the chat interface with an LLM, an external entity, is the same as talking with your Daemon, essentially externalising ones inner dialogue.
Author here: the reason I liked the analogue is because despite everybody using the same models, the output of them greatly changes based on prompting patterns and how we talk to it. If you see some people's prompting patterns you can see that they are interacting with these models in ways that they would not do with other humans. People develop some weird para social relationship with it, that is much closer to an inner monologue than talk to another human. In HDM the interaction with the daemons externalizes what normally is internal. If you see the chat logs that some people (not everybody) has with these machines it's like a weird form of talking to oneself.
Thanks for the clarification. I think what you're saying is: the way some people interact with LLM chat interfaces is how one might interact with a Daemon in Pullman's setting--in a very personal, intimate, way; rather than how I misread it: as claiming that the interactions and relationships are as one might have with a Daemon in Pullman's setting.
difficult to maintain
… on the other hand, if that forces people to finally think about churn reduction, avoiding maintenance treadmills… Maybe even we'll finally learn, after all those decades, how to structure systems to reduce damage from lava layers structure…
I think it more likely that move fast and break things mentality would lead teams to just regenerate a project wholesale from prompts whenever the cracks in the generated codebase become too significant to paper over.
I wonder whether this would be getting much further. I guess some dead-ends can be skipped, and maybe some small things can be merged — but there are apparently limits for how far you can go on a single modification workflow, and if each iteration creates roughly the same amount of issues on rerun…
A related concept I find useful here that of extractive contributions, defined by Nadia Eghbal:
Extractive contributions are those where the marginal cost of reviewing and merging that contribution is greater than the marginal benefit to the project’s producers. In the case of a code contribution, it might be a pull request that’s too complex or unwieldy to review, given the potential upside.
Yegge just wants to sell books. Going a little off the rails for dramatic effect is part of the (his) hustle.
It's more than a little, though. That Gas Town post is full of so much personal jargon that I couldn't play along hard enough to finish it.
I was pretty disappointed because I've found some of Gene Kim's previous books to be excellent. But this book felt pretty content free, there were only a handful of examples – it was basically a blog post stretched to 300 pages.
The maintainer burden issue reminds me of this piece from oxide about using LLMs for writing that was shared a while back here
LLM-generated writing undermines the authenticity of not just one’s writing but of the thinking behind it as well. If the prose is automatically generated, might the ideas be too? The reader can’t be sure — and increasingly, the hallmarks of LLM generation cause readers to turn off (or worse).
Finally, LLM-generated prose undermines a social contract of sorts: absent LLMs, it is presumed that of the reader and the writer, it is the writer that has undertaken the greater intellectual exertion. (That is, it is more work to write than to read!) For the reader, this is important: should they struggle with an idea, they can reasonably assume that the writer themselves understands it — and it is the least a reader can do to labor to make sense of it.
If, however, prose is LLM-generated, this social contract becomes ripped up: a reader cannot assume that the writer understands their ideas because they might not so much have read the product of the LLM that they tasked to write it. If one is lucky, these are LLM hallucinations: obviously wrong and quickly discarded. If one is unlucky, however, it will be a kind of LLM-induced cognitive dissonance: a puzzle in which pieces don’t fit because there is in fact no puzzle at all. This can leave a reader frustrated: why should they spend more time reading prose than the writer spent writing it?
I think this is directly applicable to code, not just prose. Sadly even seasoned programmers seem inclined to let go of previously accepted conventions in this regard, e.g. that code is a liability, not an end goal but a byproduct of a knowledge building process, and act as if it was desirable to produce as much as possible, as fast as possible.
The PR push of AI advocates does often remind me of the Anabaptist takeover of Münster. The development of Lutheran theological doctrine enabled a group of people to reconsider the entire social order. Amplified by a few charismatic leaders, they ballooned their numbers and politically seized control of the city. Their desires for societal change weren't unreasonable but their expectation that the rest of the world conform to their views, simply because of their localized power, seems almost like a Monty Python parody.
I enjoyed the Hardcore History episode on this, Prophets of Doom (costs $3 now, well worth it).
Adding links for those who haven't read Pullman's books:
In His Dark Materials, every human has a dæmon, a companion that is an externally visible manifestation of their soul. It lives alongside as an animal, but it talks, thinks and acts independently. I’m starting to relate our relationship with agents that have memory to those little creatures. We become dependent on them, and separation from them is painful and takes away from our new-found identity. We’re relying on these little companions to validate us and to collaborate with.
Without spoiling too much, it turns out that humans only locally have dæmons, and whether your dæmon is visible is an indicator of where you are currently located. Dæmons are not independent of their humans and Pullman often uses them as a heart-on-sleeve indicator of the internal emotions of a character.
Being charitable, the similarity between a chatbot and a Pullman dæmon is that they both serve as daimones: they are mirrors which allow humans to interpret themselves via idealized representations. If we stare at a mirror too long then we forget where the mirror ends and we begin, like Narcissus.
A chatbot is not part of a human brain, but it can replace part of a human mind. Previously, on Lobsters, I found that Hofstadter's caged-bird metaphor, from "I" is a Strange Loop, is the most apt way to put it. Hofstadter says that it is incorrect to put minds and brains in bijection; rather, it is possible for a mind to spill out of a brain and onto surrounding computational substrates. A mind is what a brain does, and when a brain consistently engages some physical tool to do that minding instead, the mind becomes whatever that tool does.
Aside: Your description sounds like addiction, not psychosis; psychotic episodes have the remarkable irony of being imperceptible to the sufferer. The various elements described, like intermittent rewards, unexpected delays, and the inability to stop despite understanding the long-term outcomes, are characteristic of Skinner boxes and remind me of experience reports from heavy slot-machine usage.
Speaking as a plural system who had to escape our own intero-cognitive loops in order to function as a person again (and who's read His Dark Materials), this resonates. Same reason we quit alcohol, a codependent relationship, more recently caffeine and, claws crossed, soon also weed. Everything we've seen about LLMs makes it look less like the next tech revolution and more like the next tobacco industry.
This meatbag sees y'all. Best of luck! Congratulations on managing caffeine and alcohol, which I think are respectively the hardest overall and hardest physiologially to quit.
Aside: Your description sounds like addiction, not psychosis; psychotic episodes have the remarkable irony of being imperceptible to the sufferer.
I think the description is of a reflected and somewhat controlled addiction, but also of looking at some other cases which look psychotic (vibecoded browser that has no point in history where the codebase compiles, etc.), and asking «am I sometimes like that and not noticing it?»
I think some things like completely perceptually irresistible hallucinations while having enough of reason left to say the hallucinations cannot be actually true — those things could be sometimes classified as psychotic while being caught in the moment from the inside. But anyway the text is about the author in a clearly lucid moment asking how to know if one has completely psychotic moments, consistent with that aside.
This feels like too broad a brush. Are you conflating the loudest people with the consensus opinion? Right from the beginning it says something about staying up all night to vibe code. Not a single person on my team codes outside of work. This was true of my last team as well. So off the bat this is hard to identify with. This feels like maybe a consensus of the terminally-online influencer types or something.
My team does use AI but it wasn’t hard to have the discussion about what kind of PRs are reasonable. A 100 file vibe-coded thing is just going to get rejected out of principle. The “AI slop is coming and there’s no way to stop it” narrative frustrates me for this reason. Lean into team norms. Have discussions about working relationships. It’s teamwork 101 stuff so I don’t understand the “sky is falling” talk about the inevitability of garbage vibe code.
My team has people who have shifted to ~50% LLM use and people who refuse to use LLMs out of principle. We've spelled out a pretty standard policy for LLM use: you're still responsible for completely understanding your PRs, disclose if you used agents heavily, etc. But this just codified the default everyone was already doing, it wasn't a significant change.
Most of my friends in various companies are in a similar situation: I don't know anyone who thinks "don't look at the code" is a reasonable position for any PR intended for production. Yet looking on the internet, it would be easy to believe that's the default.
The vast majority of people doing actual work don't have blogs, don't give keynote talks, don't bother with stirring up trouble on X…they just get work done that only their colleagues see. So what's visible on the internet is a very warped version of what's happening.
This applies to every field of human endeavor after the introduction of widespread social media, and computer people invented social media, so we had a head start. It's no surprise that, to quote Alan Kay, "computing has turned into pop culture".
Thanks for writing this, it's very self-aware. A few thoughts:
Particularly if you want to opt out of this craziness right now, it’s getting quite hard.
I'm not sure that it is! A key marker of addiction is the normalisation of habits and a belief that life isn't possible without submitting to them. But, I promise, it really is as easy as just turning it off.
Some projects no longer accept human contributions until they have vetted the people completely.
I suspect this says more about the circles you frequent than it does about wider trends. I have not seen such patterns playing out: in the projects I work on and contribute to, what is human is still king.
This is especially true in the workplace. If one were to make a judgement based purely on the sort of posts on this site and others, it seems easy to assume that every software house is currently undergoing a revolution, but I've simply not seen that. I've colleagues that have experimented with AI programming, only to walk away a few days later with a distinctly sour taste.
I'm sure that for someone immersed in these tools it feels like the battle has been won and the argument is over. But, week by week, as I watch people I know discover them, briefly get excited by them, get burned by them, and then turn away from them, I am becoming more confident in my belief that this is not the all-encompassing shift that its proponents believe it to be.
Maybe the answer is that we need better tools
And so, the cycle continues.
But, I promise, it really is as easy as just turning it off.
Please tell that to my CEO. Thanks.
You should try to apply pressure if you can. Find like-minded co-workers, sympathetic members of management, etc. In a job-limited market like today, the labour market alone can't be relied upon to send signals up the chain of command.
I tried going up the chain of command at my previous job (not about AI but about the insane Agile being pushed on us). The VP I was talking to suddenly left the company [1]. I left shortly thereafter.
[1] I'm still not sure if it was due to him asking the questions I was raising, or if it was unrelated, but the timing was suspicious.
Great article and I love the framing. I definitely feel a little insane myself at times!
One thing you call out is the imbalance between the time to produce a PR and to review it. I've found that Cursor Bugbot is actually excellent at finding bugs in PRs, AI-authored or not. Creating a little "PR review" Claude agent to look for bugs, code repetition, dropped comments, and other common mistakes also works OK, although whatever magic Bugbot is doing seems to work a lot better IME.
If I were to steel-man the argument of the extreme pro-AI crowd, it's that Ralph and Gas Town are aiming for "where the puck is going" of a purely autonomous self-building software project. So if you're asking "how do I review this" then you're asking the wrong question (in their view, as I understand it). If you listen to the DevInterrupted interview of Ralph creator Geoffrey Huntley, it indeed sounds a little unhinged but this seems to be what they're aiming for.
So if you're asking "how do I review this" then you're asking the wrong question (in their view, as I understand it).
This is important to recognize. I have had multiple people look at me sideways when I remark on the quality of the code I received from an agent. "Just tell it the error and make it do it again" or even "Well, if you have tests, make it iterate until they pass." They don't understand why I expected better code from the tools that are touted so highly and cost what they do. The expectation is absolutely that we'll get beyond needing to review and just trust the output.
So eventually, the prompt becomes the source code.
For the people who like their compilers to be non-deterministic and potentially to act like a historical figure that had a tendency towards genocide if they read too many references to Wagner in the prompt conversation, yes.
More seriously, I had a long conversation with someone about why I feel the integration of AI is such an issue for cybersecurity: the actions are not atomic. You can hook all manner of telemetry into Splunk and set sane alert levels. On my home network I have specific countries I do not expect my network to talk to that I alert on if a packet goes there. That's easy.
But once CONTEXT matters, those alerts go out the window. Maybe the individual prompts that warm up or fire a multi-prompt inject look innocuous and fly beneath the radar.
So in this context, it would be like the entire history of the conversation is the code, plus knowledge of the specific model in use, and the settings of it. That will take a new form of version control to figure out how to maintain. Git int his case is just like trying to use Splunk to watch for abusive prompts, it is too atomic to work.
I ended up building and building and creating a ton of tools I did not end up using much. “You can just do things” was what was on my mind all the time but it took quite a bit longer to realize that just because you can, you might not want to.
Yes, I struggle with what to even build, if building's so easy. I don't need more new stuff; I want less and better stuff. Craft and taste are what makes something worth keeping, using, and building on.
I too was blown away (and fully expected at the same time) when Cursor’s AI written Web Browser landed. It’s super impressive that agents were able to bootstrap a browser in a week!
Amazing that even while writing a post about how it's bad that people uncritically accept langlemangle garbage, the author posted about this project as if it were a real thing instead of a joke that doesn't even come close to compiling: https://embedding-shapes.github.io/cursor-implied-success-without-evidence/ There's no actual browser here! Just a bunch of browser-shaped incoherent lumps.
It’s an impressive research and tech demo, not an approach to building software people should use. At least not yet.
It's an impressive PR campaign, in that they were able to successfully get people to take their claims seriously instead of getting laughed out of town like they should have been. Calling this "research" is an insult to actual researchers.
in projects that don’t submit themselves to the slop loop, it’s going to be a nightmare to deal with all the AI-generated noise.
Well, it's a lot easier if you stop using Microsoft Github!
You quoted Armin out of context there. His very next sentence says:
But holy crap! I hope nobody ever uses that thing or would try to build an actual browser out of it, at least with this generation of agents, it’s still pure slop with little oversight.
The post about it not compiling is out-of-date now btw, I just compiled and ran it using the instructions in the new README and it's very janky but worked better than I expected it to - here's some screenshots. It's clearly not purely wrapping an existing rendering library because if it WAS there wouldn't be that many obvious rendering glitches!
No, the additional context does not help here. He was tricked by a PR campaign and repeats the absurd claim that "agents were able to bootstrap a browser in a week". His hypothetical "actual browser" that he admits doesn't exist describes "something beyond a proof-of-concept" but they did not even achieve a proof-of-concept.
He was tricked by a PR campaign and repeats the absurd claim that "agents were able to bootstrap a browser in a week".
I wasn't tricked by anything. I looked at what was posted, when it was posted, and evaluated it for what it is. I came to my own conclusion: it's a genuinely impressive research project, but it is not software that anyone should be using. Quite frankly, you attributing motives or conclusions to me that I did not express comes across as dismissive and inappropriate for what I think this community stands for.
I think the difference between your interpretations is whether «not something anyone should be using» implies that using the thing is possible at all.
Well, using is relative. It can load pages as demonstrated and I was able to compile and run it. That to me, is enough to at least say it's an impressive experiment.
Oh how interesting, so the people who failed to compile any commit of it ( https://embedding-shapes.github.io/cursor-implied-success-without-evidence/ discussed in https://lobste.rs/s/urlzyd/scaling_long_running_autonomous_coding#comments_story_01kf4m094ffwssx4433787tbqr ) were actually holding it wrong?
Thanks for the information!
(also a reply to @simonw )
No, they were trying to compile it a day ago before the author, no doubt spurred on by all the criticism they were receiving, got the build to work and added a README.
(That said, I do feel like a lot of people read that post claiming it didn't work and blind-faith believed it, which is a little ironic considering a big complaint here is that people blind-faith believed the original announcements about the project without digging in.)
@k749gtnc9l3w is being very bad-natured and overly sarcastic right now, but honestly Simon this still is a bit of a bad-faith argument. They breathlessly posted this thing claiming that AI built it from scratch all on its own and when it was investigated it showed that this AI did nothing but generate a giant uncompilable group of code that never once seemed to work. After they are called out for this, a human dev steps in and works heroically and gets it compiling, gets together a working readme, and...now we're supposed to take the original article at face value? True it's something impressive to say that it took a human but a day to get it working from its last commit, but a) why couldn't the AI have done that? Why did a human have to step in, if the entire point of the project was that it was done 100% from scratch by AI? b) how did they get screenshots of it working, if no single commit compiled? Did they have a separate fork or something that worked and they ran that, or did they fake it? It seems in bad faith.
The entire debacle was not just the fact that it didn't compile, it was the entire presentation of the whole project, and "fixing it after all their PR claims were proven false" doesn't, and shouldn't, give them a free pass on this. Saying that all of the detractors were "trying to compile it a day before it was ready" is very bad faith. If it truly was just a day away from compilation why didn't they publish it a day later? This claim seems like you intend for it to invalidate peoples' claims against the project, and while it does show it wasn't totally broken (which wasn't the claim for most; merely that it didn't compile -- also many of the detractors DID get it to compile eventually too!), which is impressive still I reiterate, it's still just an extremely bad smell and makes them look really disingenuous.
As far as I can tell the project DID compile just fine while the agent was working on it, but it didn't pass CI.
I don't see how they could have made this much progress without a compiling build - coding agents need to be able to run and test the code they are writing, otherwise they can't decide what to work on next.
I think what happened here is they failed to anticipate how much of a difference having a working demo with clear build instructions would make for how this project was perceived. That was a foolish error!
I think another subbranch here has a likely hypothesis: the environment where the agents run, the environment where the CI runs, and the environments where a person not involved in the project would try to build after cloning — these are, potentially, N+2 different environments where N is the number of persons trying to build. If the agent did setup something in its environment that made things compilable inside the sandbox, this means that a fix exists and probably can be extracted, but using the repository as provided did not initially include the fix.
I am not even sarcastic, I am sincerely losing track of what is even going on, and when I thank for the additional details it is sincere!
Ah. This uncompilable thing being fixable in a day is also somewhat impressive. Needing to be reminded that the code should probably be compilable is also impressive, but in a different direction…
Thanks for the details!
(I surely didn't want to wait for the compilation of this huge pile of code locally)
This uncompilable thing being fixable in a day is also somewhat impressive
That's not too surprising because the agent was at all points able to compile it, just not CI.
Nix or Bazel (not sure if Guix would want to touch the horrors) should steal it as a case study why reproducible build environments are useful…
Is this much different from people who anthropomorphize their computers, grow attached to their desktop themes and date other people based on tabs vs spaces?
Some good suggestions here:
better ways to share context, better ways to make the AI’s involvement visible and reviewable
Agreed, and I'd add the thinking traces too.
One way or another, the covenant between OSS contributors needs to be revisited to restore symmetric stakes.
Prior to the Internet, and especially prior to the rise of Github, forks were the norm for source-distributed software. You'd get some tapes, hack the source to work for you, and maintain the patches yourself when the next version of upstream was released. There used to be a plurality of OSes, CPU architectures, and compilers. The BSDs are an interesting vestige of this lineage: all containing similar but slightly divergent base systems. Linux and GNU's unhappy marriage made for a relatively uniquely unified system, perfect for centralized project management like Sourceforge and then Github.
I'm curious if agent psychosis will drive us back to the pre-Github days: innumerable forks existing because tweaking projects to your own purposes is just a prompt away. The asymmetry of PR generation vs reviewing that Armin mentions means maybe we just won't bother submitting the PRs anymore? Just run your fork? Let others run theirs? "Contribute" only when something triggers it (eg hearing others adding similar functionality as your fork).
This possibility scares me as it seems like "consumerism for software." Everyone consuming projects, consuming their own agent's outputs to augment it, discarding it all when it crumbles under its own weight, and repeating the process with the newest shiny project.
But I also wonder if new forms of collaboration are needed. Forms of collaboration focused more on human communication than simple code sharing. I see Ghostty's "create discussions before issues" as an interesting first step in this direction: adding a another layer of indirection between users and code: https://github.com/ghostty-org/ghostty/issues/3558 Perhaps we need a pre-PR code sharing/discussion mechanism meant for reducing the toil of dealing with slop.
Or maybe nothing will change except cultural norms around opening and closing PRs. I'm still unsure if these tools will actually revolutionize the industry or just be a better screwdriver.
I'm curious if agent psychosis will drive us back to the pre-Github days: innumerable forks existing because tweaking projects to your own purposes is just a prompt away.
I recently found someone that had 'cloned' an entire open-source library of mine with an LLM to fix a bug they found.
The worst part? They didn't even report the bug on the original project! I had no idea it existed! Then they had the gall to spend time ranting about the bug in their clone's README.
It does concern me, frankly.
This sounds right. I'm a big user of LLM tools. Something that I think is missing is a lack of responsibility. I don't mean that in the diluted way that we generally talk about responsibility, but actual accountability. The situation where your boss or someone else comes to you and says, "Hey, you're the expert and I trust your judgement. But this is really important. I'm putting my money and reputation on the line here. I will do whatever you suggest, but if this doesn't work for a reason you should have foreseen, this is going to be a problem for your employment."
If you start framing some of the LLM-generated code as the above, I think perspectives will shift appropriately. A civil engineer putting their stamp on something has real meaning and clicking the approve button does to. We aren't all just filling out sudokus to put in a drawer here, we're building things. Even in open source, we gain and lose reputation based on these decisions.
This by no means should be read as a suggestion that LLMs shouldn't be used. But that they're a tool. You're still the one hitting approve.
This was a really good read, thank you.
Two things are both true to me right now: AI agents are amazing and a huge productivity boost. They are also massive slop machines if you turn off your brain and let go completely.
I think it's possible to have that boost and still retain the sanity provided we use the agent as reference assistant at best (a context upgrade to StackOverflow/Wikipedia if you will). IMO the disaster begins when it starts becoming a black box, you no longer understand what it's churning out and you start treating it like a tutor or professor. That's when the seed for massive technical debt is sown and its result will be seen few years from now.
There is an old proverb along the lines of "mind is a good servant but a very bad master", the same perhaps should apply to your interactions with your LLM. Your objective should be always clear on what you want from the digital machine, that should help keep the daemon syndrome from ever happening.
One thing Gas Town made concrete for me is that "how much/how long can we leave these things running for and get a product the agent thinks is good?" is not what we should solve for. The goal is a product that works for users (works, does what they want, maintainable) while using human effort well.
If we get tools that are better at doing some things independently, like checking or debugging their output, cool. But sometimes you need more early human involvement where it saves later work resolving weird bugs, reworking to deal with not meeting the real-world requirements, or fixing a hard-to-maintain codebase.
I think having an agreeable sidekick is not an issue in the large. The problem of course is that it also happens to be at a time where we have social media with algorithms optimized for keeping you engaged, usually through agreeable niches and rage baits.
At the same time, we're basically easily distracted monkeys with laptops and computers. I don't see how this ends well since we have zero hygiene/healthcare guidance on how to use those systems.
I'll remain hopeful though, we can use AI as a tool but it's a sharp one indeed.
Shouldn't the Issue be the prompt? That way everyone and the reviewer can see it and refine it. Also comments can mapped to follow-up prompts. Also, wouldn’t it be good to check in the various configs and AGENTS.md for the recommended agents, just like you would lint rules and CI/CD configs? Anyone doing this? A soft reproducibility but better than nothing, right?
Good point not talked about enough: agent generated code tends to be loc heavy. We should work more at follow up refactoring of the code. And we have become dependent on these agentic “creatures”, which is why at the very least we should only use open sourced agents that can work locally (Codex cli or preferably OpenCode). And why checking in the agent configs seems like a good idea to me.
As I understand it, Gastown and Beads are an overt celebration of agent psychosis as a kind of mix between futuristic experiment and art project. The models currently have very little understanding of quality control, code design, brevity, unambiguity, clear state ownership, decomposition etc. This leads to them blowing up their context, then suffering because they can no longer hold the whole model in their heads. Gastown is what happens when you say "well, and so what if we just ignore all that and just see what happens anyway". I would not read too much into it.
I think it's possible to write code that's too optimized for humans, especially when working with AIs. AIs have a tendency to just repeat their setup/teardown by copypasting when humans would have long since abstracted it out. However, this may genuinely be an adaptation to reduce conceptual dispersal. A human can learn a codebase; a LLM can only read it; once you unload a file from the context all the information about it is erased from the agent's psyche. As such, I suspect that RL drives them to place behavioral code as locally as possible. This is the primary argument of vibecode: AI code must be written primarily for the AIs to read, which is not the same requirement as for a human to read.
I can definitely confirm the addiction is real though. :)
