Better Models: Worse Tools
49 points by mitsuhiko
49 points by mitsuhiko
A rare case where I want to put a post of mine here. Reason being that I was hunting down tool calling behavior regressions with the latest generation of Anthropic models and I found the resulting behavior both puzzling and quite problematic. Those models appear to be strongly RL'ed on their own Claude Code harness which is closed source, and when you come close in tool declarations but slightly off, you can now expect to get broken tool call behavior when older models did not yet have that defect.
Thought this might be interesting for folks here.
BTW I submitted this as ai and not vibecoding. I really struggle to understand where the boundary of vibecoding is on this website. This has much more to do with the underlying LLMs, RL and building harnesses around it. If that is now also vibecoding, then what remaining usage of the ai tag remains?
Because vibecoding around here is a giant scarlet A on anything that even vaguely looks like it might touch or have touched generative AI. Blog posts that sound funky, READMEs on major projects that ban LLM contributions, articles that have a few too many instances of "Not X, but Y", these can and have all been tagged vibecoding in just the last few of weeks. Speaking purely for myself, I've completely given up on paying any attention to the tag: the community is so reflexive about putting it on things that it's ceased having any actual meaning to me.
That all said: in this particular case, for what it's worth, I actually agree that this submission should be tagged vibecoding (it's literally about the exact models used for that!), but I also agree that ai shouldn't have been removed, for the exact reason you're highlighting. I'd suggest adding it back, but I don't have that option on this particular story for whatever reason.
I don't wanna litigate it, but the only reason I really submitted it here is to have a discussion about the AI and RL aspect of it, and the path we're on with these models. The moment it gets tagged as vibecoding it just ends up being thrown into the same bucket as every other discussion about using claude code etc. So I find this re-filing as vibecoding very frustrating.
edit: i’ll include a warning that i have basically set about litigating this issue, despite your desire to avoid that. feel free to ignore and move on~
why exactly do you find it frustrating? i ask not to be obtuse or difficult, but because i am genuinely confused by the extreme resistance some people show to this tag being used so widely. i get that some people balk at having their own hard work demeaned as effortless vibecoding, but i am not convinced that is the takeaway one should have when the tag shows up on here.
if anything, being tagged as “vibecoding” probably makes the post more likely to be filtered out by people with a deep disdain for these technologies, which would serve to improve the quality of the comments.
what cost does this tag being applied to your posts have for you?
more on topic to your questioning: the tag is probably because this post primarily discusses LLMs in the context of coding agent harnesses, rather than the fundamental techniques involved in designing and implementing LLMs[1]. my mental model is: “AI” is the broad realm of all machine learning techniques one might classify as AI, and in particular the theories of those techniques. vibecoding is the relatively narrow realm of LLMs as applied to practical problems like coding, writing, design.
not going to try and claim that the community’s application of tags are particularly consistent in that sense, or that this tagging regime is the best one. personally i would like to see some higher standards required to hijack a post’s tags, at least for tags that have garnered a controversial reputation like vibecoding.
[1]: although i will certainly grant that Reinforcement Learning complicates things here, given that it is (as far as i know) a fundamental technique applied to any kind of useful LLM, not strictly for agents or coding tasks.
i ask not to be obtuse or difficult, but because i am genuinely confused by the extreme resistance some people show to this tag being used so widely.
For a start the term is intentionally used in a derogatory on this platform and entirely non-neutral. I have largely given up engaging in conversations on the actual use of agents on this platform as a result of the general distrust in that technology here.
what cost does this tag being applied to your posts have for you?
It's quite rare that I submit a post here, and in this particular case I genuinely was curious on the thoughts that people have here. Once a vibecoding tag is slapped on, you're basically guaranteed that people will filter it out, even those who I thought would find that topic interesting and might not be familiar with the rather extreme effects reinforcement learning has on these models now.
If I would have found that tag appropriate for the article, I would have not submitted it because I already understood that this is not the community for that kind of content.
In general I find the approach of lobsters with regards to AI content incredibly frustrating. Even when I think I know where the boundaries are, I keep being negatively surprised.
thank you for sharing your thoughts, both the article and your (meta)commentary.
off topic, but fwiw, i will say that despite being relatively llm-skeptic myself, i do value your continued engagement and commentary.
In general I find the approach of lobsters with regards to AI content incredibly frustrating. Even when I think I know where the boundaries are, I keep being negatively surprised.
It's worth noting that it currently takes two users to re-tag a story via the suggestion feature. While I have a set of changes up for review to let us adjust that on a per-tag basis, it's worth being aware that currently it takes less effort to re-tag something than it does to flag it.
I'll answer this for myself.
Vibe coding is a term of art which means "writing code through prompting it without reviewing it or even needing to know anything about programming".
It's a really useful term. If I say "mind you, I just vibe-coded that" it means that I have an illustrative prototype but you should not depend on it yourself. It also gives us a way to distinguish non-engineers vibe coding neat demos from professional software engineers who are using LLMs and coding agents as part of their process for creating reliable, verified code.
Lobste.rs ignores that entirely. On this website "vibe coding" means "absolutely anything at all to do with LLMs, even using them for stuff that has nothing to do with creating software code - oh, and it's a bit derogatory too, because Real Programmers of course would never use tools like that".
I think that's bad. I think Lobste.rs should add a tag "llms" and use that instead.
I do personally disapprove of elitism and do not endorse it. I can't and won't try to stop other people from feeling elitist things, but I don't think that should be the point of pretty much anything.
Off-topic but I had similar experiences and I think we need to accept the community rules here.
Using LLMs? That’s tagged vibecoding. Period.
Using LLMs? That’s tagged vibecoding.
But that that super narrow definition then basically next to none of the currently still ai tagged posts would quality: https://lobste.rs/t/ai
My understanding of the "using AI/LLMs" was to separate it for developing / researching on AI/LLMs.
Do my posts on AI compilers (tagged ai and compiler), TIRx and Event Tensor, qualify? (I think so.) I think "using LLM -> vibecoding" is a useful definition that leaves plenty of interesting posts for ai tag, and it is also a de facto Lobsters convention.
I'm not the arbiter here but your submitted paper mentions LLMs all over the place, and well, ran inference on them. So that submission at least surely is also in violation of that tag then?
My mental model of the vibecoding tag on lobsters is as follows. Some people find LLMs depressing, and they want to live in a protected space where they don't hear about it and can mostly pretend that they don't exist -- as if we were five years ago. On Lobsters, this became implemented by filtering out entries with the vibecoding tag. So if your post breaks the illusion that LLMs don't exist, then it will be tagged vibecoding.
(I'm not saying that this scope for the tag is good or bad, or judging the practice of filtering LLMs out. This mental model provides rather accurate predictions of which post will or will not be tagged vibecoding, and I believe that it can help people accept why it is so -- rather than arguing about it repeatedly. This has become a sort of social convention / cultural practice, I think it's reasonable to accept that it exists, is practiced by a critical mass of people, and respect it.)
I can't speak for what all y'all think about this, but personally my position is that these destructive things are not the future, and are instead a waste of time. While I generally am sympathetic to safety as a concept, I think a lot of people here on lobste.rs would be quite surprised to find themselves accused of wanting safety. Many people are against generative ML for many reasons, and many of them are people more used to being on the giving end than the receiving end of dismissive "safe space" rhetoric.
Anyway, it sounds like the rubric you're using still produces correct decisions that are compatible with being part of this community, and that's all we can really ask.
In an optimistic mood, I draw the line (for generative neural networks, symbolic AI is allowed to live, and classifier neural networks sometimes get some slack) as follows:
ai contains a meaningful discussion how the weights were changed, preferably by the authorvibecoding contains a discussion of use of a model without looking inside of its weightsai tag, vibecoding blocks itThis is enough to explain what happened here.
But yeah, there are some examples of escapist tagging not explained by my content-based interpretation.
Anything close to LLMs is tainted by association. I think a lot of people see it as a mix of intellectual property theft and a form of debasement of the craft and the user as well. Lots of others see it as a tool that's here to stay and want to play around, understand and optimize. I think it's the largest schism I've seen in this forum so far and the use of the tag reflects it.
I've always viewed vibecoding as a subset of ai which is roughly related to the Attention Is All You Need paper either directly, by evolutionary lineage or by implementation. Given the amount of funding around the LLM space and it's resulting zeitgeist, vibecoding will probably end up dwarfing ai until there's some external event which prompts a change in funding or line(s) of research.
vibecoding is also being used as a media tag to indicate when content has been submitted to the site which was produced using some kind of LLM/generative ai based tool.
In case you are curious about Fable: I intentionally did not test it because I was not sure if the classifiers they are running might downgrade me to Opus silently.
My understanding is that Fable is always explicit when it downgrades now. I've certainly tripped the classifier ("rank these bugs by severity" presumably sounded too much like security research).
I will do some tests. I know they said they are no longer going to do it behind the scenes but I haven’t yet looked into the mechanism it uses for downgrades.
If this hypothesis is right, the impact probably goes beyond coding agents. Any application that depends on reliable tool calls could see similar degradation once models are post-trained around one provider’s preferred harness.
I am not surprised; vendor lock-in is likely part of the plan!
Probably with a dose of cost-cutting: everyone using Claude Code improves prompt caching.
«If you pay for Claude subscription and already transfer us all your code, you must also use our free-to-use UI, which is not necessarily good enough to be a lock-in on its own separately from the model» is not how pure vendor lock-in usually goes…
It makes sense that a model will perform optimally on the harness it was trained for, ie the one developed by the model developer. After a ton of RL on Claude Code-like traces, I would've expected it to perform a fair bit worse on a harness that doesn't behave like Claude Code. Personally, I'm more surprised how well they seem to do on third-party harnesses like Pi.
That the bug only happens deeper inside an agent trace makes sense. Earlier this year, on GPT 5.2 and 5.3 models I was wrestling with a bug where, later in a trace, the agent would output ls -ლა instead of ls -la. Briefly documented here: https://github.com/openai/codex/issues/7988. My experience was that it also only happened deeper inside a trace. It's almost like the models, over a long context, stop 'thinking' so clearly and fall back to raw 'instinct'. It's probably then where differences between the training distribution of the model and what the harness or user is forcing the model to do will affect performance most dramatically.
The question is not that they RL and now it works better on their harness, the question is that they RL so much it regresses on other harnesses compared to previous releases. This is where one should ask questions like «what other kinds of overfitting are starting to happen?»