I'm (mostly) picking models on speed now, not intelligence
33 points by martinald
33 points by martinald
This is probably going to age like spoilt milk, but right now, models around the ~Opus 4.6 level seem to be 'smart enough' for most of my daily tasks - code, pulling together research, designing slide decks and doing analytical tasks against a plethora of databases.
I wish there was more specificity to this claim. What kind of code, what kind of research?
Sonnet 4.6 was good enough for my work. Doing C# mostly and ASP.NET REST based services hitting SQL Server (this tech stack is the definition of legacy LOL). I'm mostly using Sonnet 5 now. Maybe it's not good enough for C++ or Rust but for C# and SQL it seems fine 98% of the time. I tried both Fable and Opus but didn't notice much of a difference so went back to Sonnet for my employers sake. Again this is largely backend type of work with C#. I am spending about $100/month in token usage based on my employer's API usage spend (again pure Sonnet, would be more under Opus and Fable but I'm trying to be cost conscious for my employer's sake).
Are you finding something to the contrary?
Yes, Sol xhigh xfast is neither smart nor fast enough to truly let loose on anything. It's certainly much better than what came before it, but unless all you're doing is writing extremely predictable boilerplate (and, even then...) with a very exact spec, this stuff isn't nearly hands off yet.
I have lots of questions, but similar to the comment above me I’d be interested in similar receipts, because my experience is that it certainly works well enough for me when the architecture is specified up front and the scope is well specified and bounded with success criteria. Now, I wouldn’t let it loose overnight, probably even with that initial setup if I cared about the longevity of what is being built, but I also wouldn’t let even a senior engineer build so much of a product (tens of thousands of lines) without checking in and steering as needed.
It’s not really clear to me what sort of expectations people have when using these models (and harnesses). Like, when you say “is neither smart nor fast enough to truly let loose on anything” what do you mean by “let loose” or “anything”?
I find success with understanding what I want, writing it down, having a back and forth with the agent to make sure I’m not missing anything, then have it implement behavior in an incremental fashion with success criteria and checking in. If that doesn’t count as letting loose then I think we’re in agreement. Otherwise I’m confused.
Yeah that is not what i would call let lose. But i find that these models struggle even when directed and monitored. But I am not sure what you mean by "success criteria" because I often find myself narrowing down so far that i could have just written that code myself...
But i struggle through it because it can pay off in the boilerplate. And LLMs are really good at spotting issues once you have something (although often you need a fresh context). But even then you can't let them lose because they will find things that are not actually bugs.
I would not compare these to a senior engineer. But a good senior engineer with these tools can probably be more productive than just a senior engineer alone, as long as they're left to use the tools as they please.
An example of success criteria might be “operations must be idempotent” or “the external integration is done when the sandbox endpoints all return successful and complete responses with these properties” or something like that. I agree with your point that, at a certain level of specification, youre basically at the point of writing it yourself. Regarding boilerplate, if the code has established patterns and architectural direction then a lot of agent action is very boilerplate-like. I think it’s similar to a junior in that sense, setting up the agent for success requires scaffolding and bounds and the more examples you have of established patterns in its context window (that you actually want it to continue) the more hands off you can be.
The point is that agent success on tasks is strongly correlated with code which is under-abstracted and therefore overly boilerplate.
That's an acceptable state for a codebase when you're proving out an idea, but as your codebase grows, if you don't actively abstract things properly, you'll end up with very sparse code, which is hard for humans to wrap their heads around but which also bloats the context window for agents.
The point is that agent success on tasks is strongly correlated with code which is under-abstracted and therefore overly boilerplate.
This was extremely true 6 months ago, but much less true today. My Fable test programs included some very non-boilerplate stuff, and Fable nailed it with style. Including stuff I had never even thought of but that made excellent sense. Now, I still wouldn't trust it unsupervised on a big project, and I actually don't like models that can do that much without a human even understanding it. But the current state of the art is much more capable than extending code that is "overly boilerplate".
I'm not saying it can't handle subtleties. Just that subtleties are where these things are most often introducing subtle bugs. It's easy to become blind to this if you only spot the cases where it thought of something you didn't.
An anecdote: I was trying to write a lightweight Android app a couple of weeks ago. I am mostly a systems and embedded person, so I don't have that much experience with Android. I mean, I know how to use Kotlin and Android Studio to make a modern Jetpack Compose app of some variety... ish. I even wrote an entire working modern android build system (from primitives in the form of javac, jar, zip, d8, apksigner, aapt2, zipalign, kotlinc) for fun once.
There are a lot of subtleties in Android app development. There are the UI design choices, but also how you put that together from pieces, and how you wire things together. Android goes through revisions all the time. I wanted an app which was extremely lightweight and could target a very wide range of SDK versions so I knew I wanted to avoid AndroidX, Compose, Kotlin etc.
The one-shot result was impressive, but you know, the devil was in the details. With access to visual feedback it was able to clean things up, but it seemed to only do this by default by adding code. There were a lot of things that I left undefined, and rather than taking the route of relying as much on what Android gives you, it did lots of things in a way which isn't the "happy path" (not that there necessarily is that "happy" of a path with Android XML and Java, but you know).
I appreciated the fact that I had something to start, and could iterate on, rather than having to start from the position of unknown-unknowns (plain Java android development without compose is the opposite of familiar to me). I could go through the application methodically and request refactors until it was still doing exactly what I wanted but with a fraction of the code (ungolfed).
And I appreciate the fact that I can now ask questions about the code, get answers / references to resources, and can learn about this stuff and clean it up further myself.
But the closer I get to finishing going through this code, the more weird quirks I find. Like really odd things: e.g. this app is rendering Aztec Codes for an IATA BCBP boarding pass view, and the default choice of the agent was to specify Latin-1 ECI metadata which in a real world test caused a reader to refuse to read the code (I had a pkpass fallback...). Going through the docs I found no evidence that this was something anyone expects, it makes no sense given what BCBP data is (always ASCII). I cannot for the life of me figure out where this came from. Like you have to go out of your way to do this with ZXing, I certainly never asked for this.
(As you can imagine, there isn't a great way to test this ahead of time, automatically. While the IATA doesn't specify that Aztec Codes for BCBP should use Latin-1, it also doesn't specify they should not set ECI.)
And that's just the tip of the iceberg.
So yeah, I agree these things do a great job in the subtleties, but they also simultaneously do a poor job in the subtleties. If you don't personally know the subtleties, or are unwilling to read through the subtleties and learn about them after-the-fact, you're going to have a bad time. And if you don't think that's happening, well - I don't know what to tell you.
I can tell you that I use these things more than 99% of software engineers (I am in a rare position) and while they're certainly becoming more and more capable as tools, I am continuously convinced that whenever you see code produced by them and think: "this looks fine" and it's not extremely boilerplate, you should investigate further anyway.
And whenever they produce boilerplate, you should consider investigating if there's some way they can instead help you analyse the code to find good abstraction opportunities which would necessitate less boilerplate.
so I knew I wanted to avoid AndroidX, Compose, Kotlin etc.
Does using Kotlin actually restrict which versions of Android you can target? I thought it's supposed to be mostly a drop-in replacement for writing the same code in Java.
How much speed you need depends on your workflow, and on how much your model "overthinks."
There's really no substitute for an expensive frontier model if you just want to give it a two paragraph spec, walk away, and come back to a small working tool an hour or two later. Mostly this isn't what I want, because I actually enjoy programming and design. But if this is what you're doing, then speed isn't too important.
If you actually want to remain a participant in the process, then yeah, speed is important. And as I mentioned in another comment this morning, I suspect we've actually passed the point where smarter models are actually helpful for serious development with a human in the loop. Fable just wants to lock itself in an office and come back when it's done. Opus is smart enough to make the "cognitive debt" pile up fast.
The part that interests me is actually the recently released DeepSeek V4 Flash 0731. It feels a lot like Budget Opus 4.5, companies or organizations can run it locally, and the weights are MIT licensed. It's also very close to free on Open Router, at least for a model anywhere near its abilities. And, like, Anthropic and OpenAI feel like they're speed-running the whole enshitification roadmap with things like proprietary harnesses, hidden thinking, whole new modes that default to spawning 20 Opuses when Sonnet would be fine for the task, etc.
So if we have hit a practical intelligence threshold, at least for "active human in the loop", then I think I would actually prefer to spend that on local control rather than on speed. But I do agree with the underlying argument that we've crossed a threshold where we can start prioritizing other things than intelligence.
The part that interests me is actually the recently released DeepSeek V4 Flash 0731. It feels a lot like Budget Opus 4.5, companies or organizations can run it locally, and the weights are MIT licensed.
When you say that organizations can run this locally do you mean have engineers run it locally with aggressive quantization or do you mean run an inference server of their own where they share access to the inference API? When I looked at DSV4F it looked like to run it fully loaded serving multiple requests would need 2 H200s. At the prices those demand, they would have to keep it 100% utilized for 3 years to hit a conventional depreciation schedule. Am I missing something?
See my sibling post for more details.
You can serve it on an RTX Pro 6000 Blackwell ($11,800) plus 64GB of fast system RAM ($900) using a 3-bit quant, but it's a little tight and not quite as fast as I'd like. I think a practical corporate/SMB setup might look like a pair of 6000s at 192 GB. You need 162GB for the base model, and I think another 6GB for 1M context per session (it's super good about context). Practically? That gives you maybe 4 cached, full-size sessions before you spill something to RAM? A third 6000 brings it up the price of an SUV but gives you plenty of context cache and oodles of compute.
90th percentile Claude Enterprise seat usage (e.g., the developers, not the non-developer seats) runs about $750/seat per month, with high variability. So each Claude Code user costs maybe $27,000 over 3 years. So I think the 3 card setup breaks even around 3 users, which it should clearly support. As you increase concurrent users, you also want to look at vllm instead of llama.cpp.
Three caveats:
Or just buy it on the API from a decent vendor, where it's so close to free as to make no difference. That's probably the actually rational move if your compliance people agree.
I like to program half old fashioned and half LLM generated; the model, the harness and me all working together through an MCP connection that knows how to follow me in Neovim. In this context the fast models are so much more interactive. Increasingly, the SOTA models with deep thinking will go off into a loop and trigger multiple agents and you're left to twiddle your thumbs for a few minutes. With a fast model like the Gemini or Deepseek flash series, you can work collaboratively with the model. This approach may also age like spoilt milk for the kind of code where you want to be in the loop to control quality or your own understanding and learning of the software, the model being fast key.
Is this https://github.com/paulburgess1357/nvim-mcp ? seems like a great idea as i much prefer being in the loop. Worth jumping in ..?
~~Yes, that's the one. I also made a skill to help the agent use it which includes some hints about how it can read and search the docs and so on.~~~
Sorry, it's this one https://github.com/bigcodegen/mcp-neovim-server
I'm pretty indifferent to speed when using LLMs to code. The value is being able to use multiple at once, asynchronously (here's a stack trace, try to fix it) and I try hard to avoid just staring at a spinner.
Eh, I think people only do multiple tasks at once because they're so slow. I would much rather do one task interactively than be forced to juggle 3 or 4 tasks at a time.
I wish harnesses were more sophisticated about parallelising LLM requests. Support for subagents in opencode is basic and depends on the agent to divide work instead of waiting for one subagent at a time.