Everything is about to “go dark”
40 points by untitaker
40 points by untitaker
As someone working on software security, heck, even using extremely capable LLMs to find shockingly good and complicated exploits, I don't believe we will run out of bugs.
The key to these mercenary hackers being successful is a result of complexity and understanding. In essence, those who hack a target understand the target better than those people who actually produce it. That's already the case. These people exist. Attackers who used to read code all day are now making LLMs read code, but they still have a massive advantage over the defenders. There's this old quip that attackers need to be correct just once and defenders have to be sure everywhere. But also, if you want to fix or prevent a bug as a defender, you sometimes have to walk through massive hoops. Ask three teams, navigate the org-chart, change the whole architecture, fix a dependency, work around existing code that is not in your control (e.g., drivers). All of that. The attacker's don't care about that. In fact, they benefit most by working against that. Finding bugs at the seams where either team (or either company) would call it a latent or low-severity bug that isn't worth fixing. Find enough of these bugs and chain hem together will gives them great exploit chains with high longevity.
Maybe, just maybe, I'm willing to believe that software overall might be come more secure in the longer term, as many bugs are now more easily found with LLMs. But I won't think any kind of interesting targets will have "zero bugs". With the AI-craze right now, there are too many decision makers who believe that finally, everything in software can be done. This feature creep, this "hunt for shiny" is going to create plenty of bugs. But only time will tell if it's going to result in a new equilibrium or truly zero bugs. I'm betting on the former.
A great point. Nothing yet has stopped the tech behemoths from sprinting forward as fast as they can holding a pair of scissors. Handing them a shiny new pair of even-sharper scissors and thinking "wow, this will finally get them to slow down!" seems a bit naive.
I think this is a very salient point. I've been speaking to some friends working in big tech™ recently, and it surprised me how few of them really understand the entire software stack they are working with.
We're already seeing the negative consequences of excessive complexity in software, but the tendency to complexity only seems to be accelerating. LLM coding agents make it extremely easy to design an excessively complex system, so it seems likely that such systems will therefore be harder to defend against a skilled, motivated attacker.
This seems backwards. My intuition would be that offense has the advantage. It's always easier to break things than to fix them. I worry about "going dark" where connecting your computer to the internet becomes too dangerous, and the internet as a global system ceases to exist.
If LLM usage becomes prevalent and accessible enough to that both sides have access to the same, the one used to find flaws will be used to patch them before release.
With the above in mind the game has shifted from a game of cat-and-mouse to a game of whoever-has-the-strongest-LLM; and if we assume neither party (code producer vs code attacker) is developing AI - it's a stale mate (where the defensive side wins per default simply by "being first").
Sure, one can argue that whenever a new models is released the odds are back in the attackers favor (ie. projects who have not updated their pipeline becomes vulnerable), but that window closes as soon as they re-run the pipeline... and then we are kinda pinning "dead" projects vs alive attackers, but in that area attackers have free rein regardless.
The "red team" only needs to find one meaningful exploit at any given time, while the "blue team" effectively cannot make a single mistake, and must keep that up indefinitely.
There are also many, many "blue teams", many without any incentive to spend more on security than the equivalent "cybersecurity" insurance premium costs.
Many systems like those governing traffic lights, aerospace, point-of-sale, consumer networking, etc. are also not exactly the pinnacle of continuous-deployment with a focus on security. There will be a large attack surface at any given moment, for the foreseeable future, and I don't see why any one actor in the space will have much of a reason to improve things drastically, considering everyone else is likely just as vulnerable.
How does the existence of LLMs fundamentally change any of that?
Edit: In the context of the article, I guess the number of "blue teams" is much smaller (Apple, Google, Samsung, ...?) but I still am not convinced of the idea that LLMs will so thoroughly secure these massive systems to the point where no exploits remain.
like those governing traffic lights, aerospace, point-of-sale, consumer networking,
You choose a very diverse set with very different approaches to security. Traffic lights are typically extremely simple and have known solutions for never ending in an invalid state. Aerospace is typically highly regulated and security is pretty strictly managed mostly by isolating systems. Point of sale have a wide variety. Consumer networking cares relatively little. But my point is that things that actually matter will never get continuous deployment because it's just a different environment. For example airport systems like lights will have a solution produced for specific requirements with lots of testing and docs over months and there's no such thing as just updating them with a new version continuously and never will be. (At least in the cases I know of... maybe there are exceptions)
Anyway, regarding the one mistake for the Reds to find: every time you know about some vulnerability class, you can usually protect against that class as long as it's someone's job up to it. It's not an infinite list of problems you can find - these days, given enough time or tokens you can effectively remove most types of issues completely. Some mechanically (memory safety), some by design (various injections), some by doing just-enough model verification (concurrency), etc. Soon we should have good enough templates for automatically securing applications, that finding a new issue in them will be a major undertaking. (Unless it's a huge existing system which needs to preserve compatibility and can't handle bigger changes... we won't secure Linux kernel in a week) Not everyone will adopt this immediately, but we're slowly getting there - for example automated reviews are getting a lot better. I actually see codex recent pointing out issues which would be ignored otherwise. I've been also learning how to do code verification in practice recently and it turns out you can close a lot of holes already by just pointing an LLM at some existing system, no experience necessary. (It can reproduce the failing tests for actually running systems)
Throwing LLMs at finding individual issues the way red teams would do it would be a waste of time. The developers can do way better today.
I am quite sure the argument of the article is that "everyone has access to LLMs" + "LLMs will be more effective at both patching and finding vulns than humans", with that I find it hard to reply to your comment as it seems to disregard that premise.
I believe @stringy's point is that blue teams are not incentivized to use the most expensive LLMs (plural, because they don't necessarily all find the same flaws) or to spend millions of tokens to exhaustively audit every release. Red teams just need to do it once.
Yes, I think that's a good distillation of my skepticism.
That's why I left the edit about the mobile phone duopoly. I can see the story there being a bit different, since they have the resources and the much more of an incentive.
And yet, without formal methods you don't really know "when to stop", right? You can conceivably run LLMs (plural, as there will be plenty of models capable of finding exploitable issues) over your codebase with every new patch in a loop for days and still expect to get new unique problems, they will just become more and more rare, right?
This is where "the attacker just has to find one" becomes an issue, because the shear cost to the unprincipled developer who wants to just start throwing LLMs at the problem feels like it will be enormous, and never actually come with any guarantees.
Like everyone says, I expect the opposite: I fear that we are about to enter an era of the most serious, widespread security compromises we've ever seen. The foundations of networking technology are all almost entirely memory-unsafe languages, which we've never managed to secure. Rust and GC languages and Ada Spark are a drop in an ocean of low-level C.
And even in GCed languages, today's LLM-written code is riddled left and right with security holes, because almost any LLM would prefer to duplicate a 12-line the "can user X access resource Y?" check for the 15th time rather than set up some kind of centralized authorization. And increasingly, the people giving orders to the LLM don't even know what those words mean.
It's not that you can't get an LLM to run a decent security audit. (Opus 5 isn't bad, and it's far cheaper than sending vast amounts of money to Hacker One.) It's just that we have far more memory-unsafe code than engineers with modern LLMs can possibly remediate. And only a tiny fraction of vibe coders are even going to ask an LLM to do security audits.
Meanwhile, I anticipate that anyone with a million dollar budget will soon have access to the kind of models like Mythos or the unreleased OpenAI model that broke into Hugging Face. Give it year, maybe two. The next 5 years heavily favor attackers, I fear.
This post is speculation (how big tech corporations might be prevented from using strong encryption, and what effects that might have on us) based on a hypothetical (the people with guns parts of the government might get too frustrated about not being able to get our sensitive data) based on a hypothetical (LLMs might lead to there being no security vulnerabilities).
At that order of speculation, almost anything is possible, but it's hard to have a real conversation about any of it.
My phone runs a non-updatable 2021 kernel (and i bought it just launched on 2022...) i pretty much doubt I'm the only asshole in this situation. Whate were you worried about? If only i could do something so that (for me) foreign country would not think about having access to my private data
Is there a possibility that the security implications of LLMs lead to a more local-first based future? Not that local-first software is more secure, but that there is less incentive to hack many single machines.
Or maybe a better question is, could LLMs usher in an entirely new paradigm of software that rather then favouring one side of the arms race, just circumvents it altogether?
I have doubts, starting with the framing.
Do we actually care about those capabilities notionally available to law enforcement? Whenever they announce having done something that looks worth the effort, it is more like «oh we just outcompeted a supplier known to market to criminals, then got a warrant to backdoor our own customers suspected on circumstantial evidence» or maybe «we subpoenaed a bank for all data about a money transfer». Not that either of these lack cases of abuse of power, but the hacking for law enforcement ends is just not clearly established to ever have non-abuse.
(I think some judge has even ordered law enforcement to submit evidence about hacking capabilities having had any useful outcomes — under agreement to seal that evidence — and then declared that nothing meaningful was there, and details are sealed as promised?)
If the primary way to find and fix vulnerabilities is through LLMs, then the logical next step for governments would be to backdoor models and/or the infra serving them to miss vulns that would otherwise be caught. Given that labs can't even contain their own models I'm not particularly confident in defense getting so strong vulnerabilities peter out.