The pressure
116 points by andrewnez
116 points by andrewnez
It's a really tough situation and I wish Daniel and the rest that they make it through without too many adverse health or family effects.
That being said, I think it will be super interesting to see how this pans out. This is not the first time that a novel way of automated analysis suddenly uncovered lots of new vulnerabilities in all kinds of FOSS projects. Currently it still feels reminiscent of the advent of greybox fuzzing in the 2010s, but it might take different trajectories:
A. The amount of new vulnerabilities found by LLMs eventually goes down substantially, in part because devs incorporate LLMs for security research into their own workflows. New vulnerabilities that are beyond the reach of LLMs still continue to be found (the fuzzer scenario).
B. A, but vulnerabilities basically stop being found after the LLMs are done (the "LLMs solve security research" scenario).
C. Vulnerabilities continue to be found at a high rate in big projects like curl. It turns out that the number of bugs in a project of hundreds of thousands of LOC is effectively infinite (the "Revenge of Tony Hoare" scenario).
I think scenario (a) is what will happen.
Given a snapshot of a codebase, there can only be a finite number of security bugs. The deluge will begin to subside once the search space for security bugs has been exhausted. Then we’ll be left with a trickle of bugs that are discovered from future code merges and novel harnesses/model improvements.
With regards to scenario (c) in particular to cURL project itself, they found a lower severity bugs because of their previous high bar for security testing and other conventional bug discovery techniques. This goes to show that ongoing investment into bug discovery continues to pay long term returns. Regardless of bug discovery methods available today or in the future.
Paraphrasing something Marcus Hutchins once said, “bug discovery was never the bottleneck. It was an organization’s lack of decision to invest into security researchers was the bottleneck.” LLMs have made that investment cheaper, but organizations could have always decide to invest more into bug discovery to find more bugs. It’s a policy decision.
Any company involved in building LLM tech is not just engaging in the destruction of the natural world, but the software world as well. With hardware prices skyrocketing, personal computing itself is under threat. The same goes for good-natured open source developers who want to make things for the sake of making them. It’s interesting that there seems to be infinite funds to spend on undercutting and destroying existing community-managed open source projects but zero to spend on dealing with the fallout.
I think this proves Zig right. Just refuse to engage with LLM-discovered CVEs. Let someone wiling to do so work on them.
Just refuse to engage with LLM-discovered CVEs.
Presumably Linux users would have preferred the kernel retain multiple LPEs instead?
I get that this isn't exactly the point, but like, the problem with LLM CVEs is that anyone else using the same tools can probably also find them. And if they find an actual major issue, that means more people might be able to make something harmful out of it.
Of course, that's not to say that this applies equally to low-severity or non-security issues, which curl is inundated with...but you still need to actually determine if an issue is high priority or not, which brings us back to step 1.
The problem as I see it is that the curl maintainers are burning out, and will either give up (bad) or resort to start ”fixing” CVEs using LLMs themselves (bad).
So the status quo is simply not an option.
Another way forward could be increased social pressure on companies like Google, Microsoft and Anthropic to take more responsibility for the situation that they are creating.
I also think it’s important to remember that the curl project has provided no warranties and have no customers or liabilities, whereas the companies that have built products, services, cars and moonlanders using curl do have warranties, customers and liabilities.
This makes me think that maybe companies may be more likely to default to closed source. With LLMs, it's easier to write something from scratch (ironically because it was trained on open source), and if it's closed at least it will be harder for the typical scriptkiddie to ask an LLM to inspect the code and come up with exploits that way. Of course decompilation is still possible, and who knows, maybe LLMs will learn to read machine code.
We may be seeing the end of the current software development model with an open ecosystem that has largely relied on implicit social contracts to uphold itself, and a return to the bad old pre-1970 days of all software being proprietary and platform-locked.
LLMs remove a lot of the motivations for contributing to open source:
Of course, this works for the moment but what happens when the only code available in public repos is old outdated unmaintained "foundation" or vibecoded slop?
Of course decompilation is still possible, and who knows, maybe LLMs will learn to read machine code.
From what I'm told by people who have tried reverse engineering with them, LLMs are actually quite good at this already. There's some problems like not really having a concept of reachability (so it doesn't necessarily know what's most important), but they can already tear apart proprietary binaries, especially when aided with MCPs for Ghidra/IDA/etc.
The problem as I see it is that the curl maintainers are burning out, and will either give up (bad) or resort to start ”fixing” CVEs using LLMs themselves (bad).
They might also load-shed by dropping features; curl is known for its HTTP support but it also contains implementation of POP3, FTP, LDAP, SSH, SMB, and others. Do those need to be in the same library? Maybe they could be split out into separate projects.
Or they might reduce platform support to enable expanding the use of memory-safe languages into the core library. curl supports AmigaOS and VMS, is the desire to support those platforms enough to outweigh not being able to use Rust (etc) for the code that touches hostile input?
Do all of the supported protocols need bare-metal performance? Will Daniel get complaints if the Gopher client code has a performance regression? Maybe they could take the rarely-used code and throw it into a w2c2 sandbox.
I suspect that on balance, the rare protocols probably contribute a lot fewer issues than the more commonly used ones. So dropping support for rare platforms and formats is probably less of a win than one might hope. I bet the stress of a CVE that can only be exploited on AmigaOS is a lot lower to begin with as well.
But yeah, maybe one path forward is to reduce scope and ambition.
I suspect that on balance, the rare protocols probably contribute a lot fewer issues than the more commonly used ones.
What is the reasoning behind that theory? Sounds pretty unlikely to me.
Maybe it’s not universally true, but for example HTTP/3 is a complicated protocol which would be difficult to get right, and has been added more recently to curl. Something like FTP has been around forever and is a lot simpler. But yeah it’s just a gut feeling.
Another way forward could be increased social pressure on companies like Google, Microsoft and Anthropic to take more responsibility for the situation that they are creating.
This has never worked. No amount of “pretty please” or pointing out the impact of their decisions will reverse their course. Unless they’re held to account via regulations, capitalistic organizations will continue to refuse to comply. See Apple AppStore verdict affecting only EU, Google search monopoly ruling, MaBell breakup, etc. Without regulations or breakups, they will fight tooth and nail to maintain their monopolistic practices and carelessness for society at large.
No amount of “pretty please” or pointing out the impact of their decisions will reverse their course.
This is not at all what I am suggesting. I am suggesting that the curl project can ignore CVE reports generated via LLM.
Another way forward could be increased social pressure on companies like Google, Microsoft and Anthropic to take more responsibility for the situation that they are creating.
This I'd certainly say is pretty fair. I think the main thing to note is that, in that case, the pressure is to also help with fixing the issues (even if that's just $$), which would still result in the security issues being fixed!
I think this proves Zig right. Just refuse to engage with LLM-discovered CVEs. Let someone wiling to do so work on them.
To provide more context, in the case of Zig, we're in a different situation than Curl.
Zig is still under heavy development and every time we flesh out a piece of functionality (say, incremental compilation, or async I/O for example), chances are that we are introducing new bugs, as well as removing bugs that were caused by the fact that the implementation was previously incomplete (and did not handle edge cases correctly, for example).
Given the stage of development, if somebody were to throw Mythos (or whatever) at Zig in a "find all bugs, make no mistakes" fashion, it would produce a cubic ton of reports, the entirety of which would be effectively worthless to us. At the moment the main value that bug reports bring is that they show us that there are users blocked on a given use case and, if we decide to prioritize it, we know that we'll unblock those users.
That being said, this state of things won't last forever, a lot of critical compiler features are coming together and, as things stabilize, finding and fixing all bugs will become the first priority. At that point the fact that a bug is known will have importance regardless of how that bug has been found, but that's a problem we'll tackle once we get there.
In the meantime, the policy is just no LLMs.
Let someone wiling to do so work on them.
Oh, I'm sure the blackhats will be happy to work on those security issues. Perhaps not in the way anyone hopes they would.
I understand (though don't agree with) prohibiting LLM contributions. But a security vulnerability is a vulnerability regardless of how it's found.
I think what Daniel is doing is for the best--fixing bugs that he can to make users better off, while noting the toll it takes and asking for support. I doubt he'd ever read this, but I'd also support/encourage him limiting the amount he works to prioritize his physical and mental health. I'd like to say the world will be understanding--and I think most of it will be, with a small minority of complainers.
I'd also support/encourage him limiting the amount he works to prioritize his physical and mental health. I'd like to say the world will be understanding--and I think most of it will be, with a small minority of complainers.
Yes, absolutely!
I think in practice our suggestions would have the same outcome. Once the rate of incoming reports outstrips the rate that the project (meaning Daniel) can resolve them, he might as well stop trying to keep up entirely. The question then is what that will do. Will more people or companies step in to try to help out, will some company try to replace curl entirely?
I think this proves Zig right. Just refuse to engage with LLM-discovered CVEs. Let someone wiling to do so work on them.
It's unclear how this is going to work. If the CVE is legitimate, it does not matter how it came about.
What is clear is that under the license that Zig is provided, the project is under no obligation to address even legitimate CVEs.
Just refuse to engage with LLM-discovered CVEs.
I’ve seen a similar attitude expressed towards security bugs discovered by humans on Project Zero, too.
This seems to be missing two key points:
The LLM companies or Project Zero didn’t put the security bugs in the code.
Security bug fixes aren’t for the LLM companies or for Project Zero but for users. If security bugs are exploited, users suffer.
(Especially in the case of LLM-discovered bugs, there are already signals from multiple Open Source projects that different people using the same LLM file duplicate reports, so we should assume that attackers will become aware of the LLM-discovered bugs if the defenders disengage.)
I am jealous of those projects that shipped a horrible bug at some point in the past that made the world burn for a while. They got attention and some of them then got funding and financial muscles to get them staff and hire multiple full time engineers. I sometimes think we would be better off if we also had one of those.
This also happens in many places of work. Failing teams are getting more staffed while successful team need to do more with less as it works without something horrible happening...
I don't know if it makes sens for a project like curl, but would a feature freeze for a certain amount of time help to only focus on incoming bug/CVE reports ?
I imagine that for given set of features, the amount of potential bugs/CVEs is finite and it will get close to zero as they are being fixed.
Anyway, good luck to them: it must not be an easy time to maintain a software that important.