We should all be using dependency cooldowns
118 points by yossarian
118 points by yossarian
Sounds like language-level packagers are starting to realize that it turns out distros had some good ideas after all.
Probably shouldn't pat the distro model too much on the back. That's also the model of "oh I hit a bug, let me report it upstream.... oh it's already reported... oh and it was already fixed like 9 months ago.... but why don't I have it".
The "stability" criteria in the debian model also in some sense ends up being "you get to keep the same bugs around". This is a fair tradeoff to make in some sense (because the theory is you don't get new bugs in that window), but it can be very unpleasant in many scenarios. And if people are all consuming your software via a distro, then you get a much worse feedback loop when trying to fix things.
This is influenced by me being a desktop user of linux distros, but I've hit plenty of bugs at work (e.g. pinned to some LTS, so I end up with some million-year-old copy of gettext with a bunch of random bugs... and of course I can't "just" compile newer gettext because it's tied to some libc version shenanigans) downstream of this mentality.
Don’t forget bugs that only exist in distro versions because they’ll do piecemeal backports the upstream neither did nor was informed of.
My thought exactly! It's very amusing. Sounds like the preferred average update latency is > instant and < 2 years? Maybe more things should run on a 6-month rolling-release schedule...
Who could have predicted that curation would be more secure than a free for all? Turns out the cost of sprawling dependency graphs with loose version specification is quite high!
If everyone used dependency cooldowns wouldn’t they also become less effective?
The central claim that most supply chain security vendors make is that they can proactively detect dependency compromises, i.e. even before evidence of compromise occurs on monitored user machines (EDR, etc.). So I see this as testing that claim.
(At least in the Python ecosystem, there's been an effective relationship between index scanners and the index itself -- index scanners regularly find and report malware, which then gets taken down. It's hard to predict how much that will change over time, however -- right now a lot of of packaging malware is low-sophistication, e.g. attempts to run malicious code directly via setup.py or its equivalent in other ecosystems. As attackers before more sophisticated, the accuracy of scanners may decrease.)
For:
Against:
I'm too lazy to do the rest
The xz-utils issue was discovered by an unaffiliated user running Debian testing, and noticing a perf regression with SSH logins, so we got lucky there… but I’m not sure I’d count it as Debian themselves doing the scanning etc.
At least it will give the dependency owner some window to realise that they have been compromised. They can realise this without needing others to download their packages.
I briefly spoke to Josh Bressers about this on the Open Source Security podcast (coming soon) and I think it's twofold:
Are non malicious latent vulnerabilities less common than active supply chain attacks? Is there a tension between cooling down and getting the latest version without the vulnerability? I can’t imagine you would want to exempt fixes from the cooldown, that just incentives the attacks to target the exempted releases.
To my understanding, Dependabot doesn't include security updates in the cooldown period. But you're right that this presents an interesting problem -- security updates should receive immediate bumps, but this makes bogus security updates a juicy vector for an attacker.
(Right now, the fact that a maintainer has to sign off on a GitHub Security Advisory or similar makes it hard for an attacker to take advantage of that, precluding full account takeover. But clearly that's not a perfect defense, just another hurdle the attacker needs to clear.)
Acknowledging that you’ve already acknowledged that this is not a perfect defense, after xz I gotta assume we have to be thinking about malicious maintainers when evaluating these things.
Yeah, I completely agree -- an actually malicious maintainer is sort of the nightmare scenario, in the sense that they have an unbounded window of opportunity (or at least, can employe subterfuge to keep it open until people realize that they're fully malicious).
I have a longer form thing I've been meaning to write about the taxonomy of these kinds of supply chain risks and their volumes -- I don't have great hard numbers for these things, but I kind of suspect that there are large orders of magnitude between these different types of attacks, which are inversely correlated with severity. For example, we might have 100,000 open source vulnerabilities assigned CVEs each year, 1,000 typosquatting attacks, ~10 compromised packages, and ~1 malicious maintainer. But that ~1 malicious maintainer can do a lot of damage that open source vulnerabilities on average can't!
an actually malicious maintainer is sort of the nightmare scenario, in the sense that they have an unbounded window of opportunity
Arguably it was a cooldown period that allowed catching the xz compromise before it got really bad.
Distros like Ubuntu effectively enforce cooldown periods with their release schedules.
Wait a minute, that gives supply chain attackers a trivial way to go around the delay, doesn’t it? Just mark the malicious update as a security fix, done.
Or are there additional steps to security fixes that would make this harder than it sounds?
the fact that a maintainer has to sign off on a GitHub Security Advisory or similar
Aaand I don’t know how to read. Oops.
We (Renovate) also raise security updates immediately, not waiting until the minimumReleaseAge has passed
But as noted in my other comment, we really do need package managers enforcing this as well
I think this is a band aid on the bigger problem that is the package manager independently creating mystery mixes whenever installing and/or adding a dependency.
Sure, if you're locked into a non-MVS ecosystem, then you will definitely gain something by adopting cooldowns, but this whole thing is a solution to a problem you shouldn't have in the first place.
The best moment to vet a dependency is when you're adding/updating it as a direct dependency to your project.
I find absurd that people are fine with their package manager unilaterally bumping up indirect dependencies, picking a version that potentially none of the direct dependants have ever ONCE vetted (or even just tested), and I find even more absurd this happening when users install a package without forcing usage of a lockfile.
Aren’t those orthogonal concerns? You should have the appropriate lock files to make your build reproducible, and then you can apply cooldowns on updating your lock file.
The fact that you have a lockfile means that your package manager has chosen a mystery mix for you.
With MVS you don't need lockfiles because the package manager will always select a version of your indirect dependencies that at least one of your direct deps has already seen and explicitly asked for.
With MVS you don't need lockfiles because the package manager will always select a version of your indirect dependencies that at least one of your direct deps has already seen and explicitly asked for.
Unless you put a higher one in your own go.mod which you should do if you don’t want to wait for your dependencies to apply security updates for their dependencies.
Actually you gave me the idea to maybe only update indirect dependencies automatically if there’s a security fix. Renovate can read OSV data for Go, gotta try that.
There is a methodological problem here: you look at data on attack actions that was collected before a specific defense was well-known and wide-spread, and then you extrapolate that the defense will have the same effectiveness after it becomes well-known and wide-spread.
In the very small sample set above, 8/10 attacks had windows of opportunity of less than a week. Setting a cooldown of 7 days would have prevented the vast majority of these attacks from reaching end users [...]. Cooldowns are, obviously, not a panacea: some attackers will evade detection [...]. Still, an 80-90% reduction in exposure through a technique that is free and easy seems hard to beat.
I does not sound very plausible to me that this defensive technique will remain as effective as it is today after it becomes wide-spread, including well-known to attackers. (Well, I could see this happening in the scenario were few people implement it, so they're not worth changing attack patterns for; then those people could be protected effectively as long as they remain a minority.)
Maybe there are fundamental reasons about this class of attacks and defenses that make it hard for attackers to react, and to reduce the effectiveness of the defense; and those give you confidence in your 80-90% effectiveness estimate for the future. But you haven't discussed them in the post, so the reader has no reason to believe the predicted estimate.
Maybe there are fundamental reasons about this class of attacks and defenses that make it hard for attackers to react, and to reduce the effectiveness of the defense; and those give you confidence in your 80-90% effectiveness estimate for the future. But you haven't discussed them in the post, so the reader has no reason to believe the predicted estimate.
The reason that they give in the post (and I have not decided yet that I find it persuasive) is that the businesses ("supply chain security vendors") who make money by finding vulnerabilities in github repos/npm releases/pypi releases/etc. have strong business incentives to find these issues as fast as they can because being first to find/publish it drives more business their way. I think that idea is sound; attackers are already incentivized to keep their vulnerabilities hidden for as long as they can manage, so varying cooldowns shouldn't change attacker behavior much.
The thing that I find most compelling is that I don't see a way that cooldown periods help attackers, and they have near zero cost. So even if 8/10 drops to 4/10 thwarted because of some behavior adaptation we haven't yet considered, this seems like a significant net benefit.
Personally, I'd like to see these land in uv and npm, but if they don't, this is compelling enough for me to consider adding upper version pins (which I don't generally use unless I think I need to, right now) to everything and using dependabot to enforce the cooldown.
I'm not saying the idea is without merit, I'm no expert and I am willing to believe that there are clear gains to moving the ecosystem towards this by default. My point is that the methodology used in the post to proclaim some quantitative effectiveness estimate for this idea is incorrect, so the quantitative claims of effectiveness are effectively hyperbole.
I think I agree with you about the quantitative estimate, even based only on sample size regardless of methodology. I did not gather from your comment that you were only disputing the quantitative estimate, so I replied saying why I didn't think I believed the quantitative estimate needed to be even close for the idea to be worthwhile. But if all you're questioning is whether 8/10 is accurate, I both share your question and doubt it's knowable with information currently available. I just don't think that figure changes the argument for the course of action.
For what it's worth, this is essentially my sentiment -- I was pleased to see that it was 80-90% effective against the historical record, but I would have considered even half that to be a signal for an extremely good mitigation in this context.
Attackers already want their attack window to be as long as possible. Can you sketch out how attackers might adapt to this mitigation?
The mitigation assumes that security scanners can reliably detect compromised packages before users start downloading and using those packages. If I was an attacker I would try to hide my attack logic so that it can only be detected dynamically (by running the code rather than analyzing it statically), and design a time-based activation mechanism so that no attack happens during the average cooldown period of my target userbase. Then attacks will only start happening after some delay, and will be difficult to detect before they have started affecting users, same as today. (Scanners can respond by trying to defeat the time-based logic, for example by running the packages in sandboxes that lie about the system time, and then attackers can think of getting more reliable time information from innocuous network interactions with the outside world, etc.)
FWIW, I agree that there's a methodological bias here. But I also don't think there's a counterfactual to that bias; all defenses become weaker over time, because security is an arms race. Or in other words: you shouldn't read the claim to be a permanent one, only an observed one.
With that being said, I'd expect this to remain relatively effective, for the incentive reasons mentioned in the post. Separately, it's hard for attackers in the discussed scenario to broaden their window of opportunity: there's a natural tradeoff between compromising less popular packages (less impact, but longer windows before detection) and compromising more popular ones (higher immediate impact, at the risk of closely following detection).
Now that people are slowly figuring out that automatic upgrades were not necessarily a good idea, maybe it's time to seriously consider minver by default in more ecosystems. It solves a lot of problems, can do entirely away with lock files and simplifies systems tremendously.
This is built into npm:
https://docs.npmjs.com/cli/v8/using-npm/config#before
$ npm install express --before=2024-01-01
$ npm outdated
Package Current Wanted Latest Location Depended by
express 4.18.2 4.21.2 5.1.0 node_modules/express asdf
Then it could be automated to a week ago using this:
$ npm install express --before=`date --date="last week" +%Y-%m-%d`
You can also throw this into an .npmrc file but I'm not sure how to keep it "up to date" to always be a week ago:
before=2022-06-01
One way around the "if everyone waits 72h, we just discover the problem in 72h" problem is for everyone to update deps occasionally on their schedule (so if you do it monthly, the Nth of the month with your own arbitrary N). That also batches up testing and debugging around upgrades. There'll be a distribution of how quickly people pick up a given update, but you can hope that if someone does have to be affected before a problem is caught, it's more to be one or two folks, not the whole world.
Folks may also self-sort by risk-vs-novelty preference (like how there's the range between LTS versions of things and nightly builds), potentially helping ensure that when dowstream projects do get bit it's less likely to be those used by a lot of people.
Cautious downstream projects could still use an absolute 24/48/72h delay on top of the lazy upgrade schedule. As another comment suggests some problems will be caught early independent of broad package adoption: the maintainer realized their account was compromised, a co-developer saw something fishy, a user who loves the bleeding edge caught it, that sort of thing.
Loosely related, but it could be cool to have low-stakes CI with only fast-expiring credentials and minimal to no secrets of any sort in it, where the risk of running test suites with YOLO versions is mitigated somewhat. It would help catch incompatibilities early for sure, and it's somewhere the behavior (not only source) can be seen, which...might help catch some kinds of attack?
This is all fundamentally hard! Everyone wants a really good ratio of code used to effort expended, but it inherently takes a lot of effort to check the changes happening to lots of packages. Supply chain attacks have been possible for a long time and we're kind of lucky it's taken this long for them to blow up in the way they recently have. There aren't trivial solutions and I don't think the problem is disappearing soon.
Things have been blowing up this way (and other ways) for a lot longer then it seems. It's just they nowadays the scale and visibility of all the changes has gone up by orders of magnitude.
It's one thing when a random irc server on a random university crashes, and a few thousand students can't chat about crap.
It's another thing when a few dozen million people can't work at all.
Shout out to zizmor's dependabot-cooldown for warning on repos that don't have cooldown configured
Thanks for writing this up @yossarian, have sent it to a few colleagues that were on the fence :)
By pure coincidence, I added a dependabot cooldown a few hours before reading this, because zizmor recommended it to me. It's a great tool, thanks for your work!
You get this if you don't blindly add/update dependencies in first place. Even more so, not blindly updating dependencies might make you review them and remove them, not just making it less likely, but impossible to be affected by such an attacks.
I don't think this approach is overall as good as it might sound:
Not saying that it is a bad idea at all, but I think if you don't actually do anything during the cooldown nothing much will change. Also because at least for attacks involving the original source after all the cooldown is anyways starting with something like a pull request and ending with it actually being used in your project. That's quite the stretch of time for "magic" to happen.
Again, don't get me wrong, but often when such things are talked up the industry ends up in a state where something like "we use dependency cooldowns" will gradually lead to people thinking "it's fine, we already solved that problem. See, there the ticket has been set to done six months ago". That's of course over-simplifying, but that's something that makes security improvements that only handle quite specific scenarios including to attacker having no means to avoid those mitigations a bit of a challenge. We saw that attackers indeed do have a long breath and since they have been found by accident/luck (having someone great knowing what they are doing look at it at the right time), I don't think these will will do that much for dependencies that take some care. And for dependencies that are to quote a coworker "managed like skate parks", you really shouldn't use them in first place, which is why reducing the attack surface, by reducing dependencies and then actually having time to look at what and why you are using and updating dependencies in first place will give you much better protection.
The reason why issues are found isn't magic, but people looking into things. The whole "many eyes" only works if people actually look at things, and as an added benefit you get a much better understanding of your system.
Again, I don't think the idea is but, but I do think the idea of just adding a delay isn't magically making it (a lot) harder for an attacker in many situations. And of course it all depends on the specific scenario, which is exactly why you shouldn't consider such things fixes for the problem but small improvements for the problem space. And only if you don't do stuff like ignoring security patches which very well might be used for attacks. Having an RCE fixed adds urgency, which might make pull requests, etc. to go through with less checks. So an attacker with enough time (which they seem to have) might just introduce a subtle enough vulnerability in a frequently requested, complex enough request and once released add a fix with the payload. Or be fine with just adding the vulnerability in first place.
They could even have their own repository to check if scanners detect flaws, just like a malware author probably has scanners pass over them before releasing them into the wild.
I’ve been aware of the Renovate feature but your overview of the time windows makes me actually consider doing this, so thanks :)
Yes to all of that. I am also using cooldowns l have „once a month“ for my hobby projects just because I can seldomly spend more time on it anyway.
On your "once a month" do you only pull in updates that are at least a month old, or do you pull in whatever is current at that moment? If it's the latter, it seems rather hit or miss whether it counts as a cooldown.
Very much agree with this! Although I am biased, as I've led the work to enable this by default for users of Renovate's config:best-practices users, for packages in the npm ecosystem 😉
Wholeheartedly agree that this needs to be done in conjunction with package managers - ie if a security update is created for a package, that should bypass Renovate/Dependabot's wait period. But the package manager should still block it, unless explicitly overriden (https://github.com/renovatebot/renovate/issues/39168)
One of the issues we've also found is that sometimes not every API or registry supports it equally, so ie we need to provide a safer case for "what happens if I don't have a release timestamp" which means "don't create a PR"
I prefer avoiding dependencies, and I like to manually update at the start a large release (so I have time to find bugs). But a cooldown can be fine
Strange that (large) F/OSS projects are not vendoring (and bundling) dependencies by this day. I do understand that it’s convenient to pull down packages from the internet, but if $registry, $cloudProvider or Cloudflare goes down, then how am I supposed to build e.g. Zed editor that requires ~1900 crates (combined)?
I'd like to see vendoring and "stable forking" become a more first-class workflow in language package managers. I'd say cargo and go mod do this reasonably well if you want to fork one or two libraries, but if you want to vendor several things, it's almost easier to run your own instance of something like Artifactory and proxy the entire ecosystem.
And that's what distros do: they host a copy of all the code they use, patching and upstreaming at their own pace.
Can individual projects with language-level package managers do any better here? In an ideal world you can easily vendor, patch, and contribute to ALL your dependencies.
I’m against vendoring through package managers instead of original source repositories. Also libs redistributed and compiled by distros as shared libraries for dynamic linking are not any safer.