Dependencies should be fetched directly from VCS
1 points by sanxiyn
1 points by sanxiyn
I thought this was going to argue for vendoring your deps which I daydream about doing
My primary worry here is people can decide to delete their GitHub repository. With package registry it is not just author's decision.
right, I mean, that's a feature, surely. I can't remember what it was but I do remember somebody choosing to permanently and globally un-publish their work a few years ago, and one of the big registries (npm?) made the policy decision that that's forbidden... and that felt morally offensive to me, disrespectful of the author's humanity. if people don't want to share their work, they don't have to.
Are you saying authors should be able to delete people's backups? Because that is what this is, package registry is just a backup that is convenient to use.
a public website owned by someone other than you is not your backup. in the cases of npm and github it's Microsoft's backup. if a piece of data is important to you, you should have your own copy of it.
I realize that comes off as excessively individualist, but I feel the need to parse through the question of who if anyone "owns" something on the internet, because as a privacy person I've seen in adjacent domains how making simplified assumptions about it eventually wind up enshrining injustice into law. we have to think these things through in the present so we can wind up with good practices, which will turn into good policies, which will turn into laws, over the natural course of events.
Correct.
Unkind, inconsiderate, or whatever adjective you want to use, if I say "fuck off, MY repo is now private," I am allowed to do that. "But I was using that", for any definition of "I" (me personally, my friends, my company) is irrelevant once I've made the decision to take my repo private.
Personally, I would probably give notice, try to find other maintainers, etc before I ever got to this point, but then again I've never been harassed online (please don't take that as a challenge).
As consumers of other people's code directly from VCS, tooling can help. We probably need easily accessible read-through caches (e.g. maintain a fork that follows upstream, build one into the package manager like Go, etc). If your package manager is fetching sources from VCS, you may have a copy of the repo locally, but it may be hidden and/or stored under a different name (a hash or UUID). If you're unlucky you may just have a tarball of a single commit.
yes, part of why I advocate for taking this stance now is that if we try to take it seriously, we will be motivated over time to come up with tooling that makes it easier.
If you pick a random person on the internet to host your backups you're asking for trouble.
I expect https://snapshot.debian.org/ to be available instead of maintaining my own backup. Package registry is no different.
I can tell you for certain that package registries regularly go down for unexplained reasons. I ran the CI/CD pipelines for Apple about 5 years ago, supporting tens of thousands of engineers pulling from npm, pypi, and rubygems. Upstream will go down for minutes at a time, every few weeks.
It’s unreasonable to expect the cost of hosting to be completely shouldered by these free software foundations. It’s a LOT of data, and CI pipelines hit them pretty hard, often without vendoring/caching.
I think any company over 250 engineers should start to consider vendoring their dependencies, and software foundations should institute brownouts as a rate limiting mechanism to incentivize people to vendor harder.
Also, it seems reasonable that some specific versions of some dependencies might get deleted, like in the case of a compromised package release.
I can tell you for certain that package registries regularly go down for unexplained reasons. I ran the CI/CD pipelines for Apple about 5 years ago, supporting tens of thousands of engineers pulling from npm, pypi, and rubygems. Upstream will go down for minutes at a time, every few weeks.
GitHub's doing a lot worse than that these days, sadly.
I disagree with the conclusion, but I feel like I don't even need to get that far because I also disagree with the reasoning given for the conclusion. Package-publishing workflows and package-registry accounts get targeted right now, sure. But get rid of those and switch everyone to install-from-VCS, and we'll just see the attacks switch to targeting VCS hosting and associated accounts. Things like the xz backdoor, for example.
And with a malicious commit, possibly way back in the repository's history, cleanup is likely to be difficult, and maybe even impossible depending on the amount of history rewriting required. The "Jia Tan" commits are still in the xz git repository, for example, and the malicious 5.6.0 and 5.6.1 release tags are still present, too. A system that just installs from a git repo at a specified tag or revision would go right on installing those malicious versions unless and until told to do otherwise. But a system based on publishing packages to an index can just have the index yank the malicious releases, immediately preventing anyone from continuing to install them.
You're just using Microsoft's Github as your package registry.
You have problem with trust, and you trust something that happens to talk git protocol, so you think git protocol solves trust. It doesn't. It's a very shitty protocol for downloading a package with a specific digest (your git client may not even be configured fsck the packfiles it receives!). All the package registries already have digests for the content they serve. The hard part is a discovery mechanism for trustworthy digests when a new version is released, and that's a big messy problem of trust and identity management (which seems simpler when you delegate it to Microsoft to manage). Fetching a git repo by a URL doesn't give you that!
git tags aren't even immutable. 3rd party hosts can even serve whatever they want, even different content to different users. Fetching from random domains directly is dangerous, because domain ownership can lapse at any time. If only somebody had a store of old hashes to enforce immutability and at least give you a globally-shared TOFU model for arbitrary URLs…
Docker has a similar issue, the tags are arbitrary and can be overwritten at any time, like "latest" is commonly retagged to new releases. It's just a social contract that publishers don't arbitrarily change tags from under us.