Better Batteries
50 points by olex
50 points by olex
This is the wrong question to ask. The right one is:
Which social architecture creates a high-quality standard library?
This is an extremely helpful framing, and I enjoyed the post.
I do think the availability of package managers have changed the dynamic. Python predates npm-style managers and Go has iterated on its packaging approach in recent years.
I like to think about how do we enable a single enthusiast in a domain to ship a reusable solution? Most small-medium OSS projects have a single maintainer.
Allowing our enthusiast to iterate on the API (stdlib breaking changes are hard) and easily release feels like the best option in a modern language toolchain.
I feel like
how do we enable a single enthusiast in a domain to ship a reusable solution
is, at best, half the problem. It is an important thing to have, agreed!
But then there's the question of: how does a community ensure continuity of a solution? What happens if a maintainer of a widely-used library signs off permanently, or abuses their upstream access to other projects, or...?
A lot of individual maintainers write about unreasonable asks from users. I'd hate to be putting someone in that position, where I'm relying on just their work.
Solutions that step up from "an individual's project" to "a group project"- either standard library, or a designated group, a la Tokio/Bytecode/Zig- indicate to me a committment to continuity and maintenance, and a framework to contribute other than "tip the maintainer". Even if the group is BDFN-style (a la Zig), there's still some separation of "the project" and "the person" that can in principle be held to.
So coming back to- yes, "enable individuals to share solutions"- but also, "how do we support those solutions being adopted by the community?" What is the path to inclusion in the standard library, or creating associated orgs for maintenance (ideally with appropriate financial support)?
Not only that, some packaging "systems" also just hail from a time where language releases were slow and people mostly manually downloaded packages.
PEAR for PHP was basically that. From complete chaos came a sort of organization with lots of batteries-not-included but quasi default choices and releases were a lot easier to cut than of the language runtime itself. By PHP 5 and Composer this had mostly faded, as far as problems go, composer was good and the language was evolving more quickly and with more people involved. (ignoring innate problems like the dreaded needle-haystack order of arguments here)
Yes! I've never been against languages with batteries included. In fact nowadays I prefer them. It's one of my biggest issues with Rust, but the problem also exists in Haskell. Funnily enough, in Node.js, which gave birth to npm, you can now do many stuff without external dependencies (last one I've seen are the sqlite functions).
But yes, in languages with batteries included sometimes they feel like an afterthought. It's not just Python. Java also used to have a big standard library, with many useful things including GUI programming which has been neglected a little bit.
To flip this a bit, we can consider the cost of maintaining and breaking APIs in a big standard library: we remove costs from users and move them to the maintainers. We buy stable APIs that allow interoperation across the ecosystem.
Even with infinite money, having an std crate for crypto is probably only going to stifle innovation (==go), because almost nobody needs interoperability of crypto APIs.
Having a terrible hashmap will similarly cause fragmentation because a lot of people need to build APIs around it (==cpp), having good std allocator first class will force the users to support allocator interoperability (==zig).
Of course, there are a lot of grey areas (errors in Rust is a known example in both directions), but IMO a small std is only a problem in low trust society, so improving trust is much more important that increasing the size of the std.
Even with infinite money, having an std crate for crypto is probably only going to stifle innovation (==go), because almost nobody needs interoperability of crypto APIs.
Can you expand on this? I'm not sure I understand. I wouldn't have thought of crypto as benefiting from a ton of innovation: new algorithms come around relatively rarely.
Crypto also seems like a particularly good thing to have in the standard library. It requires expert implementers, trust is super important, and you want as many eyes on an implementation as possible.
Rust is an interesting case. The 1.0 standard library APIs are brilliant. Collections and iterators are a work of art.
I agree about the collections and iterators. I think the things that ended up exclusive to std rather than core or alloc are more uneven. A few months ago, around the Ubuntu 26.04 LTS release that introduced uutils to an LTS distro, wasn't there some criticism of the Rust standard library's filesystem API?
The criticism was that the stdlib filesystem API didn't support some way to have race-free filesystem traversal and inspection (via e.g. opening a directory fd and then accessing its children relative to that fd, or opening a file fd and then performing operations on that instead of the path), but the underlying APIs for that are a bit janky, and most other languages don't support it either. Zig kinda does, for APIs where the host OS supports it (but I don't believe it supports any cases where you'd need additional userspace checks on top ootb, though I also don't think the issue uutils hit would have required this), and Go does support this well nowadays with os.Root (but that was only added last year). In Rust, you need an external crate like rustix to at least get to the equivalent of what Zig offers, or cap-std if you want something more thorough akin to Go's os.Root.
EDIT: I should mention that it's not just about being race-free, it's also about covering things like unexpected symlink traversals or relative path traversals (via things like RESOLVE_BENEATH and RESOLVE_IN_ROOT). For whatever reason, problems with races stand out much more in my mind than these other cases do.
What I take from this is that, for all of the popularity of the Unix-ish filesystem abstraction, typical implementations of that abstraction actually don't meet Rust's standard of robustness. Yet the Rust team decided, understandably, that a general-purpose programming language couldn't ship a standard library without filesystem support. I actually think that putting OS filesystem support in an external package like, say, an HTTP client or random number generation would have been a reasonable move. Unlike with interpreted languages, it's not like the standard library had to have certain functionality in order to bootstrap the installation of other libraries.
a technical rather than social problem is what support the language provides for interfaces and polymorphism. if you can provide drop-in replacements for stdlib code the cost of baking in a suboptimal solution goes down a lot (though not to zero of course; in particular it makes coming up with a better interface a lot harder)
I think that one part of why the Go standard library has been successful at integrating new changes is that the module system makes all packages feel effectively the same when using them in code. From a programmer's perspective, after go geting a package, it's just as easy to use that as a standard library package.
This means that the barrier to trying out experimental packages is very low, and then if it eventually lands in the official standard library, then the migration from something like:
import "golang.org/x/crypto/ed25519"
to
import "crypto/ed25519"
is trivial.