Memory management in 40 major C projects, a survey
59 points by eatonphil
59 points by eatonphil
Good but under analysed. It looks like programming languages mostly are using garbage collectors or reference counting; regions seem to used by postgres, and forks and extensions, and https and nginx. The rest are mostly doing it raw. There’s more to this story.
I was just finishing up a big move (NYC to Durham) last week and ran out of time to publish anything deeper. I will probably return back to this eventually. I'd also welcome (and pay) anyone else come to do a second pass on the topic for The Consensus.
Tell us more about the apis you used for this? Are they free?
As with most things on the site, I initially built it to power research for articles.
It offers exhaustive search (which few code search systems seem to offer) over all commits in the last 10 years for all projects The Consensus indexes (basically, major software infrastructure projects; there are ~400 right now and that will grow over time). It's one of the most expensive things on the site to build (only ~150 projects can be searched right now, indexing are still being built for the rest after I moved to a second version of the indexes) and host so that's why it remains a paid feature.
(The first version of the indexes only supported a few tagged releases. But being able to search all commits quickly became obviously necessary.)
Doing it only for projects The Consensus picks is good and bad. It's bad because it doesn't index the world. But mostly it's good, if you trust our (my) editorial judgement, because all your searches only go against important (and I try to be expansive here) projects. Which makes it sort of more tenable to do meaningful exhaustive search. Unlike GitHub you don't have to pick which projects to search for, since we've picked them for you. (And if anyone asks I'd probably add whatever projects we're missing, as happens over time anyway.)
You can try out the code search yourself (a few queries a day) without an account but the API access is for Standard (paid) subscribers.
I think this is an interesting subject, but it seems a bit thin on the methodology? Is it basically grep and a human classifies what style of memory management it is?
Basically yes. Part of the goal was to try out The Consensus's code search APIs (which I still need to write about). But also someone literally asked the question on Twitter: what do C projects actually do for memory management. And some people were particularly surprised about SQLite not picking a "safer" path.
i remember an old article that covered the many memory allocation strategies just for Nethack. it is a fun read.
The analysis looks lime regions work well for things that are server-like, while reference counting and garbage collection are popular among programming languages. I don’t find this surprising, but it’s still nice to see it show up in the survey.
One thing to potentially take into account: I think a common pattern in software written in C is to have subsystems that more or less manage their own memory (say, after being provided a region when they are initialized), and employ strategies which are specific to their domain. In an extreme case, this could lead to counting a single malloc plus free, despite there being many calls to the subsystem’s create and destroy functions.
Not sure if this changes the picture much but I am curious if this was taken into account or is part of the intended analysis at all
Potentially related: "Dynamic Storage Allocation: A Survey and Critical Review" by Paul R. Wilson et al. https://doi.org/10.1007/3-540-60368-9_19