POSIX wordexp(3) is dangerous
71 points by yossarian
71 points by yossarian
I have seen this in libc, and I wonder why it is there at all. No shells use it to implement anything. Tilde expansion can be implemented with a much more straightforward algorithm.
And if it just forks a shell, yes that is incredibly dangerous !! (Not to mention making it impossible for shells to use it :-) )
I think the problem is that the specification of such a function involves the entire shell, because the entire shell language can appear inside command subs:
$ echo $( f() { echo 'defined function'; }; f; f && f; f | f )
defined function defined function defined function defined function
Nobody is going to implement an entire shell inside libc ... that would be crazy.
It seems like it should be removed from libc. Not sure who put it there, and why.
Yeah, there are quite a few POSIX APIs that directly or indirectly assume the presence of a full sh lexer, a full C lexer, etc. But very few (approachably) POSIX systems are built such that the sh or C lexer is reusable from libc, so we end up with these crazy hacks instead.
Not sure who put it there, and why.
The post I linked to has an interesting explanation (quoting):
It seems the problem was that people kept demanding more and more features for glob, so it seems like the committee threw up their hands and added the far-too-broad wordexp in attempt to cover all possible bases.
From: https://www.cipht.net/2023/11/21/worried-by-wordexp.html
Yeah, there are quite a few POSIX APIs that directly or indirectly assume the presence of a full sh lexer, a full C lexer, etc. But very few (approachably) POSIX systems are built such that the sh or C lexer is reusable from libc, so we end up with these crazy hacks instead.
At least nowadays there are C-frontend-as-libraries (i.e. libclang); I'm not aware of a POSIX shell that's usable as a library. The Unix distinction between executables and libraries (and lack of an omnipresent shell like DCL) is unfortunate.
I think the problem is that the specification of such a function involves the entire shell, because the entire shell language can appear inside command subs
FWIW, glibc uses the system shell for command substitution, but implements all of the other word expansion logic itself.
If I continue to have say in this, wordexp() will never be in OpenBSD's libc. Perhaps if there is one open source project which refuses to have it, coders will continue avoiding use of wordexp().
—Theo de Raadt (2010)
A few other things that make wordexp(3) dangerous:
You can’t use it in a multithreaded program, because it usually forks a shell.
On glibc wordexp is built in, but shell expansions that set variables call setenv() which is not thread safe, and which also modifies the caller’s environment which traditional forky wordexp() does not. (libc basically never calls setenv which is why this is a notable oddity.)
It really is a lunatic function that should be documented with a big “do not use this” health warning.
A while back I came to the same conclusion when researching wordexp, because we were having discussions about adding a wordexp:: module to the Hare standard library. I was extremely against it, both because it's impossible to use safely on untrusted input (as you noted), and also because pretty much no one fully implements it correctly: implementations like musl's are hacks where it's trivial to find stuff like shell bypasses, and implementations like glibc which don't use the system shell (except for command substitution) can't handle edge cases like $(case 0 in *) echo hi; esac)
The compromise we ended up on was to ignore POSIX altogether. This means that Hare's wordexp always uses the system shell (even if it isn't a POSIX shell), and, more importantly, we don't implement WRDE_NOCMD at all; we instead document that the interface is only suitable for trusted inputs:
Note that, by design, this module runs arbitrary shell commands from user-supplied inputs. It must only be used in a trusted environment.
IMO providing a flag like WRDE_NOCMD which doesn't actually make the interface safe to use is worse than providing nothing at all, since it gives a false illusion of security.
The main motivator for us to implement wordexp was for the upcoming hare.ini configuration format, to declare dependencies and linker flags and stuff (that way the user can use pkg-config, for example). Although wordexp wouldn't have been my first choice, I'm happy with the compromise we ended up with.
implementations like musl's are hacks where it's trivial to find stuff like shell bypasses
Yes I am going to start using the term "parser differential" for this. It is often used in the context of say web firewalls trying to parse HTML incorrectly and block stuff, but I see it extremely often in relation to shell -- simply because shell is hard to parse, and people write a second, crappy, and incomplete shell parser.
https://nhimg.org/glossary/parser-differential/
And even bash can't parse shell correctly in many cases, after 30 years of trying :-) That is one reason I started https://oils.pub/
I don't have the bugs handy, but simply parsing nested () {} $() ${} in shell is extremely difficult, and the unbalanced case 0 in *) is one reason, but not the only one.
Oh and now I remember the reason I was looking into parser differentials. I noticed that there were two cases of longstanding design bugs in the Go stdlib that can be thought of as parser differentials:
(1) One them was a motivation for JSON v2 - https://go.dev/blog/jsonv2-exp (Sept 2025)
e.g. two parsers handling {"admin": false, "admin": true} differently
An example (not involving Go) is https://nvd.nist.gov/vuln/detail/cve-2017-12635 - Due to differences in the Erlang-based JSON parser and JavaScript-based JSON parser
(2) The other parser differential is in contextual auto-escaping in Go's template packages
https://nvd.nist.gov/vuln/detail/cve-2023-24538 - Templates do not properly consider backticks (`) as Javascript string delimiters, and do not escape them as expected. Backticks are used, since ES6, for JS template literals
Basically Go has a crazy HTML/CSS/JS incomplete parser / state machine, in order to do automatic escaping. But the difference between that parser and the real one can lead to missed escaping of untrusted variables, and thus a vulnerability.
If shell and HTML and other "crappy languages" weren't already entrenched enough, then LLMs really put the nail in the coffin. We will be using shell and HTML "forever".
But maybe we can do better with LLMs ... i.e. nobody wants to do all this work to migrate toward stricter languages (e.g. OSH vs bash, which is easy in 99.9% of cases), but we really should.
With bash being as widespread as it is, by doing it "correctly" in oils, aren't you creating just another parser differential?
Digging through https://github.com/apple-oss-distributions/, it looks like standard utilities shipped with macOS actually use wordexp themselves here and there, though I imagine most of this is inherited from upstream sources. For example, it's used in the :e command in less. There's also some special-case handling in sudo, but this appears to be defensive in case a different program calls wordexp via sudo.
Honestly, even without command expansion, I think it's an iffy function. I went through different implementations a decade ago on Twitter, and many just shelled out directly (as other people here have mentioned). It doesn't seem safe to use in many circumstances.
https://www.openwall.com/lists/oss-security/2015/02/10/11 (caveat here is that this is from 2015, and clearly some implantations have changed since)
One person’s dangerous is another person’s useful ;)
i3’s “include” config file directive uses wordexp and the userguide includes examples like wildcard expansion (include ~/.config/i3/config.d/*.conf) and command substitution (include ~/.config/i3/hostname.conf): https://i3wm.org/docs/userguide.html#include
Using it for things like i3 config files where any attacker who can write to them can own you anyway is fine; the problem is that
So it'd be better if there was a separate function that was guaranteed to only parse tildes, basic $VAR substitution, and maybe glob expansion (ideally with knobs for controlling which of these), and it was required to be safe on untrusted input and not just secretly shell behind your back.
There is glob() which does glob expansion. Linux allows one to do tilde expansion but that's not POSIX. No mention of environment substitution though.
There's a bit of a footgun if you don't turn off features you don't need you get features you don't want. That's poor API design. It sounds like more of the issue is that there are some buggy libcs so that if you want shell expansion.
I'm certainly not eager to rely on a libc with lines of code that look like:
if (flags & WRDE_NOCMD) for (i=0; s[i]; i++) switch (s[i]) {