ash: a hybrid between getopts and 'sh -c'

Busybox is one of many systems that use ash. It’s the Almquist shell written for 4.3BSD-Net/2 as part of the effort to get rid of AT&T code. The Debian Almquist shell dash is a fork of it.

https://www.in-ulm.de/~mascheck/various/ash/

I haven’t written Golang in some time, but I remember I didn’t dislike it the last time I wrote it.

I liked how many third-party libraries there were for useful tasks (libraries that don’t readily exist in other languages, like userspace Wireguard or SSH servers,) the performance seemed adequate, and it was nice being able to produce a single binary for distribution.

However, in putting together the example above, I (again?) got the impression that parts of the Go standard library (like flag and template) just seem very sloppily designed. Why is that? Is it just me?

For anyone who is curious what while (( # )); do case "${1}" in *) ... ;; esac; shift; done-style parsing looks like, here is a consolidated example that shows:

• short & long arguments
• default argument values
• repeatable arguments (-v)
• arguments with settable or appendable parameters (-c, --see, -d)
• concatenated-style arguments (-a -b or -ab; e.g., tar -xavf ...)
• arguments with multiple parameters (-d abc 123; e.g., bwrap --dev-bind / / …)
• arbitrary argument placement ($0 -a target or $0 target -a; e.g., rm / -rf)
• long-style arguments with = parameters (e.g., head --lines=10)
• error handling

It should be obvious that -- separation (common for disambiguation; e.g., rm -- --file-with-dashes) is just a --) break ;; to exit the while (( # )) loop, followed by a (( # )) && shift && args+=( "${@}" ).

It should be obvious that defaults can be handled either in the parsing code or in the code that calls the parsing code.

It should also be obvious that if the command were broken into subcommands (e.g., git pull,) this would be implemented by composing multiple functions. e.g., case "${subcmd}" in ...) /:/parse/args/subcmd/. "${@}" ;; esac

It’s been a while since I last used zparseopts, but I am certain that some of the above features are simply not supported (e.g., multiple parameters.)

zparseopts (unlike Python argparse) does not give you usage text for free. Since zparseopts does not give you defaults, and since it would be possible to handle defaults within this parsing code (though there are reasons why you may want to handle it in the calling code,) and since zparseopts does not give you contextual help messages (i.e., not just what went wrong but why,) we might try to be fair and say that for every additional flag, the zparseopts approach requires 1+1=2 lines of code (one for the argument to zparseopts, ½ for the error handling, and ½ for the default,) and the below approach requires 2½ lines of code (one for the short-form, one for the long-form, and ½ for the default.) The below code has about 10-lines of boilerplate, whereas zparseopts has about two. That means the effort for zparseopts is total lines = 2×argument + 2 and the effort for the below approach is total lines = 2½×argument + 10.

But the below is also relatively obvious imperative code, so we must account for the difficulty of understanding a zparseopts invocation such as zparseopts -D -E -K -M -- f+:=_f -flag:=_flag … v::=verbose -verbose=verbose h=_help -help=_help. Thus, I argue that the below option is only marginally more boilerplate, minimally more conceptual effort, and all at no loss of generality.

#!/bin/zsh

/:/parse/args/.() {
    local _1 _iarg _aarg
    local -i a=0 b=0 f=0 g=0 verbose=0
    local -a c=() d=() args=()
    local -i help=0 error=0
    local -a message=()

    while (( # )); do
        case "${1}" in
            --*) case "${1}" in
                --verbose) verbose+=1 ;;
                --see)     c+=( "${2}" )
                           (( # < 2 )) && error+=1 message+=( "Missing arg: ${1}" ) || shift ;;
                --see=*)   _1="${1#--see=}"
                           c+=( "${_1}" )
                           [[ -z "${_1}" ]] && error+=1 message+=( 'Missing arg' ) ;;
                --dee)     d+=( "${2}" "${3}" )
                           (( # < 3 )) && error+=1 message+=( "Missing arg: ${1}") || shift 2 ;;
                --help)    help+=1 ;;
                --*)       error+=1 message+=( "Bad flag: ${1}" ) ;;
            esac ;;
            -*) for _1 ( "${(A@s..)1#-}" )
            case "${_1}" in
                a) a+=1 ;;
                b) b+=1 ;;
                c) c+=( "${2}" )
                   (( # < 2 )) && error+=1 message+=( "Missing arg: ${1}" ) || shift ;;
                f) f+=1 ;;
                g) g+=1 ;;
                h) help+=1 ;;
                v) verbose+=1 ;;
                *) error+=1 message+=( "Bad flag: ${1}" ) ;;
            esac ;;
            *) args+=( "${1}" ) && <<< "x=${(q)1}" >&2 ;;
        esac
        shift
    done

    for _iarg ( a b f g help error verbose )
        <<< "${_iarg}" <<< "${(q)"${(P)_iarg}"}"
    for _aarg ( c d args message )
        <<< "${_aarg}" <<< "${(q)"$(for x ( "${(@P)_aarg}" ) <<< "${(q)x}")"}"
}

##############################################################################
/:/util/center/. () { <<< "${(pl:40::\u2500:::r:40::\u2500:::)1}" }
/:/util/heading/.() { for ln ( "${@}" ) /:/util/center/. "${ln}" }
/:/util/scalar/.() {
    local -r name="${1}" ; shift
    local -r value="${1}" ; shift
    <<< "local    ${name}=${value}"
}
/:/util/array/.() {
    local _x
    local -r name="${1}" ; shift
    local -ar values=( "${@}" )
    if (( ${#values} )); then
        <<< "local -a ${name}=("
        for _x ( "${(@)values}" ) <<< "  ${(q-)_x}"
        <<< ") #⇒${#values}"
    else
        <<< "local -a ${name}=()"
    fi
}
/:/util/associative-array/.() {
    local _x _y
    local -r name="${1}" ; shift
    local -Ar values=( "${@}" )
    if (( ${#values} )); then
        <<< "local -A ${name}=("
        for _x _y ( "${(@kv)values}" ) <<< "  ${(q-)_x} ${(q-)_y}"
        <<< ") #⇒${#values}"
    else
        <<< "local -A ${name}=()"
    fi
}

##############################################################################
typeset -gar tests=(
    '-vvv --verbose'
    '-c abc --see def --see xyz'
    '--dee def 456'
    '-a -b -f -abf'
    '-h'
    '--help'
    '-c' # error: missing
    '-d abc' # error: missing
    '--does-not-exist' # error: bad
    # '-a -b -c abc --see def --see=xyz --dee jkl 123 --dee mno 456 PRQ STU -fg -vvv --verbose'
)
() {
    for args ( "${@}" ) {
        /:/util/heading/. "${args}"
        typeset -A config=( "${(@AfQ)"$(/:/parse/args/. "${(@AzQ)args}")"}" )
        for k ( "${(@ok)config}" )
            case "${k}" in
                c|args|message) /:/util/array/.             "${k}" "${(@AfQ)config[${k}]}" ;;
                d)              /:/util/associative-array/. "${k}" "${(@AfQ)config[${k}]}" ;;
                *)              /:/util/scalar/.            "${k}" "${config[${k}]}" ;;
            esac
    }
} "${(@)tests}"

Now, I am not fooling myself that the above is readable to most people who don’t write a lot of Shell scripts. There are definitely some extreme Zsh-isms present, and most will probably consider this code very dense for the typical Shell script. However, I use this as an example to illustrate the idea of eschewing simplistic argument handling libraries which typically rob you of too much generality.

In Python, the above would look only slightly different, given the better choices we have for modelling things in Python. Likely, since our goal is to encode an arbitrary state machine, we would model this as a generator coroutine.

e.g.,

#!/usr/bin/env python3

from collections import namedtuple
from dataclasses import dataclass
from functools import wraps
from itertools import chain, repeat
from types import SimpleNamespace
from typing import Callable, Generator
from sys import argv

Binding = namedtuple('Binding', 'path value')
Error = namedtuple('Error', 'message')
sentinel = namedtuple('Sentinel', '')()

@dataclass(frozen=True)
class Parser:
    coro : Generator[[Binding | Error]]

    def __call__(self, args):
        for arg in args:
            yield from self.coro.send(arg)
        while True:
            try:
                yield from self.coro.send(sentinel)
            except StopIteration:
                break

    @classmethod
    def from_coro(cls, coro):
        return wraps(coro)(lambda *a, **kw: [cls(ci := coro(*a, **kw)), next(ci)][0])

def take(n):
    rv = []
    for _ in range(n):
        rv.append((yield []))
    return rv

def binding_or_error(values, binding, error):
    if any(x is sentinel for x in values):
        return error(*values)
    else:
        return binding(*values)

@Parser.from_coro
def parser():
    class Flags:
        @staticmethod
        def c():
            return binding_or_error(
                (yield from take(1)),
                lambda v: Binding('c', v),
                lambda _: Error(f'missing argument for {flag}'),
            )
        @staticmethod
        def d():
            return binding_or_error(
                (yield from take(2)),
                lambda k, v: Binding('d', (k, v)),
                lambda *_: Error(f'missing argument for {flag}'),
            )

    rv = []
    while True:
        if (flag := (yield rv)) is sentinel:
            break
        rv.clear()
        if flag.startswith('--'):
            match flag.removeprefix('--'):
                case 'verbose': rv.append(Binding('verbose', True))
                case 'help': rv.append(Binding('help', True))
                case 'see': rv.append((yield from Flags.c()))
                case 'dee': rv.append((yield from Flags.d()))
                case _ if flag.startswith('--see='):
                    rv.append(Binding('c', flag.removeprefix('--see=')))
                case _: rv.append(Error(f"unknown flag {flag}"))
        elif flag.startswith('-'):
            for f in flag.removeprefix('-'):
                match f:
                    case 'a': rv.append(Binding('a', True))
                    case 'b': rv.append(Binding('b', True))
                    case 'f': rv.append(Binding('f', True))
                    case 'g': rv.append(Binding('f', True))
                    case 'v': rv.append(Binding('verbose', True))
                    case 'h': rv.append(Binding('help', True))
                    case 'c': rv.append((yield from Flags.c()))
                    case 'd': rv.append((yield from Flags.d()))
                    case _: rv.append(Error(f"unknown flag {flag}"))

if __name__ == '__main__':
    tests = [
        ['-vvv', '--verbose'],
        ['-c', 'abc', '--see', 'def', '--see', 'xyz'],
        ['--dee', 'def', '456'],
        ['-a', '-b', '-f', '-abf'],
        ['-h'],
        ['--help'],
        ['-c'], # error: missing
        ['-d'], # error: missing
        ['-d', 'abc'], # error: missing
        ['--does-not-exist'], # error: bad
        # ['-a', '-b', '-c', 'abc', '--see', 'def', '--see=xyz', '--dee', 'jkl', '123', '--dee', 'mno', '456', 'PRQ', 'STU', '-fg', '-vvv', '--verbose'],
    ]
    for t in tests:
        bindings = parser()(t)
        print(*bindings, sep='\n')

I’ve included just enough “helper” code to illustrate how we can “build our way back up” to an interface like argparse. The design of the above is, of course, just a sketch. If we were to try this in earnest, we would want to make sure that as we “build our way back up” to the declarative interface, we do so at no loss of composition and without burying the state machine/even loop again. Additionally, we want to figure out how to nicely bifurcate the parts of the problem which are so regular that we can encode the state machine explicitly (i.e., as distinct, first-class nodes and edges,) from those which are so irregular that we have to encode the state machine as a generator coroutine. In the case of the former, we would use the explicit modeling to automatically generate help text (which is something that argparse does for us very nicely,) and in the case of the latter, we would handle ad hoc modalities (something with argparse simply cannot do to my knowledge.)

From this perspective, we could consider these approaches as lying somewhere on a spectrum measuring the explicitness or implicitness of the modelling of the state machine. argparse and the like are clearly toward the explicit extreme; the above approach is toward the implicit extreme. It is my belief that, following the above path, we can eventually discover the ideal design for an argument parsing library, which I suspect would feature not only modelings at both ends of this spectrum, but mechanisms that nicely compose these formulations.

Hi! Author here – just wanted to say, thanks for the detailed feedback! I’m a bit crunched for time right now, but I’ll go through your comments properly in the next couple of days. If you’re into CLIs, believe it or not, I actually have another Go project aimed to make writing CLIs easier at https://github.com/avamsi/climate. It’s built on top of Cobra, so should better address some of the pain points you mentioned.

The author uses it this way in setting up one alias for jj:

	push = [
		'util', 'exec', '--', 'ash', '-r @- revision', '--remote', ''' \
			jj bookmark move \
				--from='heads(ancestors({{.r}}) & tracked_remote_bookmarks())' \
				--to={{.r}} \
			&& jj git push --revisions={{.r}} \
				{{if len .remote}}--remote={{.remote}}{{end}}
		''']