Brat, a parallel TAP testing harness for the POSIX shell
25 points by sstephenson
25 points by sstephenson
We use bats extensively at work for testing a CLI and I’ve noticed that there’s a significant overhead to running any test because of all of the file scanning, forking, execing that it does prior to even running tests. Do you have any performance comparisons on that front e.g. startup time?
The other thing that’s bitten us multiple times is how bats creates fd 3 for diagnostic output that isn’t captured. If you spawn a background program that accidentally inherits fd 3, bats will hang forever because it waits for fd 3 to close before exiting.
As it turns out I’m also the original author of Bats. I’m very grateful to the community that adopted it and steers it today. But I haven’t used it in many years, so treat my words accordingly :)
My unscientific response is that generally Brat does less work spread over more processes. Tests feel a little pokey when run serially, but fly by when you run them in parallel. On my M1 Pro laptop, the 89-test Brat suite takes 8 seconds to run serially, but that falls to 1.8 seconds with -j 12.
Another change that should affect test performance is that the run helper in Brat doesn’t read stdout and stderr into memory, it redirects their output to files and gives you access to the paths.
Finally, Brat doesn’t use fd 3. The commands that orchestrate the test run operate on a named pipe opened on fd 9, but that happens outside the process that actually runs a test case. Diagnostic output is handled entirely by xtrace (set -x).
Excellent! Thanks for both brat and bats, we're sitting on a pile of about 600 bats tests :)
Wow this looks nice. The code looks very cleanly written (for bash/awk - It's IMO still rather hard to grasp but I guess that's just how these dense 90ies shell languages are when they're properly used...). and it looks like a lot of thought went into the architecture!
I love that you went all in with POSIX!