Integration tests are best tests
26 points by carlana
26 points by carlana
I have worked on a few systems which favour integration over unit tests, and my experience is that as time goes on, the number of tests grows, and the total run time of the test suite increases dramatically to a level where it isn't reasonable to run all of the tests often. This causes developers to find workarounds around the test suite such as avoiding to test certain behaviours, or making the test job optional on the pipeline.
That is not to say you should always favour unit tests. I find that developers tend to talk past each other regarding their test suites because it is hard to convey concisely the experience they have and that explains the choices and trade-offs they now prefer to make. I do agree with the article with the downsides of unit testing (especially mocking, which is something I now avoid to do, preferring other kinds of test doubles), but I think the article largely ignores the trade-offs one must make when implementing their testing strategy. Regarding this point, I do enjoy the article SMURF: Beyond the Test Pyramid, from a Testing on the Toilet episode.
I prefer integration tests over unit tests as they tend to require less changes over time. I also experienced the total run time going up at $PREVIOUS_JOB, so I rewrote it to allow one to selectively run tests. This let us concentrate on the immediate changes, then once the new tests were passing (or targeting existing tests relating to the code being changed), run the entire test suite. That was a decent trade off.
Another story about $PREVIOUS_JOB and integration tests---at one point the integration tests went from less than three minutes for a full run (around 17,000 tests) to about 15 minutes. It took some serious digging but it did reveal a few performance issues with both the test harness and the program being tested. Once fixed, the integration tests once again fell under three minutes.
I agree completely with the author here: unit tests just aren't worth the trouble.
I've actually never liked writing tests. I want to spend as little time as possible writing them. (That's probably a popular opinion, though not often voiced.)
I think this is, unfortunately, a pretty popular opinion these days, and I personally blame on unit tests. Unit tests were (and sometimes still are) pretty popular and recommended and a lot of people bought into the idea. But constantly fixing unit tests when you change things just sucks and I bet it killed a lot of interest in testing altogether.
Unit tests make sense when changing things should not break things. Protocols, algorithms... But testing a unit in a path through layers of business logic doesn't make much sense.
That's a cool little testing framework. Seems similar to bats, except they implemented it themselves to avoid the dependency.
I agree with the author overall. My only point of difference is that I cast a slightly wider net for what I consider "foundational libraries".
About half of the time I find that a new pattern*, new category of feature, or new major concern, will produce some amount of core, shared code. I treat those bits that like a library, even if they aren't named as such, or aren't isolated in a tidy little place of their own.
I mostly identify those by two factors:
CommonMultiStepForm or providesPermissions(), see it used in other code, might they assume that all promises its interface implies can be relied on? The more I suspect people might blindly trust it, the more I want unit tests.*Not pattern as in "design pattern", but just an emergent, repeated abstraction.
I don't think this mentions it, but it's very similar to the old concept of the testing trophy: https://www.tbray.org/ongoing/When/202x/2021/05/15/Testing-in-2021
My position is that for some particularly logic heavy codebases, you will have many small focused tests verifying algorithms or parsers; for some particularly poorly maintained legacy codebases, you may need a lot of E2E tests to validate your ball of mud; but for most other codebases, the advice in OOP and the above generally applies. Use integration tests, use fakes over mocks, etc.
I mostly build web services with server side rendered html. Nowadays i put a switch into the backend, if a client demands json via the "Accept" http header, it delivers the template data as json. This way i can easily write integration tests for the backend by validating the returned json.