Unit Tests Anti-Pattern: Leaky Mocks and Data

Unit tests should be isolated from each other. That means that it doesn’t matter if they run in any specific order, alone or in a group, we expect a consistent result. If there’s a reason for failure in the tests it should be a change in functionality.

However, things get tricky if the code, or the tests have dependencies that we can’t get rid of. The tests are no longer unit tests by definition, but if they are valuable, we want to keep them. We still need to take care of the leaks between tests. The anti-pattern is not doing so.

Leakage is not just a beginner problem. In large organizations, as we start testing bigger flows, rather than small classes. When “other people” start writing tests for areas of code that were initially written by “us”. Our assumptions of what clean up means, may no longer work, or worse, not known.

When we encounter these symptoms, they usually points to organization dysfunctions. Conway’s law works in mysterios ways.

Let’s take mocking for example. In isolated unit tests, we create the mock, configure it, and it dies at the end of the test. Then we create another one for the next test. The tests take care of the clean-up.

The “taking care of” is not really the handling of the tests. It’s just instance management – if we’re creating the mock inside a test, it will just die at the end of the tests. If we’re creating it in the @Before method (or equivalent depending on the framework), the instance used in the last test gets over-written in the current one. In a language like C++ without automatic memory management, we may still see the same affect, although the memory doesn’t just go away.

It seems that we get isolation for free, since we’re not doing anything with the mocks for that isolation we crave.

Don’t get used to those freebies

Let’s say we use Spring for dependency injection. Now, we don’t create the mocks manually. They just appear from thin air. We assume that isolation is also taken care of, out of thin air. But if the mock is injected once, it carries its expectations and behaviors from previous tests unless it gets reset (e.g. use reset() in Mockito at clean up time).

Mocks are not the only things that need leakage therapy.

As we’re testing flows with external data – database, cache, files, the registry – we need to make sure that data doesn’t leak to the next tests. We need to clean up after the test has completed.

Using Spring’s @Transactional in tests works, but maybe not enough. What about setup data? That need’s to be cleaned before moving to the next test.

And still that may not be good enough.

What if the test crashes mid-way? or has some unexpected behavior? We need to be sure we clean up correctly in every case. By the way, being sure is not that simple – we need to know how all the code behaves, from now until forever.

The other option is each test makes sure it cleans everything it needs before starting. That’s as simple as the other method.

Some of these things can be automated – base classes that contain these behaviors, clean up scripts, etc. It comes down to “how the teams works”. It assumes that the way we write code and tests is known and understood by everyone. This assumption is hardly true in most teams, and gets broken over large codebases.

When codebases break, so do the tests.

The successful path to overcome these problems is knowledge sharing and practice policies. Design reviews, pair programming, code reviews – the things that help to create a working development process. If everyone finds their own way to clean up the leaks, it pretty much guarantees that someone, somewhere will assume that “it should work like this”. Or worse – copy the method without understanding what stands behind it. And then they start noticing weird test behavir.

Leaks should be stopped, but defining how is just the first step. The next is making these methods commonly used.