The Unit Test Scope Creep

In “Should I use multiple asserts” we described the problem when using them in a single unit test. In the example I gave, while checking the result of a single operation, it still made sense to get information on both ends of the transaction. Debugging the unit test can unravel the issue, yet two separate unit test results shorten the road to the fix.

We are using asserts to lead us to the right problem to solve. But hey, we can do much worse.

We’ll write a different unit tests for checking the money transfer from the source account, but also add a check for the integrity of the account after the transfer.

@Test
public void getBalance_AfterTransfer_SrcAccountDecremented()
{
    bankAccount accountSrc = new bankAccount(5000);
    bankAccount accountTarget = new bankAccount(0);
		
    accountSrc.Transfer(500, accountTarget);

    assertTrue(accountSrc.isValid())
    assertEquals(accountSrc.getBalance(), 4500);;
}

Looks harmless. After a complex operation, the account’s state needs to be validated. Even more so, it’s part of the “unit of work” we’re testing, and the assertions are on the same object. So, there’s no apparent rule against not using two asserts in a single unit test.

What happens, months from now, when the unit test fails because the first assert fails?

We’ll see in the build report:  “getBalance_AfterTransfer_SrcAccountDecremented failed“. The reported reason would be that the source account wasn’t valid.

We don’t know the balance, because the unit test failed before that. It may be the correct balance.

It may even be that the account is not valid because of a side affect, something completely irrelevant to our balance check.

Now we are confused. Even misled.

We think the problem is with the balance, although it might not be. It will take us more time to solve the problem, than if there were two separate unit tests, with specific names.

This is “unit test scope creep”. We added a bit more scope to our unit test, and much like in feature scope creep, we got bitten.

The problem can even grow further. The more conditions we assert, we tend to generalize the unit test’s name. After all, if it checks all things in my workflow, why not call it “testWorkflow”? We lose the accuracy of the unit test. I

If we don’t need specific names, and match the assertion, we’re causing our future selves more work in the future.