What Bug Will It Find?

“Is this a good unit test?”

Usually the question refers to a test that somebody has already written. But are these unit tests effective?

Should we have written it in the first place?

That’s right, we don’t need to test (unit tests or integration tests) everything. There are a lot of examples (e.g., unit testing getters/setters or data mapping) of those. Looking at them, we smell they are not that valuable, and in fact these unit tests are more of a liability. For example, they enforce rewriting on interface change.

Are these unit tests valuable? For that, we should ask a different question:

“If this unit test fails, what bug will it find?”

Or in other words, what is the risk of not having that unit test in place?

For my first example, I’ll call the immortal Calculator. Mr. Calculator has a complex Add function that takes two numbers and adds them.

[TestMethod]
public void Add_2And4_Returns6()
{
    Assert.AreEqual(6, Calculator.Add(2, 4));
}

Now, if our unit test for the Add function fail, that is one broken main functionality. Of course, we want to unit test it.

My next witness will be the SafeDiv function. Of course, we won’t just let the calculator divide by zero. Instead, we’ll throw an exception:

public static int SafeDiv(int a, int b)
{
    if (b == 0)
        throw new Exception("b is zero, you idiot!");
    return a / b;
}

Apart from the unit test that checks the division, we can write a unit test for the error handling function:

[TestMethod]
public void SafeDiv_ThrowsIdiotOnDivisionByZero()
{
    try
    {
        Calculator.SafeDiv(3, 0);
        Assert.Fail("Didn't call you an idiot");
    }
    catch (Exception e)
    {
        Assert.IsTrue(e.Message.Contains("idiot"));
    }
}

Is the exception throwing important? Maybe. We expect an exception to be thrown anyway.

Is the message we’re asserting on important? Sometimes it is, for user experience reasons. And the experience we’re supplying here is not that dramatic (depending on how sensitive you are).

In other words: If this unit test fails in the future, it will be because the message was incorrect. That would be the bug it finds.

And to tell the truth, this is usually not a very important bug to fix. Therefore, the unit test we just wrote is not that valuable, and may become a liability in the future. The risk of not having it is very low.

When we write unit tests, we shouldn’t assert everything. We shouldn’t verify all possible calls.

We should think what bug we expect the unit test to help us uncover. If that’s not an important bug, don’t write the test.

Bonus tip: You can tell if that’s a valuable unit test just by writing the test name. It will tell you in advance, if you’re wasting your time or not.

Second bonus tip: This doesn’t work with integration tests, or longer flows. Integration tests can fail for multiple reasons, including those outside our code. The reason we write integration tests or end-to-end tests is to sleep better at night that flows are working the way we expect them.