Error handling: Checking for exceptions

When we talked about understanding what we’re actually testing in error handling, we talked about the ability to understand the behavior of our code-under-test . Just to recall, here are the three types of error implementation, and what we can check:

• No error handling – Any exception is handled somewhere else, and therefore we don’t need a unit test.
• Exception translation – The exception is caught by our code, but instead of the original exception, the code either modifies or translates it, so that someone else will have more factual information to explore. In that case we’ll assert the thrown exception.
• Error handling – The exception caught causes a change in behavior to compensate for the error. In this case, we’ll check for that change and assert it has taken place.

The first and last cases are trivial (well, nothing new here anyway – either no test or a regular test). The second one is more interesting. Let’s see how we check for thrown exceptions.

Most modern test frameworks have an API for asserting an exception was thrown. Let’s see how to test an exception was thrown. Continuing with the example from last time, we’ll check that if the guest list was null, our GuestTracker throws an application specific exception.

[Test]
public void ThrowUninitializedException_WhenUninitialized()
{
    GuestTracker tracker = new GuestTracker();

    Assert.Throws<UninitializedTrackerException>(()=> { tracker.AddGuests(3); });
}

Assert.Throws checks that a type-specific exception should be thrown when the AddGuests method gets called. Let’s unwrap it and see what’s under the hood:

[Test]
public void ThrowUninitializedException_WhenUninitialized_TryCatchPattern()
{
    GuestTracker tracker = new GuestTracker();
    try
    {
        tracker.AddGuests(3);
        Assert.Fail();
    }
    catch (UninitializedTrackerException e)
    {
        Assert.Pass();
    }
    catch (Exception e)
    {
        Assert.Fail();
    }
}

More than meets the eye, isn’t it?

The first catch is there to make sure that the error is caught. But there are two more Assert.Fails there to make sure there are no false positives (e.g. The test doesn’t pass by accident).

The first one, inside the try clause, will makes sure that the test fails if no exception was thrown. The second one, in the general catch clause, makes sure the test fails if another type of exception was thrown.

If we’d like to explore the exception information, there are additional APIs. In NUnit, xUnit and JUnit (in the upcoming version 5) Assert.Throws or its equivalents, return the exception object that got thrown, and you can assert on it.

In our case, if we wanted to check not only the right exception but also its message, it would translate to:

[Test]
public void ThrowUninitializedException_WhenUninitialized_TryCatchPattern()
{
    GuestTracker tracker = new GuestTracker();
    try
    {
        tracker.AddGuests(3);
        Assert.Fail();
    }
    catch (UninitializedTrackerException e)
    {
        Assert.Equals("Guest list was not initialized", e.Message);
    }
    catch (Exception e)
    {
        Assert.Fail();
    }
}

A few exceptions, pardon the pun

There are a couple of cases and uses that throw us off from the regular course.

The MSTest or JUnit 4, ExpectedException mechanism is a bit counter to the Arrange-Act-Assert test structure, because we declare the exception assertions up-front, before we execute the throwing code.  (Technically, also using Assert.Throws convolutes the AAA too, but at least it’s in the same line).

In both you’ll need to write your separate assertions. In MSTest it will look like this:

[TestMethod]
[ExpectedException(typeof(UninitializedTrackerException))]
public void ThrowUninitializedTrackerException_WhenUninitialized()
{
    GuestTracker tracker = new GuestTracker();
    try
    {
        tracker.AddGuests(3);
    }
    catch (Exception e)
    {
        Assert.AreEqual("Guest list was not initialized", e.Message);
        throw;
    }
}

We need to provide the assert and re-throw the exception to satisfy the type checking. JUnit 4 provides better APIs for the checking the message and cause with Hamcrest matchers, but the code is still convoluted. The new APIs are much better, so go for the updated frameworks. Or just download an extension someone already wrote.

Where there are no typed exceptions, like in javascript, you have helpers (like in QUnit raises or Jasmine’s toThrow), since there’s only Error to deal with. That makes them easier to test.

The final case I want to mention is in C/C++. Modern GoogleTest, Catch and others have the equivalents of ASSERT_THROW that are capable to catch access violations / segmentation faults. Try not to use frameworks that break when they encounter such failure.