What Are We Really Testing?

Every unit testing example starts with a calculator, so let’s do that today. Have a look at this code.

public int add(int x, int y) {
    return x + y;
}Code language: PHP (php)

Suspicious eh?

Okay, so how many tests do you think we need to cover this code?

“Cover” is pretty vague. How many as in “enough tests that you feel comfortable leaving your code with”?

I always ask this in my workshops (not just the unit testing ones). I get all kinds of answers, like 1+1, and 2+2, and big numbers and negative numbers, all good cases. We come up between 5-15 cases people are comfortable with.

Now think about it: Are these really good tests? What are they really testing, and are they useful?

This code is in Java (it can be in any language that has a + operator). Every test we would write for our function, including the cases we mentioned, would actually be testing the implementation of the “+” operator.

Not a big shock. Take for example:

return 1+1; Code language: JavaScript (javascript)

The real code that does the calculation, is deep inside the “+” operator. In fact it’s in the implicit “=” operator, and some memory action. But let’s call it “addition calculation”.

What happens when we add two very big numbers? We get an exception. Which is thrown by…

Same guys. The addition calculation guys. We can write tests for those case too, checking that an exception is thrown for the right big values.

Testing The Simple Stuff Can Be Complex

So, back to the original question: How many tests do you need to feel comfortable?

Tests are supposed to be useful. They will tell us when the expected behavior changes. When would these tests fail?

There are a couple of options.

In the first option, if we replace the operator (by mistake, of course), the test will tell us. For example if checked 4+2 before, we’d get a different result.

public int add(int x, int y) {
    return x / y;
}Code language: PHP (php)

But this is a calculator we’re talking about, and the chances of that happening in this kind of code, is pretty slim. We have a lot of hints telling us not to actually do that. Changing the code like that requires a certain… let’s say, courage.

What other option will fail the tests?

Maybe a bug in the + operator? But these operators are used by millions of people around the world, every day. If there was a bug in there, we’d probably hear about it.

In reality these tests will pass forever. And those are not good tests. They create within us false confidence, that the code is working. Tests that we haven’t seen fail, we trust less. Then, how does that reflect on our other tests? Lower trust in some tests means lower confidence overall.

Let’s get back to the original question. How many tests do we need to cover that code?

Maybe the answer is none, because if the code is that simple, we probably don’t need tests around it. When somebody goes into the code and changes it, making it more complex, then they will add tests. To guard against the complexity that they introduce.

What if I was doing it in TDD, in Test-Driven Development? I would write the test first, before I wrote the code, that means I would probably end up with a single test.

Same Code, Different Package

Let’s make it more interesting. Now, the same code does not run in a function. Instead, it runs inside an API.

This is Spring-based implementation, of a POST that adds two numbers.

@PostMapping("/add")
public CalculationResult add(@RequestBody CalcRequest request) {
    int sum = request.getNumber1() + request.getNumber2();
    return new CalculationResult(sum);
}Code language: PHP (php)

Do we stick to the “almost zero tests” answer?

On one hand, the code is the same.

But an API test will run not just our code. It will run a whole lot of Spring code. Accepting the request, returning the response, translating JSONs to objects. And don’t get me started on the non-handled exception.

This time other code, a lot more than our code, can fail the test. It can also fail because of configuration. For example, if the server is not running, the test will fail, through no fault of the code.

The question is not about “coverage”. It’s about what we want to learn from the test.

If we’re only interested in OUR code, we get back to the answer before. But this time, it comes with a price. We assume that any failure of the API test has nothing to do with code changes, and therefore, requires no debugging, no retrying. Failures can be ignored. Tests are valuable only when they are passing.

Most people can’t live like that.

The other option is consider the test to verify everything, including our code. Failure means there’s something wrong somewhere, and we spend time on debugging, reading logs and fixing configurations, perhaps with no value at all.

Most people live like that very well. Unfortunately.

When we’re testing, we need to understand what we’re testing, what would make the test pass, and what would make them fail.

Our tests are designed to give us information. If we don’t think about that before writing them, we’ll be writing not-so-useful tests, and spend a lot of not valuable time on them.

We should know what we’re testing, and what we want from the tests.

And that’s why you should go to my “Unit Testing and TDD” workshop.