Test Design – What Are We Testing?

Today we’ll talk about something we think we’re doing, but we don’t really, because like many things, we think we know what we’re doing. Or rather, we’re sure we’ll figure it out. Or, it will be ok. Test design is one of those “phases” we don’t formally go through. Instead, we’re off to write the test. We really should understand more before we touch the keyboard.

You can watch some API-related test design ideas in my “API Test Design” talk, but I want to focus on another aspect of test design today. Before we figure out what kind of tests to write, we need to understand what we’re testing. And what can go wrong.

Test design starts with the code

For starters, let’s look at a unit test and its code:

@Test
public void add_test() {
    Calculator calc = new Calculator();
    int result =calc.add(1,2);
    assertThat(result, is(3));
}

and the code that it tests:

public int add(int arg1, int arg2) {
    return arg1+arg2;
}

I know, your world has changed. Anyway, what is the system-under-test here? The add function, of course. Nothing (as far as we know compilers) can impact the result. No unforeseen side effects. Let’s look at something a bit more interesting.

public void add_with_memory() {
    Calculator calc = new Calculator();
    calc.store(1);
    int result =calc.add_to_memory(2);
    assertThat(result, is(3));
}

And the code:

int memory;
public void store(int arg) {
    memory = arg;
}
public int add_to_memory(int arg) {
    return arg + memory;
}
 
 

What is the system-under-test now? It’s the functions and the memory variable. Yes, the scope got a bit larger, but it doesn’t look like the test is at risk much. Although, we have introduced an opening for something else to effect our tests. For example, if we introduce another API:

public reset_memory() {
    memory = 0;
}

Our tested functionality can change by calling, or not calling, the reset_memory method.
But we’re still not bothered by that. Because we know in Java, and JUnit, each tests gets its own instance of the Calculator class, and we still trust compilers, so we’re sure there won’t be any problems. Each Calculator has its own memory, and therefore steps will not step on each other. And since we’re in control, writing the tests, we can’t get any safer than that. Right?

And then it got complicated

Ok, what if we do this? We change the memory field to be static.

public static int memory;Code language: PHP (php)

Ah, things are not as they were before. Since memory is now static, it can be changed by something outside the test, and the tested path. Like, by another test that ran before it, and left it in another state. In real applications, it can be something buried deep in the code, when some internal class, completely outside our visible class, will accesses static, global or shared data.

The system we’re testing got bigger, although our test and code didn’t. Test results can be affected by things outside our perceived system boundaries.

And it’s not just the code

It looks like a unit test issue, but it really isn’t. So far we’ve looked at how code impacts the system. But architecture and design can play a part here as well.

Let’s say our add code isn’t much more complex, but it’s sitting behind a REST API, and the memory isn’t just stored in-memory, but instead is stored in a database. It would be basically the same code, but because of the REST architecture (web server, database, separate hosts), we should now take into consideration delays and timeouts. Or multiple parallel operations, like a test calling a Reset API at the same time as another test runs, that doesn’t call Reset at all, will be impacted.

Test Design leads to trusted tests

Many more things can now impact the result of our tests. Yes, it may look like we’re testing a small chunk of the system. But we need to understand what can impact the results. Tests failing because of our misunderstanding is bad. How about passing tests that shouldn’t be passing, though? What’s worse?

This all goes into the basics of test design. What kind of test, what dependencies to set up, how to block interruptions – we need to think about all those considerations, so the results that we see mean what we need them to. Once we know where they system’s boundaries are, what can be impacted and what is safe, the result will be tests that we can trust. But that means thinking before rushing to typing.

Yes, for every test. You’re welcome.

If you want to learn more about Test Design, check out my Unit Testing and API Testing workshop where we go in heavy about how to test code and systems.