What’s In The Box?

What test types do we have? Unit tests. API tests. Component tests. E2E tests.

We have all kinds. The names can be a bit confusing, though.
Like, what is really a unit? Is it a method, or a class? Maybe more than two classes?

When are we crossing into component tests land? Is it like integration tests? Is it like a Venn diagram?

Maybe integration is between services. Or subsystem.s

Let’s take something simpler. API tests – they test APIs, right? Can’t get more precise than that. So here’s something to confuse you.

Let’s say this is our system. The client talks to the server using plain ol’ APIs.

An API test looks like this:

This test starts at the API, “in front” of the server, and invokes the code all the way to the database. That’s the whole white line path, our “coverage”.

But that’s just one option.

The next test, for the same API. Only instead of the real database like before, the test sets up a mocked response from the database. Obviously, the real database does not participate. The white line of coverage does not extended further.

Another option. Same test, for the same API, but this time, part of the logic is mocked, and nothing goes outside the server. The coverage is partial and only in the server.

Finally, since everyone’s missing the database, here’s another one for you:

Same API, but instead of the real database server, we replace it with an in-memoty database, for performance reasons. Real database, but running in the same process space as the server.

Are all these API tests? When I say “here’s my API test”, which one of the options above are you thinking of? Do we have the same one in mind?

Web tests have joined the conversation

Web and mobile tests suffer from the same identity crisis. Here’s another way we can say something that can (and will be) misinterpreted.

Let’s say, we’re testing a workflow of “checking the items in the store” in our web app. Innocent enough. But what do we really mean?

We can run our web tests with a workflow that starts with logging into the system, and authenticating the user, and then navigating to the store page and checking the items.

Or, we can run the same workflow, but without going through the authentication steps. Although I may be seeing different items in the store, as a “guest user”, and probably checking differnt things, the name remains the same. The coverage is different.

We’re using the same interface, and the same tools, and even the same features. Then we’re using the same names for our tests, although we’re running them in different contexts.

The test is in another castle

Let’s get back to test types. Using the same test type name, can lead us to directions we didn’t consider. Like test organization – where we group all tests of the same kind in one place. For example, all API tests go into the same folder.

When life was simple, we just threw tests into buckets based on their technical attributes. Like, the piece of code they exercised, or how we operated that code. But now, we create folders for unit, and API, and Web tests. Then we separate by functionality or structure, or a combination. And that’s probably not the only level of details.

We still refer to them with the same name, because they go in the same box. Although, the tests may be different in workflow, context and meaning.

Can we change that? Maybe group differently?

One way is to get more boxes. We’d have more accurate names. We can build semantics into our organization tree (sometimes tree is not good enough either). Then we just need to make sure everybody uses the same semantics.

Another way is to organize our tests based on completely different methods. For example, group the tests by the way they run. We want to run all the quick tests together, regardless of how they run. Then we can run all slow tests. And we can create groups by time scale.

How about grouping by shared context? All tests that require the same setup in the databsae, can be grouped together. Others that require a different setup, in another group.

There Can Be More Than One

There are different ways to group tests, other than what they call. Or what we call them.

Tests need to serve us. Technical separation looks like it might help, but if it leads to ineffective use, it is just waste.

Naming is hard. We’re trying to find distinction, and technical distinction is easy. However, it’s both ambigous, and ineffective in other ways.

We need to find what is important for us, then implement the separation. Not easy, but once you’ve got your system, and everyone understands it, there will be less errors, and less wasted time, looking for that missing test.

You know, the one we’re sure we put in this box here.

Want even more boxes? We discuss this in my “API Automation Testing For Developers” workshop.