The 4 Attributes of Testability : Observability

Testability is in the eye of the developer. And the tester. The one who’s doing the testing. When we’re talking about observability, the eye matters.

As we’ve already said, everything is testable, it’s just a matter of motivation, effort and skill. That means we’re on a spectrum. And we’d like to be on “more testable” part.

Testability depends on many things, but we can divide it to:

1. Operability – How easily we can call the tested code
2. Observability – How easily we can observe the result or effect of the operation
3. Controllability – How easily we can control the conditions of the test
4. Reproducibility – How easily we can make sure the code executes repeatably

Today, its observability time.

Testability in the Eye of the Observer

When we’re testing, we’re looking at something. It is usually the result of an operation. It could be an external operation (like calling an API), or an internal one (timed polling).

When the operation takes place, we’re expecting to notice its result, or effect, easily. We usually have options on the noticing part, but as with anything with testing, we’d like to choose the easier ones, and more importantly, the ones that build our confidence more.

Since we’ve already used this diagram for the first part, let’s reuse . Check out this diagram:

Still amazing, right?

Ok, first, the same things we talked about in Operability still apply to Observability. If I want to check the result of the tested code, it will be easier (and more accurate) the more I get close to it.

If the operation returns a result, I can unit test it and compare the result to an expected value. If an API calls that operation, I hope it returns the same result.

But here’s the difference from Operability. Sometimes we don’t have the result directly accessible.

Let’s say, our tested code stores something in the DB. Something like this:

Now, if our tested code stored that something, how would we know the operation worked?

That requires a definition of what “works” means. It can be one or (hopefully) more than the following indicators:

• It returned a success code, or didn’t throw an error.
• It gave a direct response that confirms it got the right values.
• It can be asked to retrieve that exact data it stored successfully.
• That data can be used in following operations correctly.

Let’s take an example, the POST API that called the tested code creates a new user in the system. So it can:

• Return a 201 (“created”) code
• Return an ID of the new User
• Allow a follow up GET with that ID to get the data so we can observe it
• Allow a follow up GET for all Users to show that data

The more of those we have, we can “test easily” more. The less – not so much. That means we can write or perform the test, and make a correct observation on the result.

Now, let’s say it only does the first one. What can we do to check it worked?

We May Need a Bigger Telescope

Without observing the returned data, I’ll need to find indicators the operation worked. For example, I can look inside the database, and look for the data. Or I can use a sniffer or read logs.

For that I’ll need skills (how to find the data, and how to understand its stored correctly), tools, time and motivation. And if I’m low on these, I will probably skip testing.

If it’s easy to observe, it’s easy to test. The trick is, we need to design the system to expose the information for testabilty, because after that the cost rises. Sometimes, we get this for free (e.g. the GET API is there already, or planned to be developed). Sometimes, we need to make sure it’s going to be there.

That’s observability. Next time, controllability.

Check out the my API testing and Clean Code workshops where I discuss testability: