Let’s talk about the testing pyramid. And since it’s a pyramid, I’ll also throw in a tip (see what I did there?)
You may have seen the test pyramid with different steps,or levels, in it.
It used to be called the “Automatic test pyramid”, but you might have seen it with “Manual” tests on top, like this:
The Test Pyramid is a model. It models the number of tests of each kind we should have. Unit, Component, API, UI. It can have other types of tests too.
What is the reasoning behind this model?
The number of tests in each level correlates to cost: The cost of writing, debugging and fixing. Unit tests are quicker, cheaper and easy to debug. End-to-end tests – less so. So, according to the model, we want more unit tests.
It was discovered, long ago, that our model, has a mirror-universe equivalent. Or an upside-down equivalent, depending on your favorite show.
The upside-down pyramid (or the ice-cone), is usually discussed as an anti-pattern. A lot of effort goes into end-to-end tests, a lot less into unit tests. The funny thing, is you’ll see this anti-pattern in many organizations. I mean, what’s wrong with all these people, don’t they understand that they are part of a pyramid scheme?
Upside-down testing
There are a couple of reasons people give for going to the upside-down . It’s the cost of developers (“they should be writing code, not tests”), and the pushback (“developers don’t see that as part of their job”), and also, the role of testers (“they should focus on finding bugs in the systems”).
But, it really comes down to the purpose of tests. Passing tests give us confidence. Actually, passing end-to-end tests, give us confidence in the product. We don’t perceive passing unit tests as valuable enough to build confidence.
In other words, we look at the perceived value of tests, compared to their perceived cost. And value expensive tests seem valuable more than the cheap kind.
Does that mean we all need to build upside-down pyramids?
Maybe the old model has some life in it still. Maybe not. That’s not the issue. The real question is which model is right – for you?
Let me give you an example of where the regular pyramid is not a good choice.
Let’s say you write simple CRUD applications. Just data flowing to, and from the database. In this case, there’s is not a lot of business logic in the code. And business logic is the first candidate for unit tests, so why invest there?
Unit tests won’t help here. If there’s no logic, it means we can write unit tests, pretty easily, and they will always pass. Unit tests will call the mocked CRUD APIs, returning what we want. Easy to write, but they don’t provide confidence that the product does what it needs to do. They also don’t provide a safety net, because what can fail here? Cheap to write, no value.
We can write API tests, with the real API code being run, but with a mocked database. But again, we’re testing code that “cannot” fail. Only external things may cause a failure, and our tests isolate those. Again, cheap tests (although not as cheap as the unit tests), without the value of confidence
Even tests calling the real database, may not raise enough confidence.
On the other hand, if you’re running complex algorithms, spanning different modules and services, you’d better focus on them and test them. If unit-level tests work, great, if not – write bigger tests. But the value is in covering that complexity.
The model, or any other model, is not really a plan for everyone. You need to figure out which tests you need. The model they shape out to be? Not that important.
Wait, where’s the tip?
Here’s the tip I promised: Did you know that if you wrote the code differently, your tests needs will change? For example, if your complex algorithm sat in one place, it would be easier to unit test and isolate.
If the logic is spread around , unit testing becomes challenging (I’m using kind words), and therefore, you may try your luck elsewhere, with other kinds, more expensive tests. In other words, to raise your confidence, you will need bigger, costly tests.
Your code impacts your choice of tests, You end up with a model you didn’t plan on having. That’s the magic of testabilty for you.
Remember, the model (pyramid or any other shape) is not important. It may fit your situation, or not, but you don’t get points for working according to any model, either way. You get points for testing properly.
Are your tests effective? Do they bring you confidence at the right cost? That’s the important part. Even, if it’s amoeba shaped.
I talk about the different models in all of my workshops, but API testing is one of my favorites. Check out the “API Testing for Developers” workshop.
0 Comments