When I say that testing is not just checking something is supposed to work, or doesn’t have a bug, what does it mean?
Look at this API:
Without even thinking, we already have assumptions on how it works. We assume there’ll be a new user created at the end. We assume that the API will return a 201 code – CREATED.
If it works.
If it won’t for some reason, we expect some other error code. And that this “so-called-user” data, won’t infiltrate our system.
Either we know it, or feel it, or have a premonition – we can translate the expectation into an assertion in code.
Any automated test ends with at least one of those. We have an expectation, and we want to check that we’re right. If we’re surprised, well, the real production caller will also be surprised, right?
So this is when we expect something. But have you ever went into testing a system, without any idea of how it works?
Exploratory Testing FTW
I’m sure you’ve downloaded an app, and started using it without reading the docs first. At least once in your life. But you used the app anyway. Who reads documentation?
In testing, we call that exploration. We find out ourselves. Sometimes we’ll feel ok, sometimes weird, sometimes surprised. And yes, sometimes angry. We learn how it works.
If we’re exploring, we can’t write an assertion, because there’s no expected result. That means we can’t automate exploration. We can automate part of it, just not the “testing” part, the part where compare are expectations to the results.
That’s the difference between exploration and checking. Both help us test the system, and report on its behavior. Both are needed.
Testing = exploration + checking.
Want more API testing? Sure you do. Check out the workshops. “API Automation Testing” for developers and the “API Testing with Postman” for testers.
2 Comments
DOMINIC CAPLAN · June 11, 2024 at 4:55 pm
Please take the following as observation, not criticism.
Your equation is a great way to explain it to someone outside the field.
Still, your declaration, “If we’re exploring, we can’t write an assertion, because there’s no expected result.” could be a little misleading for someone new to the field. If testing, for the reader, is to be more than just a job because one cannot get anything else, or if on the way to something else, it behooves you to push your audience’s thought process beyond that point. Thus, you CAN write an assertion before touching the SUT. And you did that in this article implicitly.
To wit, “If it works…”.
Isn’t that an assertion? Even a newborn can assert. Any assertion is merely the translation of a memory into a question. I’ve done this before, can I do it again? These things happened after that happened or after I did this… if I do it again, will it happen again (checking)? If I do something different will it not happen again (checking)? If I do something different what will happen (exploration)?
Regardless of intent, consideration of the outcome automatically implies expectation.
Non-testing example:
STATEMENT: Christopher Columbus discovers America
QUESTION: Was he exploring, or checking?
ANSWER: Both. He started with the expectation of sailing to China. He knew others had sailed to China; he knew of the theory of a spherical world. He extrapolated that there might be a better route. He stepped onto the sand of a North American beach (checking where his route had taken him), but immediately he realized he wasn’t in China, he lost the connection to his knowledge and what was required was a process for dealing with the unknown (he was exploring).
Gil Zilberfeld · June 18, 2024 at 5:15 pm
Thanks for this Dominic!
I should write more about this. Different people, with different experiences, will probably have a differnt equation. Mr Columbus, for example, was an explorer. He was looking to find a new way to east Asia ( I assume, never met the guy). He was working under certain assumptions. And assumptions need to go into the equation as well. Once hit with the truth, our perception of reality changes, the perception that are built on experience, bias, traumas. Testing is also the process of handling discoveries. Sometimes well, sometimes less.
So, I agree, this is a simplification. My intent is for people to ask “what hat am I wearing now”? Complexity (although real), may not help here.
I’ll take your feedback into account, thanks!