Let’s talk about our expectations from automation. Specifically, the expectation that our automated tests will find bugs. This expectation leads to some risky decisions, which I’ll get to in the end.
You program the code, debug it, it works. You even have an automated test that proves the code works. You’re protected forever from the bug!
Let’s look at a simple example. In my ever-persisting demo app, when you enter an entry, it stores the data it in the database.
And it works, trust me. Here’s a test for it.
This is a Playwright test, in javascript, that enters all the details, checks the “agree to terms” checkbox, clicks the submit button, navigages to a thank-you page, and goes to the database, to check the last entry contains the entered details.
FAIL!
When will the test fail? Two cases, whenever there’s a crash, that causes the test to fail. Or when someone tries to change the functionality. And, that’s the point of automation, really – to find bugs, right?
Well, not completely. Automated tests alert us that something that worked before, doesn’t now. It could be a bug.
However, think about that workflow I described. There’s code, and then, there’s a test that proves that code works. I deliberately wrote that automated test to prove my code works.
I was not looking for a bug when I wrote the test. That’s good, because I didn’t find one. But like the saying goes, testing doesn’t confirm the abscense of bugs. It just means I haven’t discovered them yet.
In my case, this scenario works as I expect it to.
Finding bugs, means I go looking for them. And automated tests prove something works, not the other way around.
Ok, potato, tomato.
Many organizations reduce the role of testers, sometimes even get rid of them completely, replacing them with automation and developer tests. They do that under the assumption that the automated tests keep quality at the same level, at least, as when they had testers around.
However, that level of quality depends (among others) on two main things – validation that the product works, and exploration to learn more about what works and what doesn’t.
Automation can help with the first, but for the second, you need a human. Someone who can design a net, throw it, look for weird ripples they see, and catch the fish. I mean bugs.
Expecting automated tests to find bugs, is both wrong and risky. Automation works well, for any scenario that is expected to work. But with the complex systems that we have today, it will not be enough to even do that.
Automation is great. Expecting it to find bugs? We’re setting ourselves up for disappointment.
To learn more about automation, check out my Testing With Playwritght Workshop. This one doesn’t dissapoint.
0 Comments