The Truth About Fragile Tests

Fragile tests.

Even the mere thought of them, makes our stomach turn. And for good reason too. We can’t trust them. And they cost us money.

First – the trust thing. We have tests that check the same code, pass most of the time, but then one build in a hundred fail. Then we run them again and they pass.

Oh it’s those tests, that’s ok. All you need to do is run them again. But the trust we have in our other tests starts to erode. If those tests “usually” pass, can I trust the other tests? What if they start failing – will I just need to run them again, and it will be ok? Maybe twice will make it alright?

We have tests to have confidence. Those things don’t build it, they wreck confidence. And not just in themselves, but also their surrounding friends.

Then, there’s the time thing. Maybe running again is ok, but then you ask “is it the regular failure, or a new one”? Can’t know until you check. Logs, debugging, running again and comparing.

And after that, you don’t find anything special, and chalk it down to “it’s those fragile tests again”. Two hours of your precious time evaporated.

Fragile tests are not really my favorite things in the world, you might gather. But you know what’s the worst thing about them?

The name.

We call them fragile tests. But really they are not fragile. It is the code. Or the design. Or the architecture.

Sure, we see the failure in the tests. That’s the smoke. But the fire is really underneath. It’s rarely the tests’ fault.

And that’s a problem, because we shift the responsibility of quality (again) from development to testing.

I don’t care about who need’s to fix it. But as long as we accept “fragile tests” as part of our lives, we condemn testers to waste more time, in finding what they did wrong. , which they probably didn’t, instead of sharing the responsibility to raise quality in the team.

If you have fragile tests, (and as I said, you probably don’t), it is a team responsibility, and they should have the shared motivation to solve the problem. The real problem.

We all benefit from quality. It gives us pride, and it allows us to speed delivery. We should all be pulling for raising it.

If you’re smart, you won’t get rid of your fragile tests. You’ll fix the fragile code they run.

And stop calling them that.

And check out my workshops on better tests: API Automations for developers and QA.