Throughout history, since the cavemen of old, we were looking for signs of quality. There was one way to do that: Look at the final product and decide.
But that takes time. Plus if your work is being assessed only after you won, and the mammoth lost, that’s a bit too late.
We’re looking for “leading indicators”. Things that will tell us now, if we’re on the right path or not. So, before we finish, it’s still not too late to fix things.
In the software world, everyone agrees that we need more of those things, and earlier. The problem with testing, is that it’s only after coding is done. So, that’s a bit too late.
What can we do earlier? Requirements, design and code reviews try to minimize the risk, that we’ll put something evil out there. But those don’t scale. So what good are they?
What does scale?
What can we measure, that gives us a way to know that everything’s going through the pipeline, gets a stamp of quality approval?
Code coverage, of course. That’s the universal software quality metric. It tells you how much of your code is covered by tests. And when the tests pass, then there’s more coverage, and that means that the product is as shiny and sparkly as it can get. Right?
Yeah, not exactly.
In order to understand why, let’s first talk about how code coverage tools work. Depending on the technology, they hook into the test automation process, and mark for themselves when you’ve entered a function, and when you’ve left it. Then they collect all this information, and give you a very colorful report. All coverage tools work the same way.
And that’s really helpful, if you want to know, if you called a function during a test. At least once.
What coverage reports don’t tell you, is if the tests were good. Or if they even had an “Assert” at the end. If they checked anything. Or anything important. There’s no connection to a use case, real or not, and if that use case is current, relevant, happens once a year or once a second, runs as a single operation, or called by 1000 concurrent users.
It just tells you, “Thank you for visiting our function. Here’s our rating”.
So what can you do with this number? Not much. Obviously, low coverage means not tested enough. But what is enough?
And what about higher coverage numbers? That means that many lines of code were executed. But was that on the right paths? the important ones? Can we connect this number to the output quality?
No, no we can’t. And some managers (I’m sure not yours, I hope so anyway), are going to hit you on the head with those numbers. If you’re not reaching coverage goals, it means that you, yes you, are hurting quality.
Code Coverage is not enough
Are coverage tools really good for nothing?
Not completely. If you want to check that a buggy area gets covered with tests, incremental coverage is a good indicator. But that’s just for a small area, that is being worked on. But again, is it tested well?
Then there are other factors. When you’re adding coverage on a certain area, but you keep watching the coverage numbers on the whole repo, and they’re pretty low, you won’t see much change, which is pretty demotivating.
So is code coverage hazardous for your health? Well, it probably doesn’t improve your quality of life. Yes, it’s easy to measure. But no, it doesn’t say much about your quality.
There are things you can do instead. Like managing the test cases. And organizing suites according to importance. And identifying those black holes areas in the first place.
That’s a lot of work, that tools don’t give you, and you’ll need a human or too. Unfortunately, quality doesn’t come from tools. But at least you know that now.
If you want to learn more about unpopular opinions about coding, there’s a Clean Code Workshop you can join. Let me know if you’re interested.
0 Comments