Testing Strategy Analysis – An Example

Each stage of implementation is hard in its own way. When we start out with a new effort or process, we think of it as: Sure, it will be hard in the beginning, but it’s going to become easier as we continue. It’s true, and then after a while a new challenge appears.

A team I’m working with has developed quite an automated suite of unit, integration (logic integration and mocked databases), and some end-to-end test (including deployments). In addition, they are also employing manual (scripted) effort by testers.

The painful part, which originated the talk, was the continuous effort in manual regression testing. They feel they are not getting the most from their process: As the applications continue to grow, the manual efforts do not scale. They feel that in their current pace, they cannot test the applications at the quality they had before.

The other issue is that while the automated suites are doing a lot of the work, during manual testing, they find most of their actual bugs when testers step off the script. That’s not the problem in itself, but it raises concerns regarding their effort in writing manual test cases.

We performed an analysis on their inventory of tests.  Automated unit and integration tests run on every build, and the automated E2E tests complete in a few hours, so the automation suite returns the needed feedback in a reasonable time.

When we drilled down deeper, we saw that the automated suite can be expanded. The test plan for each feature covers both integration, E2E and manual test cases. Currently the guidance for automation of high level flows are either happy paths and easy-to-write negative paths. We concluded we can automate more test cases and reduce the number of manual test cases. This will require more automation effort, but in the long run frees time for manual testing.

However, let’s not forget that the manual tests were not as effective as-is. The bugs that were discovered were not thanks to the detailed, prescriptive steps of the tests.

We started discussing exploratory testing to increase our chance of uncovering more of these bugs. If we currently find these “surprise” bugs, there are probably a lot more hiding. Only, we can’t find them by thinking up front of where to look for them.

In fact, it’s what the testers did in the first place. But there’s a lot to learn about doing exploratory testing continuously, even methodically. Many testers aren’t even aware of exploratory testing as methodology. While it’s still new for most testers, what I really find puzzling, is that even when the solution is popping up naturally, we tend to stick with the tried-and-true scripted tests.

Regardless of process, there’s always room for improvement. We need to analyze the big pain points and start solving them in a holistic manner. If you haven’t already, you should start now.

Quality is too important to just leave to emerge.

Want to learn more about strategy? Check out the video of a my talk “Unit Testing Strategy“.