Test Attribute #3 – Speed

This is the 3rd post on test attributes that were described in the now more famous “Test Attributes – Introduction” post.

There’s a story I like to tell about my first TDD experience. You’ll have to hear it now (some of you for the n-th time).

It was many moons ago,  when I just completed reading Kent Beck’s excellent “Test Driven Development By Example”. And I thought: This would end all my misery.

I was working on a communication component at the time, and I thought, why not use this new TDD thing?

I’ve already committed one foul ahead of writing a single line of test code, because I knew  that I was going to use MSMQ for the component. So I decided on the design instead of letting the tests drive it. My level of understanding of TDD at the time is not relevant for this story. MSMQ however, is.

For those who don’t know, MSMQ is Microsoft Queuing service, that runs on all kinds of Windows machine. An infrastructure for asynchronous messaging, that seem perfect for the job. It is, however, a bit slow.

So for my first test, I wrote a test that writes to the queue and waits to receive it from it. Something like this:

[TestMethod]
public void ReceiveSentMessage()
{
   MyQueue myqueue = new MyQueue();
   myqueue.SendMessage(new Message("Hi"));
   Message receivedMessage = myqueue.Receive();
   Assert.AreEqual("Hi", receivedMessage.Body);
}

The Death Spiral Of Slow Tests

I was so happy I had a passing test, I didn’t notice that it took a few seconds to run. Most people starting out with unit testing don’t notice
that. They keep accumulating slow tests to their suite, until one day they reach a tipping point.

Let’s take, for example, a suite that takes 15 minutes to run. And let’s say I’m a very patient person. I know, just work with me.

Up to this point I had no problem running a full suite every hour.

Then, at that 15 minute point, I decide that running the suite every hour cripples my productivity. So I decide that I’ll run the tests twice a day. One run will be over lunch, and the 2nd will start as I go out of the office. That way I won’t need to wait on my time, the results will be there when I get back to work.

That leaves me more time to write code (and hopefully some tests). So I write more code, and when I get back from lunch, there are a few red tests. Since I don’t know exactly what’s wrong (I can’t tell exactly which parts of the big chunks of code I added productively are the ones to blame), I’ll spend an hour debugging the failing tests. And repeat that tomorrow morning, and the next lunch break.

Until I realize that I now spend 2 hours a day working on fixing tests. That’s 25% of my time working for my tests, instead of them working for me.

Which is where I stop writing more tests, because I see the cost, and no value from them. And then I stop running them, because, what’s the point?

I call it “The Death Spiral Of Doom”, and many developers who start doing testing, fall downstairs. Many never climb up again.

If we reverse the process, we’ll see quite the opposite. If my suite runs in a matter of seconds, or faster, I run it more often. When a test breaks, I know what caused the problem, because I know it was caused from a change I did in the last few minutes. Fixing it may not even require debugging, because it’s still fresh in my mind. Development becomes smoother and quicker.

Quick Feedback Is Mandatory

Tests should run quickly. We’re talking hundreds and thousands in a matter of seconds. If they don’t we’ll need to do something about them.

Quick feedback is not only an important agile property. It is essential for increasing velocity. If we don’t work at it, the entire safety net of our tests can come crashing down.

So what can we do?

• Analyze. The length of tests is part of every test report, so it’s not even subjective. Look at those tests, and see which are the ones that take longer to run.

• Organize.  Split the tests to slow running and quick running. Leave the slow running to a later automated build cycle, so you’ll be able to run the quick ones without penalty.

• Mock. Mocking is a great way to speed up test. If a dependency (like my MSMQ service) is slow, mock it.

• Groom. Not all the tests should be part of our automated build forever. If there’s a part of code that you never touch, but has 5 minute test suite around it, stop running it. Or run those on the nightly cycle.

• Upgrade. You’ll be surprised how quicker better hardware runs your tests. The cost may be marginal compared to the value of quick feedback.

The key thing is ongoing maintenance of the test suite. Keep analyzing your suite, and you’ll see where you can optimize, without taking bigger risks.

The result is a quick safety net you can trust.

Next up: Accuracy.
For training and coaching on testing and agile, contact me.