Describing, Not Prescribing: A Different Way Of Looking At Test Documentation

“…my songs aren’t written on a schedule… It’s been more con-fessional than pro-fessional. – Bob Dylan

The Regression Testing Problem

I work on a commercial product. We release several times a year, and that’s been going on since I started on the team five years ago. Like (I assume) most teams that are in that situation, we have a set of regression tests that we use to verify that new work hasn’t broken old work.

We’ve automated a lot of those checks, but a portion of it is still manual work. Moreover, we have a team norm (implemented by yours truly) that states that you can’t merge new code with our production code without also putting into place a new regression test (or tests) that will allow the team to verify, at the end of this release and also future releases, that the new stuff still works when it becomes the old stuff.


The format of these regression tests will probably be familiar to anyone reading this:

I wrote down some steps. The steps tell you what to do. If you do what I say, and the product does what I said it would, then you have proven that the product still has some behavior that I decided was valuable.

It’s important to point out that we find bugs doing this stuff. Sometimes – not that often, but sometimes – we even find bugs where the actual result differs from the expected result! More often, following these instructions just happens to put us in the vicinity of bugs by taking us on a tour to the far reaches of the product, and we notice a bug while we’re traveling.

Like I said, we’ve been rolling like this for a long time. It’s been alright. We reduced the amount of mindless busywork by a large amount when we automated most of the regression checks, and nowadays – by utilizing an ATDD approach – a very significant amount of the regression tests are automated before any human being has to go through this… regimented, dystopian procedure.

But we’ve still got that norm. “Make sure you’ve written regression tests before merging code with our release code.” It bothers me. I want something better. Here’s why:

  1. Testing documentation is documentation, and all documentation goes out of date.
  2. I probably did not think of the best way to test it in the first place.

All Documentation Goes Out Of Date

Unless we’re talking about living documentation – which is a special case more related to “checking” than to “testing” – any test case with explicit procedures is going to go out of date. I don’t know how many times I’ve had this conversation: “Test case #89675 is wrong, the product doesn’t do that anymore.” “I don’t have the test case IDs memorized dude, if you see something wrong just fix it.” That seemed frustrating before because I felt like the test cases were more work than they should be. Now it feels doubly frustrating because that’s also an indicator that our brains can do better than the test cases.

How The F&#$ Should I Know How You Should Test It Three Years From Now?

At this moment I am the least qualified I will ever be to tell you how to test a feature of my product. Testing is learning about software. My team tests every day, so we learn more about the product every day: about what our customers want, about risks, about testing techniques, about the new features that came after the ones we tested before. If I write you a test case today, and you run it tomorrow, would you get the benefit of what we know tomorrow that we don’t know today? Yes – if you ignore what’s in the test case.

So what’s the point of the test cases?

The Point of the Test Cases

Software is complex, and people have imperfect brains. We need our regression test cases for two reasons: because we want to remember what’s important to test, and we want to remember how we did it last time. We need to be able to prioritize, and say that some things don’t get regression tested every month. We need to be able to say “these were the things that we changed, and this is what we wanted them to do, and here are the special setup requirements for doing that, etc.”

Like I said, regression test cases are a solution to that problem. But I think there’s a better solution.


Here’s the thing in a nutshell: You’re on my team. I want you to know what I think we need to test, which is not to say that I know best. I want you to know how I tested this stuff last time, which is not to say that you should test it that way. And if you test something, if I’m testing it next time, then I want to know how you did it, although I’m going to try to think of a better way, try to do things you didn’t do. We’re not going to prescribe tests for each other, we’re going to describe tests *to* each other.

How to Describe Tests Without Prescribing Tests

I don’t know what’s going to work best for us. The first thing that jumps out at me as a non-prescriptive, descriptive approach is SBTM, but I’ve tried that approach in my context, a LOT, without much success. People (myself included) are sloppy at writing session reports, and pretty uninterested in reading them. Plus, managing them. o_O I’m sure it could happen but it sounds… boring. Even telling people “you must submit a session report!” feels too regimented, too filled with potential for wasted effort.

This week I am starting an experiment at adapting xBTM to my context, using MindMup to work on collaboratively building up a “testing picture” of a feature that we can refer back to later. I’ll report back on how that’s going. Have you moved from prescribing to describing, in your testing documentation? What did you do? How did it go?

-Clint (@vscomputer)


4 thoughts on “Describing, Not Prescribing: A Different Way Of Looking At Test Documentation

  1. Hi Clint,

    I’m in a broadly similar situation and have used SBTM to perform exploratory regression testing too. Here’s some thoughts. I’ve been intending to blog about this for ages..

    * we do exploratory testing on a regular basis without using SBTM at all. It’s just part of the tester’s daily work. We’ve tended to keep SBTM for team-wide efforts, including regression testing.

    * many of our regression test cases are underspecified and so intended to be more like mini charters. Testers are directed to explore around the test case

    * we’ve used SBTM in a few different ways. I try to take care of the management and housekeeping myself to keep the rote work off my testers – I am the test manager 🙂 – but I do insist on session reports and also on debriefs in most circumstances.

    * Demonstrating the value of the reports is key and the debriefs and subsequent charter planning should be able to do that for you. Testers should be able to recognise that when they’re saying “I can’t remember” every time a question is asked at debrief, some notes would have been a good idea.

    * I expect all testers to both test and debrief – I don’t limit debriefs to test leads. This includes testers debriefing me on the testing I’ve done.

    * On each new SBTM round, we typically debrief as a group to start with. So, one tester will debrief another with the rest as observers. The observers typically won’t interrupt, but are encouraged to comment and suggest. This helps to generate ideas for how to debrief, new charters, etc.

    * Apart from the learning experience, I’ve found this to be quite motivational and we can build up momentum for the task too.

    * Planning new charters can be a group activity too.

    * We’ve experimented with less documentation., more flexible timing restrictions and procedure too. In some cases where we have an acute need and know we aren’t going to be iterating much we set up short regression test rounds (sometimes even using a set of bug verifications as a loose charter) and recording notes on our wiki e.g. in a shared page that all participants can watch. We may or may not debrief in these cases.

    * After each round of SBTM for regression testing we’ve debriefed about our experience of the process and tried to incorporate that feedback into the next round.

    I can also recommend

  2. Great blog, love the topic, been thinking hard and trying out new things with it for the last 2-3 years.

    Here are some things we’ve tried in the last couple of years. Not that any of these are bad or good, but I’ve found that trying to find a single solution is not useful for more then 2-3 people. Some of the people loved to use tours to describe their testing, other preferred a more RST approach, I still had one who loved to write test plans (and produced them fast AND good enough I couldn’t argue). I think attempting to find a single way for multiple people to describe might not work out, at least without getting dictatorial. (part 1) (part 2)

    Can’t wait to hear what you find.

  3. Pingback: Describing, No Prescribing – Volume 2: A Rough-Draft Recipe | Tester Vs Computer

  4. Pingback: Describing, Not Prescribing – Vol 3: Let’s Call It Expeditionary Testing | Tester Vs Computer

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s