“…my songs aren’t written on a schedule… It’s been more con-fessional than pro-fessional. – Bob Dylan
The Regression Testing Problem
I work on a commercial product. We release several times a year, and that’s been going on since I started on the team five years ago. Like (I assume) most teams that are in that situation, we have a set of regression tests that we use to verify that new work hasn’t broken old work.
We’ve automated a lot of those checks, but a portion of it is still manual work. Moreover, we have a team norm (implemented by yours truly) that states that you can’t merge new code with our production code without also putting into place a new regression test (or tests) that will allow the team to verify, at the end of this release and also future releases, that the new stuff still works when it becomes the old stuff.
I wrote down some steps. The steps tell you what to do. If you do what I say, and the product does what I said it would, then you have proven that the product still has some behavior that I decided was valuable.
It’s important to point out that we find bugs doing this stuff. Sometimes – not that often, but sometimes – we even find bugs where the actual result differs from the expected result! More often, following these instructions just happens to put us in the vicinity of bugs by taking us on a tour to the far reaches of the product, and we notice a bug while we’re traveling.
Like I said, we’ve been rolling like this for a long time. It’s been alright. We reduced the amount of mindless busywork by a large amount when we automated most of the regression checks, and nowadays – by utilizing an ATDD approach – a very significant amount of the regression tests are automated before any human being has to go through this… regimented, dystopian procedure.
But we’ve still got that norm. “Make sure you’ve written regression tests before merging code with our release code.” It bothers me. I want something better. Here’s why:
- Testing documentation is documentation, and all documentation goes out of date.
- I probably did not think of the best way to test it in the first place.
All Documentation Goes Out Of Date
Unless we’re talking about living documentation – which is a special case more related to “checking” than to “testing” – any test case with explicit procedures is going to go out of date. I don’t know how many times I’ve had this conversation: “Test case #89675 is wrong, the product doesn’t do that anymore.” “I don’t have the test case IDs memorized dude, if you see something wrong just fix it.” That seemed frustrating before because I felt like the test cases were more work than they should be. Now it feels doubly frustrating because that’s also an indicator that our brains can do better than the test cases.
How The F&#$ Should I Know How You Should Test It Three Years From Now?
At this moment I am the least qualified I will ever be to tell you how to test a feature of my product. Testing is learning about software. My team tests every day, so we learn more about the product every day: about what our customers want, about risks, about testing techniques, about the new features that came after the ones we tested before. If I write you a test case today, and you run it tomorrow, would you get the benefit of what we know tomorrow that we don’t know today? Yes – if you ignore what’s in the test case.
So what’s the point of the test cases?
The Point of the Test Cases
Software is complex, and people have imperfect brains. We need our regression test cases for two reasons: because we want to remember what’s important to test, and we want to remember how we did it last time. We need to be able to prioritize, and say that some things don’t get regression tested every month. We need to be able to say “these were the things that we changed, and this is what we wanted them to do, and here are the special setup requirements for doing that, etc.”
Like I said, regression test cases are a solution to that problem. But I think there’s a better solution.
Here’s the thing in a nutshell: You’re on my team. I want you to know what I think we need to test, which is not to say that I know best. I want you to know how I tested this stuff last time, which is not to say that you should test it that way. And if you test something, if I’m testing it next time, then I want to know how you did it, although I’m going to try to think of a better way, try to do things you didn’t do. We’re not going to prescribe tests for each other, we’re going to describe tests *to* each other.
How to Describe Tests Without Prescribing Tests
I don’t know what’s going to work best for us. The first thing that jumps out at me as a non-prescriptive, descriptive approach is SBTM, but I’ve tried that approach in my context, a LOT, without much success. People (myself included) are sloppy at writing session reports, and pretty uninterested in reading them. Plus, managing them. I’m sure it could happen but it sounds… boring. Even telling people “you must submit a session report!” feels too regimented, too filled with potential for wasted effort.
This week I am starting an experiment at adapting xBTM to my context, using MindMup to work on collaboratively building up a “testing picture” of a feature that we can refer back to later. I’ll report back on how that’s going. Have you moved from prescribing to describing, in your testing documentation? What did you do? How did it go?