WTF is a Unit Test?

I remember it like it was yesterday.  

My Company was starting a new online product, and had employed The Consultants to come in and help.  The Consultants told The Company that A: they needed to employ an iterative development lifecycle (RUP) and that B: they needed software testers to help them.  The Company, in its wisdom, had plucked me out of the support department and placed me in that position.

I had been a software tester for 2 weeks. The lead developer for the product called me into a meeting.  I went in and sat down, and the lead dev says “The Consultant says we need for you to write the unit test cases.”  To which I responded (reasonably, I think): “Ok, what is a unit test?”  “We think it has something to do with testing the design document.  We know we are supposed to have the tests before we write the code.”  I know now that she was talking about TDD, but at the time I didn’t and neither did she.

Now, at this point it’s important to note that The Company didn’t allow us to use the internet from the office.  The first anybody had ever heard of “unit tests” was when the Consultant said we needed them; now, nobody had the ability to look up what they were.  (The local university didn’t teach unit testing in the CS program.  It still doesn’t.)  So, I did the next best thing: I went and looked it up on the internet from home.

A meeting one week later: “Ok, let’s talk about unit tests.” “Yes, when can we have the unit test cases?”  “As far as I can tell, when people talk about unit tests they are talking about using something like jUnit to run tests against the code we’re building.  I don’t know how to do that.  Do you guys know how to do that?”  Dead silence.  Another developer leans in and whispers to the lead.  The lead says “Ok, yeah, we’re definitely not going to do that.  Let me call The Consultant and figure out what we’re going to do.”  

The next day we have another meeting: “We decided that we’re going to call each page a unit.  We need you to write test cases for each of the web pages.  We’ll give you wireframes, you write test cases for the wireframes, and those will be the unit tests.”  I didn’t understand the point of calling something a unit test if it didn’t mean what the internet people said it meant, but I went along with it, and wrote the “unit tests.” And that was a mistake.

I’ve brought this story up a couple of times over the years, for a laugh.  My old Company, look at how backwards we were, ha ha!  But I came rocketing back to Earth last week when I was talking to a tester from another company in my town:

“What kinds of things are you doing over there?”  “Oh, lots of testing, acceptance tests, unit testing…” “Oh yeah, what language?”  “It’s all written in Java.”  “I didn’t know you knew Java.”  “Oh, I don’t know Java, that’s just what it’s written in.”  Uh… WUT?  I dug further in, and his company, who is *not* doing its first project, is also using one of these bizarro definitions of unit testing.  He learned to test from them, and he got his wrong definition from them.

Words Mean Things

There are a lot of testing terms that are a bit fuzzy; different people mean different things when they say, say, “UAT.”  There are cases where that might be okay, but in a lot of cases it’s not.  

Words mean things when experts talk to experts.  If I have a headache, and my wife asks me if I’ve taken an aspirin, she’s asking if I took a painkiller.  If I tell her “yes” but it was really a Tylenol, it doesn’t matter.  If a doctor asks me if I’ve taken any aspirin, then I better tell him no if I took a Tylenol.  He’s not asking whether I took a Tylenol.  (I could volunteer more info than what he asked, but I better answer the question exactly.)

When a sales person asks whether the team is doing unit testing, the team can say yes if it makes the sales person feel better, and it doesn’t matter.  If a tester and a developer are having that discussion, it’s important that both of them are really talking about unit tests.

WTF is a Unit Test?

unit test is an automated piece of code that invokes the method or class being tested and then checks some assumptions about the logical behavior of that method or class.  A unit test is almost always written using a unit-testing framework.  It can be written easily and runs quickly.  It’s fully automated, trustworthy, readable, and maintainable.  —Roy Osherove, The Art of Unit Testing

So, right there, if we can agree on most of those things, we already can see that I was wrong to agree to that other definition, and my testing buddy was also misled when he thought that he could write unit tests without coding.  

On my current product, we have 1800+ “unit tests” that run every build.  We’ve got 1800+ checks that run against our system on the build server, each of which mocks out the database and then checks some functionality in the system.  I’m happy about that situation except for one thing: we’re calling them unit tests and they’re not.  We’re using a different, also wrong definition of unit testing.

How does a modular synthesizer work?

You probably were not expecting, almost 1000 words into this blog post, for me to start explaining how synthesizers work. Bear with me! This is a good analogy!

Modular synthesizers are musical instruments that are like the musical keyboards you can buy at Best Buy, except way more flexible because their components are not wired together in any fixed order. You can wire them together however you want.

I promise that you don’t need to understand synthesizers to understand the rest of this blog post, but to understand what I mean, please consider the following diagram:

As you can see, modular synthesizers are build up from modules, each of which is assigned some kind of work. Each module expects some kind of input, which it changes somehow, creating some kind of output.

Programmers, does this sound familiar? In the software world, these modules are classes. The jacks on the front of the module represent the class’s interface. The knobs on the front represent setters for class properties.

I bring this up to illustrate what is and what is not a unit test. To test a new synth module, would you build a complete working system, and then wire your new module into that system to see if it worked? No. You would first attach it, by itself, to a test harness. You would provide every jack (interface) with test voltage, change the controls, and monitor what comes out the other side. Only once you’ve proven that it’s acting the way you expect would you attach it to another module you built. Otherwise, if something wasn’t working, you wouldn’t know whether the screwup was in the new module or the one you’re using to test it.

The act of testing the modules together in concert is called integration testing. This practice is also important – you want to know whether your filter module really wires up to your oscillator the way you thought – but it’s not unit testing.

Too many times, what a team will call “unit testing” is the modular synth equivalent of replacing the keyboard with a sequencer, and then analyzing the output from the amplifier by mocking out the speakers. That’s not a unit test, it’s an integration test. It’s not the same thing.

Ok, Ok, integration tests are not unit tests. Who cares?

This is where I think Mr. Osherove’s definitions falls a little short. What he is describing is a fast automated check. Having those is great; having lots and lots of them is really great. But unit testing is not the same thing as running those fast automated checks.

Unit testing is a thoughtful analysis of an individual module’s design that helps to prove that the design works and makes sense. The fast automated checks are just an artifact of that activity, which happen to create living documentation of the system. Here’s another modular synthesis fact that didn’t make it into my diagram: Filters don’t have to filter pitched sounds… they can filter envelopes! Envelope messages don’t have to go to amplifiers… they can be routed to filters! Synth modules are decoupled in the way that software modules should be decoupled. You’re not just proving that a module’s interfaces work, you’re also proving that modules can work in isolation. You can and should test an oscillator in a test harness without a filter, or an amplifier, or a midi interface. Every one of the components should be tested on its own, without any of the other components.

Only then can you prove that each module is correct, and that the modules are properly decoupled from each other. Only then will you have the freedom to move things around in your system and try stuff without worrying that the module you’re moving won’t work without the other modules. Only then will you be able to get creative and try to plug an envelope generator into an oscillator, without fear.

That’s what I mean when I talk about unit testing, and I plan to insist on this definition in the future.


One thought on “WTF is a Unit Test?

  1. Interesting topic. One thing to bear in mind when being precise about the difference between unit and integration testing is that if unit testing is to test a class or method completely in isolation, that implies you have to have complete trust in the test harness. In your synth module analogy, if you plugged in a test voltage and then something unexpected happens, it could either indicate a fault in the synth module, or a fault in the test voltage generator (or the cable by which they’re connected). The point is that although (if you’re using a standard test harness) it’s usually far less likely that the test harness is broken than that the new software you’re testing is broken, the passing of a unit test written against something like jUnit does depend on things outside of its control, whether you like that or not! So perhaps just beware of being too purist about it all.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s