Here’s one more:
BDD-style step definitions provide a convenient opportunity to make your test automation more gray-boxy.
In one of the many real-world case-studies examined in Gojko Adzic’s book Specification By Example: How Successful Teams Deliver The Right Software, a situation is described where new features are completed using the ATDD workflow, and when those features are merged with the product codebase, the automated examples are merged with the thousands of existing examples that comprise the “living documentation” for the product. This living documentation (or, if you prefer, this “executable specification”) is seen as a work product at least as valuable as the production code.
Ok, that all sounds great. Executable specifications, living documentation, sign me up for that! WAIT: what now? Thousands of automated examples?
I started moving my product’s UI test automation into the Gherkin format about 3 or 4 months ago. We are fortunate to have a decently well-designed (in my opinion) codebase for our automation, so it was easy to do, and now we have 65 end-to-end automated examples that we can run at the push of the button.We can each one on its own, or we can run them by feature, or we can run them all at once.
If we run all 65 at once, the test run takes over four hours. This thing isn’t going to scale.
A lot of test gurus out there will tell you that UI automation just sucks. It’s too brittle, too finicky, and too slow. I don’t know that I agree that UI automation sucks, but I will grant that it’s finicky and slow. (Brittleness is something that can be conquered by good test design.)
The teams that have thousands of checked examples in their suite, they are not using UI automation for the bulk of that. It’s just not possible. That’s only possible if those tests are what Google calls “small tests”: tests that run quickly, on the order of seconds or even milliseconds… not lengthy UI tests that take more than a minute a piece.
Ok, maybe that sucks for you, because your product doesn’t have a gray box (integration) API that would make it possible for your steps to run that quickly. I’m in that situation, too.
The Gherkin-formatted step definitions represent an interesting opportunity for us.
As I described in my first SpecFlow post, Gherkin steps are great for encapsulating your test code because they’re easy to read, they’re easy to reuse, and everybody can understand what they do. Once you have lots of automated examples using your steps, you can refactor them. (Refactoring: improving the design of a function while holding its functionality constant.) As long as the refactored steps still accomplish the same work, they don’t have to accomplish it in the same way.
Let’s say your product has a concept of “users with passwords.” Your UI automation contains a step called “Given I have created a user named ‘Bob’ whose password is ‘1234.’” This takes your UI automation framework about 30 seconds, and every test must include it. This step represents a great opportunity to speed up your tests. A user-creation API is created (and tested), and then the step is rewritten to use the API instead of the user interface. Now that step take less than a second, and your entire test suite just sped up dramatically.
I mentioned before, too, that something nice about this kind of automation interface is that it gives additional power to testers. In that user creation example, it just became very easy for a tester to populate the product with hundreds of users at the push of a button, using the same Gherkin interface that he or she is used to.
So, here’s a potential workflow that seems like it could be very fruitful:
- Create automated examples using a UI framework wrapped in BDD-formatted domain language.
- Identify automated steps that are taking a lot of time.
- Replace the underlying test automation in those steps with code that exercises an integration API.
- (Exploit your new superfast steps to design new tests.)