Why is it in the software industry that I so often see teams binging on just one thing when it comes to testing?
What I mean specifically, is testing frameworks. In the Rails world the frameworks du jour are RSpec, Cucumber and Steak - with a little Capybara sauce. Mmmm sounds tasty!
RSpec has become a solid core for all layers of testing with the other parts. Cucumber was initially created to make the BDD'ness more domain specific... so that customers/project owners/BA's could read and even write them. Unfortunately, many Gherkin examples in the wild tend to be anything but non-coder friendly. This seems to have occurred due to many factors which may come to play on a team.
One of these factors seems to be part of what has attracted many people to Ruby on Rails in the first place. You often hear Rails coders talking about how they love it that RoR 'gets out of the way' and 'just lets them code'. This is definitely laudable when what is getting in the way is technical scaffolding or other technical encumbrances. It is like using a power tool over the old manual ones. Where it becomes a problem is when what is being 'gotten out of the way' is meaningful communication and collaboration across the team. Teams who have tried to get their 'business folks' involved through BDD have run into the problem that either the business loses interest or can't write the tests well enough. I believe at least a good portion of this is due to how the tests are being written. They are often too low level - basically a series of instructions for filling out web forms, clicking buttons, etc. The folks behind Cucumber have tried to force a solution by removing one of the culprits they provided early on: web_steps.rb. Web_steps was a sample collection of step definitions around many boring day to day activities. Instead of serving simply as an example, teams baked their tests around it - creating many awful to read, low level test suites.
While I applaud the removal of web_steps on the part of Cucumber, it is having the oposite effect. Teams are just moving away from Cucumber to Steak (because it is more 'coder' friendly) for their Acceptance/Integration tests. What lies at the core is a one-two punch of badness that I am seeing more and more: Teams are confusing Integration Testing with Acceptance testing AND coders are writing those tests.
Acceptance tests should be written in the domain language of the business. They are meant to be high level - to provide the coders a place to start.
Integration tests should be written to test across 'the stack' and as such are more focused on the implementation details. These are therefore in the more code like language of the implementation domain.
Unit Tests test the code directly... everyone knows the coders are supposed to write these.
Cucumber is good for BDD acceptance tests. Steak is great for integration tests. Eat your meat and vegetables!
Let me suggest a possible test driven development team and their roles in testing under the above approach.
"The Business" (Story owners/Customers/BAs/etc) - write high level Acceptance Test Scenarios - These could even be part of the story. In essence they write the Cucumber Tests. (Someone else might actually cut and paste the tests into a 'feature' file, but you get the idea).
"Coders" (Software Engineers/Programmers/etc) - write the step definitions behind the Gherkin. Step definitions can be more implementation specific. Writing these can be the first time the coders actually start doing Test Driven Design. Then the coders write their Unit tests however they see fit.
"Testing Team" (Test Engineers/QAs) - write Integration tests using Steak, etc.
The above scenarios are based upon testing using Cucumber and Steak. It could just as well apply to the myriad other testing frameworks (Fitnesse, JBehave, Selenium, etc). The essence is that integration and acceptance testing should be separate endeavors.
I can hear the objections already.
"We don't have a testing team.": Fine, there are projects which are small enough, simple enough where the coders serve this role. The point is, integration testing is a different animal from acceptance testing.
"Our business people are not involved/would never write tests)": Really? Who writes your stories? Maybe they were driven away by previous bad cukes. Listen to your business people describe the functionality they want. "So I am on the home page and when I go to create a new account then I should get some indication that the account creation succeeded". That is practically Gherkin already! If they won't physically write the tests, then the BA's could certainly be following them around 'taking dictation' as the acceptance tests drip from the Owners' lips.
On the other hand, if your business people are not involved, it is likely you have a deeper issue than what testing framework you are using. Do you have an Agility Coach?
"That is totally going to slow us down!": Again, fine! Speed by itself is not a good thing. Without direction it simply means you are going to drive off the cliff faster. If you are a business owner and the 'acceptance tests' you are being fed make you want to cry with their technical level of detail - ask why they are being written this way - or better yet - write them yourself. If the testing is being taken away from you completely, don't let them. Don't let anyone tell you that integration tests are the same thing as acceptance tests. Coders, if you want your business owners to engage more, realize that giving them pages of cryptic, coder centric tests to review does not really encourage their engagement. As stated before, the idea of BDD was to simplify acceptance gathering so that coders would know where to start. It supports Agility by giving a venue for business people and programers to collaborate frequently. In the end, the slight reduction in speed along a more prudently chosen path will likely save you way more time than a radical course change much later in the project.
All of this raises a number of other questions I have been pondering for years - such as having different types of 'Customers', What sufficiency of testing levels for a given project, optimizing testing approaches...
It also makes me hungry...
* - my wife wanted me to note that our household is vegetarian and that I am only promoting Steak in the 'software' sense. Thank you dear! :-)