My colleague Sarah Taraporewalla posted an interesting text on acceptance testing. She doesn’t believe in this technique. I’ve been thinking of acceptance tests for some months now and think that she has a valid –a bit too radical but still valid- point.
My main problem with acceptance testing is that they are too wasteful.
- They are temporary. The acceptance criteria for one Story won’t necessarily be true for more than a few iterations, days or even hours.
- They add time to the build. Adding redundant test cases to each story just to explicitly fulfill its acceptance criteria often cause builds to take more than acceptable.
That said, I still find acceptance tests useful
And then the waste starts to be a real problem. In an iterative and incremental project is very common to invalidate dozens of old acceptance criteria just by playing a single story. The developer playing that card must search the code base and decide what should be kept, what should be modified and what makes no sense anymore. That takes time.
And the build takes time as well. Given the fact that we test per-story, not per-feature or scenario, the build time skyrockets. I’ve seem in recent projects builds increasing by orders of magnitude only due to acceptance tests. Every single story finished adds more time to the build.
So, how can we still have the benefits of automated acceptance testing and get rid of the multiple problems caused by those?
I asked this question to a lot of smart folks and no one could really give an answer. I’m trying an approach that’s new for me and I can’t remember reading about something similar anywhere. This technique aims in fighting the two main causes for crappy acceptance tests.
The first and probably worst cause is the fact that developers are not used to write tests. I don’t say this as an excuse; I do think that professional software developers should know about boundaries, branches and other concepts. The problem is that as developers don’t know much about testing they usually create inefficient tests. In a recent metrics gathering I found that about 20% of the time taken by a build process in a given project was spent testing exactly the same scenario over and over again. To avoid spending effort in inefficient tests we decided that the owners of acceptance tests are the QA people.
The other cause is the fact that acceptance criteria are temporary. To avoid wasting time maintaining something that may not be valid in one hour time we decided that the acceptance test should be merged to the automated system tests after the story is done.
Our current process has, as Dahlia explained, something I like to call user story sandwich. There are some issues with this approach but it matches seamlessly with our acceptance test strategy. It starts with a kick-off with business analysts, developers, QAs and other stakeholders. This takes 15 minutes in average and in this kick-off we revisit the acceptance criteria.
Immediately after this the devs and QAs draft acceptance test cases. Those test cases will guide developers during their work and we assume a story is dev-complete when they pass. The tests written at this time are very similar to what we were using before except by the fact that now they’re generally written by a tester with the developer’s support.
When the story reaches dev-complete we do a walkthrough, a little showcase where devs present the feature working to business and testers. The main role for testers at this stage is to verify if the acceptance tests pass as expected. As devs and QAs are always communicating during the development it is very unlikely that a story will be rejected at this stage.
After the walkthrough is finished the acceptance test is given back to the QA people. To get finally to the DONE column in the story wall a card has to pass through testing and the first step they perform is to convert acceptance tests to system tests. System tests are generally created based on features instead of stories and they are often more concise and efficient than per-story testing. They are run in the build process, either in the normal dev build or as the second stage in build pipeline.
This approach is working quite well so far. Unfortunately we decided to go for this approach in the end of the project so I couldn’t see if it works for the full project lifecycle.
There are some tricky bits in that aproach.
The first and obvious thing is that your testers should not only automate their tests but also use a tool developers can use, understand and extend. This may be a problem for some but a lot of projects are already adopting tools like Twist, Selenium, Concordion and the like for acceptance tests. Even lower level API tests generally can be easily written by testers using a Domain-Specific Language.
Other attention point is that you should not forbid developers to write higher-level tests. In our team discussions it was pointed a lot of times that, if the process works, the developer should not need to write any high-level tests besides those developed together with the tester. Although I agree with this in theory, for me this is like saying that developers can’t run the website they’re building as testers will do that anyway. So far we decided to have a special source folder where developers can write their tests. In this folder currently we have several tests covering scenarios that testers didn’t consider relevant but developers still want to verify.
So far we didn’t see any improvement in the build time. The main impediment is that we still have heaps of old acceptance tests, and obviously business people won’t spend time converting those. We try to convert then to the new style when they need some maintenance but I don’t believe we will be able to convert even 50% until the project is finished. We are not making the build longer, what is good, but we are not reducing its duration.
An improvement we had was in the time cards stay in the “test” column. As the acceptance tests were written together with testers there are just a few more tests they add after converting them to system tests. Creating acceptance tests just after the kick-off is also helping to clarify some obscure topics before any code is written.
Will we continue doing that? Would I recommend that approach? It is too early to say. It is working so far but I can’t say that this is the ideal solution. I’m looking for ideas.
Got any?
I like acceptance testing because it’s an opportunity to have an unique view about what we need to accomplish, with development, testing and business.
In this way, development with that vision can build unit tests, testing can build functional tests and business can do their user acceptance tests, but all based on the same set of principles written in the acceptance testing process. And the whole team can help, because they all understand what is needed.
When thinking about testing (from a tester perspective), I believe that we need to go a little higher and think about scenarios and not only one story. Professionals focused on testing can do this, they have knowledge and can teach developers to test better. They can help to create knowledge, which is one principle we value in Lean Software Development for instance.
Also need to consider that not 100% of scenarios will be automated, so it’s important to find an approach where you have good documentation, a standardized process and a high code coverage.
Hi Daniel,
I don’t see much disagreement between your perspective and what I’ve written here. The only divergence I’d say is in that you talk about business creating “their” acceptance tests.
In theory, I see System Test as *the* business test (excluding UAT), specially because they are based on long-term system behaviour and not temporary criteria. Acceptance tests are the minimum set of features that the business consider relevant to accept a story as done.
If a scenario can’t be automated it is *very* unlikely that an acceptance test for related stories can be automated, so I can’t see an issue here.
But back to the topic, what you guys do to avoid wasting time maintaining acceptance tests that are no longer true or necessary and inflating the build duration?
The problem with tests redundant seems to be the consequences of acceptance tests applied to the GUI of the system. My take on this is that it makes it harder to understand the system from the point of view of business value. Also, as Daniel noted, acceptance tests are to enable communication. There is no point for example replace them with integration or system test IMO.
In the case of tests not updated, I don’t think they are temporary but that we are in continous learning.
Carlos,
We use GUI to drive tests since decades ago and this was never the real cause for duplicated tests. The cause for duplicated tests is because tests have bad quality.
Acceptance tests ARE temporary just because acceptance criteria ARE temporary. There’s no real doubt here as one story will break the acceptance criteria for previous stories all the time.
Example: If you play an story named “User should be able to login” that could mean that after logging in a user should be directed to the system’s home page. You acceptance tests verify this criteria.
After playing this card you get a new one where name “Improved login” where the user should be related to, say, his favourite page. Not only you have to write acceptance tests for this but you change to change the former story’s, played hours ago.
This is a simple example but it’s not hard to imagine how this scales up in a project with more than one developer. And I’m not talking about teams new to agile but with seasoned developers, BAs and QAs. The tests are not updated just because in a large codebase it is *very* hard to identify which old criteria your story just broke, specially because sometimes old tests won’t break at all.
Given that the question is that if there is any value in keeping old acceptance tests around, as they have to be changed all the time as what were their acceptance criteria change.
Also, I think you may be saying ‘acceptance tests’ but you mean ‘user acceptance test’. The former is related to a story while the latter is related to the system and its features. Stories are not features, they are just schedulable pieces of functionality.
Phillip,
A couple of comments.
First I think your original post suffered by not giving a concrete example of these wasteful acceptence tests. You did so in a comment above and that helps me understand your point.
I would counter with the idea that tests are meant to document a system. If the function of a login process changes you would expect a broken test and I would expect someone on my team to fix it or throw it away if it is no longer valuable. If I break 20 tests by introducing a change that is a different problem. That is a problem of duplication in my tests. Whether they are unit tests or any other kind of test that is unacceptable. Why would accptence tests be a special case of test? If there is lots of duplication in your acceptence tests then the job of your team is to find a way to remove that duplication. I would shy away from dumping on the nature of acceptence tests and look for problems with your particular acceptence test writing process. Maybe that is what you are doing and just describing it differently than I would.
Also the problem of build time skyrocketing. Carlos mentioned avoiding the GUI in your tests. That is often a way to speed up tests by an order of magnitude. Another strategy is to remove all this duplication in your acceptence tests.
Hi Jeff,
Thanks for your comments. I guess this point is a bit too abstract, I’ll probably write a post with an example of what we’re doing.
The argument for documentation (I prefer to refer to that as ‘executable specification’) is a nice one. Anyway, I don’t think acceptance tests add much value to that per se. I still have unit, integration and system tests so I don’t think that not having tests that describes something that is temporal in nature is missed at all (remember that system tests are part of our build and written in dev-friendly tools).
I think that acceptance tests are important as documentation for the story, not for the system. What should be represented in an permanent executable spec is not the change (the story) but the final result. Therefore I think acceptance tests *are* different. They test the very specific acceptance criteria for a story; and a story is just a modification in a system.
The change on the user login screen may break acceptance tests for stories that were played a long time ago. My point is that instead of maintaining executable specs for those old stories you should maintain executable specs for the system itself. Stories are not like use cases in the sense that you can describe a system by those. Stories are just scheduled change.
Notice that in my current project we happen to have a decent QA team that work together with the development team and make their tests part of the build. If that wasn’t the case I probably would refrain from even suggesting the ‘acceptance test promoted to system test’ strategy.
Avoiding duplicated acceptance criteria verification is a very nice idea, but I found it hard to implement in practice. It is very common for a given acceptance criterion to be repeated between stories and if you have story-based testing (in opposite of the scenario-based testing provided by system tests) it is often too hard to verify all acceptance criteria for multiple stories in a single test.
I don’t think that not using UI (or a real user API) for a system is valuable for an acceptance test. The acceptance criteria should be about externally observable behaviour and that’s the level where the acceptance tests should be written. Actually, if we don’t use UI-drivers for acceptance tests probably just unit and integration testing would be enough (and that’s Sarah’s point, I guess).
Hi Philip,
I have commented on Sarah’s article and I ran into yours. My comment is still the same, what about the financial aspect of acceptance testing (eg. clients withholding payments because of a failed acceptance). The acceptance will still be demanded by the client (where some do their best to discover flaws in the testing)…
Hi,
I’m not saying you shouldn’t do acceptance tests. What the post suggests is that after your got your story done you migrate the acceptance tests to the functional test suite.
In your example you would still have the acceptance tests demanded but once you have those working you just convert them into functional testing. That is probably good enough as you won’t play the same story twice and acceptance tests are tied to specific stories.
Anyway there is a difference between what’s the ideal process for a given project and what the client demands from you. Clients demand all sorts of things, if yours require a passing acceptance test to pay the bill then you have to create one or prove it worthless to your client.
It is the same situation when your client requires BDUF or any other non-agile (i.e. worthless) artifact. This has nothing to do with software development itself, it is about contracts and agreements between companies.
Thanks for your comment.