I’m not saying you shouldn’t do acceptance tests. What the post suggests is that after your got your story done you migrate the acceptance tests to the functional test suite.

In your example you would still have the acceptance tests demanded but once you have those working you just convert them into functional testing. That is probably good enough as you won’t play the same story twice and acceptance tests are tied to specific stories.

Anyway there is a difference between what’s the ideal process for a given project and what the client demands from you. Clients demand all sorts of things, if yours require a passing acceptance test to pay the bill then you have to create one or prove it worthless to your client.

It is the same situation when your client requires BDUF or any other non-agile (i.e. worthless) artifact. This has nothing to do with software development itself, it is about contracts and agreements between companies.

Thanks for your comment.

I have commented on Sarah’s article and I ran into yours. My comment is still the same, what about the financial aspect of acceptance testing (eg. clients withholding payments because of a failed acceptance). The acceptance will still be demanded by the client (where some do their best to discover flaws in the testing)…

Thanks for your comments. I guess this point is a bit too abstract, I’ll probably write a post with an example of what we’re doing.

The argument for documentation (I prefer to refer to that as ‘executable specification’) is a nice one. Anyway, I don’t think acceptance tests add much value to that per se. I still have unit, integration and system tests so I don’t think that not having tests that describes something that is temporal in nature is missed at all (remember that system tests are part of our build and written in dev-friendly tools).

I think that acceptance tests are important as documentation for the story, not for the system. What should be represented in an permanent executable spec is not the change (the story) but the final result. Therefore I think acceptance tests *are* different. They test the very specific acceptance criteria for a story; and a story is just a modification in a system.

The change on the user login screen may break acceptance tests for stories that were played a long time ago. My point is that instead of maintaining executable specs for those old stories you should maintain executable specs for the system itself. Stories are not like use cases in the sense that you can describe a system by those. Stories are just scheduled change.

Notice that in my current project we happen to have a decent QA team that work together with the development team and make their tests part of the build. If that wasn’t the case I probably would refrain from even suggesting the ‘acceptance test promoted to system test’ strategy.

Avoiding duplicated acceptance criteria verification is a very nice idea, but I found it hard to implement in practice. It is very common for a given acceptance criterion to be repeated between stories and if you have story-based testing (in opposite of the scenario-based testing provided by system tests) it is often too hard to verify all acceptance criteria for multiple stories in a single test.

I don’t think that not using UI (or a real user API) for a system is valuable for an acceptance test. The acceptance criteria should be about externally observable behaviour and that’s the level where the acceptance tests should be written. Actually, if we don’t use UI-drivers for acceptance tests probably just unit and integration testing would be enough (and that’s Sarah’s point, I guess).

A couple of comments.

First I think your original post suffered by not giving a concrete example of these wasteful acceptence tests. You did so in a comment above and that helps me understand your point.

I would counter with the idea that tests are meant to document a system. If the function of a login process changes you would expect a broken test and I would expect someone on my team to fix it or throw it away if it is no longer valuable. If I break 20 tests by introducing a change that is a different problem. That is a problem of duplication in my tests. Whether they are unit tests or any other kind of test that is unacceptable. Why would accptence tests be a special case of test? If there is lots of duplication in your acceptence tests then the job of your team is to find a way to remove that duplication. I would shy away from dumping on the nature of acceptence tests and look for problems with your particular acceptence test writing process. Maybe that is what you are doing and just describing it differently than I would.

Also the problem of build time skyrocketing. Carlos mentioned avoiding the GUI in your tests. That is often a way to speed up tests by an order of magnitude. Another strategy is to remove all this duplication in your acceptence tests.

We use GUI to drive tests since decades ago and this was never the real cause for duplicated tests. The cause for duplicated tests is because tests have bad quality.

Acceptance tests ARE temporary just because acceptance criteria ARE temporary. There’s no real doubt here as one story will break the acceptance criteria for previous stories all the time.

Example: If you play an story named “User should be able to login” that could mean that after logging in a user should be directed to the system’s home page. You acceptance tests verify this criteria.

After playing this card you get a new one where name “Improved login” where the user should be related to, say, his favourite page. Not only you have to write acceptance tests for this but you change to change the former story’s, played hours ago.

This is a simple example but it’s not hard to imagine how this scales up in a project with more than one developer. And I’m not talking about teams new to agile but with seasoned developers, BAs and QAs. The tests are not updated just because in a large codebase it is *very* hard to identify which old criteria your story just broke, specially because sometimes old tests won’t break at all.

Given that the question is that if there is any value in keeping old acceptance tests around, as they have to be changed all the time as what were their acceptance criteria change.

Also, I think you may be saying ‘acceptance tests’ but you mean ‘user acceptance test’. The former is related to a story while the latter is related to the system and its features. Stories are not features, they are just schedulable pieces of functionality.

In the case of tests not updated, I don’t think they are temporary but that we are in continous learning.