So, I’m not an Agile Guru of any kind -and by working for ThoughtWorks each day I find out that I still have to learn a lot about the whole agile thing- but my last post caused some interesting comments here in the blog and among my friends and colleagues so I’d like to extend a bit more on some of those topics.
I know this is a polemic topic and this text represents my personal take. Also this post got too much “meta” so feel free to skip this one.
My main concern is that I think people value Acceptance Testing too much. Don’t get me wrong: automated acceptance tests are essential to a healthy agile process, the problem is what we do with them after we deliver a story.
I think my main divergence point with the commenters is based on the fact that we face user Stories as different things. People often refer to User Stories as characteristics of a system. For me, User Stories are just schedulable change. They are just state transitions.
So, suppose you have a system in where the user logs in and is redirected to a generic, vanilla web page. There is a card in you wall that’s about changing this behaviour, instead of falling into that the user will get to a personalised home page. The Story is not about the feature, is about implementing the feature.
So now the system has different behaviour at that state and the user story is the act of transitioning from one state to the other. To most aspects but scheduling, what’s important is not the transitions but the states. One reason that stories are not that relevant is the fact that stories will often override each other.
In the example above, we probably had something like this at start:
And then we played the card the added the customised welcome page.
And let’s say that after that we had a card adding an error handling feature.
Notice that the acceptance criteria from the first transition are completely overridden after the subsequent stories. That story is not reflected by the current (yellow) state anymore. At this state it is irrelevant if the first story was ever played.
That’s for me the major difference between Stories and Use-Cases. When someone is being introduced to agile methods it is quite common to tell she that stories are like simplified Use-Cases. That’s not exactly true.
Use cases describe a system, user stories are just state transitions. Mike Cohn says something interesting:
Another important difference between use cases and stories is their longevity. Use cases are often permanent artefacts that continue to exist as long as the product is under active development or maintenance. Stories, on the other hand, are not intended to outlive the iteration in which they are added to the software. While it is possible to archive story cards, many teams simply rip them up.
So, if a story won’t outlive the iteration it was played what happens after that? I’ve got different answers from different people. Some say that the business value represented by the story is merged to the system. I prefer to think that the role of the story was to change the state of the system, after that transition is completed the story is gone forever.
As we saw in the little example above, it is very hard to go from one system to the stories that were played in its development.
In rewrites of systems developed using agile methodologies it is often the case that new stories are created based on the current state of the legacy system. The old system’s stories –if available at all- are used just as reference.
And that gets back to the point of my previous post: acceptance tests. As we saw above user stories are just state transitions and acceptance tests are tools used to verify when a state transition was completed.
If they are temporary, why keep those artefacts for a long term? In shorter projects I never had a problem in keeping acceptance tests around but in larger projects they can drive the code base into the problems stated in my previous post.
I described an approach we are using in my current project –medium-sized scope, about 20 people, 6 months long. In this project we do create acceptance tests when a story is played but we don’t keep those for a long time. When QA people accept a story to be tested they take ownership of the acceptance tests and merge them into their system tests.
That practice is far from ideal but at least it:
- Reduces duplication of effort: You can get the leaner as you want with acceptance tests and they will still just repeat the same test, at the same level, that is inside the system test
- Reduces build time: Less repeated tests over the build means shorter builds. Shorter builds mean shorter feedback cycles.
- Reduces the overhead caused by a story that overrides an older one: If acceptance tests are part of your code base you have to search for all old acceptance criteria that may or may not be valid anymore after you play a story –and they won’t always break when invalidated. If you have a large number of tests it is very easy to forget about some. By having system tests only you have to care only about the final state of the system.
- Still gets the benefits from acceptance testing: We still write acceptance tests and still use those to “tell us when something is done”, as it’s ” the best way to avoid putting too much, or too little, time and effort into it” like Mike Cohn says. We just accept the fact that acceptance tests are related to a story, thus are temporary.
One thing that is worth remembering is that we are applying this strategy in a team with competent QA people. If your team doesn’t have someone to take ownership of the tests or you can’t rely on that person I’m not sure I’d suggest something like that.
Obs: Those diagrams may be a bit misleading. For me, an user story represents a transition to a state but not necessarily specify from which state the development will start. I’d say that the “from” state is sort of defined in estimation.
To estimate one story you have to make assumptions about where you are coming from. If I tell you “implement the customised login” the estimation will vary a lot depending if there is already the vanilla login page or not. The story itself doesn’t care about the previous state but an estimative is something that binds a story with a “from” state.
Update: Pat Kua posted an article on the topic. My favourite quote from that:
As I like to say, stories represent the system in a certain snapshot of time. The same thing that lets us deliver incremental value in small chunks just doesn’t scale if you don’t consolidate the new behaviour of the system, with its already existing behaviour. For developers, the best analogy is like having two test classes for a particular class, one that reflected the behaviours and responsibilities of the system at the start, and one that represents the new behaviours and responsibilities of the system right now. You wouldn’t do his at a class level, so why should you do it at the system level?
Yes! I’ve said something like this before; it makes total sense to me. I once described a story as a “diff” to a system. Just as in your explanation, it’s impossible to reverse-engineer the diffs that created the software. Your explanation is very good, and I will share it with my co-workers. Thanks!
Phillip,
I have been working with acceptance test automation on my last few projects but our test never were at a story-based level. All the story needs to have a test, but not necessarily the test scope/size have to be limited by one story.
The user level test that we do here (we call it functional tests) are made by the developers, using Selenium RC. I think that what we do is what you call system test, where a single test method verifies a whole functionality.
In your scenario, why do the developers write the (story-base) acceptance test? Why don´t they put the test directly into the system test? Do you think that a QA team is needed for this?
Well, the size of the teams that I worked with are relatively small, so maybe I not seeing the big picture, but in our case we did not think that a QA team is needed for this.
@Dan
Love the diff example. Actually, I can’t stop thinking if source code state transition and story state transitions are just correlated or if they have some causality.
@Fabricio
System tests (or any other name for the system that tests a given functionality of the system) are different from acceptance tests. When you do a system test you verify that that feature is working end-to-end. In an acceptance test you verify only the acceptance criteria for the story, if the system has a bug in that functionality but that bug won’t invalidate the acceptance criteria the test still pass.
Phillip,
I think your new scenario looks better than the previous one. But I would go one step further: if you do not keep the acceptance tests, why do you write them? It might seen a big change, but in my experience, the system tests (or scenario-based tests), if written by the developers, are enough to say if a story is correctly implemented. You would not have the fine granularity of acceptance tests, but I think that it does not justify the overhead.
Fabricio,
I guess your question has a dangerous corollary: why do we keep stories if they’re temporary? Why not use, say, use cases that are permanent?
Being temporary doesn’t make something waste. For me the waste is on keeping these temporary artefacts forever in your build.
If you are working with user stories I really can’t see how system testing will validate your work. As I said in my previous comment System tests (that are QA’s responsibility and should only be written by developers when the project has no budget for QAs)validates the full, end-to-end, functionality. Using the words I’ve used before in this blog post system tests test end states. Stories are not end states, they are transitions.
In a more practical example, suppose two people working on different stories at once. Dev #1 finishes his task and is ready to move the card to done but at this very time dev #2 checked in some code that is not related directly to dev#1’s code and has a bug.
If we use acceptance tests it is very unlikely that dev#1’s work will be affected by that bug. If we rely only on end-to-end testing and if dev#2’s work -although unrelated directly- is part of the same functionality as dev#1’s than both cards can’t move until the bug is fixed.
In my opinion, the big advantage of any kind of test automation is the possibility to make regression testing. If you want to verify if a story is done only while you are implementing it, you can do the test manually, it’s faster. If you do not keep your acceptance tests you loose the regression part, so in this case I don’t see a big gain in automating it.
System tests and acceptance tests they both validate end states. Stories are transitions, but acceptance test validates the end state that the transition led the system to. The difference is in the scope, that is shorter in acceptance tests.
I agree that your example can happen and make system tests harder to validate a story. In my experience it happened a couple of times but it is not a very common situation. Before cheeking in any code, dev #2 is supposed to run the role system test in his code base and it helps to early identify any kind of bug.
Fabricio,
i understand your opinion but I heavly disagree with it. Manual testing for acceptance criteria is a big waste and very inneficient.
It’s a waste because you have to repeat *the same* steps over and over, and the time you spend will not only represent time that you wasted doing something that could be automated but will increase your feedback loop cycle.
It’s very inneficient because -just like any other kind of automated testing- it is very common to forget or bypass steps that are necessary.
My answer to the ‘lose the regression’ argument is expressed in this post and in the previous one. As they say we don’t throw away those tests:
If you accept my argument in this post than I can’t really understand how acceptance test would validate end tests. Stories are transitions thus you validate the transitions when accepting a story. A state is composed of dozens of stories and if you have accetpance and system tests acting in the same point they wouldn’t be different tests. I think you may not have understood my point.
About the example, tests are added all the time so one bug can be found, a test can be added and that can impact stories being developed by any other dev.
I agree that manual testing are very inefficient. I only mentioned it in case where automated tests can’t guarantee regression. And if I understood right, in your project the regression tests are guaranteed by the system tests, not by acceptance tests. But, if the system tests are made by merged acceptance tests and you have regression, I think it is a valid scenario.
In my project when a bug is found by a test, we try to isolate that test so it does not compromise the whole functionality being tested. After the bug is corrected, the test is merged. It’s not the best scenario, I know, but I think that it is best then writing acceptance test in isolation and then merging them is system tests. In any case, we have only 6 developers on the project. If we had 20, maybe this strategy would not be valid.
I may be missing something here but I can’t really any difference between what you just described (merging bug verification tests to systest) and what I described in both posts, except that looks like you do this only for bugs.
I think the main difference is that in my case story-based acceptance tests are not written. The isolated tests for bugs are not bound to any story, even though sometimes a story is written to correct the bug. But I agree it’s not a big difference: we both end up with a great coverage at a system level.
Interesting stuff!! I’m writing ’bout Stories (MundoJava, will be published on November: “stories are arrows thrown by the user that change the system”). I agree that Stories have short life cycle and acceptance tests have longer life cycle (but if they are based on stories they break and repeat a lot).
Maybe we need a Use Case BDD runner based on scenarios. I guess that we still need to automate acceptance tests but stories aren’t whe best choice because stories are too small (BDD on stories, story runner). Just something for us to think about:
1. A DSL for Use Cases (pre-post conditions, valid scenarios, etc…)
2. A DSL for Data/Object representation
3. Merge (1) and (2) to turn (1) executable (scenario runner?)
4. Forget about Acc. Test based on Stories (stories are just change requests to executable Use Cases)
5. Use Case training for every “agilists” we know…
Too complex?
Cheeeers!
Rodrigo,
I like the idea of Use Cases DSL, shouldn’t be hard to implement but I don’t work with use cases for a long time now.
About the other suggestions, I still find use cases too technical and complex to be efficient. If experienced software developers after years studying the field can’t understand those properly let alone the poor business person.
Also, I’m not sure what you mean by “Forget about Acc. Test based on Stories”. If you have stories but don’t test them I can’t see how you would accept their implementation. I guess what you are describing is pretty much what Fabricio said some comments ago and I’d point out the same arguments against it as before.
cheers
If by “characteristics of a system” you mean statements
about a problem that the system is intended to solve, yes, user stories are something about characteristics of a system.
But problems are different and has different priorities too, the login example was perfect to show how bug tracking systems can also be useful tools.
Carlos,
By that I pretty much mean “what a system do”. As the text states that user stories are change on a system, I guess it is quite clear that they are *related* to what a system does but they do not *define* those by themselves. As the text says that’s one difference between use cases and stories -use cases define what a system does, user stories define a piece of change. I think that Dan’s comparison with code changes in the first comment make make my point clearer.
I couldn’t really understand the context of your second comment. Issue trackers are great tools but I don’t understand how they’d fit in this text.
Phillip,
In his new blog post[1], Fowler exposes a scenario with this “new” approach to acceptance tests:
“Each time they play a new story, they decide how to update this suite to reflect the new behavior. This breaks the simple story-to-test relationship, but results in a much simpler and coherent suite of tests.”
The project he describes has a not so common set of characteristics, but maybe it is a nice approach for a broader range of project.
[1] http://martinfowler.com/bliki/NashvilleProject.html
Hi Philip,
great thread this one. I’m just curious about QA’s role in this scenario you depicted.
User stories changes the system, automated acceptance tests are written so that sprint progress could be monitored (cards move if acc tests go green, cards move back if acc tests go red). I suppose underlying code is crafted by TDD, so there is a suite of unit tests.
And then “(..) QA people accept a story to be tested they take ownership of the acceptance tests and merge them into their system tests.”
So, QA team is working along side with dev team, producing automated system tests? Or this system tests are manual (even if partially)?
Let’s suppose you are in 4th iteration. What’s run on build machine? Unit testing of entire code, acc tests written for this 4th iteration and *all system test ever written* (kind of a regression)?
Regards,
Camilo.