Tests with 100's of steps - am I missing something here?
Twice in the last 6 months I've met companies who told me their methodology includes writing test cases with hundreds of steps in them.
When I asked if they meant test suits with many test cases that amounted to hundreds of steps they corrected me and explained they actually have a small number of Test Cases, but each one has hundreds of steps, and they even showed me their Excel and Word docs.
For me as a tester, the idea of having a tests that expands 100s of steps seems incredibly big and even annoying (it might be only a psychological factor, but it still does!), and for me as a manager I think it is an ever biggest headache to try and understand what passed, what failed, who ran, and what do I need to re-run during the process... And these are only the 2 first things that jump to mind.
I wanted to know if I am missing something and if this is more common than what I thought up to now? Or if these are only some isolated cases...
When I asked if they meant test suits with many test cases that amounted to hundreds of steps they corrected me and explained they actually have a small number of Test Cases, but each one has hundreds of steps, and they even showed me their Excel and Word docs.
For me as a tester, the idea of having a tests that expands 100s of steps seems incredibly big and even annoying (it might be only a psychological factor, but it still does!), and for me as a manager I think it is an ever biggest headache to try and understand what passed, what failed, who ran, and what do I need to re-run during the process... And these are only the 2 first things that jump to mind.
I wanted to know if I am missing something and if this is more common than what I thought up to now? Or if these are only some isolated cases...
Tags: case, management, test
Replies to This Discussion
-
Permalink Reply by Jared Quinert on
-
As has been alluded to, it's fairly common, but the more sensible places are either -
- setting up spreadsheets with the input data and leaving out the 'how to' steps.
- automating the input.
I've worked with both approaches. It's a lot nicer when you get things automated to the point you just invoke a script from the command line that sets up the data you want (eg. create new consumer loan, 2 applicants, 1 guarantor, amount $100000). We could choose to submit the details or not, in order to manually set parameters or change things.
Whether executing with computers or by hand, abstractions are necessary if you want to reduce maintenance costs. But in my 16 years of testing, I still find they're surprisingly uncommon.
-
Permalink Reply by Nitin Purswani on -
In my case what you say one test case i will refer it as use case .Say for example. "Create a group in STC". Now this is use case and i will have separate test case like
-creating group where group name has spaces .
then separate test case for different types of group names
-creating group and adding members while creating groups
then some separate testcases how members can be added to a group(invites,subscribe,follow)
This was only small example but having separate testcase for each scenario help me in viewing my coverage , selecting testcases for regression and mapping of bugs to particular testcase.
There is one drawback i see here is to maintain testcases as they are too much to handle but to resolve i categorize my testcase and tag them to make search effective.
And once i am done with user level test cases ,i try to find what is happening behind the scene when you try to create a group and try to do some exploratory testing with that knowledge.
-
Permalink Reply by Brian Buckmaster on -
This comes from not having a generally recognized notion of what constitutes a test case. Generally speaking a test case consist of 3 or 4 things:
1. Initial state
2. Stimulus
3. Response
4. Possible transition to another state
Assuming you know what the expected response and state transition (if any) are, then you have your pass/fail criteria.
The problem is, organizations don't want to manage all of these individual tests and so they write long sequences which can encompass tens of test cases and call the whole thing a test case. The situation is not unlike software developement, where one will write a subroutine that goes on and on for hundreds of lines.
-
Permalink Reply by halperinko on -
A Test case is what Brian has described + A SINGLE set of parameters values.
If one has to repeat this test 150 times with different values - these are different Test Cases all together - though since the test algorithm is the same - this is a single Test.
-
Permalink Reply by Pradeep Soundararajan on -
I was on a pre-consulting meet for an organization whose problem was they had two and a half pages per test case. I was curious how they figured out that two and a half pages were good for each test case. On asking them if they rate testers productivity based on test cases, they said, with a lot of excitement, "Yes", hoping that I wanted them to do it.
Their magical number was 2 test cases per hour and hence testers there were forced to do write 2 test cases in an hour and ended up writing such lengthy test cases. Of course the steps in most of the test was redundant,
"
1. Open the link http://whatthehell
2. Login with proper credentials
3. You should see the home page.
25. Enter these data in the form displayed and submit
30. The system should accept valid data and reject invalid data
Phew! Unfortunately, they had some change in strategies and some consultant convinced them that Balance Score Card methodology would be more helpful to them and I lost the deal. I don't know where they are and what balance they have achieved.
However, there aren't plenty of such organizations, just a few BUT the impact of testers moving out of the organization and joining other organizations at a much higher position may lead to this idea leaking out and infecting more.
-
Permalink Reply by Siva Annapragada on -
I have seen this kind of test cases where you have 100's steps in one test case in domains like financial, healthcare, and utilities. Basically, these type of test cases address end to end flow. My observation is, these are useful though there are some disadvantages of managing these while execution and getting necessary metrics. In one point of time, we have considered each step as a test case and started capturing results accordingly. We have even customized our tool to get necessary metrics.
The other observation is, though we have large test cases like this, we still maintained small test cases which actually tests a single function separately. We used to call them as functional test cases. These test cases basically address all your basic transactions, boundary conditions, validations, negative scenarios, etc. The large test cases with 100+ steps are used for end-to-end flow testing and this can be achieved even splitting them into small test cases and linking them to get a end to end flow. We have ensured that these large test cases covers only positive business flow. (some people call it as Happy path).
Managing test cases with 100+ tests is difficult and executing them is pain becasue you have to assign each test cases to only one individual and wait for his completion. The work distribution among your team is complex and we have plan the efforts for execution effectively to address it.
Overall I found, this type of test cases are useful since they cover general usage of the system from end to end perspective and give you confidence for deployment. For business users, it is easy them to execute these test cases in UAT.
Thanks,
Siva
-
Permalink Reply by Jason McCubbin on -
One thing that always concerns me regarding test cases with a large number of steps is the tester may loose focus and become complacent with their testing. This is especially true if the same tester is continually executing the test. Breaking up a test case into manageable blocks may eliminate this.
- ‹ Previous
- 1
- 2
- 3
- Next ›
Find Us Elsewhere
Nice Things
Bloggers
© 2010 Created by Rosie Sherry
