If A=B and B=C, then A=C
Software Testing Rule #1: We can't test everything!
Well I guess in theory we could. But is it practical or feasible? Typically not. We rarely have the time or resources needed to do it. Even if we did, the product would be so expensive that no one could afford it. So how do we strike that delicate balance of testing "enough"?
Using Equivalence Class Testing or Equivalency Class Partitioning is one technique we can implement to reduce test times, and the number of test cases needed, to a more resonable number while still ensuring adequate test coverage.
Let's look at a simple test - a web-based form. The first text box on the form is a text field that is supposed to accept 5 upper-case alpha characters - capital letters (English only). If we were to fully test this text field, using every possible combination of valid characters, there would be more than 11 million possible combinations (26 x 26 x 26 x 26 x 26). Over 11 million test cases - for a single field on a form! Even with an automated test tool, and a really fast system, the task is virtually impossible (and really boring). If the field were supposed to allow both upper and lower case letters, the number of possible combinations is now exponentially higher (in the range of about 380 million). The more valid characters, the more impossible the task becomes. And that's for just one field on a form! Equivalence classes to the rescue!
Simply stated, an Equivalence Class is a set of data that we can assume will be treated the exact same way by the system. For any system input there are esentially 2 types or sets of data - valid data, and invalid data. We refer to these sets of data as "classes" In the above example, we can reasonably expect that the system will treat ABCDE as valid as well as any other 5 upper-case characters. As long as the data meets the rules of the class - it should be treated as valid. Any data that does not meet the rules of the class is invalid. The assumption is that if the system accepts any one valid member of a class, it will accept all of them. This is the valid equivalence class (Ve). Conversely, if the system rejects an invalid class member - it will reject all invalid class members. This is the invalid equivalence class (Ie). A good test uses both valid and invalid data. Sounds simple, right? Let's look at an example:
The test scenario: The test object is a simple user registration form, with the basic user information form fields (last name, first name, street address, city, state, zip code). Let's take a look at the last name field. Upon reviewing the system requirements we learn that this is a 15 character text field that allows only upper case characters. The valid equivalence class is:
Ve = {Any 15 upper case alpha characters}
Ve1 = {ABCDEFGHIJKLMNO}, Ve2 = {BCDEFGHIJKLMONP}.....and so on, Eleventy-jillion possible combinations - Pick one, any one! Basically, any set of 15 upper-case characters will meet the rules of the valid class. Pick any one set of data that meets the class rules and you have designed your first test case. You may have multiple valid classes. See, I told you it was simple. Now let's look at invalid classes.
An invalid class consists of any data that does not meet the rules of the valid class. In addition to valid data, a good test will also use invalid data. Like the valid equivalency class, the are millions of possible invalid data combinations. Are we going to test all of them? Of course not. In our example, invalid classes would include any lower-case characters, numbers, or special characters. You may, as in this case, have multiple invalid classes.
Ie = {Any 15 character combination with either a lower-case alpha character, number, or special character
Ie1 = {ABCDEFGHIJKLMNo}, Ie2 = {ABCDEFGHIJKLMN1}, Ie3 = {ABCDEFGHIJKLMN%}....
So how do we use this in testing our application? Simple. Create a positive test case for each valid class that you discover. Create a negative test case for each invalid class. In our very simple example above, if testing only this one field on a form, we have found 4 test cases to run - 1 positive, 3 negative. See I told you it was simple!
Try some on your own - what are the valid and invalid classes for:
- First Name
- Zip(Postal) Code
- Phone Number
Class dismissed! :o)
Feel free to contact me if you have any questions - dwhalen14@gmail.com!
Next: Boundary Value Analysis & Testing.
Well I guess in theory we could. But is it practical or feasible? Typically not. We rarely have the time or resources needed to do it. Even if we did, the product would be so expensive that no one could afford it. So how do we strike that delicate balance of testing "enough"?
Using Equivalence Class Testing or Equivalency Class Partitioning is one technique we can implement to reduce test times, and the number of test cases needed, to a more resonable number while still ensuring adequate test coverage.
Let's look at a simple test - a web-based form. The first text box on the form is a text field that is supposed to accept 5 upper-case alpha characters - capital letters (English only). If we were to fully test this text field, using every possible combination of valid characters, there would be more than 11 million possible combinations (26 x 26 x 26 x 26 x 26). Over 11 million test cases - for a single field on a form! Even with an automated test tool, and a really fast system, the task is virtually impossible (and really boring). If the field were supposed to allow both upper and lower case letters, the number of possible combinations is now exponentially higher (in the range of about 380 million). The more valid characters, the more impossible the task becomes. And that's for just one field on a form! Equivalence classes to the rescue!
Simply stated, an Equivalence Class is a set of data that we can assume will be treated the exact same way by the system. For any system input there are esentially 2 types or sets of data - valid data, and invalid data. We refer to these sets of data as "classes" In the above example, we can reasonably expect that the system will treat ABCDE as valid as well as any other 5 upper-case characters. As long as the data meets the rules of the class - it should be treated as valid. Any data that does not meet the rules of the class is invalid. The assumption is that if the system accepts any one valid member of a class, it will accept all of them. This is the valid equivalence class (Ve). Conversely, if the system rejects an invalid class member - it will reject all invalid class members. This is the invalid equivalence class (Ie). A good test uses both valid and invalid data. Sounds simple, right? Let's look at an example:
The test scenario: The test object is a simple user registration form, with the basic user information form fields (last name, first name, street address, city, state, zip code). Let's take a look at the last name field. Upon reviewing the system requirements we learn that this is a 15 character text field that allows only upper case characters. The valid equivalence class is:
Ve = {Any 15 upper case alpha characters}
Ve1 = {ABCDEFGHIJKLMNO}, Ve2 = {BCDEFGHIJKLMONP}.....and so on, Eleventy-jillion possible combinations - Pick one, any one! Basically, any set of 15 upper-case characters will meet the rules of the valid class. Pick any one set of data that meets the class rules and you have designed your first test case. You may have multiple valid classes. See, I told you it was simple. Now let's look at invalid classes.
An invalid class consists of any data that does not meet the rules of the valid class. In addition to valid data, a good test will also use invalid data. Like the valid equivalency class, the are millions of possible invalid data combinations. Are we going to test all of them? Of course not. In our example, invalid classes would include any lower-case characters, numbers, or special characters. You may, as in this case, have multiple invalid classes.
Ie = {Any 15 character combination with either a lower-case alpha character, number, or special character
Ie1 = {ABCDEFGHIJKLMNo}, Ie2 = {ABCDEFGHIJKLMN1}, Ie3 = {ABCDEFGHIJKLMN%}....
So how do we use this in testing our application? Simple. Create a positive test case for each valid class that you discover. Create a negative test case for each invalid class. In our very simple example above, if testing only this one field on a form, we have found 4 test cases to run - 1 positive, 3 negative. See I told you it was simple!
Try some on your own - what are the valid and invalid classes for:
- First Name
- Zip(Postal) Code
- Phone Number
Class dismissed! :o)
Feel free to contact me if you have any questions - dwhalen14@gmail.com!
Next: Boundary Value Analysis & Testing.
Add a Comment
-
Comment by Dave Whalen on December 21, 2009 at 5:21pm - I have moments of greatness....lol
-
Comment by Joe Strazzere on December 21, 2009 at 5:06pm -
"Anything short of testing ALL possible entries produces risk. The question is....is it a risk we can live with in the interest of saving time. All testing involves risk. There is always a chance we will miss something. The ultimate goal is to balance risk with time, budget, etc."
Well said!
-
Comment by Dave Whalen on December 21, 2009 at 4:42pm
-
Thanks for the discussion guys. A couple of points...
1. I tried to keep this a very simple and constrained example for the purpose of presenting the concept. Are there other classes - there may be depending on your system's individual requirements. For example, you may want to verify that a Last Name field, in real life - allows spaces, hyphens, etc. My example was extremely limited on purpose. I allowed ONLY upper case alpha characters. If you see more classes - wonderful. As long as you are not creating millions of tests, the concept still applies.
2. Its all about risk. Are we making assumptions? Absolutely! Anything short of testing ALL possible entries produces risk. The question is....is it a risk we can live with in the interest of saving time. All testing involves risk. There is always a chance we will miss something. The ultimate goal is to balance risk with time, budget, etc.
3. Ainars - Missing field tests are a whole different class of tests. For a typical form I will always do 2 positive tests. First, complete the form, using valid data, in only the required fields. Second - valid data in all fields (required and optional). When these 2 test pass, I'll start breaking it. I usually try to submit an empty form first. Next, I like to focus on required fields. Omit a required field and submit the form. Then the next required field, and so on. Then I may focus on individial fields with invalid data types (equivalency classes), field lengths (boundary value analysis), etc. Much depends on the time and tools that I have available.
I tried to keep this example extremely simple. Is it a real life scenario? Probably not. You can nit pick it to death, but remember the context of what we're doing here!
Again - thanks for the feedback. Great points!
Oh and FYI - I have rarely, if ever, seen Last Name and First Name NOT seperated as different fields. Again, it depends on the context. How are we using the data? For example - look at US Postal Codes (Zip Codes). A valid Zip Code consists of 5 numbers (we're excluding the "plus 4" for now). But is 000000 a valid zip code? Nope. Do we care enough to build code to validate it? Maybe. How are we using this address information? If it involves shipping or mail then I would say yes - we need a more extensive test. If not, maybe just checking for 5 digit numeric values is enough.
-
Comment by Joe Strazzere on December 21, 2009 at 12:58pm
-
"There's too much assumptions leading to misinterpretations "
Agreed. That's always the way with softare!
-
Comment by Ainars Galvans on December 21, 2009 at 8:04am -
Joe,
I assume (again) a single form contains both First Name and Last Name (never seen those fields separated on different forms). So if I pass any positive test for First Name it follows that I've passed positive test for Last name and V V. Perhaps I should have asked Dave what's his definition of a test and how he count them beforehand, i.e. does a fiedl test end with field filling in, leaving a field, submitting a form, retrieving the entity afterwards? There's too much assumptions leading to misinterpretations
-
Comment by Ahmed Ashraf Elkadee on December 20, 2009 at 3:36pm - Thanks alot Joe for the very simple way you used explaining this theory
-
Comment by Joe Strazzere on December 20, 2009 at 3:21pm
-
Ainars,
Whenever your goal is "ensuring adequate test coverage" (as Dave stated), you are making assumptions.
And assumptions must be made, otherwise you can never be done testing.
The trick is to take the right risks, make the right assumptions, and balance those assumptions against the risks. This is what skilled practitioners do every day.
But I don't understand how you can come up with zero test cases for any field. Are you saying you wouldn't bother to execute any tests that probe the first name field? I guess we disagree there.
-
Comment by Ainars Galvans on December 20, 2009 at 8:10am
-
I want to extend a bit what Joe started. Equivalence classes always bring an assumption - assumption that all items in one equivalence class are processed the same way. But how do you know that empty or one letter is process the same way as 10 letter input? How do you know that all special characters are processed the same way? How do you know that it does not matter than leading space or quotation mark isn't processed differently? How do you know that tabulation symbol or newline character (if copy-pasted from notepad) isn't processed differently? Paste 1 GB large block from clipboard?
You don't know, you assume that. Your assumption allows to reduce testing to 4 cases only, but introduces risk of missed bugs. As far as you know the ris it is fine. And my answer for "First Name" is - zero test cases. My assumption is that it is processed just the same way as last name and the risk to miss bugs specific for First Name is very low.
And by the way assumption that in in theory we could test everything is also wrong. As far as I remember there is a theorem that says it is impossible to have an universal testing algorithm that would be able to test any code against it's specification (even if specification is formal and thus non-ambiguous mathematically).I mean theoretically impossible - it is not problem of limited time, it is problem of limited algorithm.
-
Comment by Joe Strazzere on December 19, 2009 at 12:34pm
-
Dave,
This is a good introduction.
My main quibble is that you are leading the reader to the incorrect conclusion that your Last Name Field example has only 1 positive and 3 negative classes. I'd hate for the beginning tester to think that there are only 4 tests needed to be run!
Find Us Elsewhere
Nice Things
Bloggers
© 2010 Created by Rosie Sherry
You need to be a member of Software Testing Club - An Online Software Testing Community to add comments!
Join Software Testing Club - An Online Software Testing Community