Attitudes to scripted tests

Replies to This Discussion

Permalink Reply by Rosie Sherry on August 21, 2007 at 2:29pm

This is what I find:

Small development teams/companies first come to me asking about testing, thinking there *has* to be a formal Test Plan and Test Cases. When I tell them there doesn't *need* to be and that as testers we can plan for things and know how to find bugs without having a long list of documents to work against. Also, an exploratory style is usually a winner when I tell them it will cost them less :)

The small companies, listen, understand and are usually happy to take on board our approach (exploratory style). They can be reluctant if the project they are working on is for a blue chip or corporate client. They need to meet their clients expectations, and if having a set of documents is one of them then there is usually no way around it.

Bigger companies have bigger budgets and are more resistant to change and different approaches. I am generalising here, but there is probably a fear if something goes wrong - having a set of documents eases that fear. It doesn't matter that it's not the best approach, the main thing that matters is that they can cover themselves.

Again it can differ if the software is for critical or financial environments, think medical or banking software.

From a personal perspective, when I have written formal test cases to great detail, they get outdated *very* quickly and usually there is no budget to maintain them.

▶ Reply to This

Permalink Reply by Glenn Halstead on August 21, 2007 at 8:53pm

Hi Bill,

I work for a large company and the testing is based on formal scripts. I think there is a reluctance to be dependant on one specific individuals knowledge about the product. The perception is that with scripted tests any of the testers can be assigned to execute those tests. In my company there are many many projects on the go; priorities and staff assignments are regularly changing. I guess scripted tests provide a feeling of confidence that the same level of testing has been performed for each release of an application even if the testing staff has changed.

There is, of course, a maintenance overhead in preparing and maintaining the scripts but perhaps thats worth while in this environment. The main goal that is achieved by the scripted tests is that management can measure and feel confident that the agreed testing has been achived.

▶ Reply to This

Permalink Reply by Anne-Marie Charrett on August 22, 2007 at 12:55am

Hi Bill, I have recently blogged on this, as its a subject close to my heart.

To summarise, my opinion is that some small software development projects that are in a state of flux either because
1) their software is not yet mature and so changes a lot
2)there is a high turn-over of staff in developers or testers
3)its an area where its important to consistently produce new features (as in the mobile networks)

These reasons make it hard to sell test scripts as each time the re-work involved negates the value of re-use of test scripts. I much prefer to see a 'scope' document which outlines the purpose of the test, but leaves steps to an intelligent tester to figure out.

Its when software is in a mature stage such as a banking system, test scripts are helpful because of the small amount of change in functionality.

My post "Value of a software process" puts this into more detail

▶ Reply to This

Permalink Reply by Thomas Ponnet on August 22, 2007 at 3:18pm

Similar to what other people say I can only add it depends. Maturity of the product is one good reason to go for scripted testing as previously said. But if you have a version 1.0 product to work on it "might" be a different story. I usually ask if there will be a version 1.1, 1.2, 2.0 etc or if this was a one off. If it's a system that will definteley develop over time (no pun intended) test scripts might be a good idea.
The other thing is the experience of the testers. I'm heading a team of more or less experienced testers that write the test scripts which we then give to an outsourced team that execute them. Not ideal but seems to be cheaper on paper...
The problem that I can see with exploratory testing is that you need some good and dedicated testers who actually find the bugs and think at every step what they are doing and what impact each function has on other areas.
I worked in the pharmaceutical industry for some time and you don't get anywhere without test scripts there due to the FDA and GLP regulations. Doesn't matter if you would be better off without, but with scripts you have something to say that yes, we tested that and signed and dated it. As Rosie mentioned, the main reason there was to cover yourself and test scripts were the rule of the day. I would think that a word like "Change" is to be avoided when working with pharmaceuticals, banks, etc. Stability has a higher value than doing a good job for less money, as long as we remember which industry we work for we can adjust testing methods.

But enough for today of that ;-)

▶ Reply to This

Permalink Reply by Bill Matthews on August 24, 2007 at 10:24am

Thanks for your comments, glad to hear I'm not alone in my findings :)

I totally agree, that it is the context of the testing that will ultimately determine the approach to testing. Personally I tend to blend scripted, automated and exploratory testing on projects; even when there is a strong reliance on completing scripted and/or automated tests I try to schedule in some focused ET sessions. Seems to work well and I can usually find some good bugs with the ET sessions.

I guess the scripted tests are a valuable source of company knowledge about the product...assuming of course it is accurate and the tests were designed by skilled testers. The lack of depth in test design skills exhibited by some testers is one of my bugbears.

Something I find interesting is that while companies like to have masses of scripts for functional testing they seem to have no issues with using exploratory testing for application security or pen-testing. Perhaps it is the perceived difference in complexity and skills between the different types of testing.

I've not worked in any sectors that are highly regulated such as safety critical, avionics or pharmaceutical but I've worked with finance, banking and medical IT systems and I've always been surpised by the lack of controls in place. I've not worked with medical systems for while so perhaps they have changed but finance and banking don't seem to have. Generally, if the testing is audited, it is a case of checking whether you used a process, planned the testing, ran the planned tests and recorded the results. There has been very little emphasis on whether the planned testing was any good, the test documents are accurate, the testers followed the scripts correctly or that the results were recoreded accurately.

▶ Reply to This

Permalink Reply by Thomas Ponnet on August 24, 2007 at 3:23pm

One of my problems I had in the pharmaceutical industry was that you just had to document everything, regardless how good or bad the tests were. We skipped whole sections of functionality, some tests were laughable but this wasn't a problem, as the rest of the tests were planned and recorded accurately and could be shown to an auditor.
If you compare that to school, when asked to write an essay about trees and you wrote one about bushes in school you would get bad marks. Under GLP that was fine as none of the auditors could tell the difference and all the underlined areas where signed and dated...

Exploratory testing is always done in my projects I'm working on, regardless if it's in the test scripts or not. It's more difficult if you have testers that can't design good tests in the first place, I'm not sure if that's just a lack of learning opportunities or if it's something in the genes - some people have it, others don't. Someone please tell me that the latter isn't true :-)

▶ Reply to This

Permalink Reply by james bach on August 29, 2007 at 2:35am

I'm trying to get people to stop thinking of scripting as an either/or situation. Good testing requires a mix of scripted and unscripted elements.

Let me put it this way: Protein is required for good health. So are carbohydrates. Protein and carbohydrates are different things, but they are often found in the same meal. Real testing is like a hamburger. It has some protein, and it has some carbohydrates.

People who say they can get along without exploratory testing sound crazy to me, as if they have put themselves on a zero protein diet. I think what they mean is that they don't eat hamburgers, but they can't tell the difference between the hamburger and the protein (in other words "I don't do exploratory testing" really means, "I like to plan my tests to some extent before I do them", but you can do that and still be exploratory!) It's the same with scripted testing. Except there is a certain kind of scripting that to me is just junk food of the worst kind: step-by-step detailed procedural test scripts. Although there are situations where these actually help a test project, in general they hurt it.

Step-by-step detailed procedural test scripts, I find, are usually vacuous, insulting, and incredibly inefficient. They do not result in many bugs being found, but rather tend to discourage bug finding. They do not engage the minds of the testers who follow them, but rather tend to make those testers wish they worked elsewhere. People who are forced to write them are implicitly discouraged from doing any testing that requires, say 200 mouse clicks or 500 key presses to perform. They tend to stick to boneheaded simplistic tests.

Such test scripts are popular, I believe, simply because they are EASILY WEIGHED, and their sheer weight tends to discourage management from looking too closely at them. What I'm saying is, I suspect people who make heavy use of test scripts are doing so either because they don't know how to test, or they are actually trying to fake the testing.

The arguments in favor of scripting are potentially powerful if applied to a lighter form of scripted testing, such as using a test coverage matrix to aid exploratory testing, or to provide a structure for an improvisational form of testing, such as scenario testing or usability testing.

▶ Reply to This

Permalink Reply by james bach on August 31, 2007 at 2:28am

Jake, as I posted on the Qaforums, you are writing as if you have not apprised yourself of the research relevant to this matter. I don't know how to say that in a way that sounds non-patronizing, I'm sorry.

It is indeed true that a lot of aerospace people seem to think that writing scripts brings fortune and glory into their testing. I believe it is also true that there are situations that require heavy scripting. But, in my experience, these are rare. I have not been on an aerospace project. But I have spoken at Boeing, and I have had backchannel communication with an avionics tester at Honeywell (they do work for Boeing) who believed that the scripting he was forced to do was interfering with his testing. I've had people come up to me, one who was testing a defense radar system, and tell me that exploratory testing, which was secretly done on his project, accounted for most of the bugs found in that system.

I have also worked with government agencies (HUD, NCUA, and the FCC), military projects (I taught Rapid Testing techniques at Eglin AFB while F-15s flew overhead), nuclear software (I've taught at Los Alamos and at Lawrence Livermore National Laboratories), the telecom industry (Alcatel, Earthlink), I started a dedicated exploratory testing team for Hewlett Packard, I worked on a safety-critical factory process control system (I resigned from that project due to their amazingly bad and badly documented testing), and in the last year I have taught and consulted on ET at three medical device companies.

I've been at this for quite a long time (20 years, now), hearing anecdotes, and hearing the arguments you are making as well as those on the other side. At a minimum, unless I'm flat out lying to you, it would seem that not everyone in your world agrees with you about the value of scripting.

By the way, I have done scripting that is more expensive and more thorough than I suspect anyone does in your world. It was for a court case. I spent about $200,000 of my client's money (all of it in billed hours for myself and my team) preparing for a 10 minute test session. I used a call and response script, and practiced the test until I could do it with absolute precision and repeatability. It didn't find any bugs of course. It was intended to demonstrate a small number of functional facts about the system. As a bug finding test, it was weak. The test was filmed and annotated by a production crew. I talked through it on the stand.

I have done heavy scripting. And it is damn expensive to do well.

▶ Reply to This

Permalink Reply by dsquared on August 30, 2007 at 1:41am

James said:

"Step-by-step detailed procedural test scripts, I find, are usually vacuous, insulting, and incredibly inefficient. They do not result in many bugs being found, but rather tend to discourage bug finding. They do not engage the minds of the testers who follow them, but rather tend to make those testers wish they worked elsewhere. People who are forced to write them are implicitly discouraged from doing any testing that requires, say 200 mouse clicks or 500 key presses to perform. They tend to stick to boneheaded simplistic tests."

So, for James and all the others who share this opinion of scripts, does that mean that automation is doomed and has no place in testing? I challenge anyone, including James, to automate an exploratory test and have it be repeatable and reliable. I will also go out on a limb and make the assumption that this means that regression testing is a "boneheaded simplistic test" since it is usually very repeatable and often very scripted.

How about the whole world of performance testing? Should that be exploratory and unscripted as well?

And how about the SOX world, where documented tests and results are REQUIRED? Did we forget about that? Rosie said that she tells companies that they don't *need* a formal test plan and test cases. I think in the SOX world, that will definitely not fly and they are most definitely needed there.

And let us not forget the litigous nature of our world nowadays. Lets say that we have a mission critical piece of software somwhere and a failure results in a lawsuit. What happens when we don't have any scripted tests to back up the fact that the software was tested. Can you spell "decision for the plaintiff"?

Of course, I know that James really espouses a mix of exploratory and scripted. In reality, when doing manual testing, I follow what he describes as a "light" scripting, where I will just map out the areas I wish to explore so that I don't miss anything, but don't go to a detail level so that a "monkey off the street" could do the testing. BUT, when I am doing automation, that detail level is required.

▶ Reply to This

Permalink Reply by Thomas Ponnet on August 30, 2007 at 4:33pm

What becomes clear to me is a good balance between scripted and unscripted work works for 90% of most projects - don't nail me to that number though.
I'm used to writing test scripts and as mentioned before just writing test scripts can be an excuse NOT to do "proper" testing - what I mean with proper is in-depth testing for that particular project in question. So you can wave some test scripts in the air and say "We've done testing - here's the proof". I don't have experience with SOX but if it's anything similar to GLP there is a risk that you satisfy the documentation requirements to show that you did proper testing without actually doing it to the level the application calls for.

In general I find that scripted tests are much better when you have complicated tests with a lot of different variables or multiple data inputs. In writing down the possible (or desirable to test) combinations there is a lower risk for leaving out a combination - something where I think explorative testing falls short if the test doesn't use very, very strict discipline. One project I worked on had so complicated business rules and interactions with so many variables and different datasets that we automated the thing - this would have been impossible to test using explorative testing unless you worked in the business for years and could quote the requirements when woken up at 4 in the morning. So even "light" scripting wouldn't have helped in this case.

When we do scenario testing we note any deviations from the scenario we followed and make notes what was tested on the way. That means we also have to show something when asked "Can you show me proof that you did test the app?"

▶ Reply to This

Permalink Reply by james bach on August 31, 2007 at 2:48am

Hi Dsqared,

Let's get out of either or, and break the issues down. I'm telling you that testing is a sapient process, just as, say, driving a car is sapient. You didn't document "driving cases" the last time you drove your car, did you? But you are a skilled driver, I bet.

As a programmer, I have done a lot of automation, but why are you thinking of automation as merely the running of scripts? There's so much more to it than that, as I write in my whitepaper on agile test automation (http://www.satisfice.com/articles/agileauto-paper.pdf)

I use tools all the time to aid in my explorations. I apply automation in an exploratory way.

You can do performance testing in an exploratory and unscripted way, sure, but I would also encourage a performance tester to establish useful benchmarks, and that may call for a extreme scripting. Why would scripting be good for a benchmark, but not for other kinds of testing? Because the nature of the benchmark is such that you want to control certain variables so as to highlight changes in other variables. This is not the case with much of the functional testing challenge. In functional testing, I find more bugs by using a de-focusing approach, not one that keeps me focused and in a rut.

When Jake suggests that scripting is important in aerospace, it's because he believes that certain tests MUST be done. What he seems less concerned about is the HUGE number of tests that NEVER will be run, because the testers fritter away all their time on small coverage from carefully documented and repeated tests.

Complete testing is impossible. Therefore your tests comprise a sample. It stands to reason that paying a lot of time and effort for a tiny sample is not a way to find the most bugs. I would rather pay less, run sloppier tests, and get a bigger sample of behavior. To some degree, sloppiness amounts to variation, and variation is how I find more bugs than do you script-heads.

When it comes to SOX, first of all, the only testing that is covered by SOX relates to financial systems, if I read the law correctly. Secondly, the law does not dictate how the tests should be documented. You could, for instance, videotape all your testing. That's a document, is it not? I don't need a script in order to get documentation. Come on, guys. Get creative. Use your brains. Why are you willing to settle for bad testing that hardly finds any problems, instead of FIGHTING for good testing which DOES find lots of problems in the product? Help me with the fight, don't cave in to the bean counters.

As for litigiousness, I've only been an expert witness on seven court cases, but here's one thing I've learned from that: lawyers will advise you to document LESS, not more. Documents get you into trouble. They rarely get you out of it. And if you have documented your testing, and if, as is usually the case, you can't keep it up to date, because you have no time, then you will look pretty sorry in court, when lawyers on my side of the case start asking you why you use documentation to lie about the state of your process.

▶ Reply to This

Permalink Reply by ShaneMacLaughlin on August 31, 2007 at 9:23am

Hi James,

Interesting stuff, and the linked paper was a great read too. As someone running a small software house, I'm all for maximising the efficient use of resource through eliminating redundant processes. If it makes the staff happier as well, I guess I can live with that ;)

One of the questions DSquared raised which does not seem to be addressed in the discussion so far is the effective handling of automated regression testing. In a world where the SDKs used in my apps are changing, the OS on which my app has been hosted is changing, and the code base on which my app is based is regularly being refactored, new bugs in pre-existing well tested functionality are inevitable. Given that that app that I'm working on has been in continuous development for 16 years now, I have a substantial library of automated tests, all based on written documentation. This documentation includes examples from the user manuals, tutorials, test cases based on specific user data, and test cases covering bug replication. Much of this documentation may never be re-read, most of the automated test cases may never uncover a new bug. Does this make them redundant? Not in my book, as the testing is more about giving me the confidence that the software still works rather than finding bugs.

To a certain extent, this is based on the prioritised goals of my QA effort. The primary goal is that the software does what it is explicitly stated to do in all of the accumulated supporting documentation, stretching back over the last 16 years. I achieve this through automated regression. The secondary goal is that the software generally does what is implied either through user intuition or inference from the documentation. This is where I use exploratory techniques, and where appropriate, build more scripted tests.

The key to success in all of this, in my particular context, is to ensure that the ratio of resources required for development and testing, in order to achieve the stated goals, does not fluctuate too much. Documenting test cases, where they are not based on pre-existing documentation, does not affect this ratio anymore than developing the automation script. Where automated cases are built on a reasonably sophisticated framework that is also prone to change, I have found this documentation to be important. Good test documentation also helps prevent test code maintenance from spiralling out of control. I am open to any suggestions for a more efficient methodology than producing well documented, hand coded automated scripts for efficient long term regression testing.

I, and I guess many others, find access to good technical documentation to be a great comfort, even though I only ever read a tiny part of it. I note from your linked PDF that you consider MSDN as a valuable resource, as I do myself. I find step-by-step scripts on how to repeat a given process to be a very important part of this resource. By times they are the map that helps me explore and learn, and the recipe book that allows me cook up new solutions. Being able to discern what is useful relevant documentation from what is better employed as toilet paper possibly requires a degree of foresight and objectivity that does not IMHO exist. Does the fluff obscure the good stuff? Historically on paper, sure, but with modern document management and searching not so much. Does the cost of creating good test scripts render them redundant? I for one don’t believe so. I find well crafted scripts to be as an important, re-usable resource as well crafted code.

▶ Reply to This

• ‹ Previous
• 1
• 2
• Next ›
• Page