Attitudes to scripted tests

One of the things I hope to get out of this club is opportunity to have some in-depth discussions on interesting topics with fellow testers. So I'm hoping this will stimulate some interesting discussions...

When I'm working with smaller development outfits they seem to be quite comfortable with the idea that testing can be done without reams of scripted tests (i.e. the detailed test instructions that testers should follow). Instead they rely upon the fact that the testers know what they are doing and don't need scripted tests to find the bugs...or perhaps it is the cost of producing the scripted tests that makes them willing to accept this :-)

When I work with larger development outfits or end-user companies (e.g. banks, local government) I generally find they are more reluctant to give up on the idea scripted tests. It seems that there is a certain comfort in having a set of scripted tests to follow, particularly if they perceive that the tests will be of use in future cycles or releases.

So, in general terms...

Does anyone else find this?...or is it just me?

Do you think it is a case of the larger the project, the more there is a need for scripted testing? Can exploratory testing work on large projects?

Do you think that scripted tests are built automatically because companies think that is the right way to do it?

Do you feel that it is worth investing in scripted tests if you plan to "repeat" the tests on the next release?

Do you feel that managing the testing process is more difficult and risky if you don't have scripted tests?

Thoughts? Points of view?

Tags: scripts, test

▶ Reply to This

Replies to This Discussion

Permalink Reply by dsquared on August 31, 2007 at 12:53pm: James,

As always, your comments are thought provoking. As such, they are very valuable.

I do understand your comments and have to ask some questions.

Re documentation in court cases, do lawyers generally tend to say "less documentation is better than more", or do they say "no documentation is better than some documentation"? There is a fine distinction there, and not having been in court as you have, I have no clue where the real answer to that question is.

Re SOX. You are correct. SOX does apply to financial applications. However, Rosie made a generic statement. I was simply pointing out that the generic statement is not always true. Sometimes, documentation is some form IS required.

Re video tape of a test for SOX. Interesting concept. BUT.... Is video really the best media to document a SOX test? How do you capture things like what the various user access levels are, what test users have what levels, etc? It is an interesting concept, but I suspect that it would be a hard sell at a lot of companies, particularly because most people already have word on their machines, but video would require an investment in infrastructure.

As always, a pleasure reading your comments, even if I may or may not agree with all of them.

D^2; ▶ Reply to This

Permalink Reply by Bill Matthews on September 12, 2007 at 4:15pm: Well, I can see this turned out to be quite a hotly debated topic (both here and on QA Forums); apologies for not contributing much recently but a family emergency took me out of the loop for a while.

While I’ve come across ET on projects and I tend include it as part of my projects, I’ve not come across any large scale or complex/risky projects that have relied upon ET for significant portions of the testing.

However, recently while on a client site I had a discussion with one of their test analysts who had just returned from a conference and had become very enthusiastic about exploratory testing. At the conference he had spoken at length to a consultant who, on hearing about their scripting woes (patchy coverage and scripts going out of date quickly), suggested they abandon scripted tests in favour of exploratory testing. The consultant went as far as stating that he could sit in front of *any* application without test scripts and break it within an hour. Now I don’t know whether the consultant fully understood the context or the problems but to me it seemed to be too risky an approach and I was quite vocal about my concerns about relying upon such an approach.

It did however start me thinking about under what circumstances such an approach would be appropriate.

After all, testing can be an expensive activity and, in my experience, documenting and maintaining detailed scripts accounts for a reasonable proportion of those costs; I exclude from this the analysis and design of the tests. ET comes with the potential to reduce this aspect of the testing process without sacrificing the analysis and design of good tests.

Most of the literature I’ve read on Exploratory Testing and methods such as Rapid Testing appear to stress that free-style exploratory testing (i.e. testing without any form of guide) is very rare and it is more common to have checklists, tables, lightweight scripts and other aids to guide and measure testing. I can certainly see such approaches working for a range of domains including those where the process of testing is “regulated”. Whether it could work on some of the examples given by Jake I don’t know…safety critical projects are way out of my domain but sound like really interesting challenges!; ▶ Reply to This

Permalink Reply by Linda Yan on November 14, 2007 at 1:25am: I think scripted test is very useful in the project which dilivery time is very tight.
I always use it for smoke testing; ▶ Reply to This