Test environments - Candidate for exploratory testing?

Replies to This Discussion

Permalink Reply by phil kirkham on November 12, 2008 at 11:42am

I dont quite understand the question - what is a 'pristine test environment' ? If your customers are going to be tweaking configurations, changing browsers settings and installing/reinstalling/uninstalling then surely your testing should also reflect that ?

why do you think exploratory testing is suitable for this - and what do YOU mean by exploratory testing ?

▶ Reply to This

Permalink Reply by Srihari Palangala on November 12, 2008 at 11:53am

Agreed Phil, I think some clarification is required.

Pristine test environment = What testers consider as the environment in which software will be deployed by end customers. Usually I believe that testers will assume a "clean stack" under which software will be deployed.

Is this assumption of mine true? Is that what testers normally assume?

Exploratory in this context = Obviously it will be hard for testers to dream up all possible end customer environment configurations. Hence, exploratory testing (i.e., in my mind testers using discretion and experience to test various 'unchartered' environment configs) might make sense in this scenario. I.e., testers would use their experience to predict likely (non-pristine) customer deployment scenarios.

One of the first things we do when a customer reports a bug ... is to try and reproduce it. Doesn't that entail both environment + the customer use case? Could we have prevented those bugs that arise due to the (non-pristine) customer environment? Do you see a lot of those?

Hope this clarifies (and makes sense). I'm now wondering::

- Are variations in the test environment something that testers spend time on?
- If yes, would exploratory testing be the best way to do it?

Thanks a lot for your time on this.

▶ Reply to This

Permalink Reply by phil kirkham on November 12, 2008 at 12:02pm

OK, your question makes a lot more sense now, thanks for the clarification

Still a bit confused though...

You say it will be hard for a tester to dream up all the configurations used by a customer - but somehow an exploratory tester is able to predict them ! If your experienced tester can come up with likely scenarios then they could be tested in either a scripted or exploratory style

It will be interesting to hear other responses to see how people set up their test environments

( and speaking from painful past experience of only testing on one machine and one configuration then differing customer installations and configurations was a big source of bug reports )

▶ Reply to This

Permalink Reply by Simon Godfrey on November 12, 2008 at 1:29pm

I see what you mean. Exploratory testing could work, but I guess it depends on what you want to achieve. Why are you thinking exploratory and not scripted? What type of exploratory are you thinking?

Given our customer base, it's impossible for us to replicate their networks. We have used images of customer networks which helps find network specific defects but these images are just a drop in the ocean. We usually test on two types of networks, one which we consider the latest and greatest and one which is our oldest supported. In MS terms, this would mean we test on Vista with Office 2007 and Windows 2000 running Office 2000 (random example - assuming what we're testing is affected by either O/S or Office version!).

▶ Reply to This

Permalink Reply by Srihari Palangala on November 12, 2008 at 5:30pm

Thank you for your message Simon. I'm trying to achieve the situation where testers don't end up saying "I did not test the software in {insert customer deployment scenario}". Based on my limited understanding, I figured 'exploratory' was the way to go to minimize the above risk.

Even in your case - how often have you got hit because it was a customer environment scenario that was not replicated (for executing the test cases in)?

Other readers, Phil, Simon ... if you get a moment, could you consider casting your vote on my blog?

http://blog.vmlogix.com/2008/11/12/testing-environments-testers-ing...

▶ Reply to This

Permalink Reply by phil kirkham on November 12, 2008 at 7:24pm

I would vote except that neither option is what I would agree with

▶ Reply to This

Permalink Reply by Srihari Palangala on November 13, 2008 at 3:58am

Thanks Phil. I've added a "Other" option into the poll.

▶ Reply to This

Permalink Reply by Michele Smith on November 12, 2008 at 11:31pm

My thoughts on your revised questions:

- Are variations in the test environment something that testers spend time on?

Yes, I would hope that most testers understand the value of variations in the test environment, but in some cases the variations are limited to what the product is used for. There are some products that are designed for specific purposes, specific customers, and used specifically in a specific environment. While it is good to vary the test environment, the stakeholders may not appreciate you spending time on such activities, (you need to know your audience). Pristine environments do have their place as well. These should be factored in because they are a variation of the environment. What if a user decides to install your product first on their new operating system? If they need to use the product and do not have any other applications installed, is the product just as usable? This may sound odd to you, but I have witnessed defects in regards to first installs on clean systems.

As with all rules, depending upon the situation, you may be able to change things by using a rather devious approach to the task. For instance, one group of stakeholders, that I work with routinely, has customer base that uses a specific operating system – no ands, ifs or buts. I decided to install in another, more modern operating system, on my own time and take a look at what happens. The results were terrific so I informed them. They were quite happy with the results and as time has passed, they have decided to take a look at the full system installation on this particular operating system in order to attempt to widen their customer base. My attitude is generally “it is better to beg forgiveness than to ask permission”.

- If yes, would exploratory testing be the best way to do it?

It is no secret that I am an ardent believer in exploratory testing. I consider myself a self-learner of the “school” that James Bach “teaches” in (I am an elementary student, but I have intents of reaching the stars). With all things testing, this question raises questions of its own. What is the purpose of the software? Is it going to be life support for my grandmother? If it is, I want you to have scripted tests that you run for every release of your product. I would like you to have some exploratory testing in place as well, there will always be a “what if” scenario that you may miss with your test cases. Is your product going to take care of all my banking needs? Then I want you to have scripted testing figured into all your test cycles as well, but I also want you to use exploratory testing. Hackers are really good at it, so I want you to hire the best.

Do you see my point? There are no specific answers for any of your questions. Each product has its own set of questions that must be asked in order to evaluate what is best for the stakeholders and customers. Asking questions is the key to exploratory testing.

▶ Reply to This

Permalink Reply by Srihari Palangala on November 13, 2008 at 4:07am

Thank you Michele. Your response is comprehensive.

Looks like you (and others I'm hearing from) agree that it is important to make variations in the test environment (and with each variation you run a few basic smoke tests to make sure that stuff works).

What I'm also hearing is that testers consider this as a "nice to do" but not a "must do" (which means no one really does it in practice).

▶ Reply to This