Many detractors criticize Exploratory Testing because of lack of rigor and "seriousness".

In this paper, I'll intend to offer a real vision that help to understand that Exploratory Testing is the "NATURAL" way to test.

It is a way that has been developed and improved over the history and although it is applied in other sciences, it still seems strange in the world of IT "sciences"



Overview of the Scientific Method

The scientific method is a process for experimentation that is used to explore observations and answer questions. Scientists use the scientific method to search for cause and effect relationships in nature. In other words, they design an experiment so that changes to one item cause something else to vary in an unpredictable way.

Just as it does for a professional scientist, the scientific method will help you to focus your science fair project question, construct a hypothesis, design, execute, and evaluate your experiment. 


Steps of the Scientific Method 

  • Ask a Question: The scientific method starts when you ask a question about something that you observe: How, What, When, Who, Which, Why, or Where? And, in order for the scientific method to answer the question it must be about something that you can measure, preferably with a number. 

  • Do Background Research: Rather than starting from scratch in putting together a plan for answering your question, you want to be a savvy scientist using library and Internet research to help you find the best way to do things and insure that you don't repeat mistakes from the past.
  • Construct a Hypothesis: A hypothesis is an educated guess about how things work:

"If _____[I do this] _____, then _____[this]_____ will happen."


You must state your hypothesis in a way that you can easily measure, and of course, your hypothesis should be constructed in a way to help you answer your original question.

  • Test Your Hypothesis by Doing an Experiment: Your experiment tests whether your hypothesis is true or false. It is important for your experiment to be a fair test. You conduct a fair test by making sure that you change only one factor at a time while keeping all other conditions the same.

You should also repeat your experiments several times to make sure that the first results weren't just an accident.


  • Analyze Your Data and Draw a Conclusion: Once your experiment is complete, you collect your measurements and analyze them to see if your hypothesis is true or false.

Scientists often find that their hypothesis was false, and in such cases they will construct a new hypothesis starting the entire process of the scientific method over again. Even if they find that their hypothesis was true, they may want to test it again in a new way.

  • Communicate Your Results: To complete your science fair project you will communicate your results to others in a final report and/or a display board. Professional scientists do almost exactly the same thing by publishing their final report in a scientific journal or by presenting their results on a poster at a scientific meeting.

Even though we show the scientific method as a series of steps, keep in mind that new information or thinking might cause a scientist to back up and repeat steps at any point during the process. A process like the scientific method that involves such backing up and repeating is called an iterative process.

Throughout the process of doing your science fair project, you should keep a journal containing all of your important ideas and information. This journal is called a laboratory notebook.


Exploratory Testing as Scientific Method

  • Ask a Question: "What would happen if I put a letter in this numeric field?"
  • Do Background Research: "My experience says me that when I did that and the app didn't fail, the system showed me an informative message"
  • Construct a Hypothesis: "If I put "A" in this field, then an informative message will appear."
  • Test Your Hypothesis by Doing an Experiment: "Let's do the test and put a letter in the numeric field"
  • Analyze Your Data and Draw a Conclusion: "oh oh I think this is not OK"
  • Communicate Your Results: "I should report this problem as a bug"

So... if that happened using "A", What is going to happen using a negative number? (now I am in the first step again).


Views: 4965

Tags: exploratory, method, scientific, testing

Add a Comment

You need to be a member of Software Testing Club - An Online Software Testing Community to add comments!

Join Software Testing Club - An Online Software Testing Community

Comment by Jim Hazen on June 21, 2012 at 8:08

Michael,

Blueprints come from both a creative and technical process.  But they use a 'known' method of construction that is 'defined' beforehand so that the correct structural and load balances are maintained so that the house does not fall down on itself.

 

Of course these 'rules' for construction came about from 'experimentation' with the structural & material designs.  So eventually a loose collection of design rules became a standard method with predefined steps associated to them.

 

I find it interesting now how JB calls Testing 'science' considering in the past he called it an 'art form'.  Amazing what you learn from reading about the origins of biological classification (Taxonomy) methods.  Maybe the Buccaneer Scholar is now catching up to the other 'college trained' people.

Comment by Michael Bolton on June 1, 2012 at 2:26

"I also like to think that I need to have a blueprint first before I start to build my house."

Where do blueprints come from?

---Michael B.

Comment by David García Romero on May 31, 2012 at 10:28
James,

Thanks for sharing this with us.
Comment by James Thomas on May 31, 2012 at 5:25

On the Rapid Software Testing course I attended, James Bach said that testing is science. I found it interesting that he deliberately didn't say testing is a science. He was (I think) making the point that is covered here - the scientific method is a natural testing method.

I'd previously tried to come to a similar conclusion: http://qahiccupps.blogspot.co.uk/2011/11/appliance-of-art.html

Comment by Jim Hazen on May 5, 2012 at 6:57

David,

As someone who has a Natural Sciences degree (B.S. in Zoology), and spent many an hour learning about and using both the Scientific and Empirical Methodologies (during college and after as part of my career in Testing)  I understand your points.  A good scientist will use all of these methods to try to prove or disprove the Hypothesis (<- correct spelling).  It depends on the objective/goal of the experimental process; either prove or disprove the hypothesis they have proposed or somone elses. 

 

What I was talking about is that as part of the Scientific Methodology you also have the ' Experimental Procedure' that is defined and outlined before the actual experiment is conducted.  As part of that you will define the 'Expected Outcomes'.  Then when the experiment is executed and completed you will compare your 'Actuals' vs. 'Expected'.  Based on this you will derive a conclusion to the planned decision to either prove or disprove the original hypothesis.  If results did not match then you have to go back and determine why the actual versus expected did not match up.  You then publish your findings and defend the work done.

 

That is a 'scripted' process and method.  ET doesn't fully adhere to the 'scripting' of the test.  It allows a more emperical approach (observation and what-if scenarios) to determine if the a'priori hypothesis stands up or not. 

 

I'm not against ET, I find it very useful as a part of the 'tools' at my command.  I believe in balance of techniques and tools for the job at hand.  So in this sense I do act "contextually", or as I like to put it "using common sense".  I also like to think that I need to have a blueprint first before I start to build my house.   I don't think you would want your house built 'free-style'.  Now you may want to do some random sampling type experiments during construction and afterwards, but just relying on them alone won't give you the whole picture. 

Comment by David García Romero on May 4, 2012 at 20:14

Hi Jim, thank you for your comment.

In the post, I was talking about the general idea of Scientific Method.

The deductive reasoning, inductive reasoning, and hypothetic-deductive are the three scientific methods, which are referred to by the generic name of the Scientific Method.

"Empirical Method" as you call it, is actually the approach of using a collection of data to base a theory or derive a conclusion in science.
There are other ways to reach an hipothesis as "try and error" (experimental method).

In ET (as much as in Scientific Method), we have some thing that is suppose to work in some way.
This "way" is an hipothesis.
We can create an hipothesis based in our experience ("Empirical Method"),or  based in other experiments ("Experimental Method"), ETC.

The aim of the experiments/tests is to prove that the hipothesis is wrong under an specific situation.

Obviously every serious test must be designed (set an objetive, be aware of how reach this objetive).

Between "Freestyle ET" and "pure scripted testing" exist a lot of levels depending on the tester' freedom.

That's why every way to do testing "sounds" more or less to "scripted testing".

Personally, I think that the starting point should be "Freestyle ET" and from there going generating the documents that are needed according to the context.

Comment by Jim Hazen on May 4, 2012 at 4:17

David,

As part of the Scientific Methodology you need to mention the Experimental Procedure.  This is a key piece to being able to have designed the experiment and then ensure it is repeatable.  This requires using a procedural method, or steps, to perform the experiment (or test).  Typically this is done first (before) the experiment is done.  The procedural method/steps are derived from your prior research to formulate the hypothesis (theory).

 

Dang... this is starting to sound more like a 'scripted' test.  Sure you aren't doing more of an Empirical Method instead?  Because empirical data is gathered as part of the 'experience' of performing a set of tests in a 'what-if' scenario.  Which is what Exploratory Testing seems to be.  Experience and a'priori knowledge helping to drive direction and decisions based upon the experience at that moment.  Using Cognitive Thought processes to make observations and then making decisions based upon them. 

Comment by David García Romero on May 3, 2012 at 20:52

Thanks for the comments.

Comment by Catherine Karena on May 2, 2012 at 12:02

I like this discussion, it may help me put a description together in 400 words or less relating the relevancy of exploratory testing to her testing requirement - (it's a digital production company that creates transmedia entertainment across web, mobile, social media and games platforms).

Comment by Michael Bolton on April 28, 2012 at 20:18

I often find an opportunity criticize the detractors because of a lack of rigor and seriousness.

For one thing, it's important for us to consider what rigor (or seriousness, or structure) means. Are the local detractors able to give a plausible set of ideas about what rigor (seriousness, structure) are?  What are they including?  What are they leaving out?

Do they recognize forms of structure other than a scripted test case?  (In exploratory testing, there are dozens:  http://www.developsense.com/resources.html#exploratory)

Have the people who dismiss exploratory testing observed skilled testing closely? 

Have they noticed the interaction between formal and informal processes?

Have they notice the dependencies of formal processes on informal processes?

Do they recognize the extent to which learning and investigation of any kind is exploratory?

Have they noticed what happens when a tester runs into a problem in a script?

Are they talking about skilled exploratory testing, or slapdash testing?

Et cetera.

---Michael B.

Adverts

Ministry of Testing

© 2014   Created by Rosie Sherry.

Badges  |  Report an Issue  |  Terms of Service