Replies to This Discussion

Permalink Reply by Anand Ramdeo on May 25, 2010 at 13:52

IMO, automated exploratory testing is possible. There could be many different ways of doing it, but one of the key concept would be - outcome of previous step should determine what script/step should do executed next. Now there are various ways to look at it - one overly simplistic way of doing it would be

Do this
if X happens - Do A
else if Y happens - Do B

But problem with this approach is - you have to have knowledge of X happens, Y happens etc and at this point you defy the purpose of exploration, because you know what is expected and script according.

The way I would approach this is

Do this
Inspect application state
Are there any errors
If not, from a list of actions applicable for this application state - do something randomly.
Inspect application state
Are there any errors

May be follow this in a loop (recursion - if possible). You still need to know about the application quite a bit - for example what do you inspect in application state, what actions are possible in a given application state and how do you know if there is any error. But still, the way tests are executed - they are fairly random and can expose interesting defects.

As far as using this approach is concern, I have never used it completely - but used it it pieces. May be one day :-)

If we have something like this (Or in any other form), I dont see any reason for why they should not be running at least once a day. Integrating them with build might be an issue - because they could be fairly detailed and have (imo) potential of giving false negative and positive both. Also, I wouldn't try this from ground up - Ideally it should come out from the tests we are writing as we go along.

Will they compliment - well can certainly. If you can prepare log of what was covered as part of this on the lines of -

Did this
System is in - this state
Did that..

As far as expensive / effective is concern - I need to do it first before I can answer this, but whatever I did in pieces (It wasn't anywhere near what I described) was certainly effective. Also, it depends on how it is done and also expensive / effective in comparison to what?

Thanks,
Anand - TestingGeek.com

• ▶ Reply

Permalink Reply by Evgeniy Boreyko on May 25, 2010 at 14:16

Well, that's not Automated Exploratory Testing. What you've described is just a good automated script that covers multiple valid and invalid states and transitions. There is nothing exploratory about it

• ▶ Reply

Permalink Reply by Anand Ramdeo on May 25, 2010 at 14:41

It depends on how we look at it. Inspecting application state is probably equivalent to 'learning' about application, deciding what should be done based on the application state could be equivalent to 'design' and choosing random data / steps could be execution.

But yes, I agree that it is probably not even close to what we do in manual exploratory testing. What we usually do in our exploratory testing can not be automated most of the time.

Anand - TestingGeek.com

• ▶ Reply

Permalink Reply by Evgeniy Boreyko on May 25, 2010 at 14:49

Well, yes. If we put it that way (I mean not comparing to manual exploratory testing), then yes - creating a "smart" automation script can be considered Automated Exploratory Testing.
But in this case we'll probably face problems like:
- creating such scripts will require a long time
- maintaining them will be much more difficult than maintaining small "stupid" scripts that cover separate paths in AUT

• ▶ Reply

Permalink Reply by Torsten J. Zelger on May 26, 2010 at 20:18

Hi
What you are describing here (using the state tables) has more to do with so called test monkeys than exploratory testing.

"Monkeys" were described in STQE Magazine about 10 years ago already by back by Noel Nyman (software test engineer at Microsoft at those days). STQE does not exist anymore with this name (I guess it changed to StickyMinds and then Better Software). Anyhow, Noel described the difference of a dumb monkey compared to a smart monkey where it uses some sort of a state table, similar to what you stated in your post.

Noel Nyman also refers to a tool called "Freddie" (a dumb monkey) which is part of the CD in Tom Arnold's Visual Test 6 Bible.

The article is found in the Jan/Feb 2000 issue. You can probably no longer download it from there, but if anyone needs it, give me a note and I can post it somewhere as PDF.

Cheers
T. Zelger

• ▶ Reply

Permalink Reply by Evgeniy Boreyko on May 25, 2010 at 14:10

To my mind, there is no such thing as Automated Exploratory Testing.
Wikipedia says: "Exploratory testing is an approach to software testing that is concisely described as simultaneous learning, test design and test execution." . Let's decompose the term and see what's inside.
The three key words are "learning", "design", "execution". This means that your script will be considered exploratory only when it'll be able to learn on its own, design (upgrade) itself, and self-execute. Well, that's AI :)

• ▶ Reply

Permalink Reply by Peter Nairn on May 25, 2010 at 14:21

I remembered a set of slides that Cem Kaner put on his web site about this - see http://www.kaner.com/pdfs/ImmuneITtestTalk.pdf . Worth a look

• ▶ Reply

Permalink Reply by Rob Lambert on May 25, 2010 at 14:33

Those slides are what instigated this post :)

• ▶ Reply

Permalink Reply by Nitin Purswani on May 25, 2010 at 14:29

I am also hearing this term first time .Can you brief more about this ?

I am thinking of tuning my automation a bit so it can run on its own but i am not sure if that will be called automated exploratory testing .The way would be

-I will divide first have functions of basic usecase i test or usecases performed by user on my application. like say usecase1(),usecase2(),usecase3() and the i will have function verifyUsecase1(),verifyUsecase2() and verifyUsecase3() which will verify each usecase respectively.

-Then i will put this in a file and use random generator to run these function in random order and i will also generate data in random order too .more function i develop more function i will put in this file .

-I am also thinking of generating data which could simulate real world data .

• ▶ Reply

Permalink Reply by John McNamara on May 25, 2010 at 15:03

Automated Exploratory Testing...I like the sound of that. That's what I will now call my monkey testing.

• ▶ Reply

Permalink Reply by John Stevenson on May 25, 2010 at 15:38

Hi Rob

I am not sure what you have been drinking.........

I am struggling to understand the term automated and exploratory in the same context.

Exploratory requires sapience with some level of thinking and making a judgment.

Automated IMO requires a true/false check that requires no human interaction or thought.

I am sure there could be ways to create a very good AI that 'may' appear to do some exploratory testing but until we get to the point where an AI can pass the Turing test which we appear to still be a long way from achieving a reasonable quality AI.

The problems from trying to create an automated exploratory testing application would be:

The effort to get the AI to be part intelligent would be beyond the cost of most projects - let us assume at least £5 million to create.

You would then need another AI program (another £5 million) to test the first one - or use a human being, which then defeats the object of automating :o)

To program the automation with some form of domain knowledge (unless you intend to use just for one project) would again cost silly amounts of money.

The use of automation to aid exploratory testing is a good thing - anything that can generate lots of data random or otherwise for you is a good automation tool for exploratory testing.

regards

John

• ▶ Reply

Permalink Reply by Rob Lambert on May 25, 2010 at 20:07

Hi John,

I raised the question because I knew it was a contentious topic. I'm trying to get my head around it myself. There are elements of the Exploratory Testing that certainly could be automated, like loading data, getting to a known state, checking log files, scanning directories etc, but I'm kind of with you that the actual exploring is something only a human can do. But I'm waiting to be proven otherwise.

Some people have argued that a Domain Specific Language automation could work, where you would have all of your user actions automated. You could then piece them together, run them, then run some more, then go off at a tangent and run some others. I like the sound of this, but I'm not sure about the reality. Some have suggested sticking them together after deciding the charter and then running the automated test, but this is just a quick scripted test....right?

I'm not feeling the love towards Automated Exploratory Testing..but I'm not writing it off just yet.

Rob..

• ▶ Reply

• ‹ Previous
• 1
• 2
• 3
• Next ›
• Page