Add a Comment

Comment by Anna Baik on November 25, 2008 at 23:18

Wow, thanks James!

I mentioned Derk-Jan's talk to a couple of our test managers the other day, and they seemed pretty keen on the idea of using exploratory testing. So I'll take back your suggestions, and will see what comes of it.

Comment by James Lyndsay on November 22, 2008 at 12:39

You can do exploratory testing without an experienced exploratory tester. Indeed, that's what all exploratory testers do with a new system / technology / customer etc.; start from a position of ignorance and get exploring.

There are a couple of ideas that I'd recommend keeping in mind:
- one of the things you'll be doing is gaining experience, and (through reflection) expertise. This is part of all exploration, but it's particularly true when getting going on something new. Expertise will help you find more/better information – and you'll find fewer/poorer without expertise – but it takes time to build. However useful it is, it is neither necessary, nor sufficient, and even expert teams explore better with a newbie in the numbers. This is because...
- if a few of you are exploring, there will be great diversity in your approaches. Learn from each other - and try to take those lessons in a way that doesn't flatten that diversity.

With those thoughts, here are a couple of recipes. Adapt as you see fit. For both, you'll need to set aside some time. Regard this time as a gamble, and assess its value when you're done. If I was doing this, I'd prefer to work in a group (so for a 3 person-hour recipe, schedule 2 people for 90 minutes). I might assess value by looking at whether I'd changed my bug rate, or found any particularly useful information / nasty bugs.

1 Exploring via Diagnosis: 3 person-hours
- Pick out some current bugs that aren't clear, or that aren't reproducible. Doesn't matter if they're in your bug tracking system or not, but they should be already known, and not yet fixed.
- Explore those bugs, seeking information that will clarify them or make them more reliably reproducible. Keep track of your activity.
- Review what you've done, collate the information gained. Log new bugs, update clarified bugs.
- If you can generalise from the investigative approaches you're used, then do so.
- Tell the whole team your results.
- Schedule another 3 hours on a different day and repeat!

2 The Usual Suspects: 2 person-hours + 10 minutes preparation
- Spend 10 minutes writing down lots of different ways that bugs manifest / get into your software (Use any or all of cause, effect, location, conditions, etc.). Aim for diversity and volume, not completeness or clarity. This might be fun as a whole-team exercise.
- Leaving enough time for the review+generalise+share steps at the end, split the remaining time in two.
- In one half of the time, pick out problems that you've not yet seen in this release, and look for them. Keep track of your activity.
- In the other half, pick out places that haven't yet seen many problems, and look in those for problems. Keep track of your activity.
- Review, collate, log, update.
- Generalise and tell the whole team your results.
- Schedule another chunk of time on a different day and repeat!
- Make your trigger list publicly accessible. Invite people to contribute, share, refine etc.

That said, ET is a skilled approach, and it's easier to get those skills into a team with a bit of reading / taking a class / involving a coaching consultant. There are plenty of sources around about getting started with exploration. Niel vanEeden and I wrote a paper called Adventures in Session-Based Testing which may help. It's here: http://www.workroom-productions.com/papers.html . For pithy heuristics, Michael Bolton has recently blogged the results of a fine EuroSTAR workshop (perhaps you were in it) here, and you'll also want to check out Elisabeth Hendrickson's cheat sheet, James Bach's mnemonics, Jon Bach's questioning approach, James Whittaker's tours (hopefully turning up on his blog soon). Perhaps people could post their favourite bootstrapping-into-exploration source here.

Buzz me on skype if you want to chat about this.

Cheers - James

Comment by Mark Crowther on November 20, 2008 at 9:22

As most of the clients I work with are digital media agencies my team and I use Exploratory almost all the time. I have a number of clients that I know we're never written a 'formal' Test Case for.

We still write a Test Plan but it's never of 3 pages and to form a Test Charter we use Mind-maps and have them reviewed by the customer to ensure they provide coverage as expected. If when testing we learn there are other aspects we need to test we add them to the map as a record.

That's the basics of our approach, the customers love it and while it's got some downsides overall it works really well for rapid testing in fast turnaround environments.

Comment by Anna Baik on November 12, 2008 at 16:29

Thanks Innes - apologies for muddying the waters by bringing Agile into it, I mean it just as an example of a process (could have been anything) where it really does seem to help if you have someone (whether external or internal) who has experienced it being done "right", and can tell you if what you're doing has the right "feel" to be successful. (Yes, all very fuzzy words, I know).

Comment by Innes Gray on November 12, 2008 at 10:14

I don't think has anything to do with 'agile' as such, except in so far as it could be called an 'informal' approach. I was using exploratory testing before the term 'agile' was coined' we were doing 'rapid' or 'flexible' testing then.

I think it goes something like this :-

1. Find a subject matter expert who knows the subject domain back-to-front
2. Brief the tester on the area(s) that are to be covered during the test (they can follow other hunches of their own, that's why they are there)
3. Allow them to test for the allotted period of time
4 The tester MUST record the scenarios that they have executed and any questions/defects that they have
5. Review the tester scenarios and questions/defects and agree what is/isn't a defect and which new scenarios could be included in the existing test suite.

I don't think this is the preserve of test 'experts', the emphasis should be on the use of subject/domain knowledge to uncover defects hiding in the deep, dark recesses of the application.

Comment by phil kirkham on November 12, 2008 at 8:30

Ditto - time for a post from Simon !!

Comment by Anna Baik on November 11, 2008 at 17:58

p.s. I meant to add, I'd really like to hear what else you have to say on this, Simon.

Comment by Anna Baik on November 11, 2008 at 17:54

Part of the reason I'm a bit dubious of the idea is that I've seen the horrible carnage that results from someone saying "Hey, this Agile thing sounds KEWL! No more documentation YAY! Let's all do AGILE now!"

I.e. implementing the bits they fancy from Agile but not the corresponding bits that actually make it work.

So how do you avoid this pitfall with exploratory?

(This is all pretty theoretical for me, as I am not in a position to decide what my team does - I'm a bog standard ordinary working tester, not a test manager or test consultant. None of which stops me lobbying for stuff I think it worthwhile, mind, but I have to understand it pretty well before I put that much energy into it).

Comment by Simon Godfrey on November 11, 2008 at 17:01

I think you can do this, after all, everyone starts somewhere. Ultimately, we all do exploratory testing in some shape or form anyway so I don't think you'd have a problem bootstrapping it into your team. I've more to say on this but alas, it's time to head home.