How Do You Keep Fresh Ideas Flowing?

Replies to This Discussion

Permalink Reply by phil kirkham on January 7, 2009 at 10:18am

If you really are just retesting the same feature repeatedly then it's not really Exploratory testing is it ? The area has been explored and you have a good map of the area so isn't your co-worker really just doing regression testing ? Is any automated testing happening ?

It is easy to get stuck into a rut in such situations - maybe your co-worker should try eating some chicken fingers as I suggest in my old blog

Or try some of the games suggested here

▶ Reply to This

Permalink Reply by Michele Smith on January 7, 2009 at 11:30pm

Hi Phil,
Your comment “If you really are just retesting the same feature repeatedly then it's not really Exploratory testing is it?” has limited the feature in question to a set of limited/repeatable steps. Would your first comment be the same if the feature were the Font Feature in Microsoft Word?

“The area has been explored and you have a good map of the area so isn't your co-worker really just doing regression testing?” I think that was how he was feeling and that is why he came to talk to me. Maybe that is what he is doing, but a good map of the area may not be suitable for the different customers that are going to use the feature.

“Is any automated testing happening?” Yes, but my aim at posing the question was not to get ideas of how to alleviate his thinking. I am not saying that automation requires no thinking, what I am saying is that I think when we get in a rut we need to think about why we are there and learn how to think past it.

I really am curious how other testers think past the point where they wonder what my co-worker is wondering. I do not usually end up in that position because I think testing has many layers. And just when you think you have gotten near the bottom layers, Microsoft issues another OS/SP/.NET package to keep you busy :)

▶ Reply to This

Permalink Reply by phil kirkham on January 8, 2009 at 9:18am

Would your first comment be the same if the feature were the Font Feature in Microsoft Word?

well having just read about Font tests in the How We Test Software at Microsoft book I think it would be :)

As always in testing though, the answer is 'it depends'
What changes are being made to the feature in all these multiple builds - minor ones, major changes to functionality ? Is the feature currently being used by customers and are they finding lots of defects ?

and to turn your own post back onto you - when would you decide that enough exploratory testing has been done ?

▶ Reply to This

Permalink Reply by Alan Page on January 9, 2009 at 3:48am

Thanks for the plug!

This is a great question, and I think Joel hit on some great answers. When I'm feature testing, I try to test from as many different angles as I can (I call this 360 degree testing, but that's just me). If I'm stuck, I'll ask myself two questions: What haven't I tested for yet?, and What testing approaches haven't I tried yet? (sometimes these overlap). My initial exploratory tests may have been primarily looking for issues in functionality or negative input, so next, I may brainstorm on how the font feature may fail (fonts could overlap, fonts could be illegible, wrong font could be displayed, etc.). Then, I'll try to make those things happen in as many ways as I can think of.

When I've exhausted that avenue, I usually change directions entirely and look for memory leaks (I wonder if large fonts make word use more memory? If they do, is the memory released when I change back to a smaller font? What other ways could I cause a memory leak with fonts.

Eventually, I may examine the code or code coverage data (since I work at MS, those are available to me). Gaps in code coverage, or code review may give me more ideas on what to test. Probably much earlier than this point, I'll talk to the developers on the team who deal with fonts, as well as any other font experts I can find to see what kinds of tests they run. This ends up turning into a perpetual circle - by the time I've tried everything I can think of, I've learned enough that I can try the things I tried in the first place, because now I have a new perspective and have slightly different approaches to take.

It seems that I've flowed away from the original question, but I suppose I consider all 360 degrees exploratory (others may argue). I'm learning as I execute tests and changing my approach whenever I get stuck and want to get some "fresh ideas flowing".

▶ Reply to This

Permalink Reply by Michele Smith on January 9, 2009 at 9:51am

The 360 degree testing approach is similar, though much more in depth than I am capable of (code coverage/analysis is unavailable to us and I have no experience with it), to the "layers" I referred to.

The statement that you made "by the time I've tried everything I can think of, I've learned enough that I can try the things I tried in the first place, because now I have a new perspective and have slightly different approaches to take" is how re-testing can be done if the tester has taken the time to get to know the feature.

Your approach is smart and thorough. After reading what you wrote a lightbulb went off in my head. Maybe I misunderstood what my co-worker really needed at the time. Perhaps he just needed to remember his passion for testing. Your comment has this as an underlying theme. Thanks for posting!

▶ Reply to This

Permalink Reply by Michele Smith on January 9, 2009 at 10:00am

I have several criteria that I use to decide that enough exploratory testing has been done. Here is my short list:

1. There is a critical blocker that prohibits me from continuing.
2. A business decision is made to ship the software and stop testing.
3. My manager shifts me to a different project.
4. The lead moves me to a different feature.
5. I have covered the feature to the point where I have run out of ideas, even after de-focusing. If this is where I decide to stop I do one of two things:
A. If I am confident, I report what I have tested and provide the appropriate documentation to the test lead.
B. If I am not confident (for instance there was some behavior that suggests something is not quite right, but my testing did not reveal any proof) I will inform the test lead and the other testers so the feature and the area surrounding it can be watched.

▶ Reply to This

Permalink Reply by Joel Montvelisky on January 8, 2009 at 1:18pm

I will leave aside the valid discussion between you (Michele) and Phil, and go to ways I use to refresh my perspective, which I think is important regardless if you are working under exploratory or other methodological approaches.

I've used the following things to help me get new ideas into my old tests:

1. Pair testing, taking a peer or a developer or PM and asking him to sit with me while I plan or run my tests, getting their inputs as we run over the application always brings more ideas to the table.

2. Look at the bugs coming from other sources, specially handy if you have customers who are reporting issues via your Support Org. That way you can get visibility not only on the bugs you didn't find, but also on the areas you missed completely.

3. Read your customer-oriented documentation or see a demo of the product. Understand the examples and demos been done to your customers and understand how else can they use your product. This is specially good for platforms that allow you to customize your work broadly.

4. Talk to your customers. I wrote this article about the QA being the Customer Advocates, where I explained how we managed to get REAL LIFE SCENARIOS we were missing in our testing labs by working directly with users.


And obviously the trivial way is by getting some perspective, by working on other projects (related or not) and then coming back and trying to look at your app and work with a new perspective.

Hope this helps,

-joel

▶ Reply to This

Permalink Reply by Sharath Byregowda on January 9, 2009 at 9:12am

Michele, I feel this is a very good question.

I have had team members complaining that it's boring to test the same program with no new features. But, I have also observed a pattern, testers who complain this way, usually are full of energy when they begin testing but loose their interest may be after 2 weeks or so irrespective of the module they test. One of the reason I feel they get bored is because they run out of ideas to test. As a test lead one of my challenges have always been to keep myself and my team on the look out for information always.

Like in a recent assignment I found the number of bugs reported by my team dropping, when we sat together, I felt the ideas running out in the team – we initially focused on exploring the system with functional and domain related missions. This made me focus on Flow tests - explained the team what could be the observations to note running flow tests – and what a change I observed I felt the energy was back, they liked testing the same module again without any new changes. If they again run out of energy I still have user testing, risk, stress, variability, etc lined up.

When we feel like we are running out of ideas I feel we should change our approach/thinking a bit and the ideas start rolling.

-Sharath.B
http://testtotester.blogspot.com/

▶ Reply to This

Permalink Reply by Michele Smith on January 10, 2009 at 11:29am

Hi Sharath,
I see value in your team approach to keep fresh ideas rolling for several reasons.

1. When people talk about what they are doing and how they are testing it forces them to think about what they are doing.
2. It helps those with less experience learn new things/ideas/approaches.
3. People ask questions, which further the learning.
4. It builds confidence in the testers as individuals and as a team.
5. It keeps the team focused on the mission.
6. The passion and zeal of one tester could infect others.

Some of us, sometimes, gather and discuss these things, but we do not do it as a team.
Your team approach is great. Thank you for sharing it!

▶ Reply to This