Test case management tool vs Excel

Replies to This Discussion

Permalink Reply by phil kirkham on 19 June 2008 at 2:50pm

Can you combine the approach - carry on writing the tests in Excel and import them into the test management tool ?

▶ Reply to This

Permalink Reply by Thomas Ponnet on 19 June 2008 at 3:19pm

We could but I think it would defeat the purpose. I think that would be a waste of time but also the scripts needs to be written in a different fashion.
Importing would depend on the tool, I guess with some you could with others you can't. So figuring out how to do that takes time as well, could be worth it but again not sure.

▶ Reply to This

Permalink Reply by Dan Woodward on 25 June 2008 at 4:59pm

Do you have people on your team who are familiar with VBA, Excel and maybe some Ruby or Python experience? The place where I worked last had a very flexible way of using VBA or Ruby to scan excel files for manual test results. These results were tracked by putting the tests to be executed into a folder structure that mimicked the build releases. Once you have the files in the folders you can scan the files, look for your column names and then begin collecting results from the files. Then put that data into a web report, or a database, or another Excel file.

I can't give you any code examples, but I can assure you it is possible. To be honest, it might cost you upwards of 10-20K of resource time to develop it, but then it is your tool and your process rather than Test Director/Quality Center or other.

▶ Reply to This

Permalink Reply by Thomas Ponnet on 26 June 2008 at 8:34am

Thanks for the reply.

Unfortunately we don't have people with much programming experience in my team - one of the teams weaknesses I think so this approach is out of the question.
Getting an idea how much it would cost to do that internally is interesting though, I'll keep that in mind when we're recruiting again :-)

▶ Reply to This

Permalink Reply by Lloyd on 26 June 2008 at 11:25am

I can thoroughly recommend HP Test Director (and no I don't work for them - lol).

We went from using Excel to 'manage' our test effort in Jan 08 and have found the level of automation TD provides in the construction of Test Cases and Scripts based on the requirements entered to be excellent.

We are a small dev dept with 3 devs to 1 tester (with less than 6 testers in total) and have found that (providing I 'stuff' the requirements of each project into TD as soon as I get the signed off document) it does the job to an exceptional standard.

In summary, TD answers all Pros of using Excel (in its sleep!) and both cons listed against tools do not apply to TD, if it is setup and being used correctly.

Good luck with whatever you decide though. These kinds of tool are not for every business but my experience of using them in immediate and direct comparison to Excel has been a favouable one, to say the least.

▶ Reply to This

Permalink Reply by Charles West on 28 June 2008 at 5:24pm

Thomas,
I've been in your shoes before. Excel is certainly not an effective test management solution. Also, I'm quite disappointed with TFS's manual test management capabilities. Certainly M$ could do better than a repository of Word docs!

About 18 months ago I was managing a small QA team and looking (once again) for an affordable and effective test management tool. Having only a small budget to work with, the major tool vendors (Rational, Mercury, Segue) were all out of the question. I had been in this position before and was unable to find a tool that suited my needs. It was at that time that I finally lost my patience and started building my own.

I'm now the owner of SQA Design. We've developed a solid test management solution and are continuously expanding on it. Please check us out at http://www.sqadesign.com. From your post, it sounds like our tool may fit the bill.

▶ Reply to This

Permalink Reply by Thomas Ponnet on 30 June 2008 at 8:17am

Thanks for the link and mentioning that you're the owner. Some people 'forget' to mention their interests which then doesn't look too good. I'll download the trial version and see if that does what we want.

Cheers

▶ Reply to This

Permalink Reply by moonlight on 18 August 2008 at 9:02am

Well, no doubt nowadays several tools are available, however, I would suggest you to try Zephyr, from the last 6 months I have experience with using this. It is a comprehensive Test Management System with very effective test case repository which are neatly organized & accessible, importable & exportable and Drag-able and drop-able.

▶ Reply to This

Permalink Reply by Ido Schacham on 19 August 2008 at 9:07am

That's interesting. Our company, Testuff, was also founded by people who grew tired of managing tests in Excel, but couldn't find a good and simple test management tool. So, we built an on-demand test management exactly for that stage.

Testuff might suit your bill. You could import your test cases from Excel into Testuff via the import feature. We also offer to do it for you for free if you wish. Copying-pasting tests is easy in the test editor since tests are fully textual, each step simply separated with a blank line. Recently we added integration with TFS so that you could continue to report issues there from Testuff.

Currently we don't have requirements management and reporting, but they're both on the road map. For your offshores you could export tests back to Excel or to HTML and send it to them. We would like to add user privileges in the future so that there will be "test runners" that will only be able to run tests they are assigned and nothing else.

So please forgive me once again for the shameless self promotion, but I'd love it if you gave Testuff a try and let me know what you think:
http://www.testuff.com

▶ Reply to This

Permalink Reply by Stephen Kay on 19 August 2008 at 11:13am

First off, my company also make a test management tool, there seem to be quite a few around, my sales pitch is at the bottom of this post.

Now onto your actual post.

I use "Ken TestMan" (our tool obviously), my test cases are generally grouped by components in a tree structure. For example I may have a folder for the "login" component and in there I have all the cases for the login function. This is similar to folders in Windows Explorer and makes sense to most people.

My cases are quite general, I'm not a big fan of highly scripted tests (click X button, this happens; click Y button, this happens;) so my cases are more like:

Test condition: "login with invalid username"
expected outcode: "'invalid username' message appears".

This is partly because the maintenance of highly scripted test cases just kills me, partly because I work with people who have the skills and experience to know which buttons to click (and if 3 people do it 3 different ways then great), and partly because if I'm going to that level of detail I may as well automate the test.

I probably create 50% of cases beforehand based on requirements documents, but most of them are created while I'm actually testing. So it's important to me that a tool allows me to add cases very easily when I'm executing a test.

Even though our tool allows us to map test cases to requirements, I rarely do it, but I know lots of people do.

The features I use most are:
- tree structure to organise my test cases
- quick and easy way to enter test cases both before and during testing.
- version control and record locking to allow my team to work on the same project at the same time.
- web based interface (I can get to it from any PC)
- test executions allowing me to record the details of the environment I'm testing in and the status of the tests for that execution.
- reporting:
- test cases created by day (tells me if we're still discovering new features)
- test cases passed/failed/untested (tells me how much of the planned tests we've done)

Although our test tool (and most others) does more, I can operate quite happily and effectively with just this.

One of the things I've encountered is that the full glorious solution (from requirements management, through testing to defect management and resolution) requires buy in from the whole project team. Many of the people I've met can't get this, and so want a solution that works for the test department, but can then extend to the full solution once the project team see how effective it is.
(sales pitch:
to support this view we do 2 things:
- we have a free version that provides this basic functionality
- our commercial version is licensed per server, not per user as we want to encourage the project team to get involved, not deter them with higher license costs)

That answers the bit about what I do and how/why.

As for you, there doesn't seem too much in your cons that would make me rush away from Excel, although I appreciate that you may have simplified the problem in order to make your post readable.

You can get counts of test cases, passes, fails by using the Excel function "Countif", your example script seems to lend itself to this. A simple summay sheet with some counts would give you basic statistics. (One note, if you do go down the VBA route and come up with a heavily customised solution then remember to factor in ongoing maintenance.)

Version control you could manage through TFVC, but I don't know much about that.

Requirements linking, most of the tools I've seen that support this require you to store documents in a certain way or certain format, if your analysts don't buy into this process/tool then this could still be a very manual process.

As for failing steps. If you fail a step within a test case then generally the test case status is "failed" too, but I don't think that means that the test execution (plan, scenario, suite or whatever you call it) fails. I'm not sure I understand your concern here.


For me, the most important thing is that the test tool has to get out of my way. It has to support me, not constrain me. So I think finding one that makes sense to you and allows you to organise your work in the way you want is the most important thing.

sales pitch:
If you want to see our tool we have a website http://www.kentestman.com, but I'd encourage you to look at the other tools too, most of them have downloads or online versions.

Good luck,
Steve

▶ Reply to This

Permalink Reply by halperinko on 21 August 2008 at 7:51pm

Hi Thomas,

First I would like to say that you are on the right track - Test Management Tool can benefit your work.

There are more issues to consider before selecting a test management tool.
1st, one should consider other activities your company does, and relations to those.

If possible, I would suggest using a tool that manage Requirements, Tests, Bugs in a single tool in order to improve abilities of building a testing / version Dash Board.
This means you should consider which tools are currently being used for other activities, will other groups be willing to switch those into another, and what are the abilities to interconnect between the tools you select and existing tools if not replaced, as well as other tools such as Customer Support DB, and SW Configuration Manager (both should be well linked to the bug tracking part)
Mainly consider Req way of writing, as you wouldn't want to have to rewrite Req. into the DB just for the sake of traceability.

Also, one should first have a list of expectations from the tool, in order to define which tool will work best for you (I think you might find some initial data in Stickeyminds articles).

As people suggested several applications, I will suggest considering some free applications available in sourceforge, and also XStudio by XQual.
And also one COTS application I have enjoyed using - QPack by Orcanos.
(I have no relation to either of those...)

I think that having 50 steps in a single test might not be the right way to work, but as I do not know your exact implementation I will leave that note as that.

Indeed, as I view it, one of the main limitation of most if not all Test Management tools (though I managed to make Orcanos have some form of support for that), is the lack of ability to use Parameter Value tables as part of the DB in order to be able to write one steps script, and execute it with different sets of values each time, while having a place for the results per each set within the DB (in QC/TD you might be able to use external Excel sheets, and some say maybe the BP module can do the same)
This is the only disadvantage in compare to Excel, as I can see.

▶ Reply to This