Why do we write test cases? To manage testing? Yes I used to believe it is easier to manage defined test cases rather than people. I’ve developed a different approach. However you will have to wait for my next blog to read about it. In this blog I want to share my (2 years of) experience with managing test cases in JIRA.
Hint 1: JIRA is for management, not for content
First of all we don’t store test case content (sometimes referred to as test steps though I don’t like this definition as my test cases tend to be more than instructions) in JIRA. We store content in confluence WIKI which integrates perfectly with JIRA. Our JIRA test cases has a custom field Test URL. This gives additional bonuses:
1) WIKI provides complete version control
2) WIKI provides labelling, hierarchy of pages, some scripting, etc. Thus we could organize tests for huge projects in manageable way
3) If we need to repeat test execution we don’t have to copy-paste content. It saves space. It also helps if we need to fix a content mistake.
4) Test Case content review may be done in parallel to execution. Wiki labels may help to track review status. We sometimes create additional JIRA (tasks) to manage test case content review (in past years we’ve found that it is wrong to postpone test execution only because someone has not finished the review)
Hint 2: We use link feature extensively
We link test cases with bugs found running them. We sometimes link defects with test cases that are blocked due to a defect (so one really bad defect may be linked with hundreds of test cases). We sometimes link defects with development JIRAs (New Feature). It sometimes appear quite helpful for traceability and for defect retesting.
Hint3: think about indexing
If you add Test Case JIRAs to the same project where you store bugs, you could reuse fields like Component. There is a bit of confusion with Affects and Fix Version however. We use Fix version to indicate which software version (build) test case is tested against.
Hint4: you may want a custom JIRA Life-Cycle
I agree with Doug Hoffman: tests don’t simply pass or fail .
We may want richer status information. Even with only pass and fail some projects don’t want to allow transition Failed->Passed, but others do (with mandatory to change affects version to indicate that test case failed on previous build is passed on this one). Some would need “In Progress“ status (not there by default).. Some want to distinguish between failed and blocked statuses. Some want status reopened. Some want to have the same test case reopened for regression tests others want to create new JIRAs to easier get statistics.
So it is better if you can define your own life-cycle and screens for a test case.
Issues
There are some issues however:
1) When I test a test case and find a bug I have to create new bug, fill in it’s index information, and link with test case. This may be automated – a lot of index information could be copied from TC, and link could be created automatically. We are considering to write some custom code to manage this…
2) When you have a lot of test cases you may want to organize them into test suites (also known as test plans). We have some ideas how to do this in a semi-automated way (either as sub-tasks, links or custom field), but not yet done.
To Be continued
For small projects I prefer not to use JIRAs for managing test cases. WIKI is so wonderful tool that I could report my test progress directly in wiki content. See a few samples here
However on large projects if our management want to see overall status and requirements traceability and other things that Test Case Management system may provide – it may be a good idea to still use JIRA TC. So I just started using JIRA to address those two things: to provide requirements traceability and progress (but not to direct testers). I’m using term “Test Case” in a way that would hardly be accepted by most of testers. Details in my next blog.