I just spilled my coffee on the table… what a shame. But wait a bit – I’m the only one in an office and if I clean up before anyone notice - the shame would be all gone, wouldn’t it?
If we fix bug in a sprint where it was introduced then our customer don’t need to know about the bug, do they? We are lucky to have JIRA defect reporting system with it’s ability to mark some bugs confidential hiding them from customers, aren’t we?
There is but one problem when you hide some bugs – how do you report test status? A test case that fails due to hidden bug - should it be reported as passed (how about the risk of hidden bugs) or not tested yet (QA is the bottleneck)? There are even more problems if test case pass/fail is not all/what your stakeholders want to know.
Agile magnified the problem, didn’t it?
Agile methodologies (and especially scrum) are all a great collection of good engineering and/or management ideas. As far as I remember only one thing surprised me and one intrigued. Collective estimation and commitment (i.e. planning game) surprised me (I lived in soviet union and thus am intolerant about anything collective). Daily visualized progress (task board) intrigued me.
I’ve never worked in a project using task board. However the need to provide project sponsors with weekly or even daily progress reflection (reports, that sponsors are able to understand and map with project business value) are there in most projects I’ve been in for last few years. I can’t do 2 weeks of “test design” anymore (i.e. “please don’t bother me for next 12 working days – I have to finish my tests before accepting any input and providing any status reports).
So if on waterfall project we only need to provide upper management with the report at the end of the project and could hide from them any problem that was fixed during the project. Or recalling the spilled coffee analogy we were alone in the room until the end of the project, but not any more…
I never lie, I just don’t say all the truth
Politics, that’s what test managers eat for breakfast. Even if I talk to a developer face to face or write them an email without cc their boss – I have to think twice not to offend the developer. I never say “this does not work, have you tested it yourself?” I say “I can’t manage to get this working, could you help me, maybe share your test data, your tests”? Quite often it appears the reason is apparently insignificant different in test data, environment or even the way we start application (i.e. they start it from IDE, I start it from command line).
I have to think even more when I talk to lead developers, engineering managers. I don’t want to escalate small issues we could resolve without a manager. I may not want to tell that we are a bit behind the schedule if I know that developers are busy fixing defect we found so far anyway.
Even harder is to communicate test status with executive management or customers – whenever I have to. It is a very good idea to agree on the list of issues with engineering manager before communicating them with executives or customers. Maybe some of them could be solved in an emergency mode and removed from the list to communicate. And it is always better, if a resolution plan and ideally date is added to each issue before communicating it.
Problem: extended reporting effort
So let’s return to the problem with test case pass/fail. Assume I’ve just tested a feature or a user story and found a single problem which is major from business point. Developer promise to fix it tomorrow morning. Now what’ the status of the User Story testing: passed, failed, on hold, blocked, not tested? It’s on hold for me (I’m waiting for developer to provide fix), failed for project manager (developer must fix it), in progress for developer (a pending task), not tested for customer (it is not done, done, done yet), isn’t it?
Well this is not a problem while it’s one bug only. But as a tester I’m used to report 10 bugs a day and retest even more a day. Maybe we could ask for a test case management system that generates different reports for different roles? Meanwhile I spend quite a lot of time hiding a part of the truth from people who better don’t ever learn that truth.