Management asks you to estimate testing. You try hard but your estimate is declined because it significantly extend what they had in mind before asking you.... Ever been in such situation?
There is nothing wrong about it! The goal of test estimation isn’t always the estimate. “Test estimation” in a lot of cases is actually a testing (service) quality (how much will we test) negotiation process; i.e. if you estimate twice as much you know you have to test less carefully than you would like to or do less documentation, or anything else.
In this blog I’m going to take a meta-look (from another angle, from large distance) at testing estimation. That’s something that helps me not to cry looking at what happens to my estimates (and my estimation efforts).
Experience: My first real estimation effort
I still remember time when I was reinforced to a project to help test team with effort estimation. We spend a lot of time with the team. We came up with something like three point estimate for a task brake-down list. i.e. we divided all the features required to test into items and estimated minimal, recommended worst-case estimate for each. And you know what? They got half of the estimated minimal effort. I could mention that the quality of the delivery was so terrible than they increased test team approximately 4 times (and got me to manage the team) for the next release, but that’s a different story…
Why do we do an estimate?
There are at least two reasons to make any estimate: to plan (i.e. estimate how much time the urgent new feature request implementation takes so I can tell customer how much will we delay the release) and to decide (i.e. estimate how much time this feature implementation will take and based on that we will decide with customer weather to implement it).
Testing is somewhere in between. They would never decide not to test, but they are not going to give you all the time you estimate, are they? They want to decide “how much” instead of “weather do or not”. And that’s why testing estimation is so hard. That’s why classic estimation methods don’t work; at least for me.
Illustration: what’s so special about Testing
Let’s take an example: I have to estimate two tasks (and I mean testing is much more like the first one):
1. I need to clean up C disk on my computer. I know that I have some software installed that I don’t use any more and that there are some folders that I don’t need any more. How much time it would take to clean it up a bit?
2. I need to install MySql and latest version of tomcat and websphere so that they could work simultaneous on my machine. I have to deploy our webapp on both of them and configure it to work with MySql. How much time do I need?
If I don’t have experience installing this stuff the second estimation may tricky, but if I’ve done that multiple times already I could give you some very precise answer. The first task is tricky by itself - it is not very well defined. What does it mean to “clean up a bit”? Well even if I change that to “clean up 500 MB” it wouldn’t make estimation much easier, would it? Perhaps I could uninstall a single application or delete a movie file and get 1 GB freed up, but I can as well work all day … especially if I have done the same on this computer a week ago already. And perhaps it is easier to by more space or even reinstall windows that trying to clean up a mess collected for years…
Either fortunately or unfortunately we don’t have to test until we “clean up” 500 bugs. We test until we clean it up “a bit”. So estimation get’s two dimensions: estimate “a bit” and an effort. And you can’t estimate until you start actually doing the task, can you? Just the same with testing, right?
What else is special about testing estimates?
As far as I know the only way to estimate development is so called test-task-driven approach where you estimate each sub-task. In testing there is an alternative: development-effort-driven (i.e. testing will take x% of development time). Of course, you apply your knowledge, past experience or even metrics to estimate each task in first case and decide value x in second case. In my experience it’s quite typical to do both and compare
Further reading and thinking
If you liked this blog and it provoked some thinking you may consider thinking about next step – control that the estimates are implemented. Adjust estimates as you go. There is a big problem specific for testing again. Maybe I’ll have a need and time to think about this topic this year – I’ll blog my thinking then.
Meanwhile you could see my older posts related to the topic: if you work in waterfall-like methodology then this one
I wrote 3 years ago; or recent one
for an agile (frequent deliveries, frequent estimation, detailed monitoring).