I’ve never been recommending doing too detailed test design (scripted tests) before test execution. The only exception was the tests we do to address regression risks. For years I believed that the outcome of a new feature testing must be well chosen tests, documented to the level to make it as fast as possible to do the regression testing. I hated to do the brainless and boring regression tests, but I did them because I knew I should – this is the best practice to follow.

Today I want to add this practice the the "things must die" list for me.

Facts/statistics from my experience:

1. While analyzing (RCA) regression bugs discovered in production we had almost 9 out of 10 bugs escaped into production due to the lack of the right/sufficient checks [1]  and less than one out of 10 – due to fact that a check were not run since regression occurred
2. On projects with good automated unit test coverage 9 out of 10 regression bugs I (or even more if don’t count those found by automated smoke test on each new build) are discovered while testing new functionality (because new and old functionality is always integrate and you have to use old functionality to do preparation steps)

But we did risk based testing, didn’t we?

My best practice was not just to design regression test checks during initial testing of a feature. We prioritized checks in low medium and high risk of a regression based on: information about (automated) unit tests, project failure patters (typical features to break, typical types of bugs to occur, the bugs found in initial testing, etc.). I would run higher risk checks more often later, making it more efficient usage of our testing resources.

There was an issue with this approach Old tests are blind to new problems. It ignored the new features and bugs-fixes to be done later. It ignored further project decisions and events, such as need to support new operational system, decision to switch to a free IDE, change one of 3^rd party vendors, increase unit test code coverage, etc. It ignored all the risks generated by further events. Today I know the things we ignored has major impact.

Risks have changed haven't they?

Things have changed: our requirements and software are more flexible, our development practices better address regression risks and I think the scripted regression testing is no more any effective (and never been efficient) way of using your testing resource

Why it’s still so popular?

Checks-based regression testing is easy to manage, i.e. forecast how much time it would take and monitor progress and report status. You could count the number of checks done/not done, passed/failed. Though it does not (in general) correlate with regression risk mitigation (i.e. 50% of tests don’t necessarily mean we have done 50% of the risk planned mitigation …). But test managers tend to fool themselves into believing it does (I did this mistake myself once) and have a very good feeling of control over regression risk.

But there are no better practices, are there?

In 2007 I published an alternative practice, where I was amazed how regression-testing could be 10 and more times as efficient and effective as regression-checking. I mean testing that is not guided/managed/driven by checks listed during previous tests cycles, but rather a process of “exploration, discovery, investigation, and learning”. There were a limitation to that practice however, a huge one – it would work fine it tester is only one on a project, or if you can split functionality into completely unrelated parts and assign one tester to each part. But you run the bus factor of 1 – should this tester get sick or take vacation you can’t do testing anymore. And testers got overwhelmed with information to analyze on long running projects, so only the most experienced and talented testers were able to do good regression testing on large projects.

Probably we need a better way to collect and manage the information and risks…

More blogs that inspired this one

One By Martin Jansson

Another oneby Anne-Marie Charrett