“The aim of ISO/IEC 29119 Software Testing is to provide one definitive standard for software testing that defines vocabulary, processes, documentation, techniques and a process assessment model for software testing that can be used within any software development life cycle”.
Oh dear! That's the introduction to “ISO/IEC 29119 Software Testing - the new international software testing standard”.
I'm afraid my hackles rise when I see phrases like “one definitive standard” and “used within any software development life cycle”. It immediately triggers an adverse emotional reaction as I remember this rhyme from Lord of the Rings, about the One Ring that would give the holder power over all.
“One Ring to rule them all, One Ring to find them,
One Ring to bring them all and in the darkness bind them
In the Land of Mordor where the Shadows lie”.
More pertinently, I also think of this piece of ancient software development wisdom, from McCracken and Jackson in 1982. I'm afraid I don't know of a free source.
“Any form of life cycle is a project management structure imposed on system development. To contend that a life cycle scheme, even with variations, can be applied to all system development is either to fly in the face of reality or to assume a life cycle so rudimentary as to be vacuous”.
The people who are putting ISO 29119 together are experienced, well intentioned professionals, and I'm sure that they are well aware of many of the potential pitfalls that they face.
However, I'm far from convinced that the exercise will ultimately prove worthwhile. Is there a happy middle ground between an excessively prescriptive standard that will be an irrelevant nuisance in many situations, and a high level standard “so rudimentary as to be vacuous”?
Perhaps the framers of the standard will get the balance right. However, even if they do, I worry about what will happen when the standard is released. Even if the standard's creators envisage a set of useful guidelines, from which people can select according to their need, the reality is that other people will interpret them as mandatory statements of “best practice”.
Lawyers and large consultancies will see a business opportunity, and write contracts that require the production of all the documentation examples offered by the standard. Regulators and legislators will pounce on the standard as evidence of professionalism, with failure to comply being interpreted as deviation.
I believe that guidelines are extremely valuable, but allowing them to mutate into rigid standards is dangerous. I've written about this here, and I won't repeat my arguments in any detail.
I have three basic concerns about standards in software development.
1 – They inhibit the growth of novices and constrain experts.
2 – They are consistent with, and in practice encourage, a fundamental misunderstanding about the nature of software development, which is an iterative process of design and discovery, not a linear production process.
3 – Standards, by their very name, encourage non-experts to insist that they should be mandatory, without any understanding of whether they are relevant in a particular context. In my experience goal displacement poses a major threat. The focus of projects can become production of mandatory documentation and adherence to standards, rather than the real work.
The question, as posed in a tweet by Michael Bolton the other day, is “What can we do, then, to stop ISO 29119? A worldwide movement is called for”.
Forums like the Software Testing Club are great, but often we use them to preach to the converted. Testing conferences may reach a more diverse audience, but the real problem is neither the testers who want to introduce standards to our profession, nor the standards themselves. The problem lies with the way that they are applied. We therefore have to look to those outsiders who will turn the standard from guidelines to rules.
So what do we do?
I think we should be reaching out to these outsiders, networking, cultivating contacts and influencing those who can help us, our who might make our lives difficult if they don't understand what we are doing.
I used to work as a computer auditor, so I believe strongly that good auditors can be a great ally. They are often, wrongly, blamed for imposing mandatory standards. Managers often demand slavish adherence to standards, especially detailed documentation, “because the auditors insist”. Only bad auditors insist that software development standards should be applied without exception, in every case, regardless of context. Likewise, the US Sarbanes Oxley is often wrongly blamed for demanding excessive documentation. That's something I expand on in the last past of my article (referred to above) and also in this blog about the similarities between testing and auditing.
What do you think? Are you bothered? Are you happy enough to work with whatever standards are produced? If so, why?
If you are worried, what are you going to do?
This should really be a blog posting, but I want to start a discussion and a forum post works far better for that.
I hope that this is something that people will want to discuss. A new testing standard will affect the way that huge numbers of testers will have to work, whether they like it or not.
I have two questions:
1.Do you know exactly the date that ISO 21119 will be officially available to public?
2. Do you also know if a specific department (Software Quality Assurance & Control) of a company not the whole company can be certified?
The timeline on the ISO site is vague, but it looks like the three final drafts will come out late this year or sometime next year, with the final versions being released in the second half of 2013.
There's no suggestion (as far as I know) of companies being certified as complying with the standard. The idea is that the standard is released as being "best practice". Big corporations will then adopt it and insist on suppliers complying. Suppliers in turn will be happy to claim they' will comply, not because it will help them build better systems, but because the standard will allow them to carry out work that is unnecessary, but lucrative.
The publication of a new standard bothers me. My current employer is unlikely to insist on its application, so it will not immediately affect me in that sense, but I believe it could be harmful to the profession as a whole.
My main gripe is that because this is being presented as an ISO standard, it means the recommendations it makes are likely (in my experience) to be 'forced' on testers, regardless of whether they are suitable for the task at hand.
I haven't read through it all yet (although on first glance I note it seems to be quite documentation-heavy) but I hope that management teams choose to scrutinise standards prior to insisting on them, in order to ensure what is being used is applicable and adds value to the project in question. Lengthy test plans and rigorously-documented test cases are no use unless someone is reading them.
As tempting as it may be to denigrate any standard that encourages testers to think in "one true way", I wonder if a knee jerk reaction to some flow charts is helpful?
I would encourage anyone who wants to challenge this standard to spend some time researching it to understand what its trying to achieve.
Then apply a testers mindset to it in a thoughtful and considerate manner.
Or blast it beyond this galaxy..the choice as a context driven tester is all yours...
Thanks for joining in Anne-Marie. I was hoping you might because you have exactly the perspective needed to provide a real debate, as opposed to a slagging fest.
Ok, this is a provocative post to stimulate debate and not a sober academic article. I accept that the creators are trying to produce something worthwhile, and I think that they will come out with a lot of material that will be good if testers are allowed to select what's useful for them at any given time. The problem isn't the content.
Perhaps you're right that my reaction to the flowcharts was a bit "knee jerk". However, my concern is the way that others will interpret them, and they may see the linear flow and ignore the comment about rework being necessary.
I think a reasonable question is; how will the creators and supporters of the standard try to prevent them being abused by non-experts who want to insist that they're applied universally and often inappropriately?
That is the likely consequence of the standard coming out of ISO, as Matt B has said.
I should add - I don't want to bash those defining the standard or the standard itself. I am sure the content and the suggested practices will have value in context. I think the issue is more related to how some portions of the industry view standards in general.
I wonder if this will feed back into the ISTQB/ISEB certification as well.
I certainly don't want to knock any of the people who've drafted the standard. Perhaps I erred on the side of impact rather than thoughtfulness in the first few paragraphs. I doubt if I'd disagree with Anne-Marie on the content of any particular part of the standard, and if I disagreed with her about how it should be applied then my first reaction would be to question my own judgement. That's not really my point, however. It's the bigger picture of how the standard will be used and abused that concerns me.
Actually I have read parts 2 and 3 of the standard (they are still in a draft state) and what amazed me most was the lack of "one true way", especially in part 3 "Test Documentation".
They did a good job encouraging thinking, for example the words "variations", "lighter" or "Agile" can be found in the examples Annexes.
That's interesting. The drafts don't seem to be publicly available. What I've seen isn't encouraging. The only downloadable document is a presentation from 2010 that doesn't mention "variations" or "lighter". It refers to Agile and iteration only in passing when it mentions the need for trialling the standard in all project types.
I'll be very interested to see how they strike the balance between being overly prescriptive and losing relevance on the one hand, and being too high level to offer practical guidance on the other. And above all I'm keen to see whether they can define the standard, and make it tailorable, in such a way that it supports testers who are trying to adapt it to their particular project, and frustrates lawyers, senior management et al who want to use it to provide a suffocating blanket of documentation.
search the ISO store for it http://www.iso.org/iso/store.htm
part 1 and 4 are not for sale yet, part 2 costs 66CHF and part 3 costs 98CHF.
Ah... the word "context" appears 28 times in part 3 of the document :)
I see the standards like the filling in a sandwich - the bottom slice of bread are those that don't really know what they are doing, just starting, or looking for info, and hopefully use the standards to rise to an acceptable level of foundation, whereas the top slice are those who have experience and know how, and use what they require from the standards while applying their own methods where necessary. For those that will mandate the standards I think you will find will find something to mandate, so it may as well be the testing standard (better the devil you know than the devil you don't right?).