Book Review: How We Test Software At Microsoft?
I ordered the book when I searched on Amazon for books on Software Testing. I tend to do that every so often as it provides me information if there is anything new! The difference this time was that I really wanted to read something different, basically to give myself a break from reading research papers! What really got my attention was the title! It is simple, but yet every tester would be curious to find out how they test at Microsoft! Yes, we can be sarcastic and ask ‘Do they Test at all at Microsoft? But the fact is that if you are a software tester, at some stage you have most likely used one of their products! Just like I used MS Word to type this out!
So now moving on to the book itself! After reading the first few pages I thought it was going to be more Microsoft centric and less about the testing (I was proven wrong later in the book)! Although, I must admit I enjoyed every bit of the first part. The book has been divided in four parts, Part I is ‘About Microsoft’, Part II is ‘About Testing’, Part III is ‘Test Tools & Systems’ and Part IV is ‘About the Future’.
These parts consist of chapters, ranging from Software Engineering at Microsoft in Part I to Analysing Risk with Code Complexity in Part II to Testing Software Plus Services in Part III and concluding with Solving Tomorrow’s Problems Today in Part IV. I won’t discuss each of the chapters, as I have highlighted that in the presentation I gave at the SIGIST conference. However, in this review I will highlight what appealed to me most, what was thought provoking and what was simply good reading!
In Part I, of course I enjoyed reading about the details of software engineering at Microsoft, but I particularly enjoyed the tester DNA term! I think it could be used very usefully to demonstrate testing needs specialised skills! Also, the career path diagrams are particularly useful to understand what career paths potentially exist in software testing. I was probably more drawn to it, because I started in the industry went back to academia, completed my PhD and am back in the industry. So different career paths at different organizations allow you to understand what possible options could be there! I particularly like the concept of IC (Individual Contributor) and how a VP can still be an IC. One other interesting point is about the test lead being the best tester! Increasingly the role of test lead tends to take away the thrill of testing, and may be the skill set.
One term that is mentioned in Part I and one I really intend to read more about is ‘Technical Debt’. In software testing, this term is definitely needed and probably its usage needs explanation! As I see it, it could be a topic for a separate discussion. But I can only write more, once I have read more.
And lastly, what really made me laugh in Part I was an advice ‘not to listen to the HiPPO’ in the ‘War Room’. So firstly, war room is simple, a room where decisions are taken but then who is the HiPPO? It is the Highest Paid Person in the Organisation!
Part II, focuses more on software testing techniques. I will not discuss this in too much detail as the material contained is what you would expect. But what makes these chapters easier to read than testing techniques for a certification is the fact that the techniques are illustrated with practical examples, the authors explain techniques which we as testers are assumed to know completely. So this part is particularly useful for testers who are involved with test case designing. Still from a personal perspective what appealed to me was the mention of the Triangle problem and how big companies like Microsoft still use the well known problems, and what is more interesting is that they are actually collecting data to support the empirical evaluation of a very well-know problem in testing. While I was reading about this, I think it was just the researcher in me that got very excited at the prospect of the availability of real data!
The only other thing about Part II that I have to inevitably mention is the chapter on Model Based Testing. I say inevitably purely because my PhD is on Finite State Machines Testing Method. I was keen to read about MBT at Microsoft, but I was slightly disappointed. Simply because, there was not much new for me in it, but from another perspective I thought it would be an ideal reading for people to introduce them to MBT who do not necessarily understand my research. I know the authors have already addressed this point in reply to another reviewer’s comment, and I do agree that if there is one chapter that could have a separate book on the topic it would be this one! And it is the only chapter that actually has a list of recommended reading.
Part III of the book focuses on Test Tools and Systems. It starts with an excellent chapter on ‘Managing Bugs & Test Cases’. And I say excellent, because I just couldn’t put the book down when I was reading this chapter. The concept of the Bug bar is very interesting, though I am not sure how that would be applicable to a scenario where the developer is only assigned for bug fixing. It also discusses the concept of bug triage, and I find it useful because I am actually trying to use formal triaging in my current project with a colleague who is very keen on it. I could go on about this chapter for a while, but I would say it is a must read for all testers but more so for test leads and senior testers.
The remaining chapters discuss automation, non-functional testing and information about tools, customer feedback, and testing software plus services. And going back to my reasons for picking this book up to read, I was very curious about the customer feedback chapter. The chapter actually made me think about why I kept hitting ‘Don’t Send Report’ button when I saw the Window Error Reporting pop up. Also, I have never signed up for Customer Experience Improvement Program CEIP that Microsoft runs. It did make me wonder whether I was being a responsible user or because I was the user I didn’t have to worry about feedback and errors! But someone has to. And I suppose they do, that is why Microsoft has 8000 people involved in the CEIP program.
Part IV is all about the future. It briefly describes the challenges that Microsoft faces, and I suppose by the time you are reading this chapter you are most likely thinking about challenges that your team or you as a software test professional would be facing. How much can the challenges being faced by Microsoft generalised. But the best thing about the last part is this quote by Machiavelli and its parallels in software testing, with which I will conclude this review!
Machiavelli wrote, “In the beginning of the malady it is easy to cure but difficult to detect, but in the course of time not having been either detected or treated in the beginning, it becomes easy to detect but difficult to cure"
So now moving on to the book itself! After reading the first few pages I thought it was going to be more Microsoft centric and less about the testing (I was proven wrong later in the book)! Although, I must admit I enjoyed every bit of the first part. The book has been divided in four parts, Part I is ‘About Microsoft’, Part II is ‘About Testing’, Part III is ‘Test Tools & Systems’ and Part IV is ‘About the Future’.
These parts consist of chapters, ranging from Software Engineering at Microsoft in Part I to Analysing Risk with Code Complexity in Part II to Testing Software Plus Services in Part III and concluding with Solving Tomorrow’s Problems Today in Part IV. I won’t discuss each of the chapters, as I have highlighted that in the presentation I gave at the SIGIST conference. However, in this review I will highlight what appealed to me most, what was thought provoking and what was simply good reading!
In Part I, of course I enjoyed reading about the details of software engineering at Microsoft, but I particularly enjoyed the tester DNA term! I think it could be used very usefully to demonstrate testing needs specialised skills! Also, the career path diagrams are particularly useful to understand what career paths potentially exist in software testing. I was probably more drawn to it, because I started in the industry went back to academia, completed my PhD and am back in the industry. So different career paths at different organizations allow you to understand what possible options could be there! I particularly like the concept of IC (Individual Contributor) and how a VP can still be an IC. One other interesting point is about the test lead being the best tester! Increasingly the role of test lead tends to take away the thrill of testing, and may be the skill set.
One term that is mentioned in Part I and one I really intend to read more about is ‘Technical Debt’. In software testing, this term is definitely needed and probably its usage needs explanation! As I see it, it could be a topic for a separate discussion. But I can only write more, once I have read more.
And lastly, what really made me laugh in Part I was an advice ‘not to listen to the HiPPO’ in the ‘War Room’. So firstly, war room is simple, a room where decisions are taken but then who is the HiPPO? It is the Highest Paid Person in the Organisation!
Part II, focuses more on software testing techniques. I will not discuss this in too much detail as the material contained is what you would expect. But what makes these chapters easier to read than testing techniques for a certification is the fact that the techniques are illustrated with practical examples, the authors explain techniques which we as testers are assumed to know completely. So this part is particularly useful for testers who are involved with test case designing. Still from a personal perspective what appealed to me was the mention of the Triangle problem and how big companies like Microsoft still use the well known problems, and what is more interesting is that they are actually collecting data to support the empirical evaluation of a very well-know problem in testing. While I was reading about this, I think it was just the researcher in me that got very excited at the prospect of the availability of real data!
The only other thing about Part II that I have to inevitably mention is the chapter on Model Based Testing. I say inevitably purely because my PhD is on Finite State Machines Testing Method. I was keen to read about MBT at Microsoft, but I was slightly disappointed. Simply because, there was not much new for me in it, but from another perspective I thought it would be an ideal reading for people to introduce them to MBT who do not necessarily understand my research. I know the authors have already addressed this point in reply to another reviewer’s comment, and I do agree that if there is one chapter that could have a separate book on the topic it would be this one! And it is the only chapter that actually has a list of recommended reading.
Part III of the book focuses on Test Tools and Systems. It starts with an excellent chapter on ‘Managing Bugs & Test Cases’. And I say excellent, because I just couldn’t put the book down when I was reading this chapter. The concept of the Bug bar is very interesting, though I am not sure how that would be applicable to a scenario where the developer is only assigned for bug fixing. It also discusses the concept of bug triage, and I find it useful because I am actually trying to use formal triaging in my current project with a colleague who is very keen on it. I could go on about this chapter for a while, but I would say it is a must read for all testers but more so for test leads and senior testers.
The remaining chapters discuss automation, non-functional testing and information about tools, customer feedback, and testing software plus services. And going back to my reasons for picking this book up to read, I was very curious about the customer feedback chapter. The chapter actually made me think about why I kept hitting ‘Don’t Send Report’ button when I saw the Window Error Reporting pop up. Also, I have never signed up for Customer Experience Improvement Program CEIP that Microsoft runs. It did make me wonder whether I was being a responsible user or because I was the user I didn’t have to worry about feedback and errors! But someone has to. And I suppose they do, that is why Microsoft has 8000 people involved in the CEIP program.
Part IV is all about the future. It briefly describes the challenges that Microsoft faces, and I suppose by the time you are reading this chapter you are most likely thinking about challenges that your team or you as a software test professional would be facing. How much can the challenges being faced by Microsoft generalised. But the best thing about the last part is this quote by Machiavelli and its parallels in software testing, with which I will conclude this review!
Machiavelli wrote, “In the beginning of the malady it is easy to cure but difficult to detect, but in the course of time not having been either detected or treated in the beginning, it becomes easy to detect but difficult to cure"
Add a Comment
-
Comment by Anna Baik on June 29, 2009 at 8:43pm -
Hi Sarah,
Thanks for a great book review - I hope you don't mind if I link this from the Test Book review group?
-
Comment by Ken Johnston on June 26, 2009 at 9:40pm -
I found the link to the video on chanel 9 and though its from 2004 it still a good example - http://channel9.msdn.com/posts/TheChannel9Team/Alex-Kipman-Inside-a-MS-Build-Bug-Triage-meeting/
This is a post from DevDiv and you can see they have the rising bar concept but they use some different terms. http://blogs.msdn.com/release_team/archive/2005/06/21/431333.aspx
I can't find much else published on our web sites. I know there is some on this in the MS Press Book I.M. Writes Hard Code. If you have it check page 94. If you don't I'll find out from Eric Brechner if he's posted anything on the subject.
-
Comment by Hariharan Sekhar on June 25, 2009 at 8:39am
- Hi Sarah the post was very informative....cheers :-)
-
Comment by Sarah Salahuddin on June 24, 2009 at 9:54pm -
Thanks for the message Ken. I really enjoyed the first part and in fact I presented the book review at my team meeting and it is amazing how little people actually know about Microsoft! I picked out the little notes that were present in the book and presented them as questions to the audience, it was amazing to see the difference between the actual numbers and what happens at Microsoft.
I will definitely be looking at the videos, I am very keen to read up and understand more about triage. Is there any recommended reading you could suggest?
I am not on Twitter as yet but have tried to read the blogs ever since I read the book. Thanks for such an interesting and different read on software testing!
-
Comment by Ken Johnston on June 24, 2009 at 5:13am
-
Sarah, section one of the book gets mixed reviews. Some individuals feel it was too rah rah (and having re-read recently I see some places to curb the enthusiasm) but many other readers have expressed how they enjoyed learning more about the way Microsoft runs as a company. It was kind of a risk to mix content more typical of a business book into a testing book but it seems to have worked. As two of those were my chapters I am very happy you liked them.
There are some good video podcasts on http://channel9.msdn.com/. I know there are some posts where teams let you sit in on a small team bug triage but I couldn't find the one I was looking for. It was posted a few years ago and I hope it's still on the site. I'll try again to find the specific video and get you the link.
If you're on Twitter or track blogs on MSDN Alan Page, Bj, and I are active on those sites.
You posted some good comments in your review and Alan answered the questions already so i'll just say thank you once again. I'm glad our book was a good read for you.
-
Comment by kashif ali on June 23, 2009 at 7:54am - thanks Alan, its really simple definition, i got the idea about these terms, i really appreciate ur concern :)
-
Comment by Sarah Salahuddin on June 22, 2009 at 8:04pm
- Thanks for the detailed reply Alan, of course you would be the best person to answer Kashif's question!
-
Comment by Alan Page on June 22, 2009 at 4:44am -
Thank you Sarah for posting this review. I'm glad you had enjoyed the book (and that you had fun talking about it). I hope you don't mind, but I'll take a stab at Kashif's questions..
IC = Individual Contributor - in other words, a non-manager
Bug Bar - most common meaning is the criteria for whether a bug is going to be fixed or not. As you get closer to release, the bug bar gets higher - meaning that bugs have to reach a certain level of severity in order to get fixed.
Tester DNA - its what makes us want to test more than any other role in software engineering. Those with Tester DNA like to figure out how things work...so they can figure out how to make it NOT work
Technical Debt - this is the work you really should do, but put off until later. E.g. "the documentation is out of whack, but we can still ship this release without it". Now you have work to do later because you put it off today. Another example is all of the "postponed" bugs that didn't hit the bug bar (see above) that are now waiting for you to address in the next release.
-
Comment by kashif ali on June 20, 2009 at 9:53am
-
hi sarah,
I read book review from someother URL about "HWTAMS", but your review is quite informative. i really liked rather enjoyed the term :Hippo" as you have mentioned, it made me laugh as well,:)
I dont have this book at the moment so i would like to ask you kindly post some informative material regrading terms like "IC" , " Bug Bar","Tester DNA" and "Techhnical dept" etc, it would be quite helpful.
Congrats for your Phd, and good luck for your future. Good Work :)
-
Comment by Sarah Salahuddin on June 19, 2009 at 9:34pm
-
Thanks Mark. I am really glad that people enjoyed the book review as I didn't want it to be academic :)
Well I suppose completing the PhD is just a start to a very exciting career I hope...that's why I enjoy SIGIST conferences, you get to see what a great future testing has!
Btw, I jotted a few points from your talk that I really found very interesting and would like to discuss with you. I noted your email during your presentation, but glad found you here! Will write to you over the next few days!
Find Us Elsewhere
Nice Things
Bloggers
© 2010 Created by Rosie Sherry
You need to be a member of Software Testing Club - An Online Software Testing Community to add comments!
Join Software Testing Club - An Online Software Testing Community