Software Testing Club -  online QA & Software Testing Community

I made a brief and rather dismissive comment about uTest on Twitter, and Rosie Sherry asked me to expand.

Firstly, my comment was a first impression, which I admit wasn't very good. However, I do want uTest to be a good idea, and I want to be persuaded that it's going to be of value to experienced professional testers. It would be great if something like this could really take off. However, I'm sceptical.

I can see how uTest would be valuable to development companies, especially for web applications. It will introduce another layer of testing that they'd have difficulty providing in house.

However, it strikes me that such testing is more suited to exploratory testing at a late stage in the lifecycle, after more formal testing has been completed in house. In effect, the uTest cycles look like a sort of beta release.

Exploratory testing is surely performed better by experienced testers, otherwise it can be pretty random button pressing. The uTest model, ie payment by bugs detected, and at a low rate, isn't likely to attract many experienced professional testers, especially if applications really are at a beta release stage, and most of the bugs have been detected.

Also, at this stage many of the problems would be related to usability, and this can be subjective. Testers could work on a defect report and find it's not accepted, because the client's subjective judgement is different, or because the defect is dismissed as cosmetic and of no great significance.

This leads on to another concern. The possibility of spending time detecting and reporting defects, then finding that many are not accepted, or that they have already been reported. No payment in such cases.

Testers would presumably have to invest some time in establishing how the application is supposed to work, and checking defects that have already been reported. They could then find that they find few defects, and that not all of these are accepted. Given that the rate per defect is fairly low it's hard to see that there's much money to be made for the time that would be required.

The majority of testers could therefore be inexperienced, even beginners. If the quality of testers is low then that forces the payment level down, and you've got a vicious circle. Or is my assessment of the payments based on UK levels, and are they set at a level that is going to get good, experienced testers from Asia?

When I was at school I used to work at a big racecourse, selling papers at big race meetings. We worked on a commission-only basis. We got 30% of the price of each paper we sold. I packed it in because I realised that the newspaper company had every incentive to take on as many sellers as possible, and that was what they were doing, making it extremely difficult for any seller to get a worthwhile wage for their day.

I fear that uTest could go the same way, and that the temptation for uTest and the clients would be to throw as many bodies as possible at the testing because there's no marginal cost to using an additional tester. If payment is only by results, and if the skill level of individual testers is low then that might seem a rational, though short term approach.

This isn't a considered essay on the subject, just a quick explanation of my fears based on my first impression. Like I said earlier, I want to be wrong! This could be great for contractors in between contracts, and testers who would really prefer to work from home if that's possible.

Is someone going to pick these sketchy thoughts apart and show why they're wrong?

Reply to This

Replies to This Discussion

You've possible seen it, or involved in it but there is an interesting discussion going on at QAforums about this as well, including one of the VP's from utest.

http://www.sqaforums.com/showflat.php?Cat=0&Number=510481&p...

I did sign up to have a look, but have not got involved in any of the projects since then. However, you can view the projects on offer and what other testers have been raising and I have to say I still remain sceptical as well.

Just a couple of quick points

The first thing I noticed is the amount of "Experts" involved. As there is no certification or checking involved (a point brought up in the discussion above) everyone seems to have classed themselves as an expert. However, if you look at some of the defects raised (and the subsequent 'disputes' over validity) the quality of some of these experts is a matter of debate.

You have mentioned what I believe to be a critical point, and one of the reasons I have so far invested no further time in the initiative, namely the time to 'learn' the application, read release notes and review previously raised issues. All this time is unpaid.

In defence of the companies involved some do seem to be going out of their way to provide notes, areas of concern to concentrate on and in some cases pre-prepared test cases. Unfortunately, as seems to have become increasingly common in the internet age, people are not prepared to read and learn and want to jump straight to the end result, in this case payment!

Reply to This

Thanks for that link. I hadn't seen the discussion.

I can see the benefits for development companies using it as beta release testing, but uTest seem to have wider ambitions than that. Can they really offer a full testing service, as James Whittaker implies in his blog (the link for which was in the QAforums debate.
http://blogs.msdn.com/james_whittaker/archive/2008/08/20/the-future...

I just don't get it. Either that or he just doesn't understand testing.

Reply to This

RSS

© 2010   Created by Rosie Sherry

Badges  |  Report an Issue  |  Privacy  |  Terms of Service

Web Analytics