Test/bug video recording - innovation or just hype?

Replies to This Discussion

Permalink Reply by phil kirkham on October 1, 2008 at 3:44pm

I'll try and respond to the list of reasons you gave on your original post about this

Easier to view a video than write a description - a lot of testers will already have the tests written so how hard is it to send the failed test case to the devs ?

Devs might not understand the description you wrote - that seems to be more a problem of your writing skills, learn to improve them rather than use a tool. ( and again, this 'problem' is not a problem if you already have a test case written )

Devs might not believe you - have a good relationship and build your credibility with the devs and this shouldnt be a problem. Problems like this could also indicate a problem with devs and test having different environments and a video wont fix this problem

Devs wont understand bug severity - wont this be decided by someone other than the devs ? Surely the product manager/business owner is the person to be deciding on this and it will be set in the bug db rather than an argument with a dev ?

I can see the use of it when exploratory testing and helping track down the hard to reproduce bugs and I would like tester education videos

For the big testing shops that write then run 1000's of testing scripts then I cant see video adding much value to their process. As to whether that process is valuable is a whole other discussion...

▶ Reply to This

Permalink Reply by Ido Schacham on October 1, 2008 at 4:52pm

Thanks for the response, you do present interesting points. However, they're not entirely true.

Quite often the test case isn't a description of the bug. Many bugs are outside the test cases or reproduce in a slightly different way than the test case suggests. Also, quite often the team leader, devs, and project managers need a more specific description of the bug and its symptoms, so often I find that the test case isn't enough.

About devs not understanding, it takes 2 to Tango. I try to make my descriptions as clear as possible, but quite often devs misunderstand them, are too lazy to read the full description properly or don't have the concentration for it, miss a certain important point about the bug, or have some natural language disabilities.

Currently one of the devs I'm working with in the company is one of my best friends from high school, and even he sometimes needs to see bugs with his own eyes! It's not just about the relationship, it's also about the fragile dev ego which is obviously out of QA control.

Testers often have a big say on how severe a bug is, team leaders and project managers depend upon it. Some bugs may appear as small problems, but they could prove to be extremely annoying to the user. There seems to be a difference in reading about a bug than seeing it happen, it might even be analogous to reading about a movie than seeing it for yourself.

Still, you didn't answer the question on the personal level - why don't you record test videos? Are you in the company house doing 1000s of tests so that recording is too much of a hassle?

▶ Reply to This

Permalink Reply by phil kirkham on October 1, 2008 at 5:39pm

they are discussion points so it's not a case of true/false, it's all dependent on context, experience etc

on a personal level, I havent felt the need for it apart from a few occasions when trying to nail hard to reproduce bugs and I'm now moving on from being a hands-on-tester to managing the testing process. Would I recommend the use of test videos ? Not been convinced yet

hopefully there will be some more viewpoints added soon...

▶ Reply to This

Permalink Reply by phil kirkham on October 2, 2008 at 10:21am

couple more thoughts

You say devs are too lazy to read the bug description - isnt there a danger that testers will become too lazy to describe/narrow down/reproduce the bug and just file a video as the bug report ?

What would the QA process be - should a video be logged for every bug report ? If not, how would you decide which ones ?

Somewhat off-topic and personal point - you dont seem to have much respect for devs - lazy, cant concentrate, fragile egos. Do you have a good relationship with them ?

Personal question for you - what is your experience in the testing industry in terms of size of projects, size of test teams ?

▶ Reply to This

Permalink Reply by Ido Schacham on October 2, 2008 at 10:42am

The danger of testers becoming to lazy to narrow down or reproduce bugs might not be a danger at all. It may be that it takes too much time for testers to narrow down and reproduce bugs, and even then some info that the developers need to figure out where's the problem may be missed. Testing videos with a short description and probably other debug info may be more than sufficient. Should testers really be spending so much time on reproductions rather than moving on to new more in depth tests?

The QA process should be as you would like it to be. If you find that recording test videos is too much of a hassle and that the test cases are enough 90% of the time as bug descriptions, then there is probably no need for them. If however test videos end up saving time and money since the bug descriptions are more whole that way and don't demand further inquiry from QA and capture hard to reproduce bugs, then record test videos. Or maybe just use them for your exploratory testing, or however it suits you.

I have plenty of respect for devs, I'm even somewhat of a developer myself and not just a tester. Do note that I wasn't making any silly generalization such as "all devs are lazy" or something like that. I was saying sometimes they are. In fact there are plenty of lazy testers and plenty of lazy people. It doesn't have to do with the profession but with the personality and mood at a point in time.

I've previously worked in a QA group of around 100 people. The teams I worked in usually had around 5 people in them or less. Projects were big. Not only was my employer rapidly expanding their existing products, but also working on parallel versions and other new products on the side.

▶ Reply to This

Permalink Reply by Sherilyn Tasker on October 1, 2008 at 10:24pm

The big one I can see is the amount of space potentially required to store these videos. I have no idea of how big these files would be and disk space may be cheap to some but you try fighting with your IT department to get a bit more space on the server to store videos and see how you go! We struggle to even get enough space to store dump files.
I haven't found a need to move to video as our developers are just down the hall..if we need them to see something happening we go and get them. I would reconsider if our developers were miles away and a visual would help.

▶ Reply to This

Permalink Reply by Tony Bruce on October 2, 2008 at 2:18pm

I have used video recording previously as an aid to logging defects. I used it in VMWare. I'd never use a video in place of a good descriptive log though, I'd use it as part of the log.
It also helped me when describing the steps I followed to get to the defect. I could double check I didn't miss anything and it would help me create a more decriptive log.
Essentially the more info I can supply the better and I see a video as an aid to defect logging, not a method.

▶ Reply to This

Permalink Reply by Ajay Bhagwat on October 2, 2008 at 9:16pm

The problem often is size of the avi file- it eats a lot of space.

If you are working as an independent tester, sometimes the video file makes a lot of sense to avoid communication issues that can cause due to time zone difference.

▶ Reply to This

Permalink Reply by halperinko on October 3, 2008 at 2:10pm

As stated in the past, the main limitations of video are:
1. file size
2. Serial viewing order - harder to skip and get to the point.
3. No search options.
4. Requires detailed attention in whole viewing process.

Eventually, If I was a developer looking at such video, I would have tried making a list of actions in my head if it was a short video, or start writing it down on a long video - so why don't we just jump to the next step - record the actions as a log.
This will take much less space, will allow easier search & fast look-up abilities, may speak the "developers language" if taken similar to OO method, keeps exact timing values (in case of parallel process that might be important), can be combined with other logs giving a better view of the whole system status / errors etc.

The only advantage of video as I see it, is for the limited amount of issues which describe visual effect (like in testing flash).

▶ Reply to This

Permalink Reply by Steve Swanson on October 4, 2008 at 9:52pm

So I first saw the post at the testuff blog, and I admit I have not read through all of the comments made thus far but wanted to address the comments made by halperinko. Firstly I want to address what the main 'limitations' are.

1. File Size.
If you get a decent tool, you don't have to worry about this. You can export your videos in flash (if high res isn’t necessary) making them ~100KB. Now a days at a normal big box store you can get a 500GB hard drive for under 100$, so you are looking at about 5 million videos for 100$ (or 500 000 videos at 1 MB each). To me that isn’t really what I'd consider a limiting factor.

2. Serial Viewing order - harder to skip and get to the point.
The way that I use videos (described below) this is not an issue. A test video used properly should not be long enough for this to be a problem.

3.No search options
Again if you have a short video, and you use the video accompanying another bug report, you will not have any problems.

4. Requires detailed attention
Again if you are using videos correctly this should not be an issue.


So how should videos be used in reference to bug reports...?

Firstly in my opinion a bug report will never be replaced by a video (well unless it is mashup of video with text) because there are things that you can easily and quickly describe in text. Videos are only helpful for certain things, a few of them are as follows:
- Remembering your course of actions when you hit a bug. I am sure it has happened to everyone once where they notice a bug but they have no idea how they got there.
- Helping to illustrate time sensitive bugs; for example you find something where, in a web form, you press save back save quickly and it causes a problem.
- When a developer may not believe you. Often I have filed bugs that developers do not believe happen (sometimes because it is intermittent behaviour), a video is proof, words are just that... words.
- When someone else on the test team has to explain a bug you filed. Wording can be tricky and filing a bug report is somewhat of an art. At times it is difficult to understand what the person that filed the bug meant, but if you have a video you can often figure it out.

There are other uses for videos, those are just the ones that jumped to the fore front.

What has to be remembered is that, like anything else, it is all fluid! Sometimes videos are appropriate, sometimes not, it is always a judgement call; you have to think to yourself will a video add value. If you are unsure maybe ask the developers or your team lead what will be most helpful to them! I think video is extremely useful with certain cases, so try not to disregard it too quickly.

--Steve
My Blog

▶ Reply to This

Permalink Reply by Bob French on October 3, 2008 at 8:00pm

We use VMware movies as an attachment to the bug report for those that are difficult to explain on paper or visual in nature. This is the rare exception rather than the norm.

I don't think a movie can or should ever replace a well written informative defect report for the reasons people have stated above. However, it can be a valuable "clarification" tool.

▶ Reply to This

Permalink Reply by Glenn Halstead on October 5, 2008 at 11:28am

Hi Ido,

I have used bug videos from time to time.

Like most in develpment and testing, I have worked with great people and incompetent bafoons. Until a person has convinced me of their compentence I am sceptical about their claims. I expect others to treat my claims / bug reports the same way.

I find that a bug video is particulary useful when I'm new to a project and have not yet built trust and credibility with the developers. I try to put myself in the developers shoes, I ask myself; why is the developer going to spend time on this defect report? Why is he going to believe that the issue I'm reporting actually happened. Is he going to understand the situation I'm trying to describe in my written text? Am I going to be filed in his 'great tester' or 'incompetent bafoon' bucket?

From time to time I have found a video to be a great solution for a particular communication need. Like most tools and practices, it's contextual.

Glenn

▶ Reply to This

• ‹ Previous
• 1
• 2
• Next ›
• Page