You're NOT supposed to get outside of your box!

This is the story of bug reporting.

The story began one day when QA Engineer (automation consultant) first time connected to a service module that was signed for functional test automation. Typically, some exploratory manual testing is a good thing to start with. As a part of investigation it's also recommended to look under the cover to see HTML structure and scripts. It's proven quite useful if you want quickly identify areas with a potential defect like field value validation (implemented with a front-end script).

This time expectations were fulfilled.

First, in the hard-coded (defined right in the code) sequence of restricted characters some discrepancies were found and immediately tried. Voila, first fails: "9.0" + "1,5" isn't supposed to be "10", and "1.1,5" isn't a number at all.

Second, one critical field (it supposed to be a unique digital code, like account number) was not validated at all, and submitting combinations like "angle bracket" plus wrong code effectively broke the security allowing to bypass the web page, and see on the next one half-parsed HTML with some secured information that regular user is definitely not supposed to see.

Since the build was on UAT already, and the previous build was in Production, the Character notified QA Lead and Developer of the module of a high impact issues discovered.
So here is the scene we begin from.

QA Lead.
"Why did you do those negative tests? We need to automate only positive tests.
We have an outsourcing company that performs security testing for us".

Developer.
"You, testers, are supposed to do black box testing! You should have not looked into my code!"

QA Engineer.
"..."

I left the Character’s reply undiscovered because I’d like to substitute my thoughts here, and I invite everyone to express theirs. I’m only asking not to drive in a judging direction.

“Quality is everyone’s responsibility”, so what would you do in that immediate situation (as well as going forward in given environment) to make your personal contribution and bring a change, no matter how small it would be?

If someone wants to take other sides’ opinions (i.e. Developer’s, QA Lead’s, etc.) you’re welcome too.

0 members like this

▶ Reply to This

Replies to This Discussion

Permalink Reply by Jim Hazen on July 31, 2009 at 20:22

My reply would be this: "I found this while working on the automation. I looked at the code to see what I needed to get at and noticed a potential issue. I figured I would try it to see if it really was a problem and it turned out to be true. Manual exploratory testing is part of the process of figuring out what can be automated. Also, aren't you glad that I found the problem before a user in the field did?"

▶ Reply

Permalink Reply by Glenn Halstead on July 31, 2009 at 20:55

My reply... I know how to keep these guys happy...

"OK. I'm Sorry. We'll just pretend we don't know about these problems in the product."

▶ Reply

Permalink Reply by Shobh on August 1, 2009 at 19:41

Whatever the approach, either negative or positive, it is important to understand, QA are the first internal customers,
Only a skilled tester can do negative testing, understanding the real time scenarios from many different perspective, specialized skills of negative testing can reveal flaws that’s fatal to customer relationship.

Well I would support negative testing, thinking its not negative test based on the severity and the time of issue we find... :-)

Baseline reply for this is ‘My test cases include, testing validations of the fields which is the basic tests for any requirement; it failed and I found an issue!!’

▶ Reply

Permalink Reply by Rob Lambert on August 3, 2009 at 16:17

It seems to me your example is of someone working in an environment where people do the minimum (and less) that is required of them.

It also sounds like the managers and developers have little understanding, consideration or care for quality. I'd hazzard a guess that there are few unit tests and little respect to a tester who actually raises a defect. My guess is that the project also doesn't appreciate defect being raised, missed deadlines or the people they actually employ.

It sounds suspiciously like somewhere I used to work :)

It's sad that questions are still being raised about these sorts of topics (not sad of you, but that the situation still exists).

The QA engineer is facing a tough choice. Do as told and nothing more. Or have integrity in the career they have chosen and stand up for quality.

I would stand up and fight and take people down with me, but that's me. I've known people take all types of abuse, hatred and disrespect and still continue to work places - so I guess people are different.

I tried to change the culture at the place I worked that was like this. I tried for about 1 hour until I was 100% positive nothing could be done. It was a culture that had been there too long and there was no chance of changing it. Not at the position I was at. People were being sacked for finding too many bugs.

So I did the decent thing and left. I managed a whole 6 days there before throwing in the towel. And when I did it was amazing how many people congratulated me for having the sense to leave. Yet, most of them are still there.

So in answer to your question after my little ramble. My immediate suggestion would be stand up for quality...champion it...fight the corner, but if the QA Engineer has a clear moment of realisation that nothing will change the culture other than a new board of management - then to leave and keep their career on the right path. (assuming there are some jobs to move to)

It's the QA engineers career on the line and if they are working in a culture that supports only positive automation and actively enforces a white/black box testing culture then they need to sit and think hard about what's important to them. It's not about quality, it's about roles and responsibilities. It wont be long until that QA Engineer has all of the passion, creativity and positivity sucked out of them.

Careers and enjoying your job are crucial as is professional integrity...

Rob..

▶ Reply

Permalink Reply by Jim Hazen on August 3, 2009 at 19:37

Now we are starting to venture into the realm of economic impact. Both to the end-user and the company that made the software.

Think about this. What was the cost of you finding the problem vs. if the end-user had found it, AND what was the cost to the company overall in both scenarios. Key thing here is Rework. How much rework, and the cost involved in it, would need to be done in both situations. Do the math and explain it to some of those people and you might get some credibility, and leverage in the future.

When all is said and done at the end of the day the final thing that really matters is money, period. Show how you can help a company retain its money (and possibly more) and you will be in a much stronger position. Especially with the people that really matter, senior management.

▶ Reply

Permalink Reply by Michael Bolton on August 6, 2009 at 14:19

"If I found a problem, even though I was working in a way that you didn't expect, would you be interested?"

---Michael B.

▶ Reply

Permalink Reply by Albert Gareev on August 6, 2009 at 14:59

Thank you everyone for your detailed comments, analysis, and personal experience shared!

I have nothing to add to immediate reply by Jim Hazen.
For a QA Engineer's contribution to an overall project quality attitude my suggestions are the following.

1. Going extra mile
It has begun already in the example given since negative tests were not the objective. So the next step would be to document and e-mail defects discovered or submit into defect tracking system. At first it could be ignored or even met offensively but with the time it may become an established practice.

Another good practice is expanding the original test data set provided by QA. With a low effort here the coverage could be greatly extended. (Of course, low effort advantage is gained only with a properly developed data-driven test scripts).
The best candidates here will be boundary functional tests and SQL validation tests.

Personally I have an experience establishing automated boundary testing within the Functional Testing phase while originally scripts were intended for positive Sanity Testing on UAT phase only.

"Verba volant scripta manent". Or documentation, documentation, documentation. Whether it's test cases written nowhere but shown to you by QA or BA, setup or validation procedures, any other knowledge or analysis - it's worth to take a time to write it down in a soft copy (and share it with the team).

2. Knowledge forward and knowledge transfer

Under "knowledge forward" I meant sending within the team short e-mails with a various quick tips, articles, advices, and checklists.
I have a positive experience with this practice.

As for knowledge transfer, it doesn't have to be an officially scheduled session. Sometimes 5 minute sharing of ideas or practices (or just asking the right questions) can make a team mate thinking of the ways to raise the bar of quality.

Thank you.

▶ Reply

Permalink Reply by Tony Bruce on August 7, 2009 at 8:37

Mohinder Khosla asked me to post this for him while waits for his membership

This post also appeared on Test Repulic at the same time as this and surprisingly the added comments portray the same sentiments and conclusions are fundamentally the same.
In my experience, Tester’s brief has always been to test the system and there is hardly any mention of positive or negative tests. It is his responsibility to report faults that he finds whether he has to sneak into the code to come up with good test cases but in the end it is QA job on the line if sensible judgements are not made before the product is released. I do not agree that we should fight our corner just to gain brownie points so that our findings are accepted. Tester is in a good position to comment ‘I told you so’ when these bugs are discovered during UAT or production live.
I have to admit I am one of those who regularly look through code when I find requirements not complete or not clear to see whether the developer, who has direct link to BA, understood the requirements or he sought clarification before coding. I used to be responsible for code walking and just can’t break the old habit in order to find more information not available elsewhere and to see whether the developer written efficient code which may have performance issues later. You do need good relationships with the developer so you can report code related issues directly to him if required.