Test Coverage – A Concept!

24 10 2011

These days I am trying to work on a concept known as Test Coverage. I call this a concept, as it starts off with something in the mind of the Management, fetters down to the Manager and finally is handed down to the Tester to carry out the said instructions. Without actually realizing, soon a graphical representation of our work comes out, in something which people call Business Intelligence (another much-hyped word these days, but will come to it later). The graphical representation goes on to show that the current set of tests which have been implemented/created, cover either “X” lines of code or “Y” number of Business Screens.

Is this a true representation of the complete scenario? Not what a Test Manager or a Dev Manager, who has enough thought process would like to think so. The above is a misnomer of how we go about treating an important issue like Test Coverage. Let me take you through a typical “Software Test Life Cycle” (don’t even start me off on that one). The requirements come out in the form of a BIG bunch of documentation, which has gone through various iterations and reviews with the Business people and the other Stake Holders involved (but rarely the Test Team). This bunch of neatly typed bundle is handed over to the Test Team in an official ceremony, which we call the “Beginning of the Test Cycle”. The Test Manager goes over this vast bundle of joyous documentation and then based on his “past” experiences, provides an estimate of what all will need testing and what test cases can be broadly done. This is called the “Estimation Period”, as usually a rough time period is provided,Β on when the Test Team will finish – includes Automation, Manual, Performance, Security and the jig-bang.

Once this “Estimation Period” is through, the task is handed over to the Leads to break down and offer an estimate, but based on what the Test Manager has already provided. Till this time, the actual team members are usually not taken into consultation, but the seniors of the Team are the confidants who will decide on what the underlings do. Finally a document starts taking shape, which for the sake of convenience we call the “Test Plan” or the “Test Strategy“, for want of a better name. This soon becomes the golden Bible/Vedas for the Test Team and they have to adhere to what has been said in it. Thereby the official STLC starts!

Once you have converted the BRD (Business Requirement Document) or the PRD (Product Requirement Document) to your test cases, you need to start actually implementing those test cases. This is the place where you start bringing in concepts like Test Matrix and Test Vectors, which in layman parlance (developer speak) mean the way that your tests are structured across the various data points for a particular view on the application. Now comes the really good part! This also lies the place where the above mentioned superior tester comes out and says that we are doing a Test Coverage of “X” lines of code, or a “Y” number of business screens (for GUI applications, which usually is 90% of tested applications). But does he actually know what he has covered with his test cases? Some do, while some have just made the assumptions, after reading blogs such as this one or from their superiors, who again might have obtained their knowledge from such places. The test cases are sorted out and some go over to the Automation Team to put in their regression suite, while others are manually vetted out and put through the paces of the “Bug Life Cycle”! (what this means to the globally scattered teams, depends on how much the management has spent of procuring a good issue reporting tool. My recommendation would be to look into Joel Spolsky’s FogBugz: http://www.fogcreek.com/fogbugz/). But to each his own …

Once the case of creating test cases and shoving them into the Automated Test Suite is completed, the Test Manager will jump and click a variety of buttons on his console (something which has been created by his Team to make life a brisk walk for him Or the Management has spent some more Money into procuring another one of those efficient tools out there). Thus, voila, a beautifully colored report of what passed and what failed, and specially “How much of Code/Screens were covered by our Testing”. Definitely a piece of Beauty for the Management!

But what is the realΒ usefulness of such a report! In my honest opinion (IMHO), zilch… NIL! We did a good job of covering all the lines of code which were there, but did we cover the paths through which the code would be executed, I don’t think that is thought of even 25% of the time. Did we make sure that boundary values are covered? it might be that we have a few test cases making sure of this, but do they map to our coverage? Did we take care of the definite values that a few fields on our screen work on? No, this would be a definite gap most of the time… What we did do was this – a) Ensure that at least 85-90% of the code lines are covered by our test cases, executed using the Automated Scripts (Good! This might be an issue with doing through Manual tests, so no offence to Manual Testing here). b) Made sure that all the GUI screens are covered.

But, did we make sure that all the fields on that screen are covered, usually not. These are the places where we get issues. Also, most of time Negative testing is not given enough importance in such cases. The usual rant being – a) Did not have time. b) Is not that important, as such a case would not happen in Production? But these are important things and they convey the coverage of our tests. I will try to bring out more facets of this testing type in my next few posts and hopefully those are more helpful, than this one, which just rants about what is not being tested and/or how badly we test things …

When do we say testing is complete?!

1 06 2010

This is one of the favourite questions asked during an interview for Test Professionals. The answers that I have heard in various interviews have been as varied. The answer to this question could be a simple – “When a tester has exhausted all the test cases he built for the application under test”. But, recounting on my experience, the answer is not this simple. The requirement for ending a test cycle and saying that the product is “Ok Tested”, takes in a lot more than this. It depends mostly on what the team perceives as the “testing” being completed. It may be as basic as running a single command to as complex as creating a whole new suite of test frameworks to check the product/application.

A good answer from my end would be more like, all the basic user interfaces (user can be internal or external, api or ui, etc.) have been verified as working and all the user scenario’s possible (with regard to the business requirements documented) have been validated. This exercise should result in zero P1/S1 bugs and x% of other bugs (as defined differently across projects and/or Organizations), which have been carefully filed and are reproducible with the given steps. Comments to the contrary are welcome πŸ™‚

Job Interviews – Taking

20 05 2006

Today was quite a hectic day. We had gone to Chandigarh for hiring a few people, and I woke up at 3 AM in the morning to get ready and reach Chandigarh at 1030 Hrs. As the interviews were going to start at 11. We had a lovely journey in a Innova with tubeless tires, one of which kept getting depleted of air and there seemed to be no way that it could be repaired until we got back to delhi, except that we had to keep getting air filled every time we saw a petrol station… πŸ™‚

The interviews went off well at the Picadilly in Sector 22. We even managed to select a couple, then had a good lunch, courtesy the HR Manager [along with a bottle of beer]. On moving back, just before Panipat, we got caught in a BIG jam, created due to the construction of the road!! Here the road is being widened by L&T and was full of traffic trying to get through the jam and creating more of a Jam in the process.

It was a LONGGGG day when I finally reached home around 2300 Hrs… that constitutes about 3 hours sleep if I sleep at 2300, as last night I slept at 0115 in the morning. But, here I am writing and updating this blog entry and talking to someone special… πŸ˜›

Bangalore – What it Holds!!

22 10 2005

I had always been fascinated by the lure of Bangalore – A Software Company in every street and every third house housing another small startup, also along with these varied software and IT companies an Engineering/Arts College to match the growing needs of these upcoming companies; at places where even you would not think of.
That was the lure that bought me to Bangalore, when I was offered a job here. That illusion broke down when I arrived in Bangalore. Going everyday to Office from a place called HSR(Hosur-Sarjapur) layout on either the bike or by Bus; the first thing I realized was that it was not going to be anywhere near a easy task. What with having to face the daunting local and corporation buses and to top it all the unruly auto rickshaw drivers; who drove a hard bargain to get you to the destination of your choice at the most exorbitant rates that they could come up with. During one of these sorties after missing my bus that I found that if you ignore them when they say these exorbitant prices they automatically come down to the real rates at which it is reasonable to take an autorikshaw… 😦

The Bangalore that i arrived at was full of people trying to find a place – to park, to drive, to walk, to stop, to enjoy and to generally get on with work and life; the image I had in mind was completely different…
Fortunately for me I had a few like-minded friends with me and they made my first few days enjoyable, before I finally got down to the nitty-gritty gritty of what I had arrived at Bangalore for – Work!!!

Already too much has been said of the excellent traffic and the traffic conditions of our Dear Ole Bengalaru… So I thought I should also add my two bits… πŸ™‚

I have been in Bangalore for almost 18 months now and have seen the situation go down to the worst… by nature I am a foot traveler and enjoy the walk through the city where you can get to observe many things which you otherwise miss on. This was my way when I arrived to this city and started with going to my Office on foot. I used to enjoy these walks when I arrived in Bangalore. Those were the days when we used to go from the hostel in which I was staying to the Office building which was in Kormangala industrial area, the walk was a good 1 1/2 to 2 Kms and both used to enjoy the same, with the relatively cleaner air in the morning, when we used to either catch the bus going towards Raheja Arcade or walk it down if we were early enough (early used to be 0845 and late used to be 0900… πŸ™‚ ). Then while coming back from Office we used to enjoy some time in Forum, which had just about opened up and then walk back, almost always. Then came the days that just after 39 days of coming to Bangalore, I changed the company I was working for and joined another which was just starting operations in India (the first center for development outside of the US). Here too I meet like-minded people who enjoyed the walk (not to the office, as that was in Whitefield area; on the ITPL road). When we got down at the Raheja Arcade, and walked it down a bit to either the Aiwa’s or Forum building; 200 yards or so… :-). Here after having a snack or two we used to head back to our homes, me waiting for my friend working a few yards from Forum to join me so that we would then walk it down to the hostel.

Well! now back to the traffic area. The second reason for coming back walking from the location of my first company, was that the inner roads were cleaner and we could easily walk across without hearing the honking of horns and the smoke and smog of the vehicular traffic which used to be coming our way. This usually was the norm, until 2 months down the lane the roads were segregated and some even became 1 ways. The traffic volume increased and so did the problems related to it. We changed routes and went through the more quieter regions of Kormangala, which has some real good architectural beauties which seem to be left empty by their creators. I really enjoyed those days as compared to nowadays when all you hear is the honking and people getting irritated at the people who share the roads with them…

I have a lot more on this and there will be a second edition for this soon… πŸ™‚

Lazyness and Sleeping on Weekends!!

27 08 2005

So, what do we all do these days, when we get our weekly breaks on the Saturdays and Sundays… for me it is SLEEP, and what I have heard from all those whom I have encountered till date, although my mates in Macromedia make that a habitual thing for most of the weekdays also… πŸ˜›

I was trying to get a consensus on this and found that some people actually get up early on Saturdays!!! And then I found out that these were the f(a)tness buffs… πŸ˜€ They actually think that getting up real early on a weekend (something like 7 in the morning) and going for a walk/jog will actually turn then into f(i)t, well I am as yet to achieve that for now (the fit and the fatness regime) I am fat enough now and hopefully should be going on the fitness regime soon enough if my roomie’s have their way and manage to lure me out to the cool clean environs of the morning air soon enough and at a regular pace, then I also think I will join in the ranks of those who think that waking up in weekends early is good… !!! πŸ™‚