Test Coverage – A Concept!

24 10 2011

These days I am trying to work on a concept known as Test Coverage. I call this a concept, as it starts off with something in the mind of the Management, fetters down to the Manager and finally is handed down to the Tester to carry out the said instructions. Without actually realizing, soon a graphical representation of our work comes out, in something which people call Business Intelligence (another much-hyped word these days, but will come to it later). The graphical representation goes on to show that the current set of tests which have been implemented/created, cover either “X” lines of code or “Y” number of Business Screens.

Is this a true representation of the complete scenario? Not what a Test Manager or a Dev Manager, who has enough thought process would like to think so. The above is a misnomer of how we go about treating an important issue like Test Coverage. Let me take you through a typical “Software Test Life Cycle” (don’t even start me off on that one). The requirements come out in the form of a BIG bunch of documentation, which has gone through various iterations and reviews with the Business people and the other Stake Holders involved (but rarely the Test Team). This bunch of neatly typed bundle is handed over to the Test Team in an official ceremony, which we call the “Beginning of the Test Cycle”. The Test Manager goes over this vast bundle of joyous documentation and then based on his “past” experiences, provides an estimate of what all will need testing and what test cases can be broadly done. This is called the “Estimation Period”, as usually a rough time period is provided,¬†on when the Test Team will finish – includes Automation, Manual, Performance, Security and the jig-bang.

Once this “Estimation Period” is through, the task is handed over to the Leads to break down and offer an estimate, but based on what the Test Manager has already provided. Till this time, the actual team members are usually not taken into consultation, but the seniors of the Team are the confidants who will decide on what the underlings do. Finally a document starts taking shape, which for the sake of convenience we call the “Test Plan” or the “Test Strategy“, for want of a better name. This soon becomes the golden Bible/Vedas for the Test Team and they have to adhere to what has been said in it. Thereby the official STLC starts!

Once you have converted the BRD (Business Requirement Document) or the PRD (Product Requirement Document) to your test cases, you need to start actually implementing those test cases. This is the place where you start bringing in concepts like Test Matrix and Test Vectors, which in layman parlance (developer speak) mean the way that your tests are structured across the various data points for a particular view on the application. Now comes the really good part! This also lies the place where the above mentioned superior tester comes out and says that we are doing a Test Coverage of “X” lines of code, or a “Y” number of business screens (for GUI applications, which usually is 90% of tested applications). But does he actually know what he has covered with his test cases? Some do, while some have just made the assumptions, after reading blogs such as this one or from their superiors, who again might have obtained their knowledge from such places. The test cases are sorted out and some go over to the Automation Team to put in their regression suite, while others are manually vetted out and put through the paces of the “Bug Life Cycle”! (what this means to the globally scattered teams, depends on how much the management has spent of procuring a good issue reporting tool. My recommendation would be to look into Joel Spolsky’s FogBugz: http://www.fogcreek.com/fogbugz/). But to each his own …

Once the case of creating test cases and shoving them into the Automated Test Suite is completed, the Test Manager will jump and click a variety of buttons on his console (something which has been created by his Team to make life a brisk walk for him Or the Management has spent some more Money into procuring another one of those efficient tools out there). Thus, voila, a beautifully colored report of what passed and what failed, and specially “How much of Code/Screens were covered by our Testing”. Definitely a piece of Beauty for the Management!

But what is the real¬†usefulness of such a report! In my honest opinion (IMHO), zilch… NIL! We did a good job of covering all the lines of code which were there, but did we cover the paths through which the code would be executed, I don’t think that is thought of even 25% of the time. Did we make sure that boundary values are covered? it might be that we have a few test cases making sure of this, but do they map to our coverage? Did we take care of the definite values that a few fields on our screen work on? No, this would be a definite gap most of the time… What we did do was this – a) Ensure that at least 85-90% of the code lines are covered by our test cases, executed using the Automated Scripts (Good! This might be an issue with doing through Manual tests, so no offence to Manual Testing here). b) Made sure that all the GUI screens are covered.

But, did we make sure that all the fields on that screen are covered, usually not. These are the places where we get issues. Also, most of time Negative testing is not given enough importance in such cases. The usual rant being – a) Did not have time. b) Is not that important, as such a case would not happen in Production? But these are important things and they convey the coverage of our tests. I will try to bring out more facets of this testing type in my next few posts and hopefully those are more helpful, than this one, which just rants about what is not being tested and/or how badly we test things …

Appraisal System

14 11 2009

Was talking with a colleague on the current appraisal system that is adopted across most of the Organizations. They usually go with an yearly system, in which the person to be appraised is told to write out his goals for the year and then he gets to finish whatever he has stated in the goals over a period of a year. Well! most of the time, when the final appraisal is done, it is for the work done in the past quarter… and the person who has been consistent throughout the year loses out.

Work @ Big Corporates

20 12 2006

So, I have started work at another one of those so-called Big Corporates which they say have the best policies for their employees. But does the Work Culture and the kind of work being offered also effect the employee’s or are they to be silent spectators in the grand wide scheme of things of churning out one badly conceptualized product after another?
Even the best of organizations are unable to offer the type of employee satisfaction that is required when working. I think it mostly depends on the seniors to ensure the type of quality work and environment which compels the employee to stick around and do the work given sincerely and to his own satisfaction level rather than that of the Organization. We do have a responsibility towards all those who are working with or for us to ensure that they are constantly energized to do the kind of work that is required from them. work and Money do not go hand-in-hand everywhere, but with the rising expectations [on the Money front] of the new breed of IT people who are coming into the field has fueled a sort of a mad rush where everyone just seems to be running for the money and not the work that is offered.
We are ready to take up a downright insulting work to our intellect rather than compromise on the Money part. Then the cribbing starts…
Why do we do this? Have we been ignoring these issues in the past so that the monster has become uncontrollable? I think so, we try to think of everything in terms of Money and in this we tend to spend a lot of it even when we might have not earned it, this way we start looking for avenues to make more and end up in the classic trap of making more Money without looking at the Work we may be doing to earn the same. I think it is high time we understood our priorities and started doing what we really wish for instead of chasing that elusive Money which might never give us the satisfaction we need.

Then again, it is also the prerogative of the seniors who need to start making the Work more interesting so as to reduce the problem of Attrition being faced by all. If we start making work interesting along with the money then we might just manage to curb this fascination of people to splurge and then look for avenues to retrieve back that lost Money. This way we might even manage to save the ever impeding problem of salaries in India crossing the profitability mark for the Corporates and save ourselves from a Big Depression…