Test Coverage – A Concept!

24 10 2011

These days I am trying to work on a concept known as Test Coverage. I call this a concept, as it starts off with something in the mind of the Management, fetters down to the Manager and finally is handed down to the Tester to carry out the said instructions. Without actually realizing, soon a graphical representation of our work comes out, in something which people call Business Intelligence (another much-hyped word these days, but will come to it later). The graphical representation goes on to show that the current set of tests which have been implemented/created, cover either “X” lines of code or “Y” number of Business Screens.

Is this a true representation of the complete scenario? Not what a Test Manager or a Dev Manager, who has enough thought process would like to think so. The above is a misnomer of how we go about treating an important issue like Test Coverage. Let me take you through a typical “Software Test Life Cycle” (don’t even start me off on that one). The requirements come out in the form of a BIG bunch of documentation, which has gone through various iterations and reviews with the Business people and the other Stake Holders involved (but rarely the Test Team). This bunch of neatly typed bundle is handed over to the Test Team in an official ceremony, which we call the “Beginning of the Test Cycle”. The Test Manager goes over this vast bundle of joyous documentation and then based on his “past” experiences, provides an estimate of what all will need testing and what test cases can be broadly done. This is called the “Estimation Period”, as usually a rough time period is provided, on when the Test Team will finish – includes Automation, Manual, Performance, Security and the jig-bang.

Once this “Estimation Period” is through, the task is handed over to the Leads to break down and offer an estimate, but based on what the Test Manager has already provided. Till this time, the actual team members are usually not taken into consultation, but the seniors of the Team are the confidants who will decide on what the underlings do. Finally a document starts taking shape, which for the sake of convenience we call the “Test Plan” or the “Test Strategy“, for want of a better name. This soon becomes the golden Bible/Vedas for the Test Team and they have to adhere to what has been said in it. Thereby the official STLC starts!

Once you have converted the BRD (Business Requirement Document) or the PRD (Product Requirement Document) to your test cases, you need to start actually implementing those test cases. This is the place where you start bringing in concepts like Test Matrix and Test Vectors, which in layman parlance (developer speak) mean the way that your tests are structured across the various data points for a particular view on the application. Now comes the really good part! This also lies the place where the above mentioned superior tester comes out and says that we are doing a Test Coverage of “X” lines of code, or a “Y” number of business screens (for GUI applications, which usually is 90% of tested applications). But does he actually know what he has covered with his test cases? Some do, while some have just made the assumptions, after reading blogs such as this one or from their superiors, who again might have obtained their knowledge from such places. The test cases are sorted out and some go over to the Automation Team to put in their regression suite, while others are manually vetted out and put through the paces of the “Bug Life Cycle”! (what this means to the globally scattered teams, depends on how much the management has spent of procuring a good issue reporting tool. My recommendation would be to look into Joel Spolsky’s FogBugz: http://www.fogcreek.com/fogbugz/). But to each his own …

Once the case of creating test cases and shoving them into the Automated Test Suite is completed, the Test Manager will jump and click a variety of buttons on his console (something which has been created by his Team to make life a brisk walk for him Or the Management has spent some more Money into procuring another one of those efficient tools out there). Thus, voila, a beautifully colored report of what passed and what failed, and specially “How much of Code/Screens were covered by our Testing”. Definitely a piece of Beauty for the Management!

But what is the real usefulness of such a report! In my honest opinion (IMHO), zilch… NIL! We did a good job of covering all the lines of code which were there, but did we cover the paths through which the code would be executed, I don’t think that is thought of even 25% of the time. Did we make sure that boundary values are covered? it might be that we have a few test cases making sure of this, but do they map to our coverage? Did we take care of the definite values that a few fields on our screen work on? No, this would be a definite gap most of the time… What we did do was this – a) Ensure that at least 85-90% of the code lines are covered by our test cases, executed using the Automated Scripts (Good! This might be an issue with doing through Manual tests, so no offence to Manual Testing here). b) Made sure that all the GUI screens are covered.

But, did we make sure that all the fields on that screen are covered, usually not. These are the places where we get issues. Also, most of time Negative testing is not given enough importance in such cases. The usual rant being – a) Did not have time. b) Is not that important, as such a case would not happen in Production? But these are important things and they convey the coverage of our tests. I will try to bring out more facets of this testing type in my next few posts and hopefully those are more helpful, than this one, which just rants about what is not being tested and/or how badly we test things …





What does testing require?

26 09 2010

Testing is not an easy job. In India, software began in a big manner due to the test capabilities that were advertised for gaining a ground in the software field. That does not mean that we were not making good software, but test capabilities were the ones which catapulted us. It is not as simple as just writing a few scripts in shell or for the GUI. As James Whittaker has written, in his article on testing, it takes skill and a good knowledge of the domain that you are testing for. It is tougher than development of the same. Developers need to know the technology and they know the domain. Testers need to know much more. They need to know the workings of the application and the domain, along with how the user will use it.

From the viewpoint of the tester, it is never just a small portion of the feature that is being done work on. He has to know what all inputs can come into the product/feature and what kinds of output are expected by the downstream/next to make it to work. I learnt it that way, and that is why I love the profession of test. I know the product from the user viewpoint and also from the viewpoint of the Dev (the inner workings). Along the way, I made a lot of learnings. Although I would say that I lost the ability to program in any specific language; I learned a lot about logic and analysis of a problem.

Along with the above, testing also requires a lot of understanding of the tools that you need to use to implement the tests. This may be in the form of commercially available tools (Mercury QTP, SilkTest, Rational, WinRunner, etc.) or open source tools (Selenium, Watir, Fitnesse, etc). You can create your own using scripting languages or regular languages. After testing for a few years on different technologies and platforms, you should be able to shift from one to the other, which is not as easy for the developers who are working on a particular technology, but they find it easier to shift domains. What do you think…?! 🙂





Bidding Adieu to Yahoo!

16 06 2010

It has been an awesome journey for me in Yahoo!. I have made some great friendships, which would last a lifetime and also managed to get to know some good people with whom I would like to be associated with now and in the future. This was a long journey (one of my longest tenure in a single organization), which was filled with a lot of learning and strives, some I faced with confidence and others where I did falter. A good journey altogether, which matured me and made me look at life from the eyes of others who mattered. During this journey, I met the person with whom I plan to spend the rest of my lifetime and have been managing to steer myself through mood swings and great loving times.

Yahoo! as a company, is one Great place to work for. You get all the freedom to do what you want, get to travel (and expense it to company accounts – when you go for Campus or Regular Interviews). You imbibe a lot of learning, most of which is available to you through the labyrinth, which in Yahoo! yodel, we call the TWiki (the Technical Wiki), although most of the material available freely on it is no where near ‘technical’ 🙂

The Yahoo! culture is one great learning force, which enhances the knowledge of many a fresh faces and minds. Many have tried to introduce some sort of process into this madness, in the form of ‘Yet Another Technical Place’, but the culture of adhoc’ism prevails and drives the energy and work force to yet newer and innovative levels. (I know I am going to get a reprimand from my oldest mentor, for such a long sentence, but couldn’t help it 🙂 )

Yahoo! is a brand name, which lives in the life’s of online people around the world, it is synonymous with the Internet, and might have lost its sheen in search, but is still a force to contend with when it comes content. It supplies the best on the web, without doubt. It was “Jerry and David’s Guide to the Web” and has remained “THE Guide to the Web”. Critics have said that Jerry should have taken the decision in favour of Microsoft, but they are not the people who have nurtured the Organization to the stage at which it is today. Yes, Jerry does have sentimental values for a company he created, but he took the right decision. Yahoo! is a BRAND, which stands out on the big bad Web, and Jerry knew he could turn it into a profitable venture. He has brought in the right person to bring about this change. She may be a bit outspoken, but she is making the right decisions to keep all happy.

Ok, enough of this rant. To conclude, I would say that Yahoo! has great internal talent pool and a culture to moot for. Now that is being cultivated to bring about the required changes to make Yahoo! happen.

My colleagues and friends made it a GREAT day for me and I was for once really feeling hollow and devoid of words on what to thank them with. It really does not feel that I am leaving, and saying good-bye to all that I have enjoyed and nourished. I do not wish to do so, specially after 3 1/2 years of being here. But, as they say – “All Good Things Have To Come To An End!!”, so does my really great innings at Yahoo!.

Thanks All for being there and all the help and love rendered over this long fruitful journey!

Adieu, Afscheid and Sustantivo…





Work Culture

12 01 2010

I am still not sure as to why people want you to be political in the work life. They want you to cut corners, claim credit for what you have not done and generally make you into something which you were not meant to be. My aspirations have been always to grow technically in whatever I do. Since my childhood I was interested in breaking open all sorts of things just to figure out, how in hell did they work. I was fascinated by computer games and miniatures and loved playing with computers (although managed to keep my hands away from opening them) and making miniature things. This fascination led me to join Engineering and carry forward my interests. I skipped getting into the armed forces as I thought it would curb my passion for electronics, but I landed into something worse… the cut throat competitive and politics filled world of the Indian Corporate sector.

People want you to change and start claiming points for what others have done, they want you to cut down others, as to them that is the only way that you can hope to grow. Why I ask? Why should this be the way of the corporate world? Can’t we live life on our achievements alone?





Appraisal System

14 11 2009

Was talking with a colleague on the current appraisal system that is adopted across most of the Organizations. They usually go with an yearly system, in which the person to be appraised is told to write out his goals for the year and then he gets to finish whatever he has stated in the goals over a period of a year. Well! most of the time, when the final appraisal is done, it is for the work done in the past quarter… and the person who has been consistent throughout the year loses out.





Interviewing the Interview Process

12 10 2009

Do people realize that when they go for an interview on the impression they make on the person[s] sitting across the table, trying to evaluate them? I think not. Most of the people I know ignore the basic fact that they are the ones who are looking out for an opportunity and it is they who need to create an impression. They regard the basic interview process as one in which they just need to write something on their CV and expect the interviewer to understand that although, it is there, they have not really worked on the same.

This practice and activity is predominantly seen with the senior members rather than the juniors. The seniors, when they reach a certain stage in their work career [5 years in Service companies and 7-8 years in Product based ones], they tend to lean towards managing people, rather than managing themselves. Hence, the failure to clear even the simplest of technical questions, which if given to a fresher, he/she would have done the same in a jiffy.





Yahoo! As A Technology Company

28 10 2007

The perception of Y! as an Organization has for a long time been that of a Internet/Web-based Company. From a Business point of view on what Y! does, this seems like a perfectly correct view point and that is what it is famous for. Well! I agree to the same that it is a Internet/Web-based Company, but as an Organization, all that require a lot of technological thought and expertise to sustain and grow, along with an extreme sense of innovation within the Organization. It is the perception of the Web tag along with the unique name (made more so famous by the song & dance of ‘Shammi Kapoor’, in India’s film Industry of yore), which has given the Company, although an excellent work culture, a wrong perception/view in the people’s mind, of it not doing anything except to build and maintain Web Pages.
The above perception is something which needs to be taken care of whenever a new hire is made. Lots of people do not/can not visualize the technology which is going into making the Web and Y! a better place to be in, when the Internet is growing at this pace. From the outside they vie this as a place where people churn out Web pages and goof off as if still in their teens and College days. From the inside, it is more of an Academia. There are people who do churn away on Web pages, but those are pages made from a lot of technological thought and innovation. They have to be optimized for the maximum User Experience and minimal load times. Each page has a function to take care of, and a lot of thought process and sweat goes behind each. Along with the above, an optimized delivery platform is also there to be taken care of. Cable companies claim that their bandwidth is being used, but I think a better way to put it is, that it is being utilized (and not for free, as they wish the rest of the World to believe). An immense amount of innovation and technological thought process goes into each component of a Web page and this is where people don’t/can’t appreciate the value of the same. They see a great looking interface and will go and visit it, but not give a thought to the work and sweat which has given them the page.
All I say, Y! is more technologically oriented than any of the various other product Organizations I have gone through. Also, they have a truly great environment and work culture, which is to be appreciated and hopefully preserved for the future of the Organization. At the same time, trying to dispel the wrong notion developing in people’s mind that technologically, it might not be the best place to work ad grow from.