Test Coverage – A Concept!

24 10 2011

These days I am trying to work on a concept known as Test Coverage. I call this a concept, as it starts off with something in the mind of the Management, fetters down to the Manager and finally is handed down to the Tester to carry out the said instructions. Without actually realizing, soon a graphical representation of our work comes out, in something which people call Business Intelligence (another much-hyped word these days, but will come to it later). The graphical representation goes on to show that the current set of tests which have been implemented/created, cover either “X” lines of code or “Y” number of Business Screens.

Is this a true representation of the complete scenario? Not what a Test Manager or a Dev Manager, who has enough thought process would like to think so. The above is a misnomer of how we go about treating an important issue like Test Coverage. Let me take you through a typical “Software Test Life Cycle” (don’t even start me off on that one). The requirements come out in the form of a BIG bunch of documentation, which has gone through various iterations and reviews with the Business people and the other Stake Holders involved (but rarely the Test Team). This bunch of neatly typed bundle is handed over to the Test Team in an official ceremony, which we call the “Beginning of the Test Cycle”. The Test Manager goes over this vast bundle of joyous documentation and then based on his “past” experiences, provides an estimate of what all will need testing and what test cases can be broadly done. This is called the “Estimation Period”, as usually a rough time period is provided, on when the Test Team will finish – includes Automation, Manual, Performance, Security and the jig-bang.

Once this “Estimation Period” is through, the task is handed over to the Leads to break down and offer an estimate, but based on what the Test Manager has already provided. Till this time, the actual team members are usually not taken into consultation, but the seniors of the Team are the confidants who will decide on what the underlings do. Finally a document starts taking shape, which for the sake of convenience we call the “Test Plan” or the “Test Strategy“, for want of a better name. This soon becomes the golden Bible/Vedas for the Test Team and they have to adhere to what has been said in it. Thereby the official STLC starts!

Once you have converted the BRD (Business Requirement Document) or the PRD (Product Requirement Document) to your test cases, you need to start actually implementing those test cases. This is the place where you start bringing in concepts like Test Matrix and Test Vectors, which in layman parlance (developer speak) mean the way that your tests are structured across the various data points for a particular view on the application. Now comes the really good part! This also lies the place where the above mentioned superior tester comes out and says that we are doing a Test Coverage of “X” lines of code, or a “Y” number of business screens (for GUI applications, which usually is 90% of tested applications). But does he actually know what he has covered with his test cases? Some do, while some have just made the assumptions, after reading blogs such as this one or from their superiors, who again might have obtained their knowledge from such places. The test cases are sorted out and some go over to the Automation Team to put in their regression suite, while others are manually vetted out and put through the paces of the “Bug Life Cycle”! (what this means to the globally scattered teams, depends on how much the management has spent of procuring a good issue reporting tool. My recommendation would be to look into Joel Spolsky’s FogBugz: http://www.fogcreek.com/fogbugz/). But to each his own …

Once the case of creating test cases and shoving them into the Automated Test Suite is completed, the Test Manager will jump and click a variety of buttons on his console (something which has been created by his Team to make life a brisk walk for him Or the Management has spent some more Money into procuring another one of those efficient tools out there). Thus, voila, a beautifully colored report of what passed and what failed, and specially “How much of Code/Screens were covered by our Testing”. Definitely a piece of Beauty for the Management!

But what is the real usefulness of such a report! In my honest opinion (IMHO), zilch… NIL! We did a good job of covering all the lines of code which were there, but did we cover the paths through which the code would be executed, I don’t think that is thought of even 25% of the time. Did we make sure that boundary values are covered? it might be that we have a few test cases making sure of this, but do they map to our coverage? Did we take care of the definite values that a few fields on our screen work on? No, this would be a definite gap most of the time… What we did do was this – a) Ensure that at least 85-90% of the code lines are covered by our test cases, executed using the Automated Scripts (Good! This might be an issue with doing through Manual tests, so no offence to Manual Testing here). b) Made sure that all the GUI screens are covered.

But, did we make sure that all the fields on that screen are covered, usually not. These are the places where we get issues. Also, most of time Negative testing is not given enough importance in such cases. The usual rant being – a) Did not have time. b) Is not that important, as such a case would not happen in Production? But these are important things and they convey the coverage of our tests. I will try to bring out more facets of this testing type in my next few posts and hopefully those are more helpful, than this one, which just rants about what is not being tested and/or how badly we test things …





What to Automate?

19 09 2011

I had this interesting conversation on Automation the other day with my colleagues at my new job. It started off fairly innocently on how the automation should be thought out and what needs to be done to automate. We already have an existing framework and test scripts in place, which very efficiently work and report issues. The problem comes when we need to provide data on what automation actually does. How do you prove the effort spent on automation? You could easily say that it saves time and resources by checking for faults early in the development cycle, but how can we be sure that it actually covers scenarios that check the application? This is where a business requirement document becomes a necessity.

The issue which we increasingly face today is how do we relate the business requirements to what we test. There are a few things which are given in textbooks and across such certifications organizations like ISTQB, which provide information on such. The thing is ultimately it depends on the person who is sitting and working on the application (which most of the Program/Product Managers miss out on) and the person who is writing the scripts to automate the application testing. The best way to figure out for a restricted zone (propriety) application is to go and sit with the users and find out what they use the most (or run a key stroke capture software and see where all it goes, limitations of this later). With a public/global reach software, it is best to give out Beta versions of the same, like most of the Big Organizations do and see what is reported back.

The other end of the spectrum is propriety software, which might not have a user interface (system tools like Compilers is one example). For this the technique is basically to read through the Software Requirement Document/Specification and have confidence in your abilities to decipher the jargon written in those and convert them to simple English. I got my initial training on writing code on these only, and the golden rule for us was [in a summary]:

  • Read through the document and write what you have understood for each function
  • Create the Algorithm for each in plain English pseudo code
  • Convert these into test cases, and run those test cases
I think the above rules of going through the steps of Why, What and How has helped us out a lot to become what we are today 🙂




What does testing require?

26 09 2010

Testing is not an easy job. In India, software began in a big manner due to the test capabilities that were advertised for gaining a ground in the software field. That does not mean that we were not making good software, but test capabilities were the ones which catapulted us. It is not as simple as just writing a few scripts in shell or for the GUI. As James Whittaker has written, in his article on testing, it takes skill and a good knowledge of the domain that you are testing for. It is tougher than development of the same. Developers need to know the technology and they know the domain. Testers need to know much more. They need to know the workings of the application and the domain, along with how the user will use it.

From the viewpoint of the tester, it is never just a small portion of the feature that is being done work on. He has to know what all inputs can come into the product/feature and what kinds of output are expected by the downstream/next to make it to work. I learnt it that way, and that is why I love the profession of test. I know the product from the user viewpoint and also from the viewpoint of the Dev (the inner workings). Along the way, I made a lot of learnings. Although I would say that I lost the ability to program in any specific language; I learned a lot about logic and analysis of a problem.

Along with the above, testing also requires a lot of understanding of the tools that you need to use to implement the tests. This may be in the form of commercially available tools (Mercury QTP, SilkTest, Rational, WinRunner, etc.) or open source tools (Selenium, Watir, Fitnesse, etc). You can create your own using scripting languages or regular languages. After testing for a few years on different technologies and platforms, you should be able to shift from one to the other, which is not as easy for the developers who are working on a particular technology, but they find it easier to shift domains. What do you think…?! 🙂





Testing is Complete!

23 09 2010

A lot of time during the test process, we get to hear the phrase ” When will testing be complete?!”. The common refrain we can think of is – “Never”. In a way this might be the correct phrase; but that does not alienate us from not being responsible if the product gets a bug. As test people, we do not have the liberty to say – “Look I told, testing was not completed!” 🙂

There are many different aspects of judging when a particular test cycle is complete. One criteria depends on the test plan. The Test Plan, which is created using the design and engineering specifications document, should cover the common scenario’s. Along with the common scenario’s, we should also have what we in test parlance call the border cases and negative cases. The border cases are those which are not so common scenario’s, while the negative ones are if we use the application in a way it was not meant to be used, what is the behaviour.

Another very very imporatant aspect of development and subsequent testing, I was taught early on in my software education, was documentation. Whatever happens, the documentation should be kept on top. May it be a specification discussion or related to any new test that you add during development or testing. Mostly, one should not feel it as an attack on your ego, if someone else informs you on it, and then to mention why it was added and not have pangs of being crucified for missing it in the initial draft. That is the reason a draft is created. But mostly it is not so, and we miss some critical testing in this manner.

Coming to the topic of being “test complete”. The best bet is to write a comprehensive test plan based on the specifications and get it reviewed by all – dev and test teams, as well as the product managers. Once this exercise is done, rest assured that you have covered a good percentage of scenario’s on how the user can use your application. As they say in test – “Think like the user, not the owner of the product/component” 🙂





Bidding Adieu to Yahoo!

16 06 2010

It has been an awesome journey for me in Yahoo!. I have made some great friendships, which would last a lifetime and also managed to get to know some good people with whom I would like to be associated with now and in the future. This was a long journey (one of my longest tenure in a single organization), which was filled with a lot of learning and strives, some I faced with confidence and others where I did falter. A good journey altogether, which matured me and made me look at life from the eyes of others who mattered. During this journey, I met the person with whom I plan to spend the rest of my lifetime and have been managing to steer myself through mood swings and great loving times.

Yahoo! as a company, is one Great place to work for. You get all the freedom to do what you want, get to travel (and expense it to company accounts – when you go for Campus or Regular Interviews). You imbibe a lot of learning, most of which is available to you through the labyrinth, which in Yahoo! yodel, we call the TWiki (the Technical Wiki), although most of the material available freely on it is no where near ‘technical’ 🙂

The Yahoo! culture is one great learning force, which enhances the knowledge of many a fresh faces and minds. Many have tried to introduce some sort of process into this madness, in the form of ‘Yet Another Technical Place’, but the culture of adhoc’ism prevails and drives the energy and work force to yet newer and innovative levels. (I know I am going to get a reprimand from my oldest mentor, for such a long sentence, but couldn’t help it 🙂 )

Yahoo! is a brand name, which lives in the life’s of online people around the world, it is synonymous with the Internet, and might have lost its sheen in search, but is still a force to contend with when it comes content. It supplies the best on the web, without doubt. It was “Jerry and David’s Guide to the Web” and has remained “THE Guide to the Web”. Critics have said that Jerry should have taken the decision in favour of Microsoft, but they are not the people who have nurtured the Organization to the stage at which it is today. Yes, Jerry does have sentimental values for a company he created, but he took the right decision. Yahoo! is a BRAND, which stands out on the big bad Web, and Jerry knew he could turn it into a profitable venture. He has brought in the right person to bring about this change. She may be a bit outspoken, but she is making the right decisions to keep all happy.

Ok, enough of this rant. To conclude, I would say that Yahoo! has great internal talent pool and a culture to moot for. Now that is being cultivated to bring about the required changes to make Yahoo! happen.

My colleagues and friends made it a GREAT day for me and I was for once really feeling hollow and devoid of words on what to thank them with. It really does not feel that I am leaving, and saying good-bye to all that I have enjoyed and nourished. I do not wish to do so, specially after 3 1/2 years of being here. But, as they say – “All Good Things Have To Come To An End!!”, so does my really great innings at Yahoo!.

Thanks All for being there and all the help and love rendered over this long fruitful journey!

Adieu, Afscheid and Sustantivo…





Functional Testing, is it truly functional?!

2 06 2010

Most people who have been into the game of testing, drop words like functional, regression, end-to-end, system, integration testing like they are the ones who invented them. You ask the person on what exactly he wishes to attribute that kind of testing to, and you get a blank stare, which states – “You Stupid! Don’t you even know this much? And you wish to test my knowledge of it??!!” 🙂

Well! To me the concept of Function Testing is rather vague. It usually starts when the developers complete a component and hand it over to the test team. What needs to be checked and verified is the functionality of the component. How does one go about doing this? In the true sense, most of the time, functional is not about functionality, but more about the interface between multiple components in an application.

(more coming up…)





Regression Testing

28 05 2010

Regression with an automated test suite is a frequently used/abused phrase. People need to first understand what “regression” actually means.

Regression testing in a nutshell relates to the activity, by which we check a new application/product, for all the previous functionality; while keeping the newly added features unused. Basically, we validate that the new code has no effect on the older functionality. Regression is not used/should not be used to check for a functionality which has changed. As a change in functionality means that it is a new feature and needs to be verified first before it is validated.