Comparing Commercial Test Tools

1 03 2011

This post is more about the comparison of 2 tools, which finally made the cut for an application/product, I have to Automate for my current Organization. They are a renowned name in the Smart Grid domain and have their own Smart Meters manufacturing. The application is the software API on top of these meters and their firmware, which allows the readings from the multiple meters (mostly in the thousands), to be collected and provides a Business Intelligence abstraction layer for the actual Hardware and Firmware. There were certain criteria which were needed to be implemented and taken into consideration before the final tool choice was to be made.

The team went through many .NET enabled Software Test tools – both commercial and open source, before finalizing on the below two, due to long-term stability and robustness. Also, we had to cater for emulators being used to test things which were critical to the business. All these points might not be mentioned below in the actual comparison of the tools, which has been made more generic for the purposes of posting on the blog.

Criteria on which the tool has been analysed HP QuickTest Professional v11 Visual Studio Test Professional 2010
Actual end user simulation: Is the test conducted using this tool equivalent to an end user action? QTP claims to perform end user simulation, in other words executing QTP scripts are equivalent to a person performing those steps manually on the application. Using the Coded UI tests, we can create UI test cases as they have been done using actual user interaction. You can execute tests with the browser minimized also, like Selenium, as it can use XPATH and DOM.
Support for UI Components QTP requires extra add-ins (plug-ins, not free) to work with .NET and other components, like Java, JavaScript, etc. Visual Studio natively supports .NET components. Also, JavaScript and other web scripting languages support is present, without additional plug-ins.
Object Management & Storage QTP comes built-in with Object Repository. Object Repository management is quite easy in QTP. Objects are recorded and added automatically to the Object Repository. Visual Studio Coded UI interface provides a limited set of Object Repository. It creates the user interaction internally in XML format and can be used in conjunction with screen position or the object name and ID.
Support for Dialog Boxes QTP supports all kinds of IE dialog boxes. These are helpful, when parsing error messages in the application under test. Especially when we expect a popup dialog to appear. Good support for embedded and IFrame dialog boxes. This has better support for IE browsers, being a Microsoft product.
Support for web browsers Cross-browser support is lacking in QTP. Scripts created for one browser may not run on another. It has cross browser support for IE, Safari and Firefox. These have been built by the specific vendors themselves.
Object Oriented Language Support & Scalability (as in Integration with External tools utilities and libraries). VBScript has limited OO support and QTP has limitations with using any other language for framework development. Supports C# as the major language. It is very similar to Java and has full OO support. Also, there is a large base of resources who are working with C# and .NET
Integration with Test Management tool With HP Quality Center and Test Director Integrated with Visual Studio Test Manager and Team Foundation Server.
Types of application supported Web, Windows (.NET, VB,  Power Builder, TCL/TK), Terminal Emulation, Command Prompt, Windows Desktop Native .NET, Command Prompt, Windows Desktop Native, Web Applications.
Support for different Operating Systems / Platforms QTP only supports Windows Current implementation of Coded UI, can support test cases on Windows and Linux boxes, as the application creates XML based code.
Technical Support QTP offers technical support by phone and mail, HP also has a web-forum. QTP user community is vast and questions posted on online forums get answered quickly. Although the technical support is available through phone and mail, the forums are not that intuitive now. But Microsoft has made efforts to have multiple Evangelists create blogs and forums to discuss user issues.
Cost Costly. $9,000 per seat license. Separate costs for Quality Center and other development and SCM related tools from HP. Costs $11,000, when bought with the Visual Studio Ultimate edition. But has other products bundled with it  – TFS, Test Manager, Visual Studio, etc.
Test Development Environment Reasonable but not the best. QTP tests can only be developed using QTP or Notepad like application. Best in the world. (my opinion, after Eclipse, it is the best and those who use it love the interface) 🙂
Integration with development process No real integration possible. Has plug-ins now which can integrate with other tools and development processes. Tests developed using VS TP can be easily part of the development project. Using tools like TFS and VS Build, Continuous Integration is easier.
Data Driven Testing Support for Data-driven and Keyword-driven testing, implemented using MS Excel. Good support for both Data-driven and Keyword-driven testing. With XML as the base for Keyword-driven.
Database Testing SQL (Structured Query Language) is integrated with QTP. Can make use of SQL statements from within QTP. Native SQL Server DB API’s are present. Allows command-line driven testing for validation and verification of the DB Integrity also. Support for Oracle is also present. (Will need to investigate this further)

Update: Integration of HP QTP scripts with Microsoft TFS is now also possible, as has been given in this excellent Lecture Series: StickyMinds.com Lecture Series

http://testingcircus.com





What does testing require?

26 09 2010

Testing is not an easy job. In India, software began in a big manner due to the test capabilities that were advertised for gaining a ground in the software field. That does not mean that we were not making good software, but test capabilities were the ones which catapulted us. It is not as simple as just writing a few scripts in shell or for the GUI. As James Whittaker has written, in his article on testing, it takes skill and a good knowledge of the domain that you are testing for. It is tougher than development of the same. Developers need to know the technology and they know the domain. Testers need to know much more. They need to know the workings of the application and the domain, along with how the user will use it.

From the viewpoint of the tester, it is never just a small portion of the feature that is being done work on. He has to know what all inputs can come into the product/feature and what kinds of output are expected by the downstream/next to make it to work. I learnt it that way, and that is why I love the profession of test. I know the product from the user viewpoint and also from the viewpoint of the Dev (the inner workings). Along the way, I made a lot of learnings. Although I would say that I lost the ability to program in any specific language; I learned a lot about logic and analysis of a problem.

Along with the above, testing also requires a lot of understanding of the tools that you need to use to implement the tests. This may be in the form of commercially available tools (Mercury QTP, SilkTest, Rational, WinRunner, etc.) or open source tools (Selenium, Watir, Fitnesse, etc). You can create your own using scripting languages or regular languages. After testing for a few years on different technologies and platforms, you should be able to shift from one to the other, which is not as easy for the developers who are working on a particular technology, but they find it easier to shift domains. What do you think…?! 🙂





Testing is Complete!

23 09 2010

A lot of time during the test process, we get to hear the phrase ” When will testing be complete?!”. The common refrain we can think of is – “Never”. In a way this might be the correct phrase; but that does not alienate us from not being responsible if the product gets a bug. As test people, we do not have the liberty to say – “Look I told, testing was not completed!” 🙂

There are many different aspects of judging when a particular test cycle is complete. One criteria depends on the test plan. The Test Plan, which is created using the design and engineering specifications document, should cover the common scenario’s. Along with the common scenario’s, we should also have what we in test parlance call the border cases and negative cases. The border cases are those which are not so common scenario’s, while the negative ones are if we use the application in a way it was not meant to be used, what is the behaviour.

Another very very imporatant aspect of development and subsequent testing, I was taught early on in my software education, was documentation. Whatever happens, the documentation should be kept on top. May it be a specification discussion or related to any new test that you add during development or testing. Mostly, one should not feel it as an attack on your ego, if someone else informs you on it, and then to mention why it was added and not have pangs of being crucified for missing it in the initial draft. That is the reason a draft is created. But mostly it is not so, and we miss some critical testing in this manner.

Coming to the topic of being “test complete”. The best bet is to write a comprehensive test plan based on the specifications and get it reviewed by all – dev and test teams, as well as the product managers. Once this exercise is done, rest assured that you have covered a good percentage of scenario’s on how the user can use your application. As they say in test – “Think like the user, not the owner of the product/component” 🙂





Bidding Adieu to Yahoo!

16 06 2010

It has been an awesome journey for me in Yahoo!. I have made some great friendships, which would last a lifetime and also managed to get to know some good people with whom I would like to be associated with now and in the future. This was a long journey (one of my longest tenure in a single organization), which was filled with a lot of learning and strives, some I faced with confidence and others where I did falter. A good journey altogether, which matured me and made me look at life from the eyes of others who mattered. During this journey, I met the person with whom I plan to spend the rest of my lifetime and have been managing to steer myself through mood swings and great loving times.

Yahoo! as a company, is one Great place to work for. You get all the freedom to do what you want, get to travel (and expense it to company accounts – when you go for Campus or Regular Interviews). You imbibe a lot of learning, most of which is available to you through the labyrinth, which in Yahoo! yodel, we call the TWiki (the Technical Wiki), although most of the material available freely on it is no where near ‘technical’ 🙂

The Yahoo! culture is one great learning force, which enhances the knowledge of many a fresh faces and minds. Many have tried to introduce some sort of process into this madness, in the form of ‘Yet Another Technical Place’, but the culture of adhoc’ism prevails and drives the energy and work force to yet newer and innovative levels. (I know I am going to get a reprimand from my oldest mentor, for such a long sentence, but couldn’t help it 🙂 )

Yahoo! is a brand name, which lives in the life’s of online people around the world, it is synonymous with the Internet, and might have lost its sheen in search, but is still a force to contend with when it comes content. It supplies the best on the web, without doubt. It was “Jerry and David’s Guide to the Web” and has remained “THE Guide to the Web”. Critics have said that Jerry should have taken the decision in favour of Microsoft, but they are not the people who have nurtured the Organization to the stage at which it is today. Yes, Jerry does have sentimental values for a company he created, but he took the right decision. Yahoo! is a BRAND, which stands out on the big bad Web, and Jerry knew he could turn it into a profitable venture. He has brought in the right person to bring about this change. She may be a bit outspoken, but she is making the right decisions to keep all happy.

Ok, enough of this rant. To conclude, I would say that Yahoo! has great internal talent pool and a culture to moot for. Now that is being cultivated to bring about the required changes to make Yahoo! happen.

My colleagues and friends made it a GREAT day for me and I was for once really feeling hollow and devoid of words on what to thank them with. It really does not feel that I am leaving, and saying good-bye to all that I have enjoyed and nourished. I do not wish to do so, specially after 3 1/2 years of being here. But, as they say – “All Good Things Have To Come To An End!!”, so does my really great innings at Yahoo!.

Thanks All for being there and all the help and love rendered over this long fruitful journey!

Adieu, Afscheid and Sustantivo…





Functional Testing, is it truly functional?!

2 06 2010

Most people who have been into the game of testing, drop words like functional, regression, end-to-end, system, integration testing like they are the ones who invented them. You ask the person on what exactly he wishes to attribute that kind of testing to, and you get a blank stare, which states – “You Stupid! Don’t you even know this much? And you wish to test my knowledge of it??!!” 🙂

Well! To me the concept of Function Testing is rather vague. It usually starts when the developers complete a component and hand it over to the test team. What needs to be checked and verified is the functionality of the component. How does one go about doing this? In the true sense, most of the time, functional is not about functionality, but more about the interface between multiple components in an application.

(more coming up…)





When do we say testing is complete?!

1 06 2010

This is one of the favourite questions asked during an interview for Test Professionals. The answers that I have heard in various interviews have been as varied. The answer to this question could be a simple – “When a tester has exhausted all the test cases he built for the application under test”. But, recounting on my experience, the answer is not this simple. The requirement for ending a test cycle and saying that the product is “Ok Tested”, takes in a lot more than this. It depends mostly on what the team perceives as the “testing” being completed. It may be as basic as running a single command to as complex as creating a whole new suite of test frameworks to check the product/application.

A good answer from my end would be more like, all the basic user interfaces (user can be internal or external, api or ui, etc.) have been verified as working and all the user scenario’s possible (with regard to the business requirements documented) have been validated. This exercise should result in zero P1/S1 bugs and x% of other bugs (as defined differently across projects and/or Organizations), which have been carefully filed and are reproducible with the given steps. Comments to the contrary are welcome 🙂





Regression Testing

28 05 2010

Regression with an automated test suite is a frequently used/abused phrase. People need to first understand what “regression” actually means.

Regression testing in a nutshell relates to the activity, by which we check a new application/product, for all the previous functionality; while keeping the newly added features unused. Basically, we validate that the new code has no effect on the older functionality. Regression is not used/should not be used to check for a functionality which has changed. As a change in functionality means that it is a new feature and needs to be verified first before it is validated.