Automation Tool across Web, Mobile and Web Services!

26 03 2012

Earlier in the week, I was sent across a request from one of our Senior Management on what could be a best tool, used for the automation of tests across the spectrum of Web (HTML & Flash), Mobile (iPhone, Android, Windows, etc.) and Web Services. What I could come up on this is the following. People may disagree with these options and may have different opinions and views on it… please feel free to comment and put them through, to improve on the content 🙂

Looking into the problem from the requirements viewpoint, I believe Selenium would be the tool best suited for the above automation work. The issue which might go against it, is that their Mobile product is still in Beta, and they are not the best for Web Services Testing, Watir being the frontrunner in the Open Source (i.e., Free) tools in that category. There are other Commercial Tools also which are available with good support and good interface, making it easier for the Automation to be maintained; which is somewhat of a problem with the Open Source tools, if not properly designed initially. Commercial products also have a big following and hence are cost-effective in the long run, although they might be expensive to procure, but getting a resource who is great in an Open Source product can sometimes be a big recruitment headache.

That said, Flash/Flex is a group, which almost with all tools requires a debug/special build to be provided for testing. Each tool has their own quirks and libraries with which the Flash/Flex application needs to be compiled with. So, you might wish to go more into each tools individual ability and reviews of their Flash library functionality; especially for Web Based applications.

Coming to mobile applications, the market for these exists as a very fragmented field for testing successfully. With Android Browser, iPhone Safari, IE Mobile and Firefox being the major browser contenders for the Automation tools available, along with testing of the Apps within the iOS, Android, Windows Phone and the various other vendors out there. I have seen many people refer to the Experitest SeeTestMobile tool, which might be becoming a tool of choice for many, these days.

I plan to go over some of the tools which might help out in each group, and some which might have multiple categories covered below. These opinions are my own through what I have experienced with them, and all are free to criticize and cajole me into making changes as is reflected “great” for them…

Selenium

Advantages: Good for Web GUI Testing. Great tools available for Firefox browser and the new WebDriver combined with PageObjects concept make it a great cross-browser test tool for the HTML/JavaScript Web. It even has a Flex/Flash plug-in for compatibility with the [debug/developer] flash applications. Can be coded in multiple languages (Java [most popular], Perl, PHP, Python, C#, etc.). This is a Free Open Source Tool.

Disadvantages: Not very intuitive, depends on coding skills and good design. New WebDriver is good, but there are not many in the market who can create some really good frameworks and know how to use it properly. Requires knowledge of XPath and JUnit type of coding to do anything great with the tool. Mobile product is still in Beta. Not many people available and consultation fees with consultants and resources can be high.

HP Quick Test Pro

Advantages: Well supported and lots of resources available who have certifications, but mostly used in Financial Institutions. Integrated add-ons for Flex, Web Services, Silverlight, and Web HTML. Framework issues can be easily taken care of with Odin AXE framework, which uses XML and simple interface.

Disadvantages: Ability to recognize complex UI and dynamic content hinders the tool. Mostly used in Data-driven web testing, which makes use of Excel sheets; easy for the user to use, but may cause issues in maintainability. Windows System only focused. Not suitable for Unix-Clones and Mac OS. High deployment costs.

MicroFocus / Borland SilkTest

Advantages: Good tool for Web and Flash. (MicroFocus has recently bought it after Borland failed, not sure of its development path going into the future). Has support for other platforms and operating systems.

Disadvantages: Learning curve, due to its test coding language. Not many people available with the tool knowledge.

Watir

Advantages: Good Open Source Tool for Web Services and Web Testing. Used with Fitnesse, produces easy to create and support web tests and web services tests. Not too good with Flash and Mobile.

Disadvantages: Uses Ruby as the language of choice, which is a skill getting hard to find for Testing.

 

SAHI

Advantages: Great tool for Web testing. Has good variety of plug-ins for the various other technologies. Available as Free version and supported paid version. Support for the same is great, the Developer of the tool is quite helpful in working out the issues with the Test Team. Good for complex websites, where other tools may sometimes fail. Unlike Selenium, it does not make use of XPath to identify objects; and can be used across browsers for recording tests.

Disadvantages: Only used for Web Testing for now. [not sure if it has been updated with plug-ins for others]. Limited use, thus not many people know about it.

 

SmartBear SoapUI

Advantages: Great tool for Web Services Testing from Smart Bear.

Disadvantages: Only useful for Web Services Testing. (but this might be an advantage, as they plan to make this a separate activity)

TestComplete

Advantages: Good tool, very similar to HP QTP, with a good interface and price. Overall good for Flash/Flex, with the included Libraries. SmartBear has a full stable of tools, which if bought together may be helpful in pricing and overall deployment and support. Uses VBScript/VBA for coding. People with QTP Experience may find it easy.

Disadvantages: Flash/Flex testing is still not very stable, sometimes fails to recognize the separate objects.

Microsoft Visual Studio Test Professional

Advantages: Is natively attached to the Visual Studio product line. Great for Cloud and .NET application testing. Good is you have Windows Phone applications. “CodedUI” is an excellent tool for testing cross-browser and web HTML testing. MS does deals to get the testing community to start using their tools 🙂

Disadvantages: Only for MS Technologies mostly. Not good for Firefox and Android. Only uses C# or Python.

Odin AXE Framework

Advantages: Great tool for building a wrapper over the existing tools scripts; actually it converts the tools identified objects into a XML recognizable format and has a great and easily understandable format for Automation testers.

Disadvantages: None that I can think of for now, except the use of a tool is somewhat a compulsory need for the framework created in AXE to work. Odin has done a good job of making the tool robust for Web Testing tools and it is compatible with almost all other commercial tools available.

Tricentis TOSCA

Advantages: Combines the best of Requirements, Test Case Design and Test Case execution, all in one single application. Good when there are business testers who know what the application is doing and there is good documentation available for doing it.

Disadvantages: Not very flexible when it comes to handling of unexpected behaviour within the application. Likes to have a clean interface to run through test cases and offer a “happy” path.

I can provide some more research into the new tools (and some less known but good ones), but the above are some of the common ones in use.

I am not advocating the use of any one tool above and to each depends on what he has worked with and would be comfortable in using.

Advertisements




What to Automate?

19 09 2011

I had this interesting conversation on Automation the other day with my colleagues at my new job. It started off fairly innocently on how the automation should be thought out and what needs to be done to automate. We already have an existing framework and test scripts in place, which very efficiently work and report issues. The problem comes when we need to provide data on what automation actually does. How do you prove the effort spent on automation? You could easily say that it saves time and resources by checking for faults early in the development cycle, but how can we be sure that it actually covers scenarios that check the application? This is where a business requirement document becomes a necessity.

The issue which we increasingly face today is how do we relate the business requirements to what we test. There are a few things which are given in textbooks and across such certifications organizations like ISTQB, which provide information on such. The thing is ultimately it depends on the person who is sitting and working on the application (which most of the Program/Product Managers miss out on) and the person who is writing the scripts to automate the application testing. The best way to figure out for a restricted zone (propriety) application is to go and sit with the users and find out what they use the most (or run a key stroke capture software and see where all it goes, limitations of this later). With a public/global reach software, it is best to give out Beta versions of the same, like most of the Big Organizations do and see what is reported back.

The other end of the spectrum is propriety software, which might not have a user interface (system tools like Compilers is one example). For this the technique is basically to read through the Software Requirement Document/Specification and have confidence in your abilities to decipher the jargon written in those and convert them to simple English. I got my initial training on writing code on these only, and the golden rule for us was [in a summary]:

  • Read through the document and write what you have understood for each function
  • Create the Algorithm for each in plain English pseudo code
  • Convert these into test cases, and run those test cases
I think the above rules of going through the steps of Why, What and How has helped us out a lot to become what we are today 🙂




Functional Testing, is it truly functional?!

2 06 2010

Most people who have been into the game of testing, drop words like functional, regression, end-to-end, system, integration testing like they are the ones who invented them. You ask the person on what exactly he wishes to attribute that kind of testing to, and you get a blank stare, which states – “You Stupid! Don’t you even know this much? And you wish to test my knowledge of it??!!” 🙂

Well! To me the concept of Function Testing is rather vague. It usually starts when the developers complete a component and hand it over to the test team. What needs to be checked and verified is the functionality of the component. How does one go about doing this? In the true sense, most of the time, functional is not about functionality, but more about the interface between multiple components in an application.

(more coming up…)





Regression Testing

28 05 2010

Regression with an automated test suite is a frequently used/abused phrase. People need to first understand what “regression” actually means.

Regression testing in a nutshell relates to the activity, by which we check a new application/product, for all the previous functionality; while keeping the newly added features unused. Basically, we validate that the new code has no effect on the older functionality. Regression is not used/should not be used to check for a functionality which has changed. As a change in functionality means that it is a new feature and needs to be verified first before it is validated.





Language Barrier!!

13 05 2007

There is one field where I have not as yet encountered a language barrier and that is testing. An interesting concept came out the other day while I was attending a conference – “Whatever comes out of the Developer’s desk, for testing the same, we require a simple understanding of the application and what is desired from it. Innovation is thing which is best left to Developers, but an analytical understanding is to be acquired for testing out anything. This is and should be the language of the Tester. And for this, the Developer can write in any human readable language, but the thinking mind will always be able to decipher it and work out the wrong it in… “.

Although, the above is true; we should not forget that the Developers also have the common logical thinking language similar to the analytical thinking language… 🙂