Selenium is a suite of tools used to automate web application testing across many platforms. It is an open source tool developed in Java Script and browser technologies and hence supports all the major browsers on all the platforms. It was developed by ThoughtWorks Inc.

There are 4 components of Selenium:

1) Selenium Core:

This is the original JavaScript-based testing system. It is now used primarily as a component of Selenium Remote Control, but it can also be used as a pure JavaScript/HTML testing system. To run Selenium Core tests, they normally have to be installed on the server on which you want to test.

2) Selenium IDE:

Selenium IDE is an integrated development environment for Selenium tests. It is implemented as a Firefox (2+) extension, and allows you to record, edit, and debug tests.

Features:

• Easy record and playback
• Intelligent field selection using IDs, names, or Xpath, as needed
• Autocomplete for all common Selenium commands
• Walk through tests
• Debug and set breakpoints
• Save tests as HTML, Ruby scripts, or any other format
• Support for Selenium user-extensions.js file
• Option to automatically assert the title of every page

Drawbacks:

• The biggest drawback of Selenium IDE is its limitation in terms of browser support. Though Selenium scripts can be used for most of the browsers and operating systems, scripts written using Selenium IDE can be used for only the Firefox browser if it is not used with Selenium RC or Selenium Core.
• Selenese, the simple scripting language used by IDE, is somewhat primitive as it has no conditionals (no “if” statements), and no loops (no “for” statements).
• Does not help with the launching and closing of the browser

Note: Selenium IDE can be configured to repeat a specific test by using flowControl extensions: goto, While, and so on, by installing the selenium-goto extension (download it from http://wiki.openqa.org/pages/viewpageattachments.action?pageId=379). To install it, open the Options panel in the Selenium IDE browser extension and specify the location of the user-extension.js file.

3) Selenium Remote Control:

Selenium Remote Control (RC) is a test tool that allows you to write automated web application UI tests in any programming language against any HTTP website using any mainstream JavaScript-enabled browser. The RC server also bundles Selenium Core, and automatically loads it into the browser.

Selenium RC comes in two parts:

a) A server, which automatically launches and kills browsers, and acts as a HTTP proxy for web requests from them.

b) Client libraries for your favorite computer language (Java, .NET, Perl, Python, C#, PHP and Ruby).

Click on selenium for a simplified architectural representation.

As a test suite starts, the following happens

• The client/driver reaches out to the Selenium-Server.
• The Selenium-Server launches a browser (or reuses an old one) with a URL that will load the Selenium core web page.
• The Selenium-Core gets the first instruction from the client/driver (via the HTTP Proxy built into the Selenium RC Server).
• The Selenium-Core acts on that first instruction, typically opening a page of the AUT.
• The web server is asked for that page, and it renders it in the frame/window reserved for it.

4) Selenium Grid:

Selenium Grid runs tests on many servers at the same time, cutting down on the time it takes to test multiple browsers or operating systems.

Installation:

1. Selenium IDE:
   • Selenium IDE can be downloaded from http://seleniumhq.org/download/.
   • Open Mozilla Firefox.
   • Drag and drop the file into it.
   • Install it and restart Firefox
   • If installed properly, Selenium can be accessed from Tool –> Selenium IDE in your browser toolbar.
2. Selenium RC:
   • Selenium RC can be downloaded from http://seleniumhq.org/download/.
   • Unzip the file and extract it in the directory of your choice.
   • JRE 1.5 or higher version should be installed on your machine. To check the version, type the following command at the dos prompt: java -version.

Bibliography:

http://seleniumhq.org/

http://www.testinggeek.com

By Diana Dsouza

This blog will focus on the best practices to be followed for Internationalization & Localization testing of web based applications.

Introduction

Internationalization testing is the process of testing software which is supposed to work uniformly across multiple regions and cultures. There are two of the major aspects of international software – world readiness (globalization) and localization. World readiness refers to the process of designing, coding and testing the product such that it can be easily localized for different regions. Localization involves translating and customizing the product for different regions.

Ensuring world readiness is different from testing localized versions of the product, and it is broader than just testing the functionality. It also includes realizing the implications of globalization, and verifying that those implicit requirements are met throughout all the design and development steps.

Internationalization testing

The goal of Internationalization testing is to detect potential problems in software globalization. It makes sure that the code can handle all international support without breaking functionality that would cause either data loss or display problems. Globalization testing checks proper functionality of the product with any of the locale settings using every type of international input possible. There are two basic testing requirements which ensure that the product works well in all regions for which it was developed:

• Testing to check if the product is locale aware (globalization).
• Testing the localized versions of the product.

The following diagram depicts the breakdown of the product with respect to internationalization and localization testing:

Best Practices in Internationalization Testing

• Feature based Testing

The product may be marketed with some features applicable and others not applicable to specific cultures. While testing, it is important to ensure that these features are turned on or off as required while switching locales.

• Testing for Localized content

Localized content includes both text and graphic artifacts on the user interface. Textual content to be localized includes the static text on controls such as menus, buttons, etc. and user specific messages. Language experts are required for testing the localized content in each language that the product supports.

• Testing for culture awareness

User locale settings define items such as number, date and currency formatting. Input locales should be handled and the user be allowed to input data in that language.

In many cases, data would need to be converted from one encoding format to another. Correct handling and understanding of encoding formats is required in such cases. If coded incorrectly, in addition to operational errors, it can also result in loss of data. For example, converting from a multi-byte character set to a single or double-byte character set can result in data loss if not handled properly. User locales do not impact conversion of data from one encoding format to another.

• Testing rendering

Internationalization testing should check whether an internationalized product properly displays all supported scripts in accordance with the linguistic characteristics associated with them. The characteristics include bi-directionality, character reordering, contextual shaping, combining characters and special rules in terms of word breaking, line breaking and text justification.

• Testing user interface

Internationalization testing should cover whether the user interface, on which the fonts are rendered, supports the different fonts and character widths as well as the varying spacing requirements introduced by translation. For example, it is a well known fact that German text occupies more space to convey the same information when compared to English. Thus, the user interface should be adaptable to accommodate user messages in both languages.

Following are the recommended guidelines for user interface testing:

1. Check if textbox labels appear on top of the textboxes in order to handle text with larger widths. If it is still required to have the label beside a textbox, enough room should be left for the text to grow.
2. Check if dialog boxes expand while localizing. Additional space (ideally, 30%) should be left between the end of messages and the edge of dialog boxes for further expansion.
3. Check the text embedded in images and icons. Having text in images will make localization difficult and require the efforts of a user interface designer/tester.
4. Buttons and labels should have a larger width to accommodate for longer text in non-English languages.
5. Images embedded within resource files of forms should be moved out as separate image files.
6. Mirroring: Mirroring is the process of localizing the user interface to handle right to left (RTL) languages such as Arabic, Hebrew, Farsi, etc. To give a perfect RTL look and feel to an application’s UI, both the text and the UI elements need to be laid out from right to left once they are translated into RTL languages. A complete discussion on Mirroring is not within the scope of this paper.
7. Graphical images: Graphics can be difficult and expensive to translate. Hence it is preferable to have graphics which are universally acceptable.
8. Following are some of the key user interface elements that need to be localized:
   1. Menus
   2. Messages
   3. Dialog boxes
   4. Images
   5. Toolbars
   6. Status bar

• Storage testing

Internationalization testing should also cover data storage and retrieval in multilingual applications, which warrants careful attention. The key sources of storage include files and databases.

• Testing File Transactions

While reading from a file which contains Unicode characters, it is important to specify the encoding. If the encoding is not explicitly specified, a default encoding may be used (UTF-8 in case of .NET framework) resulting in misinterpretation of data. For example, if a text file stored in UTF-16 is read without specifying the encoding, .NET will interpret the UTF-16 characters as UTF-8 resulting in unintelligible input. Please note that ASCII and UTF-8 are interoperable if the file contains only English characters having code point less than 128

• Database Testing

Testing should cover the support of Unicode characters in databases; special data types are used which are defined for this purpose. For example, SQL Server defines nchar, nvarchar and ntext data types to allow you to store Unicode text. (The n prefix for these data types comes from the SQL-92 standard for National (Unicode) data types). Use of nchar, nvarchar and ntext in SQL Server 2000 is the same as char, varchar, and text, respectively, except that:

• Unicode supports a wider range of characters.
• More disk space is needed to store Unicode characters.
• The maximum size of nchar and nvarchar columns is 4000 characters and not 8000 characters as for char and varchar.

Unicode constants are specified with a leading N (for example, N<a Unicode string>). The following example shows how to insert Unicode values (Russian) into an nvarchar column.

Once you have defined a Unicode compatible column in the database, inserting and retrieving multilingual data is the same as working with regular (ASCII) data. In database migration, care should be taken while mapping the source and target data columns. If the source data type is nchar, and the target data type is char, data could be lost upon migration.

In summary, there are several aspects to be kept in mind while designing software for a global audience. These aspects cover not only coding and testing, but also the interface and layout of the software as well as factors such as the file formats and data encoding to be used.

Do developers also have to test? How do developers test since they are supposed to be good at coding and not at testing? Well, then the solution is for them to code the test as well!

This blog covers the basics of Junit – a framework designed to write repeatable tests. It is based on the xUnit architecture, which is basically a collection of code-driven testing techniques, allowing testing of various software units such as functions and classes and eliminating the need to write the same tests multiple times.

JUnit is simple to use and write TestCases (written as one word and different from a ‘test case’) that could cover almost all testing scenarios if planned well and covered by the tests. Moreover, it plays a vital role in code maintenance and reduces regression testing usually added by new features or code fixes.

So how do we start about JUniting? Well, the simplest rule of thumb to follow is to plan the input and output for any given scenario, be it a positive scenario or a negative one, and then write a test involving each of these scenarios. This ensures that each condition is checked well before the code release. Moreover, whenever the test suite is run for regression checks, it identifies immediate issues, introduced either due to additional fixes or addition of new features. This may mean that either the new code broke some valid scenario, or the broken scenario is no longer valid and you may turn it into a positive scenario by tweaking that test.

JUnit is based on two key Java design patterns: Command pattern and Composite pattern. A TestCase constitutes the Command object whereas a TestSuite comprises Composite TestCases.

Writing JUnit code is very simple and we would not get into the basics of it in this blog. If you are using Eclipse or any other IDE, it supports creation of JUnit tests for a given class in just a few seconds. However, we shall look at ‘from the scratch’ approach wherein we want to add a customized test suite.

The prerequisite for this would be downloading JUnit 4.x and adding the junit-xxx.jar in your classpath.

Implementing a TestClass

A quick annotation preview for TestClass

A TestClass is implemented using the ‘@Test’ annotation. There is no need to extend the TestClass now with JUnit 4.x and the method names need not start with ‘test’ as required by earlier versions of JUnit.

The ‘@BeforeClass’ annotated method is executed as soon as the TestClass is initialized and the ‘@AfterClass’ annotated method is executed before the TestClass is being finalized. They run only once in the lifetime of the tests.

The ‘@Before’ annotated method is executed before each test is run and the ‘@After’ annotated method is executed after each test has completed execution.

The ‘@Ignore’ annotation is used to ignore the test for execution.

The ‘@Timeout’ annotation takes the time in milliseconds as a parameter and fails if the test does not complete execution within the stipulated time.

A simple TestCase example is given next.

import static org.junit.Assert.assertEquals;
import static org.junit.Assert.fail;

import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;

import test.com.xoriant.testsuite.TestFinalizer;
import test.com.xoriant.testsuite.TestInitializer;

import com.xoriant.SingletonClass;

public class SingletonClassTest
{
    /**
     * @throws java.lang.Exception
     */
    @BeforeClass
    public static void setup() throws Exception
    {
        if (System.getProperty(TestInitializer.TEST_INITIALIZED) == null)
        {
            TestInitializer.setUp();
        }
    }

    @Test
    public final void testGetInstance()
    {
	try
      {
        SingletonClass singletonClass1 = SingletonClass.getInstance();
        SingletonClass singletonClass2 = SingletonClass.getInstance();

        assertEquals(singletonClass1, singletonClass2);
      }
	catch(Exception e)
{
	  fail("Execution failed due to: " + e.toString());
}
    }

    @AfterClass
    public static void tearDown()
    {
        if (System.getProperty(TestFinalizer.TEST_FINALIZED) == null)
        {
            TestFinalizer.tearDown();
        }
    }
}

Implementation of the TestInitializer class

This is a common class utilized by the test cases for loading test data for the tests prior to any execution of the tests. This class may also be invoked for independent test cases.

import org.junit.BeforeClass;
import org.junit.Test;

import static org.junit.Assert.assertTrue;

public class TestInitializer
{
    public static final String TEST_INITIALIZED = "com.api.testsuite.initialized";

    static
    {
        logger.debug("Setting system property...");
        System.setProperty(PropertyManager.BUNDLE_PROPERTY_NAME, "test_config");
        System.setProperty(TEST_INITIALIZED, "Y");
    }

    /**
     * Sets up the test
     */
    @BeforeClass
    public static void setUp() throws Exception
    {
	  // Initialize the data required for the tests
    }

    @Test
    public void dummyTest()
    {
        assertTrue(true);
    }
}

Implementation of the TestFinalizer class

This class is a common class for removing test data after all the tests in the TestSuite have completed. This class may also be invoked for independent test cases.

import org.junit.AfterClass;
import org.junit.Test;
import static org.junit.Assert.assertTrue;

public class TestFinalizer
{
    public static final String TEST_FINALIZED = "com.api.testsuite.finalized";

    static
    {
        System.setProperty(PropertyManager.BUNDLE_PROPERTY_NAME, "test_config");
        System.setProperty(TEST_FINALIZED, "Y");
    }

    @AfterClass
    public static void tearDown()
    {
          // Remove the test data that was required for the tests
    }

    @Test
    public void dummyTest()
    {
        assertTrue(true);
    }
}

Implementing a TestSuite

A TestSuite is implemented using the ‘@SuiteClasses’ and ‘@RunWith’ annotations. It takes a list of TestClasses as a class array parameter and executes each of them in that order.

Here, we initialize the TestSuite by adding required test data into the database in the TestInitializer class. Similarly, we cleanup the data using the TestFinalizer class. The ‘@RunWith‘ annotation allows us to define a customized TestSuite with listeners that get triggered at appropriate events in the test execution cycle.

import org.junit.runner.RunWith;
import org.junit.runners.Suite;
import org.junit.runners.Suite.SuiteClasses;

import test.com.xoriant.SingletonClassTest;

@RunWith(CustomSuite.class)
@SuiteClasses( { TestInitializer.class, SingletonClassTest.class, TestFinalizer.class })

public class AllTests
{
}

Implementing a custom TestSuite

import org.junit.runner.RunWith;
import org.junit.runners.Suite;
import org.junit.runners.Suite.SuiteClasses;

import test.com.xoriant.SingletonClassTest;

@RunWith(CustomSuite.class)
@SuiteClasses( { TestInitializer.class, SingletonClassTest.class, TestFinalizer.class })

public class AllTests
{
}

Some of the best practices for JUnit tests that should be followed

• Do not run tests with interdependent test data. This could create maintenance problems later on.

• The source folders of the actual classes and the test classes should be different so that the tests are separate from the original source files and do not need to be shipped along with production code.
• Keep the test class packages same as the class in question to be tested. This allows protected methods to be tested.
• JUnit can be integrated with nightly builds and executed as an ant task. This would enable running the tests on a daily basis and avoiding production problems.
• Utilize proper initialization and destruction methods. Never initialize test data in constructors.
• Keep tests up-to-date to avoid revisiting the tests to be fixed.

Summarizing, in this blog, we saw how to leverage the JUnit testing framework and its derivative API TestCase to increase the testing efficiency for Java based projects.

It is to be noted that apart from Java, JUnit has also been ported for several other popular programming languages such as C#, C++, Perl, PHP and many others. The ultimate objective of using JUnit is to test before you code so that you can rest after you code!

Overview

Today, non brick-and-mortar companies such as Google, Amazon and Yahoo! generating revenues in billions of dollars is quite well-known. What may not be so well-known, though, is the level of investment these companies have to make in ensuring that their systems and servers are always available to their customers 24×7, and to make sure that their data centres are able to handle the amount of traffic needed to be processed to keep their business going.

Little wonder then, that testing forms an important part of managing their business infrastructure.

Introduction

This blog is intended to explain to the reader the most important factors that are involved in the testing of high-traffic Web portals. For such Websites, security and performance become the key factors in testing along with other concepts such as usability, functionality and navigation.

Unit Testing

Unit testing must be performed by the developers against their components prior to integration. Custom unit tests only exercise functionality but neglect performance, which becomes a bottleneck later; hence component performance needs to be analyzed during their unit tests. This is usually done using the following tools:

• Memory profilers
• Code coverage and coding standard tools

Usability Testing

Usability testing helps to create and maintain a user-friendly and user-centric portal that a visitor will find easy to use and will want to visit again. Some of the usability practices tailored for high traffic Websites are listed below:

• User friendly layout: Use language and concepts which are familiar to the user. Try to present information in a sequential and logical order.
• Maintain consistent conventions: Concepts which are similar should be shown using similar terminologies and graphics. Details of maintaining uniform conventions for aspects such as layout, formatting, typefaces, labelling, etc should not be ignored.
• Minimize users’ memory usage: Try not to force visitors to remember important information across multiple documents.
• Using a flexible and efficient design: Try to arrange screens in a way such that frequently required data is found easily, providing instructions and directions wherever necessary.
• Keeping an aesthetic yet simple design: Do not present or display information which may be distracting or irrelevant.
• Arrange a progressive level for details: Information should be organized such that generic information is presented first, followed by more details, as requested or searched for.
• Provide navigational feedback. Moving between related topics should be made easy. The user should be allowed to find out where he/she presently is on the website (by providing a site-map) and it should be easy to return to any previously visited point on the site.

Globalization

For high traffic portals, there is increasing demand for supporting local languages as the end user base is increasing from every corner of the world. Globalization testing is the process of testing the Website which has to work uniformly across multiple regions and cultures. There are two aspects to an international website: world readiness (globalization), and localization. World readiness refers to the process of designing, coding and testing the website such that it can be easily localized for different regions. Localization involves translating and customizing the products for different regions. Ensuring world-readiness is different from testing localized program versions, and it is broader than just functionality testing. It also includes realizing the implications of globalization, plus verifying that those implicit requirements are met throughout all the design and development steps.

AJAX Bridging

It is well known that AJAX applications can only connect back to the website from which they have originated and this is a security measure in AJAX. For instance, JavaScript code developed with AJAX and obtained from a certain site cannot launch a connection to another site. The AJAX service bridge was developed to enable a third-party Website connection in this way.

This is how it works: Inside the bridge, first, a host provides a Web service acting as a proxy and allowing traffic forwarding between the JavaScript code running on the client and the third-party Web site. Since a bridge can be considered as a ‘Web service to Web service’ type of connection, an attacker can use this to gain access to sites with restricted access. Hence, it is required to check whether the website is vulnerable to attacks or not.

HTML/AJAX Injection

The heart of AJAX is XMLHttpRequest which allows for synchronous server communications and browser updates. The browser can be updated with just simple HTML (DOM), XML, or another structured data format. These XMLHttpRequest calls are just normal HTTP requests. Check if every request for authentication checks for the behavior of application after introduction of HTML injection. The same security vulnerabilities and controls apply to AJAX websites.

• AJAX often requires additional or stronger controls because they are usually complex, bidirectional, and asynchronous.
• AJAX applications often have weak authentication, session management, and error handling.

Cross Site Scripting

Additionally, check the Web application for XSS (Cross site scripting). Any HTML, such as <HTML>, or any script such as <SCRIPT> should not be accepted by the application. If it is, the application can be prone to an attack by Cross Site Scripting.

Cross-Browser Compatibility

Cross-browser compatibility generally is a big issue. It is magnified a few times when you consider high traffic Websites with AJAX. Browsers either do not support AJAX or render AJAX differently. More stress on browser compatibility testing of high traffic Websites comes when you consider different browsers with different versions such as Internet Explorer, Netscape, Firefox, Safari, Opera and Chrome.

Security Testing

Because AJAX is still a recent technology, several of its security issues are still to be fully understood. AJAX must not be deployed in security-sensitive applications, such as credit card verification portals. Some of the security concerns to be considered while designing the test cases while using AJAX are listed below:

• It increases the available attack surface and requires many more inputs to be secured.
• It exposes the internals of the application.
• It enables access to third-party resources for the client without any built-in security and encoding mechanisms.
• It has no mechanisms to avoid failures in protecting authentication information and sessions.
• The line between client and server-side code is highly blurred, which results in security mistakes.

Testing Search Engine Optimization

Testing teams of such high traffic portals need to deeply understand the basics of search engine optimization (SEO). This is all the more important, especially because Website technologies like AJAX and Flash are not search engine friendly. Listed below are some important SEO items that the testing teams should be aware of:

• Ensure that static URLs are used widely and IP addresses are used only when absolutely necessary.
• Ensure that there is no excessive use of tables which is considered a bad SEO practice.
• Ensure that non-compliant HTML is not used.
• Ensure that images have ‘ALT’ and ‘Title’ attributes.
• Ensure proper use of ‘Meta’ tags and keywords.
• Ensure the presence of an ‘Admin’ screen to dynamically update keywords based on feedback from search engines.
• Check hyperlink updates for use of domain name instead of IP address.
• Avoid excessive use of JavaScript based menus which have a negative impact on SEO.
• Ensure that all searchable text is available to search engines.
• Test the Website on all search engines for which the site is optimized.

Special Test Cases for Websites Heavily Loaded with AJAX and Flash

• Absence of an IDE for AJAX makes unit testing and debugging a discouraging task. Hence, excessive dependency is always there on the regression testing teams who need to factor this and design the test suite more elaborately and cover each single unit for all its possible functionalities.
• High traffic Websites with AJAX will not work if JavaScript is disabled. Make sure that the user is prompted to enable JavaScript and even if the user doesn’t do so, the site should not become totally inaccessible with JavaScript turned off.
• The ‘Back’ button on the browser may not work until developers provide that functionality. Testing teams need to ensure that this functionality is available on pages where there is a high probability of users hitting the ‘Back’ button.
• It may be difficult to bookmark a particular section of an AJAX site. Testing teams must ensure that pages users may want to bookmark are free from this limitation.
• The Flash based interface must be tested on different speeds such as modem, broadband, LAN etc.
• The audio and video synchronization for flash movies and animations must be checked.
• Finally, the Flash/AJAX application on the machine with minimum configuration must be tested.

Bandwidth Dependency

Both AJAX and Flash have a heavy bandwidth dependency and do not work well on low bandwidths. Hence, testing the response on varying bandwidths to determine and ensure the comfort band is an important requirement. This adds a whole new dimension to testing such sites, and demands additional discipline, time and resources.

Performance Testing

For such high traffic Web portals, thousands of requests are usually being made to the server and during particular events and times, this count can go in millions. With such a high volume of traffic, server performance may become an issue and hence to ensure satisfactory performance, the following criteria should be met during performance testing:

• It must be ensured that your site’s performance requirements and goals represent the needs of your users.
• Load tests must be designed such that they replicate the actual workload at both normal and peak times.
• Performance testing must be conducted using parameters such as data types, distributions and volumes which are similar to those used in actual business operations under actual production.

• Whether the system can handle multiple concurrent requests performing the same transactions must be checked.
• To properly identify the bottlenecks during performance testing, using a good and optimised profiling tool is also important.
• Automated testing tools such as LoadRunner, Keynote and JMeter should also be used for evaluating various performance aspects.
• Time-critical transactions in the performance tests must be included.
• Even under peak traffic conditions, it must be ensured that the user can transfer large volumes of data.
• The performance tests must also be conducted while regular system processes, such as a batch job execution or a virus definition update download, are in progress.
• The performance must be measured under various load levels and mixed scenarios.
• It has to be validated that all of the correct data was displayed and saved during your performance tests.
• An important testing technique, known as Soak Testing, also has to be executed. This involves testing a system with a significant load extended over a significant period of time, to discover how the system behaves under sustained use.

Summary of the Factors Important in Testing High-traffic Web Sites

Standard testing methodologies must be adapted to the unique nature of Website testing environments. The following table lists some of the key aspects that testing teams should consider while designing test strategies for high traffic Web sites:

Type of Check	Parameters to be Validated
Validation checks	Validating HTML, input forms, check for broken links, validating CSS (such as fonts, colors, spacing etc)
Security checks	Testing cookies
Performance checks	Testing bandwidth dependencies
Browser compatibility checks	Using different browsers such as Internet Explorer, Netscape, Firefox, Safari, Opera and Chrome
Other checks	Testing the sever side interface, configuration, recovery and accessibility

Phases in Testing Applications Running on such High-traffic Web Portals

To ensure that the application is able to withstand the high volume of traffic, it must go through the following phases during specific points in the software development lifecycle:

• Unit testing
• Integration testing
• Functional testing
• End-to-end testing
• Distributed testing
• Usability Testing
• Globalisation testing
• Exploratory testing
• Regression testing
• Compatibility testing
• Conformance testing
• Fault tolerance and security testing
• Load and stress testing
• Performance testing

Summarizing, we have seen in this blog that testing high-volume portals requires careful planning at every stage, starting from the site’s design right up to considerations such as the end user’s bandwidth availability.

Introduction

In today’s competitive and ever changing world, where most software applications are complex in nature having distributed, multi-tiered, Web-based architectures, one of the key success factors is the application’s actual performance.

Imagine a scenario where your application is great in design with a fantastic look and feel and the most user friendly GUI, but is really poor when it comes to performance, with users needing to wait for several minutes to complete a single transaction. Add to that a very high system down time due to failures, frequent restarts and incorrect operations under heavy traffic conditions. All these factors would understandably lead to losses in revenue, market share and the company’s reputation.

Software performance testing thus plays an essential and significant role in the entire software testing cycle; the quicker the problems related to performance are found in the development cycle, more easily and cost effectively they can be fixed by providing the necessary hardware and/or software solutions.

This blog focuses on the performance engineering practices that we follow here in Xoriant to deliver high quality software on-time that meets the required performance objectives within the allocated budget.

Performance testing types

Following are some of the phases of software testing that an application usually goes though during the testing cycle:

• Load Testing: Determine if the application can meet a desired service level under real world volumes.
• Spike Testing: Simulate a sudden increase in the number of concurrent users performing a specific transaction to determine the application’s behavior under abnormal traffic conditions.

• Endurance Testing/Soak Testing: Determine if the application can sustain a continuous and expected load for an extended period of time. Usually during endurance tests, memory utilization is also monitored to detect potential memory leaks, since small amounts of memory leaks on a system can turn out to be a severe problem during peak traffic conditions leading to performance degradation and resulting in the system running out of memory very often.

• Stress Testing: Determine the application’s breaking point under maximum load conditions that the application is designed to handle (typically specified in terms of number of concurrent users/transactions, memory and disk space becoming full etc).

• Scalability Testing: Determine whether the application under test is able to gracefully handle increases in work load by upgrading the hardware (for example, by adding extra memory and disks to the system).

The performance testing process

The following diagram shows the processes involved in the entire performance testing cycle along with the flow between them.

Processes involved in the performance testing cycle

Requirements analysis

During the requirements stage, the system performance criteria are captured and documented to get them approved by the customer. This process helps to identify testing objectives, understand system components and its configuration, and comprehend system usage in the production environment.

Performance test planning

Based on the requirements plan, the key activities of performance testing are planned, which include:

• Test tools’ identification and finalization.

• Preparing various checklists to be used during actual execution.

• Indentifying test resources includes hardware, software and human resources.

• Indentifying the baseline for the test setup.

• Identifying the risk involved in performance testing.

• Estimating and planning the performance test cycle.

Testing tools’ identification and finalization

Based on the requirements, identify the tools which can support the number of simultaneous virtual users as fixed in the test plan, the cost of testing, the number of features to be supported in the particular build and the platform it will support. Because most of the times these requirements are fulfilled by good open source performance testing tools, investing in proprietary and licensed tools must be properly thought over.

Some of the popular performance testing tools in use these days are Load Runner by HP, SilkPerformer by Borland, Webload (open source), JMeter (open source) and Microsoft Web Application Stress by Microsoft

Preparing the various checklists

Having a checklist for performance testing is extremely important. This helps in creating a better definition of the required performance and tracking the requirements. In the absence of properly defined performance testing requirements, teams may have to spend a great deal of time either in getting the information during actual testing which could delay the actual testing, or in rework required in the project because of insufficient information.

Checklist to ensure availability of the correct performance testing environment

The following is the list of points to be verified to ensure that the performance testing environment is correctly set up:

• Does the test environment exist?

• Is the environment self-contained?

• Are all required licenses in place?

• Are systems for the load testing identified?

• Is the environment modeled close to the production environment?

• Is a copy of production data available for testing?

• Are replacement servers available?

• Are the tools and utilities identified compatible with the environment?

• Is load balancing available?

• Are system deployment diagrams in place?

• Are testers well versed with the environment?

Checklist to ensure proper performance test planning

• Are people with the required skill sets available?
• Have version control processes been used to ensure the correct versions of applications and data in the test environment?
• When will the application be ready for performance testing?
• How much time is available for performance testing?
• How many iterations of testing will take place?
• Have external and internal support staffs been identified?
• Have back-up procedures been written?
• Have any contingency situations been considered and provisioned for?

Identifying the test resources

Identify suitable hardware and software which should replicate the production environment because using the same hardware and software as in the actual production environment is neither cost effective nor feasible. Also, identifying the human resources in the beginning will help you plan the required trainings for the application’s understanding and its usage as well as using the testing tools and learning any third party software that is required in the testing phase. This in turn helps the testers feel confident while executing the performance tests.

Identifying baselines for the test setup

Identify the baselines for the test setup for all types of tests that you are going to perform. The setup should be a replica of the real environment. This has two advantages: First, the results from various categories of tests do not reflect the type of hardware you are using. Second, you can depend on the test results because the test setup is close to the production environment.

Identifying the risks

Performance testing is a risk-driven and time consuming activity. If the level of risk is small, the effort involved in performance testing activities can be correspondingly small and if the risk is high, then a significant amount of effort is needed. Hence, identify these risks in the beginning and document them in the plan to build a common undersigning for all stakeholders, and define the clear expectations for each throughout the performance testing cycle.

Estimate and plan the performance test cycle

Because performance testing is a risk-driven and time consuming activity, precise estimation is very much important which will help define the early performance testing cycle so that any bottlenecks found can be fixed and retested during the test cycle itself.

Test design

This phase includes designing and developing the test cases for each performance test type. Prioritize these test cases and test scenarios according to the critical functionalities, the risk involved in the use cases and the high traffic/volume conditions that need to be tested first. The deliverables for this phase are the test case design document which is the input criterion for the test scrip and test suite creation activities.

Test script and test suite creation

This phase includes developing the test scripts and designing a test suite. This includes actual performance test execution scripts as well as system monitoring scripts.

Test execution

The test execution phase includes running the actual tests and collecting statistics for analysis. The following items needs to be considered while actually executing the performance tests:

• Always execute the performance tests in a controlled environment. Testing without dedicated servers and good configuration management will yield test results that are not reproducible. If you cannot reproduce the results then you cannot measure the improvements accurately when the next version of the system is ready for testing.

• Clear the application and database logs after each performance test run since excessively large log files occupy the disk increasing the chances of the system running out of space during the test. It can also artificially skew the performance results.

• If the application uses a database, then both the application and the database should be installed on two different systems.

• Validate the test setup by sending a few requests before you actually start the complete test.

• Prioritize the test cases and test scenarios that need to be executed first so that important bottleneck issues are identified early.

• Always generate the load by using different client computers.

• Make sure that the client and server systems are located on the same network; otherwise it may introduce latency in the request/response times.

• Always refer a single client to capture client-side data such as response time or requests per second. You should consolidate data at a single client and generate results based on the average values while a load is generated on the system.

• Include a buffer time between the incremental increases of users during a load test.

• Use a zero think time if you need to fire concurrent requests. This can help you identify bottleneck issues.

• Monitor all computers involved in the test, including the client that generates the load. This is important because you should not overly stress the client.

• Do not allow the test system resources to cross resource threshold limits by a significant margin during load testing, because this may affect the test data and test results.

• Do not run tests in live production environments that have other network traffic. Use an isolated test environment that is representative of the actual production environment.

• Do not place too much stress on the client test computers. The processor and memory usage on your client computers should be well below the threshold limit (CPU: 75 percent). Otherwise, chances to die/hang for the client are very high and destroying the test result and necessitating re-execution of the tests.

• Do not try to break the system during a load test. The intent of the load test is not to break the system, but to observe performance under expected usage conditions. You can stress test the system to determine the most likely modes of failure so they can be addressed out.

During the test, the system should be closely monitored to capture the following metrics and statistics at regular intervals of time:

• CPU utilization
• Memory usage identifying memory and thread leakages
• Disk space
• All queues and IO waits
• Network usage on individual clients, application and database servers (bytes, packets, segments, frames received and sent per sec, bytes total/sec, current bandwidth, connection failures, connections active, failures at network interface level and protocol level)
• Web server (requests and responses per second, services succeeded and failed, server problems if any)
• Request and response times
• Throughput
• Time (session time, reboot time, transaction time, task execution time)
• Hits per second, requests per second, transactions per second
• Database problems (settings and configuration, usage, read/sec, write/sec, any locking, queries, compilation errors)

Test analysis

This is the most important phase. At the end of the test execution, the test results and its data are analyzed for system performance and bottlenecks are identified.

A preliminary snapshot of these analyses is sent to the development group to fix the issues or performance tuning recommendations. Once fixes or tuning is applied on the test system, the tests are re-executed and hence, test execution and test analysis processes are iterative processes and need to be executed until the desirable performance is achieved.

Identifying bottlenecks and tuning the system

If any bottlenecks are found in the system, then the performance of the system can be enhanced by improving the architecture, design, code and configuration. And tuning of the code can be done using developer support tools such as profilers and code coverage analysis tools.

The following considerations should be taken into account while applying any performance optimization:

• Always practice change control. Make backups of all configuration files you alter, so you can revert to an older version if required.

• Take a baseline of the performance before and after changing any settings.

• Document the changes. This helps others understand why the changes were made.

• Don’t apply any changes to the test system unless you have a reason to. If everything is working well enough and performance data are acceptable by the customer, then why to make any changes? In such a case, you should contact the customer with the suggestions and apply the changes only after they have been approved by the customer.

Test results and reporting

At the end of the performance tests, test results are correlated and summarized in an executive summary report which includes the following contents:

• Objectives of the test

• Reference documents

• Test setup (hardware configurations, software configurations, tool settings, list of transactions, scenarios tested, client machine details and its configurations, transaction generation pattern etc.)

• Test results (list of significant runs, scalability statistics, request and response time statistics, server side statistics etc.)

• Findings, recommendations and test log

Knowledge repository

Maintaining a proper knowledge repository is the most important part of the performance testing cycle since it is a link between the current and future activities as it contains the record of all the activities performed during performance testing. The objective of this is to capture the knowledge gained during the project which can be used in future releases.

So summarizing, we saw in this blog that there are several steps and processes that need to be followed to ensure that a software development project progresses in the intended manner and within the available time and resource limits. Thankfully, many of these processes have now been made easy by good quality and easily available software tools.

Overview of a Plug-in

A plug-in is a software component that resides in a software application (known as the parent application) and allows users to perform specific, customized activities within the application. Most software applications support plug-ins for many reasons. Some of the main reasons include:

• to enable third-party developers to create capabilities which extend an application
• to support easily adding new features
• to reduce the size of an application
• allow for a large number of people to contribute to and share stored data
• control access to data, based on user roles. User roles define what information each user can view and edit
• aid in easy storage and retrieval of data

This blog is intended to highlight some usability guidelines which can be followed to overcome some of the common challenges associated with plug-ins and also make the plug-in usage a more user friendly experience.

Installation Testing

Installation is the first step towards using any desktop plug-in. A plug-in should generally allow the users to configure it during the initial installation. To achieve this, the installer should be self explanatory and guide the users through the various screens to configure the settings and complete the process.

The areas to be focused on during installation testing are:

• User interface: Should be self-explanatory and kept short
• Installation steps: Should be self-explanatory and kept short
• Pre-requisites: Many desktop plug-ins need a set of supporting software to be pre-installed in order to integrate with the application and run the plug-in successfully. If such software is not installed on the target machine, then the installer should ideally download and install the necessary pre-requisites automatically, or prompt the user to install them, or navigate the user to the website from where he can download and install the pre-requisites manually. For example, desktop plug-ins designed in VSTO need .NET Framework 2.0 and Visual Studio for Office Runtime Second Edition to function properly. This aspect should be checked by the plug-in and it should perform the steps mentioned above.
• Graceful rollback: The user can cancel the installation at any step during the installation process. Cancelling the installation should clean up all residual entries from the installation folder, registry etc. in a graceful manner. The user should be informed about the un-installation sequence.
• Versioning: The user should be able to install a newer version of the plug-in without manually uninstalling the older version, meaning, the installer should automatically take care of deleting and removing the older files and entries before coping the newer ones.
• Availability for Admin users and standard users: After Microsoft created the Big Divide between the Standard and Admin users from Windows Vista onwards, the installers need to be intelligent to detect the current user type and install the plug-in for the relevant user.
• Respect the UAC (User Access Control): The installer should be compatible with the O/S’s UAC settings and should be intelligent to adjust the installation process accordingly.

Creating exhaustive log files: This can save a lot of debug time and become your life saver. The installation log files should be generated at a location where both the admin and standard users have access.

Compatibility Testing

The desktop plug-in user base is generally very wide with different flavours of OSes and application versions, and so compatibility testing is one of the critical and most important tasks while testing desktop plug-ins. It may happen that a button-click on a plug-in in one environment can generate an error or no response in another environment.

Planning this activity is critical to the success of the plug-in. The first task is to identify the target environments depending upon the user base and the various O/S combinations.

Setting up a QA lab of virtual machines using tools like VMware greatly eases the task. Using this, you can set up different environments with various combinations of OSes and application versions. Although this cannot be termed as an exhaustive testbed but atleast it gives an assurance to the customers that the plug-in will work on the majority (if not all) of the target machines when released worldwide.

The primary areas that need to be focused on can be summarized as:

• Identifying the environments.
• Operating Systems: Windows XP (32 bit and 64 bit) with SP1, SP2 and SP3 packs, Windows Vista (32 bit and 64 bit) and Windows 7 (32 bit and 64 bit).
• Application versions: The application for which the plug-in is installed. For example, if the plug-in is designed for Microsoft Office products, then it needs to be tested with Office 2003, Office 2007 and Office 2010.
• Different screen resolutions: It is very important that the user experience remains the same irrespective of the OS or application versions or varied screen resolutions.
• Different font sizes: The font size should not change or affect the functionality of the plug-in.

Functional and UI Testing

Functional testing of any software or application includes testing carried out to meet the business requirements as per the application’s specification document. In this case, the functional testing is very specific to the plug-in and needs to be planned according to its requirement documents. The features provided in the plug-in should leverage the features of the parent application and the user Interface should resemble the parent application.

Performance Testing

In performance testing, some technical parameters of the plug-in are tested against defined benchmarks. A few of these parameters are listed below:

• When a plug-in is opened in multiple instances of the same application.
• Keeping the application running for long periods of time with the plug-in loaded.
• The plug-in’s initial load time and functionality validated on various OSes for latency.
• Evaluating the performance of a similar plug-in from another vendor installed on the user’s environment.

The above factors coupled with a properly planned functional and performance testing process will ensure smooth operation of the plug-in on all the environments and make the plug-in usage much more user-friendly.

Introduction

Within the software industry, once a product reaches the stable manual testing phase, every organization usually thinks of automated testing to save on the costs involved in manual testing. Since testing cost is an important factor for any project, organizations have started preferring open source test automation tools (which have reached a stage where they now rival the commercial ones) instead of investing in costly commercial testing tools. With no licensing costs, open source automation testing tools provide competitive features for automating the testing of software applications as well as Web portals.

A variety of open source automation testing tools is available for almost all types of testing such as functional, Web, UAT, regression, performance etc. Because of the extent to which these open source tools have matured, it’s time to think about them and have them in your QA automation kit. There are also various open source tools available to support the different testing types such as White Box Testing (Unit Testing, for e.g., using JUnit) and Black Box Testing (system/regression testing, for e.g., using Selenium, Sahi, Watir, TestMaker, LogiTest, TestGen4J, FitNesse etc).

The scope of this blog is to cover the basic flow for one of these open source Web testing automation tools – Selenium.

Why Selenium?

Selenium is probably the best option for automated testing of Websites today. It is becoming increasingly popular and it is the first choice of automation testers as well as organizations for automating the testing of Web-based applications for both the GUI as well as the functionality. Selenium can also be used as a unit testing tool for JavaScript. We at Xoriant have been working on Selenium for automating the testing of Web sites developed using AJAX for the pharmaceuticals and travel industries. The following graph shows the popularity of Selenium along with other open source automation testing tools.

Selenium Test Package

Selenium is a package of various test components which consists of the following three major tools. Each one has a specific role in aiding the development of test automation for a Web application.

1. Selenium IDE – A Firefox extension to record test cases and suites.
2. Selenium RC – Used to run tests on different browsers and systems.
3. Selenium Grid – Runs multiple instances of Selenium RC at once.
4. Qualitia and Tellurium – A wrapper for the Selenium engine.

Selenium Modes:

Bases on the components Selenium has following three modes for executing the test cases and test suites:

• Record-Playback mode (Selenium IDE)‏:

In this mode only Selenium IDE is used to record the test scenarios in terms of test cases in firefox. This is a great way to get started to writing tests and group them together to form the test suite. The recorded tests can be exported to many programming languages so that we can tweak them and put them in the testing framework. The test cases and test suites can be replayed back to check the verifications and validations or sent to Selenium RC or Grid for further tweaking.

• Selenium Remote Control (RC) Mode

In this mode Selenium starts multiple browsers (one at a time) and then runs the recorded test-cases which are saved in your language of choice. This helps to enhance the test cases with looping and programming techniques to cover all the required test scenarios and checks.

• Test Runner Mode

In this mode the test cases are recorded ad replayed in the form of HTML tables. This is just one more facility to execute the Selenium IDE as well as RC test cases. This helps to check the test results reports in better manner if not formatted already.

Testing AJAX with Selenium

Selenium IDE – Recording and updating a script

Selenium IDE is the FireFox Add-on provided by the Selenium group. This is very simple and easy to use add-on so that non-programmers can record and create the test scripts for automating web components. These automated test scripts are used as Selenium RC test cases by choosing the language code. i.e. Selenium IDE makes easier to create Selenium RC test cases. (Install Selenium IDE and Selenium RC from Selenium download page).

Selenium IDE is used for:

• Recording and updating the test cases or write them manually in table tab.
• Creating Test Suite by grouping the test cases under one group
• Exporting Test Cases/Suites the supported language and save for Selenium RC to enhance them
• Finding reference of every API and Selense commands
• Debugging test cases by toggling breakpoints through the commands

How commands generated in IDE for AJAX pages:

1. Almost all of the web pages are developed with AJAX technology since web2.0 evolution. While recording the page elements, Selenium IDE locates the page elements/objects using XPath/DOM/CSS. XPaths are found automatically, but you may have to update the specific elements in Xpath manually by looking into the source of the page to make them unique. Firebug is the helpful firefox Add-on for this purpose.
2. Every command has maximum two arguments. The first is usually target element pointed by its ID, name, link, DOM, CSS or XPath and the second is the value to that target element if exists. The selenium IDE commands when saved in the programming language of our choice for selenium RC purpose are called as Selenese.
3. The common commands are open, click, type, assert, verify. Commands beginning with ‘assert’ or ‘verify’ are for checks and verifications. If verify command is used and it fails, it fails the test case and continue to run the remaining tests. If assert command is used and it fails, the test case is aborted and stops to run the remaining tests.
4. For synchronizing the flow of test execution Selenium has commands ending with ‘AndWait’, it tells Selenium to wait for the page to load after the action has been done. However, it fails when using AJAX calls. In this case you have to use ‘WaitFor’ commands.

Selenium IDE – Most commonly used commands

The following are the most commonly used commands in the Selenium IDE:

open: Opens a page using a URL.

click: Clicks the element/object on the page.

clickAndWait: Performs a click operation, and optionally waits for a new page to load.

verifyTitle: Verifies the expected title and continues to run if it fails.

assertTitle: Verifies an expected page title and stops the execution if it fails.

verifyTextPresent: Verifies that the expected text is present somewhere on the page.

verifyElementPresent: Verifies an expected UI element, as defined by its HTML tag.

verifyText: Verifies that the expected text and its corresponding HTML tag are present on the page.

waitForPageToLoad: Pauses execution until an expected new page loads.

waitForElementPresent: Pauses execution until an expected UI element, as defined by its HTML tag, is present on the page. Used with AJAX calls.

Get more details about Selenium IDE here: Selenium IDE documentation.

If you need a full reference of all Selenium commands, read the Selenium commands.

Conclusion

In the first part of this blog, we have covered the basics of a typical test setup required for Selenium for automated testing of a Web-based application.

In the next part, we shall see automated testing in action – i.e., a practical example of how to actually use the Selenium IDE and RC to execute the tests. The reader is advised to check out the links highlighted in this blog to build the foundation needed for part two.

Introduction

This is the second in a three-part blog on Selenium – a free and open source automated software testing tool.

In part one of this blog, we saw the basics of Selenium, its test packages, test case execution modes and some of the commonly used commands in the Firefox-based Selenium IDE. We had also seen the basics of testing AJAX with Selenium, with an overview of how commands are generated in Selenium IDE for AJAX pages.

In this blog, we shall have an overview of AJAX testing using the Selenium IDE.

Commands available in a test script recorded using the Selenium IDE

To know about the commands available in the Selenium IDE, download the IDE and add it as an extension to the Firefox browser, after which, it will be available from under the Tools menu in Firefox.

In order to record test cases using the IDE:

1. Start the Firefox browser.
2. Go to the Tools menu of the browser and select Selenium IDE to launch the IDE.
3. In the Base URL field of the IDE, enter the URL of a Website on which user actions are to be recorded for automated testing purposes.
4. Click the red Record button on the top right side below the Base URL field.
5. Record the script for the test cases/test scenarios as per the test plan.
6. A Base URL field to enter the URL of the site to be tested.
7. A Record button to record the tests.
8. A Replay button with two options – one for replaying a particular test-case and one for replaying the entire test-suite.
9. A Pause/Resume button to pause or resume the tests under execution.
10. A Step button for step-wise debugging of the test cases.
11. A collapsible test case list area, test case tab area and test case detail area with Table and Source modes.
12. Text columns such as Command, Target and Value.
13. Log, Reference, UI-Element and Roll-up tabs at the bottom for reference.
14. Table mode, and
15. Source mode.

Selenium IDE has the following components to record and replay test scripts:

Let’s see a simple example of the test script generated using the Selenium IDE. The script displayed in the IDE has two modes:

Table Mode

This mode displays three column lists – Command, Target and Value, as shown in the previous screenshot. The Command option has recorded commands for the events as per the test cases while recording. The Target option has the target to be tested i.e., text, XPath, DOM and CSS. The Value option has the value for the command and the target for each event. We can also have text in the command line as a comment. The Table mode has three text fields, one each for the command, target and value. The Command field is an auto-complete dropdown used to enter and update/enhance the commands. The Target field is used to enter and update/enhance the value of the XPath/DOM/CSS object. The Value field is used to enter the specific value required for the commands recorded.

Source Mode

By default, the recorded script is displayed as HTML under the Source mode. However, the script can be exported in any supported/required target language for the Selenium RC framework of your interest. To do this, go to Options > Format and select the supported language for your framework from the available (configured) options, as shown in the following screenshot.

However, Selenium-IDE does not directly support:

• Conditional statements and iterations.
• Error handling, particularly unexpected errors.
• Database testing.
• Test case grouping and re-execution of failed tests.
• Test case dependency.
• Logging and reporting of test results.
• Capturing screenshots of test failures.

Although these tasks are not supported by Selenium directly, all of them can be achieved by using programming techniques with a language-specific Selenium RC client library – a subject that we shall look into in more detail in the third and concluding part of this blog.

Basics of Selenium RC

Selenium RC starts a Web server that provides APIs for client libraries as well as Selenium Grid – a tool which transparently distributes your tests over multiple machines allowing parallel execution of the test cases increasing the overall testing efficiency. Selenium-RC uses the full power of programming languages to create more complex tests like reading and writing files, querying a database, and emailing test results.

Selenium RC comes in the following two parts:

1. A server which automatically launches and kills browsers, and acts as a HTTP proxy for Web requests from them, and
2. Client libraries for your favorite computer programming language.

In order to execute the Selenese code in the programming language environment, first install the environment for that language. We will be using Java, so install Eclipse for Java, install JUnit and then install Selenium RC. The detailed steps are outlined below:

1. Download and unpack the Selenium RC archive from Selenium RC.
2. Set up a programming project for the Java client driver, as listed below:

• Download Selenium-RC from the Selenium HQ downloads page.
• Extract the file selenium-java-client-driver.jar.
• Open your desired Java IDE (Eclipse, NetBeans, IntelliJ, Netweaver, etc.)
• Create a new project.
• Add the selenium-java-client-driver.jar files to your project as references.
• Add to your project classpath the file selenium-java-client-driver.jar.
• From Selenium-IDE, export a script to a Java file and include it in your Java project, or write your Selenium test in Java using the selenium-java-client API. The API is presented later in this blog. You can either use JUnit, or TestNg to run your test, or you can write your own simple main() program. These concepts are explained later in this section.
• Run Selenium server from the console.
• Execute your test from the Java IDE or from the command-line.
  • Run console and go to the selenium-server folder.
  • Type: java -jar selenium-server.jar on the above path.

3. Install the Selenium-RC Server:

You may configure the Selenium server based on the java -jar selenium-server.jar –help help file. Now, save the sample code from our example above in Java format in your workspace directory and open it in Eclipse. A snapshot of the code that can be exported in Java for JUnit and TestNG looks like the one shown below:

package com.example.tests;
import com.thoughtworks.selenium.*;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import java.util.regex.Pattern;

public class Sample extends SeleneseTestCase {
@Before
	public void setUp() throws Exception {
	//code to start browser test
	}
@Test
public void testSample() throws Exception {

//Open the url

//code to verify the logo on the page

// code to get the auto-complete list
  	selenium.click("query");
	selenium.waitForPageToLoad("30000");
	selenium.type("query", "san");
	selenium.waitForPageToLoad("30000");
	selenium.keyDown("query", "40");
	selenium.mouseDown("query");
	selenium.waitForPageToLoad("30000");

//Select the item from the list

selenium.click("//div[@id='destination_selections']/ul/li[8]");
	selenium.waitForPageToLoad("30000");

@After
// code to stop the test case and browser
}

The autocomplete feature of AJAX

In the above example, we have automated the auto-complete dropdown scenario, which works like this:

Enter the prefix of the name of a city in the text field, e.g., “san” (as in “San Francisco”). The auto-complete list of cities is then displayed for “san”. Click on one of the items from the dropdown list. The clicked item will be given as an input to the field to display the next page. Verify a few items on the page to check whether the page displayed is indeed for the input which was given.

Conclusion

In part two of this blog, we saw how to test AJAX components in Selenium with a code example. We also saw the autocomplete feature of AJAX – a useful tool for testing textbox fields and dropdowns on Web pages.

In the third and concluding part, we shall see some advanced features of Selenium RC along with selense, and  wrappers for the Selenium engine.

Introduction

This is the third and concluding part of a three-part blog on Selenium – a free and open source automated software testing tool.

In part one of this blog, we saw the basics of Selenium, its test packages, test case execution modes and some of the commonly used commands in the Firefox-based Selenium IDE. We also had an overview of how commands are generated in the Selenium IDE for AJAX pages.

In part two, we had an overview of AJAX testing using the Selenium IDE.

In this part, we shall see some of the more advanced features that Selenium offers, such as Selenium RC, and wrappers for the Selenium engine.

Selenium commands vs. Selenese commands

Commands generated in the Selenium IDE Table mode are known as Selenium commands which are simple, type/click-type commands, whereas the commands generated when the script is exported in a user-selected programming language for Selenium RC are known as Selenese commands.

Using XPath/DOM/IDs for automating UI tests with exact position of objects

Selenium is capable of capturing the test items/objects both with and without the XPath values. Capturing the item with its absolute XPath value helps to test the exact position of the object on the page, whereas without the XPath value, the item can be located anywhere on the page. The items which are captured from the dropdown/combo lists always have an XPath value associated with them. These values are in the //div[n]/div[]/h2, //div[n]/ul/li format. In the following subsections, we will analyze the command flow with XPath to select some items from the auto-complete lists.

Synchronizing the script with the mouse and keyboard related commands

While recording, the generated script only has commands related to the user’s mouse clicks and keystrokes, but these commands are not sufficient and fail when the recorded script is replayed. Only clicked and typed commands are not sufficient to recognize the auto-complete list items displayed on the screen. In order to get an item selected from the auto-complete list and execute the script successfully, we need to modify the script with some keyboard and mouse related commands such as keypress, keydown, mousedown, mouseup and mousedownAndWait etc. The flow of script execution is synchronized with such commands.

A sample script generated while recording a test case is:

click query
type san
click //div[@id='destination_selections']/ul/li[8]

This script can be modified with some keyDown and mouse events to synchronize the selection of items from the auto-complete list so that the script executes successfully:

clickAndWait query
typeAndWait query san
keyDown query 40
mouseDownAndWait query
clickAndWait //div[@id='destination_selections']/ul/li[8]

A user possessing knowledge of the basic XPath concepts will easily understand that the position of the item is in ‘li’ of ‘ul’ of ‘div’ with id equal to ‘destination_selections’. This simulates a click on the 8^th ‘li’ item in ‘/ul/’ which is “San José del Cabo” for our example. However, we can generalize it further using regular expressions so that it will select the item based on the given ‘li’ id.

Some of the basic commands along with their Java Selenese equivalent commands are listed next.

Commands for synchronization of the script execution

clickAndWait commands with XPath:
clickAndWait //div[@id='destination_selections']/ul/li[8]

The equivalent Java Selenese for this command is:

selenium.click("//div[@id='destination_selections']/ul/li[8]");
selenium.waitForPageToLoad("30000");

Another example of wait command:

waitfor commands with XPath:
waitForText  //div[@id='leftColumn']/div[3]/div/h2 Best hotels, facts and information in San José del Cabo

The equivalent Java Selenese for this command is:

for (int second = 0;; second++) {
if (second >= 60) fail("timeout");
	try {
if ("Best hotels, facts and information in San José del cabo".equals(selenium.getText("//div[@id='leftColumn']/div[3]/div/h2")))
break; } catch (Exception e) {}
	Thread.sleep(1000);
		}

Commands for verification and validation – verify commands with XPath:

verifyText  //div[@id='leftColumn']/div[3]/div/h2 Best hotels, facts and information in San José del Cabo

The equivalent Java Selenese for this command is:

verifyEquals("Best hotels, facts and information in San José del Cabo", selenium.getText("//div[@id='leftColumn']/div[3]/div/h2"));

This is how simple and easy it is to understand the script converted in the Selenese (JUnit) code. For further automation tasks, we need to apply regular expressions, parameterization, and enhance the script with conditional loops for functional logic and this is the actual challenging task for the QA engineer.

Selenium Grid

Selenium Grid manages several Selenium RCs through API commands which are called Selenese commands. It is a tool which dramatically speeds up functional testing of Web applications by leveraging the user’s existing computing infrastructure. It allows the user to easily run multiple tests in parallel, on multiple machines, in a heterogeneous environment. If the user wishes to test Selenium Grid, additional information may be obtained from here. Setting up Selenium RC on different systems and connecting them using Selenium Grid is a pretty complex task, but it can be made easier by buying services that run in clouds. An example of such a service is http://saucelabs.com.

Wrappers for Selenium: Qualitia™ and Tellurium

Writing UI module-based Web automation scripts in Selenium is not yet supported. To achieve this, there are some wrappers available for the Selenium engine. Qualitia and Tellurium are two popular such wrappers used for Selenium and a few similar other tools. While presently acting as wrappers for the Selenium test units, Qualitia and Tellurium seem to be developing into independent test tools going forward.

Qualitia™

Qualitia™ is a thick client application based on .NET. It has seamless integration and the ability to work as a wrapper with market leading test automation tools such as Rational Functional Tester, QTP, and Selenium. It also has common test case management methodologies which helps the Selenium engine to manage the test cases and execute them as suites as well as scenarios. The fully loaded and easy to use graphical user interface helps to access the functionalities, building test cases, execution, error reporting, diagnosis and analysis. Qualitia can generate both XML based HTML reports.

Tellurium

Because Selenium does not support the ‘UI module-based’ Web automated testing, Tellurium is used as a wrapper for Selenium unit tests. Tellurium is a portable software testing framework for Web applications that runs on top of Selenium. The statement Tellurium is just a wrapper for Selenium was true only till version 0.6.0. Tellurium will be an independent testing framework while still acting as wrapper to Selenium unit tests (for backward compatibility) from version 0.7.0 onwards. Tellurium focuses on Groovy objects for runtime locator mapping, and then uses Selenium RC under the hood to drive the tests. The diagram below explains how Tellurium acts as a wrapper along with the test execution flow. The Tellurium framework is a separate topic altogether and is not within the scope of this blog.

Using Tellurium with Selenium RC and Selenium Core

Best practices for using Selenium

• Make sure that all tests are source controlled.
• Make use of nightly runs for most of the tests.
• Try to use one test per function, and not one test per page.
• Always use separate environment specific variables.
• Automate only the fixed functions, keep the other ones manual.
• Try to abstract out and reuse the functionality.
• Maximize the use of parameterization in your tests.
• Use ids wherever possible (XPaths make the tests brittle).
• In order to generate unique ids, use timestamps.
• Use the right level of granularity.
• Use data-driven tests.
• Use JSP/PHP pages to generate the tests.

Benefits of Selenium

• Free and open-source tool.
• Helps automate testing on AJAX/CSS applications.
• Supports testing on multiple browsers.
• Contains record and playback facility.
• Best for GUI-intelligent field selection (uses IDs, names, or XPaths as needed).
• Auto-complete feature available in IDE for all common commands.
• User-friendly features for debugging and setting breakpoints.
• Available in multiple languages of the user’s choice; can save tests as HTML, Ruby scripts, or several other formats.
• Support for Selenium user-extension .js files for explicitly looping add-ons.
• Option to automatically assert the title of every page.
• Mimics actual user experience.
• Continuous integration:
  • Runs Selenium tests as part of the build.
  • Can generate HTML reports and publish them to the entire team.
  • Helps catch bugs ASAP.
• Large user community.

Limitations of Selenium

Problems in low level testing:

• Performance issues (HTMLUnit, WebTest).
• Reporting issues (WebTest).
• CSS/AJAX needs tricky coding for correct and efficient usage.
• Data dependency needs to be avoided.
• Slow when testing edge cases.
• Slow for testing fine-grained features (since set-up and tear-down are called for each test).

Limitations applicable to Selenium Core:

• Hard to automate and generate alerts.
• Cannot go with “Same Origin Policy”.
• File operations such as file uploads are difficult to handle.

Common problems in Web applications testing:

• Record/replay feature is not reusable and is fragile.
• Test case management is not flexible.
• UI changes make tests breakable and need rework.

Conclusion

In this concluding part of the blog on Selenium, we saw several advanced features of the Selenium suite, such as Selenium RC and Selenium Grid. We also had an overview of some of the wrappers available for Selenium, which help the user in performing tasks not supported by the Selenium core itself. We finally ended the blog with a quick overview of Selenium’s advantages and disadvantages.

Overall, the popularity of Selenium is increasing by the day since many software applications are slowly migrating from the traditional desktop platform to the Web platform. The open-source advantage of Selenium along with its large present user base will ensure that it remains one of the most popular tools for testing Web-based applications for years to come.

In this blog I will be introducing FitNesse tool, an automated testing tool for user acceptance testing. This blog will give an overview of the tool which is a Wiki built on top of Fit framework and used for automating Acceptance test cases.

Existing User Acceptance Testing Processes

• Manual Acceptance testing: User exercises the system manually using his system knowledge.
• Acceptance testing with “GUI Test Drivers” (at the GUI level): These tools help the developer perform functional/acceptance testing through user interface such as a native GUI or web interface.
• Table-based Approach for acceptance testing: Starting from a user story (or use case or textual requirement), the customer enters in a table (spreadsheet application, html, etc) the expectations of the program’s behavior. The customer can manually insert inputs in the system and compare outputs with expected results.

Why FitNesse?

The Existing Manual User Acceptance Test process as described above is time consuming, prone to errors and incurs cost. Manual testing becomes exhaustive because of the large test data and numerous test cases. This process can be improved by automating the test cases for acceptance testing phase. FitNesse is the only Open source automation tool available for automating the Acceptance test cases/scenarios. This tool is Simple and has Robust Test Frameworks and also provides simple and user friendly result reporting.

What is FitNesse?

• FitNesse is an Open source integrated testing framework for automating the User Acceptance Testing and Regression Testing process.
• FitNesse creates a feedback loop between customers and programmers and allows customers and testers to use tools like MS Office Excel to give examples on how a program should behave.
• FitNesse automatically checks those examples against the actual program, thus building a simple and powerful bridge between the business and software engineering worlds.
• FitNesse enables customers, testers and programmers to learn what their software should do, and to automatically compare that to what it actually does.
• FitNesse is used by agile web teams to create comprehensive test suites that make system-level assertions to test in the TDD process.

FitNesse is basically

• Open source User accepting Testing tool written in Java 1.4
• Tests written in the form of HTML tables in wiki pages
• Makes use of two frameworks – Fit and Slim frameworks
• Test tables references fixture classes that connects to the business logic, Fixture classes/code are written in either – Java, .Net, Ruby, Python, C#

Benefits of FitNesse

Reusability of Test Cases based on dataset – Same Set of Test Cases can be used to test positive as well as negative scenarios with different dataset.

Reusability of Framework and Fixtures – As the Fixture classes are developed using Java, they can be reused for similar scenarios and fixture tables. Frameworks are also reusable for similar types of scenarios and functionalities.

Unit Testing and Regression Testing – FitNesse can be used and implemented for Unit Testing as well as Regression testing. Generation of fixture classes and test table can be incorporated as a process in the development life cycle. This gives advantage of using the same test cases for unit testing which will save time required for next levels of testing by finding the defects at early stages which will reduce testing cost.

Script Maintenance Activity – The developer, who works on change request can also modify the test cases in Fitnesse as a part of process. This will help in upgrading the test scripts quickly as per the new features in the application.

Multiple Test Suites – Once all the test cases are created in Fitnesse, different Test Suite can be created based on the selection of test cases that are required for execution. This will help in creating multiple test suites like- Unit Test, Regression Test and user Acceptance Test.

Independent Test Cases – The Test Cases in Fitnesse are independent of each other. During script execution, if a specific test case is not correctly executed it will not affect other test cases in the suite.

Reliability – Automation Process is always more reliable than the Manual Process

FitNesse Testing Process/Framework

1. Write unit/regression/acceptance tests:  These tests are written in a form of executable tables with Wiki syntax.
2. Write fixture code:  After the customers define the acceptance tests in wiki tables, the developers work on implementing the test cases with fixtures which communicates with business logic.
3. Run acceptance tests:  Acceptance tests are executed through the FitNesse Test runner.  The Test runner parses the test tables to find the key words and link them together to construct the class name of the fixture code which communicates with the business logic and returns the actual data.
4. View test results:  After running the acceptance tests on the FitNesse server, the results are shown in the form of the original test table with the appropriate value cells colors that represent the acceptance testing results.

The Table styles

There are two types of test-systems/frameworks available in FitNesse- Fit and Slim. The test/wiki pages have wiki text written in wiki syntax, the input and expected output is written in the form of tables. The Fit and Slim Test System has it’s own particular kind of Test Table styles. First Row- first cell always refers Fixture Name. Subsequent cells are used to pass the input parameters and expected values.

The Most Common Fit Table/Fixture Styles

Column Fixture	This is the style you may end up using most: rows of data represent inputs and expected outputs.
Row Fixture	This is good for testing queries that should return an exact set of values (Order-independently).
Action Fixture	This style allows you write a script that emulates a series of events (Such as controls manipulated on a user interface).
Comment Tables	Sometimes you want a tabular comment that is not executed as a test.

The Slim Table styles

The first cell of a slim table tells you what kind of table it is. Here are the table types so far:

Decision Table	This is similar to Fit Column Fixture which supplies inputs and outputs for decision.
Query Table	Supplies the expected results of a query. This is similar to the Fit Row Fixture
Subset Query Table	Supplies a subset of the expected results of a query.
Ordered query Table	Supplies the expected results of a query. The rows are expected to be in order. This is similar to the Fit Row Fixture
Script Table	A series of actions and checks. Similar to Do Fixture.
Table Table	The design which you want it to be!
Import	Add a path to the fixture search path.
Comment	A table that does nothing.
Scenario Table	A table that can be called from other tables.
Library Table	A table that installs fixtures available for all test pages

Advantages of FitNesse

• Fitnesse is webserver – requires no extra configuration or Setup
• FitNesse is a wiki –You can easily create and edit the testcases on wiki pages
• Tests can be written before the code so that this approach can support TDD-Agile approach
• Requirements Engineering –
  • Easy to describe the Requirements themselves for existing frameworks using tables
  • Good tool for requirements engineering
  • Easy for the developer to glue the requirements with business logic using Fixtures
• FitNesse is light weight and Open Source framework that makes it easy for software teams to:
  • Collaboratively define Acceptance Tests- wiki with table of inputs and expected outputs
  • Write the Fixture code to communicate this tests with Business logic
  • Run those tests and analyze the results
• FitNesse has features adapted to system tests, like:
  • CommandLineFixture available to execute special cases
  • DoFixture and Ordered Query table executing some methods in order
• FitNesse is perfectly suited for regression testing
• The biggest advantage Fitnesse has over other integration testing frameworks is that the entry point into the code stack is not the (web) frontend – fixtures are written in java/C# and call out to whatever part of the codebase you are testing. This allows us to test separate layers as well as testing the entire code stack working together.

Limitations of FitNesse

1. No capture & replay possible
2. Cannot explicitly test a web frontend
3. Web interface can make setting up expectations time consuming
4. Learning curve to understand the framework is steeper if we have to modify the framework.

This was a brief introduction to the FitNesse tool, in the subsequent blogs we shall see how to use FitNesse tool to Create Test pages/wiki pages, Write Fixture classes, etc along with code samples

Introduction

As explained earlier, FitNesse is a wiki server; it helps us to organize all our test scripts in the form or Wiki pages. It is a web server, it enables multiple people to work on automation scripts by keeping them at a central location. Even when fixture is doing all the work, at some point of time when the fixture has to be invoked, FitNesse wiki is the calling application. It supplies the fixture right automation scripts at right time as and when the tester wants to do the testing. FitNesse receives the statistics about the test status (pass or fail) and creates a report in wiki page itself which is easy to debug. For creating the wiki pages we should follow the following steps:

Downloading and Installing FitNesse

It really takes very little time and effort to get FitNesse running on your machine.

Go to http://fitnesse.org/FrontPage.FitNesseDevelopment.DownLoad and click on the most recent fitnesse.jar file. Download the fitnesse.jar to some location in local machine.

Type java -jar fitnesse.jar

Start up a browser and go to http://localhost:8080

The main FitNesse screen should come up. That’s it. You’re ready to start using FitNesse.

Process for creating/editing new wiki page:

• Go to the home page http://localhost:8080
• Click on edit link on LHS side
• Enter the CamelCase text as a name of the page link e.g. QueryTest, NewTestPage

(The names: newTestpage, newTestPage are invalid and the names: NewTestPage, NewtestPage, SampleTestPage, SampleTestpage, SampleExample are valid)

• Click on save button
• Check the frontpage displayed with the text QueryTest? With a link on “?”
• Click the link on “?”
• New page displayed with URL http://localhost:8080/QueryTest?edit&nonExistent=true

OR

• For creating or editing new wiki page directly through URL:
  • current-url.NewPageName
  • current-url.NewPageName?edit
• click on save and again go to this new page and click on ‘Properties’ link on LHS panel
• This displays the page for setting up the page properties for the page
  • Select Test from Page type, check all check boxes from Actions and keep others default
  • Click on ‘Save properties’ button at the bottom
  • Check the new page with the buttons like Test, Edit and so on… at the LHS panel

For more detailed commands related to creating and editing the new wiki pages check out the page: http://fitnesse.org/FitNesse.UserGuide.QuickReferenceGuide

The ever increasing demand of mobile devices has given a push to software developers in taking the traditional web applications to mobile environment. The challenge is to provide user experience as similar and seamless across various mobile devices as possible in spite of the limitations which the mobile environment poses, adopting an agile methodology to develop the mobile applications for a diversified device environment, hardware and networking considerations.

Mobile device markets that includes Smartphones, Tablets, PDAs etc. is growing dynamically making the mobile application developers strive to deliver most robust, scalable applications with quality assurance Every device platform creates a unique testing environment challenging the mobile application developers to follow different testing strategies. Here we shall see how different types of testing approaches can be taken up for a variety of mobile platforms.

Numerous different mobile platforms available for mobile applications to name a few:

• Apple’s iOS
• Google’s Android
• Nokia’s Symbian, Maemo and MeeGo
• Palm/HP’s WebOS
• Samsung’s Bada
• RIM’s BlackBerry OS., and many more.

As mentioned, every platform needs a different testing approach. A combination of manual and automation testing can be done for an effective outcome.

Following are different types of manual testing for mobile environment:

• Functionality Testing:

Functional testing mainly includes finding device specific bugs and navigational issues of application. This type of testing should be done at the initial stage when the application is under development. In functional testing we can check database or network queries with response times, crashes and memory issues. Functionality testing of a mobile application is a black-box type of testing to assure that the application is functioning as per the business specifications.

• Usability Testing:

Usability testing encompasses mobile interface testing, application navigation testing, and intuitiveness of the application, consistency, and soberness of color scheme. An ISO standard defines usability as “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use”.

In Usability testing ease and efficiency of the user and content verification will be tested as defined in common Usability guidelines.

• Network Connection Testing:

Every mobile application which utilizes internet needs to be tested under different networks. These are a few networks for testing applications:

2G (GPRS, CDMA, EDGE), 3G and WiFi.

This is a necessity because most of the times the server responses are different for different type of networks. This testing helps us check application behavior under different networks.

• Installation Testing:
  

While installing, the application should install required softwares automatically on the device. And while uninstalling the application, it should remove all the available content and databases from the device which are used by application.

• Performance Testing:

Performance testing is required for finding memory related issues in application when data is excessive, due to devices having less memory as compared to simulator, this testing is required to be done on device only. By this testing approach, we can find the stability and performance of application under excessive data.

• Stress Testing:

This testing is required for getting the application response time while clicking on different buttons regressively and adding more data on device. We can get different crashes and hangings of application while running the application for long time.

In absence of mobile devices, simulators have always played a vital role for testing mobile device applications, although device testing is always the ideal way as it represents more likely end user scenarios, the significance of simulators in testing cannot be ignored. To have an effective testing with the help of a simulator, it’s necessary to explore all the capabilities of simulator.

Simulator versus device testing:

Simulators are mainly used for functional testing of basic flows. Simulators are not used for performance and usability testing, but the final testing is conducted on actual devices so crashes and hangings can be identified. One cannot get quality application while relying only on simulator. Device testing is necessary for all the applications, as with device testing we can understand the application behavior on different networks.

Simulator:

Is a software application that can accurately imitate a mobile phone. Simulator is mainly used by developers to check the functionalities implemented under development phase.

Let’s discuss the pros and cons of using simulators:

• Pros:

- Helps in isolating issues which are not volatile network connection dependent
- Provides a wide variety of testing over different types of device simulators for the same build
- Allows to test the same build in multiple device screens.

• Cons:

- Simulators of older generation handsets don’t resemble the device as closely
- Some issues which are hit by the speed at which input was given cannot be reproduced easily
- Hardware/Firmware environment variations detectable in device testing only
- Device testing is always preferred as it represents more likely end user scenarios.

Device: Is the actual handset where application installed and runs.

There are some pros and cons while using real devices for testing as well.

• Pros:

- Finds actual issues of application.
- Finds crashes, memory leak issues which can not found on simulators.
- Checks application over 2G and 3G and different networks
- Checks application behavior while incoming call, SMS, MMS and alarm.

• Cons:

- Expensive for compatibility testing of application over wide range of devices
- Consumes more time for adding excessive test data for testing purpose.

Checklist to follow while testing a mobile application:

Following is a basic checklist which is required while testing mobile application for any platform:

1. Installation & Uninstallation: To verify whether the application can be installed & uninstalled successfully.

2. Network Connectivity:

• The application can use simultaneous connections properly
• The application follows the GSM Offline profile correctly when making connections.
• When GSM Offline profile is selected, application cannot take network connection or send an SMS/MMS
• The application can utilize WLAN, 2G and 3G networks correctly.
• Performance of application during network connectivity problem.

3. Call/SMS/alarm handling: Verify that Application pauses and resumes for the same state when there is an incoming phone call/SMS/MMS/Alarm notification.

4. Check the look & feel of the application

5. Content: Check if enough information is displayed

6. The application must function as defined in the Help, user’s guide, or functional specification

7. Performance : Application and inner pages load time.

Automation Testing:

Automation testing is usually extended to perform tasks impossible with manual testing in large applications. An automated software testing tool is capable of playing pre-recorded and predefined actions. The results are mapped to the expected behavior and report the success or failure of these manual tests. Once automated tests are created they can easily be repeated. Having experienced the efficiency of automated software testing, it has become an important part of an application testing.

Automation tools available for testing:

Mainly mobile testing is done manually on actual devices. Some of the following tools are available in to test the functionality as well as usability of application.

1. Robotium for Android
2. Testquest, try, and digia for Symbian
3. FoneMonkey for IPhones
4. Memory sweep for IPhone
5. Other tools: eggplant, VNC Robot, Hopper and TestQuest.

Advantages and Limitations of Automation Tools:

Advantages:
- Multiple tests can run in less time with fewer resources
- Automated Tools run tests faster than human users
- Can reuse tests on different versions of an application.

Limitations:
- Unable to test all the functionalities and Usability of application through automation tools
- Proficiency is required to write the automation test scripts for application different functionalities
- Debugging the test script if any error is present in the automation test script
- Storing and maintenance of test data is difficult.

I believe that this comprehensive information will be quite helpful to you if you are looking at some methods for an effective approach to test a mobile device application. You can leave your comments and/or questions; I would be looking forward to having an interactive conversation.