BLUEPRINT OF A REUSABLE TEST AUTOMATION SYSTEM
Blueprint of a Reusable Test Automation System Reference Architecture for Implementing a Reusable Test Automation System A Test Automation System (TAS) manages the automated execution of tests without human intervention, compares the automated test results with expected results and publishes the results. A TAS may also cover script-less test automation by allowing the test designer to create automated tests using more simplified set of reusable keywords, English like test specification or simplified model of the application under test. Reusability is the quality of a TAS that makes it independent from a specific testing problem and allows it to be used across a wider set of testing problems. The Return on Investment (ROI) of a well-implemented TAS continues to increase over time. Such a platform serves not only as an automation platform for multiple products, but also as an enterprise-wide test automation platform.
INTRODUCTION Test automation is usually used to automate repetitive manual
test design, automatically executes tests, compares the
testing tasks. Beyond labor saving, test automation also speeds
automated test results with predicted results and publishes the
up software testing cycles and thereby accelerating the delivery
test execution results. Reusability refers to the quality of the TAS
of value to the end-customer. Software testing lifecycle consists
that is sufficiently general and is independent of the current
of following steps – requirements review, test planning, test
testing problem for a specific application under test. This white
designing, test execution and test reporting. The ultimate goal of
paper provides a set of guidelines and a reference architecture
software test automation system is to automate all steps of the
for creating a reusable, extensible and maintainable TAS that
software testing lifecycle enabling faster and economical
can be used to test Web application, mobile application, desktop
delivery of applications. The whitepaper provides insights on the
application and Web services.
reusable Test Automation System (TAS) that allows script-less
PRINCIPLES OF INDEPENDENCE OF TEST AUTOMATION SYSTEM The guiding principles for implementing a reusable TAS are quite
significantly reduce the RoI that could potentially have been
elementary. Still many projects have gone about implementing
reaped from the spending towards implementing the TAS.
test automation without fully appreciating the fundamental
Separation of the TAS from the specifics of a particular testing
principles behind the TAS. This may not necessarily lead to
problem of an application under test is an important concept
failure of the test automation activity, nonetheless, it can
that is outlined in the four independence principles.
Blueprint of a Reusable Test Automation System
1
Test Designer
Test Data Test Design Test Report
Test Automation System (TAS)
Test Automation Tools
Web GUI
Desktop GUI
Mobile GUI
Web Service Application Under Test Figure 1: Test Automation System
Principle 1: TAS is Independent of Test Design
shaped
The test design specifies the sequence of steps for testing a
with each application that it tests. An enterprise-wide TAS is a
specific business function of an application under test, input
must for implementing Test Center of Excellence (CoE) as it
data and expected results. Test design activities are the same
allows a common pool of engineers to move unhindered across
whether the test is executed manually or automatically. Where-
multiple projects in the enterprise without having to deal with
as, TAS is an execution environment for automated execution of
several different project-specific TASs.
by
individual
project
schedule
and
deadline
commitments. We are better served by a single TAS spanning multiple projects in an enterprise that continuously evolves
the tests. Quite a large number of projects have approached test automation as an unbudgeted side activity of test design engineers. Creating a TAS, to support automated execution of the cases is a full-scale software development activity which requires software development skills and is independent of test
Principle 3: TAS is Independent of Test Automation Tools There are many test automation tools available in the market
design activity.
for automated testing of Web application, mobile application,
Principle 2: TAS is Independent of Applications and Projects
desktop application and APIs. Quite a lot of these tools tend to point solutions that solve a specific problem very well e.g. Apache Selenium for Web GUI testing and Apache JMeter for Web service testing. Also, each automation tool supports a set
Each application implements a unique functionality or
of programming languages that are used to create test scripts
implements similar functionality in a unique manner. Rather
using the programming interface provided by the tool. A real-life
than focusing on uniqueness of each application, TAS uses the
application under test may require test automation across
common
test
multiple digital channels such as Web, mobile, desktop, social
are Web
platform and Web services. To address test automation of such
applications, mobile applications, desktop applications and
applications, multiple complementary tools from various
Application Programming Interfaces (APIs). For example, all
vendors have to be integrated under a single umbrella to satisfy
Web GUI applications have some form of window, menu,
all the test automation requirements. The TAS provides an
toolbar, textbox, push button, combo box etc. TAS achieves
abstracted interface with a common vocabulary spanning
application independence by focusing on these common
multiple test automation tools. It is important to create
components. This is a well-understood principle. Yet, many
vendor-neutral TAS so that the system components can be
enterprises implement a separate TAS for each project
replaced without impacting overall solution.
constituents
automation. Typical
of
these
applications
applications under
tests
for
resulting in multiple partially reusable frameworks that are
Blueprint of a Reusable Test Automation System
2
Principle 4: TAS is Independent of Test Automation Technique Test
automation
techniques
initial
language-specific test scripts. Hybrid approach utilizes
approaches using record and playback liner scripting and
lower-level keyword-driven and data-driven approaches to
structured scripting approaches to newer techniques involving
provide a high level set of domain-specific keyword library.
data-driven, keyword-driven, hybrid, behavioral-driven and
Behavioral-driven testing approach allows tests to be specified
model based approaches that shield the test designers from
in a domain-specific English like language based on the
the intricacies of scripting. Linear scripting approach records
high-level software requirements. Model-based testing relies on
the interactions with the application under test and plays it
creating simplified models of the application under test to
back to achieve test automation. The issues of duplicity,
automatically generate the tests based on the model, thus
maintainability and reusability with record and play linear
liberating the test designers from designing the tests for each
scripts are solved by either modifying the recorded scripts or by
feature. Instead, the effort is spent in creating a simplified
manually creating structured scripts that are modular and
model that adequately represents the application under test.
reusable. Data-driven approach solves the issue of reusing the
TAS provides a common framework for multiple test
same script with different data for similar kinds of test by
automation
externalizing the test data. Keyword-driven testing provides a
behavioral-driven and model-based approach that utilizes the
library of keywords that can be used by non-programmers to
basic building blocks of hybrid framework built on lower-level
create
keyword-driven and data-driven approaches.
automated
tests,
have
evolved
without
having
from
to
create
techniques.
A
layered
TAS
can
provide
REUSABLE TEST AUTOMATION SYSTEM ARCHITECTURE Software architecture is the structure, organization and
breakdown of a system into its parts; the decisions that are
interactions of the key component of a system that meets all of
hard to change; there are multiple architectures in a system;
the technical and operational requirements, while optimizing
what is architecturally significant can change over a system's
common quality attributes such as performance, security and
lifetime; and, in the end, architecture boils down to whatever
manageability [2]. Martin Fowler
the important stuff is� [3].
says, “the highest-level
Test Data Generator
Behavioural Driven Testing Data Driven Testing
Hybrid Testing Structured Scripting
Reusable Business Operations (Auto-generated, Application Specific)
Layer 2 - Test Operations Layer Application 1 Application 1 Application 2 Application 2 Application N Application N REST API Web API REST API Desktop API Mobile GUI SOAP API Abstraction Abstraction Abstraction Abstraction Abstraction Abstraction Element Operations (System Defined, Application Independent)
Layer 1 - Test Adaptation Layer Graphical User Interface Adapters (Web App, Desktop App, Mobile App)
Application Programming Interface Adapters ( REST, SOAP, XML)
Other Interface Adapters ( Log files, SQL DB, No SQL DB, Emulator, Simulator,
Selenium Adapter
HP UFT Adapter
Jmeter Adapter
SoapUI Adapter
ELK Adapter
JDBG Adapter
MS Visual Studio Adapter
Appium Adapter
Gatling Adapter
Load UI Adapter
Cassandra Adapter
Cisco CLI Adapter
Application TAS Code Code
Test Suite/ Test Test Case Data
Web Page Object Service Model Definition
Test Design
Test Management
Test Logging Test Execution Test Workflow
Test Orchestration
Test Reporting
Layer 3 - Test Case Layer Model Based Testing Keyword Driven Testing
Configuration Management & Version Control
Test Automation System
Test Execution Report
Test Setup Deployement & Teardown Scripts
Application Under Test
Figure 2: Test Automation System Reference Architecture
Blueprint of a Reusable Test Automation System
3
Test Automation System Reference Architecture Layering and partitioning are architectural techniques to realize a weakly coupled decomposition of a system into the constituent sub-systems. Alternate implementations of the same layer or partition, conforming to same interface, can be used interchangeably or may even coexist. This allows enhancement of the system implementation over time. Layered architecture technique isolates the sub-system in hierarchical layers so that changes can be made in one layer without affecting the others. Each layer is highly cohesive that provides services to a higher layer. Each higher layer provides an incremental level of design abstraction. A layer depends on
Layer 2: Test Operations Layer Test operations layer provides an incremental level of abstraction based over test adaptation. The GUI pages of the application under test are modelled as page objects. The tests use the methods provided by the page object for interacting with the GUI page. This allows better maintainability of the test cases as all the GUI page level implementation is kept separate from the test case implementation and contained in a single place rather than being scattered throughout the test case. For example, a login page object model provides approaches to type text in username text field, type text in password field, click submit button, etc. Similarly for Web services API, this layer abstracts the API services so that any changes to the API are externalized from the test cases.
services provided by the lower layers. There is loose coupling between layers that allows the layers to be deployed independently even across distributed computing systems. Partitioning divides the system vertically into independent weakly coupled sub-systems at the same level of design abstraction. Figure 2 depicts the decomposition of the TAS into the constituent layers and partitions.
Layer 3: Test Case Layer One of the key measure of a successful TAS is the ease with which the test designers can create and manage test cases. Test case layer provides services to the test designer to create test cases, test suite and test data utilizing the services provided by the underlying layers. Based on the capabilities, the test
cases
maybe
created
using
structured
scripting,
data-driven, keyword-driven, hybrid, behavioral-driven or
Layer 1: Test Adaptation Layer Test adaptation layer provides the lowest layer of abstraction for the TAS. It provides the reusable elementary actions that are common to all applications under test. These actions are not decomposable into simpler operations on objects of the same type. Example of the elementary operations include checking a toggle box, setting focus on a window, maximizing a window, press down control key on the keyboard, release control key on the keyboard, and many more. A TAS may utilize multiple commercially licensed and open source test automation tools. Test adaptation layer offers a vendor-neutral interface to the upper layers so that the automation tools can be replaced without impacting the upper
model-based approach. For example, the test case for validating that registered users are able to login into the Web GUI application contains following steps: Open the Web browser Type the URL of the application user test Navigate to login page Type valid user name Type valid password Click on the submit button Verify that the home page of the application is launched Parameterizing the browser type allows the same test case to be run across multiple browsers based on parameters provided in a data file. Similarly, parameterizing the username and password allows the same test case to be run against multiple registered user, based on parameters provided in a data file.
layers. For example, HP Unified Functional Tester (UFT), IBM Rational Functional Tester and Apache Selenium are alternate tools to test Web GUI interface. During the life of the project, a decision can be taken to shift from using HP UFT to Selenium. The change is realized with minimal impact by shifting from using HP UFT adapter to Selenium adapter, as the test case implementation is independent of the tool vendor specific API and implementation language. Test cases can be distributed over multiple adapters of the same type to achieve scalability.
Blueprint of a Reusable Test Automation System
Test Orchestration Test orchestration allows creation of test workflows for continuous integration and continuous delivery, execution of the tests specified in the workflow, generating test log and test reporting. Test log is a chronological record of all significant details about execution of a test to aid troubleshooting in case
4
of failures. Test report is the summarized results of the tests
execute sanity test suite on build server: execute regression test
performed against a specific software build. An example of a
suite on small infrastructure configuration: execute test suite for
continuous delivery workflow activities triggered from the
the new feature on a small infrastructure configuration and
version control system as a result of a code check-in of the
execute performance test suite on production like infrastructure
application under test can be as follows: build the software
configuration.
release; perform unit tests; perform static code analysis:
EXTERNAL COMPONENTS TAS integrates with third-party configuration management and test management system for comprehensive test automation.
Configuration Management
Test Management
All artifacts relevant to the project across building, deployment,
Test management allows requirements traceability, test reports
testing and operations are stored in the configuration manage-
and test logs to be accessed for a test operation. They integrate
ment system. These artifacts include the documentation for
with bug tracking tools to track bug status and collaboration
test specification, application source code & build scripts, test
across the team for project tasks.
automation system source code & build scripts, third-party tools for development & testing, test cases, test scripts, test data, simulators, emulators, test adapters, test results, test reports, all application & third party tool configuration, database scripts and environments used for development, testing & operations. This ensures compatibility across the versions of the artifacts used for application under test, test automation system and automated tests.
CONCLUSION The architectural blueprint of a reusable test automation system
to make it usable in live projects. It can be developed over time
presented in this whitepaper can be used to implement a TAS
with incremental functionality as the automation benefits are
that is reusable for testing a large variety of projects. It has
realized in the projects even with minimum functionality availa-
weakly coupled sub-system that can be developed independent
ble in layer 1 test adaptation layer.
of other sub-systems. Not all sub-systems have to be developed
Blueprint of a Reusable Test Automation System
5
ARICENT’S SOFTWARE TEST AUTOMATION SERVICES Aricent has over 20 years of experience in implementing a wide
automation as well as performance test automation services
variety of test automation projects for Web applications, mobile
implemented in Agile methodology to address your test
applications, desktop applications and Web services for small as
automation needs.
well large sized projects. Aricent offers both functional test
Learn More To find out more about Software Product Quality Engineering Services, please visit https://www.aricent.com/ or email us at: info@aricent.com
Dinesh Singh Author
Blueprint of a Reusable Test Automation System
6
-
REFERENCES 1. Test Automation https://en.wikipedia.org/wiki/Test_automation 2. What is software architecture? https://msdn.microsoft.com/en-in/library/ee658098.aspx 3.Patterns of Enterprise Application Architecture, Martin Fowler
Aricent is the world’s #1 pure-play product engineering services firm. The company has 20-plus years experience co-creating ambitious products with the leading networking, telecom, software, semiconductor, Internet and industrial companies. The firm's 11,000-plus engineers focus exclusively on software-pow ered innovation for the connected world. The company’s key investors are KKR & Co. and Sequoia Capital. info@aricent.com
Š 2016 Aricent. All rights reserved. All Aricent brand and product names are service marks, trademarks, or registered marks of Aricent in the United States and other countries.