Automated Software Testing Magazine - August 2013 by Automated Test

An A U TO M AT E D T ESTING I NSTITUTE Publication - www.automatedtestinginstitute.com

A utomated ....... S T oftware esting

MAGAZINE

August 2013 $10.95

Planned

Agility The Testing Tsunami

Continuous Load Testing

Rethinking the Role of Automated Software Testing

Agility Applied to Automated Performance Testing

Selecting the Right Tool

The Pyramid Approach to selecting a tool to meet your needs

Good Things Come: Syncronization in HP Functional Tester (QTP)

TestKIT

ONDEMAND If you canâ&#x20AC;&#x2122;t be live, be virtual

ONDEMAND Sessions Anywhere, anytime access to testing and test automation sessions >>>> Available anywhere you have an internet connection <<<< >>>> Learn about automation tools, frameworks & techniques <<<< >>>> Explore mobile, cloud, virtualization, agile, security & management topics <<<< >>>> Connect & communicate with testing experts <<<<

ONDEMAND.TESTKITCONFERENCE.COM

A utomated S T oftware esting

August 2013, Volume 5, Issue 2

Contents Planned Agility Repeatedly dealing with the concepts of “continuous change”, “fast-paced environments”, and “agility” in isolation can be detrimental to the success of our projects because they have the potential of leading us to become too jittery and careless in the name of speed. This issue is dedicated to providing a more balanced approach of addressing continuous change coupled with an appropriate investment in getting the job done efficiently and effectively.

Features

Continuous Load Testing

This article addresses the importance of a well-thought-out, methodical, continuous approach to load testing in an Agile environment. By Michael Punsky

Selecting the Right Tool

This article provides a real-world model for selecting a test tool with confidence in the tools ability to meet your needs prior to investing in that tool.

By Bernd Beersma

The Testing Tsunami 28 Read this article to learn how one of the largest providers of healthcare information solutions in the U.S. leveraged the cataclysm of changes in their industry to rethink and revamp their test automation organization.

Columns & Departments Editorial

Planned and Well-Thought-Out Agility Balancing pace and planning.

Authors and events

Learn about AST authors and upcoming events.

TestKIT TIP

Good Things Come To Those That Wait Employing synchronization in HP Functional Tester

August 2013

Open sourcery

By Rubina Ansari

Automation Just Got Fuzzier! A new fuzzer has been brought to the public!

I ‘b’log to u 36

Read featured blog posts from the web. The Automated Software Testing (AST) Magazine is an Automated Testing Institute (ATI) publication. For more information regarding the magazine visit http://www.astmagazine.automatedtestinginstitute.com

www.automatedtestinginstitute.com

Automated Software Testing Magazine

Editorial

Planned and Well-Thought-Out Agility by Dion Johnson We talk a lot these days about “continuous change”, “fast-paced environments”, and “agility”, as well we should. Today’s IT environments have an abundance of these attributes and as software engineers we must be able to thrive in this new reality. Repeatedly dealing with these concepts in isolation, however, can be detrimental to the success of our projects because they have the potential of causing us to become too jittery and careless in the name of speed. A bull can probably move pretty quickly through a china shop, but that may not be the best course of action. Don’t get me wrong, I’m not suggesting that we try to slow the world down. For one, nobody would listen to such advice. But I also don’t think that would be good for technological advancement. I am suggesting that as we address these “speed-based” concepts we also mix in an adequate focus on other concepts such as investment, precision and correctness. Focusing on these concepts helps to provide a necessary balance within our IT activities so that we don’t hide poor quality and excessive rework beneath the guise of continuous activity.This issue of the magazine focuses on this type of balance by addressing continuous change coupled with an appropriate investment in getting the job done efficiently and effectively. The first feature entitled “Continuous Load Testing” by Michael Punsky 4

Automated Software Testing Magazine

I am suggesting that as we address these “speedbased” concepts we also mix in an adequate focus on other concepts such as investment, precision and correctness. www.automatedtestinginstitute.com

describes the importance of a well-thought-out and methodical approach to load testing in an Agile environment. Next, we take a look at a well-thought-out and methodical approach for selecting tools that may be used for implementation of load, functional or any other type of testing. In a feature article entitled “Selecting the Right Tool: The Pyramid Approach to Test Tool Selection,” Bernd Beersma provides a real-world model for selecting a test tool that you can be confident will meet your needs prior to investing in that tool. Finally, we discuss how to rethink and revamp your automation approaches to fit into a new mode of thinking in an article entitled “The Testing Tsunami: Rethinking the Role of Automation” by Rubina Ansari. August 2013

5 Annual th

ATI Automation Honors Celebrating Excellence in the Discipline of Software Test Automation

Voting Begin August 19th! www.atihonors.automatedtestinginstitute.com

Authors and Events Whoâ&#x20AC;&#x2122;s In This Issue? Michael

Punsky

Michael Punsky is Product Manager, Performance Solutions at SmartBear Software, provider of software quality tools used by more than one million developers and testers worldwide. Michael has had years of experience conducting performance tests, using a variety of tools. In addition, he has 20 years of experience in developing, implementing, growing and improving professional services practices. Michael has served in various roles, including sales and services, launch management, consulting, project management and technical support for Webapper Services, Brightcove, Adobe, Macromedia and Allaire. He was also QA/Training Release Manager for the Commonwealth of Massachusetts.

A utomated S T oftware esting

Managing Editor Dion Johnson Contributing Editors Donna Vance Edward Torrie Director of Marketing and Events Christine Johnson A PUBLICATION OF THE AUTOMATED TESTING INSTITUTE

Bernd Beersma

is competence leader in test automation and senior test consultant with Squerist. He has over 11 years experience with different forms of test automation and performance testing for different companies. Bernd holds a bachelor degree in software engineering and became acquainted with software testing during this period. During his numerous customer assignments Bernd created different frameworks and solutions for test automation with a broad variety of tools. These different approaches let to creating a generic approach for test automation. Because of his knowledge about test automation, he also gives advice on how to implement test automation and does workshops and trainings. Bernd is a member of the TestNet Test Automation workgroup and co-initiator of the Test Automation Day. His goal is to keep on learning and thus improving his skills on test automation enterprise.

Rubina Ansari has been a hands on Automation architect, implementing automation testing and testing tools at the organization level for over 15 years. Her career has spanned from the telecommunications industry to her current expertise, which is within the healthcare software arena. Rubina has been an Automation guru, building the Quality Center of Excellence in mid to large sized corporations. She is currently the Associate Vice President at Trizetto. Her current responsibilities include building and implementing the automation framework while rolling out Agile Tools that support and implement Scrum and Scale Agile across all Trizetto development centers in North America and overseas. 6

Automated Software Testing Magazine

CONTACT US AST Magazine astmagazine@automatedtestinginstitute.com ATI Online Reference contact@automatedtestinginstitute.com

ATI and Partner Events Now Available TestKIT On Demand Virtual Conference http://ondemand.testkitconference.com

August 19, 2013 ATI Honors Voting Begins

http://www.atihonors.automatedtestinginstitute.com

March 3-5 TestKIT 2014 Conference

http://www.testkitconference.com

The Automated Software Testing (AST) Magazine is an Automated Testing Institute (ATI) publication. For more information regarding the magazine visit http://www.astmagazine.automatedtestinginstitute.com

www.automatedtestinginstitute.com

August 2013

TestKIT Conference 2014 Testing & Test Automation Conference

Save the Date! March 3 - 5, 2014 Crowne Plaza Hotel Arlington, VA

The KIT is Koming... Back www.testkitconference.com August 2013

www.automatedtestinginstitute.com

Automated Software Testing Magazine

TestKIT Tip

Good Things Come To Those That Wait Tips For Employing Synchronization in HP Functional Tester (QTP)

TestKIT

is the name used by ATI for describing one’s testing toolkit. A TestKIT is filled with knowledge, information and tools that go with us wherever we go, allowing our projects and organizations to quickly reap the benefits of the practical elements that we’ve amassed. This section provides tips to add to your TestKIT.

C by

in A

a apt

ati

m uto

until the complete wait time has elapsed Example: 1

hings are not always ready when we need them, so we spend a lot of time waiting on things. Many times we are happy to wait for things that we want like waiting in line to get into a concert or waiting for a steak fresh off the grill. Sometimes we wait for things that we don’t really want, like waiting in lines at the Motor Vehicle Administration (MVA) or the airport for a late arrival. Your QTP test execution is the same. Sometimes objects or pages take a bit longer to load than usual. That’s probably not a defect. How do you make QTP understand that good things come to those who wait? During test execution the QTP scripts run as fast as they can, but some applications may need some extra time to render, enable an object or open a window or popup. Keeping this in mind, the script should not fail if the test condition is not immediately ready. QTP Tests can be taught to wait for the application to respond through a variety of synchronization methods.

Automated Software Testing Magazine

The following are some of the different types of synchronization tips and tricks you can use in your QTP scripting: • • • • • •

Wait Statement WaitProperty Exist property Sync Method Synchronization Points Increasing Tool Default Synchronization Time

Tip 1: Wait Statement Syntax: Wait(n) where “n” will be the seconds to wait. You can also include “,n” to add Milliseconds The “Wait” statement tells the script to pause execution for the specified length of time. Usage: “Wait” can be easily used anytime you need the test script to pause for a fixed time interval, but it is not smart enough to continue if a certain or desired condition is met. The disadvantage to using this is that even if the event you are waiting for occurs quickly the test will not continue www.automatedtestinginstitute.com

2 3

Browser(“MySite”).Page(“MyPage”). WebList(“FavoritePet”).Select “Dog” Wait(5) Browser(“MySite”).Page(“MyPage”). WebList(“FavoritePet”).Select “Monkey”

Tip 2: Wait Property Syntax: object.WaitProperty (PropertyName, PropertyValue, [TimeOut]) The “WaitProperty” pauses execution for the specified time then checks to see if the expected object property condition is met. The execution then continues or fails based on expectations. Usage: “WaitProperty” is smarter than “Wait” in that it can be linked to an operation and will fail if certain conditions are not met and not just a certain time frame. This is used to check not only that an object is available but that its property is also correct. The length of time used to search for the object can be specified, but if none is specified it will wait the time specified in the global settings. Example: 1

If .WaitProperty(“Enabled”, True,

August 2013

TestKIT Tip 5000) = False Then 2 Reporter.ReportEvent 1, “Calendar”, “Object Disabled” 3 Else 4 .Click 95, 100 5 End If 6 7 If .WaitProperty(“Enabled”, True, 5000) = False Then 8 Reporter.ReportEvent 1, “Calendar”, “Object Disabled” 9 Else 10 .Click 95, 100 11 End If 12 13 If .WaitProperty(“Enabled”, True, 5000) = False Then 14 Reporter.ReportEvent 1, “Calendar”, “Object Disabled” 15 Else 16 .Click 95, 100 17 End If 18 19 If Browser(“MySiter”).Page(“MyPage”).WebEdit(“MyPet”). WaitProperty(disabled”, 0) Then 20 Browser(“MySite”).Page(“MyPage”).WebEdit(“MyPet”). Set “Pet” 21 End If 22 23 If .WaitProperty(“Enabled”, True, 5000) = False Then 24 Reporter.ReportEvent 1, “Calendar”, “Object Disabled” 25 Else 26 .Click 95, 100 27 End If

Tip 4: Sync Method Syntax: object.Sync [Milliseconds] The “Sync Method” is used to wait for the object before performing the next operation. But it will stop and fail if the global timeout is met and the object is not found. Usage: The “Sync Method” is used for synchronizing Web Browsers, Web Pages, Oracle Applications, SAP and Terminal Emulator applications and not specific properties as “WaitProperty” does. Example: 1 2 3 4

Browser(“MySite”).Page(“MyPage”).Sync Browser(“Mercury Tours”).Page(“Mercury Tours”).Sync Browser(“Mercury Tours”).Page(“Mercury Tours”).Sync Browser(“Mercury Tours”).Page(“Mercury Tours”).Sync

Tip 5: Synchronization Points If you do not want to manually code a synchronization method you can take advantage of the QTP built in functionality. Usage: To have QTP automatically add your synchronization points simply display the page or object you want to synchronize with a click:

Tip 3: Exist Property Syntax: VirtualButton.Exist([Timeout]) The “Exist” method will wait a specified time and check to see that an object exists. The length of time used to search for the object can be specified, but if none is specified it will wait the time specified in the global settings.

Insert | Step | Synchronization Point This will create the code for you. The length of time used to search for the object can be specified, if none is specified it will wait the time specified in the global settings.

Usage: “Exist” can be combined with the “Wait” statement to loop until a certain object exists. 1 2 3 4 5 6 7 8 9 10

y=Window(“MyPage”).Dialog(“Welcome Home”).Exist counter=1 While y=0 Wait (5) y=Window(“MyPage”).Dialog(“Wlecome Home”).Exist counter=counter+1 If counter=10 then y=1 End if Wend

2 3 4 5

if Browser(“MySite”).page(“MyPage”).webbutton(“Feline”). Exists then Msgbox ”Cat” Else Msgbox ”Dog” End if

August 2013

While most of the examples allow you to customize the wait time, if no wait time is specified QTP will use its global system value located in: File | Settings | Run Object synchronization Timeout You can change the wait time depending on your specific needs, to short will create many failures and to long will extend the test execution time. Usage: Keep in mind that this is the maximum wait time and if the object if found before the time expires the test will continue and not wait the entire time.

Example: 1

Tip 6: Increasing Tool Default Synchronization Time

Wait! There’s More!

Read more about HP Functional Tester (QTP) on ATI’s QTP Blog: http://www.automatedtestinginstitute.com

www.automatedtestinginstitute.com

Automated Software Testing Magazine

Open Sourcery

Automation Just Got A Little Fuzzier! Welcome the Newest Open Source (Non Evil) Fuzzing Tool

existing data or generate new data

randomly

3. Record the state of the input data (in the event of a failure, this will ensure the data causing the error is known) 4. Deliver the data to the system under test (SUT) 5. Monitor the system for failures

Weâ&#x20AC;&#x2122;ve come to think of a Minion as an evil side kick, but it now has a new definition. Minion is also the name of the newest automated fuzz testing tool.

uzz Testing (aka Fuzzing) is a testing approach that relies on the concepts of negative testing, but typically approaches the creation of negative test cases in a random manner. It delivers randomly sequenced and/or structurally bad data to a system to see if failures occur. Fuzzing may be done on various areas within a 10 Automated Software Testing Magazine

software system including files, network protocols, API calls, Graphical User Interface (GUI), and more. The Fuzzing process may be similar to the following: 1. Identify types and sources of inputs 2. Randomly

cycle

through

www.automatedtestinginstitute.com

This process lends itself perfectly to automation, because an automated tool can efficiently generate large quantities of data, and an oracle can be used to effectively identify system failures that occur. Automation may also be ideal simply because it often allows for the creation of random (or at least semi random) data combinations and permutations that a human may not even conceive of, which, again, is the point of Fuzzing. Recently, the world of automated fuzzing tools became a little more crowded, with an announcement by Mozilla regarding their latest addition to the tool group. Mozilla, a non-profit organization that is perhaps best known for its Firefox open source web browser, has announced the launch of itâ&#x20AC;&#x2122;s security automated testing platform. This platform, known as Minion, has been under development for a year but has August 2013

Open Sourcery

https://wiki.mozilla.org/Security/Projects/Minion

results. The Front End is a web application that provides the ability for users to perform configuration tasks necessary for setting up and reviewing results of Minion scans. The Plugins Component contains automation scripts that handle the key security tool features such as tool configuration, scan initiation, and results collection. Skipfish is one of the key plugins maintained within Minion’s Plugins Component, and it is the platforms web fuzzer.

Recently, the world of automated fuzzing tools became a little more crowded, with an announcement by Mozilla regarding their latest addition to the tool group. just recently been made available to the public. According to the Mozilla Wiki, “Version 0.3 release of Minion allows Development, QA, and Security team members to perform automated web security scans with a set of tools, and reexecute those scans as needed.” August 2013

The Minion architecture is composed of three major components: the Task Engine, the Front End and the Plugin Component. The Task Engine provides an API for managing and configuring collections of plugins, users, sites and services, and execution

www.automatedtestinginstitute.com

Wait! There’s More!

Learn more at: https://wiki.mozilla.org/Security/ Projects/Minion

Automated Software Testing Magazine

By Michael punsky

Continuo

Load T

12 Automated Software Testing Magazine

www.automatedtestinginstitute.com

August 2013

ous

Testing Agility Applied to Performance Testing

August 2013

www.automatedtestinginstitute.com

Automated Software Testing Magazine

onsider the amount of time spent on finding and fixing problems in production. Even with the most comprehensive functional testing process, you still discover elusive bugs under the load of thousands of users going live simultaneously. Early testing and rapid feedback, as well as effective communication, are cornerstones of the Agile approach to software development. In many applications, performance is a critical factor for success. This is certainly the case with pretty much any serious Web or Cloud application. The need to accelerate delivery and improve user experience also leads many development teams to embrace Agile as the development process of choice. While performance is crucial for application success, what we often see in the industry is that in absence of additional resources during development, performance testing gets pushed aside until the very end of the project. Historically, and certainly in waterfall types of setups, performance would be scrutinized at the end, with the expectation that only minor tweaks and tune-ups are needed to meet the performance requirements. Lastminute changes, bugs that prevent test execution, infrastructure limitations, or lack of dedicated testing resources all contribute to these flawed practices. In the meantime, application delivery took a quantum leap towards flexible, redundant production systems – often hosted in the Cloud with full support of virtualization. This additional flexibility comes with added complexity, and performance issues detected late are likely not to be small tune-ups, but may lead to significant contract renegotiations, if not to re-architecture of the application.

Four Best Practices for Agile Load Testing First, choose a tool for test automation One of the main goals of Agile is to allow you to effectively manage change in 14 Automated Software Testing Magazine

both external and software functionality, and to accelerate the development of quality software. Part of the challenge with managing change is providing quick feedback on when the change in software happens, and the impact of the change. One of the important organizational points of Agile is the concept of a team producing a usable version. This approach should put Dev, QA, and tech writers on the same page, and in the case of performance testing, it should designate a resource who will be responsible for the testing. The other aspect is that all stakeholders, including those on the business side, need to be tightly involved. The concept of performance testing in Agile is different from “test early and often” because that concept still refers to waterfall. In the waterfall terminology, “early” means “before the software is transitioned to QA.” With Agile, you test all the time. Functional testers define functional tests to match the sprint functionality. Functional testing is an opportunity to introduce performance testing scenarios. Here are four handy suggestions for Agile Load Testing: 1. Test Incrementally 2. Establish Performance Goals 3. Reuse Functional Tests 4. Include Performance Tests in the Build Process

Test Incrementally Incremental testing starts at the lowest functional levels. Test a small function, and then build from there. If all of the individual components function and perform correctly, chances are they will work well together and this knowledge will help you analyze and debug performance bottlenecks when you test on the system level later on. Look at performance testing in Agile as testing of different performance layers that can be approached in an onion-like fashion in the rhythm of the sprints. In www.automatedtestinginstitute.com

order to achieve flexibility, you also need to plan your performance testing efforts and break a bigger challenge into smaller tasks. As a comparison, it is difficult and expensive to solve a 12 man-year problem at once, but one can solve 12 one man-year problems in sequence. The same approach can be applied to signing off the performance of an application. Later in the document you can learn some concrete methods to plan for incremental performance testing.

Establish Performance Goals Create performance tests to work alongside your functional tests, and to incorporate the performance test into your build process. Start by establishing performance goals when writing user stories. For example, let’s say you’re creating an online store. You want users be able to add an item to their carts, and want this process to take no more than half a second. The first thing you should do is create your performance test alongside your functional test to allow you to measure performance and continuously test it.

Reuse Functional Tests Always check if you can reuse existing or future functional tests as performance tests. It may be possible to incorporate timers into an automated test to give you a feel for how the application is responding. Review your functional tests with a designated performance expert and make the decision. This can be one of the key anchor points for Agile performance testing. This also drives development of work scenarios. These can later be used as building blocks for the final load tests.

Include Performance Tests in the build Process Finally,

you

should

include

your

August 2013

Continuous Load performance tests in the build process so that each build goes through some level of performance testing. By doing this, it’s going to be easier to determine what caused the slowdown in your application’s performance. Also, if you need to roll back to an earlier build, you know how well that particular build performs.

Considerations/ Obstacles on the path to Agile load testing Agile performance testing is a strong value proposition but, there are some significant obstacles on the way. Here are a couple of suggestions to keep in mind when considering Agile performance testing.

Is Your Test (staging) Environment an Accurate Representation of Production? There are some things that are hard to do until later in the delivery cycle. Stress and high-capacity load testing is an example. If you are performing a test at the very high end where you’re finding out how many users can use the application, how quickly they can ramp up on the system, and how many errors are generated when you are at your peak that is something that you want to save until the end. The staging environment may not support that level of capacity. You will need to take that into consideration when you are performing your tests. You may need to perform some of these tests in your production environment. One of the things that is important to consider early on is whether it is feasible to test the functionality of all or some of the application components before you perform an integrated system test. Often, your staging environment and production environment differ significantly in the amount of resources that are they are allocated. This will force

August 2013

Performance Planning Tips

• Put performance tasks on a SCRUM board • Make fixing functional bugs that block performance test a high priority

• Do • • • • • • •

not delay fixing performance issues identified by

tests

Retest after fixes are applied to know if they worked You cannot guess where the next bottleneck is Don’t pre-optimize before testing Understand the production environment Analyze logs Know when performance is good enough Apply low-level load to the system under test during manual testing

• Use load test scripts to find functional and stability bugs, memory leaks, reproduce intermittent problems you to perform those tests in production environment. Also, if you’re performing tests in production, you may need to test from outside your firewall, coming in from the Cloud. To allow for higher velocity of your performance testing efforts, identify tests that can be done inside your firewall in order to eliminate all of the other noise that might come into play during your test. If you do not have the available resources to make your staging environment an exact replica of Production, at a minimum, have staging represent a scaled-down version of Production.

Is Your Test (staging) Environment an Accurate Representation of Production?

up on the system, and how many errors are generated when you are at your peak that is something that you want to save until the end. The staging environment may not support that level of capacity. You will need to take that into consideration when you are performing your tests. You may need to perform some of these tests in your production environment. One of the things that is important to consider early on is whether it is feasible to test the functionality of all or some of the application components before you perform an integrated system test.

There are some things that are hard to do until later in the delivery cycle.

Often, your staging environment and production environment differ significantly in the amount of resources that are they are allocated. This will force you to perform those tests in production environment. Also, if you’re performing

Stress and high-capacity load testing

tests in production, you may need to

is an example. If you are performing a test at the very high end where you’re finding out how many users can use the application, how quickly they can ramp

test from outside your firewall, coming in from the Cloud. To allow for higher velocity of your performance testing efforts, identify tests that can be done.

www.automatedtestinginstitute.com

Automated Software Testing Magazine

Crowdamation Crowdsourced Test Automation

will offer the flexibility to

use a tool of choice (open source and

commercial),

have

teams

operate out of different locations, address the challenges of different platforms introduced by mobile and other technologies, all while still maintaining and building a cohesive, standards-driven automated test implementation that is meant to last.

“It’s OK To Follow the Crowd” Inquire at contact@automatedtestinginstitute.com

“It’s OK To Follow the Crowd” 16 Automated Software Testing Magazine

www.automatedtestinginstitute.com

August 2013

w w w. a u tom atedtestinginstitute.com

ATI Local Chapter Program Enhance the awareness of test automation as a discipline that, like other disciplines, requires continuous education and the attainment of a standard set of skills

Help provide comprehensive, yet readily available resources that will aid people in becoming more knowledgeable and equipped to handle tasks related to testing and test automation

Offer training and events for participation by people in specific areas around the world

ATIâ&#x20AC;&#x2122;s Local Chapter Program is established to help better facilitate the grassroots, global discussion around test automation. In addition, the chapter program seeks to provide a local based from which the needs of automation practitioners may be met.

Start a Local Chapter Today Email contact(at)automatedtestinginstitute.com to learn more

ATI - Meeting Local Needs In Test Automation August 2013 www.automatedtestinginstitute.com

Automated Software Testing Magazine

inside your firewall in order to eliminate all of the o t h e r noise that

might come into play during your test. If you do not have the available resources to make your staging environment an exact replica of Production, at a minimum, have staging represent a scaled-down version of Production.

Are You Testing With Real Data, or Will You Create Dummy Data? Another important question to consider early in order to correctly scope the effort is: Are you going to use real data? Are you going to create dummy data and if yes, do you need to create it? In a lot of environments out in the industry, especially in the medical world, using live user data for testing may be difficult. This factor may also determine how data can be accessed, and if you can load from outside of the corporate network. If you do have access to real data, don’t pass this opportunity. After tests are done, make sure you can set your database back to its pre-test condition. Since your database is going to cache a lot of information you want to use data that’s as realistic as possible. 18 Automated Software Testing Magazine

Furthermore, you need to consider altering data between t e s t cases. I f

you r u n a load test that is logging-in as the same user every time and it’s filling in a form field, or a set of form fields the same way every time, and doing the same inserts into a database, you’re not really testing your system at all. There may be some data that will not be caught or will cause an error. And you might not catch that if you just use homogenized data. One of the situations you may need to work around is having a test running for a long period of time and using large numbers of virtual users and your data set is limited. The best scenario, of course, would be to have a very large set of data and not have anything ever be repeated. Depending on the test you’re running, however, having data you need and having it repeated at some point, might be a possibility. If you’re using live data, you may not have the ability to scale it up as large as you might like.

Who Needs To Be Involved During the Testing, and In What Capacity? Clearly, a dedicated performance engineer www.automatedtestinginstitute.com

should definitely be involved from the beginning, and be an integral part of the agile team. When tests are run, the performance testing results should be available to all of your team members – the Database Administrator (DBA), the network specialist, the developers. Different people on the team may provide different perspectives on the observed performance problems or any errors that come up. A DBA is always going to see a performance problem as being related to the database. A network person might tend to think that bandwidth is the problem. Your developers might think that it was the way that something was structured in the code, and it may need to be refactored. This becomes even more important when you run your load tests at the end of the whole application, because sometimes it takes a number of different sets of eyes to provide the clarity you need to find the actual problems.

What Criteria Will Be Used to Determine Success of the Test? From the very beginning, when you want to run a load test, you need to determine what your criteria for success. Whether you are looking for an absolute response time, or a relative change from a previous test, you need to define the criteria and have a clear expectation before you start testing.

Typical Pitfall: Overestimating the # of Virtual Users One thing that you can do if you’re running your larger tests is look at your analytics and determine historical performance. If it’s a completely new application, then the rule of thumb should be to start small and build from there. If you have tools that can generate load with a large number of users like SmartBear’s LoadComplete it may be August 2013

Continuous Load results can be obtained with a moderate number of virtual users, it makes sense to begin with running many moderate tests.

Small Does Not Refer Only To Virtual Users In addition to moderating the number of virtual users, it also helps to run single scenarios (one script as opposed to multiple scripts) during the iterative testing. From there, build your way up to multiple scenarios. The smaller your test, the greater your insight into where your errors and bottlenecks are coming from.

Inside or Outside of the Firewall

Figure 1: Workspace Custom Ramp-up Shape tempting to start with a large number of users, but it is important to remember that you can get a lot of good results at very low levels of load. By starting small, you’re going to be able to correct a lot of problems before you do your heavy testing. And this is going to ensure better results when you to do your heavy testing. Not following this best practice and falling into the trap of overestimating the number of virtual users early on will likely force you to repeat that large test numerous times in order to get the results you’re looking for.

Initially, consider looking at one virtual user executing one scenario, running from your local workstation so that you’re able to get your numbers. This way, you’re able to tweak and tune. Additionally, you know that your component is working to its best capabilities, but that it’s not putting a lot of mud into your test. You can get many good results at very low levels of load. By starting small, you can correct a lot of problems before you do your heavy testing, ensuring better results.

Plan

Define Your Workload Start out doing small, incremental tests. Later, combine these into a more realistic whole. For example, we have a test set up for an e-commerce site. And here, you’ll see that this is coming in from the Cloud. Also, it has two scenarios running at the same time. August 2013

One is a group of users that is only browsing your site. The other group of users is actually purchasing something and checking out. This is a great way to have a realistic load test of your system. However, this is something to save for the very end when you are hitting production.

Start Small There is a tendency to start out throwing a large number of virtual users against the site being tested. Since actionable

www.automatedtestinginstitute.com

Some load tests are generated inside the firewall with software, and others are generated outside the firewall by a service. It seems that most people champion one over the other. There are valid reasons for both types of testing. Internally run software is best suited for iterative testing. Load origin is key here. There have been a lot of people band-standing about having load come from the Cloud versus having load come from real browsers. Either way, there is a need and a use case for all types of load testing. For small, iterative load testing, the best tool to use is software that you can run inside your firewall. You’re paying one licensing fee, and you can run as many tests as you want for no additional charge. Run as many tests as possible to get the best results possible. Tweak and tune at every step. By running the tests from within the firewall, you won’t get any of the noise or X-factors of additional users coming in from the outside or any of the other pieces such as your providers and external influences. Once your application is tuned as well as it can be,

Automated Software Testing Magazine

present during testing (whether inperson, remote or on-call). Design and follow a process when performing iterative tests, and compare results of various iterations.

The Importance of Parameterization When defining workload for a test, the importance of using a broad spectrum of data cannot be overstated. Many factors can only be properly tested by using either actual data, or at least a very good representation of it.

Figure 2: Task Chart run something from outside the firewall, generally from the Cloud. You can have users from different Cloud regions. You can scale really high, and, you could find out how your application performs under real-world conditions. Test in real browsers as well as staged tests to get very accurate results. How does the page appear? How is the JavaScript loading? How is the CSS looking under this load condition? The key here is to use the right tool for the task at hand. If you’re doing unit testing, application server tuning, smoke testing, and bottleneck analysis, these are all good candidates for inside the firewall. You can do some bottleneck analysis from outside the firewall, as well as heavy load testing, stress testing, and capacity testing all from outside the firewall.

Some Examples Flexibility: To be flexible is to rapidly respond to change. In an Agile environment, things can change quickly as new requirements come in or are modified. As this happens, performance can be left by the wayside. However, if you’re testing continuously, you will be more likely to catch problems sooner and make the needed corrections.

20 Automated Software Testing Magazine

Data Preparation It’s probably easier to prepare relevant data gradually as the functional scenarios become available. Smaller scale, single-user performance tests are key. These are not scaled up, but provide some metrics that are. If you test at the smallest increments and you use a small amount of load, you still get valuable information as you perform your iterations. Let’s say you have five users, and your performance is half a second. If the same five users produce a response that is double that after some changes, you’ve got a problem. With Agile testing, however, you can always see the delta between where you were and where you’re going to be. If the success criteria for a sprint include delivery of a working software version, performance and capacity should be a sign-off metric. If there’s bootstrapping of functionality in multiple sprints, then you need to, at least, look at these items as challenges. The main goal of Agile was to set up an approach to more effectively manage change in both external and software functionality.

Test It is important to have key personnel

www.automatedtestinginstitute.com

Another factor here is parameterization. Many factors can only be properly tested if you’re using actual or, at least, realistic data. Some examples of things that you may want to parameterize are user IDs and passwords, different search terms. It’s possible certain search terms will take longer to return than others. You want to test a wide variety of those search terms and in different quantities. By testing this way, you may uncover some bugs in the application as well. Additionally, any time a form is going to be inserting data, test varied information on the form to maximize your results.

Analyze Analyze Generated Test Reports A particular art in testing is the ability to analyze your generated test reports. You want to be able to look at your test reports, understand everything being presented. Come at it from different angles as well, and have the business owners look at your reports. Also, always be asking network people, DBAs, developers, and anyone else involved to provide their own perspectives. Multi-disciplinary Approach analysis from a multidisciplinary perspective, considering the following disciplines: • •

Developer DBA

August 2013

Continuous Load • •

Network Admin Business Owner

Determine Bottlenecks or Areas of Concern Missing or non-optimal indexes can really slow down any application. With non-optimal code loops, it’s very easy for the developers to write code that does things in a brute force sort of way. The paradigm that’s being used for a particular loop or a section of code might not be as optimized as possible. Finding these things and correcting them can help to reduce bottlenecks.

you add virtual users, and see it flat line, then chances are something is limiting your amount of bandwidth. This could be a piece of hardware acting up, or your provider. Again, testing frequently is the best way to catch these issues. Third-party Calls These days, third-party calls and content are all over web pages. Whether it’s social media, advertising, or analytics, your company’s personal content might be 20 percent of your home page. 80 percent of that could likely be coming from other sources.

C ontent C oday T

ontribute

Another thing that is very key in your applications is caching. There are so many different layers of caching that can be involved in your application. Part of your application can be 100 times faster if it’s cached than if it’s not cached, depending on what it is and what level of caching is implemented.

Unfortunately for you, a customer coming to your web site thinks of everything that’s on your site as being you. So if one of your third parties is not performing well, it reflects directly on you.

Community Comments Box

You can cache pages of your site. You can cache queries. You can cache your results to RAM or to disk.

Fix

Announcements & Blog Posts

When remediating these issues, you’ll be adding or rebuilding your indexes, and adding or optimizing your cache. You’ll be upgrading hardware, adding new hardware, optimizing network components and settings. You’ll work with your third parties to optimize any of those calls. You’re going to need to optimize code loops, refactor code.

Automation Events

And then there’s the question of what’s the duration that you’re going to be caching for. Is this information that can be cached for a day? Is it information that can be cached for five minutes? There are cases where you need pages and queries to be as current as possible. However, if there is a two-minute delay where it can be cached, you could actually gain a lot of performance. Hardware is usually seen as one of the first things people want to look at when it comes time for performance. Ease your mind – it’s usually not the hardware. If you get everything else working fine, your current hardware might be completely adequate for the task. Sometimes, hardware can be the problem. Testing frequently and analyzing results can help establish where problems lie. Networks If while running a test from outside your firewall, you notice load is building as August 2013

All of these things take time to remediate. To cut down on that time, test constantly and provide your corrections diligently. The sooner you begin your remediation, the better. By using continuous testing, there will be fewer costly surprises at inopportune times. One thing that you see all the time now is a development group that puts together an application, and releases it without load testing. Much to their dismay, they find out the amount of traffic on that site is going to take it down once it’s up. Include performance testing continuously, and there will be no surprises at the end.

www.automatedtestinginstitute.com

As a registered user you can submit content directly to the site, providing you with content control and the ability to network with like minded individuals.

Learn more today at http//www.about.automatedtestinginstitute.com

Automated Software Testing Magazine

Selecting the Right Tool The Pyramid Approach to Test Tool Selection

By Bernd Beersma

22 Automated Software Testing Magazine

www.automatedtestinginstitute.com

August 2013

“

A test tool is one of the best tools

to help us throughout the test cycle because test tools can support almost all activities we do in testing: test management; test design; etc. But if we do a Google search, we get hundreds of thousands of hits for test tools. So how do you as a tester, test manager or test specialist know how to select the best

“

tool for your day-to-day business – one that fits your purpose?

electing a tool, that eventually fails to meet your requirements or expectations, can lead to:

1. Long List

•

Poor automation

4. Pilot

•

Frustration

•

Lack of commitment

Phase 1: Long List

•

Bad business case

•

Shelfware

The outcome of the long-list phase is, as the name states, a long list of test tools. But getting there requires a few steps:

The tool selection pyramid is a structured selection approach that will help to avoid these pitfalls.

The Tool Selection Pyramid The Tool Selection Pyramid is composed of four different phases:

August 2013

www.automatedtestinginstitute.com

2. Short List 3. Proof of Concept

Identify your stakeholders.

If we want to implement a test tool in our organizations, this will influence the test team and other departments, IT and the maintenance departments, for example. We want to identify all those who are involved 2. Define measurable, feasible goals for Test Automation (TA). Automated Software Testing Magazine

Be sure to involve all the stakeholders. This again creates commitment, but the other departments can also add goals of their own. For instance, business wants to decrease the time to market when you select a tool for automated regression testing.

regression test or need to support a defect-management process. These are on a high level, and mind mapping is a good way to gather these requirements. This approach helps to get the high-level requirements for the tool in a fast and efficient way.

Create a business case for TA.

We need to set up a business case to be sure the project has enough support in the organization, and can be brought to life.

Gather general tool information.

After defining the general requirements, explore the tool market. Look at toolrelated websites and magazines, and events.

general requirements. In this phase, we need to go a little deeper into these requirements. For example, we needed a tool for automated regression testing. Now, with a mind map, we find the need for tools to support the use of external data, or tools to support cross-browser testing. 2. Gather the detailed requirements. Add them to a scoring card and determine weight and priority. The scoring card is the basis for the next step, the Request

After drafting the requirements we invite the vendor(s) to set up a plan for the pilot together with members of the project team. This needs to be a solid plan, with fixed timelines and dates, because this pilot phase needs to have an end, after which the project team can start the evaluation. 4.

Set up a project team.

The next thing we need to do is set up a project team for the tool selection project. Again, be sure that people from the different identified departments are on this project team, although the majority of the team can be from the test department. Someone from IT and maintenance â&#x20AC;&#x201C; but also a developer â&#x20AC;&#x201C; can be of real value to your team not only to create the support within the organization, but they need to help identifying requirements that are not test-related but are an essential part of the selection process. 5.

Define general requirements.

In this step, focus on defining general requirements like the need to automate 24 Automated Software Testing Magazine

Create a long list.

This list contains the tools and vendors we think should enter the next phase in our tool selection pyramid, the short-list phase. Before entering the next phase, there is always a moment of consideration about whether or not we are ready to enter the next phase. This go/no-go moment gives us the possibility to restart the previous phase or start the selection process over.

Phase 2: Shor t List This phase also consists of several steps: 1.

Define detailed requirements.

In the long-list phase we came up with www.automatedtestinginstitute.com

for Information (RFI). 3. An RFI goes to all the vendors on the long list. Sometimes this is not possible if there is no vendor or party to whom to send the RFI. But if you still want to score, you can always see if you know an independent consultant who has experience with that tool to help you to fill out the form, or find a third party experienced with the tool and ask them to fill in the RFI. In the worst case, the project team has to fill it out. After getting back the RFI, you need to determine overall score. As you can see, this is only an example, but already 20 different main categories are identified here. So this shows tool

August 2013

Selection Pyramid selection is not that easy.

time. Be sure that all systems are ready when starting with the next step in this phase, the actual execution of the POC.

4. A demo. Invite at max the top five scored tools to do the demo. The demo is like first contact, and you never get a second chance to make a first impression. Be sure to invite all the project members, stakeholders to let them see the demo. They can help you to decide afterwards which tools to put on your shortlist. Preferably three vendors will end up on the short list and this completes our second phase. Again, before entering the next phase, there always is a moment of consideration if we are really ready to enter the next phase. This go/no-go moment will always give us the possibility to restart the previous phase or even start all over again with the selection process.

Phase 3: Proof of Concept Now we entered our Proof of Concept or POC phase, during which several vendors or consultants are invited to do a proof of concept. This is like a test drive for the tool, to see how the tool performs in our situation and environment, and to see if the tool meets our requirements. Again, this phase is completed by executing several steps. First of all, we draft the requirements for the successful completion of the POC. These requirements are the same for all participants of course, to create a fair competition. The tool must support all technologies that are part of a certain business process chain. After completion, the requirements drafted will be also part of the acceptance criteria for the POC. Second we invite the vendors to do the POC, but this is a combined effort between us and the vendor. We need to be clear on what we expect of the vendor. This not only counts for the availability of the Software Under Test (SUT) but also of internal resources who will be executing the POC with the vendor. We need to setup a plan so we are sure the POC will be executed and finished on August 2013

Keep in mind that user accounts have to be set up, as does a test date, etc. If you are sure that everything is good to go, start executing the POC. This is also a combined effort. The more you involve the team by the selection process, the better the results of the selection will be. After you and the vendors finished all the POCs, an evaluation of the POCs needs to be conducted. We will do a session with the project team and will see if the POC meets our requirements. The goal of this step is to choose the best tool, preferably one tool; you have selected two tools that meet all the requirements. If this is the case, no worries, we will come up with the definitive selection in the pilot phase. But, we haven’t completed this phase yet, the next thing to do is to send a Request for Proposal. Even though I strongly believe that money is a bad driver, we still want to know what licenses, trainings, maintenance cost. We need this input to validate our business case. In the case of an open-source tool, the same procedure as for an RFI is started. Now with all this information gathered we are ready to update the business case and confirm we are still on track with our tool selection. This completes this phase and we will be entering the pilot phase, but not before we do a go/no-go, but again, we are always allowed to restart the phase or even the whole selection project.

Phase 4: The Pilot During this phase, one or two tools will be used in the pilot project for a longer period of time. It’s best to setup a separate project for this phase in a test environment. Like in all of the other phases, we have to follow a few steps to complete this phase. But, after completing these final steps we have selected the best tool to help us with our testing, so completing this phase is worth the wait.

www.automatedtestinginstitute.com

Again we start with the requirements for completion of this phase. We want to evaluate the outcome of course against our acceptance requirements. An example of a pilot requirement can be, “testers must be able to automate the regression test by themselves.” After drafting the requirements we invite the vendor(s) to set up a plan for the pilot together with members of the project team. This needs to be a solid plan, with fixed timelines and dates, because this pilot phase needs to have an end, after which the project team can start the evaluation. After setting up this plan, we start the actual pilot with a kick off, so all participants in the pilot know what is expected of them, but also to create a feeling of collaboration between the involved parties. No we are ready to actually start with the execution of the pilot. This is a combined effort with shared responsibility between the vendor and the project team. The goal is to prove that the tool can be implemented within the project and the organization. Now that we have executed the pilot, final step is evaluation of the pilot. Gather the project team and see if the selected tool meets the requirements and acceptance criteria. And if all requirements are met and all systems are go, we have completed our selection project and the best tool for the job is ready to be implemented. That leaves me with some final words of advice. Whenever you start selecting a tool, be sure to create awareness within your organization. Make it a combined effort and a shared responsibility between you and all the possible tool vendors or consultants, and always keep in mind that if you don’t feel comfortable after completing a phase, start again.

Wait! There’s More!

Learn more at TestKIT OnDemand: http://www.ondemand. testkitconference.com

Automated Software Testing Magazine

Training Thatâ&#x20AC;&#x2122;s Process Focused, Yet Hands On

Software Test Automation Training www . training . automatedtestinginstitute . com

Public Courses

Software Test Automation Foundations Automated Test Development & Scripting Designing an Automated Test Framework Advanced Automated Test Framework Development Mobile Application Testing & Tools Virtual Courses Automated Test Development & Scripting Designing an Automated Test Framework Advanced Automated Test Framework Development Mobile Application Testing & Tools Come participate in a set of test automation courses that address both fundamental and advanced concepts from a theoretical and hands on perspective. These courses focus on topics such as test scripting concepts, automated framework creation, ROI calculations and more. In addition, these courses may be used to prepare for the TABOK Certification exam.

Public and Virtual Training Available

26 Automated Software Testing Magazine

www.automatedtestinginstitute.com

August 2013

Crowdamation Crowdsourced Test Automation

will offer the flexibility to

use a tool of choice (open source and

commercial),

have

teams

“It’s OK To Follow the Crowd” Inquire at contact@automatedtestinginstitute.com

“It’s OK To Follow the Crowd” August 2013

www.automatedtestinginstitute.com

Automated Software Testing Magazine

The Testing Tsu Rethinking the Role of Automation

nprecedented

market dynamics challenge traditional thinking in many ways, including around test automation. What used to work may break down under new pressures. The key to succeeding in trying times is to be willing to reexamine and, if necessary, overturn established assumptions and experiment with new approaches. Learn how one of the largest providers of healthcare information solutions in 28 Automated Software Testing Magazine

www.automatedtestinginstitute.com

August 2013

unami by Rubina

â&#x20AC;&#x153;That Was Then...This is

Nowâ&#x20AC;?

A nsari

the U.S. leveraged the cataclysm of changes in their industry to rethink and revamp their test automation organization, architecture and deployment to achieve new levels of productivity and quality. Whether you are involved in healthcare or not, there are important lessons to be learned about reinventing your approaches to stay ahead of upheavals in any market. Their discoveries and decisions may surprise you and should certainly inspire you. August 2013

www.automatedtestinginstitute.com

Automated Software Testing Magazine

The Dynamics of Healthcare Quality The healthcare market is one of the fastest growing job sectors, powered by an aging population, advances in technology and expansion of government regulations and incentives. It is hard to drive down the street without encountering billboards advertising for this hospital or that doctor, or watch television without being encouraged to see our doctors about some new miracle drug (pay no attention to that steady stream of scary side effects), or read the news without being drawn into the healthcare reform debate. All of this activity portends a hot market for skills in every area of technology, and given the risk profile – both medical and financial - of most healthcare applications, the need for testing and test automation is even hotter.

And that’s not all. Providers and insurers are now going to be held accountable for outcomes. Not only must hospitals measure and publicly report their outcomes in many key areas, health insurers must ensure that at least 8085% of all premium dollars collected by insurance companies are spent on health care services and health care quality improvement. If insurance company administrative costs or profits are too high, they must provide rebates to consumers. If all of this is making your head spin, imagine the impact on healthcare providers and payers who must keep their businesses running while absorbing all of these new requirements, understanding their impact on interrelated systems, implementing the required changes and – of course – testing them all.

The more people use it, the more demand there is for new capabilities In particular, there are a number of initiatives that are driving extensive modifications to the massive systems that manage patient information and insurance. For example, new regulations will require heath plans to adopt electronic patient records and implement secure exchanges of health information. Also, ICD-9 is being replaced by ICD10 and ICD-10PS. These are coding taxonomies that classify diagnoses and related procedures that form the backbone of how insurers determine which medical costs are covered and for how much. The US version has some 68,000 diagnosis codes and is coupled with the ICD-10 PCS, a procedure code system that contains 76,000 codes. On top of all these changes, the format for electronic claims that structures the communications among providers and payers has also been revised. 30 Automated Software Testing Magazine

This testing tsunami forced TriZetto (www.trizetto.com), one of the largest providers of healthcare information solutions in the U.S. whose solutions touch half of all insured customers, to fundamentally reevaluate their approach to software engineering, testing and test automation. Their journey offers insights that apply to all industries but especially those in highly regulated, complex sectors where content and deadlines are often externally driven.

That Was Then Despite being geographically distributed, TriZetto had a unified testing organization that comprised all test resources across all products. This structure evolved to provide not only process improvement and consistency of metrics, but also to serve www.automatedtestinginstitute.com

as a counter-balance to the engineering organization; both QA and engineering reported up to the same vice president as equals. Managers were aligned with development centers and team leads were aligned with projects. Like many companies, TriZetto also had an Automation Center of Excellence (ACOE) that sought to centralize skills and expertise around test automation. With over a dozen different solution areas serving all aspects of healthcare providers, insurers and benefits administrators, they needed automation across a wide range of products, technologies and customers. The ACOE was responsible for selecting, installing and supporting the automation toolset and promulgating standards across the multiple groups. This organization resulted in a twotier test approach. Most QA resources were recruited from the industry, such as benefits administrators and other subject matter experts who were either past users of TriZetto solutions or experienced in the market domain. Their knowledge base was essential to designing effective tests. The number of combinations of product features, customer configurations and data was mind-bending, so the majority of the test effort usually came down to selecting or creating the precise set of conditions that would trigger cascading business rules and result in a specific outcome. For example, each insurer offers a set of plans and options to their customers that specify the business rules governing benefits. Within each plan, each individual participant’s own profile further affects coverage: dependent family members, their ages, pre-existing or excluded conditions, deductible balances and other factors have to be considered. Each claim filed by an individual is even further circumscribed by whether the provider was in or out of network, the diagnoses was covered and the procedure authorized, and the contractually negotiated cost and co-pay amounts. August 2013

Testing Tsunami The next time you receive an insurance reimbursement with an “Explanation of Benefits” (EOB), pause to reflect on the complexity that went into the end result. These valuable and knowledgeable QA resources, however, were unlikely to also have coding skills. Thus the second tier included test automation engineers who sought to identify and automate test case candidates for regression. But despite engaging with on and offshore outsourcing and consulting teams, the automation team could simply not keep up. The flood of changes meant the library of automated tests was under constant maintenance and there was not enough bandwidth to keep adding coverage. Eventually market dynamics began to stress this structure and approach. The escalating rate of change made it difficult to manage priorities among projects. Continually adding QA resources was not financially viable and automation could not make up the difference. It was time for a change.

Inflection Point Like most reorganizations, this one started at the top. A new CTO was recruited with a background in the fastpaced world of online solutions. After careful study, extensive internal and customer interviews and a professional assessment, he implemented a fundamental restructure. Product teams were vertically integrated with all functional areas reporting to a product manager, including QA. This created a scramble by the product managers to recruit the top QA resources for their teams and exposed the inadequacy of the automation engineering capability. There were simply not enough to go around and the overall QA headcount already exceeded industry norms. What to do? Should the product managers replace their subject matter experts with automation experts? The implications were profound. Did it make sense to trade test quality for August 2013

quantity? Many of the QA resources had years of experience not only in the industry but with the TriZetto products as well. The loss of this knowledge was a real risk. New engineers might be able to develop more automated tests, but if the tests lacked proper design would anything be gained? For the newer, smaller product teams with only one or two QA resources on their team, compromise was not an option. A decision had to be made.

to allow otherwise manual SME testers to create their own design steps using point and click drop-down lists, which in turn called the reusable components to automate the execution. “Now the SMEs can use a familiar tool to do something completely new: automate their own tests,” said Rubina. “Instead of replacing or deprecating these valued experts, we are enabling and supporting them to leverage their knowledge.”

Breakout Thinking

This breakout thinking completely changed the dynamics of test automation. Rubina was promoted to lead the Enterprise Automation and Agile Tools Center of Excellence for Trizetto where she now focuses on adding platforms and components so the product teams can focus on test design and automation while leading the charge in implementing Agile SDLC tools to the organization. Everyone is able to exploit the resources and skills they have and quality does not have to be sacrificed for quantity.

Rubina Ansari, one of the top QA leaders at TriZetto with extensive experience managing both manual and automation resources, wrestled with this question and came to the realization that it did not have to be either/or. After wide-ranging research and consultation, she concluded that by adopting a different framework architecture, she could enable the SMEs to do their own automation. The existing framework approach called for reusable components that were mapped to business functions within an application. Once these components were developed, theoretically the SMEs could use them. The problem was that developing the components was a substantial effort – over half a million lines of code costing $1 million for a single application – and these had to be maintained as the functionality changed. There was not enough time, money or resources to develop and maintain components for all the different products. Instead, Rubina chose to implement a new framework that was mapped to the application platform. Reusable components would be developed around the classes of objects and their behaviors. The key was that these components were reusable across all applications for the same platform, and they did not change with the functionality. Thus, developing a Web library addressed multiple applications with little or no additional coding. This radically reduced the amount of coding and all but eradicated the component maintenance. But the real power was unleashed when she modified the test management system

www.automatedtestinginstitute.com

This Is Now Only a short year after the journey began, TriZetto is running thousands of automated tests every day and the ACOE is continually expanding the scope, power and flexibility of the framework. New tools and platforms are being considered, such as testing for restful web services, and reusability is moving towards data as well. “The more people use it, the more demand there is for new capabilities,” Rubina noted. And of course success breeds new challenges: Now TriZetto implementation services teams and their customers are clamoring for the same benefits. And with all the changes headed TriZetto’s way, there will never be a lack of demand for more and better tests. In an industry as dynamic as healthcare – and many others, for that matter - there will always be trade-offs and tough decisions to make. But as TriZetto’s experience proves, sometimes you don’t have to choose: You can have it all. Automated Software Testing Magazine

TestKIT Conference 2014 Testing & Test Automation Conference

Save the Date! March 3 - 5, 2014 Crowne Plaza Hotel Arlington, VA

The KIT is Koming... Back www.testkitconference.com 32 Automated Software Testing Magazine

www.automatedtestinginstitute.com

August 2013

Are You Contributing Content Yet? The Automated Testing Institute relies heavily on the automated testing community in order to deliver up-to-date and relevant content. Thatâ&#x20AC;&#x2122;s why weâ&#x20AC;&#x2122;ve made it even easier for you to contribute content directly to the ATI Online Reference! Register and let your voice be heard today!

As a registered user you can submit content directly to the site, providing you with content control and the ability to network with like minded individuals.

Community Comments Box

Announcements & Blog Posts

Automation Events

August 2013

>> Community Comments Box - This comments box, available on the home page of the site, provides an opportunity for users to post micro comments in real time. >> Announcements & Blog Posts - If you have interesting tool announcements, or you have a concept that youâ&#x20AC;&#x2122;d like to blog about, submit a post directly to the ATI Online Reference today. At ATI, you have a community of individuals who would love to hear what you have to say. Your site profile will include a list of your submitted articles. >> Automation Events - Do you know about a cool automated testing meetup, webinar or conference? Let the rest of us know about it by posting it on the ATI site. Add the date, time and venue so people will know where to go and when to be there.

Learn more today at http//www.about.automatedtestinginstitute.com www.automatedtestinginstitute.com Automated Software Testing Magazine 33

34 Automated Software Testing Magazine

www.automatedtestinginstitute.com

August 2013

August 2013

www.automatedtestinginstitute.com

Automated Software Testing Magazine

I ‘B’Log To U

Latest From the B

Automation blogs are one of the greatest sourc automation information, so the Automated Tes decided to keep you up-to-date with some of th posts from around the web. Read below for so posts, and keep an eye out, because you never will be spotlighted. Blog Name: QTP (HP Functional Tester) Blog Post Date: May 29, 2013 Post Title: QTP Test Run Like Greased Lightning P3 Author: Captain Automation

Blog Name: Test This Blog Post Date: April 1, 2013 Post Title: Do Automated Checks Run Unattended? Author: Eric Jacobson

The QTP object repository is a database of object properties and values. Descriptive programming is a way to call object properties and values from the command line without the object repository. In general execution using descriptive programming is faster than execution using the object repository. A descriptive programming example entering the word “Testing” into the Google search box would look as follows...

Automators, be careful. If you tell too many stories about unattended check-suite runs, the non-automators just might start believing you. And guess what will happen if they start running your checks? You know that sound when Pac-Man dies, that’s what they’ll think of your automated checks. I remember hearing a QA Director attempt to encourage “test automation” by telling fantastical stories of his tester past...