2011 Manchester , UK 21 - 24 November 2011
Testing ‘Slow Flows’ Fast Automated End-2-End Testing Using Interrupts
2011
Dominic Maes Sr. Test Consultant
In an ‘online’ world, everything seems to be going faster and faster. Too often, we forget about ‘slower’ processes that need to be tested in our information services. Automated end-2-end testing on these flows, which use independent systems and timers, is not that difficult. Using a record and playback tool with a framework is all it takes. You’ll simply have to add an ‘action queue’ to your framework and you’re back on track for testing thousands of test cases.
Testing ‘Slow Flows’ Fast - Automated End-2-End testing using interrupts
Contents 2
Goal
2
The Challenge
2
The Quest for a Solution
3
A Short Evolution of Record - and Playback Tools
4
Setting up the Solution
5
Let’s Rock ’n Roll!
8
Drawbacks
9
The Summary
10
References
11
Biography
11
PAGE
1
Introduction
w w w. e u ro s t a rc o n f e re n c e s . c o m
Testing ‘Slow Flows’ Fast - Automated End-2-End testing using interrupts
Introduction We live in a world where instant fulfillment is a keyword when doing business. Everything seems to be going faster and faster in our ‘online’ world. Customers don’t want to wait for services and products anymore: ‘Wow, I need that iPhone right now!’, ‘I need those tickets for your concert tomorrow’, etc… Modern economics are based on these consumer principles. Too often, we forget about ‘slower’ processes that need to be automated in our information services. And let me be clear, ‘slower’ is still relatively fast. We are not talking days or months here, but minutes and hours.
PAGE
2
Test – automating those processes can be a bit tricky. Why even bother using test automation tools to complete a flow that can take several hours? Automated end2-end testing on these ‘slow flows’, that use independent systems and are perhaps sensitive to different timers, is not that difficult. Using a record and playback tool combined with some scripts is all it takes. You simply need some kind of scheduler and you’re back on track for testing thousands of test cases. By using an ‘action queue’ in your automated test framework, in combination with a scheduler, you can easily adapt your automated scripts to perform many actions while the tool is waiting for processing to finish. This kind of ‘interrupt processing’ is a solution for every application flow that can take up many hours in total before having finished and where interaction between independent systems is reflected in a database. This framework can even check timers and delays of processes without much hassle. To me, this type of framework has proven its reliability and usability. It has become a standard for automated testing of high volumes in ‘slow processing’.
Too often these kind of ‘slow processes’ are considered difficult and impossible to test and are often only tested in a system testing phase. With this approach, these often critical processes can be tested end-2-end which will give you the extra security, by reducing the risks involved in using multiple independent systems with time critical processing. TWEETABLE
Too often these kind of ‘slow processes’ are considered difficult and impossible to test and are often only tested in a system testing phase.
Goal This paper is the representation of my experience in my quest in finding a ‘fit for purpose’ framework in automated testing and dealing with time-related issues while testing. It describes a short evolution of record and playback tools and the final solution I implemented.
The Challenge Some 10 years ago, I was working in a company where I got a special challenge to automate test cases with a record and playback tool so that we could execute 10,000 test cases a day. As I had already quite some experience in developing test automation frameworks, it was not a big challenge for me. Creating a test data file, setting up the framework to repeat input and check database contents would do the trick. The pitfall came in the process-flow to test: as the real operational flows unfolded, I discovered that processing time for a single test case could vary strongly according to the
Testing ‘Slow Flows’ Fast - Automated End-2-End testing using interrupts
context in which the test case was executed. Due to the great number of communications with different subsystems, execution time of a test case could be between 3 minutes and 8 hours. In a worst case scenario, you could end up with only 3 test case executions per day, hardly worth automating. But the project manager was pretty clear: aim was to execute 10,000 test cases a day. To protect the privacy of my client, I changed the context and the business, but kept the challenges in the examples that I have used in this paper. TWEETABLE
A first question you should always ask in any project is: “Do I really need test automation in my project?”
PAGE
3
The Quest for a Solution Why use automated testing? A first question you should always ask in any project you’re testing is: ‘Do I really need test automation for my project?’ The answer can go either way. Initially, test automation tools were seen as a way to speed up the test process, once testing got on the critical path of the project. In a world of rapid application development and ‘releases that should have launched before the project started’, a tool that shortens the project lifecycle would come very handy. As there are many different testing tools, taking care of many aspects of testing (i.e. test management, test data creation, comparators, record- and playback, etc...) we will only focus on test execution tools. If you look closer to the testing work inside a project, you cannot deny the advantages
that come with the use of a test automation tool: a) A tool is less error prone than a human tester: i.e. it will do exactly what you have told it to do, time after time, with a great accuracy.
b) It brings more work enjoyment: Executing test cases over and over again under time-pressure is most of the time a real tedious job. A tool will never complain.
c) As modern test execution tools have possibilities to ‘speed up’ test case execution, a higher volume of test cases can be processed in the same timeframe (compared to human test execution).
d) By using a tool, you will have the opportunity to monitor more aspects of your test execution, which were otherwise unable to follow (manually). E.g. in stress testing, CPU and memory usage can be closely monitored while executing test cases.
Unfortunately, test automation has pitfalls, waiting to counter your test automation efforts: a) ‘If you automate chaos, you’ll only get more chaos’. Make sure you have a structured way of organizing your test process before you start automating it. b) ‘A fool with a tool is still a fool’: Be smart in what you automate. Avoid replicating your application logic in your test automation framework. A test automation framework should be there to input and check test data. Nothing more, nothing less.
c) Test automation is a separate process/ activity. Don’t think you can automate your test cases ‘in between’ your normal testing activities. Consider
Testing ‘Slow Flows’ Fast - Automated End-2-End testing using interrupts
PAGE
4
a test automation project as a development project, needing proper people to handle it. After all, you’re developing/updating a framework.
d) Test execution tools cannot test every application. Look for a balance in possibilities of the tool(s) at hand and the effort you need to automate your test case execution.
e) Test tools are not ‘magical’. Unlike what vendors might tell you, testing with a press on the button is not possible unless you invest a great deal of time to program the framework behind the button.
f) Test tools are not cheap. Ideally test execution automation should be used mostly for regression testing. Try to avoid using them in situations where new applications are being developed or where your framework will only be used once. Calculate an initial cost of at least 3 times the effort of creating and executing manually your test cases when you plan to automate them (with or without a framework). Ideally test execution automation should be used mostly for regression testing.
g) And don’t forget to calculate maintenance costs for your automation framework. Your automated tests are also sensible to changes.
So think wisely at the beginning of your project whether or not you will use test automation tools. In our example, the decision was taken to go ahead and start the development of a test automation framework.
A Short Evolution of Record - and Playback Tools
As I had experience in building test automation frameworks for different platforms, I went on a search of existing frameworks that would allow me to complete my challenge. It is nice to know that also automation frameworks have evolved in time. Here is a short introduction: Plain Record and playback In the beginning, there were record and playback tools. They were just tools to record key strokes (and later mouse movements) that re-played them in the exact same sequence, kind of like an old VHS tape recorder. In more advanced earlier tools, there were extra’s present like fast re-play and some very basic editing functions (as these tools worked with script-like scenarios) and you could of course save the recorded scenarios. Advantages were that test cases could be replayed more accurately and faster than a tester could do. Disadvantages were: the difficulty in maintaining the scripts created by the record and playback tool, especially when there was a need of extensive scripts, which manifested the lack of adaptability of the scripts to new test data. Data driven In a later stage, tools were equipped with the possibility to use variables in the recorded scripts, so that changing test cases/test data could be done very easily. The base recorded script is kept, but variables are used to replace the hard coded test data. This functionality allowed re-using the same base-script with different test data sets. It was the first step towards a test harness or test automation framework. Although this was a step in the right direction, it was far from an ideal situation. Maintenance on the scripts remained difficult.
Testing ‘Slow Flows’ Fast - Automated End-2-End testing using interrupts
Modular framework Just as in development languages, the scripting languages for testing tools were also evolving towards more modular programming. The ability to make small chunks of code that could be re-used made this framework very flexible and maintenance-friendly. These small modules of code were then used in a hierarchical fashion to construct larger tests, realizing a particular test case.
PAGE
5
Keyword driven In the search of a framework that would be easy to maintain but could also give the possibilities to less experienced testers to create test cases at an abstract level, the keyword driven approach was born. The strength of this type of automation framework is its reusability and the fact that once created (by experienced test automation engineers) it can be used by testers with no specific automation experience. All you need as a tester is a list of keywords to create a complete scenario for test execution. The framework then translates this into understandable actions performed on the application under test. The biggest disadvantage is that for every new function or action to be executed by the tool, an appropriate new keyword section needs to be added in the framework. Model Based In search of a more efficient testing approach, model based testing was born. In model based testing, the testers start from a set of rather abstract test cases, derived from a (sometimes mathematical) model. The model is a partial representation of the behavior of the application under test. Finally the abstract test cases are transformed into real test cases to test the application. In some model-based testing tools, the abstract test cases are automatically transformed to an executable test set, provided that there is enough information available. In other cases, transformation to a usable test case set has to be done manually.
Hybrid testing framework The Hybrid Test Automation Framework is what most frameworks evolve to over time and after repeated use. The most successful automation frameworks generally accommodate both keyword-driven testing as well as data-driven testing. This allows data driven scripts to take advantage of the powerful libraries and utilities that usually accompany a keyword driven architecture. In these kinds of frameworks, a careful balance between maintainability and completeness needs to be drawn. Too often, these frameworks become almost a copy of the applications they are bound to test, but that should never be the purpose of a testing tool. None of the above mentioned Frameworks would be sufficient to fulfill the challenge given to me; I needed to find my own solution. TWEETABLE
Too often, these frameworks become almost a copy of the applications they are bound to test, but that should never be the purpose of a testing tool.
Setting up the Solution
Preparing the business flow As a first step, it is necessary to divide the (business) flow into little (sub-) processes. The best way to do that is to look at points in the process where you expect responses, or need to wait. It is not really necessary to have a screen or graphical user interface, although you can use screens to check your test case status instead of going directly into the database. By simply using a status screen (i.e. for checking the order status in the example) you avoid the use of queries
Testing ‘Slow Flows’ Fast - Automated End-2-End testing using interrupts
to dive into the databases. The business flow would look like this, once divided into (sub -) processes:
The interrupt automation framework The framework I usually start from is a hybrid form. It’s a keyword and data-driven framework that is setup in a modular way for maintenance purposes. Due to my interest in hardware and electronics, I searched here for answers to my challenge. I remembered interrupt requests or IRQ’s, a way to stop the normal action of a computer component and get it to do something ‘in between’. Interrupts are a way to achieve multi-tasking in real time computing and it contained similarities to my challenge. If we look a bit closer to a typical interrupt mechanism, it will look like the following diagram overleaf:
PAGE
6
Suppose this is the workflow that every order has to pass. There are 10,000 test cases. Total testing time can run up to 10.000 times 3h and 45 min. This is impossible to do manually because of the cost and the time of the test execution. The test cases to execute are listed in a file containing the start time of each work flow, i.e. every test case workflow starts with Process A. A complete flow would contain the following sub-processes: A) Online Purchase B) Check Supply C) Check Client status D) Order missing items E) Send Items F) Invoice
‘System Startup’: The initial blocks are there for initialization of the framework and loading initial data. At this point the action queue is filled for the first time with the test data of the test cases that we want to execute. The initial test data set can come with timings prepared, i.e. each test case has its scheduled time already included, or the System startup module can take care of automatically spreading the test cases in time. It will however respect the order in which the test cases have been offered. Then the control is handed over to the ‘Central system’, containing a scheduler. The scheduler is comparing the system clock with the timing of the first action in the queue. If the system time is greater than or equal to the timing mentioned in the first action’s data set, it will start processing the test case. The different parts of a process flow are represented in the diagram by a ‘pile’ centralized around a ‘Task block’. Each ‘Task Block’ contains a ‘context restore’ before and a ‘context save’ after the task. A task in this diagram is representing a (sub-) process of the entire flow to test. The context restore is taking data out of the test data set to go back to the previous point in
Testing ‘Slow Flows’ Fast - Automated End-2-End testing using interrupts
System startup System initialization Central system Scheduler
Context restore
Context restore
Context restore
Context restore
Task : Process A
Task: Process B
Task : Process E
Task : Process F
Context save
Context save
Context save
Context save
calls Reporting Generic functions Error Handling
Switch
PAGE
7
the test cases’ flow or the initial test data arguments for the first (sub-process) of the flow to test. The context save of the block is responsible for saving the status of the test case back in the action queue, ready to process the next part of the flow in the test case. The ‘Switch block’ is deleting the executed
action out of the list and sorts the action list again to prepare it for the next loop. The ‘Calls block’ on the right contains modules to address specific functions in reporting and error-handling. Also a big part of generic functions (e.g. date handling, database access and querying, etc.) is handled in this block.
The Action Queue
The second important part of the framework is built around the action queue. We start from a regular keyword based test data file. This could look like this:
Action
Param 1
Param 2
….
Add Order
12455
Mr. Smith
….
Add Order Add Order
12456 12457
Ms. Simpson Mr. Doe
…. …
By adding an execution time/date stamp, you’re turning it in a simple action queue. After loading the initial list of test cases, the action queue can look like this: Execution Timestamp 01/12/2010 - 08:00:00 01/12/2010 - 08:01:00 01/12/2010 - 08:20:30
Action Add Order Add Order Add Order
Param 1 12455 12456 12457
Param 2 Mr. Smith Ms. Simpson Mr. Doe
…. …. …. …
Add Order
12455
Mr. Smith
….
Add Order Add Order
12456 12457
Ms. Simpson Mr. Doe
…. …
Testing ‘Slow Flows’ Fast - Automated End-2-End testing using interrupts Execution Timestamp 01/12/2010 - 08:00:00 01/12/2010 - 08:01:00 Execution Timestamp 01/12/2010 - 08:20:30 01/12/2010 - 08:00:00 It’s01/12/2010 time to- 08:01:00 roll up 01/12/2010 - 08:20:30
Action Add Order Add Order Action Add Order Add Order Add Order your sleeves. Add Order
Param 1 Param 2 …. 12455 Mr. Smith …. In a conventional framework, the record 12456 Ms. Simpson …. Param and 1 Param …. would wait before 12457 Mr. Doe2 … playback tool 12455 Mr. Smith …. continuing. When using an action queue, 12456 Ms. Simpson …. Here 12457 the tool Mr. can Doe perform … actions instead of
Let’s Rock ’n Roll!
is where the action (queue) starts, let’s ‘Play Computer’ (or better ‘Automation Framework’):
Suppose that inserting an order into Process A (the start of the flow) takes as much as 15 seconds. Then the order will take about 10 minutes before it has completely passed the whole Process A.
PAGE
8
waiting within the time freed up. The only thing we have to add at the end of the ‘Add Order’ action in our tool is the creation of a new action to check the result of Process A some 10 minutes later. So after the initial insertion of an order (Add Order 12456), the action queue could look like:
Execution Timestamp 01/12/2010 - 08:00:00 01/12/2010 - 08:01:00 Execution Timestamp 01/12/2010 - 08:20:30 08:00:00 01/12/2010 - 08:10:15 01/12/2010 - 08:01:00 01/12/2010 - 08:20:30 01/12/2010 - 08:10:15
Action Add Order Add Order Action Add Order Add Order Check Process A Add Order Add Order Check Process A
Param 1 12455 12456 Param 1 12457 12455 12456 12457 12455
Param 2 Mr. Smith Ms. Simpson Param Mr. Doe2 Mr. Smith …. Ms. Simpson Mr. Doe ….
…. …. …. …. … …. …. … ….
01/12/2010 - 08:01:00 01/12/2010 - 08:10:15 Execution Timestamp 01/12/2010 - 08:20:30 01/12/2010 - 08:01:00 01/12/2010 - 08:10:15 01/12/2010 - 08:20:30
Add Order Check Process A Action Add Order Add Order Check Process A Add Order
12456 12455 Param 1 12457 12456 12455 12457
Ms. Simpson …. Param Mr. Doe2 Ms. Simpson …. Mr. Doe
…. …. …. … …. …. …
The 15 seconds are from the processing at 8:10:15. To finish off it will delete the of (sub-) Process A, while the 10 minutes action (order) that was just executed. come from the expected finishing time Before taking the next action out of the of the process. So the automation queue we sort the action queue by the framework will check whether the result is Execution Timestamp (ascending) to correct (orTimestamp the order has the correct state) this: Action Param 1look like Param 2 …. Execution
When 08:01:00 has passed (scheduler will check with system clock), Order 12456 can be processed. And a similar Execution Timestamp 01/12/2010 - 08:01:00 01/12/2010 - 08:10:15 01/12/2010 - 08:20:30 01/12/2010 - 08:11:15
Action Add Order Check Process A Add Order Check Process A
‘Check Process A’ action can be added to the queue.
Param 1 12456 12455 12457 12456
Param 2 Ms. Simpson …. Mr. Doe ….
…. …. …. … ….
Execution Timestamp 01/12/2010 - 08:01:00 01/12/2010 - 08:10:15 01/12/2010 - 08:20:30 At01/12/2010 08:10:15- 08:11:15 ‘Check
Action Add Order Check Process A Add Order Check Process A Process A’ for
Param 1 12456 12455 12457 12456 new order
Testing ‘Slow Flows’ Fast - Automated End-2-End testing using interrupts
12455 will be done. This can be anything from checking databases to inserting Execution T 01/12/2010 01/12/2010 01/12/2010 01/12/2010
imestamp - 08:10:15 - 08:11:15 - 08:20:30 – 08:10:30
Action Check Process A Check Process A Add Order Start Process B
Param 2 Ms. Simpson …. Mr. Doe …. into the info
…. …. …. … …. system.
At the end of this, a new action can be inserted in the queue ‘Start Process B’ for order 12455.
Param 1 12455 12456 12457 12455
Param 2 …. …. Mr. Doe …
…. …. …. …
Drawbacks
PAGE
9
The strength of this action queue is that you can use ‘fixed’ checking times (see Process A in the example flow, that will This ‘Interrupt-driven’ test automation have a processing time of 10 minutes) or s. has proved already its merits framework ‘variable’ checking times (see Process in several occasions, but it is far from B, C and D in the example flow, that -F in the flow. The queue will fill itself These after the initial load of test casestimes, and willbut control its ownperfect. processing. When all are some important are unknown processing with cases have finished, the queue will be empty and automated testing will stop. drawbacks to this framework: a Every upper processing time).appropriate You tasklimit can beinmonitored by integrating reporting functionality inside, so that the tester can, at any followD -up the progress of the test can even check Processes B,time, C and 1) of Sometimes, execution. Additionally he has a proof of the order of execution the different you just can’t cut your intermediately to see whether they have test cases. business flow into (sub-) processes. already been finished. If they have not yet Most of the time, this is the case for finished, you ks can add the same ‘Check Drawbac online applications. It does not mean Process’ action again with an interval of that theitsframework will not work with This ‘Interrupt -driven’ test automation framework has proved already merits in a several few minutes. occasions, but it is far from perfect. These are some important drawbacks this kind of applications, but it’s just to this framework: not designed to unleash its full We can implement this for every Process 1) Sometimes, you just can’t cut your business flow into (sub in these -)process es. potential circumstances. A-F in the flow. The queue will fill itself Most of the time, this is the case for online applications. It does not mean after thethat initial load of test cases andthis will the framework will not work with kind of applications, but it’s just 2) Timing accuracy can be a bit tricky. control its processing. When all cases notown designed to unleash its full potential in these circumstances. have finished, the queue will be empty 2) Timing accuracy can be a bit tricky. a. The system is not designed and automated testing will stop. Every to take into account the individual a. The system is not to take into account the individual task can be monitored bydesigned integrating processing time of each action. i.e. processing time of each action. I.e. if the framework is executing appropriate reporting functionality inside, an action and in the meantime there is an action the top if due theatframework is executing an so that the tester can, at any time, followof the action queue, th action and in the meantime there is up the progress of the test execution. an action due at the top of the Additionally he has- a proof of the order action queue, the latter one will of execution of the different test cases. have to wait till the framework has finished the execution of the first one. This is the trade-off that has to be made between ‘Quantity’ (of test cases to execute) and ‘Accuracy’ in timing.
Testing ‘Slow Flows’ Fast - Automated End-2-End testing using interrupts
b. You can try to manipulate execution time stamps in such a way that test cases won’t impact each other too much, but it will come at a price: the number of test cases that can be executed in a given timeframe will be lower.
c. Fine-tuning the timings in your framework is necessary to optimize execution speed of your test cases. Some iterative runs with the same test data will give you enough information to adjust the parameters of your framework to work as optimized as possible.
PAGE
10
3) Processing order of your test cases can be different, even if you use the same set of test data multiple times. It all depends on the reaction time of the (sub-) processes you have in your business flow. This is not necessarily a drawback, as we are focusing on completing a number of test cases anyway, rather than executing them in a specific order. The exact execution order of the different tasks (actions) is logged in the test reports generated by the automation framework. TWEETABLE
As a first step, it is necessary to divide the (business) flow into little (sub-) processes.
The Summary By using an action queue in your automated test framework, you can easily adapt your existing test automation framework to perform many actions while the tool is waiting for processes to finish. This kind of ‘interrupt processing’ is a solution for every application process flow that can take up many hours in total before having finished and where interaction between independent systems is reflected in a (or several) database(s). It can even check timers and delays of processes without much hassle. To me, this type of framework has proven its reliability and usability several times. It has done miracles in automated testing of high volumes of ‘slow processes’. It’s a simple and straightforward asset for your hybrid testing framework with a lot of added value. Too often these kind of ‘slow’ processes, being time-sensitive or using asynchronous communication are considered difficult and impossible to test and are often only tested in a system testing phase. With this approach, these often critical processes can be tested end-2-end and will give you the extra security, by reducing the risks involved in using multiple independent systems with time critical processing.
Testing ‘Slow Flows’ Fast - Automated End-2-End testing using interrupts
References -
Practical Model-Based Testing: A Tools Approach, Mark Utting and Bruno Legeard, Morgan-Kaufmann 2006.
-
Improving the maintainability of Automated Test Suites, CemKaner, Paper presented at Quality week 1997.
- Automatisering van de testuitvoering, Bart Broekman, Christiaan Hoos, Mark Paap, 2001.
PAGE
11
- TMap Next, for result-driven testing, Tim Koomen, Leo van der Aalst, Bart Broekman, Michiel Vroon.
Biography
Graduated as Master in Commercial and Financial sciences, Dominic started his IT-career in 1997 as one of the first, specially trained, professional testers in Belgium. He has worked his way up through several dozens of projects in different domains and economic sectors. He grew from junior tester to test expert, specialized in test methodology and testing tools. Dominic has gained experience in sectors such as retail, telecom, finance and government in Belgium as well as in the Netherlands. Dominic has an excellent knowledge of TMap Next® and is ISEB Practitioner and TMap® certified. He is experienced in coaching and supporting, giving training, test and people management, developing test strategies for projects and large programmes, writing down (master-) test plans and the pragmatic implementation of test execution and test automation. Dominic is communicative, diplomatic and pragmatic. He is a known speaker at universities, seminars and conferences. Dominic is member of the board of the Discussion group Software Testing (TI KVIV) and is co-founder of the Belgian Testers.
Join the EuroSTAR Community‌ Access the latest testing news! Our Community strives to provide test professionals with resources that prove beneficial to their day-to-day roles. These resources include catalogues of free on-demand webinars, ebooks, videos, presentations from past conferences and much more...
Follow us on Twitter @esconfs Remember to use our hash tag #esconfs when tweeting about EuroSTAR 2011!
Become a fan of EuroSTAR on Facebook Join our LinkedIn Group
The EuroSTAR Blog
Contribute to the EuroSTAR Blog
Written by Testers for Testers
Check out our free Webinar Archive Download our latest ebooks
w w w. e u r o s t a r c o n f e r e n c e s . c o m