TEST Magazine - March-April 2010

IN TOUCH WITH TECHNOLOGY

T H E E U RO P E A N S O F T WA R E T E S T E R Volume 2: Issue 1: March 2010

HIGH HEELS IN HIGH TECH

Devon Smith with a woman’s view of a hi-tec industry

Inside: Risk-based testing | Checking automation | User testing Visit T.E.S.T online at www.testmagazineonline.com

Test Case Management

Satisfy your quality obsession.

TestTrack® TCM

www.seapine.com/testmag

[

Test Planning & Tracking

© 2009 Seapine Software, Inc. All rights reserved.

TestTrack® Studio

Surround SCM®

Configuration Management

Seapine CM®

Change Management

Automated Testing

QA Wizard® Pro

[

Full-Time Quality Assurance Manager—Immediate Opening

• Achieve complete traceability between test cases and defects with seamless TestTrack Pro integration.

• Ensure all steps are executed, and in the same order, for more consistent testing.

Test Case Management

• Streamline the QA > Fix > Re-test cycle by pushing test failures immediately into the defect management workflow.

• Manage suites of platform-specific compliance tests, functional tests, and performance tests in one central location.

• Assign tests to your QA team, track results, and report on performance and workload.

Issue Management

• Know instantly which test cases have been executed, what your coverage is, and how much testing remains.

• Use test variants to target multiple platforms with the same test case for more efficient test case management.

TestTrack® Pro

TestTrack TCM puts you in control of test case planning and tracking, providing better visibility over the testing effort and giving you more time to manage your team. With TestTrack TCM your team can write and manage thousands of test cases, select sets of tests to run against builds, and process the pass/fail results using your development workflow.

TestTrack® TCM

Don’t work yourself to death. Use TestTrack® TCM to manage your testing effort.

www.testmagazineonline.com

Leader | 1

T H E E U RO P E A N S O F T WA R E T E S T E R

IN TOUCH WITH TECHNOLOGY T.E.S.T THE EUROPEAN SOFTWARE TESTER

T H E E U RO P E A N S O F T WA R E T E S T E R Volume 2: Issue 1: March 2010

VOLUME 2: ISSUE 1: MARCH 2010

HIGH HEELS IN HIGH TECH

Devon Smith with a woman’s view of a hi-tec industry

Inside: Risk-based testing | Checking automation | User testing Visit T.E.S.T online at www.testmagazineonline.com

Editor Matthew Bailey matthew.bailey@31media.co.uk Tel: +44 (0)203 056 4599 To advertise contact: Grant Farrell grant.farrell@31media.co.uk Tel: +44(0)203 056 4598 Production & Design Dean Cook dean.cook@31media.co.uk Toni Barrington toni.barrington@31media.co.uk Editorial & Advertising Enquiries 31 Media Limited, Media House, 16 Rippolson Road, London, SE18 1NS Tel: +44 (0) 870 863 6930 Fax: +44 (0) 870 085 8837 Email: info@31media.co.uk Web: www.testmagazineonline.com

Did Toyota lose its ‘Way’?

S

peculation has been rife as to why Toyota, for so long a byword in quality and continuous improvement with ‘The Toyota Way’, has had catastrophic quality problems at the end of 2009 and into 2010.

Last month it revealed – as some in the software testing community had predicted – that a significant part of the problem stemmed from software. Following the floor mat and accelerator pedal recalls in 2009, Toyota issued a separate recall for hybrid anti-lock brake software last month. It was quoted as investigating the reports, and said that it would be “premature to comment,” although later the same month the Japanese automotive giant said it was investigating problems with the Prius's brakes, which it confirmed were caused by a software glitch.

It seems Toyota has done the right thing in suspending sales and recalling all affected vehicles, but the effect on its hard-won reputation for total quality can’t be underestimated. We can only hope that application of the Toyota principles will effect a rapid resolution of the problems. Sometimes it’s not the problem itself that causes the long-term damage; it’s the way

Printed by Pensord, Tram Road, Pontllanfraith, Blackwood. NP12 2YA © 2010 31 Media Limited.All rights reserved.

it’s dealt with. A couple of famous examples prove this point. In the ’90s Coke had problems in its Benelux region with kids complaining of illness and vomiting after drinking the product. Coke denied all knowledge and responsibility taking a highhanded attitude that the cause was mass hysteria triggered perhaps by a non-toxic residue of fungicide picked up from wooden pallets. They may well have been right, but the damage was done. Their reputation suffered and the global CEO’s head rolled as a result and a more people-oriented chief was promoted to the top spot.

Then there was a famous case with Tylenol in the US. In September 1982, a Tylenol scare started when the first of seven victims died after taking Extra Strength Tylenol that had been deliberately contaminated with cyanide. The company immediately recalled 31 million bottles of the tablets from retailers. The crime was never solved and, as you would expect, Tylenol sales collapsed. The brand was rescued by the introduction of the first inherently tamper-proof capsule, recapturing and even building on the sales lost after the cyanide incident.

The scare led to the introduction of tamper-evident packaging and ‘gelcaps’ in the drug industry.

Toyota has many systems and procedures to promote total quality, but perhaps having to fight this type of rear-guard action is a new experience for its quality engineering gurus.

T.E.S.T Magazine is edited, designed, and published by 31 Media Limited. No part of T.E.S.T Magazine may be reproduced, transmitted, stored electronically, distributed, or copied, in whole or part without the prior written consent of the publisher. A reprint service is available.

Opinions expressed in this journal do not necessarily reflect those of the editor or T.E.S.T Magazine or its publisher, 31 Media Limited.

Matt Bailey, Editor

ISSN 2040-0160

Published by:

It seems Toyota has done the right thing in suspending sales and recalling all affected vehicles, but the affect on its hard-won reputation for total quality can’t be underestimated. We can only hope that application of the Toyota principles will affect a rapid resolution of the problems. Sometimes it’s not the problem itself that causes the long-term damage; it’s the way it’s dealt with.

Matt Bailey, Editor

www.testmagazineonline.com

March 2010 | T.E.S.T

In Touch With Technology

SUBSCRIBE TO T.E.S.T. GY TH TECHNOLO IN TOUCH WI

Supported by

IN TOUCH WITH TECHNOLOGY

THE EUROPEAN SOFTWARE TESTER

ARE TESTER EAN SOFTW THE EUROP

T.E.S.T THE EU ROPEAN SOFTW ARE TESTER

T.E.S.T

T.E.S.T

TESTER N S O F T WA R E THE EUROPEA

Septe mber 2009 Volum e 1: Issue 3:

Supported by

IN TOUCH WITH T ECHNOLOGY

T H E E U RO P E A N S O F T WA R E T E S T E R

Volume 1: Issue 4: December 2009

THE EUROPEAN SO F T WA R E T E S T E R

Volum e 2: Issue 1: March 2010

Dave Whalen takes on the cult of Agile the agile approach James Christie takes as a service testing | Crowd Inside: Risk-based

VOLUME 2: ISS UE 1: MARCH 2 010

VOLUME 1: ISSUE 4: DECEMBER 2009

009 PTEMBER 2 ISSUE 3: SE VOLUME 1:

BRIDGING I HATE HIG AG ILE ! H HEELS P A G E H T IN HIGH TECH testing | Testing Devon Smith with a wom Inside: Data security | Testing virtual worlds | User accepted an’s view testing

ne at T.E.S.T in now onli

online.com www.testmagazine Visit T.E.S.T online at www.testmagazineonline.com

of a hi-tec industry

Inside: Risk-based testing | Checking automation | User

testing

Visit T.E.S.T online at www .testmagazineonline.com

Simply visit www.testmagazine.co.uk/subscribe Or email subscriptions@testmagazine.co.uk *Please

note that subscription rates vary depending on geographical location

Published by 31 Media Ltd Telephone: +44 (0) 870 863 6930 Facsimile: +44 (0) 870 085 8837

www.31media.co.uk T.E.S.T | March 2010

Email: info@31media.co.uk Website: www.31media.co.uk

The European Software Tester www.testmagazineonline.com

Contents | 3

CONTENTS APR 2010

1

Leader column

A recent problem with the cruise control on Toyota cars is linked to software.

What quality lessons can the car giant learn?

4

Cover story – High heels in high tech

Lead QA for Agora Games, Devon Smith explains the challenges and benefits of being the only female in the testing department.

10

Turning up the heat on risk-based testing Neil Pandit of Sopra Group, winner of the Logica Triple Star Award for Innovation at the EuroSTAR 2009 conference, explains the benefits of a visual approach to risk-based integration testing.

18

Automation: Your check mate

According to Keith Braithwaite much of the routine checking that makes up the bulk of testing is ripe for automation, freeing up testers for more important, high-value work.

22

Is automation the answer?

Efficient management of resources is a considerable challenge for network and

4

communications testing, especially in large or distributed testing labs. Eitan Lavie explains the benefits of an automated resource management system.

26

Your most valuable software testing tool

There’s no excuse in this day and age for a non user-friendly interface and often the only source of reliable insight into this area is the user. Dave Tait puts the case for the user as invaluable testing tool.

28

Putting the user at the centre of things

26

A strategy that puts the user in control at the centre of things is, according to Hammad Khan, the principal benefit of a user-centred approach.

32

The case for customisation

It’s high time our cumbersome monolithic operating systems offered increased agility, enhanced security and lower support costs. Steve Harris puts the case for customisation.

35

Tools you can trust

Automated testing tools can save time and resources in the rapidly expanding agile world. Senior systems engineer Allen J Fisher looks into the future of testing tools.

38

35

A common language for testing

Mike Smith, managing director of Testing Solutions Group explains how Learntesting online training is building a common language for testing.

42

T.E.S.T Directory

48

The Last Word – Dave Whalen After taking on Agile in our last issue, Whalen takes on a construction project which is at times oddly reminiscent of the testing world.

www.testmagazineonline.com

38 March 2010 | T.E.S.T

4 | Test cover story

High heels in high tech It’s different for girls according to the song. Lead QA Devon Smith explains the challenges and benefits of being the only female in the testing department.

O

h! You’re a girl!” said a remote member of a QA team I worked for during our first phone call. “I didn’t expect that.” Taking people by surprise is something you get used to when you work with computers.

Millennium aside, the software field is still dominated by men, and there is a good chance that as a woman, at some point in your career (in my case, a few times) you will find yourself one of the few – if not the only – woman on your team.

That is something they do not prepare you for in college. Your co-workers may be the nicest guys ever, but at first glance, you look more like their sister than their teammate.

Will your bugs be taken seriously?

Will you ever fit in? It feels a little like walking in to the locker room on the first day of gym class – except you accidently walked in on the guys’ side.

Being the only (or the first) woman in

T.E.S.T | March 2010

your company is a unique experience.

There you are, laptop in hand, the lone woman in a room full of programmers.

"I walk in the door and everyone thinks I'm the secretary," said one female programmer I talked to. A QA manager kept threatening to hang her degree on her office wall to prove she belonged. I have shown up for interviews and had people say “actually, I’m waiting for an interviewee right now, but he isn’t here.” I did get to enjoy the surprise on their faces when I explained that I was, in fact, that interviewee.

Integration Certifications, degrees, experience, and testing philosophies mean nothing if you cannot integrate into the team, and in some interviews, it feels like they are evaluating your skills as well as your nail polish. If they take you on, what other obstacles will they have to face? What does it mean to www.testmagazineonline.com

Test cover story | 5

have a woman working on the team?

Sometimes the subject is delicately breached before hiring, as the company tries to gauge your ability not only to do the job, but also to work with all men.

You move into your new office space (which you share with a guy named George) and start to unpack and set up your computer. Your boss seems nice, your office has a window, and the lunchroom fridge is stocked with soda. You have made a mental note to request Diet Coke. George is friendly, but a little awkward as he clears off his collection of army men that have been launching a full-scale attack on one corner of your desk. He offers to show you around and introduce you.

Somewhere between introducing you to Mark, Brian, Greg and Dan, he leans over and says “I’ve always wanted to walk around the office with a girl on my arm.” It is right then that you realise you are in strange territory. This is not an ordinary office – this is a tech office and until you started – this was a place for jeans, “that’s what she said” jokes, and hot wings lunches. While you are introduced to Dan, you can tell people are looking at you, and it is not because you are the new person – it is because you are the first woman on their team and everyone wants to know what that means.

You are the new dynamic. In that awkward silence on your first day, your co-workers might be a little worried that you will radically change the office culture. Are you the new house mom? Will you be uncomfortable if they make jokes like they used to. Do you know what you are doing? Can they keep wearing that stained t-shirt they love so much?

Differentiation They have lots of questions, but so do you. How do you act around these men? Are you supposed to fit in and be one of the boys or do they expect you to be a lady? What do you wear?

You might be uncomfortable wearing feminine (office appropriate) clothes.

Nylons, necklaces, and necklines are www.testmagazineonline.com

It is right then that you realise you are in strange territory. This is not an ordinary office – this is a tech office and until you started – this was a place for jeans, “that’s what she said” jokes, and hot wings lunches. While you are introduced to Dan, you can tell people are looking at you, and it is not because you are the new person – it is because you are the first woman on their team and everyone wants to know what that means. all new here. Is a knee length skirt too short when no one else is wearing them? There may be no HR policy about open toed shoes. When the guys wear shorts in the summer – is it ok if you do? How sensitive should people be about professionalism? Will there be a women’s bathroom or will you all share? Most of that depends on the culture of the office – but a lot of it you will define yourself as you find your spot on the team.

If you have come from an office with more gender diversity, you may find yourself suddenly wearing lots of turtlenecks, questioning the length of your skirt, and feeling very aware of the sound of heels clicking on the office floor. It is obvious you are in a different world here – the office is like a bachelor’s apartment (ok, honestly it is sometimes like a frat house) empty toilet paper rolls and all. Some days you are one of the boys, and some days you get asked out three times by the same co-worker. At first one of them tries explain to you how March 2010 | T.E.S.T

6 | Test cover story

You might look for bugs in new places, or question the wording of a spec that never would have been considered. If we think differently, work together differently and test differently, it brings a valuable new dimension to the team.

T.E.S.T | March 2010

to work a mouse and a file menu, but eventually they learn that you can match them in discussion about requirements and software behaviour.

I read an article that studied women in male dominated workplaces.

They found that women would subconsciously speak in a lower voice in an attempt to fit in with the men. I believe you do not need to be one of the boys to be one of the team. You may need to learn to communicate in a slightly different way with the developers, but that is part of QA anyway. Besides, often the guys will tend to communicate with you a little differently at first. They do not want to hurt your feelings, but they do not always know how to talk to women.

It is important to remember that even though you are all co-workers, women and men are still wired a little differently, but that is a good thing!

You might look for bugs in new places, or question the wording of a spec that never would have been considered.

If we think differently, work together

differently and test differently, it brings a valuable new dimension to the team.

Man management At some point, you may end up managing men; sometimes much older men. When I was 23, I found myself, as the youngest team member and the only women in the group, in charge of men old enough to be my father. Age is a factor, but so is gender.

Being open about any uncomfortable dynamics there is the best policy – even if you have to be the one to bring it up. There were a few rocky releases for us, but eventually, we were all up together on late nights testing. We learned to work together and overlook the generational and gender gaps to make a great team. It feels strange to talk about it, but gender is defiantly a factor.

When a woman I know started a new job to lead a group of programmers, she told me that one of the biggest challenges was www.testmagazineonline.com

A Global Virtual Learning Environment for the Software Testing industry Positioned to guide you to success in testing education and certification ■ ■

■

■

■

Options to fulfil your requirements – from start to finish Access to online and classroom courses, virtual classrooms, ebooks and a range of other testing-related content Sample exam questions, papers, revision sessions and live exam service Scalable to support individuals to the largest corporations Supported 24/7 with a global network of accredited training providers

www.learntesting.com For details of your local Learntesting provider, visit www.learntesting.com and begin your journey to success with Learntesting

8 | Test cover story

coming on and taking over a man’s role on a team of all men. She had to work to find her place – both as a new employee, and the only female lead. It can be uncomfortable for you and the people you have been assigned to manage. There are different boundaries; different ways of interacting that are often overlooked.

It can be intimidating to stand up and take charge in that situation, but as a leader, it is important to address the issues and face them so that your team does not suffer.

New territory When you have been on the team a while, navigated through shared bathrooms (was that a Maxim Magazine in there?) and a few awkward conversations, you think you have made it through the hard part.

Nope! Even when your team settles in to routine or hires a few more women, there is still new territory to cross.

Sometimes that territory is a little uncomfortable, especially if you are working for a small company that has not had to deal with a lot of office-place issues yet. When the first woman goes and does something like getting pregnant, the company scrambles to put together a maternity policy. The first woman to have a baby at my first job had to pump in a bathroom stall because people walked in on her when she tried to use an office. Eventually there was another new clause in the company manual about working mothers, which they created after her.

You have to be more honest about what you need- even if it is embarrassing- like covered trashcans in the co-ed bathroom for example (that one took me about three months to implement). When you travel, you might need to remind them that while you are fine sharing a rental car, you would prefer your own hotel room.

It seems like regardless of how progressive your co-workers are, someone is going to ask you to plan the monthly birthday party, or the T.E.S.T | March 2010

baby shower, or bring in cookies, or decorate the office lobby. It is not always a negative thing. Let’s face it, sometimes we need help reaching the plastic silverware on the top of the cupboard, and sometimes your male co-workers need help picking out a cake for birthdays. The trick is to get everyone involved, so you do not feel like QA, party planner, and interior designer all rolled in to one. Besides, guys are apparently really good at hanging streamers and everyone needs a break from the laptop glare sometimes.

Diversity Once you find a comfortable fit though – the joys of being one of a few is apparent. I have found the offices to often be more laid back and less cliquish. After few late releases together (or after you win your first chicken wing eating contest), you are suddenly part of a team that can use its differences – in age, gender or background – to its advantage and work together. It makes for stronger QA teams and a more interesting work place.

Then one day you are sitting in a meeting debating requirements documents and one of your co-worker says, “you are like the most awesome girl ever” and you realise that you have done what not a lot of women even attempt. You have made a place for yourself on your team and brought something new to the work environment. Everyone has relaxed.

Even though when you go to the company Christmas party every other girl there is someone’s date, you know that you hold a special place in the office.

Maybe future new girls will have it just a bit easier. A woman in the workplace is not such a daunting idea anymore. You have proven yourself as a good QA and a good teammate. Your co-workers know that you can file bugs, play pranks, and debug a test suite with the best of them – only backwards and in heels.

Devon Smith Lead QA Agora Games http://ladybug010.wordpress.com/

www.testmagazineonline.com

The Whole Story Print Digital Online

For exclusive news, features, opinion, comment, directory, digital archive and much more visit

www.testmagazineonline.com www.31media.co.uk

The European Software Tester

10 | Risk-based testing

Turning up the heat on risk-based testing Neil Pandit of Sopra Group, winner of the Logica Triple Star Award for Innovation at the EuroSTAR 2009 conference, gives us a taste of his winning presentation where he described the benefits of a visual approach to risk-based integration testing.

W

ith reduced budgets, resources and a need to deliver projects faster, risk-based testing (RBT) is an area that is increasingly being seen as an approach that stakeholders and project managers are keen to adopt.

Often, project stakeholders are attracted towards risk-based testing as the benefits are easily promoted: reduced testing costs through targeted testing and reduced risk of failure in production.

RBT is about developing an objective technique to prioritise the testing T.E.S.T | March 2010

effort, with a view to reducing the overall testing effort, while minimising any increase in risk exposure to the organisation. It is, therefore, an ideal approach to be applied to projects where there is a combination of limited time, money and people available.

Unfortunately there are often misconceptions about the use and objectives of RBT, which can lead to it being perceived as a ‘taboo’ technique within some organisations.

This short article will provide details of the presentation that I delivered at EuroSTAR 2009 entitled “A Visual Approach to Risk-Based Integration www.testmagazineonline.com

Risk-based testing | 11

The objectives of RBT can often vary in organisations, depending on who you speak to. From a testing perspective the objective is typically to mitigate risk while reducing overall testing effort, from a project manager’s perspective it is often to reduce costs but not compromise on quality, whereas from the business area, RBT can be viewed as a reason by the project team to do less testing, and is sometimes therefore perceived as actually increasing the overall risk. www.testmagazineonline.com

Testing.” The visual approach highlights potential risks using colours annotated onto a system architecture diagram resulting in the creation of a ‘heatmap’. The process considers both the business impacts and technical complexities of the systems, with the architecture diagram being mutually agreed by all stakeholders within the project. This article will also highlight some of the misconceptions and problems encountered when using RBT and highlight other testing areas where the concept of heatmaps can be applied.

However, before I discuss the detail of using this visual technique, it is useful to consider a relevant concept which highlights why two key roles in a project (developer and business user) have very different perspectives of RBT. This observation together with the consequences mentioned below is the primary reason for developing this approach to RBT. A developer that is asked to apply a risk-based approach to testing for Unit Testing or Unit Integration Testing will tend to focus on the technical nature and complexity of the modules or solution. As far as the developer is concerned, the highest risk will be in areas that are considered new, unstable or technically complex. The developer’s focus when applying a risk-based approach to testing is therefore more at a technical level

rather than on any knowledge of business criticality of the functionality being developed. If asked which areas the system integration or user acceptance testers should focus their testing efforts on, the developer would generally point to the technically complex areas as these are believed to be of the highest risk.

From a business user’s perspective, however, these technically complex areas are of little importance and his/ her focus will tend to be on ensuring that the critical business processes required to keep the business operational are functioning correctly.

This difference in focus can subsequently result in two parties with very different perceptions on risks and how testing might mitigate that risk with both developer and business user having ‘supposedly’ performed a risk-based approach to testing; this can result in a combination of two serious consequences: 1. The business user encounters a critical defect in an area of functionality that has not been tested by the developer due to the functionality being perceived as having a low technical likelihood of failure.

2. The business user does not test a non-critical process and so fails to detect a defect in a technically complex area that the developer would have expected the business March 2010 | T.E.S.T

12 | Risk-based testing

user to have tested, resulting in unanticipated production problems.

This is where some RBT approaches may be biased towards only the business impact of failure and ignores the technical likelihood of failure.

From my experience on a client where this specific observation was made, both the business and technical teams were overwhelmed by the number of supposedly ‘noncritical’ technical production defects and instability in the systems to the point where it was significantly impacting the business. This was despite the business users having tested the ‘critical’ processes based on business impact. The inability to fully grasp what RBT can provide and the further amplification of problems during System Integration and User Acceptance Testing leads us into the key misconceptions of RBT.

Misconceptions of RBT Clarification of objectives of RBT: The objectives of RBT can often vary in organisations, depending on who you speak to. From a testing perspective the objective is typically to mitigate risk while reducing overall testing effort, from a project manager’s perspective it is often to reduce costs but not compromise on quality, whereas from the business area, RBT can be viewed as a reason by the project team to do less testing, and is sometimes therefore perceived as actually increasing the overall risk.

Independent of software delivery: Another misconception is that the RBT approach can be developed independently of software delivery.

A prioritised order of execution can be produced, but if your systems and components are subsequently delivered in a different order by your development partner, this may crucially affect both the scheduling and effectiveness of your test execution.

No defects in later test phases: There is a misconception that RBT will find all the major defects, so there is great T.E.S.T | March 2010

surprise when a high severity defect is found in UAT. The two issues here are: 1. It is ‘risk’-based, which implies there will be risks remaining and therefore potential defects.

2. The defect may be a major severity defect but this may be in a non-critical system or process. Again as a result of defects remaining, the perception by the business is that RBT still leaves you with defects.

Problems with RBT In addition to the above misconceptions, testing of major system integration projects can result in a number of other problems: Obtaining ‘objective’ risk assessment of multiple integrated systems: It may be straightforward to apply an RBT approach when testing a single system as it usually involves a single technical / business resource assessing the risks, but with a large number of systems and interfaces, and potentially a number of resource involved, it will be harder to objectively assess the risks of failure of systems.

Focus is on systems and not interfaces: Typically when applying RBT, the focus tends to be on systems and so interfaces and data transfers can get overlooked. Often, the business users may not know that data is being transferred, as they may not be aware of the lower technical architecture of how data is transferred from one system to another. This may result in too much emphasis being placed on testing individual systems and not enough on interface testing.

Reporting does not focus on risks mitigated but on ‘numbers’ of scripts executed: Although execution is riskbased, progress reporting tends to continue to focus on the number of scripts that have been run. Typically, project managers are still looking for how many scripts have been run, rather than the risks that have been mitigated. Progress reporting also lacks visible representation to key stakeholders and clarity over where the problem systems or interfaces

are within the architecture / process.

By stating in a weekly report that the interface between system X and system Y is not working means very little when out of context with the overall end-to-end business process.

Heatmaps – A potential solution An approach that I have successfully applied uses the concept of heatmaps, which takes into consideration both the misconceptions and problems highlighted above of testing large system integration projects.

A heatmap is a visual representation of data or information through the use of colours. This method of communication is accessible to all levels of users and is commonly used in other sectors outside of software testing.

So where have you seen them before? I am sure you are all familiar with weather maps which use different colours to indicate warmer areas than others or different colours to indicate heavier rainfall across different areas.

Other examples are in the stock market where you see that blue typically means that shares are rising and red means that they are falling.

Our approach to applying heatmaps for RBT similarly provides a visual representation of the testing priorities through the use of colours. In terms of what we use as our ‘map’, as the most commonly understood diagram on a project is the system architecture www.testmagazineonline.com

14 | Risk-based testing

be a system or interface which may have manual workarounds that can be put in place immediately or internal systems and interfaces which are nonApproach customer facing.

In terms of our approach, the first So, at this point we have a system activity is to obtain the system architecture diagram with the architecture diagram for the system, technical likelihood of failure and an an example of which is shown in architecture diagram with the business Figure 1.

We then set up a workshop with the impacts of failure of each system and interface. Using a matrix as shown technical experts or owners of the in Figure 2, we take the business and systems to get an understanding of the technical complexities and risks of technical priorities for each system and combine these into an overall failure of the systems and interfaces prioritised approach. So a system so that we can annotate these onto that is considered to be high (orange) the diagram using colours. The from a technical risk perspective and a principle is to apply different colours high (orange) from a business impact to indicate different levels of priority to the system architecture diagram to perspective will have its overall rating increased to a critical (red) rating. demonstrate the technical likelihood This now gives us a combined system of failure. architecture diagram. A simplified This provides a technical heatmap example of a system architecture based on four levels of priority such diagram that might be produced is that critical may relate to a new shown in Figure 3. system or interfaces being created, As a result of using this technique, unstable existing production system, we end up with a final combined new technology / supplier etc. A agreed diagram that not only takes low likelihood would be where there the business impact into consideration were minor / no changes to a system but also the technical complexities or interface or where a system or of the solution. This now satisfies not interface has a low production defect only the business users in addressing history. their concerns but also addresses the Having created the technical technical areas that testing is focusing heatmap, we set up another on potentially less stable areas. workshop with the business areas to This diagram also tells us that the understand firstly the critical business areas of focus for testing need to be processes and map this to the system architecture diagram. We then perform on broker sales system, customer database and policy management the same prioritisation exercise using systems as well as three critical four levels of prioritisation with the interfaces. You will still be testing remainder of the systems to agree other lesser priority systems and a prioritised business impact of the interfaces along the way as part of systems and interfaces. any end-to-end test, but the key point So a critical priority system may be here is to understand the systems and one where failure incurs a significant interfaces that form the primary focus regulatory fine, has direct financial of our testing against those that we / customer impact or reputational simply use to help us in our test. impact. A low severity impact may diagram, this is used as the basis for applying the heatmap approach.

T.E.S.T | March 2010

fig. 1

fig. 2

fig. 3

www.testmagazineonline.com

Risk-based testing | 15

However critical the system might be, we still need to look at the technical risks of failure rather than simply testing a system or interface because the business has identified it as critical.

Benefits of heatmaps There are many benefits of using a visual approach as described above. In terms of the key benefits, these are: Highlights the RBT approach visually: The primary benefit is that it enables you to visually prioritise and focus your System Integration Testing.

Provides ease of understanding for all stakeholders: This is a simple method of communicating the RBT approach to all stakeholders who may not have much knowledge around testing.

Incorporates technical risk into test design: Provides testers with a clear focus on test design of scripts ensuring scripts are created that cover the maximum number of hotspots while incorporating the technical risks. Your test design will still cover lower priority systems but you can adapt the design so that you are not spending significant time identifying data variants in areas of the integrated system that are of lower priority.

Provides risk-based reporting against specific interfaces and systems: Enables you to use the heatmaps to provide visual progress reports on specific interfaces and systems.

Considerations and challenges when using this approach In order to apply this technique successfully it is important to be aware of some considerations and challenges that are typically encountered throughout the process.

Being aware of these challenges will www.testmagazineonline.com

help you to apply this technique successfully.

Obtaining full buy-in from subject matter experts and technical resources: This is not an activity that testing can do in isolation and so needs full buy-in from the business users and technical architects to attend workshops. The demand for key resources to attend workshops requires them to get away from their day jobs.

Understanding impact of IT-driven processes: There may be some activities that are not business-led processes and so these may get omitted from the assessment. These may be things such as maintenance activities, updating rating tables, postcode tables that are run by separate IT teams, which could have a major impact to system functionality if failed but are not shown in the architecture diagram or identified as a business process.

Agreeing organisational business impact: It can be tempting for some business areas to exaggerate the organisational impact of failure of systems as they do not want to be supporting unstable processes. If this results in additional business resource performing corrective action or manual workarounds then budgets may also need to be protected. My experience has shown that some personalities exaggerated the impact of certain issues in order to have their business areas tested to a greater depth, rather than taking into consideration the wider impact on the organisation.

Ensuring use of up-to-date system architecture diagrams: We need to ensure that the architecture diagrams include all the systems that are being used by the business and not just those supported by IT. They may

We can use the testing priority of systems and interfaces to help with determining the priority of both project and production defects such that the approach can be used to prioritise the order in which defects should be fixed first.

March 2010 | T.E.S.T

16 | Risk-based testing

not include bespoke systems that are potentially created / purchased by the business or even redundant systems that are no longer supported by IT, but still being used by the business!

Lack of availability of fully integrated test environments: Finally we have agreed the RBT approach using the system architecture diagram but in reality it may not be possible or feasible to create a fully integrated test environment. The alternatives such as test harnesses, manual checking of files etc, all need to be assessed when planning an RBT approach as this may impact the risk assessment itself if you are not actually using the correct systems.

Heatmaps and any RBT approach cannot be performed by testing in isolation, and so it is essential to get full buy in from both business subject matter experts and technical resource.

This process is about getting all relevant parties involved from the outset in achieving the common objective of agreeing the scope, priority and focus of System Integration Testing.

T.E.S.T | March 2010

Neil Pandit Senior test consultant Sopra Group www.sopragroup.co.uk

developments: For incremental developments, your RBT approach may change as you may have new systems and interfaces being developed for each increment. If an increment is adding new interfaces and functionality, the heatmap can be revisited to consider the new impact of the increment.

The key points

This has been a brief introduction to using heatmaps for providing a visual approach to RBT but the key points to take away are: Agree the objectives of RBT: RBT will continue to be used within organisations, but if not applied correctly will be disregarded by the same people it is intended to benefit Other uses of heatmaps (project managers, stakeholders, the So while this article and approach has business users). focused primarily on using heatmaps More effective focus of System for system integration, this same Integration Testing effort: We need technique can be successfully applied to have more effective focus on to other areas: System Integration effort as poorly Can be applied to prioritise system testing: We have applied this approach planned System Integration testing and incorrect focus could result in at a system integration level but this significant time and effort being information can also help prioritise wasted. any lower level testing such as Heatmaps - a possible solution: This system testing ensuring that there is maximum coverage within the specific approach can provide an objective riskbased and visual approach to System systems.

Integration Testing which combines Application to third party software: both the technical risk with the You can also apply this to third party business impact. commercial off-the-shelf (COTS) Ensures common understanding: It products where there may be a core provides clarity to stakeholders over product and you may want to know what has and what has not been tested which components have changed and can assist with reporting progress as a result of the bespoke changes / visually. configuration that is required.

Finally, in terms of the key challenges Prioritisation of defect fixing: We can use the testing priority of systems and with this approach, we must remember interfaces to help with determining the that the heatmaps and any RBT priority of both project and production approach cannot be performed defects such that the approach can be by testing in isolation, and so it is essential to get full buy in from both used to prioritise the order in which business subject matter experts and defects should be fixed first. technical resource. This process is Regression testing prioritisation: about getting all relevant parties Through identifying prioritised involved from the outset in achieving business processes and end-to-end tests it can also inform you of what to the common objective of agreeing the scope, priority and focus of System include in your regression pack.

Integration Testing.

Reassessing RBT for incremental www.testmagazineonline.com

t.

18 | Automated testing

Automation: your check mate According to Keith Braithwaite, principle consultant with Zuhlke Engineering, much of the routine checking that makes up the bulk of testing is ripe for automation, freeing up testers for more important, high-value work.

T

o obtain the highest confidence in a software system at reasonable cost, in terms of correctness and robustness, that system must be tested. Other ways of gaining high confidence do exist but are rarely economic given the state of the art.

That may change as the capability and ease of use of formal verification techniques continues to improve.

Right now though and for, say, the next five years at least, testing will T.E.S.T | March 2010

remain a keystone activity for systems builders. A shame then that so much testing effort is wasted, and that so many opportunities to gain greater value from testing are missed.

How is testing effort wasted? It is wasted by devoting so much human attention to checking, rather than testing. Checking is a valuable activity, so it’s good that so much of it is done. But it is also exactly the kind of activity that people are bad at, and so it’s a shame that so many of them www.testmagazineonline.com

Automated testing | 19

and have as their main responsibility interaction with other systems (the user is a system) should not also have responsibly for functional processing.

The UI should have no business logic in it. Servlets exposing a web service should have no business logic in them. Instead, the computation that Checking produces functional correctness should Go and look at your testing be held in some interior components. department, you'll find that a lot This is not to say that the ‘domain of their activity is really checking. model’ beloved of object-oriented Showing that the system as built conforms to a specification is checking. programmers is always the right All verification that a system provides answer. Those interior components could very well be manifested as outputs in agreement with the database tables and procedural code specification when provided inputs (although that hopefully not in stored that the specification states are valid, procedures), or a declarative rule set, is checking. All regression testing is checking, checking that the behaviour or a collection of transformations to a data stream. The technology and of the new system agrees with that architectural stance can vary widely of the old. Again, this is good and valuable activity, but people shouldn’t but the overarching principle is to observe that a system has an inside be doing. and an outside with a surface between Checking can, and should be, highly automated. A very few characteristics them (please, not a ‘stack’ with a ‘top’ and a ‘bottom’). That surface of a system, particularly those has transducers on it that mediate surrounding the user experience, communication across the surface of are hard to automate. Others are the system, and functional correctness not. All ‘functional’ requirements with respect to the problem domain can be checked automatically, that’s lives inside the surface; perhaps far pretty much what ‘functional’ means: inside. the output can be described as a A checker (are you getting used to mathematical function of the input. It that by now?) who can only determine might not be trivial to check that the functional correctness through the functions which the system actually external interfaces of the system, via implements are the ones which the specification requires, but it is possible, those transducers, is like an eighteenth century doctor trying to diagnose and desirable. a patient through purely external Go look at your test organisation observations. again. How much of the checking, Of course those transducers, such now that you recognise checking, of as UI components, must be checked functional correctness is being done themselves. But this is really the job through the external interface of the of a UI toolkit developer or vendor. system (the user interface, if there is “Does the list box correctly display the one)? This is a very common and very entries in its backing data structure? inefficient way to determine that the Does it correctly pass on events computations done by the system representing user gestures?” These provide the expected output for questions and other like them should acceptable input. have been answered long before that Current good engineering practice UI toolkit was accepted for use in recommends that those components building a system. The better focus of a system which sit at its periphery spend their working day engaged in it. Checking involves sustained close attention to many repetitions of almost identical actions. That’s hard for people to do well, but easy for machines.

www.testmagazineonline.com

Checking is a valuable activity, so it’s good that so much of it is done. But it is also exactly the kind of activity that people are bad at, and so it’s a shame that so many of them spend their working day engaged in it. Checking involves sustained close attention to many repetitions of almost identical actions.

That’s hard for people to do well, but easy for machines.

March 2010 | T.E.S.T

20 | Automated testing

for the checker’s attention is: does the logic correctly update the backing data structure when a certain kind of event takes place? This requires access to the internals of the system in the checking environment.

Stepping through Again, how many of your checkers are running through scripts? Howe many of them are doing this by hand? I hope very few. The majority of commercial ‘test automation’ tools really provide a facility for automated checking at the system boundary by stepping through scripts. That’s much better than manual stepping through scripts, but stepping through scripts is not especially valuable in the general case.

Sometimes it is required. Sometimes correctness of a system relative to a specification is all about sequences of steps, sequences of question/answer interactions at the system boundary.

But not always. Stepping through the process of entering a deal into a trading system and then stepping through the process of the end of day batch job is a very inefficient way of finding out if the pricing engine gives the right answer for that trade.

If scripts are not always the answer, what to do in the other cases? Recall that what we check is functional correctness and the word “functional” gives us a clue. A mathematical function can be expressed as a certain kind of mapping between two sets of values, the domain (usually corresponding to inputs) and the range (usually corresponding to outputs). The function itself is then a subset of the Cartesian product of these two sets.

In all but the most trivial cases this set is intractably vast. This observation is the origin of the oft-repeated claim that no amount of testing (including checking) can prove the absence of defects. Of course, that is literally false: one can calculate the exact amount of testing which would prove the absence of defects. Doing that immediately shows that there is no possible way of doing that amount of testing. Try this for yourself—imagine you are to check the multiplication T.E.S.T | March 2010

function in your favourite language.

Take the square of the number of distinct numerical values that a variable in your language can hold, this is the size of the set that the multiplication function exists in. Could you possibly do that many checks, even for so simple a function?

In fact, through the use of large multi-processor machines such exhaustive checking has been tried in very simple cases and did find some interesting defects that might have been hard to find any other way.

But you can’t afford to do it for your system, I confidently assert.

However, those who work as checkers have a lot of expertise in finding interesting cases to check, to finding cases that are strongly representative of a large equivalence class. Of finding cases that sit near the awkward corners of both the problem domain and the proposed solution embodied by the system. It is possible to intensively check those illustrative examples of desired system behaviour.

And with smart automation which, like modern diagnostic equipments, can look inside the ‘patient’ the checking of those examples can be very quick.

Missed opportunities What opportunities are being missed?

‘Checking’ is by definition checking against a specification or requirement.

Let’s say that we are going to gain confidence in the correctness of a system by intensive automated checking. We know from Royce’s original waterfall process paper of forty years ago that putting what he calls testing at the end of the development lifecycle “is risky and invites failure”. Royce recommends that testing starts as early as possible, certainly before development is complete. And, as it happens, he recommends that coding starts before design is complete and that design starts before analysis is complete and that analysis starts before requirements gathering is complete and he recommends to do these activities more than once so that we can learn through experience. www.testmagazineonline.com

Automated testing | 21

It’s a shame that more of the authors of software development standards that cite the waterfall process seem not to have read Royce’s paper, where he tells us that a waterfall process is largely bogus and what to do instead.

Anyway, in the limit of Royce’s recommendation we might very well want to start testing (checking) very early. Very, very early. There is a great opportunity here. Since checking is checking against a requirement and smart checking is against illuminating examples of required behaviour, then producing examples to check can become part of the requirements activity.

If we do this, then several advantageous side effects arise.

It becomes hard for requirements to be vague, since an example must be explicit. It becomes hard for requirements to be dropped in to a document without thought because producing examples requires careful thought. It becomes hard for requirements to be hard to check because we can even make it a project rule that a requirement without examples to check is not accepted.

The checkers are now up at the front, at the business end (literally, the business facing end) of the development process, bringing their skill and expertise at finding illuminating examples to bear at exactly the point where survey after survey shows that project after project goes off the rails—during requirements capture. Most projects fail, and most projects that fail, fail not because the developers did a bad job of building the system (although that does happen) but because they turn out to have built the wrong system.

If requirements come with (in advanced teams, requirements are) examples to check, and if smart tooling allows examples to be checked quickly, reliably and intensively, then a much smarter way of running projects becomes possible. This can extend as far as reporting project status.

Count the examples crated: this measures something like scope to be delivered. Count the examples that www.testmagazineonline.com

check against the system as currently built: this measures something like scope delivered. The first time I had a project report to its board this way as it happens the news was not good but a senior project manager on that board said afterwards that this was the first status report from a project he had ever seen that he believed. And because the board believed the report they were able to respond and assist that team to improve. Which they did, and they had the evidence to show it.

What’s left to test?

That’s a lot about checking, so what is left for testing? Michael Bolton has written extensively about this distinction and he characterises testing as what happens when “we’re trying to find out about the extents and limitations of the product and its design, and when we’re largely driven by questions that haven’t been answered or even asked before” Note that part about questions that haven’t even been asked before.

Testing isn’t about the requirement, it’s about things that aren’t in the requirement. Checking an example gives a binary result, the output was as expected or it wasn’t. Doing a test can give any kind of result at all, but mostly doing a test results in learning.

It might result in learning a more interesting question to ask next time.

This is not a kind of activity that lends itself well to automation.

Machines don’t (yet) have the imagination required to figure out new and interesting ways to stress a system, nor to usefully interpret the very wide range of results that might ensue. Bolton cites James Bach who characterises testing as the action of “questioning a product in order to evaluate it” and notes that this is not a quality assurance process (checking is) but a process which informs quality assurance.

Testing is a very high value activity that requires very high value people to do it. And they will be free to if all the checking is automated away from them.

Keith Braithwaite Principle Consultant Zuhlke Engineering www.zuehlke.com

In all but the most trivial cases this set is intractably vast. This observation is the origin of the oftrepeated claim that no amount of testing (including checking) can prove the absence of defects. Of course, that is literally false: one can calculate the exact amount of testing which would prove the absence of defects. Doing that immediately shows that there is no possible way of doing that amount of testing.

March 2010 | T.E.S.T

22 | Testing automation

Is automation the answer?

Efficient management of resources is in particular a considerable challenge for network and communications testing, especially in large or distributed testing labs. Eitan Lavie, director of technologies at QualiSystems, explains the benefits of an automated resource management system.

F

or network equipment manufacturers and service providers, network testing is a mission-critical, multifaceted task that is essential if products and services are to be high quality and priced competitively. Testing efficiency and efficacy directly impacts time-to-market, product quality and overall profitability including capital and operational expenditures.

Testing teams find themselves needing to manage thousands of local or networked resources. These resources can be used by a single station or shared between multiple users, in a single location or across different labs. If these expensive resources are not managed properly, they are under-utilised or end up in usage conflicts within and between test teams. This translates directly into expensive bottlenecks or delays.

Resource management challenges Let us consider in more depth some of the issues that can arise from a lack T.E.S.T | March 2010

of effective resource management.

They add up to a pretty long list of problems for the test engineers themselves and for the organisation as a whole. Some of the bigger issues include: Utilisation and cost: Equipping and running test laboratories is a costly operation. Physical resources - equipment like traffic generators or switches can run into millions of dollars, and are expensive to replace if they are damaged or become outdated. Logical resources (managed functionalities like switch node configurations or software licenses) are often costly as well. So any level of utilisation under 100 percent is an instant money-loss for the organisation.

Setup: As network environments and equipment become more complex, it gets more and more time consuming to configure and set up equipment for testing. Required configuration adjustments for different test batches or Devices Under Test (DUT) often mean our engineers have to manually wire www.testmagazineonline.com

Testing automation | 23

up and re-cable the equipment, and sometimes this can lead to time consuming configuration errors.

Sharing: In most organisations, test resources must serve multiple test teams. Sharing these resources can be quite frustrating. Making sure the test equipment you need is available when you need it, and furthermore – guaranteeing your test is not being interrupted just a second before it ends, is sometimes an impossible task. Resource use conflicts between individual testers and testing teams, plus inefficient use of expensive equipment result in longer, unpredictable testing times and efforts.

Control: When sharing resources across large or distributed test labs, resources are often stored in testequipment farms. Control over the equipment becomes increasingly difficult as more and more Layer 1 Switches are required to automatically manage resources cabling and connections. Control gets even trickier as large clusters of switches with complex topologies are required.

Centralised, automated resource management Meeting with network equipment manufacturers and telecom service providers’ testing teams reveals these challenges are faced on a daily basis.

While trying to figure out potential solutions to these pains, one can consider that the continuous evolution of automated testing solutions can be deployed to the rescue here as well.

Testing teams can improve the way they manage multiple test resources, while save time and avoid frustrating www.testmagazineonline.com

efforts by using a centralised, automated resource management system.

Some of the crucial issues I mentioned above can be solved by ‘good friends’ like a scheduling and reservations mechanism, an application that automatically calculates switch topology paths and a well-managed cluster of Layer 1 Switches.

Automatic topology configuration: As mentioned earlier, to save re-configuration and manual re-cabling efforts, Layer 1 switches can come in very handy. The Layer 1 switch provides a wire-once technology, which users can use to connect any port to any other port within the system using software commands. Once the lab network and equipment is connected, test topology changes are accomplished by a simple mouse click, and there is no further need for manual cabling.

As multiple labs share the same resources, multiple Layer 1 switches are required and topologies get complicated once again. One solution would be to wrap all the switches into one cluster or cloud and then control it using a background service to automatically calculate the optimal topology, according to the test requirements and definitions. Ideally, the test themselves remain unaware and unaffected by the solution in the background, and doesn’t require any modification.

Automatic reservation and scheduling: An automated service that can lock and release test resources according to predefined rules can help avoid those situations where different

An automated service that can lock and release test resources according to predefined rules can help avoid those situations where different test engineers are stepping on each others’ toes, and interrupting each others’ tests. With all resources under a centralised repository, managing resource reservations between test executions, test development teams and other resource customers is highly effective and helps eliminate resource usage conflicts.

March 2010 | T.E.S.T

24 | Testing automation

test engineers are stepping on each others’ toes, and interrupting each others’ tests. With all resources under a centralised repository, managing resource reservations between test executions, test development teams and other resource customers is highly effective and helps eliminate resource usage conflicts.

On top of a locking service, a scheduling capability can upgrade lab control. Being able to automatically schedule and allocate specific resources at specific times, can help guarantee optimal resource usage, and minimal test interruption.

However, some flexibility is required.

For example if a test overruns its allocated time and requires an additional few minutes to finish a long and complex test process, it would be a real shame not to allow it to run to completion by reallocating a resource to another test. In order to avoid cases like this and to assure that the test with the highest priority gets all the resources it needs, a rules engine that defines a hierarchy for tests and resource users, and that also provides an automated conflicts management system is a great plus.

Coordinated Execution: When dealing with growing multi-site and large scale testing environments, achieving timely and cost effective test execution requires not only well managed resource sharing but also control over test stations, test requirements and an ability to match and coordinate all these components together. In order to automatically allocate tests to the appropriate available station, we need a centralised application that can dynamically T.E.S.T | March 2010

abstract resources by defining their structure (chassis, card, port and so on) and capabilities like protocols and rules etc; and to remotely control all resources and stations. With these tools, a well-coordinated execution can be achieved.

Overcoming the challenge To return to our initial set of challenges, we have seen that in network testing labs where multiple users design and execute large numbers of tests for different product components, poor test resource management directly impacts on test efficiency and from there affects product quality and overall bottom line profitability. To maximize testing, test labs need to share their resources efficiently and avoid lengthy test setup times.

The centralised automated resource management system that I describe, combining dynamic resource definition, resource reservation and locking, and automatic test scheduling and topology configuration, with remote control over test resources and test stations for coordinated execution, is not just some futuristic concept. It is a concrete solution that leverages the value of physical, logical and human test resources and increases overall profitability by maximizing test lab capacity.

In the final analysis, an automated resource management solution can help manufacturers and service providers achieve consistent high quality products and services that will ensure their ability to succeed in the highly competitive world of networking and telecommunications.

Eitan Lavie Director of technologies QualiSystems www.qualisystems.com

www.testmagazineonline.com

26 | User testing

Your most valuable software testing tool There’s no excuse in this day and age for a non user-friendly interface and often the only source of reliable insight into this area is the user.

Dave Tait* puts the case for the user as invaluable testing tool.

G

aining maximum market share with a software product means selling to inexperienced as well as experienced users and making both of these types of user feel like it was designed with them in mind. The ‘Windows 7 was my idea’ campaign appears to demonstrate that Microsoft have taken this on board, in a move reminiscent of Nintendo’s creation of the Wii, a games console even non-gamers can use.

Software developers themselves can be hamstrung by their own expertise when it comes to testing since by definition they only represent expert users. That is why it is important to get end users involved in the design process as early as possible. Not only can they find bugs in software by interacting with it in unexpected ways, but they can also reveal fundamental problems with its usability.

T.E.S.T | March 2010

Blame the user In the past, users accepted the blame for their inability to use technology. It was seen as the domain of people with knowhow and if you were struggling with a piece of technology it was either because you were too lazy to read the manual or just plain stupid.

Oddly, this acceptance of blame extended even to devices such as VCRs and central heating controllers, devices which, on the face of it, should not need specialist knowledge to operate.

Designers, manufacturers and even suppliers all seemed happy to go along with blaming users. However, with the increasing ubiquity of electronic devices and electronic interfaces in general, including web pages, it is becoming evident that this belief is increasingly untenable. There are just too many interfaces to learn and not enough time to consult the manuals for all of them.

Of course, many of the usability problems of older technology were associated with the cost and size of components, yet others were plainly down to a lack of user testing.

Even now parking ticket machine manufacturers, for example, come up with some of the most baffling designs; designs that suggest that their machines are never used hurriedly or in a rainstorm, by non-native speakers, visually impaired people, or by people who are simply carrying something.

Software interface and website design can be just as baffling for the same reason - that testing was limited to making sure the product did not fail. While catching bugs is a crucial part of software development, the fact that software works is something that users have come to expect as standard. They are now evaluating software based on whether they had a positive or negative experience of using it, not just whether it worked at www.testmagazineonline.com

User testing | 27

all. Building the product right is not the same as building the right product.

Why should this concern developers?

Because customers are being presented with an increasing number of products to choose from and those that are easy and even pleasurable to use will win out. Customers implicitly blame the developers of those products they cannot use by choosing products that they can use instead. Gone are the days of struggling manfully with technology solely because it has superior technical specifications to its rivals.

all the users’ requirements that makes software unusable for the target user group, but also the extra complexity and cost of unnecessary functionality.

In UCD, prototypes are developed and presented to users only after the user requirements have been established. In the early stages there may be many prototypes and they will probably be lo-fi, perhaps consisting of sketches or storyboards. The most appropriate prototype may go through many design iterations, depending on the responses of users, as it progresses towards a hi-fi prototype.

Users are consulted throughout Keep it simple the whole design process from early Even large companies purchasing bespoke software are coming to realise concept phases to the final product, that simple-to-use systems reduce the with each design iteration being traceable back to user goals. Where costs associated with training their possible, users are studied interacting personnel to use them, as well as the with the prototypes in order to costs of onsite helpdesk provision. highlight any usability issues.

Simple systems can also increase productivity due to fewer errors in Historically, system design and user-system interaction. testing has tended to focus on Software focussed on users’ functional requirements – what the needs is also cheaper to develop system should do. UCD expands and maintain. According to a 2003 this focus to users as well. It takes review of surveys on software into account their varying attitudes effort estimation, the respondents towards and abilities with computers. (typically project managers) in one Systems are designed to be helpful, survey believed that half of cost memorable and clear for novices or schedule overruns were due to changes in design and implementation. and infrequent users, while retaining power, flexibility and speed for expert Another survey suggested overruns users. were caused by frequent requests By taking the UCD approach for changes from users, overlooked developers get access to two very tasks, users’ lack of understanding valuable resources – expertise in the of their own requirements, and poor process that the software product communication and understanding seeks to streamline, and a sense of between users and analysts. Fixing what the solution looks like from the a problem after a product has been point of view of someone who has no released is far more expensive than idea what is going on behind the user identifying it at the design stage. This interface. In addition, involving users is where a user-centred approach to early on in the design process means software design comes into its own. that developers can make sure they are building the product their customers A user-centred approach need, and instil a sense of ownership User-centred design (UCD) involves of the product in the users. This can users in the development process drastically reduce the cost of the right from the start. Users’ tasks product since the maintenance phase and goals are the driving force of is often the most expensive phase of UCD and it establishes what these the software lifecycle. This in turn is are with questionnaires, interviews, focus groups and so on. Users’ current good for customer satisfaction and that has a positive effect in terms of behaviour and the context of the customer loyalty. product’s use (for example whether it is to be used in a noisy environment) * Dave Tait is a recent graduate are also studied to see if this will of the University of Sussex where affect the design. The combination of he studied Human-Computer this information establishes the user Interaction as part of a Music requirements. Getting these right is Informatics BSc. crucial as it is not only failing to meet www.testmagazineonline.com

Dave Tait davidatait@googlemail.com

Software developers themselves can be hamstrung by their own expertise when it comes to testing since by definition they only represent expert users.

That is why it is important to get end users involved in the design process as early as possible. Not only can they find bugs in software by interacting with it in unexpected ways, but they can also reveal fundamental problems with its usability.

March 2010 | T.E.S.T

28 | User testing

Putting the user at the centre of things A strategy that puts the user in control at the centre of things is, according to Zabisco director Hammad Khan, the principal benefit of a user-centred approach.

T.E.S.T | March 2010

www.testmagazineonline.com

User testing | 29

H

ave a conversation with almost anybody and most would agree that making your end-users perspective a leading factor in how you approach your business makes sense. So why is it that when you look closely, so many businesses (and non-profit organisations and public bodies for that matter) don’t actually adopt this seemingly obvious idea?

Taking ‘The user’s point of view’ doesn’t have to be quite as opened ended as it sounds. The idea of User Centred Design (aka UCD) is a well-established practice; it even has an ISO standard (ISO 13407), so your managers can be confident of implementing it into existing business process. When I’ve been asked to come into an organisation and get them to start taking a UCD approach to a project, we’ve found it to be one of the easiest change management practices to go through, as it’s driven by common-sense as opposed to other methodologies that rely on a decision making process that not everybody agrees on. Of course, it also delivers the best user experience at the end and that is no bad way to create a closer and more engaging relationship with your customers.

Collaboration User Centred Design is not just about ‘blue-sky thinking’ over what people might want (which can be dangerous in its own right as opinion quickly becomes fact during internal brainstorms). UCD encourages the inclusion of your target audiences in the process. Requirements capture, prototyping, revision cycles, testing and ongoing feedback – almost every part of a project’s lifecycle is an opportunity to include your users into www.testmagazineonline.com

the process and keep things focused in their favour.

The result is that you and your team are collaborating, rather than competing on how to drive out your business goals more effectively; with the end users being the grounding voice throughout. All those well established business mantras like ‘the customer is always right’ exist for a reason, so adopting UCD allows us to pay homage to this, while being realistic at the same time.

One way to quickly see how much UCD influences a company’s outward facing communications and services is of course by looking at its website. Being a digital agency, this is something we analyse and implement regularly. How a business chooses to label and organise its content and the navigation around it is often a good indicator as to how much the end users have been taken into consideration and whether this has been given preference over the needs of the business itself.

Those taking an inside-out view will often have their navigation and calls to action reflecting how their organisation is structured; its departments, locations and sales contacts. Conversely, organisations who have taken a more user-centred perspective are more likely to have a navigation that better reflects the needs of their audience; how they can help, softer language etc.

Good intentions So, why is it so many businesses don’t get this right? Is it because they don’t care about their customers?

That’s not often the case. Projects often start with good intentions, but the user’s needs and viewpoints get overshadowed by politics and

So, why is it so many businesses don’t get this right? Is it because they don’t care about their customers? That’s not often the case. Projects often start with good intentions, but the user’s needs and viewpoints get overshadowed by politics and personalities.

March 2010 | T.E.S.T

30 | User testing

Seldom does one group or department back down in favour of another in a ‘straight up’ fight. By using UCD as the methodology throughout, it’s a lot easier (and quicker) to get the compromise needed, as everybody knows up front that if a particular feature, function or requirement isn’t high on the audience agenda it can be re-thought.

personalities. Working in corporate web design and development, this is a pattern we come across regularly and it’s easy to see how it can occur.

In a typical organisation, there are usually competing groups vying for the space, prominence and functionality that meet their specific needs: a marketing department wanting to push the latest product line for example; the CEO wanting to communicate the new strategy; or the Finance team needing to publish reports. Each a valid business need, but ultimately serving that of the organisation first, rather than what their audience needs.

This internal struggle also comes with political connotations and in extreme cases can even get personal.

Seldom does one group or department back down in favour of another in a ‘straight up’ fight. By using UCD as the methodology throughout, it’s a lot easier (and quicker) to get the compromise needed, as everybody knows up front that if a particular feature, function or requirement isn’t high on the audience agenda it can be re-thought.

Similarly if the terminology used for navigation isn’t driving the user to their goal, then it can be re-architected. An instant business benefit of taking this approach is to getting a better return on investment (ROI) from your pay per click advertising on sites such as T.E.S.T | March 2010

Google or Bing. Why waste money on bringing traffic to your website, if people will get lost, confused and ultimately leave before fully engaging with what you had to offer? This kind of optimising is something we do a lot of and it doesn’t always have to come attached to a big redesign project either.

To give an example, getting the balance right can be as simple as promoting a product’s benefits instead of listing its features; a subtle difference, but one that engages in a totally different way. Clearly thought out benefits can inspire somebody to think about how a product or service fits with them as an individual.

Defining features relies on the hope your market can find a connection with it. In a world that is ever more competitive and sceptical, the question is – why risk it?

User-focussed structure In website design (or more accurately, information architecture), one of the most common patterns we see is clients who want to know why a navigation structure that that looks similar to example A below is not working well for them (ie low traffic, poor sales, high drop-out rates etc): About us | Products | Services | Partners | News | Contact us Simple enough and not wrong in any tangible way, but actually this www.testmagazineonline.com

speaks volumes about how the organisation thinks – pretty much about themselves predominantly! By looking at these choices, the user is not educated about what’s on offer or whether the information they are after is even available.

Let’s apply this to a fictitious business as an example – say Acme Watches.

If a potential customer came along looking for the latest prices, would they be in ‘products’? Maybe, but what if Acme Watches doesn’t sell direct to the consumer and you have to find an authorised reseller (which happens to be inside ‘partners’). The user might search around and find the appropriate content they need, but more than likely they’ll switch off, or turn back to Google for a new and more obvious direction.

If however Acme Watches got the team together and brainstormed on

how to better communicate their proposition, they may have come up with something like example B: Who we are | What we do | How we can help | Who we work with | Get in touch If however, they adopted UCD and took both internal and external perspectives, they’d more likely end up with something not too different from example C. About Acme Watches | Mens’ Watches | Ladies’ Watches | Repairs & Servicing | Where to buy Again, a small change, but a huge difference when it comes to keeping users engaged. I’m sure I don’t need to patronise by spelling out the improvement, but if you know of a sceptic and fancy making a quick few quid, put these examples through some A/B testing and put your money on C!

Hammad Khan Director Zabisco www.zabisco.com

Clearly thought out benefits can inspire somebody to think about how a product or service fits with them as an individual. Defining features relies on the hope your market can find a connection with it. In a world that is ever more competitive and sceptical, the question is – why risk it?

Over half Of the UK’s tOp 50 e-COMMerCe sites* test with Site ConfidenCe. shOUldn’t yOU? *source: comscore Media Metrix (dec 09)

For web site monitoring & load testing: Call 08445 380 127 Email info@siteconfidence.com Web www.siteconfidence.com

siteconfidence an ncc group company

32 | Testing environment

The case for customisation It’s high time our cumbersome monolithic operating systems offered increased agility, enhanced security and lower support costs. Steve Harris, senior director for Data Centre Solutions at Novell puts the case for customisation.

F

or far too long, users have been bogged down with cumbersome operating systems (OS). Mass customisation – the process of customising a product to meet individual needs while leveraging mass production efficiency – is now commonplace in the manufacturing of everything from cars to laptops. Yet, the OS has remained monolithic, even while there has been an explosion in hardware platforms, mobile devices and applications.

To date, the ability to customise, test and deploy a tailored OS for a

T.E.S.T | March December 20102009

specific use case or device has been slow, expensive and complicated.

Furthermore, those who customise the OS do so at their own risk, as the result of the process is often unsupported by the OS vendor. This changing now as IT departments now recognise the need to tailor the OS to gain agility, drive down support costs and enhance security.

This is particularly true in the Linux environment where new tools are making the promise of a tailored, fully-supported Linux OS a reality. In fact this is one of the most compelling elements of Linux: its flexibility and

adaptability. With tools such as Novell SUSE Studio and SUSE Linux Enterprise JeOS, developers and ISVs can now take just the components they need to run the application or workload. As virtualisation and cloud become ever popular the desire for stripped down, configurable systems will grow. This new paradigm is part of an emerging market we call intelligent workload management (IWM).

The drive toward mass customisation Historically, major OS distributions have been notorious for their bulk www.testmagazineonline.com

Testing environment | 33

and complexity. The time, effort, and resources required to install, maintain and upgrade a complete OS are substantial. Operating systems are designed to support every possible hardware device and software function, yet most applications require only a fraction of the full capabilities of these heavy operating environments. The extra, unused assets become a liability from a security and management perspective. For example, in financial institutions, everyone from bank tellers to the CEO often have access to the same operating systems, with far more than is needed for the majority of the staff. Given this level of access, an unauthorised employee could unwittingly walk away with proprietary customer information, which he or she should not have had the ability to access in the first place.

The infiltration of computing into everything from cars to coffee makers is driving new mobile and desktop computing platforms to be fully customised for the user experience.

This mindset is also permeating into the OS as users are beginning to demand that the OS provide the same fully supported mass customisation they have come to expect elsewhere.

A better approach is to create an environment using only the OS resources needed for a particular application set. This is known as JeOS (just enough operating system). The question is: How do we move to a JeOS environment?

Mass customisation and Linux Ironically, although Linux is modular and is designed to be customised, relatively few organisations have taken advantage of these capabilities due to a lack of vendor support. That’s why businesses are moving toward the new approach of mass customisation.

By using the existing building blocks www.testmagazineonline.com

within Linux, and applying them in different combinations based on each user’s needs, businesses of all sizes as well as the public sector gain the benefit of a tailored OS environment with the confidence of having a fully supported OS. Linux’s modular architecture is ideal for creating JeOS because it can be easily stripped down and modified.

Highlighting the shift towards open source, Linux and indeed, JeOS, is the recent decision of the UK Government to look at open source as the building blocks for its future IT requirements.

While several major Government departments including the NHS, which runs most of its core network on Linux, have already embraced open source, the new proposal indicates a major move away from relying on proprietary software and to embrace open source.

One of the key drivers behind this move is the desire for the UK Government to embrace the open source culture of sharing, re-use and collaborative development across government and its suppliers. At the core of this is the Government Application Store (G-AS), which is geared towards re-using programmes that have already been written and can be re-applied. This approach works as we can testify with our own Appliance Program, which has seen over 250,000 appliances built in SUSE Studio since its launch in mid2009 and tens and even hundreds of thousands of downloads of these appliances. Sharing and re-use is the power of Linux.

The benefits Mass customisation leverages the cost benefits and efficiencies of mass production, coupled with the ability to tailor an OS for different use cases.

Importantly, this tailoring is achieved through the arrangement of standard

Highlighting the shift towards open source, Linux and indeed, JeOS, is the recent decision of the UK Government to look at open source as the building blocks for its future IT requirements. While several major Government departments including the NHS, which runs most of its core network on Linux, have already embraced open source, the new proposal indicates a major move away from relying on proprietary software and to embrace open source.

March 2010 | T.E.S.T

34 | Testing environment

Linux components and it is fully supported by the community and vendors alike.

One of the most obvious benefits is the ability to deploy a compact OS quicker and more easily than a fullblown version. In the just enough OS model, deployment and testing time can shrink from weeks to minutes. A preconfigured, self-contained package with no external dependencies is easy to transfer through testing into the production environment, and is highly portable between virtual environments, making it beneficial for disaster recovery and other scenarios.

As we move to an IT model that will have physical, virtual and cloud environments, this easy portability is critical and one of the central reasons why developers, ISVs and IT directors must look for a strategy built around IWM. Another benefit is that mass customised Linux includes only the packages needed for a computer to function properly for a given use, vastly simplifying maintenance. By replacing a cumbersome OS with a smaller set of packages, IT staff benefits from a simplified software management lifecycle.

These compact, finely tuned JeOS environments allow applications to run more efficiently, with all components tested and optimised to run together.

This is similar to how devices such as DVRs and wireless access points work. They are purpose-built devices equipped with only the OS and applications needed to perform a specific set of functions. As the devices run on a ‘trimmed down’ OS, they are easier to set up, operate and maintain.

With mass customisation security is a critical component and this where the need to look for solutions that can blend OS and security together. With businesses needing to comply with a host of regulations organisations need to control access to sensitive data. By deploying a trimmed down environment tailored to include only the applications, components and associated data needed by each user, organisations can build a foundation that prevents access by unauthorised users. Moreover, since JeOS contains T.E.S.T | March 2010

fewer packages, there are fewer attack points for potential intrusion.

Finally, mass customisation helps IT departments become more agile as they are able to speed the time-todeploy new servers and respond to changing business needs faster. New use cases can be precisely configured and deployed in near real-time to address market shifts. Additionally, these highly portable environments can be scaled up or down to rapidly respond to resource peaks and valleys.

As a result of mass customisation, IT is better positioned to tailor and adapt its systems and processes quickly and easily in order to adapt to business demands.

The future of mass customisation In the future, virtually all computers will be tailored to meet an individual’s unique profile, from the OS on up through the technology stack. The use of mass customised operating environments will proliferate as enterprises and public sector organisations realise the considerable financial, security and maintenance benefits.

Virtual appliances are one example of mass customisation, with $1.1 billion in revenue expected to be spent on software and virtual appliances by 2012 worldwide according IDC. Likewise, IDC predicted that there will be more than seven million thin client devices on the market worldwide by 2012. As demand for mass customisation of Linux increases, we’ll see businesses explore new use cases, from USB stick-based computers to cloud computing.

With the launch of software appliance tools and programmes, the days of monolithic OS is over. We are now able to create mass customised OS using Linux that will fuel a rise in new applications and services. In doing so IT will need to shift their mindsets towards building, securing, managing and measuring intelligent workloads within a mass customised Linux environment. As organisations look to become more fleet of foot and efficient, mass customisation of Linux is the logical path for IT.

Steve Harris Senior director for Data Centre Solutions Novell UK www.novell.com

In the just enough OS model, deployment and testing time can shrink from weeks to minutes.

A preconfigured, selfcontained package with no external dependencies is easy to transfer through testing into the production environment, and is highly portable between virtual environments, making it beneficial for disaster recovery and other scenarios. www.testmagazineonline.com

Testing tools | 35

Tools you can trust Automated testing tools can save time and resources in the rapidly expanding agile world. TestPlant senior systems engineer Allen J Fisher looks into the future of testing tools.

T

est automation is not going away. The faster turnaround of the web and the rapid development of new technologies such as the Apple iPad and Google’s NexusOne are requiring the tester to do more with much less. Added to this challenge are new technologies that are rapidly being developed to satisfy the need for not only functional, but usable applications.

As technology becomes more and more mainstream, graphically rich applications are required as users demand usability as well as functionality. This makes testing the true end-user experience even more important. Skinnable and customised user interfaces also add to the challenge. For example, iGoogle allows the user to add modules based on different technologies and then rearrange them to suit the individual user’s comfort.

www.testmagazineonline.com

It used to be you had Windows which ran on a big lumbering machine sitting on your desk. Now, with Apple and Linux gaining market share, testing more operating systems becomes crucial. Each time you add an OS without dropping one, you’re doubling your testing time.

A large chunk of my testing career has been spent testing a 20+ year old codebase that has to run on several different versions of Windows and the Mac OS. With a huge codebase with lots of interdependencies, adding a platform is no small investment, even a new version of the OS could introduce many problems that could triple testing time. Add to that things like smart phones, netbooks, iPod Touch, and the iPad and the amount of testing time can start to spiral out of control.

Another challenge in automated testing is the initial investment. The initial investment is really hard to

March 2010 | T.E.S.T

36 | Testing tools

justify as release dates loom. Managers may be willing to part with the money for an automated tool, but often the tool becomes shelfware—destined to sit on a shelf and maybe be used occasionally instead of being a critical part of any testing strategy. This is in large part due to an attitude of ‘we need it done now, not done right.’

QA blinders My experience is that testers just want to test (I am no exception). They want to hit the date, and tend to fall back into their comfort zone of working a few extra hours to meet the deadline. Also, a condition I’ve called ‘QA blinders’ can creep into the mix. This happens when a tester has looked at an application for so long that they do not see problems because they are doing the same test that they’ve always done and don’t see side effects. Tests need to be repeatable, to guarantee an accurate report of the quality level of a given release to management, regulators, certification boards, and users.

T.E.S.T | March 2010

My experience is that testers just want to test (I am no exception).

They want to hit the date, and tend to fall back into their comfort zone of working a few extra hours to meet the deadline. Also, a condition I’ve called ‘QA blinders’ can creep into the mix. This happens when a tester has looked at an application for so long that they do not see problems because they are doing the same test that they’ve always done and don’t see side effects. Tests need to be repeatable, to guarantee an accurate report of the quality level of a given release to management, regulators, certification boards, and users.

Also, cost to train a tester to use an automated tool can be quite high.

The automated tester often has to either have a computer science degree or needs to go through two to three weeks of training in order to have success with the tool that they’ve been tasked with. After that, it may still require developer intervention to ensure that the objects are available for the tool to work with. This plays to team dynamics. Testers and developers often have adversarial relationships that make working together or getting information difficult. A tool should be easy enough for a lower level tester to learn, but powerful enough for the

journeyman tester to use successfully.

Lastly, there’s platforms; many tools are Windows-centric. My testing career has been spent in a crossplatform, multi-environment world.

It is difficult to find tools that test older systems as well as cutting edge ones. This means that you will not get the full value of an automation effort if your tests only run on a single platform. In my specific case, I had to cover six operating systems: three versions of the MacOS and three versions of Windows. The code base was 20+ years old and largely the objects were custom, there was not good model separation, and much of the code was dependent upon which OS you were running. We could not assume that if a test passed on five platforms that it would pass on the sixth.

Tools you can trust I’ve been using our eggPlant on projects for the last seven years. I find it flexible and easy to use. To start with, it is a two-computer system, consisting of a controller machine connecting to a system under test using VNC technology: This makes it able to connect and test any system that supports VNC, including smart phones. It is a truly black-box automation tool, testing what the user sees, by using images of different screen elements to move about the screen as opposed to having to learn what objects are part of the codebase before being able to test it. This means that technologies that other tools can’t reach such as Flash, Flex, or Silverlight are well within its reach. Even if you already have a set of automated tests, it can piggy-back www.testmagazineonline.com

Testing tools | 37

onto that effort and fill in the gaps.

New platforms and technologies are also not a challenge due to this flexibility. Many tools require updates to support emerging operating systems or technologies. Since VNC support is the only requirement, new operating systems and platforms can be inserted into the testing process as soon as they are available, saving valuable time for other efforts.

This flexibility also lends itself to the problem of highly customisable, graphic rich user interfaces. If a screen is rearranged, such as the widgets on an iGoogle page being moved around, it will adapt and locate the image, wherever it is on the screen. As long as it can ‘see’ something, it can interact with it – exactly like a human would.

The image matching schemes are flexible enough to allow you to write a single script and be able to execute it on several systems, regardless of the OS. So with few changes to a script, you can have an automated testing system such as this: Having a single script do the exact same test on six platforms gives confidence that the code is clean regardless of platform. As an example, we had a test that took four hours to run manually. The test consisted of opening hundreds of proprietary files in a desktop publishing system for music and pressing the ‘play’ button to hear the song play. If the program did not crash while preparing to output sound, the file would begin playing back and the test was successful. By spending eight hours automating this tedious procedure, the time to execute this test dropped to 20 minutes, and could be run at any time at the press of a button. www.testmagazineonline.com

Ease-of-use is very important, especially for the tester new to automation. The software has its own proprietary scripting language called SenseTalk. It is a high-level, readable scripting language with all the features you’d expect from a language such as Perl or Python, but with the readability you’d expect to find in a manual test script. I like to say that SenseTalk is so easy, even a manager can read it. A very simple example of the SenseTalk language follows: The time to ramp up is very short.

When I first evaluated eggPlant for my project, I was able to write a simple smoke test of my application in a couple of hours. By the end of a week, I had a meaningful set of tests that I could show as a proof of concept to the R&D team. Other, single platform tools that I had evaluated had a much more significant learning curve, in the order of a couple of months.

Moving on In the future, I see the need for automation of tests expanding, not contracting, needless to say. teams are going to have to become leaner and faster, and the advent of things such as cloud computing are only going to make the demand for high-quality, quick turnaround only higher. Money and lives can be lost if software is not released at its highest quality.

The progression of software and technology from a select few to everyone having a computer in their home has brought with it a demand for highly visual, highly customisable experiences. The tools of the future will be those that can adapt to a rapidly changing world.

Allen J Fisher Senior systems engineer TestPlant www.testplant.com

March 2010 | T.E.S.T

38 | T.E.S.T profile

A common language for testing Mike Smith, managing director of Testing Solutions Group explains how Learntesting online training is helping to build a common language for testing.

L

earntesting was established as a separately branded online training division of specialist IT consulting firm Testing Solutions Group (TSG) in 2003. TSG is one of the largest providers of bespoke and certificated software testing training in the UK, as well as providing consulting and project services to its large network of clients in the UK and Europe.

At Learntesting, we wanted to establish an online training and global distribution service for software testing and certification aimed at both corporate and individual users.

With the rapid global expansion of the certification scheme, Learntesting could provide a highly accessible service and offer an opportunity for T.E.S.T | March 2010

those looking to reduce costs and minimise time out of the office.

The certification schemes are widely recognised as setting a benchmark for the industry, providing a ‘common language for testing’ across the globe and demonstrating a level of competence to employers. Over 130,000 people are now certified in the ISEB/ISTQB schemes and there are 47 member boards in ISTQB. Following the experience of two earlier implementations of online learning technology, the Learntesting Virtual Learning business model was developed to incorporate: • An online warehouse of high quality content to support the testing certification schemes accredited by BCS and the International Software Testing Qualifications Board (ISTQB). Due to the nature and scale of this online content warehouse, it needed to be supported by a robust and highly available learning management system; • A franchise distribution model that would allow the content and online training to be localised and implemented by global training providers and resellers; • e-commerce capability in order for corporate organisations and individuals to purchase Learntesting content as and when needed; • A multi-channel delivery solution offering a range of content and delivery options to support all levels of learning objectives and to prepare www.testmagazineonline.com

T.E.S.T profile | 39

students adequately to pass industry exams including: Online self-study tutorial videos, exercises, quizzes and other content; Live virtual classrooms (LVC) to support higher-level learning objectives, practical exercises, collaboration and exam preparation; Traditional classroom and blended learning options; eBooks.

• A 24/7 global support network, staffed by local accredited training providers.

A collaborative partnership Learntesting chose the CLIX learning management platform from IMC (UK) Learning Ltd to support the service.

CLIX was chosen as a key element of the Learntesting platform due to its ‘industrial strength’ engineering track record. Its professional user interface, highly configurable functionality and scalable performance made it the perfect choice to support the Learntesting business model.

Learntesting worked in collaboration with IMC who took the Learntesting business model and matched CLIX, to work to this. From this point on, IMC became heavily involved in the project providing consultancy and expertise.

Learntesting forged other collaborative partnerships to build its Virtual Learning solution: • HSBC global e-learning centre of excellence for training content development; www.testmagazineonline.com

• K wantlen University (Vancouver) for instructional design; • Skillsoft for a private library of testing and related ebooks; • iSQI and Pearson VUE for global exam service delivery.

Learning Styles There are many different ways that students can prepare for certificated exams. Study options range from simply reading the syllabus to sitting accredited courses, whether in the classroom or online.

There are many books, sample questions and other support materials widely available at a range of costs.

From our experience, a student’s best option depends upon their previous experience, the level of certificate they are studying and their preferred learning style.

Some experienced testers decide not to do any formal study, preferring to rely on their experience. While this works for some, others actually have real problems because their experience may be too narrow, or they have used a proprietary approach to testing not in line with the standards upon which the schemes are based.

Even with a prime objective of just passing the exam, choosing not to study can be a false economy leading to expensive exam re-sits.

With the above in mind, when providing online learning it is important to cater for a range of

We wanted to establish an online training and global distribution service for software testing and certification aimed at both corporate and individual users. With the rapid global expansion of the certification scheme, Learntesting could provide a highly accessible service and offer an opportunity for those looking to reduce costs and minimise time out of the office.

March 2010 | T.E.S.T

40 | T.E.S.T profile

backgrounds. The most inexperienced students will need a lot of material available and ways of reinforcing the knowledge that they are acquiring.

The more experienced students need ways of assessing their knowledge levels and easy access to the areas of learning in which they are weakest.

For all learners, it is important that courses provides an interesting learning experience. This is the most difficult challenge to anyone trying to build online training solutions.

Although there is an increasing range of ‘rapid authoring’ tools available, it requires expert subject matter knowledge, online content design skills, a lot of time and a large budget to build successful online training materials.

Interactive quizzes and exam assessments are critical and in many respects, the most important feature for the success of online training.

This involves a huge amount of development work, since setting good exam questions is hard enough – and providing the supporting information as to why each answer option is correct or not is an even bigger challenge!

With Learntesting, we have chosen to get our courses leading to certification accredited. This means we cannot take short cuts and need to cover all aspects of the syllabus as well as ensuring the online learning experience is good.

Students of Learntesting are buying a guarantee that our courses properly prepare them for the exam. Individuals T.E.S.T | March 2010

and organisations will soon be unhappy if the training fails to deliver this, and our pass rate at Foundation level of over 95 percent in the past five years is testament to the quality of the learning experience.

Implementation In April 2009 we market-tested our new service with limited distribution capability and gradually introduced more content, distributors and technical enhancements.

The Learntesting Virtual Learning Environment (VLE) is now available to anyone in the world, with subscribers able to self-register on the system and gain immediate access to a range of free content in the form of ‘Testers Treasure Chests’. This includes: • Sample online course extracts; • Sample certificated exam style questions; • Sample video extracts, high-quality papers and other value-added material;

Lifetime service A unique feature of Learntesting is that it is not just providing ‘one-off’ limited access to course materials and other content. It is designed to support corporations’ and individuals’ learning needs from start to finish.

The Virtual Learning Environment built

by Learntesting using the CLIX engine gives us full control over the content and service delivery, allowing us to offer the following: 1. Free evaluation, planning and

support service to help testers self-assess their capability, define their goals and plan their learning objectives, with selfassessment exam style questions; self-assessment questionnaires; and personal consultation with experienced, accredited tutors.

2. Free content in the form of ‘Testers Treasure Chests’ to help all testers in their day-to-day work and also provide an insight into the type of content they can purchase.

3. Guaranteed on-going access to content – while we recommend optimum study periods for our courses, they are all extendable with no extra charge. If we agree testers’ learning objectives, our guarantee is to provide all content until these objectives are met.

4. Alumni programme – Learntesting will be introducing an alumni programme in May, featuring ‘lifetime’ access to content after gaining certificates.

5. Private e-book library. Anyone purchasing any content from Learntesting automatically gets access for one year, 24/7, to the library of 50 testing and related titles. This incorporates many of the best known software testing books including some specifically written to support certification schemes.

6. 24/7 support provided by a global network of accredited training organisations, tutors and administrators, plus collaboration tools facilitating the creation of a Learntesting ‘family’. www.testmagazineonline.com

T.E.S.T profile | 41

It is designed to support corporations’ and individuals’ learning needs from start to finish. The Virtual Learning Environment built by Learntesting using the CLIX engine gives us full control over the content and service delivery.

7. A range of different content to suit learning styles, budgets and study availability, including courses fully accredited by BCS/ISEB and ISTQB. 8. Proven track record of pass rates – 95 percent at Foundation Level (compared with global pass rate of 75 percent).

9. ‘End-to-end’ service delivery.

• Self-register via a local Learntesting provider; • Select content (immediate access provided to a range of free content); • Subscribe for content using PayPal (or other methods); • Gain immediate access to subscribed content; • Self-register when ready, to Live Virtual Class (LVC) course components; • Exam vouchers provided for use in 5,000 test centres around the world.

10. The confidence to offer all TEST Magazine readers a full money back guarantee if anyone is not satisfied with the courses that they purchase (Quoting reference MSBS01).

Independent recognition – award finalist After only six months of operation, Learntesting was selected as a finalist for the prestigious ‘Learning Technologies Solution of the Year’ award from the Institute of IT Training and was presented with a finalist trophy at the awards ceremony held at the Dorchester Hotel in February.

Other key indicators of the early adoption and success of Learntesting include: www.testmagazineonline.com

• O ver 30 traditional classroom clients adopting the new service in the first six months with a high level of positive feedback; • Acceptance and delivery within the independent contractor market via Testing Agency Portal Owners; • Take up by large-scale global IT services providers; • Adoption by eight local portal distributors in the UK, Europe, Far East and Australasia.

Courses and content now available includes: • ISEB/ISTQB Foundation (English and German); • ISEB Intermediate; • ISTQB Advanced Level; - Test Manager; - Test Analyst - Technical Test Analyst • Other separately available - Exam style questions; - Test techniques packs; - Agile testing seminar video; - Testers

templates and other value-added testing related content.

Into the future Learntesting is continuing to invest in its testing and testing-related content, including: • Multi-language courses (building on the German localisation of Foundation); • Agile Development & Testing; • User Acceptance Testing; • Sample exam questions & answers; • Papers & presentations; • Enhanced interactive content.

Mike Smith Managing director Learntesting msmith@learntesting.com www.learntesting.com

March 2010 | T.E.S.T

42 | T.E.S.T company profile

ISEB The Information Systems Examinations Board (ISEB) is part of the BCS, the Charted Institute for IT, and is an international examination body created to raise the standard of competence and performance of people working in IT. We’re leading the way in qualifications for IT professionals – delivering more than 380,000 exams in over 200 countries.

Our qualifications are internationally recognised and cover eight major subject areas in: Software Testing, ITIL /IT Service Management, IT Assets and Infrastructure, Systems Development, Business Analysis, Project Management, IT Governance, Information and Security and our new qualification Sustainable IT. These are available at Foundation, Practitioner and Higher Level to suit each individual candidate. ISEB Professional Level is also available.

For more information visit www.iseb-exams.com.

These qualifications are delivered via a network of high quality accredited training and examination providers.

The breadth and depth of our portfolio is one of its key strengths as it encourages knowledge, understanding and application in specific business and IT areas.

Candidates develop their competence, ability and aptitude – and therefore their professional potential – giving employers the edge they’re looking for

BCS BCS, The Chartered Institute for IT, promotes wider social and economic progress through the advancement of information technology science and practice.

We bring together industry, academics, practitioners and government to share knowledge, promote new thinking, inform the design of new curricula, shape public policy and inform the public. As the professional membership and accreditation body for IT, we serve over 70,000 members including practitioners, academics and students, in the UK and internationally. A leading IT qualification body, we also offer a range of widely recognised professional and end-user qualifications.

BCS membership for software testers BCS membership gives you an important edge; it shows you are serious about your career in IT and are committed to your own professional development, confirming your status as an IT practitioner of the highest integrity.

Our growing range of services and benefits are designed to be directly relevant at every stage of your career.

Industry recognition Post-nominals – AMBCS, MBCS, FBCS & CITP – are recognised worldwide, giving you industry status and setting you apart from your peers.

BCS received its Royal Charter in 1984 and is currently the only awarding body for Chartered IT Professional (CITP) status, also offering a route to related Chartered registrations, CEng and CSci.

Membership grades Professional membership (MBCS) is our main professional entry grade and the route to Chartered(CITP) status.

Professional membership is for competent IT practitioners who typically have five or more years of IT work experience. Relevant qualifications, eg a computingrelated degree, reduce this requirement to two or three years of experience. Associate membership (AMBCS) is available for those just beginning their career in IT, requiring just one year’s experience.

Joining is straightforward – for more information visit: www.bcs.org/membership where you can apply online or download an application form.

Best practice By signing up to our Code of Conduct and Code of Good Practice, you declare your concern for public interest and your commitment to keeping pace with the increasing expectations and requirements of your profession.

Networking opportunities Our 44 branches, 16 international sections and over 40 specialist groups including Software Testing (SIGIST) and Methods & Tools, provide access to a wealth of experience and expertise. These unrivalled networking opportunities help you to keep abreast of current developments, discuss topical issues and make useful contacts.

Specialist Group in Software Testing (SIGIST) With over 2,500 members SIGIST is the largest specialist group in the BCS. Objectives of the group include promoting the importance of software testing, developing the awareness of the industry’s best practice and promoting and developing high standards and professionalism in software testing.

For more information please visit: www.sigist.org.uk.

Information services The BCS online library is another invaluable resource for IT professionals, comprising over 200 e-books plus Forrester reports and EBSCO databases. BCS members also receive a 20 percent discount on all BCS book publications. This includes Software Testing, an ISEB Foundation and Intermediate. As well as explaining the basic steps of the testing process and how to perform effective tests, this book provides an overview of different techniques, both dynamic and static, and how to apply them.

Career development A host of career development tools are available through BCS including full access to SFIA (the Skills Framework for the Information Age) which details the necessary skills and training required to progress your career.

BCS, First Floor, Block D, North Star House, North Star Avenue, Swindon, SN2 1FA, United Kingdom Tel: +44 (0) 1793 417655 Fax: +44 (0) 1793 417559 Email: isebenq@hq.bcs.org.uk Web: www.iseb-exams.com

T.E.S.T | March 2010

www.testmagazineonline.com

T.E.S.T company profile | 43

Parasoft Improving productivity by delivering quality as a continuous process.

For over 20 years Parasoft has been studying how to efficiently create quality computer code. Our solutions leverage this research to deliver automated quality assurance as a continuous process throughout the SDLC. This promotes strong code foundations, solid functional components, and robust business processes. Whether you are delivering Service-Orientated Architectures (SOA), evolving legacy systems, or improving quality processes – draw on our expertise and award winning products to increase productivity and the quality of your business applications.

Parasoft's full-lifecycle quality platform ensures secure, reliable, compliant business processes. It was built from the ground up to prevent errors involving the integrated components—as well as reduce the complexity of testing in today's distributed, heterogeneous environments.

What we do Parasoft's SOA solution allows you to discover and augment expectations around design/development policy and test case creation. These defined policies are automatically enforced, allowing your development team to prevent errors instead of finding and fixing them later in the cycle. This significantly increases team productivity and consistency.

End-to-end testing: Continuously validate all critical aspects of complex transactions which may extend through web interfaces, backend services, ESBs, databases, and everything in between.

Advanced web app testing: Guide the team in developing robust, noiseless regression tests for rich and highly-dynamic browser-based applications.

Application behavior virtualisation: Automatically emulate the behavior of services, then deploys them across multiple environments – streamlining collaborative development and testing activities. Services can be emulated from functional tests or actual runtime environment data.

Load/performance testing: Verify application performance and functionality under heavy load. Existing end-to-end functional tests are leveraged for load testing, removing the barrier to comprehensive and continuous performance monitoring.

Specialised platform support: Access and execute tests against a variety of platforms (AmberPoint, HP, IBM, Microsoft, Oracle/BEA, Progress Sonic, Software AG/webMethods, TIBCO).

Security testing: Prevent security vulnerabilities through penetration testing and execution of complex authentication, encryption, and access control test scenarios.

Trace code execution: Provide seamless integration between SOA layers by identifying, isolating, and replaying actions in a multi-layered system.

Continuous regression testing: Validate that business processes continuously meet expectations across multiple layers of heterogeneous systems. This reduces the risk of change and enables rapid and agile responses to business demands.

Multi-layer verification: Ensure that all aspects of the application meet uniform expectations around security, reliability, performance, and maintainability.

Policy enforcement: Provide governance and policy-validation for composite applications in BPM, SOA, and cloud environments to ensure interoperability and consistency across all SOA layers.

Please contact us to arrange either a one to one briefing session or a free evaluation.

Web: www.parasoft.com Email: sales@parasoft-uk.com Tel: +44 (0) 208 263 6005

www.testmagazineonline.com

March 2010 | T.E.S.T

44 | T.E.S.T company profile

Seapine Software TM

With over 8,500 customers worldwide, Seapine Software Inc is a recognised, award-winning, leading provider of quality-centric application lifecycle management (ALM) solutions. With headquarters in Cincinnati, Ohio and offices in London, Melbourne, and Munich, Seapine is uniquely positioned to directly provide sales, support, and services around the world.

Built on flexible architectures using open standards, Seapine Software’s cross-platform ALM tools support industry best practices, integrate into all popular development environments, and run on Microsoft Windows, Linux, Sun Solaris, and Apple Macintosh platforms.

Seapine Software's integrated software development and testing tools streamline your development and QA processes—improving quality, and saving you significant time and money.

TestTrack RM TestTrack RM centralises requirements management, enabling all stakeholders to stay informed of new requirements, participate in the review process, and understand the impact of changes on their deliverables.

Easy to install, use, and maintain, TestTrack RM features comprehensive workflow and process automation, easy customisability, advanced filters and reports, and rolebased security. Whether as a standalone tool or part of Seapine’s integrated ALM solution, TestTrack RM helps teams keep development projects on track by facilitating collaboration, automating traceability, and satisfying compliance needs.

TestTrack Pro TestTrack Pro is a powerful, configurable, and easy to use issue management solution that tracks and manages defects, feature requests, change requests, and other work items. Its timesaving communication and reporting features keep team members informed and on schedule.

TestTrack Pro supports MS SQL Server, Oracle, and other ODBC databases, and its open interface is easy to integrate into your development and customer support processes.

TestTrack TCM TestTrack TCM, a highly scalable, cross-platform test case management solution, manages all areas of the software testing process including test case creation, scheduling, execution, measurement, and reporting. Easy to install, use, and maintain, TestTrack TCM features comprehensive workflow and process automation, easy customisability, advanced filters and reports, and role-based security.

Reporting and graphing tools, along with user-definable data filters, allow you to easily measure the progress and quality of your testing effort.

QA Wizard Pro QA Wizard Pro completely automates the functional and regression testing of Web, Windows, and Java applications, helping quality assurance teams increase test coverage.

Featuring a next-generation scripting language, QA Wizard Pro includes advanced object searching, smart matching a global application repository, data-driven testing support, validation checkpoints, and built-in debugging. QA Wizard Pro can be used to test popular languages and technologies like C#, VB.NET, C++, Win32, Qt, AJAX, ActiveX, JavaScript, HTML, Delphi, Java, and Infragistics Windows Forms controls.

Surround SCM Surround SCM, Seapine’s cross-platform software configuration management solution, controls access to source files and other development assets, and tracks changes over time. All data is stored in industry-standard relational database management systems for greater security, scalability, data management, and reporting.

Surround SCM’s change automation, caching proxy server, labels, and virtual branching tools streamline parallel development and provide complete control over the software change process.

www.seapine.com United Kingdom, Ireland, and Benelux: Seapine Software Ltd. Building 3, Chiswick Park, 566 Chiswick High Road, Chiswick, London, W4 5YA UK Phone:+44 (0) 208-899-6775 Email: salesuk@seapine.com Americas (Corporate Headquarters): Seapine Software, Inc. 5412 Courseview Drive, Suite 200, Mason, Ohio 45040 USA Phone: 513-754-1655

T.E.S.T | March 2010

www.testmagazineonline.com

T.E.S.T company profile | 45

TechExcel TechExcel is the leader in unified Application Lifecycle Management as well as Support and Service solutions that bridge the divide between product development and service/support.

This unification enables enterprises to focus on the strategic goals of product design, project planning, development and testing, while enabling transparent visibility with all customer-facing initiatives.

TechExcel has over 1,500 customers in 45 countries and maintains offices in UK, US, China and Japan.

Application Lifecycle Management DevSuite is built around the best-practices insight that knowledge is central to any product development initiative. By eliminating the silos of knowledge that exist between different teams and in different locales, DevSuite helps enterprises transform their development processes, increasing efficiency and overall quality.

DevSpec DevSpec is an integrated requirements management solution that is specifically designed to provide visibility, traceability and validation of your product or project requirements. DevSpec provides a framework to create new requirements, specifications and features that can be linked to development and testing implementation projects.

DevPlan DevPlan is a project, resource, and task management tool. It allows users to plan high level areas of work, assign team members to work in these areas, and then track the tasks needed to complete the activities.

DevTrack DevTrack is the leading project issue and defect tracking tool that is used by development teams of all sizes around the globe. Its configurable workflows allow DevTrack to meet the needs of any organisation's development processes.

DevTest From test case creation, planning and execution through defect submission and resolution, DevTest tracks and manages the complete quality lifecycle.

DevTest combines the test management features of DevTest, DevTrack and TestLink for test automation into one integrated solution.

KnowledgeWise KnowledgeWise is the knowledge management solution at the core of the entire suite.

It is the centralised knowledge base for all company documents including: contracts, processes, planning information and other important records as well as customer-facing articles, FAQs, technical manuals and installation guides.

More information at: www.techexcel.com/products/devsuite.

Service and Support Management Service and Support Management solutions provide enterprises with total visibility and actionable intelligence for all service desk, asset management and CRM business processes.

ServiceWise ServiceWise is a customisable and comprehensive internal Helpdesk, ITSM- and ITILcompliant solution. Automate and streamline services and helpdesk activities with configurable workflows, process management, email notifications and a searchable knowledge base. The self-service portal includes online incident submission, status updates, online conversations and a knowledgebase. ServiceWise includes modules such as incident management, problem escalation and analysis, change management and asset management.

CustomerWise CustomerWise is an integrated CRM solution focused on customer service throughout the entire customer lifecycle. CustomerWise allows you to refine sales, customer service and support processes to increase cross-team communication and efficiency while reducing your overall costs. Combine sophisticated process automation, knowledgebase management, workflow, and customer self-service to improve business processes that translate into better customer relationships.

AssetWise AssetWise aids the process of monitoring, controlling and accounting for assets throughout their lifecycle. A single and centralised location enables businesses to monitor all assets including company IT assets, managing asset inventories, and tracking customerowned assets.

FormWise FormWise is a web-based form management solution for ServiceWise and CustomerWise.

Create fully customised online forms and integrate them directly with your workflow processes. Forms can even be routed automatically to the appropriate individuals for completion, approval, and processing, improving your team's efficiency. Web-based forms may be integrated into existing websites to improve customer interactions including customer profiling, surveys, product registration, feedback, and more.

DownloadPlus DownloadPlus is an easy-to-use website management application for monitoring file downloads and analysing website download activities.

DownloadPlus does not require any programming or HTML. DownloadPlus provides controlled download management for all downloadable files, from software products and documentation, to marketing materials and multimedia files.

More information at: www.techexcel.com/products/itsm/

Training Further your investment with TechExcel, effective training is essential to getting the most from an organisation's investment in products and people. We deliver professional instructor-led training courses on every aspect of implementation and use of all TechExcel’s software solutions as well as both service management and industry training. We are also a Service Desk institute accredited training partner and deliver their certification courses.

More information at: www.techexcel.com/support/ techexceluniversity/servicetraining.html

For more information, visit www.techexcel.com or call 0207 470 5650.

www.testmagazineonline.com

March 2010 | T.E.S.T

46 | T.E.S.T company profile

31 Media 31 Media is a business to business media company that publishes high quality magazines and organises dynamic events across various market sectors. As a young, vibrant, and forward thinking company we are flexible, proactive, and responsive to our customers' needs.

www.31media.co.uk

T.E.S.T Online Since its launch in 2008 T.E.S.T has rapidly established itself as the leading European magazine in the software testing market. T.E.S.T is a publication that aims to give a true reflection of the issues affecting the software testing market.

What this means is that the content is challenging but informative, pragmatic yet inspirational and includes, but is not limited to: In-depth thought leadership; Customer case studies; News; Cutting edge opinion pieces; Best practice and strategy articles The good news is that the T.E.S.T website, T.E.S.T Online has had a root and branch overhaul and now contains a complete archive of previous issues as well as exclusive web-only content and testing and IT news.

At T.E.S.T our mission is to show the importance of software testing in modern business and capture the current state of the market for the reader.

www.testmagazine.co.uk

VitAL Magazine VitAL is a journal for directors and senior managers who are concerned about the business issues surrounding the implementation of IT and the impact it has on their customers. Today senior management are starting to realise that implementing IT effectively has a positive impact on the internal and external customer and it also influences profitability. VitAL magazine was launched to help ease the process.

vital Inspiration for the modern business

www.vital-mag.net

VitAL Focus Groups VitAL Magazine, the authoritative, thought provoking, and informative source of information on all issues related to IT service, IT delivery and IT implementation is launching a specifically designed programme of Focus Groups that bring together senior decision makers for a series of well thought out debates, peer-to-peer networking, and a supplier interaction.

Held on the 21st June 2011 at the Park Inn Hotel, Heathrow, the VitAL Focus Groups promises to be a dynamic event that provides a solid platform for the most influential professionals in the IT industry to discuss and debate their issues, voice their opinions, swap & share advice, and source the latest products and services.

For more information visit: www.vitalfocusgroups.com or contact Grant Farrell on +44 (0) 203 056 4598

31 Media Limited www.31media.co.uk info@31media.co.uk Media House, 16 Rippolson Road, London SE18 1NS United Kingdom Phone: +44 (0) 870 863 6930 Fax: +44 (0) 870 085 8837

T.E.S.T | March 2010

www.testmagazineonline.com

T.E.S.T company profile | 47

iTrinegy Network emulation & application testing tools iTrinegy is Europe’s leading producer of network emulator technology which enables testers and QA specialists to conduct realistic pre-deployment testing in order to confirm that an application is going to behave satisfactorily when placed in the final production network.

Delivering more realistic testing Increasingly, applications are being delivered over wide area networks (WANs), wireless LANs (WLAN), GPRS, 3G, satellite networks etc, where network characteristics such as bandwidth, latency, jitter and packet error or loss can have a big impact on their performance. So, there is a growing need to test software in these environments. iTrinegy Network Emulators enable you to quickly and easily recreate a wide range of network environments for testing applications, including VoIP, in the test lab or even at your desktop.

Ease of use Our network emulators have been developed for ease of use: • No need to be a network expert in order to use them • Pre-supplied with an extensive range of predefi ned test network scenarios to get you started • Easy to create your own custom test scenarios • All test scenarios can be saved for subsequent reuse • Automated changes in network conditions can be applied to refl ect the real world • Work seamlessly with load generation and performance tools to further enhance software testing.

A comprehensive range to suit your needs iTrinegy’s comprehensive range of network emulators is designed to suit your needs and budget. It includes: • Software for installation on your own desktop or laptop (trial copies available) • Small, portable inline emulators that sit silently on the desktop and can be shared amongst the test team • Larger portable units capable of easily recreating complex multi-path, multi-site, multi-user networks for full enterprise testing • High performance rack-mount units designed to be installed in dedicated test labs • Very high performance units capable of replicating high speed, high volume networks making them ideal for testing applications in converged environments.

If you would like more information on how our technology can help you ensure the software you are testing is ‘WANready’ and going to work in the field, please contact iTrinegy using the details below:

Email: info@itrinegy.com Tel: +44 (0)1799 543 345 Web: www.itrinegy.com/testmagazine

The Software Testing Club The Software Testing Club is a relaxed yet professional place for software testers to hang out, find likeminded software testers and get involved in thoughtful and often fun conversations.

Interesting things happen at The Software Testing Club. It started out as an experiment. Now two years on it has turned into vibrant online community of software testing professionals.

You'll find members are dedicated to their profession and you can find them in deep conversation within the forums. However, it's more than just forums and your standard niche social network. As the club grows we are finding things happening. This includes things like a Job Board, a Mentoring Group, a collaborative Software Testing Book and a crowd sourced testing initiative called Flash Mob Testing.

The Software Testing Club is a grassroots effort. It's for the members and grows according to what we believe they want.

Come join and let us know what you think.

Rosie Sherry – Founder & Community Manager Email: rosie@softwaretestingclub.com Tel: +44 (0)7730952537 Web: www.softwaretestingclub.com

www.testmagazineonline.com

March 2010 | T.E.S.T

48 | The Last Word

the last word...

We don't need no stinkin' test plan!

A weekend spent as a volunteer on a construction project has resonances with testing for Dave Whalen. Who always has a plan!

M

y wife's boss bought a small two-bedroom house just west of Denver. He wanted to add to the house and recruited volunteers from his employees, their spouses and kids. A ‘weekend project’ I was told.

So one gorgeous, cool, crisp, Saturday morning in July, we drove up to his house in the foothills to help out.

Since I had some experience with construction in a prior life, I was made project foreman. We were in serious trouble!

It was a bit more than an addition.

He had added a four-car Garage to the existing house and wanted to build a new house on top of it. So I asked the boss where his plans or blueprints were. He pulled out a spiral notebook and a window catalogue and began to draw his vision.

Soon a parade of trucks arrived and dropped off loads of construction materials and lumber. How he knew what to order and how much is beyond me. We began to build and raise walls with rough openings for the windows (which had not been ordered yet). As we broke for lunch another truck arrived carrying premade roof trusses. A few minutes later, a crane arrived. We finished raising the walls and had the crane operator hoist the trusses to the top of the structure. As we were nailing them in place, the boss looked out at the magnificent view of Denver below us and declared: “We need another floor!” Construction came to a halt, and we packed up with plans to return the next weekend to build another floor.

Next weekend the scenario replayed itself. We built the third floor using the same process, with only a drawing on a piece of notepaper to guide us. We put on the roof this time. Needless to say the project was a nightmare. Most of us had no idea what we were doing.

T.E.S.T | March 2010

We ran short of supplies, and had to stop to wait for them to be delivered or go get them ourselves.

We had too much of some items, which ended up being wasted. Some things were built incorrectly only to be torn down and rebuilt. When the code inspector arrived, we failed miserably (did I mention we didn't have a building permit). We had to tear off the roof, order new trusses, and completely rebuild it. It had to have cost a fortune, even with volunteer labour!

Always have a plan!

As Dr Covey says in his ‘Seven Habits’: “Begin with the end in mind!” The same thing applies to software testing.

Take the time to write a test plan, Identify the resources you need, the time you have, what tools you need, etc. It can be an exhausting mental process, but it's a valuable one. It ultimately saves you time and money and the more you do it, the easier it becomes.

Sadly, most managers will take a wrong approach to test planning – they throw a bunch of people at it, typically in the last couple of weeks before release (which is then cut to a week because “development is a little behind schedule”) then figure it will all work out. Never has, never will!

Take the time to think it through and write a comprehensive test plan.

I'm always asked: “Should I have it approved and signed?” I personally like to, I like to have all the group leaders sign it: project managers, development managers, business analysts, etc; basically, anyone with a stake in the outcome.

I view the test plan as my contract to the project team as to what we will test, how we will test it, how long it will take, how many people we will need, tools, etc. Will they read it?

Probably not! But, when things don't

I view the test plan as my contract to the project team as to what we will test, how we will test it, how long it will take, how many people we will need, tools, etc. Will they read it? Probably not!

But, when things don't go according to plan, it’s nice to pull it out and point out the signatures.

I always keep a signed copy handy. It should be a working document, continually updated as the project changes. go according to plan, it’s nice to pull it out and point out the signatures.

I always keep a signed copy handy.

It should be a working document, continually updated as the project changes.

A word on templates – I like them, but they are just guidelines. Don't just cut-and-paste your project name into an existing template. Use it as a guide to write your own project-specific plan. Personally, I'm a great collector of test plans. I save them and refer to them often, but always write one from scratch for each project. I may steal a little from this one, or a paragraph from that one, but at the end of the day, I have a unique, project-specific plan. A roadmap to the future; my vision of the end.

Dave Whalen President and senior software entomologist Whalen Technologies http://softwareentomologist.wordpress.com/

www.testmagazineonline.com

Subscribe FREE to the most VitAL source of information for the modern business VitAL : Inspiration

vital mod ern busin ess Inspi ratio n for the 2009 er / December Volume 3 : Issue 2 : Novemb

nds The future is in theithinrksha of IT eration’ What the ‘realtime gen

2 : November/December Volume 3 : Issue 2009

Courting disaster

Are we just paying lip service to DR?

A healthy reliance on ITSM Taking the temperature of IT in healthcare

ET .NET AG.N -MAG AL-M AT WWW.VitAL VitAL IS NOW ONLINE

News, Views, Strategy, Management, Case Studies and Opinion Pieces

vital Inspiration for the modern business

www.vital-mag.net/subscribe 31 Media will keep you up to date with our own products and offers including VitAL Magazine. If you do not wish to receive this information please write to the Circulation Manager the address given. www.testmagazineonline.com Marchat2010 | T.E.S.T Please tick here ■ if you do not wish to receive relevant business information from other carefully selected companies.

Promote yourself...Test yourself ISEB Software Testing certification • • • • •

International benchmark of skills and experience for Software Testers 80% of all Software Testing exams worldwide taken through ISEB Hierarchical structure providing a basis for skills and career progression Over 70,000 Software Testing exams delivered worldwide Supported by ISEB Foundation and Intermediate level publications www.bcs.org/books

ISEB is part of the BCS, The Chartered Institute for IT, and is an International examination body.

We have delivered over 380,000 exams worldwide and our exams are available in over 200 countries.

For more information visit www.iseb-exams.com or call 01793 417655

Book a training course at www.iseb-exams.com/st Book an exam at www.iseb-exams.com/computerbasedexams The British Computer Society (Registered charity no. 292786) MTG/AD/689/0909

T.E.S.T | March 2010

www.testmagazineonline.com