Test Magazine January 2016 by 31 Media

C O N T E N T S

T E S T

M A G A Z I N E S E C T O R

F O C U S :

J A N U A R Y

2 0 1 6

H E A LT H C A R E

36 10

40 NEWS

MOBILE APPLICATION TESTING

Global software testing news 5

Putting all users first

AGILE TESTING

Mobile testing: a mission critical function

The spotlight stays on agile

CAREER CORNER

Women in testing

Wearable tech in the healthcare arena: a testing environment The A and E of healthcare testing

PROJECT CASE STUDY

What a transformation!

TEST AUTOMATION

Is the future of testing a developerâ&#x20AC;&#x2122;s discipline?

HEALTHCARE SECTOR

DEVOPS

Driving the DevOps agenda 44 PREVIEW

TEST Focus Groups

T E S T M a g a z i n e | J a n u a r y 2 01 6

E D I T O R ' S

C O M M E N T

WHAT'S ON THE AGENDA IN 2016? A s 2015 drew to a close I received many a news item about the software/cyber security/DevOps/ testing trends for this year. December and January are, naturally, a good time for reflection, anticipation and forward planning. Buzz words such as ‘agile’, ‘DevOps’, ‘big data’, ‘IoT’ – all the usual suspects – will remain at the forefront of conversations in board rooms, as well as at conferences in 2016. The challenge will lie in translating these into viable enterprise strategies. On p.10 in this issue, you can hear from testing professionals from very different sectors discussing what ‘agile’ means in their organisations and how they can adopt it and, crucially, adapt to it. Cyber security dominated headlines in 2015 – with many brands suffering from reputational damage after well‑publicised hacks. The recently released World Economic Forum’s Global Risks Report 2016 highlighted cyber security as a chief concern, particularly in the US. The report also showed that, globally, the perceived importance of cyber security was uneven, with some countries not placing emphasis on this boundary‑less risk. To ensure that more brands do not suffer humiliating headlines and public inquiries, topics such as security testing must be pushed to the forefront. I believe this year will also see continued friction between tech firms and governments over state access to digital data, through controversial methods such as back door encryption. Whilst the governments of the Netherlands and France have said nee and non, respectively, to these forms of state monitoring, in the UK and the US, they are still being hotly debated. Another area where I hope the debate remains (and grows louder) is that of the gender imbalance in IT. This issue features five women in testing (p.14), who outline ideas on how to help encourage more women into testing and IT in general. Internally, we’ve also had some time to reflect and plan here at TEST Magazine. We

CECILIA REHN EDITOR OF TEST MAGAZINE

are excited to continue serving the European (and increasingly further afield) testing community. The end of last year saw a major redesign of the magazine and our news portal (www.softwaretestingnews.co.uk) to make it easier for you, our readers, to glean the information and insight you want. We also launched the Software Testing Forum, a member’s only community offering exclusive research, invitations, networking, discounts and much more. For 2016, we begin the year with a new event, the TEST Focus Groups (see more on p.46), which will bring senior testing and QA professionals together for a one‑day event of roundtable discussions. Then the popular National Software Testing Conference will return to the British Museum in London on 17‑18 May. The European Software Testing Awards will return in November, and entries will open in April. A full calendar of what we are up to in 2016 is online: www.thesoftwaretestingforum.com/calendar As still a relative ‘newbie’ to this industry (I only joined you last summer), I have been positively taken aback by how friendly, supportive and interactive the software testing and QA community is. We take our responsibility of serving this community seriously, and are always looking for more ways to give back. This year we’ll be reintroducing our independent industry Benchmark Report, which will be ready towards the end of the year. The more we all get together in 2016, collaborate, share ideas and good practices, the better the industry as a whole will be. I hope you enjoy this issue, and I look forward to working with you, meeting you at industry events, and interacting online (@testmagazine/LinkedIn) this year!

JANUARY 2016 | VOLUME 7 | ISSUE 6 © 2016 31 Media Limited. All rights reserved. TEST Magazine is edited, designed, and published by 31 Media Limited. No part of TEST Magazine may be reproduced, transmitted, stored electronically, distributed, or copied, in whole or part without the prior written consent of the publisher. A reprint service is available. Opinions expressed in this journal do not necessarily reflect those of the editor of TEST Magazine or its publisher, 31 Media Limited. ISSN 2040‑01‑60 EDITOR Cecilia Rehn cecilia.rehn@31media.co.uk +44 (0)203 056 4599 ADVERTISING ENQUIRIES Anna Chubb anna.chubb@31media.co.uk +44 (0)203 668 6945 PRODUCTION & DESIGN JJ Jordan jj@31media.co.uk 31 Media Ltd, 41‑42 Daisy Business Park 19‑35 Sylvan Grove London, SE15 1PD +44 (0)870 863 6930 info@31media.co.uk www.testingmagazine.com PRINTED BY Pensord, Tram Road, Pontllanfraith, Blackwood, NP12 2YA

@testmagazine TEST Magazine Group

cecilia.rehn@31media.co.uk

T E S T M a g a z i n e | J a n u a r y 2 01 6

TESTING RECONSIDERED OF PROJECTS WILL BE CANCELLED 31.1% BEFORE THEY EVER GET COMPLETED [1] ONLY

ARE COMPLETED ON-TIME

16.2% AND ON-BUDGET [1]

$312BN

? Y H W

W, OO SLO S AN T S I G TESTIN UAL AND LET OF N R TOO MA PTABLE NUMBE UNACCE ECTS THROUGH DEF

POOR QUALITY REQUIREMENTS

56%

OF DEFECTS STEM FROM POOR QUALITY REQUIREMENTS [3]

64%

SPENT ON DEBUGGING PER YEAR [2]

CA TEST DATA MANAGER LETS YOU FIND, CREATE AND PROVISION THE DATA NEEDED FOR TESTING.

189%

[1]

MANUAL TEST CASE DESIGN 6 HOURS TO CREATE 11 TEST CASES WITH 16% COVERAGE

A NEW APPROACH TO TESTING

UNAVAILABLE OR MISSING DATA UP TO 50% OF THE

AUTOMATICALLY GENERATE AND EXECUTE OPTIMIZED TESTS

THE RIGHT DATA, TO THE RIGHT PLACE, AT THE RIGHT TIME

(GRID-TOOLS AUDIT AT A LARGE FINANCIAL SERVICES COMPANY: HTTP:// HUBS.LY/H01L2BH0)

AVERAGE TESTER’S TIME IS SPENT WAITING FOR DATA, LOOKING FOR IT, OR CREATING IT BY HAND (GRID-TOOLS EXPERIENCE WITH CUSTOMERS)

TESTING CANNOT OF TOTAL DEFECT REACT TO CHANGE

COSTS ORIGINATE IN THE REQUIREMENTS ANALYSIS AND DESIGN PHASE [4]

THE AVE COST OVE RAGE RRUN IS

TWO TESTERS SPENT TWO DAYS UPDATING TEST CASES AFTER A CHANGE WAS MADE TO THE REQUIREMENTS

(GRID-TOOLS AUDIT AT A LARGE FINANCIAL SERVICES COMPANY: HTTP://HUBS.LY/H01L2C_0)

BUILD BETTER REQUIREMENTS

4 HOURS TO MODEL ALL BUSINESS REQUIREMENTS AS AN ACTIVE FLOWCHART AND MAKE THEM “CLEAR TO EVERYONE” [5]

2 BUSINESS DAYS TO GO FROM SCRATCH TO EXECUTING 137 TEST SCRIPTS WITH 100% COVERAGE [5] 60% IMPROVEMENT IN TEST

DATA QUALITY AND EFFICIENCY WITHIN 3 MONTHS USING SYNTHETIC DATA GENERATION

(GRID-TOOLS CASE STUDY AT A MULTINATIONAL BANK: HTTP://HUBS.LY/H01L2G50)

AUTO-UPDATE TEST CASES AND DATA WHEN THE REQUIREMENTS CHANGE

5 MINUTES TO UPDATE TEST CASES

AFTER A CHANGE WAS MADE TO THE REQUIREMENTS

(AUDIT AT A LARGE FINANCIAL SERVICES COMPANY: HTTP://HUBS.LY/H01L2HP0)

DON’T DELAY, START YOUR FREE TRIAL TODAY

GRID-TOOLS.COM/DATAMAKER-FREE-TRIAL

[1] STANDISH GROUP’S CHAOS MANIFESTO, 2014 – HTTP://HUBS.LY/H01L2JK0 | [2] CAMBRIDGE UNIVERSITY JUDGE BUSINESS SCHOOL, 2013 – HTTP://HUBS.LY/H01L2KY0 | [3] BENDER RBT, 2009 – HTTP://HUBS.LY/H01L2L80 | [4] HYDERABAD BUSINESS SCHOOL, GITAM UNIVERSITY, 2012 – HTTP:// HUBS.LY/H01L2MC0 | [5] CA A.S.R CASE STUDY, 2015 – HTTP://HUBS.LY/H01L2NJ0 Copyright © 2015 CA, Inc. All rights reserved. All marks used herein may belong to their respective companies. This document does not contain any warranties and is provided for informational purposes only. Any functionality descriptions may be unique to the customers depicted herein and actual product performance may vary. CS200-160313

S O F T W A R E

T E S T I N G

SOFTWARE BUG SEES US INMATES RELEASED EARLY A 13‑year old software error at the Department of Corrections, Washington State, USA, has seen the accidental early release of inmates. The error was first discovered, but not acted upon, three years ago. Governor Jay Inslee’s office revealed that around 3% of inmates, approximately 3,200 incarcerated criminals, released since 2002 had been erroneously let out of prison early because of a bug in the software used to compute sentence lengths. It has been reported in the US that at least one inmate mistakenly released early had gone on to commit further crimes – robbery and murder. “I’m very concerned about what we will uncover as we move forward and there's likely to be more crime that has been committed during that window,” Corrections Secretary Dan Pacholke said. The Department of Corrections is now calculating release dates by hand and an external

UBER WANTS TO PARTNER WITH MORE APP DEVELOPERS Ride‑hailing service Uber is looking to entertain its passengers while they are in the car. Recently, Uber opened up for smartphone app developers to create ‘trip experiences’ for riders. The goal is for Uber passengers to experience tailored information and entertainment in transit. A few examples cited included timed music playlists or news briefings. “We opened our API to developers globally over a year ago. Since then we’ve seen the creative ways that other apps have used Uber’s API to help get riders from A to B,” Chris Saad, Head of Product on Uber’s developer platform said in a blog post. In the past, the firm has established partnerships with companies such as StubHub, United Airlines, OpenTable and

N E W S

investigation has been launched into how the software error came about, and why nothing was done to fix it three years ago. “I have a lot of questions about how and why this happened, and I understand that members of the public will have those same questions,” Governor Inslee said.“ I expect the external investigation will bring the transparency and accountability we need to make sure this issue is resolved.”

Facebook, to allow users of these services quickly book a ride. “These integrations help make life simpler and easier for people to get around,” Saad said. Uber is inviting developers to submit and test their ideas, but retains final say over whether a trip experience will be allowed.

SMARTBEAR PARTNERS WITH TECHARCIS SOLUTIONS SmartBear Software has announced a partnership with specialised QA and testing solutions company, TechArcis Solutions. Headquartered in Atlanta, USA with offices and technology centres in India and Australia, TechArcis is now delivering SmartBear’s testing and quality assurance solutions including the company’s API Readiness platform, ‘Ready! API’, as well automated testing tool TestComplete. “SmartBear products complement our strategy to help our clients build software for the connected world,” said Sunil Sehgal, Managing Partner at TechArcis Solutions. “Our objective is to address our clients QA and testing needs, deliver testing transformation and assure continuous improvements. SmartBear tools help us do that, ensuring quality throughout and at every step of the development lifecycle. We are excited to use SmartBear’s Ready! API unified platform and provide a differentiated solution to our clients.” “With today’s applications deploying on mobile, web, desktop, internet of things (IoT) and even embedded computing platforms, the connected nature of these applications through public and private APIs presents a unique set of challenges for developers, testers and operations teams,” said David Parker, Director, Global Alliances at SmartBear. “This partnership helps mitigate the challenges and risks for clients, while helping them to ‘get it right the first time.’”

T E S T M a g a z i n e | J a n u a r y 2 01 6

S O F T W A R E

T E S T I N G

N E W S

NEARLY US$430 BILLION IN ASSETS WERE AFFECTED BY SOFTWARE BUGS IN 2015 Despite the apparent ‘death of testing,’ Tricentis’ newly published report, Software Fail Watch: 2015 in Review, makes a strong case for the continued necessity of enterprise software testing. The report features statistics aggregated from a collection of over 450 software bug news articles published in 2015. These 450 stories revealed that nearly US$430 billion in assets, and over 4 billion people were affected by software bugs in 2015. The government sector saw the most fails at 145 reported incidents, with transportation and retail close behind. However, the government sector generated the least media coverage as a whole. The finance sector appeared uniquely skilled at burying their software bugs; their stories tending to be particularly vague on the facts. Several of the finance related stories that did come to light however, blew up to be among the biggest stories of the year, the report states. The latter half of the year saw a significant up‑tick in software bug frequency, which appear to match the travel and consumer purchasing trends of the year. All enterprises, it would seem, are vulnerable. “Nobody is immune to bugs, especially not digital enterprises,” says Wolfgang Platz, CPO and Founder of Tricentis. “This report is a crucial reminder of how easily a software bug can destroy enterprise value. Risk‑based testing is key to ensure that you can keep your enterprise out of reports like these.”

T E S T M a g a z i n e | J a n u a r y 2 01 6

WALMART DEBUTS ITS OWN MOBILE PAYMENTS SOLUTION Right at the end of 2015, Walmart began rolling out Walmart Pay, making it the first retailer with its own mobile payments solution that works on any iOS or Android smartphone. The retail giant does not allow customers to use Apple Pay, Google Wallet, or Samsung Pay, and the news of its propriety solution came shortly after Merchant Customer Exchange began public testing. MCX is a partnership of major store and restaurant chains including Walmart, Target and CVS Health, created to counter Apple Pay. It has been speculated that since MCX has no set launch date, Walmart did not want to wait any longer, and risk missing out on mobile payment revenues. Forrester Research predicts that mobile payments by US consumers will increase from US$52 billion

in 2014 to US$142 billion by the end of 2019. Notwithstanding Apple Pay’s popularity, analysts say that Walmart’s system could work. “Nothing’s really taken off,” James Wester, an analyst at IDC said. “There is still room in the market. And this also allows them to negotiate with credit card companies on their own.” “Walmart Pay is not about improving payment for just payment’s sake,” Daniel Eckert, Walmart US Senior Vice President of Services, said. “It’s about how we can use payment to create a better shopping experience.” Available in all US stores by this summer, Walmart Pay uses a camera that reads a code that appears on cash registers at the end of a transaction to enable payment. Credit card information is not stored on the app but rather on the company’s servers. Eckert insisted the launch did not mean Walmart was wavering in its commitment to MCX, and technically the door is still open to working with other digital wallets, should Walmart want to.

FITNESS WEARABLES WILL REMAIN THE PRIMARY WEARABLE DEVICE New data from Juniper Research shows that fitness wearables will remain the primary wearable device, with smartwatches being less commonly used for the next 3 years. However, fitness wearables are expected to be used by approximately 110 million people worldwide at the end of 2019, while smartwatches will have more than 130 million users. The new research observed that the lines between the categories are starting to blur, with fitness wearables offering a range of call‑handling and notification functions that can also be found in smartwatches. The existence of app‑enabled fitness trackers, such as the Samsung Gear Fit and Microsoft Band, lessens the distinction even further. The research also highlighted the role of monetary incentives in driving future device

CENTRE4 TESTING AND THE TEST PEOPLE MERGE Centre4 Testing has merged with The Test People, positioning the new combined entity as the largest privately‑owned UK software testing company. The organisation now offers a 350+ strong, UK‑based team with offices in Leeds, Manchester, London and Brighton. The merged company will be led by Miles Worne, CEO, Centre4 Testing, with existing directors from both companies combining to make an experienced leadership

WIKIPEDIA FOUNDER SAYS GOVERNMENTS MUST STRIKE A BALANCE ON CYBER SECURITY Speaking to Sky News to mark the 15th anniversary of the online encyclopaedia, Jimmy Wales commented that when it came to cyber security, part of the problem was that politicians "don't really understand the technology very well."

T E S T M a g a z i n e | J a n u a r y 2 01 6

S O F T W A R E

adoption, emphasising its increasing role in corporate wellness schemes. This is even more expressly the case in the professional sports world, where wearables are becoming part of the training regimes of many teams, and form the majority of the market for clothes with integral fitness tracking. Over time, Juniper expect that wearables‑measured performance will become a standard part of hiring practices, and potentially also players' contractual obligations. The research also noted that while future electronic healthcare records will drive the use of dedicated wearables, the price of the devices and dependence on smartphones will hold them back from full adoption by universal healthcare systems. “The use of wearables to track health shows promise, but such devices will not reach their full potential until they can become less dependent on mobile devices to relay their information, in addition to meeting healthcare data storage and relay requirements,” research author James Moar noted.

T E S T I N G

N E W S

TWITTER CODE BUG CAUSES SIX‑HOUR GLITCH Twitter has blamed a failed software update for a fault that caused its service to become unavailable or glitch for over six hours in January. An "internal code change" had caused the problem, which lasted six hours and 10 minutes, the social media site said. "We reverted the change, which fixed the issue," it added. Many of Twitter's 320 million users were unable to log in, or if they could, saw the regular service interrupted during the affected period. Twitter's shares fell by nearly 7% following the disruption. "The current market malaise and the recent site outages are compounding the negatives and having a very negative reaction on the shares,” Victor Anthony, Analyst at Axiom Capital Management. said.

team. Private equity firm, Livingbridge, who announced their investment in Centre4Testing in March, helped to structure the merger, and is working with the new management team to help them continue to grow.

The UK government proposed new legislation on surveillance, the so‑called ‘Snooper’s Charter’ last November. The Investigatory Powers Bill is intended to bring existing surveillance powers up to date and to introduce new powers, all under one comprehensive piece of legislation. Wales acknowledged that cyber security is very important, but politicians do not fully understand what they are trying to legislate. "Sometimes they pass laws that sound good to them on paper, but we know they

won't work. It won't work/it puts a lot of people's data at risk,” he said. "We need to be encouraging businesses and everybody else to be more to secure, to use encryption everywhere to save our data from criminals,” he added. There needs to be balance, the Wikipedia founder said. If the government wants insist on controversial measures such as backdoors to monitor citizens, then they are potentially enabling criminals to do the same.

S O F T W A R E

T E S T I N G

N E W S

Featuring stimulating, intriguing articles and features from experienced software testers and leading vendors, you can be sure that you will stay up‑to‑date with the software testing industry.

www.softwaretestingnews.co.uk

INDUSTRY EVENTS TESTBASH

Date: 10‑11 March 2016 Where: Brighton, UK www.ministryoftesting.com

DEVOPS FOR US DEPARTMENT OF HOMELAND SECURITY Vencore, Inc. has been awarded the Joint Engineering Teams – Sustainment (JETS) Benefits Portfolio task order under the Department of Homeland Security (DHS) Enterprise Acquisition Gateway for Leading Edge Solutions (EAGLE) II contract. JETS is a five‑year contract with a total ceiling value of US$96 million.

read more online

CYBER ATTACK RISK HIGHLIGHTED IN GLOBAL RISKS REPORT In a recent risk report, cyber attacks were ranked as a major concern by US and European business leaders. The 11th edition of the World Economic Forum’s The Global Risks Report 2016 draws attention to ways that global risks could evolve and interact in the next decade. In this year’s annual survey, almost 750 experts assessed 29 separate global risks for both impact and likelihood over a 10‑year time horizon.

read more online

★★★

TEST FOCUS GROUPS

Date: 22 March 2016 Where: Park Inn by Radisson, London, UK www.testfocusgroups.com R E C O M M E N D E D

★★★

CLOUD EXPO

Date: 12‑13 April 2016 Where: ExCeL, London, UK www.cloudexpoeurope.com ★★★

APPS WORLD GERMANY

Date: 20‑21 April 2016 Where: CityCube Berlin, Germany www.germany.apps‑world.net ★★★

GAMES QA

Date: 26 April 2016 Where: Kings College, London, UK www.tiga.org ★★★

PRINCETON SCIENTISTS LAUNCH ‘BUG‑FREE SOFTWARE’ CAMPAIGN A team led by Princeton computer scientist Andrew Appel aims to exterminate software bugs, aided by a US$10 million five‑year grant from the National Science Foundation (NSF). Appel plan to develop integrated tools to eliminate uncertainty from the complex task of software development.

read more online

SMART HOME PLATFORM COMING TO THE UK O2 has announced that it is working with AT&T Digital Life and tado° to bring a unique smart home proposition to the UK in summer 2016. After a limited trial of AT&T’s home management and automation service in Europe last year, O2’s parent company Telefonica has now confirmed that UK will be the first commercial market for the smart home platform.

APPS WORLD NORTH AMERICA

Date: 11‑12 May 2016 Where: Santa Clara, USA www.na.apps‑world.net ★★★

THE NATIONAL SOFTWARE TESTING CONFERENCE

Date: 17‑18 May 2016 Where: British Museum, London, UK www.softwaretestingconference.com R E C O M M E N D E D

read more online

T E S T M a g a z i n e | J a n u a r y 2 01 6

THE SPOTLIGHT STAYS ON AGILE

TEST Magazine hosted an industry roundtable late last year, where senior testing and QA professionals from end user organisations got together to debate testing challenges in an agile world. Representatives from the financial, gaming, travel and mobile sectors argued that as organisations transition to agile systems, testers will benefit but many challenges still remain.

A G I L E

T E S T I N G

he driver to adopt agile is different across sectors, and some are noticeably further ahead – retail, ecommerce and leaner startups have been quicker on‑board the agile train as they are less encumbered by complex legacy systems, or heavy regulation. A senior test manager at a leading gaming firm noted that time to market is a huge change agent for them. Transitioning to agile will speed up delivery in an increasingly competitive market. The gaming sector has seen unprecedented competition in recent years in the form of online and mobile/tablet games, and there is pressure to put out new content every couple of months. “We’ve been talking about agile for many years; the need to help our developers do things smarter and faster,” she said.

AGILE CANNOT WORK IN A WATERFALL ORGANISATION “You can’t embed an agile dev process in a waterfall organisation, it won’t succeed,” said a senior testing manager at one of the UK’s largest low fare airlines. Other executives were in agreement. Unless the support comes from the top, and the entire IT department has been shifting to agile, testing and QA cannot drive this change alone. In the banking sector, legacy systems stand in the way for some but not all. At one major investment bank, top down direction is leading the way for one of the largest reverse engineering processes. “It’s succeeding because we’ve got the buy in from the top. That way you get the budget,” said a head of testing director.

THE DEFINITION CHALLENGE There are many varied definitions and interpretations of agile. A vice president at a large investment bank lamented that they are not implementing pure agile, and the many hybrid definitions floating around complicate things. “You have different teams operating in waterfall and agile, and different combinations of both, meaning there is no cross‑company standardisation or organisation,” she said. At another bank, terms such as water‑scrum‑fall are floating down the corridors, highlighting the confusion. There is an emphasis on trying to avoid bookending agile construction between waterfall requirements and waterfall testing and acceptance. Overcoming the ‘buzzword’ hurdle is an additional challenge. Senior management can, and often have, fall in love with agile – and point to high profile companies such as Facebook and Google and turn around and demand the same immediate transition within their own organisations. A comparison to a flashy tech company such as Google is unrealistic if you’ve got financial regulators on your back or are weighed down by legacy systems. A senior manager for a gaming company explained that in her industry, you can see agile being talked about more and being incorporated in the dev teams, but QA is still seen

A senior manager for a gaming company explained that in her industry, you can see agile being talked about more and being incorporated in the dev teams, but QA is still seen as very separate – something that happens later.

T E S T M a g a z i n e | J a n u a r y 2 01 6

Rarely does a company have the luxury of putting three testers on one piece of code, and for many, even getting testers in the same time zone as the developers is a challenge.

T E S T M a g a z i n e | J a n u a r y 2 01 6

A G I L E

as very separate – something that happens later. Games developers are very creative and many do not want to be ‘contained’ or ‘stifled’ by others; there is a strong black box testing mindset. “I hear about some studios being more open, but a lot of them are more old school,” she said.

TESTERS NEED STRONG POSITIONS Testers might need a push to help empower their position in an agile environment explains one senior test manager. At this leading low cost airline, small focused sprint teams have been set up and the attitude and framework are in place for agile. However, as much as people like to talk about it in team meetings, in practice QA is still led by the dev team. “I have to tell my testers, go sit next to the dev team. Don't wait for an invitation. Test while he’s writing code. Testers need to feel empowered to say: ‘I’m not waiting to be asked, I’m coming along,’” the senior testing manager said. The ‘three amigos’ system of putting BA, dev and test together to create

T E S T I N G

BDD scenarios is currently in place at the airline. By physically being able to get together and meet, communication is clearer.

COLOCATION CAN AID COLLABORATION A head of QA manager at a leading high street bank notes that in his experience, successful agile teams were able to effectively include QA during pre sprint planning meetings. QA ask the ‘what if’ questions, adding much value to the meeting and ultimately saving time and costs as these questions were considered in advance. “I would definitely do this again if setting up a new agile team. The value we got out of these pre sprint planning meetings was enormous,” he said. A testing professional in the mobile community highlighted an example where one mobile company had the traditionally separate iOS and Android developers working together. They were encouraged to look over each others’ shoulders to learn from each other. Had QA been involved in this equation as well, then there would have been increased value.

A G I L E

T E S T I N G

Indeed, most testing professionals seem to recall a time where a small group of testers and developers were locked in a room to produce quality results, but this is not a sustainable use of resources. Rarely does a company have the luxury of putting three testers on one piece of code, and for many, even getting testers in the same time zone as the developers is a challenge. “Our QA team is in India and the developers are in New York. There’s no way that these guys can get in one room,” a vice president at an investment banking firm said. “We operate in high cost locations, and no one would want to sign off on a QA person in New York,” she added. “And we can’t move the developers out to India because then they would be considered ‘too far away’ from the core business.” One potential solution to a situation like this would be to find a programme in isolation and set up a small, colocated team, to prove that it works. However, for the financial firm in question, complex systems mean that there are no obvious programmes to isolate. A good question to consider is how to calculate the cost difference of having a colocated team on the ground in a high cost location such as London or New York, presumably resulting in a shorter time to market (and a higher quality product) versus using cheaper offshore locations and maintaining the status quo. A senior testing manager from a gaming company noted that they also do not have colocation capabilities, with cost being a major factor, but are looking into it. In the gaming industry you do find examples of successful colocation, but agile is also seen as something that can work very well as long as you have many strong communication channels in place.

DISADVANTAGES There can be some disadvantages for testers when organisations adopt an agile methodology. “When people don’t implement agile properly you have a problem,” a chief quality auditor commented. “If you only focus on functionality testing you risk losing sight of the QA mindset or the user experience.” In the world of mobile applications you can see many examples of lean startups, who despite benefitting from agile working conditions, deliver products that users dislike and/or simply cannot use.

“Apps still get poor reviews despite meeting all the specifications, but requirements and specifications are all assumptions. We need testers to challenge and test these assumptions,” the auditor said. When QA is misunderstood as only equalling testing requirements, which can be automated, a mentality of ‘testing was an engineering problem that we’ve now automated away’ can occur. Subsequently, there is a belief that there is no need for human testers anymore. This mentality threatens to undermine the profession as a whole. Another potential drawback is that agile does not provide the same illusion of certainty as waterfall. In a waterfall system, you can have a project plan, which gives the illusion of organisation – what’s going to happen and when. Although many would argue that there are multiple issues with relying too heavily on project plans that can see numerous revisions or simply become obsolete before completion. But there can be a level of discomfort with agile, because you don’t know when things will be delivered. “Say you’re getting an extension built on your house, you’d like to know a project end date, you don’t just want to be receiving a bill every week,” commented one head of QA. Agile without discipline can be hard and can lead to scope creep. If you don’t have the right processes around agile it will fail. “Business wants something, then they change, add more and more, meaning dev and test takes longer – and they wonder why time to market is affected, since ‘you’re agile’”, a vice president at a leading investment firm said. If QA is not incorporated at the forefront of the SDLC, and is not allowed to participate in the business decisions, then there is a danger that it can become a 'free for all' as new demands filter down every week, or even every day.

CONCLUSION As companies continue to explore transitioning to an agile methodology, testing departments should embrace the change and help shape their organisation’s definition and implementation of agile. Quality is quickly becoming everyone’s responsibility, and although there are still some challenges ahead, agile is opening up lines of communication and the end products should benefit greatly.

“When people don’t implement agile properly you have a problem,” a chief quality auditor commented. “If you only focus on functionality testing you risk losing sight of the QA mindset or the user experience.”

Join the roundtable Interested in attending similar roundtable discussions? Become a member of The Software Testing Forum for exclusive invitations.

www.thesoftwaretestingforum.com

T E S T M a g a z i n e | J a n u a r y 2 01 6

O U T S O U R C I N G

WOMEN IN TESTING Numbers of women in IT in the UK have fallen from over 20% to 16% in recent years, despite much effort from government and the private sector to encourage more diversity in the workplace. We see it here on our platforms also; for example, the gender split on www.softwaretestingnews.co.uk averages 70% men to 30% women.

C A R E E R

C O R N E R

ith these statistics in mind, TEST Magazine’s Editor Cecilia Rehn gathered up some talented senior women professionals in the software testing and QA space, to ask their opinion on the gender gap in our industry. Interviewees include Paula Cope, Global Head of QA, Tullett Prebon; Donna Firks, Test Manager, New Look; Bhuvaneswari Gangadharan, Senior Test Manager, Harman Connected Services; Sally Goble, Head of Quality, Guardian News & Media Limited; and Asia Shahzad, QA Manager, Hotels.com. Our roundtable covered topics such as the need for more inclusive recruitment campaigns and encouragement of younger women and girls to get interested in tech early on; advice for women interested in pursuing a career in software testing and QA; and how to break the stigma that IT is a man’s world. How did you get into software testing and what motivates you to keep going? Asia Shahzad: Like a lot of people I fell into software testing. When selecting university courses at the end of my schooling, I knew IT was the growth area for the future and so decided to pursue a degree in Computing and Information Systems. My first job allowed me to get a foot in the door with a technology company, and nine months into my role they decided to change their CRM system and needed some SME testing resource. I jumped at the opportunity without fully understanding what the role actually entailed, I haven't looked back since. Bhu Gangadharan: I learned computer programming when I was pursuing an Engineering degree in Computer Science, and I found my calling in testing when I was working as a developer cum tester in the initial days of my career. I realised that my forte lies in ensuring users have a product that meets the user’s expectation and requirements as well as being easy to use. Paula Cope: For me, it was something I progressed into after managing a support project and realising how important testing was to the end user experience. I think the most exciting part of working in software testing is that you get a bird’s eye view of so many different projects; often, for example, a developer will not have much of a view outside the area of code they are working on. It’s also a very challenging position and you have to get it right.

Sally Goble: My journey into QA developed over time. I was one of the first people in London to start using Adobe InDesign when the publishing industry started adopting it as the DTP software of choice. From there I consulted with the Guardian newspaper who were developing custom plugins for Adobe InDesign. I liaised with developers in Holland to make sure the software we had commissioned met our specifications, and gradually became the person who did acceptance testing, reproduced and reported bugs, and liaised with the devs. Once I moved over totally to the digital side of the company, I joined the QA team where I’ve been working ever since.

My advice is: be passionate about the work you do and also do not take things at face value. Testers should advocate for customers' requirements and expectations.

Donna Firks: I’d been working in Application Services for a number of years (the majority of my career) and the opportunity was offered to work as a tester. Although it is something I had not considered before, I was ready for a change. It was the best decision I made. SG: What keeps me going is working in an environment where change, continuous learning, being flexible and problem solving are the things that are most valued. I can't remember the last time I was bored at work. PC: Testing as a discipline has risen massively over the last 20+ years, it is now gaining the recognition it deserves through events such as The European Software Testing Awards. However, there is always further to go and more to learn over a diverse range of areas, which keeps me interested and motivated. BG: Thinking beyond the requirements for a project, anticipating the problems faced by users in situations, getting in the user’s shoes to get an insight into the user’s expectations from a product are a few points from the list of things that keep me motivated. DF: What I love about testing is the ever‑changing platforms on which you are working, the variety in applications we can test, and the ongoing knowledge I gain. No day is ever the same. What advice would you give to someone trying to get into software testing? PC: A good tester will have an enquiring mind; a high level of technical understanding in terms of product architecture and design, coding standards and current testing methodologies including BDD. There is a very real opportunity to shine as a software

PAULA COPE GLOBAL HEAD OF QA TULLETT PREBON

Paula has worked in QA for over 20 years, working at Oracle for 15 years prior to moving to Tullett Prebon. Paula is passionate about QA and ensuring QA is seen as everyone’s responsibility with the importance of building in Quality from Project Initiation. Although a practitioner of many SDLC methods, Paula specialises in agile and its various flavours.

T E S T M a g a z i n e | J a n u a r y 2 01 6

C A R E E R

C O R N E R

We need to be pushing forward all of the time within our organisations and communities. Never shy away from an opportunity, if you think the job fits, go for it and be a role model for your colleagues.

Bhuvaneswari Gangadharan with her manger and mentor Nihar Nayak, Director of Technology at Harman Connected Services at The European Software Testing Awards 2015.

tester, the scope of the role continues to grow and is increasingly valued. SG: You don't need to have a formal qualification or a degree to be a great tester. Some of the best testers I've worked with haven't had those skills. However you need to be extremely curious and logical and be interested in technology more broadly. DONNA FIRKS TEST MANAGER NEW LOOK

Donna has worked in the IT industry for the past 21 years, starting her career working for an independent hardware and software computer solutions company in Dorchester. She chose to return to New Look six years ago, to become part of the Test Team, working on a large number of projects and programme enhancements to support New Look’s pipeline vision for the future.

T E S T M a g a z i n e | J a n u a r y 2 01 6

BG: My advice is: be passionate about the work you do and also do not take things at face value. Testers should advocate for customers' requirements and expectations. SG: I think the main thing is to be adaptable and to be willing to learn new skills and technologies all the time. And to never ever make assumptions. PC: The old myth “anyone can test” has been exposed as entirely untrue, in fact finding highly skilled QA professionals is becoming harder and harder. You must have passion to release the highest quality product possible to meet your customers’ requirements and provide them with an extraordinary user experience.

DF: From a personal perspective I think that a good foundation in application support offers you a great grounding for roles within this sector. Understanding how applications work, how they are used (without the textbook) and looking at applications with a logical mind is the best advice I can offer. AS: Make the most of the wealth of information available around you and ensure that your knowledge of software testing and its practices is as cutting edge as possible. Take any opportunity available to get your first break within an IT organisation, it may not be the role of your choice or your ideal career, however it is much easier to find new openings once you are within an organisation. SG: Learn some programming principles if you don't have them already – JavaScript, HTML and CSS will always be useful, as will Java or Python. Do an online course in your spare time – Coursera or Code Academy offer good ones. DF: Testing is an ever moving target so if you are adverse to change then it probably isn’t the career for you.

C A R E E R

C O R N E R

What are your career aspirations? BG: At the outset, I had clearly defined personal and career milestones. Support from my family and from colleagues and mentors have been hugely instrumental to my success. I feel fortunate for having been given an opportunity to work with my manager who has given me the carte blanche when it comes to planning, managing and executing project tasks. His confidence in my ability did wonders to help me perform to my full potential. AS: I have had a very successful and enjoyable time in software testing. As my career has developed I have become less hands on and more of a mentor and coach to those with whom I work alongside. I want to continue in this trend and develop cross‑functional teams where quality in the deliverable is their primary and shared goal. SG: I'm not sure I have ever had formal 'career aspirations', or a 'five year plan' or anything. I think it's good to be flexible and open about what might be around the corner, both in your career but also in your personal life – that way you are always open to new possibilities and challenges.

women bring to the table are seen as well suited for the software industry, particularly in testing and QA, and these are now becoming more recognised and appreciated. AS: I have never personally experienced such a stigma, however we do need to encourage women who currently work in IT to make themselves known, and to share their experiences in the community at large (not just at women’s forums). SG: You can theorise to your heart's content about why women or men might be better suited to any role, or about why they aren't doing them. Women are just as suited to software testing as men are. It's a great career and an exciting industry, so there is no reason why women shouldn't see testing or tech as a career option. To help spread awareness of this, the women in our tech team at the Guardian regularly host initiatives: we were involved with talking to schoolgirls about careers in tech – holding an open day on Ada Lovelance day. We also host Ladies who Code events.

BHUVANESWARI ‘BHU’ GANGADHARAN SENIOR TEST MANAGER HARMAN CONNECTED SERVICES

Working in software testing since 2002, Bhu has a Bachelor of Engineering in Computer Science from Amrita University, Coimbatore, India. Her most recent achievement is receiving the Testing Manager of the Year Award at The European Software Testing Awards 2015. When not at work, Bhu spends time with her family and her 13‑month old baby.

DF: I have been very fortunate with New Look that they have offered me a lot of opportunities and scope to grow. We have some large programmes of work coming up over the next couple of years of which to I am keen to be involved as a Test Manager. Supporting the company in this growth and seeing the benefits of this success will be a real personal gain. PC: Testing opens so many roles and opportunities; my present role has a wider remit and I believe the increasing focus on quality has been part of this. I greatly believe that there are so many opportunities in today’s IT organisations as complexity and customer expectations continue to grow that the options are endless. As such my aspirations are boundless and I would loath to limit myself by stating what they are, I always search out new opportunities.

SALLY GOBLE HEAD OF QUALITY GUARDIAN NEWS & MEDIA LIMITED

IT is perceived as a man's world, how can we help to break this stigma? A portrait of Ada Lovelace on display at the

BG: I believe this perception is changing. The opinions, qualities and perspectives that

Science Museum in an exhibition celebrating the bicentenary of her birth, ©Science Museum/SSPL.

Sally is charged with maintaining the quality of the Guardian's digital products. Under her leadership the QA team at the Guardian are redefining themselves – less as testers and more as quality advocates who encourage everyone involved in the product development to think about and be involved in quality. When Sally is not working, she mostly swims. She swam The English Channel solo in 2006 and currently organises the Guardian's winter swimming club.

T E S T M a g a z i n e | J a n u a r y 2 01 6

C A R E E R

Attendance at grad fairs, conferences, sponsored events, and meetups can help promote your organisation’s ethos on women in IT.

PC: The IT domain is largely male dominated; however, there are now more women at all levels than when I first started my IT career. I think there has been recognition that the most effective teams are those with a mixture of backgrounds, cultures and work ethics. There are particular traits, which are often better demonstrated by one sex than the other, though this, of course, is a generalisation it is often the case that a combined team brings out the best in all members – male or female.

being taught programming in primary school, this will remove barriers to entry and allow all to make a choice of IT as a career.

SG: Indeed, the industry needs to make sure that recruitment processes don't discourage women from applying for roles. Then, once we employ women we need to have workplaces and environments and cultures where women (and other under‑represented groups) are welcomed and made to feel valued and supported.

BG: It is important that organisations strive to pick the best person for the job. To this end we ensure that women account for around 50% of the hires at Harman. Additionally, efforts are being made to ensure a work culture that recognises potential leaders and develops them to succeed. Companies must seek out incentives to both attract and retain staff.

DF: I think IT has been a man’s world in the past but I do believe times are changing. Women are now more empowered within their careers and have many more equal opportunities open to them. We need to be pushing forward all of the time within our organisations and communities. Never shy away from an opportunity, if you think the job fits, go for it and be a role model for your colleagues. What can be done to encourage more women into IT and software testing?

ASIA SHAHZAD QA MANAGER HOTELS.COM

At Hotels.com, Asia is responsible for the QA strategy for several teams across multiple locations in Europe. Prior to this role Asia was a freelance tester working on some challenging high profile projects across ITV, BBC, TalkTalk, AOL and Cable & Wireless. Asia is passionate about all areas of agile software delivery, and helps promote the importance of QA to all areas of the business.

T E S T M a g a z i n e | J a n u a r y 2 01 6

C O R N E R

DF: I think we need to quell that fear around technology being a male dominated environment. One of the organisations I am involved in is STEMNET (Science, Technology, Engineering and Mathematics Network) of which I have recently become an ambassador. Some of the work I do is working with children and young adults in schools, colleges and universities, talking about my role in testing, technology and breaking down the barriers. AS: I was quite clueless about computers and IT until I joined university; we need to ensure children are educated in technical skills from a young age. The positive trend has already started in schools, my nieces are

PC: While there are already a number of female role models for IT and software testing, including Janet Gregory and Lisa Crispin, companies should look to recruit a more even balance of female to male employees, particularly at the higher levels to inspire and drive the next generation of female employees.

PC: We live in such an IT centric world that I can only assume that IT is more accessible to young people which I hope in turn will encourage more women to follow IT career paths. AS: But until the new generations can join us in the workforce, employers must consider giving opportunities to people, irrelevant of educational background or career history, who show interest and acumen for a role in IT. Additionally attendance at grad fairs, conferences, sponsored events, and meetups can help promote your organisation’s ethos on women in IT.

C A R E E R

C O R N E R

WEARABLE TECH IN THE HEALTHCARE ARENA: A TESTING ENVIRONMENT

With wearable tech advancing at lightening pace, fast, reliable and unfailing testing has never been more important. Here, Mark Thomas, CEO, Coderus, highlights the testing technique that can bring real benefits for wearables in the healthcare arena.

H E A L T H C A R E

S E C T O R

ast year was a great year for the development of wearable tech and we’ve seen a raft of game changing products and devices entering the market. While this is no surprise, there is still enormous potential for growth and development, with wearable tech entering virtually every industry and arena. What may be a surprise, especially given the surge of health and wellbeing products this year, is the relatively slow advance of medical grade healthcare wearables into the marketplace. Digital health offers a huge opportunity for both the public sector and for private business. With individuals being better able to monitor and control health and wellbeing through heartrate, blood pressure and other key indicators, as well as the obvious advantages to both individuals and the healthcare industry of being able to monitor and control drug intake and compliance, the benefits for pre‑emptive healthcare alone are clear. Pre‑emptive care and health direction is certainly the area in which wearables are able to have an immediate and positive impact and, with less testing required before they can reach the marketplace than wearables that may be used for medical treatment purposes or diagnoses, these are set to be seen more and more in the short‑term future. Whether for general health and wellbeing or for medical purposes, though, as with all technology, healthcare wearables need vigorous, consistent and dependable testing before they reach the general public. Testing can take a number of forms, while in this area, software developers need to look further than traditional testing methods. This is further demonstrated by the needs of both consumers and the healthcare industry.

WHAT CONSUMERS WANT FROM THEIR WEARABLES As well as wanting wearable tech that looks good, consumers need their health and wellbeing wearable tech to really deliver. In this arena, wearable tech needs to be: • Informative – measuring and reporting on activity, heart rate, blood pressure, etc. • Intuitive – delivering outcomes and useful information that can be acted upon. • Integrated – with other devices and with the consumer’s lifestyle. Cost is also an important factor for consumers choosing health and wellbeing wearables,

and while healthcare partners may look into subsidising products, testing methods can help to drive down costs to reaching market too.

WHAT THE HEALTHCARE INDUSTRY NEEDS FROM WEARABLES Focusing on pre‑emptive healthcare, professionals also need wearable tech to deliver. Enabling real changes in patient behaviour in terms of nutrition, exercise and general self‑care, the benefits to GPs and the like in terms of patient education, understanding and awareness, as well as the collection of immensely valuable data, are many. For healthcare professionals, wearable tech needs to offer: • Accuracy – giving confidence in the outcomes flagged. • Consistency – allowing correct advice and guidance to be given. • Cost efficiency – enabling devices to be rolled out to a larger audience.

Whether for general health and wellbeing or for medical purposes, though, as with all technology, healthcare wearables need vigorous, consistent and dependable testing before they reach the general public.

DEVELOPING WEARABLES FOR PRE‑EMPTIVE HEALTHCARE From smart health patches that stick to the skin like a tattoo to monitor vital statistics through to tracker bands designed specifically for people with certain conditions, such as epilepsy, healthcare wearables are on the cusp of a big swell, just waiting for the supporting technologies to become less cost prohibitive. With so much potential for data capture, coding, too, needs to be well developed to ensure memory space is not greatly restricted and that battery life is optimised rather than inhibited. Power itself is another consideration, and solar power, kinetic energy and the like are all considerations that may impact on price. Radio frequency is another big challenge, both due to the crowded frequencies and radio noise and due to human interference, with every individual giving off different electrical properties. All of these present challenges to meeting user expectations and market requirements, making testing and accuracy of data even more important and necessitating

MARK THOMAS CEO CODERUS

Mark Thomas is the CEO of Coderus Limited. Mark is a developer/software designer in both his private and professional contracting career, and prior to founding Coderus he spent many years in the telecoms industry.

T E S T M a g a z i n e | J a n u a r y 2 01 6

H E A L T H C A R E

the inclusion of a range of programmable features to ensure the device is tailored to the individual.

CONTINUOUS DEVELOPMENT TESTING IN THE HEALTHCARE ARENA While there is a wide range of methods and aspects to take into account when testing any new technology, wearables in the healthcare market need even more testing and greater vigilance to meet the stringent regulations in the industry, including trials, which can take at least three years to complete. As we have touched on, traditional testing cannot cater for all of the aspects required for devices designed for the healthcare industry. Continuous delivery testing, though, especially when coupled with user experience testing simultaneously, can ensure both high quality and efficacy of delivery. Whatâ&#x20AC;&#x2122;s more, this agile testing method can cut the time of testing from weeks to hours.

S E C T O R

Continuous delivery testing can also be integrated early into the development process. Running in the background as products are worked on, it helps to identify and rectify issues almost immediately, considerably reducing the time to market phase. In conclusion, continuous delivery software testing for wearable technology in the healthcare industry needs to be implemented to reach the highest possible standards. This includes the high standards set by individual countries before a product can enter the healthcare industry. Continuous delivery will play a very important role in ensuring testing is executed to the highest possible standards to meet and exceed any internal and external standardisation requirements. The human element of testing is also very important. It has been reported the drop off levels for wearable technology can be high. The interaction, functionality and overall experience between the user and device have to be well thought through and developed. User experience testing will make sure the devices are user friendly and promote longerâ&#x20AC;&#x2018;term usability.

SOFTWARE TESTING OUTSOURCING Delight your users with a consistent, high-quality experience â&#x20AC;&#x201C; everywhere, every time.

We hate bugs as much as you do â&#x20AC;&#x201C; and go all out to catch them. Get your mobile, web, desktop, and wearable apps tested by our award-winning team. Contact us for a 4-week risk-free trial of our software testing services.

Why Mindfire?

16 years of award-winning experience

1000+ global projects delivered for startups and enterprises

Market-proven Automated and Manual testing experience

ISTQB Platinum Partner

Well-equipped QA lab with prevailing hardware and software

www.mindfiresolutions.com +1 248.686.1424 sales@mindfiresolutions.com

Specialist serving the testing needs of small & midsize businesses

Strong Selenium, SoapUI, WebAPI, Protractor, QTP, Appium, Robotium, Jmeter skills

90+ certified test engineers with 5+ years average experience

4 weeks risk-free trial

Founded in 1999, Mindfire Solutions is an award-winning provider of software development and testing services to the global market with 650+ talented software engineers at 3 centers in India. For its people and its work, Mindfire has won coveted international awards such as Deloitte Technology Fast50 India Award 2013 and 2014, Dun & Bradstreet Fastest Growing SME 2013 Award, Red Herring Top 100 Asia Award and Zinnov GSPR 2014. Mindfire has been recognized with ISO 9001:2008 and ISO 27001:2005 certification, is a continuous member of NASSCOM, and has established a strong track record of 2000+ projects successfully delivered for 500+ technology clients.

OF HEALTHCARE

TESTING

Cassy Calvert, Testing Services Manager, BJSS, takes us through the lessons theyâ&#x20AC;&#x2122;ve learnt working on the NHS Spine 2 project â&#x20AC;&#x201C; which won the Best Overall Testing Project in the Public Sector at The European Software Testing Awards 2015.

H E A L T H C A R E

Spine 2 operates 24x7x365, supporting over 27,000 organisations, including hospitals, GP surgeries and pharmacies. It has over 1.1 million users, handles 1800 transactions per second and incorporates a national demographic database of more than 80 million patients.

CASSY CALVERT TESTING SERVICES MANAGER BJSS

Dedicating her career to the art of Testing since 1999, Cassy joined BJSS in 2012 as Test Manager responsible for the development and growth of company’s test capabilities. Her role sees her managing enterprise‑scale projects, ensuring quality test delivery across multiple onshore and offshore work streams in a high‑pressure, agile, delivery‑focused environment. Outside of IT, Cassy’s spare time is filled with enjoying life with her family and coaching at her local gymnastics club.

T E S T M a g a z i n e | J a n u a r y 2 01 6

n 2014, the International Longevity Centre confirmed what society already knew: the UK is ageing. While those aged between 15 and 64 will grow on average by 29,000 every year until 2037, those aged 65 and over will rise by 278,800 a year. This reality is staring healthcare providers in the face. The older we get, the more healthcare services we consume. Now is the time for healthcare providers to review their IT estates and ensure that their technology matches up to this challenge. This is where robust testing becomes even more important. Any outages, data loss or integration failures could have a material impact on day‑to‑day operations and would severely impact patient care. Recently I was involved in delivering the successful replacement for NHS Spine – the world’s largest public healthcare platform. NHS Spine 2 supports critical NHS business applications, providing interoperability and data sharing across its various healthcare and management systems. As the technological backbone of the NHS, Spine 2 operates 24x7x365, supporting over 27,000 organisations, including hospitals, GP surgeries and pharmacies. It has over 1.1 million users, handles 1800 transactions per second and incorporates a national demographic database of more than 80 million patients. A significant focus on test was key to the successful delivery of this system which today routinely exceeds SLA targets. While the combined test team for NHS Spine 2 had over 75 years of experience working in both public as well as private sector healthcare providers, its unique set of requirements challenged existing test practices and provided valuable learning experiences.

AUTOMATION Automation ensures stability in any large agile delivery project. To enable small regular deliveries, automation should be applied from deployment to unit, integration and system tests. While this may add complexity to maintenance, it reduces the perception of bottlenecks, builds early confidence and delivers a stable codebase. The additional effort required to create and maintain an appropriate level of automation must be planned for at the project’s outset. Spine 2’s delivery, at a high level, included technical (performance, resiliency, soak, stress and volume), UI, data migration, functional, user, bespoke automated

S E C T O R

messaging (synchronous and asynchronous), and considerable unit testing. This approach allowed for testing to occur early in the project. It also became more straightforward to adapt to the individual needs of each phase. Largely bespoke automated testing formed part of overnight builds and resulted in around 25,000 integration tests occurring across the project’s lifetime. Page‑by‑page, object oriented, defect specific and user journey tests were automated. Mocks simulated Spine 2’s thousands of entry points and users. As a result, there was very high confidence in the system especially with its ability to accurately replicate the API and behaviour of its predecessor. Automation artefacts were managed with the same rigour applied to the codebase. Artefacts were delivered to the same coding standards, allowing them to benefit from clear annotation and become part of the build. There are benefits to defining a clear framework for tests, ensuring that items such as error handling and multi‑browser capability are considered. The project also reaffirmed some important principles: 1. Be clear on what test data will be used, where it will come from and how often it will be updated. 2. Agree a language style and enforce consistency. 3. Having and managing test technical debt enhances automation. 4. Continuously maintain and refactor the codebase – both the framework as well as the tests.

BASELINE The non functional team for NHS Spine 2 was in place very early – even before an environment was made available to use – enabling the team to define which types of tests would be required, how they would be monitored, which tools would be used and so on. From this point onwards, as soon as an environment became available, a baseline for those tests was defined. Time was spent experimenting to discover and understand tests such as the rate or distribution of requests over the different endpoints or URLs. A baseline establishes a starting point where there is none. Non‑functional requirements (NFRs) are often not considered until well into the project. Establish them as early as possible – even if much of the project is a functional ‘like‑for‑like’ replacement.

H E A L T H C A R E

S E C T O R

COLLABORATE In any project, there are often many stakeholders with varying perspectives and requirements. This is especially true in a heterogeneous industry such as healthcare where doctors and nurses, receptionists, support staff and management organisations are all defined as stakeholders. This breadth of interested parties impacts practically all aspects and decisions. With Spine 2’s exceptional size, the team had to consider its functional capability, performance and resiliency. To achieve this, development and test teams from both the NHS and BJSS worked together in blended teams. The level of collaboration in these teams made it virtually impossible for outsiders to differentiate between the two groups, enabling issues to be identified and dealt with quickly. Testers were also involved in regular iteration planning sessions. This allowed for detailed capacity planning. Business and assurance representatives attended stand‑ups where progress could be understood, and hidden issues identified. Collaboration was extended to subject matter experts (SMEs) and users who were identified at the outset of the project and invited to show and tell sessions. Small, regular deliveries helped them to understand and contribute to the decisions around design and delivery, and it also helped suppress corporate politics. Large organisations often use more traditional delivery methods and the prospect of introducing an agile approach is akin to turning around an oil tanker in a small port. Working closely with blended teams introduces many of the agile ceremonies, but at a pace that is more comfortable and allows for small course corrections.

DATA

in the healthcare sector where there are legitimate concerns about the use of patient identifiable data (PID). There is no easy answer to what is the acceptable balance of production data to test requirements. In many cases, legislation prevents even depersonalised data from being used. Starting with canned data, a dataset can be built through analysis of data journeys. Done early enough, it becomes possible to build a rich set of transitioned data alongside test cases as the project progresses. Where automated deployment processes are used, take into account the length of time the data will take to load. Consider what size will the data be at go live, how quickly it will grow and how it will look in several years’ time.

ENVIRONMENTS Access to appropriate environments for testing is vital. While any tester would dream of having a fully scaled, like‑for‑like system, the reality is almost always different. Instead, an environment that is appropriate for the stage of testing should be sufficient. Defining the purpose of testing in that stage drives the specification of the environment. Spine 2 combined virtual and production environments. For the automated integration testing the team was able to use virtual machines, automatically deploying the code using open source tools such as Puppet. During the GUI phase, the code was deployed to existing integration environments with endpoints stubbed out. Spine 2 used completely new technologies and therefore allowed the test team to physically use the production environment prior to go live for non‑functional testing. It can become easy to abuse test environments. Old data and test code

can pollute the environment to a level that it becomes unusable. Roles such as ‘TestOps’ with responsibility for the testing infrastructure can help ensure that environments are available when required, and are maintained throughout the lifecycle of the project.

AND FINALLY… An ageing UK will add to the demand on the healthcare sector’s IT estates, making it likely that replacement and expansion projects will become increasingly complex over the medium term. While projects such as NHS Spine 2 are very large, ‘once in a generation’ engagements, they provide valuable lessons and challenge the existing testing status quo. For example: • Decide from the outset how non‑functional requirements will be tested. Ensure this occurs across all stages of the project. Collaborate as much as you can. Healthcare is heterogeneous and there are many stakeholders. Invest in building strong but frank and honest relationships. • As challenging as it might be, always invest in good quality production‑like data – it’s common to find many defects in edge cases and without the rich data they can be difficult to locate. • Finally, automate wherever possible and wherever it makes sense. If you can’t use a production environment, then set up a virtual environment that is as near to production as possible. IT is renowned for failing to live up to its hype; however, when skilled practitioners carefully test solutions, implementations can deliver lasting operational benefits for healthcare providers and improve patient care.

Tests need data. It’s common to find lots of defects in edge cases and without rich data they become difficult to locate. As with many large organisations, the NHS generates considerable volumes of data. NHS Spine 2, for example, handles 70 million patient records and transfers 1800 messages every second. Data volume is a challenge not only for the richness of scenarios that a large dataset can provide, but for the effects of volume on any platform. One of the biggest challenges in any testing project is being able to obtain production data. This is particularly sensitive

T E S T M a g a z i n e | J a n u a r y 2 01 6

PUTTING ALL USERS FIRST Almost one in five of the world's population lives with some kind of recognised disability â&#x20AC;&#x201C; Martin Wrigley, Executive Director, AQuA, explains the work taking place in partnership with MMF to allow all to be able to use phones and apps.

M O B I L E

A P P L I C A T I O N

ooner or later, everyone will develop at least some limitations in vision, hearing, dexterity or learning. To improve usability for those of us with sensory or physical limitations, phones and tablets have features for accessibility, which are continually improving and becoming more prevalent as technologies advance. The Mobile Manufacturer’s Forum (MMF) and the App Quality Alliance (AQuA) has teamed up to help developers check that their apps are accessible to people with such limitations. If your app is limited to only reaching 80% of its potential audience, it could be a simple fix to extend that reach. The MMF work with the handset manufacturers on features built into the devices, and AQuA have extended that to produce Testing Criteria to allow developers to run through a simple series of checks to ensure that they aren’t accidentally blocking those one in five of us living with a disability. As a couple of quick examples: • Allowing a choice of colours used and control of the contrast can make a huge difference for people with colour blindness. • Correctly labelling the buttons in the app enables the screen readers to work. AQuA has gone through the various areas identified by the MMF and W3C WCAG guidelines and built tests to find any issues in your app.

CATEGORIES OF LIMITATION Not all accessibility issues are the same. For example, what makes a great interface for someone with a visual impairment may not work well for someone with impaired hearing. To help break this down AQuA has worked with MMF on various categories of limitation. MMF uses these same categories to help consumers find the best handset for their needs. This is embodied in the Global Accessibility Reporting Initiative (GARI), a project run by the MMF to help consumers learn more about the accessibility features of mobile devices and to help them identify devices with the features that may assist them with their particular needs. The GARI site looks at several categories of limitation, and what handsets and apps can do to help:

HEARING

In 2005, the World Health Organization estimated that some 278 million people

T E S T I N G

worldwide had moderate to profound hearing loss. For these people using a telephone of any type can be a real challenge. Most mobile phones offer a range of features that can certainly make a noticeable difference to those with hearing difficulties, including: • A range of visual alerts to notify the user of incoming calls/messages. • Adjustable volume control. • Display of missed, received or dialled calls through call logs. • Visual or tactile indicators showing what’s been pressed on the keypad. • Text based messaging options. And most have the potential for video conferencing that could be used for sign language. There are also optional accessories from phone manufacturers that are designed to work well with tele‑coil equipped hearing aids.

However not all accessibility issues are the same. For example, what makes a great interface for someone with a visual impairment may not work well for someone with impaired hearing.

VISION

There are many degrees of visual impairment ranging from difficulty in reading small characters through to complete blindness. With all mobile phones, providing a range of options for users with visual impairments is important. In particular, the following features are likely to be of interest: • Tactile markers to help orient your fingers on the keypad. • Audible or tactile feedback to confirm a button has been pressed. • Adjustable font sizes. • Audible cues for low battery, caller waiting or ending a call and volume level. • Adjustable brightness/contrast controls for the display. • The size of the main display. • Backlit display. • Voice recognition and commands. Another simple aid is to assign different ring tones to different stored numbers to identify the caller without the need to look at the phone.

SPEECH

People who have speech related accessibility needs will tend to rely on the text‑based features of phones to help them communicate effectively. Pre‑prepared messages are a handy help to speed up communication.

DEXTERITY

For people with limited dexterity such as those with arthritis, operating the keypad or simply holding the phone can be difficult.

MARTIN WRIGLEY EXECUTIVE DIRECTOR AQUA

Martin has more than 25 years of experience in telecoms and IT, with a wide background of IT development, solutions architecture and delivery and is now an independent consultant and Executive Director of AQuA.

T E S T M a g a z i n e | J a n u a r y 2 01 6

For these people, the following features may be of interest: • Ability to use the phone in 'hands‑free' mode. • Predictive text input that predicts the word and minimises the number of key presses. • Any key answering which allows the call to be answered by pressing any key. • Voice recognition for dialling or accessing features within the phone. • Design where the controls of the phone do not require pinching, twisting or rotation of the wrist. Optional accessories such as a Bluetooth headset or keyboard can also make life easier through making texting and talking much easier.

COGNITION

Mobile phones are becoming increasingly complex devices offering a variety of features that can be bewildering for many people. Ensuring menus and instructions are clear and simple to understand, providing simple instructions when something is required from the user and providing enough time for people to enter the required information are all features that most people will benefit from.

THE ROLE OF TESTING In November 2015, AQuA released a new set of Microsoft App Accessibility Testing Criteria to complement the existing Android App Accessibility Testing Criteria. Both sets of criteria help a developer assess their app and find any areas that will prevent people from fully using their app.

T E S T M a g a z i n e | J a n u a r y 2 01 6

M O B I L E

A P P L I C A T I O N

Starting from the categories of hearing, vision, speech, dexterity and cognition, AQuA has put a structure on the testing looking in each case at navigation, control, feedback, display, adjustment/settings and external devices. Each individual test is set out with a description, a set of pre‑conditions, a set of testing steps and a desired result. Tests can be marked as ‘critical’ – with pass/fail, or ‘warning’ – with a range of results from ‘annoying’ to ‘difficult’ to ‘impossible’. Several warnings can add up to an overall failure, depending on the severity. After running through the tests, and marking up the document, you have a ready test report that will give confidence that the accessibility issues have been checked. The Testing Criteria are independent of the specific functionality of the app, so they can be used for virtually any app running on that platform. The tester needs to look at the criteria and devise how they will use them on the app in front of them. Whilst the AQuA Accessibility Testing Criteria can help make an app more widely

T E S T I N G

accessible, it isn’t trying to tell you how to design your app. The criteria don’t describe exactly what you might need to change to make your app accessible. Clearly that is strongly dependent on the functionality, but AQuA highlight the needs for the developer to address. If your app is specifically aimed at a disability group as your market, then the AQuA criteria may be a good starting point, but you will need to go on to further testing amongst your target market. That really is no different to any other app.

SUMMARY The Accessibility Testing Criteria are today provided free as PDF downloads, ready to use, and AQuA also provides an online tool allowing testers to go through all of the AQuA Baseline Testing Criteria on‑screen and produce a report at the end. The organisation expects to add the Accessibility Testing Criteria to the online tool in early 2016. The Accessibility Testing Criteria will be expanded also to cover the Apple iOS platform, and AQuA is currently planning a set of App Security Testing Criteria for 2016.

MOBILE TESTING: A MISSION CRITICAL FUNCTION Joe Luthy, Marketing Director, LogiGear, considers a holistic mobile device testing strategy.

M O B I L E

A P P L I C A T I O N

obile apps are a necessity for companies of all sizes, and apps are getting more complex all of the time. That fact, along with the dizzying array of devices, requires a well thought‑out mobile testing strategy. And it will involve a bit of risk/reward analysis. Mobile apps come with inherent risks. What might be considered a minor issue on a laptop could be critical on a mobile device. People are generally hurrying, multi‑tasking and have limited time and attention spans when using mobile devices, so it’s not just bugs in apps that are not well tolerated. Buttons, menus and forms that are easy to access on a desktop can be small and frustrating to use when resized for mobile. Testing too many devices creates unnecessary expense. Too few devices risks lost revenue from app abandonment. However, taking time to understand the device ecosystem and the customer the application is designed for will enable creating a test strategy that will balance risk and return.

PLATFORM MATRIX The diversity in devices, operating systems and screen resolutions makes determining the right mix of devices to test complicated. A little basic data analysis will provide a great deal of insight into determining the best device matrix. Three manufacturers account for 80%1 of devices used in the US: Apple (43.5%), Samsung (28.7%) and LG (8.2%). Using that information and then looking at specific target demographics can give a pretty good composite picture of the devices predominantly used by them (which will provide insight into the operating system version) and hence where the majority of testing should be focused. After identifying the device matrix, there is also the option to use a mix of emulators and real devices. Hardly anyone would argue that nothing takes the place of testing on actual devices. Usability testing on emulators and browsers with any extensions are getting better, but won’t always represent what will be seen on the actual device. Emulators can be good for testing new functionality or a new component design, and have some advantages over using actual devices. They are also ideal for running preliminary functionality tests on new devices and OS versions. Logging faults and capturing screenshots are much simpler when working from a desktop, and some conditions that are hard to duplicate on real devices, like low battery power, are easy to simulate.

T E S T I N G

It’s important to note that emulators tend to be slower than real devices. Depending on what type of app is being tested and whether the tests are manual or automated, may limit the response time and the amount of testing that can be done with emulators.

TEST EXECUTION Normally, it is not practical or cost‑effective to conduct full testing or full functional testing on multiple devices. A practical approach consists of running a full set of tests on one or two primary devices then running a smoke test on additional devices to identify any obvious issues across the range of targeted devices. However, it depends on the nature of the application. If the app is cutting‑edge and could possibly stress the device’s capability (processing power, memory) then extensive testing is in order. One thing to keep in mind when running a basic test is that most handheld mobile devices give priority to the communication environment. For example, an incoming phone call always receives priority over a running application. This makes it important to test the various events, which can occur when an application is running, such as incoming calls, text messages, auto power saving mode and alerts. A mobile testing strategy is not complete without testing the integration between the application and the backend system. This is especially true when the release cycles of mobile apps and backend systems are very different, and they typically are.

A mobile testing strategy is not complete without testing the integration between the application and the backend system. This is especially true when the release cycles of mobile apps and backend systems are very different, and they typically are.

MANUAL OR AUTOMATED A great deal of basic compatibility and basic functional testing can be carried out efficiently with manual testing, but when it comes to testing several devices and applications that need frequent retesting, automation can be an efficient way to scale. The efficiency gain will depend on the experience and skill of the automation team – the standard disclaimer ‘results may vary’ is even more applicable to mobile test automation due to all the variables.

JOE LUTHY MARKETING DIRECTOR LOGIGEAR

Joe Luthy is the Marketing Director

DEVICE MANAGEMENT A big challenge with mobile testing is sourcing and managing devices. Creating the initial matrix is just the beginning.

for LogiGear. Joe has over eight years experience in product management and quality assurance for mobile devices, applications and wireless services.

T E S T M a g a z i n e | J a n u a r y 2 01 6

Accessing devices in‑the‑cloud can be a good option. The growing number of cloud service providers makes it possible to access a broad range of devices and pay only for the time needed. However, there are limitations to relying solely on device rental.

M O B I L E

A P P L I C A T I O N

It’s common for each manufacturer to introduce three or more new devices each year, and, on average, devices are upgraded every two years. For most companies this makes it impractical to maintain an inventory of devices. Accessing devices in‑the‑cloud can be a good option. The growing number of cloud service providers makes it possible to access a broad range of devices and pay only for the time needed. However, there are limitations to relying solely on device rental. It’s important to thoroughly research the different plans to see which will work with the testing and timeframes surrounding when the devices will be needed. A hybrid approach is often the best way to go from both an efficiency and economic standpoint. A hybrid model involves purchasing a small, or very manageable, number of devices that are maintained in‑house for executing full sets of tests, and then utilising devices in the cloud for basic compatibility and functional testing.

FULLY OUTSOURCED OPTION Completely outsourcing mobile testing is a strategy that can work well for organisations of all sizes. This approach’s big advantage is that it eliminates the challenges and headaches of managing and maintaining an inventory of mobile devices. Firms with mobile specialists typically understand the testing nuances, and likely have mobile automation expertise, which will likely translate to broader test coverage and improved testing. Better firms, because of their experience, can also help develop the device and testing matrix that will provide the optimum test coverage at the lowest cost. Mobile is rapidly becoming the primary user interface, which means mobile testing will continue to increase in importance. Applying a thoughtful approach and rational analysis will go a long way in developing a mobile strategy that will provide the optimal level of testing.

Reference 1.

T E S T M a g a z i n e | J a n u a r y 2 01 6

T E S T I N G

ComScore August 2015 US Smartphone Subscriber Market Share.

In 2012, Virgin Media began one of the largest mobile transformation programmes ever undertaken in Europe. Beverley Wells, Portfolio Test Lead â&#x20AC;&#x201C; Strategic Programmes and Heather Cumming, Test Service Portfolio Lead â&#x20AC;&#x201C; Digital Entertainment, Virgin Media, share the successful role the testing department played.

T E S T M a g a z i n e | J a n u a r y 2 01 6

P R O J E C T

C A S E

S T U D Y

he ambitious three‑year project (two years in test), culminated in migrating 6 million+ customer records (containing 4.5 years’ historical data), more than 10 million mobile handsets and over 68 million equipment items (i.e. mobile accessories) onto a brand‑new IT system over a single weekend. Test’s role was to be instrumental to the project’s delivery, requiring the department to undertake nine testing phases (FAT, IOT, SIT, E2E, OAT, ORT, UAT, performance and security) and run over 22,000 test scripts across 71 systems, whilst fulfilling the following project‑critical objectives: • Quality: Ensuring zero business‑critical defects and that any non‑critical defects had agreed resolution paths at go‑live. • On‑time and on‑budget delivery: Technology testing needed to be completed within a fixed budget by 12 January 2015 to enable the business to complete UAT before go‑live.

APPROACH TAKEN When devising the approach, the Test Department had to ensure it addressed challenges such as a lack of clarity between the main parties as to areas of responsibility within Test. There were also time constraints to take into consideration, as testing needed to be completed by 1 December 2015 to allow time to complete UAT before go‑live. The fact that there was no single end‑to‑end design of the solution available was also a challenge. And the diversity of the workforce, which was spread across multiple locations (the UK, Central Europe, India and Asia) and time zones, meant communication was critical. Additionally, disjointed requirements that failed to support the end‑to‑end customer journey, as well as incomplete non‑functional requirements, needed to be combatted. Finally, in parallel to delivering the most ambitious test programme the team had ever undertaken Virgin Media was also aiming to achieve TMMi Level 3 certification. To address the challenges above, the test approach aimed to: • Defining the boundaries, scope and ownership of testing: This involved sitting down with the Asian business partner and the outsourcer at the outset of the project to agree the scope of testing and areas of responsibility. • Introducing risk based testing (as per TMMi Level 3): To deliver within the limited test window, market and customer

data were assessed in order to prioritise customer journeys, and devices and browsers for testing. The scope was then agreed with key project and test stakeholders. • Creating a set of principles upon which testing would be based: This required the team to: ££ Test smart – automation and tools would be used wherever possible to allow the test department to achieve the required coverage, improve efficiency and time taken. ££ Shift left where possible to embed quality into the requirements and design. ££ Shift right to ensure subsequent phases are delivered with the same high level of quality, e.g. UAT management, deployment support and warranty. ££ Set up strong lines of communication within the project to avoid misunderstandings and assumptions. ££ Implement strong governance and clear reporting to enable effective project management. This enabled Virgin Media to be responsive to issues which could have otherwise led to a ‘test miss’. ££ Work proactively and collaboratively within the project where key test inputs were missing to resolve these issues. Additionally during 2014, the team successfully implemented the requisite quality processes and standards necessary for them to attain TMMi Level 3. Early in 2015, during the most critical period of test execution on the project, the company decided to aim for TMMi Level 5, the highest level of accreditation, and attained accreditation in November 2015.

In parallel to delivering the most ambitious test programme the team had ever undertaken Virgin Media was also aiming to achieve TMMi Level 3 certification.

BEVERLEY WELLS PORTFOLIO TEST LEAD – STRATEGIC PROGRAMMES VIRGIN MEDIA

Beverley has over 20 years' test management experience in telecoms and media and has recently been awarded UKIT Manager of the Year.

EXPANDING THE ROLE OF TEST No one within the business had undertaken a project of this scale or nature before, therefore there was no precedent to follow: quite simply, innovation was essential. To deliver, Virgin Media therefore expanded the role of Test, going far beyond the function’s usual scope. This proved essential not only because it enabled the team to deliver against its own project objectives but because it kept the entire project on track. This was achieved through the following two means:

RELEASE MANAGEMENT

Firstly, an approach that was akin to release management was adopted: The sheer

HEATHER CUMMING, TEST SERVICE PORTFOLIO LEAD ‑ DIGITAL ENTERTAINMENT VIRGIN MEDIA

Heather has over 20 years' test management experience in utilities, telecoms and digital entertainment. Her current role is Test Portfolio Lead for Digital Entertainment at Virgin Media.

T E S T M a g a z i n e | J a n u a r y 2 01 6

Tactical objectives and a clear strategy for UAT were also clearly outlined by the team to ensure everyone was clear on what they were working towards and how this would be achieved.

P R O J E C T

volume of defects logged during testing (over 12,000) presented a significant project challenge. To address this, for each issue, the team: • Articulated what the issue was: As the Asian systems were new to the team, this often involved establishing whether a software defect was genuine or attributable to a lack of understanding of the system. • Identified the root cause: If a defect was established as genuine, there was a high probability it was a result of integration issues between the existing and new systems. To test this theory, we adopted the role of facilitator. This involved getting the two teams concerned together in a room and white boarding the flow of data to identify the cause of the issue. • Proposed a solution: Having identified the root cause, the team then articulated a solution, got stakeholder agreement and then either made the necessary changes or co‑ordinated when they would occur. • Due to the lack of non‑functional requirements, the team identified all the required SMEs within the business to document and feedback the targets required.

USER ACCEPTANCE TESTING

Secondly, Test took on UAT. UAT on this scale (involving over 100 people across multiple locations) had never been attempted before by the business. Test’s remit was originally simply to support the UAT process, but it quickly became clear that they were ill‑equipped to deal with testing on such a large scale, potentially impacting quality. Without waiting to be asked, the team stepped up to the challenge, seconding four people from within the Test team to work with the UAT teams. Once in place, the four members co‑ordinated the project across the UAT teams, introduced hourly checkpoints and ensured everyone was clear on their areas of responsibility and completion dates. One Business Readiness Manager said: “Since working with Beverley and the team on the journey to go live I have developed new found respect for the work that they do in Test. Quite simply, without their support and guidance we would not have been able to complete UAT in time for go‑live.” The team introduced planning and reporting into UAT in order to ensure they completed the project on time. In total UAT1 had 3000 scripts to check within 10 weeks and UAT2 had 2600 scripts to check within four weeks across 12 different business areas.

T E S T M a g a z i n e | J a n u a r y 2 01 6

C A S E

S T U D Y

The team also introduced good governance through: • Daily progress calls. • Daily reporting of progress to plan. • Monitoring of the impacts where progress to plan is behind schedule. • Monitoring of defects by business priority in ‘real time’. • Quality check daily reports which included: tests executed; tests not linked to defects; open defects not linked to a test case; failed or blocked tests that can be retested; number of tests blocked by data defects. Tactical objectives and a clear strategy for UAT were also clearly outlined by the team to ensure everyone was clear on what they were working towards and how this would be achieved.

PRIOR TO SANITY & UAT2 START

Tactical objectives included confirming all workstreams publish their execution plan for the 3‑week testing period, that all team members completed their training and readiness preparations, i.e. desk space, hardware, resources.

DURING EXECUTION

During the execution period, key strategies that were enforced included ensuring business stakeholders attended all the progress calls and Business Test Leads managed the daily execution and promptly raised any issues. Additionally, business impacts needed to be included in defects and quantified, and same day retesting of business and highly critical defects were implemented. Other objectives included having the QC up to date by 3pm daily, encouraging team members to not rely on email and instead pick up the phone if needed, and authorise overtime/weekend work if progress was behind schedule.

COMMUNICATION TACTICS The project required the testing management team to co‑ordinate 650 test professionals (120 in‑house) spread across three continents. Effective communication was therefore essential. Tactics to communicate effectively included: • Defining terms: One of the first actions was to sit down with the Asian business partner and establish, for example, what key terms such as ‘end‑to‑end testing’ meant to them. By doing so, it quickly

S T U D Y

TIME MANAGEMENT Virgin Media already produces a standard daily test report; however, Beverley designed a second, more comprehensive report, both to use as an effective means of updating senior leadership on progress, and in order to easily plot where Test was against projections. Figure 1 gives an example of a graph included in the 7‑page weekly executive summary report showing the overall open defect trend (comparison of Actual vs. Expected). In Figure 1, the red line shows actual open defects (including defects that are newly

Approx. 60% through projects test schedule by this point of time

18-Aug

25-Aug

11-Aug

04-Aug

21-Jul

28-Jul

14-Jul

07-Jul

23-Jun

30-Jun

16-Jun

09-Jun

02-Jun

19-May

26-May

12-May

28-Apr

05-May

21-Apr

14-Apr

07-Apr

31-Mar

24-Mar

519 open defects on the 28th of April

17-Mar

became apparent that we had different understandings of basic key terms, requiring us to jointly redefine them to avoid confusion. • Conducting theoretical test scenarios: These were undertaken with the Asian partner at the beginning of the project to see how they’d manage different testing scenarios and to ensure there would be full end‑to‑end coverage, given the lack of an end‑to‑end solution design. This quickly uncovered missing scenarios that the management team either ensured the partner addressed or included within the test scope. • Avoiding idioms: Given that the team included a mix of Indian, Chinese, English, Russian and South African members, it was essential that expressions which could be misunderstood by non‑native English speakers were avoided. • Clear reporting: A comprehensive new daily test report (since adopted by the other test managers) was introduced to provide senior‑leadership with progress updates. (Test’s role became so central that Beverley as project lead met with the CIO weekly in the run‑up to launch to give status updates). • Two‑way communication: Close communication between peers and senior leaders within the project’s test team was encouraged, harnessing collaborative technologies such as video and web conferences for effective communication between on‑ and offshore teams as well as a lot of face‑to‑face communication onshore. As a result, in an anonymous employee survey the team rated project communication as 8.1/10 (10=excellent); the way the different parties worked together 8.6/10; the extent to which their ideas were listened to 8.1/10; and communication on the project overall 8.1/10.

10-Mar

C A S E

03-Mar

P R O J E C T

Figure 1. An example of a graph included in the weekly executive summary report showing the overall open defect trend (comparison of Actual vs. Expected).

raised, or open with the development team or in re‑test). This also includes defects that are closed on a hotfix and have yet to be re‑tested against a formal patch release. The vertical dotted line represents the point in time where one would expect the numbers of defects to start coming down, i.e. approximately 60% through testing. The grey bell curve is designed to show the shape of how the open defects should vary over time. The maximum point on the graph is aligned to approximately 60% through the test phases (excluding UAT). At this point one would expect the numbers of defects to start to come down. The purpose of this graph is to monitor the shape of the actual curve rather than analyse the numbers of open defects.

There were no issues in the first week that were attributable to a test miss. Whilst no target was set for this phase, this was beyond expectations given the scale of the project delivered. As a result, there was virtually no impact on the end‑customer, with the company’s contact centre reporting that queries did not break expected levels during the cutover and in the period immediately following. The agents left work on Friday with one system and returned on Monday to the brand new system, up and running. Despite the scope changing part way through to include a new mobile phone financing option for customers, the test project was delivered on its original deadline and within test budget.

RESULTS Thanks to the team either meeting or exceeding their targets (including achieving a 93% defect acceptance rate against a contractual target of 85%) the project was successfully delivered with zero open business critical and highly critical defects at go‑live. The cutover and launch was successfully achieved including the full customer migration and IT implementation within the expected 47‑hour deployment window.

T E S T M a g a z i n e | J a n u a r y 2 01 6

IS THE FUTURE OF TESTING A DEVELOPERâ&#x20AC;&#x2122;S DISCIPLINE? Wolfgang Platz, Founder and Chief Product Officer, Tricentis on how the future of test automation affects us.

T E S T

A U T O M A T I O N

e have a problem. Although agile methodologies and DevOps have reduced development cycle times significantly, today’s software testing is still dominated by manual tests. This is how the testing efforts of a typical enterprise looks today:

demand. It is not an exaggeration to say that literally all of the companies I have talked to want to increase their test automation rates. The testing landscape is prone to seismic shifts. Every time the landscape changes the question appears again: Is software testing dead? Everyone wants to know whether robots have finally come to steal their jobs. It’s a valid question, particularly to those who fill the two major roles typically found in enterprise testing:

MANUAL TESTERS

Manual testers define test cases and execute them manually. They identify the required test data objects and also do a fair amount of exploratory testing.

Manual testers and automation engineers will see their job descriptions changing – and as a result, become more valuable to their respective companies than ever before.

AUTOMATION ENGINEERS

What do we see? • Test automation rates are low – they barely exceed 30%. • The majority of automated test cases use UIs as the access to the apps to be tested. • API tests are the exception rather than the standard. Obviously, this testing model will not fare well once DevOps is introduced into the enterprise. Whether they like it or not, companies are being forced to adopt test automation early on in the development process just to keep up with the market

Automation engineers design sophisticated test frameworks and are skilled software developers. They automate, maintain, and run test cases. These roles are bound to evolve with continual changes to the software landscape. As more enterprises increase automation rates, less test cases will be left for testers to run manually. As more manual testers learn to automate their tests, automation engineers will find that the focus of their work changes as well. Manual testers and automation engineers will see their job descriptions changing – and as a result, become more valuable to their respective companies than ever before. Let’s take a deeper look at this by first predicting how software testing will change as enterprises adopt higher rates of test automation.

WOLFGANG PLATZ FOUNDER AND CHIEF PRODUCT OFFICER TRICENTIS

Wolfgang Platz, the Founder & Chief Product Officer of Tricentis, has over 20 years of technology experience. Wolfgang co‑founded Tricentis in 2008 as a testing consultancy and laid the cornerstone for the development of its enterprise software testing product, Tosca Testsuite. Today, he is responsible for global product management, R&D and delivery of product‑related support and services to customers.

T E S T M a g a z i n e | J a n u a r y 2 01 6

Once the tools allow test automation to be embraced as a business, rather than a developer’s discipline, today’s manual testers will shift the bulk of their work to automated testing.

T E S T

WHAT DOES THIS MEAN FOR THE FUTURE? • The share of testing in IT budgets will not increase,1 despite the demand for speed. Testing will become much faster and more efficient in comparison to today. • Test automation rates will increase dramatically, simply because they have to in order to keep testing in parallel with the increased pace of agile development. • Remaining manual tests will be predominantly exploratory. • Automated UI tests will be more important than they are today – but they will require a smaller share of the effort in relation to full automation. • API tests are much more stable and faster than UI tests. They will be the dominant method of testing apps in the future. • Service virtualisation will play a vital role as an enabler of high test automation rates in complex and interconnected landscapes.2 It will be an integral part of test automation. Assuming that these predictions are accurate, how would these changes affect today’s manual testers and automation engineers? If we have a brief look at the history of test automation, we recognise that maintenance efforts have always been the biggest threat to automated test cases: when the automated testing maintenance efforts outweigh the manual execution efforts, why stick with automation? Using a sophisticated framework eases the pain, but on top of being extremely expensive to implement and maintain, they turn test automation into a software

T E S T M a g a z i n e | J a n u a r y 2 01 6

A U T O M A T I O N

development project – excluding business testers from the automation process.

BOTTOM LINE

Any successful test automation in the past has required heavy software development. Does that mean that all our manual testers of today need to be replaced by ‘development‑testers’ in order to meet the rigours of agile and DevOps practices? To complicate things further, API testing today is purely the domain of the technically savvy. How can enterprise API testing be expanded without first hiring more developers? Don’t forget service virtualisation! Which will soon be a fundamental requirement in complex system landscapes. Without service virtualisation, testing environments will not be available to support the systems in continuous test – who will handle that?

SUMMARY Is the future of testing a developer’s discipline? The answer is no. Testing will not become a developer’s discipline because it simply can’t. You may be aware of how hard it is to find suitable software developer talent these days. The sheer number of developers that would be required for continuous testing simply does not exist on the market. Not to mention that software developers themselves are not usually too fond of testing. If we were to ask software developers to do 85% of the testing, the testing would either never happen, or distract developers of their primary job – which is to innovate.

T E S T

A U T O M A T I O N

The key to success doesn’t lie in restructuring the testing department, but in changing the way we leverage the tools of software testing. Besides solving the maintenance issue of test automation, software testing tools of the future will need to enable and encourage today’s manual testers to take on a more proactive role in test automation. Better yet, these tools should enable manual testers to automate all kinds of tests: UI tests, API tests, and enable service virtualisation. This is a true paradigm shift: test automation needs to become a business discipline. At Tricentis, we firmly believe that the following role model can (and has) become a reality.

TODAY’S MANUAL TESTERS ARE THE AUTOMATION SPECIALISTS OF TOMORROW Once the tools allow test automation to be embraced as a business, rather than a developer’s discipline, today’s manual testers will shift the bulk of their work to automated testing. As a result, their job profile will be enriched and their career pushed forward: while still doing some manual testing, particularly exploratory, they will also have the capability to automate UI and API tests, as well as service virtualisation. The educational path to becoming an automation specialist is straightforward: after a few days of training with the testing tool, manual testers will be able to deliver automated test cases.

automation specialists productive: writing custom‑control steering for user‑interfaces, and providing business abstractions of messages on the API and service virtualisation level. They will even be required to ensure that object‑oriented principles in the set‑up of automated test cases are being applied by the automation specialists. Automation engineers of today will find that their technical skills become all the more valuable in the future as they enable the work of the automation specialists.

CONCLUSION The test automation rates of most enterprises today are way too low to enter the age of DevOps. Attempting to solve the problem by turning manual testers into savvy technicians dealing with sophisticated test automation frameworks simply does not work. Instead, the tools of the future needs to embrace manual testers and make them productive for any type of test automation, be it UI test automation, API tests or service virtualisation. Making automated testing a business discipline was the dream that inspired the development of Tricentis’ software testing suite, Tosca Testsuite. With tools like this, manual testers will become automation specialists while automation engineers provide the foundation for automation specialists to work from. What will be the result of this new approach to automated testing? Not only will today’s manual testers and automation engineers find that they have become more valuable than ever, but that together they will achieve the highest test automation rates we have ever seen.

TODAY’S AUTOMATION ENGINEERS WILL FIND THEIR SKILLS LEVERAGED With manual testers becoming automation specialists, will today’s automation engineers lose their jobs? No, the need for technical skills will never decline. Automation engineers will need to provide the customer specific basis to make References 1. 2.

IT’s share on overall budgets is increasing, and currently even QA budgets are increasing faster than IT budgets – but this trend will flatten (see World Quality Report 2015/16). The border between API tests and service virtualisation is not a sharp line but an area of crossover.

T E S T M a g a z i n e | J a n u a r y 2 01 6

DRIVING THE DEVOPS AGENDA Iain Chidgey, Vice President International at Delphix, explores the rise of DevOps and the associated challenges to its adoption.

n today’s culture of immediacy, achieving speed, agility and continuous delivery is the holy grail of IT. Tasked with delivering updates and new software applications as a point of competitive differentiation, the ability to deliver on weekly, daily or even hourly releases has become a prerequisite for success. Yet teams are increasingly restricted by the various ways in which key technology stakeholders operate. Software development and IT operations teams use different tools and approaches and are driven by different incentives. This means breaking down the silos between them is the only way

T E S T M a g a z i n e | J a n u a r y 2 01 6

to improve collaboration and help them to adopt best practise from each other. Doing so enables the business to deliver faster, reduce delivery cycle times and increase agility, while retaining a high quality and professional approach to IT.

THE RISE OF DEVOPS DYNAMICS In response to this, many organisations have hailed DevOps as the answer – seeing the benefits of bringing development and

infrastructure teams closer together. In fact, Delphix research into the state of DevOps has found that 77% of UK organisations have introduced dedicated budgets and support teams for DevOps, with 35% spending £1 million or more on DevOps initiatives per year. So what is DevOps and how is it helping businesses to embrace change? DevOps is a movement that promotes the faster, more efficient delivery of high‑quality software by engaging different teams on mutual business goals, using a range of best practices surrounding technology, people and processes.

D E V O P S

Success of this movement often includes names of tech titans including Google, Facebook and Netflix. The industry has seen reports of these high performers carrying out deployments 30 times more frequently and up to 8000 times faster than the average company using this approach. That means the innovation and customer experience pay‑off for those enterprises is overwhelming. Gene Kim, author of ‘The Phoenix Project’, comments that “high performers can execute a deployment in minutes or hours, whereas lower performers need weeks, months, or quarters. Of course, we have to take investment to go from weeks, months, and quarters to on‑demand, executing within minutes. But the value of doing that is too high to ignore.”

EMULATING DEVOPS SUCCESS A number of contributing factors make DevOps leaders particularly successful. The secrets behind effective approaches can provide some valuable lessons for those DevOps practitioners looking to emulate the speed at which high performers can innovate and compete. First of all, automation is a central factor in DevOps, used to build, test, release and deploy software. It means teams can gain control and governance over processes while ensuring speed, agility and the quality expected of modern‑day systems. When it comes to employees and processes, businesses need to shift toward an outcomes‑driven approach, creating an environment that thrives on collaboration. Teams can then remove the barriers presented by process‑orientated IT, which focuses on stability and risk reduction and results in the slowdown of delivery. Despite growing adoption, DevOps is inconsistently defined. Common among leaders is the understanding that DevOps is not simply limited to encouraging collaboration between teams. In fact, it’s about relying upon developers embracing operations functions and deriving scale through automation. Given that developers drive the value creation in IT, DevOps should be used to help them clearly communicate what they need for faster software delivery. An extension of this trend is determining who owns the responsibility for driving initiatives. Findings indicate that high performers were four times more likely to support DevOps groups and as a result, 200% more likely to say they are completing DevOps practices effectively. By empowering

dedicated teams, DevOps leaders accept that it shouldn’t be a trade‑off or cost‑cutting exercise and that having the right volume of staff to carry out tasks is imperative. Core to this is understanding that, though technology has a significant impact in speeding up processes, it is no substitute for human resource. Finally, high performers clearly define what success looks like using measurable outcomes. According to the Delphix State of DevOps report, they are 12 times more likely to select “uncovering defects in the development cycle” as a central part of defining DevOps – which is a tangible metric. This is in preference to collaboration and automation, which while both are critical, are much more difficult to measure. DevOps practitioners must focus on how they define, measure and prioritise ownership for DevOps initiatives to ensure they are on track to compete and make their foray into the movement worthwhile.

Highlighted as the biggest challenge uncovered in the research report, 46% of DevOps leaders say that testing environments are limited due to data issues.

DEVOPS THE DATA The final piece of the puzzle underpinning all DevOps success is access to secure data on demand. Highlighted as the biggest challenge uncovered in the research report, 46% of DevOps leaders say that testing environments are limited due to data issues, with more than half of survey respondents waiting a week or longer to refresh non‑production environments from production. That’s time organisations simply cannot afford to lose when competing on time to market. Ultimately, the only way to reduce the time taken to provision data for business‑critical applications and limit handoffs between teams is to make the underlying data more agile. An effective method for tackling this is to insert a new layer into IT infrastructure to do the heavy lifting for you. Using virtualisation, data copies no longer have to be duplicates and data sets can be refreshed and reset on demand. As a result, environments can be served up in minutes not months, bookmarked and shared between users, and rolled back to any point in time. This streamlines the data supply chain and means projects can be completed far quicker, and a self‑service approach is promoted to empower users to copy and share data without fear. By championing data as a service (DaaS), data is finally put into the hands of those that need it, when they need it, to support developers in driving innovation and competitive advantage.

IAIN CHIDGEY VICE PRESIDENT INTERNATIONAL DELPHIX

Prior to joining Delphix, Iain was VP and General Manager EMEA for ArcSight, a leading global provider of compliance and security management solutions. In his 20 years of experience in the IT industry, Iain has held senior sales roles for various high growth software organisations, including Portal and Oracle both in EMEA and the US. He has a BSc with Honours in Systems Modelling from Sheffield Hallam University.

T E S T M a g a z i n e | J a n u a r y 2 01 6