Test Magazine November 2015

NOVEMBER 2015

THINKING OUT OF THE BOX DOES SIZE MATTER? THE EUROPEAN SOFTWARE TESTING AWARDS 2015

SPECIAL

FLAVOUR OF THE MONTH

1

C O N T E N T S

T E S T

M A G A Z I N E S E C T O R

|

F O C U S :

N O V E M B E R

2 0 1 5

U T I L I T I E S NEWS

Global software testing news 5

20

THOUGHT LEADERSHIP

Staying ahead of the game 10 DEVOPS

12

Flavour of the month TEST AUTOMATION

A new era in test design

16

COVER STORY

How one of the Big Six ensures business as usual 20 OUTSOURCING

26

Does size matter?

SECURITY TESTING

Security and compliance – friend or foe?

28

SUPPLIER PROFILE

32

CA Technologies CAREER CORNER

Thinking out of the box

16

26

36

AGILE IMPLEMENTATION

Fail better

39

SUPPLEMENT

The European Software Testing Awards 2015 Supplement 41

28

41

T E S T M a g a z i n e | N o v e m b e r 2 01 5

3

L E A D E R

SOFTWARE IS NOW PRIMETIME TV

A

short while ago I caught glimpse of the latest instalment in the popular Crime Scene Investigation franchise: CSI Cyber. It’s already on its second series in the US, so this might be old news, but as someone who rarely watches live TV nowadays, the show was new to me.

A hotel was hacked, a murder was covered up, phones were physically rebuilt – screen and all – “to access the hard drive”, and viewers at home learnt what a decoy router was, and how, ostensibly, you could be framed for murder should you accidentally connect to it.

In-between shots of the laboratory and complex computer simulations seemingly generated in minutes, the show focused on detectives out arresting suspects and, typical of the crime show genre, interrogations. But then, to reinforce the ‘cyber’ theme, the dialogue was composed of phrases such as “let’s see what the digital evidence shows” and “your smartphone places you at the crime scene.”

While the show has come under fire for simplifying the technical aspects, the larger cultural effect cannot be ignored. For millions of viewers around the world – the show has been licensed to over 200 territories – an awareness of the new digital world and personal safety/security online might take root.

In a recent interview with The Telegraph, forensic cyber psychologist, Professor Mary Aiken, whose work CSI Cyber was based on, was quoted as saying: “We need to start thinking of the online world as a real place.”1 By showcasing cars and baby monitors being hacked, the TV show is bringing topical concerns to the public. Making them aware of the real consequences of online behaviour.

Another show that has caught my eye is the new documentary series Tomorrow’s Food on BBC One, which investigates technical advances in the agricultural space. The programme presents robots in restaurants and the latest

1.

CECILIA REHN EDITOR OF TEST MAGAZINE

in nutritional science, but the most fascinating part to me was learning about researchers in Australia who are utilising drones and advanced software to replace farmers – from monitoring and herding cattle to scanning the crops in order to spot and eliminate weeds.

Whether entertainment or educational, TV shows like these demonstrate just how much the public is craving knowledge and understanding of the potentials and pitfalls of software. Of course, we’d prefer the dialogue to feature more prominently the role of testing and QA, but we can only hope those who are new to software in general and might only have just learnt about hackable devices continue to read on and find out about us.

I haven’t just been watching TV in the last month! We recently held our third annual European Software Testing Awards, which was a huge success. You can see photos, a list of all the finalists and winners, and read my thoughts on the night in our special supplement beginning on page 41.

Additionally, during the Awards day we hosted three Executive Debates for a select audience in central London. Feedback from the day was that the roundtable sessions were very informative and engaging. If you’re interested in similar events, then don’t miss our TEST Focus Groups coming up in March: www.testfocusgroups.com.

Finally, read on in this issue for discussions on security testing, DevOps and a great piece questioning if an exclusively agile background is a problem amongst new recruits.

I hope you enjoy the issue!

© 2015 31 Media Limited. All rights reserved.

TEST Magazine is edited, designed, and published by 31 Media Limited. No part of TEST Magazine may be reproduced, transmitted, stored electronically, distributed, or copied, in whole or part without the prior written consent of the publisher. A reprint service is available. Opinions expressed in this journal do not necessarily reflect those of the editor of TEST Magazine or its publisher, 31 Media Limited.

ISSN 2040‑01‑60 EDITOR Cecilia Rehn cecilia.rehn@31media.co.uk +44 (0)203 056 4599 ADVERTISING ENQUIRIES Anna Chubb anna.chubb@31media.co.uk +44 (0)203 668 6945 PRODUCTION & DESIGN JJ Jordan jj@31media.co.uk 31 Media Ltd, 41‑42 Daisy Business Park 19‑35 Sylvan Grove London, SE15 1PD +44 (0)870 863 6930 info@31media.co.uk www.testingmagazine.com PRINTED BY Pensord, Tram Road, Pontllanfraith, Blackwood, NP12 2YA

@testmagazine

cecilia.rehn@31media.co.uk

TEST Magazine Group

‘The real-life spook behind CSI: Cyber’, The Telegraph, http://www.telegraph.co.uk/culture/ tvandradio/11972732/The-real-life-spook-behind-CSI-Cyber.html (3 November 2015).

T E S T M a g a z i n e | N o v e m b e r 2 01 5

S O F T W A R E

T E S T I N G

ESPORTS TOURNAMENTS GAINING POPULARITY New data from Juniper Research has found that online eSports services, such as Twitch, are estimated to reach over 310 million viewers by 2020, increasing from 133 million this year and bringing in almost US$1 billion in revenues from subscriptions alone.

The researchers believe that by 2020 eSports audiences will surpass the levels of the NFL tournament and approach that of Formula 1: • eSports: 102 million global viewers 2014.

• NFL tournament: 220 million unique viewers in 2014.

• Formula 1 racing: 400 million global viewers in 2014.

Juniper Research’s latest study, Digital Games: eSports 2015-2020, forecasts that eSports is on the cusp of significant growth, with tournaments and prize pools expanding in size and value; key eSports tournaments such as last year’s League of Legends World Championships final drew in over 27 million viewers.

RUSSIA BANS FOREIGN SOFTWARE IN STATE AND MUNICIPAL AGENCIES Russia’s Prime Minister Dmitry Medvedev, has signed an order prohibiting state and municipal agencies from using all foreign produced software.

According to the Russian government’s website, where the signed order was published, state bodies and organisations purchasing software necessary for their work will have to check their procurements are included in the special register of Russian software. Should no Russian-made alternatives exist, exemptions would be allowed.

This special register is to be created by an expert group from the Communications Ministry. It will comprise of software copyrighted by the Russian government, state agencies, municipal bodies, Russia-registered non-profit groups or commercial companies of which at least 50% is owned by Russian nationals.

Additionally, any software procured for official use must be available all over the country and the portion of license fees paid to

N E W S

In particular, advertisers are set to benefit, with revenues forecast to increase by almost 250% from 2015 to 2020. This will mirror an increased willingness by consumers to subscribe to online eSports channels and broadcasters, with subscription revenues reaching almost US$1 billion by 2020.

Lauren Foye, report author, said that “An increased audience, as well as greater impact from the most popular eSports broadcasters will see user adoption of the subscription model increase. Consumers will also begin to spend more time viewing eSports content in general, leading to an increase in terms of ad impressions.”

However, there are key challenges facing eSports, primarily the alleged widespread use of drugs to enhance performance. As a result, the Electronic Sports League (ESL) have recently implemented drugs screening. If eSports is to be seen as a professional sport, then it must be regulated and controlled as such, in order to attract the necessary sponsorships and partnerships to drive growth.

foreign legal entities must not surpass 30% of overall proceeds.

The signed order also details measures of government support for Russian software companies and plans to promote the development of the country’s own operating systems.

However, promoting internal software development may not be the only prompt for Medvedev’s order. Russian politicians and officials have a history of discussing and fearing the dangers of surveillance and data leaks that come with the use of foreign computer programs in official organisations.

Only last month, a group of MPs from the Communist and Liberal-Democratic Parties promised to draft a bill that would ban government employees, under threat of fines or even being fired, from using products

by Google, Yahoo and WhatsApp. The draft bill was justified “because the international situation was uneasy and it was important to prevent any possible data leaks.”

Last year, citing fears of eavesdropping, members of the centre-left party Fair Russia proposed officially advocating that all parliamentarians to stop using iPhones and iPads.

5

THE BANK OF ENGLAND INCREASES CYBER SECURITY SPENDING Recently, the Bank of England has increased spending on cyber security in a move to combat the augmented threat of cyber attack. The Bank has also improved cyber security training for staff across the organisation, which includes warning employees to be wary about revealing their roles at the Bank.

Training methods such as fake phishing campaigns have been implemented to see if staff took the bait. According to the minutes of the latest Bank of England Court of Directors meeting, the campaign has been largely successful, with fewer and fewer staff clicking on suspicious links and attachments.

“Significant progress had been made in applying controls, but at the same time external threats had been increasing. The Bank had numerous information assets and was a key part of the UK critical national infrastructure,” the report said.

Furthermore, the report stated: “A £20 million three-year investment programme had been agreed in 2013 and there had also been a substantial increase in day-to-day resources in the IT Security and Information Security Divisions, with an uplift of 74 FTE [full-time equivalent] staff”.

Recognising the importance of the combination of technical solutions and staff awareness, the report highlighted staff training, including warning staff of discussing their work roles on social media accounts, which are easily hacked.

“Technical controls put in place had strengthened the Bank’s ability to prevent, detect and respond to attacks. But no technical fix could guarantee security 100%, so at the same time significant effort had been made to improve security awareness among all staff, and incident handling procedures had been strengthened,” the report said.

T E S T M a g a z i n e | N o v e m b e r 2 01 5

6

MULTIPLE DEVICES WILL SUPPORT UK CHRISTMAS SHOPPERS According to recent research, Britons are planning to spend £24.4 billion on Christmas gifts this year, with the average person spending around £489.

The report showed that found that 25-34 year olds will be spending the most.

Research firm RadiumOne found that among those who know how they’ll buy their gifts, 45% plan to research and buy presents online while 13% said they will do both research and shopping in-store.

Nearly 25% of online Christmas shoppers said they will use at least two devices to make their purchases. And a majority 72% of respondents said that desktop computers are their preferred choice for online shopping.

Rupert Staines, RadiumOne’s European Managing Director, said: “Christmas is, by far, the biggest sharing event of the year and it’s fascinating to see how influential technology now is, with its multiple devices, in the whole of the Christmas decision-making process.

With the UK population being the most

REFORMS IN THE US WILL SEE THIRD PARTY RESEARCHERS ABLE TO EXAMINE MEDICAL DEVICES FOR SOFTWARE FLAWS Last month, the Library of Congress issued exemptions to the Digital Millennium Copyright Act, which will allow independent researchers to begin examining medical devices for software defects.

Up until now, independent research of software in medical devices has been banned under a nearly 20-year-old law known as the Digital Millennium Copyright Act (DMCA). One provision bans anyone from sidestepping systems designed to protect copyrights for any reason, ensuring pirating a movie or inspecting an implanted defibrillator remained illegal.

Now, following copyright reforms passed by the Library of Congress, which revises the DMCA every three years, independent researchers will be lawfully allowed to study

T E S T M a g a z i n e | N o v e m b e r 2 01 5

S O F T W A R E

sophisticated e-shoppers in the world, it’s absolutely critical for brands and retailers to better understand these consumer connections and their fast-evolving relationship with technology to ensure their advertising delivers the ideal Christmas sales boost.”

most medical devices, as well as automobile software and other consumer devices.

Consumer and tech groups have pushed for changes to the law, especially after research has revealed software vulnerabilities in cars and other connected devices. Device makers, on the other hand, have argued that permitting third parties to examine software will invite copyright infringement, which could seriously harm their business.

The Library of Congress ruling comes with some stipulations. The rule will not take effect for a full year, and it will not allow all forms of research.

Researchers will be allowed to study devices that are not implanted inside the body. Advocates did push for the right to study implanted devices providing the patients gave consent. However, the Library of Congress came down on the side of medical device manufacturers who argued that batteries – and therefore functionality – would be negatively affected if researchers regularly accessed devices. Patients require additional surgeries to replace failing implant batteries.

T E S T I N G

N E W S

MALAYSIA LOOKING TO EMPLOY 30,000 CERTIFIED SOFTWARE TEST PROFESSIONALS To help achieve the nation’s economic target by 2020, Malaysia is looking to employ 30,000 certified software test professionals.

Speaking at a recent conference at the Universiti Malaysia Sabah (UMS), Faculty of Computing and Informatics Dean Dr. Ag Asri Ag Ibrahim, said under the 11th Malaysia Plan, software testing has been identified as one of the focus areas to be developed as a new source of economic growth for the nation.

“Software testing has indeed become a major topic of interest among universities and industry in Malaysia today thanks to the continuous awareness and promotion activities carried out by MSTB (Malaysian Software Testing Board) under the MSTH (Malaysian Software Testing Hub) initiative,” Ag Asri said.

Ag Asri said UMS is also actively involved in contributing towards achieving the target number of software testing experts.

“As a start, we have conducted a four-day training course for our students in June. The training is designed to prepare them for the International Software Testing Qualifications Board’s (ISTQB) Certified Tester Foundations Level examination.

“Thirty-four participants successfully completed the training, and 30 of them, students including lecturers, are CTFL-certified.”

The seminar was organised in conjunction of the second adjunct lecture visit by the German Testing Board (GTB) team to Malaysia as part of the academic outreach collaboration under MSTH. Germany, according to Ag Asri, has long earned the reputation for quality and innovation, particularly in engineering and other technical disciplines.

Additionally, Germany is ahead of Malaysia as the country surpassed the 30,000 mark for certified software testers back in 2013.

“We are indeed fortunate to have GTB as partner in the MSTH Academic Outreach programme,” Ag Asri said. He is confident that the seminar participants will gain more knowledge from Germany’s experience in software testing and “perhaps we can emulate some of the successes in our own quest for excellence.”

TESTING RECONSIDERED OF PROJECTS WILL BE CANCELLED 31.1% BEFORE THEY EVER GET COMPLETED [1] ONLY

ARE COMPLETED ON-TIME

16.2% AND ON-BUDGET [1]

$312BN

?

Y H W

, O SLOWAN O T S I G TESTIN UAL AND LETS OF N R TOO MA PTABLE NUMBE UNACCE ECTS THROUGH DEF

POOR QUALITY REQUIREMENTS

56%

OF DEFECTS STEM FROM POOR QUALITY REQUIREMENTS [3]

64%

SPENT ON DEBUGGING PER YEAR [2]

CA TEST DATA MANAGER LETS YOU FIND, CREATE AND PROVISION THE DATA NEEDED FOR TESTING.

189%

[1]

MANUAL TEST CASE DESIGN 6 HOURS TO CREATE 11 TEST CASES WITH 16% COVERAGE

A NEW APPROACH TO TESTING

UNAVAILABLE OR MISSING DATA UP TO 50% OF THE

2

(GRID-TOOLS AUDIT AT A LARGE FINANCIAL SERVICES COMPANY: HTTP:// HUBS.LY/H01L2BH0)

AVERAGE TESTER’S TIME IS SPENT WAITING FOR DATA, LOOKING FOR IT, OR CREATING IT BY HAND (GRID-TOOLS EXPERIENCE WITH CUSTOMERS)

TESTING CANNOT OF TOTAL DEFECT REACT TO CHANGE

COSTS ORIGINATE IN THE REQUIREMENTS ANALYSIS AND DESIGN PHASE [4]

THE AVE COST OVE RAGE RRUN IS

TWO TESTERS SPENT TWO DAYS UPDATING TEST CASES AFTER A CHANGE WAS MADE TO THE REQUIREMENTS

(GRID-TOOLS AUDIT AT A LARGE FINANCIAL SERVICES COMPANY: HTTP://HUBS.LY/H01L2C_0)

1

BUILD BETTER REQUIREMENTS

4 HOURS TO MODEL ALL BUSINESS REQUIREMENTS AS AN ACTIVE FLOWCHART AND MAKE THEM “CLEAR TO EVERYONE” [5]

AUTOMATICALLY GENERATE AND EXECUTE OPTIMIZED TESTS

2 BUSINESS DAYS TO GO FROM

SCRATCH TO EXECUTING 137 TEST SCRIPTS WITH 100% COVERAGE [5]

3

THE RIGHT DATA, TO THE RIGHT PLACE, AT THE RIGHT TIME

60% IMPROVEMENT IN TEST

DATA QUALITY AND EFFICIENCY WITHIN 3 MONTHS USING SYNTHETIC DATA GENERATION

(GRID-TOOLS CASE STUDY AT A MULTINATIONAL BANK: HTTP://HUBS.LY/H01L2G50)

4

AUTO-UPDATE TEST CASES AND DATA WHEN THE REQUIREMENTS CHANGE

5 MINUTES TO UPDATE TEST CASES

AFTER A CHANGE WAS MADE TO THE REQUIREMENTS

(AUDIT AT A LARGE FINANCIAL SERVICES COMPANY: HTTP://HUBS.LY/H01L2HP0)

DON’T DELAY, START YOUR FREE TRIAL TODAY

GRID-TOOLS.COM/DATAMAKER-FREE-TRIAL

[1] STANDISH GROUP’S CHAOS MANIFESTO, 2014 – HTTP://HUBS.LY/H01L2JK0 | [2] CAMBRIDGE UNIVERSITY JUDGE BUSINESS SCHOOL, 2013 – HTTP://HUBS.LY/H01L2KY0 | [3] BENDER RBT, 2009 – HTTP://HUBS.LY/H01L2L80 | [4] HYDERABAD BUSINESS SCHOOL, GITAM UNIVERSITY, 2012 – HTTP:// HUBS.LY/H01L2MC0 | [5] CA A.S.R CASE STUDY, 2015 – HTTP://HUBS.LY/H01L2NJ0 Copyright © 2015 CA, Inc. All rights reserved. All marks used herein may belong to their respective companies. This document does not contain any warranties and is provided for informational purposes only. Any functionality descriptions may be unique to the customers depicted herein and actual product performance may vary. CS200-160313

8

S O F T W A R E

www.softwaretestingnews.co.uk

IN-GAME WEARABLE TECHNOLOGY IN NFL SHOWS HOW SMART CLOTHING WILL AFFECT FUTURE SPORTS VIEWING The American National Football League is capitalising on in-game wearable technology to enhance spectators’ and viewers’ experiences, with more leagues to follow, according to the latest research.

A new study has found that the fitness wearables sector will generate over US$10 billion in hardware revenues by 2020, up from an estimated US$3.3 billion this year.

read more online

T E S T I N G

N E W S

Featuring stimulating, intriguing articles and features from experienced software testers and leading vendors, you can be sure that you will stay up‑to‑date with the software testing industry.

MAINTAINING QUALITY AND GAMBLER LOYALTY IN AN OMNI-CHANNEL WORLD Gambling is big business. According to the Office for National Statistics (ONS), in 2013 the average UK household spent £166 a year on some form of gambling. With the rise of online channels, the customer base is only set to grow. But to take place and appease customer demand, the gambling industry needs to understand the importance of each channel and how to survive in today’s omni-channel world.

read more online

INTUITIVE WEBSITES A MUST FOR RETAILERS, NEW RESEARCH SHOWS

TECHNOLOGY CONVERGENCE IS LEADING TO AN AGE OF BRANCHLESS BANKING

Today’s retailers are faced with the challenge to balance sites with richer content, making them responsive across every device while being efficient to navigate. Understanding the user and the user experience, is key.

When it comes to attracting the new Generation Consumer, comprehending the trade-off between complexity of technology and simplicity of experience online is a battleground for strategists and marketers alike.

The banking and finance sector is being fundamentally transformed by the global digital explosion, fuelling the sector’s transition from traditional to omni-channel operations. The trend towards smart banking will continue to gather speed as banks look to minimise operational expenses and improve customer experience. New analysis has identified five key technologies central to the shift to smart banking.

read more online

T E S T M a g a z i n e | N o v e m b e r 2 01 5

read more online

STAYING AHEAD OF THE GAME Siva Ganesan, Vice President and Global Head of Assurance Services, Tata Consultancy Services, gives insight on the complete reimagination of the assurance space and the innovations that are fulfilling the changing corporate and customer expectations.

I

t might seem like a cliché that testing and quality assurance (QA) have moved away from being a reactive pursuit. But it is definitively important to reiterate and reinforce that the philosophy of testing today is unequivocally proactive. Whether it be in the use of tools and methodologies, or in testing consumption patterns, QA organisations are now involved right at the outset of the product’s (or services’) lifecycle in assuring end user satisfaction and fulfilment.

T E S T M a g a z i n e | N o v e m b e r 2 01 5

The shift in focus on customer experience has led to a fundamental change in the delivery of products and services. Digitisation has become the norm. Brands are being consumed in many different digital guises with omni-channel at the forefront of such consumption. Using this new knowledge of how products and services are consumed, QA specialists are simultaneously engineering to deliver to product quality and a fulfilling experience. Customer feedback and sentiment has, in fact, become critical in deciding on the ‘official’ version of the product.

Assurance now means considering the customer’s experience from the beginning.

It sounds controversial, and perhaps a little preservationist, but without this evolution to the modern proactive assurance model, there would be no marketplace left for testing alone.

There simply isn’t room for forces that slow down delivery today. Gone are the days when the QA focus was merely on preserving KPIs! Therefore, as we see speed to market growing in influence, we find assurance specialists innovating and keeping pace; which is why it’s all about the business impact and the customer experience now. And powering these imperatives is a complete reimagination of QA – intelligent automation, agile assurance, DevOps, and a whole new generation of QA tools and frameworks.

SHIFTING LEFT THE RIGHT WAY The most talked about concept in testing in recent times – shift left – is also the most important innovation in quality assurance.

There has been a huge change in attitude towards and within testing through the adoption of this trend. It has had a far reaching impact not only on the time taken for testing, but it has also helped bring down costs, increase overall quality through early detection and fixing, and helped breathe life into rapid release cycles (which are critical for a number of businesses today).

On the face of it, shift left is a fairly straightforward process change. But just like most ideas, it is deceptively simple to understand, yet gloriously difficult to execute. While it espouses early performance testing and more unit testing, it requires the right combination of business, development and testing knowledge. Which is why most QA folks do not shift left the right way!

UNDER THE HOOD STRUCTURAL QUALITY Another innovation in testing has been the introduction of structural quality, which has contributed immensely to optimisation in cost of quality. By analysing the cracks prevalent in the code itself, testing and assurance have taken on a proactive role at the building block level, carrying out pre-emptive checks and fixing problems even before they can arise. Structural quality assurance has aided in acceleration and velocity of production in a big way. Today, there are plenty of tools in the market which help QA organisations look for structural anomalies in the existing ecosystem.

At the 2015 European Software Testing Awards, Debapriya Jena from Tata Consultancy Services was awarded the SQS Testing Innovator of the Year.

T H O U G H T

11

L E A D E R S H I P

VELOCITY-WISE SERVICE VIRTUALISATION AND INTELLIGENT TESTING Two of the most exciting innovations have been that of service virtualisation and intelligent testing. Through the power of virtualisation, QA organisations can carry out end-to-end testing, without the erstwhile need to wait for someone or a system to process the request on the assembly line. Service virtualisation has enabled a greater number of testing cycles to run in the same amount of time. It has ensured that testing and QA are no longer obstacles or impediments to business. When meeting today’s speed to market demands, this aid to acceleration has been a vital innovation, along with the evolution of intelligent testing. New automated tools have completely reimagined the way QA organisations provision test data and how they handle test analysis. With elements of automated test bed preparation, self-generating and self-rectifying test scenarios from requirements, completely independent test data and test case preparation and of course, rapid test execution, intelligent testing systems are all set to become the norm.

The two together have increased test coverage considerably and have helped secure a lifelike production environment, assisting assurance professionals in safeguarding quality with more accuracy than ever before.

THE DEVOPS CONTINUUM The most recent development in the QA arena has been that of DevOps. Adding a whole new dimension to ‘shift left’, DevOps takes the concept to an entirely different plane. It not only has an impact on the QA organisation, but to ensure maximum gain, needs to envelop the entire IT organisation as well. The potential of this innovation in process and people aspects is gaining ground.

THE ASSURANCE PEOPLE TREE And now, perhaps the most under rated development in testing: the way in which assurance professionals have reinvented themselves. From rapid skilling beyond traditional testing methods to equipping themselves to cope with the need for agile methods in delivering to the contours of quality and speed to market, assurance professionals have redefined what it means to be in QA. The shift isn’t just in the development lifecycle; it

is a shift in mind set as well – from reactive to proactive, from detection to prevention, from manual to automated, from bugs to business!

Multi-disciplinary teams with techno-functional expertise and skills around assuring customer experience are being created. And a career of choice in this discipline is seen as a viable proposition for many engineers.

FUTURE OF ASSURANCE INNOVATION

There simply isn’t room for forces that slow down delivery today. Gone are the days when the QA focus was merely on preserving KPIs!

Looking ahead, it is clear that a wide world of possible innovation awaits us all. Robotics has become more mainstream, and when you consider the world of smart devices, wearables and the internet of things (IoT), we’re seeing a different ball game altogether. Whether you’re dealing with products that are worn on human bodies, or smart technologies inside the private sphere, there is opportunity for testing and QA to innovate.

It is already at a stage where it is no longer just about assurance for requirements, design, test cases and test data. It is now getting into the realm of virtual reality with more simulations of real life scenarios taking place. Cognitive and behavioural sciences too, along with hardware, software and firmware are coming together to create the most optimum testing arena for the best consumer experience.

The future will also see a change in how we deliver solutions. Cloud services have become popular, and will become the default. Brands and consumers alike will expect no delay in delivery, and the first time right mentality will prevail.

CONCLUSION The practice of quality assurance lies at the convergence of what the brand wants and what the consumer wants. And it will only get more important as future technology gets developed and more areas of our lives become digitised.

Technology disruptions in the future promise complexity for those who want to avoid it and opportunity for those who embrace it. It is important to stay grounded and to listen to the market forces around us.

Siva’s extended article on how the future of innovation varies across different business sectors is now available online.

SIVA GANESAN VICE PRESIDENT AND GLOBAL HEAD OF ASSURANCE SERVICES TATA CONSULTANCY SERVICES

Siva runs the Assurance Services Unit (ASU) business for Tata Consultancy Services (TCS), which serves a large, diversified customer base globally. With over 25 years in the IT industry, Siva has considerable experience in building

www.softwaretestingnews.co.uk /the-future-of-innovation/

relationships ground up and helping customers unlock tremendous value from their existing testing estate.

T E S T M a g a z i n e | N o v e m b e r 2 01 5

FLAVOUR OF THE MONTH OR TOO HOT TO HANDLE?

It looks as though DevOps is here to stay, and with business stakeholders under increasing pressure, perhaps, argues Renato Quedas, Vice President – Solutions Marketing and Enablement at Micro Focus, it is time to give them a more active role in leveraging DevOps to deliver better results to the business.

T E S T M a g a z i n e | N o v e m b e r 2 01 5

13

D E V O P S

U

sers and customers interact with more software and applications than ever before – and their tolerance for a poor experience is fast diminishing. This assessment is determined by a number of factors including their expectations, product performance and ease of use.

In a complicated landscape of fast‑moving IT trends such as mobile, cloud, consumerisation and rising complexity, organisations are working in more fragmented, agile ways to meet increasingly unrealistic customer expectations and any practice that promises to streamline the ‘development to production’ process is viewed favourably. DevOps is the latest flavour of the month.

Properly executed, DevOps can help IT deliver changes faster and more effectively and create the products that the business has identified as being what the market wants. DevOps is – naturally – focused on Development and Operations and excludes the full scope of the software development supply chain and, equally, does not factor in the views and requirements of the people who determine what must be done and why – namely the business stakeholders.

BUSINESS STAKEHOLDERS – DEFINING THE WHY AND THE WHAT The business stakeholder plays a key role. They must define how IT can meet each expectation and establish and, once underway, communicate any progress to the organisation itself and beyond to customers and partners. While cross‑organisational research and collaboration determines the ‘why’, communicating the ’what‘ to DevOps teams remains a challenge.

This is a fast‑paced environment requiring business stakeholders to make business‑critical decisions quickly. Time spent evaluating the potential impact of change in business priorities could make or break a successful product or service.

This is where DevOps teams and business stakeholders can have ideological differences.

DevOps teams choose their technology how they work with it. But these are strategic decisions that may not factor in the time a business stakeholder needs to pass on the key planning and delivery information required to deliver a market‑leading product or service.

In addition, a DevOps team will need enough detail to estimate, plan and deliver an incremental solution to a business need.

So any shift in business priorities will extend delivery time – and feel at odds with the principles of business agility in responding to change.

Properly executed, DevOps can help IT deliver changes faster and more effectively and create the products that the business has identified as being what the market wants.

MAKING DEVOPS WORK FOR THE BUSINESS To successfully bring DevOps to the delivery process, the business stakeholder must understand and successfully communicate the business needs and priorities to the dev teams and express it as a change. They will, in turn, suggest a realistic delivery date for the change. That’s the ideal scenario.

However, the reality can be a little different.

Once the requirements have been captured, the dev teams translate them into bite‑sized stories. These stories quickly disappear among others into the backlog. So while the work is being done, it is not visible as such to the business stakeholder. The problem becomes one of a communication shortfall – a gap in understanding between the development and operations people delivering the work, and the business stakeholder who initially requested it.

RENATO QUEDAS VICE PRESIDENT SOLUTIONS MARKETING AND ENABLEMENT MICRO FOCUS

Renato has over 20 years of services, sales and development experience with application development lifecycle products and their related technologies and processes, helping organisations developing best practices and approaches to successfully adopt modern development and testing practices and tools. He is a prolific speaker, addressing a broad range of topics around the globe, and a published author with a wide range of technical publications and presentations to his credit.

T E S T M a g a z i n e | N o v e m b e r 2 01 5

14

The problem is finding a tool that tracks every sprint and offers full backlog visibility to overcome the communication gap, and crystallises the distinction between what the stakeholder wants and what the dev and ops teams are preparing to deliver.

D E V O P S

If the business does not understand how the work is progressing – and when it will be delivered – then a disconnect occurs.

Understanding becomes lost in translation, with the developers working on ‘executing stories in the backlog’ and the business stakeholder needing to track progress towards their original requirement.

The process becomes even more convoluted when the almost inevitable change in business priorities shifts schedules around still further.

The challenges are many and varied.

There are many hurdles for the business stakeholder and the development and operations teams including defining requirements – and reporting progress – in a mutually‑understood way. Multiple teams managing multiple stories add to delivery complexity, as does tracking every sprint to ensure a full ‘paper trail’ of completed work. And the ‘communication breakdown’ extends beyond the stakeholder and teams implementing the work. How can he or she interpret the progress into meaningful news that the rest of the business can understand and act upon?

SEEING THE BIGGER PICTURE The key is to provide the big picture. To give the business stakeholder a clearer understanding of how the work will progress in the future – and better understand the potential impact of any changes – by providing a full retrospective view. Because it is only by understanding why something has happened that the business can make sound decisions and establish strategies based on a realistic delivery date. The problem is finding a tool that tracks every sprint and offers full backlog visibility to overcome the communication gap, and crystallises the distinction between what the stakeholder wants and what the dev and ops teams are preparing to deliver. In short, to discover a technology offering the insight the stakeholder wants to create the foresight the business needs.

There are no tools dedicated to DevOps per se. However, Borland offers a piece of engineering that brings business transparency to DevOps. It enables business stakeholders to achieve the 360˚ perspective on where their business needs are in the development lifecycle and get closer to establishing the business agility improvements that DevOps promises.

Borland Atlas from Micro Focus brings business agility to the development and operations function. If DevOps is all about establishing agile practices in the development lifecycle, then Atlas provides the universal window on the process, enabling full visibility – and complete understanding for everyone.

SUMMARY Software success starts with effective requirements management. It is the cornerstone of building a product that meets all the criteria the business wants to meet, a need that the market demands.

Atlas is the enabler that ‘fine tunes’ the communication and decision‑making process between business and development teams, offering a full history of the work completed and transparency over the current and future work to be delivered.

So if you see DevOps working for your development teams, then consider the support you might need, because only with a full retrospective view can organisations successfully plot the way forward.

T E S T M a g a z i n e | N o v e m b e r 2 01 5

A NEW ERA IN TEST DESIGN

Sharon Elgarat, Test Solution Architect, Amdocs Testing, discusses the evolution of automated test design.

T E S T M a g a z i n e | N o v e m b e r 2 01 5

T E S T

17

A U T O M A T I O N

A

s test design can take up to 50% of a testing project, the ability to reduce the test design process can reduce costs and contribute to faster time to market. Test design ensures that the full scope of a project is covered and highlights key problem areas, all of which have a significant effect on testing quality.

As testing methodologies evolve, the importance of test design grows, driving it to the forefront of the project, ahead even of code development. Effective testing begins with test design that is clear, easy to understand, and which covers the full project scope – all of which are essential for software project success.

THE EVOLUTION OF TEST DESIGN Traditional test design methods describe test design as a list of individual test scenarios. As a result, users lack a high level understanding of the project’s scope, making it increasingly difficult to properly determine the exact list of scenarios required to cover the full scope of a project. Figure 1 depicts the complexity of this process, which not only demands an extensive amount of resources and time to maintain and understand but is also impossible to reuse or provide additional value. As a result, resources are wasted on the current project and are denied from any future endeavours that could benefit from best practices.

Devising a solution to this complexity resulted from first-hand experience where project managers continuously dealt with

projects that, with each new version, were becoming increasingly complex. It became clear that in order for projects to succeed, advanced tools not yet available were needed.

The complexity shown in Figure 1 portrays not only the monstrosity of some projects but also the need for reuse. Think of it like designing a closet – while standard designs are available, there are also tools such as brackets and fixtures with customisations for the specific needs and dimensions of the closet. Software testing vendors that are industry-specific benefit from their industry expertise, and may use a knowledge library as a base for generating graphic flows. By doing so, they reuse both technical and business knowledge accumulated by the testing team. Again, going back to the closet – the main elements needed are known and it’s a question of how they are put together to customise for the customer’s unique needs.

MOVING AHEAD TO A NEW ERA OF DESIGN

The ability to include an automated, algorithm-based breakdown of the diagram into scenarios enables test designers to produce high quality design with the confidence that the customer’s needs will be met with complete satisfaction.

What started off as a solution for a major transformation project for a large European service provider has been successfully implemented in four projects tested by Amdocs over the last year where automated test design reduced design efforts by 65% in comparison to other projects similar in size. The ability to include an automated, algorithm-based breakdown of the diagram into scenarios enables test designers to produce high quality design with the confidence

SHARON ELGARAT TEST SOLUTION ARCHITECT AMDOCS TESTING

Sharon Elgarat has a B.Sc. in Computer Science and Mathematics from Bar Ilan University. He has over 10 years of experience as a testing architect and submitted over 20 patents during Figure 1. Example of a traceability matrix used in test design to illustrate complexity.

2014 - 2015.

T E S T M a g a z i n e | N o v e m b e r 2 01 5

18

It is at the heart of the professional testing service to be able to ensure that testing quality is at the top – reassuring the customer’s business continuity with complete testing coverage and defect identification.

T E S T

that the customer’s needs will be met with complete satisfaction. In addition, analytical capabilities enable the system to fine-tune recommended flows with each test conducted by any team utilising the system.

Drag and drop test design tools, as shown in Figure 2, introduce the use of graphic elements to describe the testing scope at a higher level.

Test designers gain a new stage in the design process to present the target scope the testing scenarios will cover. When the testing scope is described as a diagram, the customer business and IT managers gain a clear, high level view of the testing flows without having to drill down to individual tests. This enables the customer to (a) review the testing artefacts at a very early stage of the design; (b) verify the testing team is considering the entire scope of the project as part of the testing scope; (c) provide input from a business perspective to help prioritise activities according to their specific needs; and (d) work and design according to business processes which can be adjusted according to industry needs.

The diagram is further broken down into linear flows with algorithms, which help the user select proper scenarios. Algorithms can range from being very generic, based on pairwise testing to algorithms led by industry business rules, and may also offer artificial intelligence solutions based on similar test results. By properly representing the use cases encountered after the system goes live, users will not meet scenarios not covered by tests.

The resulting recommended set of scenarios may take testers, development teams and

A U T O M A T I O N

customer experts back to the drawing board while the design team is likely to make changes in the diagram and re-run the automated recommendation; they may also manually revise the automated recommended list of scenarios.

Proper test design also includes functional testing, performance, load and security testing which don’t get lost along the way.

REASSURING BUSINESS CONTINUITY This test design system can be utilised release after release. Entities are saved from all levels – from the individual generic test used to a full diagram – ensuring full coverage of business processes. In the next release, should changes arrive in an already tested business process, the tester can reuse the entities generated, gaining both high level business flows and individual test boxes with a full technical description. The focus therefore shifts toward required updates according to the changes introduced in the new release – saving both time and resources.

The use of multiple technologies and business processes that may not be fully understood up front make the creation of good test design challenging. It is at the heart of the professional testing service to be able to ensure that testing quality is at the top – reassuring the customer’s business continuity with complete testing coverage and defect identification.

Automated test design not only improves design outputs, but also reduces the test design effort.

Figure 2. Example of a test design graphic display with a drag and drop propriety tool.

T E S T M a g a z i n e | N o v e m b e r 2 01 5

Pete in Q&A thought his testing had been thorough... ...shame he forgot to test in a realistic network!

Virtual test networks are the smart way Virtual test networks are the smart way to manage the risks of deploying critical to manage the risks of deploying critical applications into potentially hostile applications into potentially hostile network environments. network environments.

etworks are the smart way

Mimic every perceivable network Mimic every perceivable network topology and scenario. Tweak and topology and scenario. Tweak and repeat. repeat.

Mimic every perceivable net

he risks iTrinegy of deploying critical topology and scenario.

Twea is the leading provider of networked application risk management solutions, iTrinegy is the leading provider of networked application risk management solutions, trusted by governments, military organisations and enterprises around the world. trusted by governments, military organisations and enterprises around the world.

into potentially hostile

ronments.

repeat.

Contact us today to arrange an online demonstration or trial Contact us today to arrange an online demonstration or trial ( +44) 01799 2 52 2 00 ( +44) 01799 252 200

WWW.IT RINEGY.CO M WWW. I T RI N E G Y. C O M

INFO @ITRINEGY.CO M I N FO @I TRI N E G Y. C O M

e leading provider of networked application risk managemen

HOW ONE OF THE BIG SIX

ENSURES BUSI

AS USUA T E S T M a g a z i n e | N o v e m b e r 2 01 5

21

INESS

AL

Scott Arnold and Prabhu Raman share a British Gas case study in partnership with Cognizant Technology Solutions, on successfully implementing the SAP Solution Manager Toolset at Britain’s largest energy and home services provider.

T E S T M a g a z i n e | N o v e m b e r 2 01 5

22

C O V E R

BPCA is an application which helps in executing change impact analysis and allows the customer to carry out risk based test planning and execution.

I

n the past few years, the IT industry has embarked on a fast paced transformational journey to deliver accelerated business value. It is more important than ever for business and IT teams to align and develop synergies for a faster time to market.

Serving around 12 million homes in the UK, British Gas is the biggest UK energy supplier and is considered one of the ‘Big Six’ in the domestic gas and electricity market.

In the British Gas Information Systems department, the Business As Usual (BAU) test team is responsible for carrying out regression testing against all core SAP releases.

Historically, any fix or change proposed to the SAP application required a manual assessment of impact to existing processes and customer journeys.

This manual process was time consuming and as with any manual process, it could be open to human error. This could result in testing too much or not testing enough, which could result in production incidents.

The SAP Solution Manager product contains a tool called Business Process Change Analyser (BPCA), which the test team identified could potentially be used to address this challenge. Since its introduction, it has resulted in a number of benefits for the business.

year round. Regression testing is essential for all of these releases and it is critical that this is conducted in the most efficient and reliable manner possible.

Manual scoping of changes going live can potentially cause the following issues: • Difficulty in identifying critical business process affected by change events.

• Difficulty in arriving at test recommendations for those change events.

• Missing transparency about the business process/scenarios to be included in the BAU regression pack.

• Missing transparency on the business processes that are going through the change across releases.

• Justifying the priority of regression test coverage. If timelines were shortened for any reason, it could lead to incident leakage to production.

• Withdrawal of some changes of a release due to conflicts, meaning a time consuming and repeated manual test scoping process.

• Test scope optimisation was risky due to the fact that the manual process was highly dependent on variables such as correct documentation or availability of key resources.

THE TECHNOLOGY

THE SOLUTION

BPCA is an application which helps in executing change impact analysis and allows the customer to carry out risk based test planning and execution. It is part of the end to end integration testing standard for SAP solutions. The different phases of BPCA include preparation, change impact analysis and risk based testing scope.

British Gas’ Testing Services department worked with Cognizant Technology Solutions to trial usage of the BPCA tool. Scott Arnold (Testing Service Manager for BAU) led this initiative working with Prabhu Raman from Cognizant Testing Services. The main purpose of the tool is to analyse the objects included in any number of SAP transports and automatically compare these to the objects contained in the Technical Bill Of Materials (TBOMs) of the target system.

Since every TBOM is clearly assigned to a certain transaction in a certain scenario, business process or process step, it is then possible to determine precisely which parts of a business process hierarchy are affected by the change. As test cases are mapped to

MAJOR DRIVERS FOR BPCA IMPLEMENTATION IN BRITISH GAS ED REID HEAD OF TESTING SERVICES BRITISH GAS

S T O R Y

British Gas makes a high number of production SAP releases of varying sizes all

Ed joined British Gas 16 years ago, originally working in Customer Services and moved into IT and testing in 2003.

Preparation

He worked his way up through the ranks and has been Head of Testing Services for the last 2 years.

T E S T M a g a z i n e | N o v e m b e r 2 01 5

BPCA phases

Change Impact Analysis

Risk-Based Testing Scope

C O V E R

23

S T O R Y

SCOTT ARNOLD BAU TESTING SERVICES MANAGER BRITISH GAS

The British Gas BAU team now delivers innovation through better project planning, and delivers scoping faster through a better assessment of the testing effort using the ‘Effort Analyser’ feature of BPCA business processes this provides a rapid and reliable method of test scoping. The trial was successful and is now used comprehensively.

IMPACT OF BPCA The British Gas BAU team now delivers innovation through better project planning, and delivers scoping faster through a better assessment of the testing effort using the ‘Effort Analyser’ feature of BPCA. Other advantages include streamlined business process hierarchy documentation across customer journeys.

BPCA also gave the team the ability to provide technical evidence to the regression test scoping activity for every round of regression execution. It also enables smart ranking of business processes/process steps, with highest impact using the test scope optimisation feature of BPCA. British Gas has reduced its test effort across test phases by prioritising testing of most impacted business processes.

For example, on SAP upgrades, such as enhancement pack or service pack upgrades, it is almost impossible to manually go through each and every release note to determine the impact. Whereas BPCA provides an automated way of providing the business process impact.

COST SAVINGS SCOPING AND EXECUTION OF BAU REGRESSION

Scope identification efficiency has been increased after implementing BPCA. Below are the main types of releases to production for SAP‑centric changes in British Gas for which regression testing is run: • Operational release (OPR): Frequency is three to four times monthly. They contain small changes and incident fixes, but typically approximately 40 changes. The BAU regression team normally executes one round of regression testing after

Scott has worked for British Gas IS for 17 years in a variety of business & IS roles, including 11 years in testing. His current role includes focusing strongly on transforming the automation testing strategy across both the core SAP and strategic landscape. Scott studied Business Information Systems at the University of Teesside.

PRABHU RAMAN TEST MANAGER COGNIZANT TECHNOLOGY SOLUTION

Prabhu is an IT Professional with 14 years’ experience in software quality assurance and testing. He has experience of a variety of roles and environments spanning the entire software lifecycle and his domain experience includes UK & EU Information media & entertainment, oil & gas, CRM, supply-chain, case management, network and educational arenas covering both back-office systems and front-end user applications.

T E S T M a g a z i n e | N o v e m b e r 2 01 5

24

BPCA has enabled testing effort to be more accurately focused on the processes at risk.

This has resulted in improved defect identification, which in turn has prevented incidents occurring in production that may otherwise have been missed in testing.

C O V E R

Release

Manual scoping Scope count

BPCA

Defects

Scope count

Defects

25th Mar OPR

68

3

25

3

7 Apr OPR

45

0

70

0

th

PB15.Apr.01

249

4

108

6

23rd Apr OPR

100

0

60

1

29th Apr OPR

85

1

45

1

21 May OPR

125

1

44

2

28th May OPR

129

0

32

1

PB15.May.01

367

1

227

2

st

08 June OPR

128

0

20

0

29th June OPR

78

2

78

1

PB15.June.01

387

4

52

3

06 July OPR

64

0

94

2

PB15.July.01

362

2

177

2

th

th

the final merge build going live, as part of the release.

• Planned build release (PB): Monthly planned builds with projects, small changes and incidents that are going live as part of the release with a larger capacity than an operational release.

The BAU regression team executes two rounds of regression testing against the merge build going live as part of the release.

• Rapid release (RR): Ad‑hoc frequency and an ability to quickly promote urgent fixes to production without going through major assessment from technical teams. The BAU team executes one ‘mini’ round of regression testing against the against the merge build going live as part of the release.

REDUCED DEFECT/INCIDENT LEAKAGE INTO PRODUCTION

As a result of this work, defect detection efficiency has improved because of the extra scope identified using BPCA. The company can now focus on the specific test cases required based on technical impact analysis.

This helps in the identification of defects early, by running regression against a more targeted and accurate set of business process based test scripts when compared to manual regression scoping. The table above shows the actual defects found so far in BAU regression using both manual scoping and also added BPCA scoping.

Significant cost savings are now made as more defects are being found via BAU regression testing because of the more accurate and efficient scoping.

T E S T M a g a z i n e | N o v e m b e r 2 01 5

S T O R Y

SUMMARY Once the British Gas BAU testing team embraced SAP Solution Manager Toolset and BPCA a number of advantages followed very quickly.

“Through best use of technology in SAP Solution Manager we have transformed our regression capability resulting in increased quality and reduced cost. Scott and Prabhu have done a great job in making this happen.” said Ed Reid, Head of Testing Services at British Gas. By using BPCA, the main benefits include:

COST OPTIMISATION:

The BAU regression team has seen an approximate 25% reduction in the cost of comprehensive SAP regression testing.

REDUCED TIMELINE:

There has been a 30% reduction in test scoping and execution time.

INCREASED QUALITY

BPCA has enabled testing effort to be more accurately focused on the processes at risk. This has resulted in improved defect identification, which in turn has prevented incidents occurring in production that may otherwise have been missed in testing.

By reducing the manual effort and optimising the scoping process it has benefited the team by enabling staff to work on developing the automation pack further. This is resulting in increased automation coverage for the British Gas SAP changes.

7th INTERNATIONAL TESTISTANBUL CONFERENCE Test Data Management April 26th, 2016 Istanbul Turkey

The largest software testing conference in South East Europe and Middle East, hosting nearly a thousand local and foreign testing professionals

MEET INSPIRING KEYNOTES

SUPER EARLY BIRD FEE

EARLY BIRD FEE

REGULAR FEE

180 £ + VAT

210 £ + VAT

250 £ + VAT

Until November 23rd

November 23rd - January 18th

After January 18th

Sponsored by

Organized by

REGISTER

www.testistanbul.org

26

O U T S O U R C I N G

DOES SIZE MATTER? Mark Bargh and Richard Simms at ROQ IT, discuss drivers for IT outsourcing and the growth of the ‘supersized’ service providers.

T

he traditional reasons cited to outsource IT functions include a desire to focus on a company’s core competencies, reducing costs, streamlining finances, strategic IT thought leadership, standardised processes, scalable infrastructure and access to a wider IT talent pool.

However, as business dependence on IT increases, IT has grown to be one of the largest cost centres for many organisations.

Combine this with the fact that IT employees are generally considered an overhead and it is clear that cost reduction and streamlining

T E S T M a g a z i n e | N o v e m b e r 2 01 5

finances are the primary drivers for IT outsourcing.

With cost as the primary driver, it is then no surprise, that the outsourcing industry has chased the lowest cost resources in the global market. As a result, the past 15 years have seen the explosive growth of offshore IT services and the emergence of ‘supersized’ IT service providers.

This article is not intended to consider the merits of outsourcing (the evidence suggests that the model has proven successful for many companies) nor document the inevitable high

profile examples of failures that occur. Here we are concerned with the importance of selecting the right scale of outsourcing partner, particularly in the specialised and increasingly important area of software testing.

WHY IS TEST OUTSOURCING SPECIAL? Testing is predominately a downstream project activity, very dependent on other teams which require collaborative working, often under

27

O U T S O U R C I N G

severe time constraints (if earlier project stages have over run), and consequently slippage in the testing phase generally impacts go‑live dates. The acceptance testing phases in particular require close partnership between the test, business, technical and infrastructure team to be successful.

The value of independence in testing and having a fresh pair of eyes to challenge assumptions facilitates the detection of defects that other teams have overlooked: this sets test outsourcing apart from other IT functions.

Test outsourcing is clearly not just about technology and process, it is as much about the people and building relationships. With offshore teams facing the additional challenges of distance, working hour alignment, language and cultural diversity: communication is absolutely key for successful testing delivery.

CHALLENGES POSED BY THE ‘SUPERSIZED’ DEALS TO SUCCESS IN TESTING To deliver the negotiated cost savings in large‑scale outsourcing deals a high percentage of the work has to be completed offshore. This is not always appropriate to certain test types or stages. Indeed, this may pose a challenge to activities such as acceptance testing or performance testing, where close collaboration with other teams is essential.

These major outsourcing deals may pose a particular challenge to smaller projects within the IT portfolio. It is not uncommon for teams to work under the constraint of having to deliver at least 80% of all work offshore.

As an example, a small project within a large outsourcing contract required only two test resources: stipulating that one resource ‘had’ to be offshore was inappropriate for the successful delivery of this project.

Organisations may believe that by outsourcing to a large service provider they have alleviated risk and gained security in size. However, this may not always be the case. The contract needs to be considered against the multi‑billion pound portfolio of the outsourcing provider. This has implications for the availability and quality of the resources devoted to the contract and the attention it receives from the service provider.

These ‘supersized’ providers may be just too big to be truly flexible. They are typically constrained by internal bureaucracy, processes and procedures: testing often requires a quick response to provide quality onsite testing resources, which is difficult for large providers to achieve due to contract constraints.

Similarly, large outsourcing providers have internal review processes which may prevent them from providing timely estimates to clients. When the inevitable ‘change requests’ arises, the ability to provide impact analysis, including revised costs and timescales, may be hampered by these internal procedures.

The implications for quality and the true overall cost may be impacted if the emphasis of the initial contract is focused on the offshore ‘rate card’.

Effectively, the cost of provision of onsite resources is often prohibitively high to encourage the use of offshore resources.

The offshore test teams may also experience difficulties when resources are required onsite for relatively short periods of time. This requirement may be for activities such as face‑to‑face workshops, knowledge sharing and on‑boarding. The length of time taken to secure visas and the cost of travel are often barriers to effectively supporting these activities.

THE ADVANTAGE OF A MANAGED TEST SERVICE DELIVERED FROM AN OFFSITE, UK LOCATION Recent trends have seen more IT service functions return back to the UK from offshore locations. This is especially the case for areas of business that require collaborative working or close resources.

Building and sustaining working relationships is a key factor for successful test project delivery. This creates an environment where a project can evolve through constructive challenge and collaboration.

A UK‑based test lab offers the advantage of offsite working, yet provides an effective viable cost alternative to offshore working.

They can provide solutions that are innovative, scalable and flexible to meet the changing demands of clients. A UK based location allows face‑to‑face working where appropriate, which builds strong inter and intra team relationships. Working with a medium‑sized service provider provides the foundations for a mutually beneficial relationship where the provider is small enough to care, but large enough to deliver.

When considering outsourcing strategies or looking for the provision of managed testing services, organisations should review the size and distance. UK based test labs provide a realistic alternative with many advantages over ‘supersized’ offshore providers.

MARK BARGH FOUNDER AND DIRECTOR ROQ IT

Mark has over 20 years IT experience covering the full project lifecycle.

Prior to founding ROQ IT six year ago he was Vice President of European Operations for AppLabs. Within ROQ IT Mark is responsible for the operational and delivery aspects of the business, including the development of methods, processes and services.

RICHARD SIMMS TEST ARCHITECT ROQ IT

Richard has spent over 25 years in IT, the last 12 of which were spent in test management roles. Including five years as Quality Manager overseeing testing and change management at Morrisons.

He joined ROQ IT in 2012 and has developed test strategies and delivered projects for many clients across a whole range of industries and wide variety of solutions.

T E S T M a g a z i n e | N o v e m b e r 2 01 5

SECURITY AND COMPLIANCE – FRIEND OR FOE? Kevin Foster, Testing Services Manager, MTI Technology, discusses how to get more out of security and compliance.

T E S T M a g a z i n e | N o v e m b e r 2 01 5

S E C U R I T Y

29

T E S T I N G

In some instances, an over‑emphasis on a single compliance standard can focus security budget on a small number of areas, leaving other important areas devoid of much‑needed resources.

T

oday, CIOs and IT decision‑makers are facing more and more security challenges on a daily basis, and often in the public eye. From recent high‑profile enterprise hacks to risks associated with wearables, mobiles and IoT, there are constantly issues that need immediate addressing. At the same time, many IT budgets are stagnant or shrinking, leaving IT departments scrambling to squeeze resources, cut costs and streamline processes.

This tightening of resources has forced IT decision‑makers to take a long, hard look at their security and compliance priorities. For many businesses, this has resulted in a ‘mere compliance’ security approach being taken.

Security and compliance activities have become simple box‑ticking exercises with limited perceived value and strategic implementation.

Much of this can be attributed to a lack of understanding around why security and compliance activities are needed, and how they apply to different organisations.

MOVING BEYOND THE BARE MINIMUM Organisations often perceive security and compliance activities such as penetration testing to be a bare minimum requirement, and nothing more. They will merely comply with regulations and side‑step where possible. However, from a consumer perspective, it can be appealing to know that you’re dealing with an organisation that values your personal data protection.

Security is a particularly important consideration for consumers who use e‑commerce sites or have their personal

records disclosed online. The security aspect will be even more critical for consumers who have had their personal information stolen, leaked or defrauded in the past. Just ask the millions of Ashley Madison customers who had their names, sexual preferences, credit card numbers, email and physical addresses dumped online! Tragically, a pastor who was outed via that leak committed suicide, and there is speculation of other suicides linked to the case.

Regardless of the view taken, the reality is that security and compliance measures deliver value and prevent loss. Compliance activities including penetration testing, ethical hacking projects and security assessments are all tried and proven methods for identifying security risks, preventing hacks and reducing business damage.

KEVIN FOSTER, TESTING SERVICES MANAGER MTI TECHNOLOGY

Kevin Foster is responsible for developing MTI’s penetration‑testing business. Since joining MTI in 2004, he has developed testing specifications to address emerging technologies and compliance standards. His role sees him conduct and capture initial requirements, review designs and documentation, as well as liaising with client teams and MTI’s penetration testing consultants, to produce tailored test specifications.

T E S T M a g a z i n e | N o v e m b e r 2 01 5

30

Instead of cutting costs across your entire security and compliance operation, consider larger cuts to small number of areas.

This will allow you to prioritise areas that require the most attention.

THE IMPLICATIONS OF COMPLIANCE STANDARDS There are several compliance standards that exist to protect consumers where organisations fail. For example, the Payment Card Industry Data Security Standard (PCI DSS) stipulates the controls required by businesses to safeguard customer credit card details.

For the IT and information security industry, these standards present a number of benefits and drawbacks. On one hand, compliance standards force organisations to invest in security solutions, consultancy advice and services. On the other hand, they often see organisations implement security measures just for ‘compliance’s sake’.

A mere compliance approach often results in underwhelming outcomes.

Organisations can miss the intent and benefits of compliance standards entirely. In these cases, compliance can easily become the enemy, as money is wasted and budget owners fail to see return on investment.

Take public sector websites and applications, for example. These platforms host and process an abundance of personal data about the public. In some instances, an over‑emphasis on a single compliance standard can focus security budget on a small number of areas, leaving other important areas devoid of much‑needed resources.

One commonly overlooked area is web application security testing. These tests explore potential OWASP‑related

T E S T M a g a z i n e | N o v e m b e r 2 01 5

S E C U R I T Y

vulnerabilities, internet network defences and business logic flaws. If exploited, these areas can lead to fraudulent use of and unauthorised access to public data.

GETTING THE MOST OUT OF AN IT BUDGET With security budgets as tight as ever, how can organisations ensure that they’re prioritising security and compliance activities effectively? Where should money be going and how can investment be maximised?

Some organisations manage to apply a strategically sound and cost‑effect approach. Others struggle to get the balance right.

For organisations that have to implement security and compliance activities, especially penetration testing, here are some tips to consider.

LINK COMPLIANCE TO BUSINESS GOALS

In order to get the support needed, highlight how specific compliance activities link to overall business strategy. To this end, it helps to move beyond the language of mere ‘compliance’ towards speaking about compliance activities in ‘goals’, ‘KPIs’ and ‘growth’ terms.

REDUCE AND OUTSOURCE

Can you change any organisational processes to reduce the areas that need compliance? By reducing your organisational footprint, you’ll be able to do a better, quicker and cheaper job across a smaller number of areas. It’s also worth looking into options for outsourcing business process and compliance risk to third parties.

COST‑CUT STRATEGICALLY

Instead of cutting costs across your entire security and compliance operation, consider larger cuts to small number of areas. This will allow you to prioritise areas that require the most attention.

DOUBLE UP, WHERE POSSIBLE

Savvy IT strategy can see organisations save time and money by combining IT and security compliance work streams. In many cases, a business can implement security measures

T E S T I N G

to address multiple, overlapping compliance standards at the same time. For example, the PCI DSS and Data Protection Act require organisations to implement many of the same controls when dealing with credit cards and sensitive authentication data.

SERVICE LEVEL AGREEMENTS CAN OUTSOURCE THE RESPONSIBILITY

Always aim to include security SLAs when procuring third‑party services and new IT solutions. Doing such will help to ensure that new systems remain secure for the contractual period. It will also mean that compliance obligations stay with the third‑party for the length of the contract, reducing your potential costs and liability.

BE REALISTIC

If you lack the money and resources needed to cover the‑scope of work, it’s important to let the budget‑holder know.

Weigh up the potential costs of non‑compliance or security breaches, and consider whether the organisation’s investment is sufficient.

DON’T INVEST UNNECESSARILY

If the intent behind particular compliance standards doesn’t apply to your type of organisation, it’s worth raising evidence of this to relevant auditors and regulatory authorities. You may be able to avoid unnecessary work and costs.

SEEK ADVICE

If you’re unsure about the security compliance standards or challenges you face, it’s worth consulting an independent industry professional. They can provide an objective and informed perspective on your security and compliance activities to help uncover shortfalls and potential risks.

BE SAFE, NOT SORRY

Even if you don’t operate in a heavily regulated industry, it’s still worth considering investment in a comprehensive security and compliance regime. Taking into account the regularity of public data security breaches nowadays, and the potentially huge costs that such breaches can incur, it may be worth overinvesting a little bit rather than falling.

UP, UP AND AWAY! Earlier this summer, CA Technologies announced that it had acquired Grid-Tools Ltd, a company specialising in test data management, automated test design and test optimisation solutions. The acquisition is poised to assist CA Technologies in expanding its DevOps portfolio with an enhanced focus on quality acceleration.

T E S T M a g a z i n e | N o v e m b e r 2 01 5

S U P P L I E R

33

P R O F I L E

T

EST Magazine’s Editor Cecilia Rehn recently got together with Huw Price, the former MD of Grid‑Tools and now VP at CA Technologies, to discuss continuous integration, what it means to offer test data excellence, and what the future holds following the acquisition.

Tell us about Grid‑Tools, and its history?

Grid‑Tools was a classic British software company, founded 11 years ago. It grew very organically, and we chose to specialise in test case design and synthetic data generation capabilities early on. We felt as though test data was one of the final frontiers left in IT. Companies had traditionally approached it as a logistical problem – how can we manage all of this data? – but to us it was all about test driven development.

Grid‑Tools really helped to redefine this market, and once we had a good understanding of test data, we progressed into test coverage. In order to move away from simply collating, moving and masking data, we focused on clarifying test case requirements to remove ambiguity and save time. Put simply, we aimed to find and send over the right test data to developers and testers, as quickly and efficiently as possible.

During Grid‑Tools’ history, we developed our two signature products: Datamaker, the ‘synthetic’ test data generator, and Agile Designer, the test case design tool.

Since the acquisition, these two tools have now been rebranded CA Test Data Manager (formerly Datamaker) and CA Test Case Optimizer (formerly Agile Designer).

And how did your business relationship with CA Technologies start?

We had been working alongside CA Technologies for two years when they came to us with their CA Service Virtualization product. Test data is a crucial component of service virtualisation, and they needed us to help find the test data.

So we began working together regularly, sort of joined at the hip! Then the acquisition was almost a formal extension of this successful partnership. Grid‑Tools had around 30 employees at the time, most of which have stayed on, eager to see how the products will grow and develop with the support of CA Technologies.

There has been a big push towards continuous delivery and agile methodologies; what are the challenges

here and how can they be solved?

The key challenge, I believe, is solving the issue of language hops. Every time you have to explain something, there’s a 50% reduction in quality and doubling of time.

What we’re seeing is that, typically, the communication chain goes something like this: the client speaks to the developer, who has to interpret the initial brief. Then a tester enters the scene, and has to further interpret and create a test case. You might also have a test data engineer, some test automation people and a performance tester – who all will invariably reach different interpretations. Organisations are essentially creating inefficient production lines, where valuable information is getting lost or miscommunicated as it hops from end to end.

The goal, and indeed the overall push in the industry, is towards continuous delivery and agile methodology.

The goal, and indeed the overall push in the industry, is towards continuous delivery and agile methodology. If you can set it up so that all are working truly in parallel, then the tester will only ever be five minutes behind the code being delivered, and the automation framework 10 minutes behind that. This is the ideal, dream scenario.

We’ve focused on creating one asset, which can be reused across the spectrum.

CA Test Case Optimizer allows for a more collaborative workflow. Out the back you get the user story, test cases and coverage techniques. It all flows at the same time, ensuring you’re covering all the decisions of the test cases.

By reducing the language hops, introducing clearer communication and making sure all teams work together, companies will be able to drive continuous integration forward.

Case in point, I was speaking with a customer recently, who had a very well orchestrated test automation framework set up. Despite this, it was apparent that as soon as a group of analysts changed any requirements, there was a sense of urgency in responding. The new test framework needed to be up and running in a few hours. We’re calling the ability to do this ‘reactive automation’.

Grid‑Tools was well established as an innovator in the testing and test data management space, how is CA Technologies’ portfolio now laid out to support these areas?

Following the acquisition, Grid-Tools’ products and engineering talent has been incorporated into CA Technologies, and our product

HUW PRICE VICE PRESIDENT, TEST DATA MANAGEMENT, AND BUSINESS UNIT EXECUTIVE, APPLICATION DELIVERY, CA TECHNOLOGIES

Huw joined CA Technologies in 2015 as Vice President of Test Data Management, when specialist testing vendor Grid-Tools was acquired into the CA DevOps portfolio. During his 30 year career, Huw has gained a deep understanding of the challenges faced by modern organisations, and, with an understanding of the science of testing, how to solve them.

T E S T M a g a z i n e | N o v e m b e r 2 01 5

34

Following the acquisition, Grid-Tools’ products and engineering talent has been incorporated into CA Technologies, and our product portfolio has a strong focus on requirements definition and test case design.

S U P P L I E R

Figure 1. CA Test Data Manager Solution Architecture.

portfolio has a strong focus on requirements definition and test case design.

CA Test Case Optimizer is a full requirements definition tool and headlines our portfolio.

It can be used for many different tasks, including creating requirements documents, but it’s primarily used to create test cases.

Through the use of flow charts and models, CA Test Case Optimizer is able to calculate the minimum amount of test cases and decisions needed for coverage.

Take for example the London Underground.

Should you wish to test the system based on the question “can I get from every station to every other station?” you do not want to physically criss‑cross the network, making redundant journeys. Instead, CA Test Case Optimizer would highlight the minimum amount of journeys to ensure that all the junctions have been tested.

We’re in the business of helping to save time, by reducing the amount of test cases down to the smallest number possible without sacrificing coverage.

With CA Test Case Optimizer, you can also create automation frameworks in Selenium, Gherkin, Protractor script, or anything you like. The tool has been designed to reduce the amount of manual testing, save time, and ensure a structured approach to test data requirements and test data creation.

T E S T M a g a z i n e | N o v e m b e r 2 01 5

P R O F I L E

And once the test data has been created, how can it be managed?

Our second flagship tool is CA Test Data Manager. It is essentially split into a few different components. Firstly, it delves into the logistics of test data – moving test data out of production.

Secondly, the tool manages test data automatically. In most projects you’ll see 200 developers and 100 testers all fighting for test data. We generate a Test Mart, which acts as a real time index of the test data, searching through it and allocating it to various team members. This way the test data – which is linked to the test criteria, ensuring it’s easy to understand – is shared more efficiently, and ultimately the test mart makes life a lot easier for the testers by bringing the test data directly to them when they need it.

Finally, CA Test Data Manager is revolutionising the way we produce synthetic test data. Testers, of course, are already generating some level of synthetic data during testing, but we’ve brought a more structured approach to this. Additionally, when using CA Test Data Manager, any data that’s previously been created becomes available, reducing time spent on duplicates.

We recently worked on a big project for a theme park, where we were able to use CA Test Data Manager to build test data

S U P P L I E R

35

P R O F I L E

Figure 2. If testing the London Underground system, CA Test Case Optimizer would calculate the minimum amount of journeys to ensure all junctions are fully tested. CA Test Case Optimizer saves time by reducing test cases without sacrificing coverage.

objects, to speed up their testing operation.

You can easily build anything from a visitor to the theme park, to a hotel, or the billing system. The customer could then assemble the test data objects as and when needed.

Ultimately, CA Test Data Manager provides this structured approach to save time and reduce errors. We’ve found that a lot of bugs that testers find are not bugs at all, but are more a problem with the set up of the test data. So instead of treating test case data as an ad hoc component of the SDLC, we aim to simplify and automate it as much as possible, so that testers can focus on other parts.

How do you work with your clients?

Along with providing the tools, we are trying to help promote more collaboration and automation in the industry. We want to be evangelical. I don’t want to say that testing is broken, but we need visionaries on the scene to help push us all forward.

I consider the role of vendors, like CA Technologies, to be part of the larger

conversation. We encourage our clients to actively participate in the industry, and listen to and read about opinions from thought leaders such as Paul Gerrard and Dorothy Graham.

Moving towards more agile methodologies and continuous integration is helping testing and QA departments be more structured, but it’s difficult when traditionally the model has been quite siloed.

Our role is to come in and try to break down these silos, and add value to a project as quickly as possible.

3 ‑ 4 months, and we’ve seen a lot of immediate support from CA Technologies, including increased sales support. With this new support, our main tasks are to continue improving on performance, as well as focusing on integration efforts. For example we’re integrating CA Test Data Manager and CA Test Case Optimizer with other products in the CA Technologies family. The way we see it, the more our products can talk to each other and work together, the more efficiently we’ll be able to help our customers.

Taking the Test Mart for example, which has proven a popular solution to allocating test data more efficiently. For such an installation, we would typically be involved for 3 ‑ 6 weeks, with the emphasis on beginning work and delivering value after only a few days, or even less!

We’re also working on integrating the tools with other outside products, and we’re launching the Microsoft versions soon.

What excites you about being part of CA Technologies, and what are the plans for the next 12 months going forward?

It’s been a whirlwind over the last

The Grid‑Tools products will soon be represented on the CA website as part of the DevOps story. For more information, please visit www.ca.com.

Finally, we have a backlog of 20 ‑ 30 ideas that we’ll be bringing out in the future, so stay tuned!

T E S T M a g a z i n e | N o v e m b e r 2 01 5

THINKING OUT OF THE BOX The way in which testers are working in agile teams means they can be constrained in their approach by the narrow confines of user stories. Steve Watson, Test Manager, Reed Business Information, highlights a few shortcomings found during tester interviews, and the serious implications these could have.

T E S T M a g a z i n e | N o v e m b e r 2 01 5

C A R E E R

37

C O R N E R

A

re testers losing the ability to think outside the box? Are we encouraging test analysts to only focus on what is documented in a user story?

These are two questions that I find myself asking after speaking to testers who have experience working only within an agile background.

A question I like to ask during candidate interviews is how they would do an exploratory test of a web page (e.g. BBC News) if they had no user story, no specification or any prior knowledge of the application. I ask candidates to tell me the types of tests that they would focus on so I can see whether they have a general idea how to approach testing.

What I am looking for is a considered list covering some of the following (basic) tests: • UI – text, fonts etc.

• Page layout, scrolling.

• Links, to other pages and other sites.

• Button functions.

• Field input tests (positive and negative).

• Images.

• Videos.

• Field tab order.

• Search function.

• Page load times.

• Cross browser tests.

• Usability tests.

A lot of candidates reply that it depends on the user story. I can understand that, as they are used to working in an agile way, but I stop them and ask them to do some blue‑sky thinking. Imagine there are no requirements (a look of shock did appear on one candidates face!) and you are just placed in front of a website. What general tests would you do?

To my surprise, around a third of the testers struggle with this concept – the security blanket of a story has been removed and they do not know what to do.

Another third do have a try but miss really key aspects, or invent tests that are not appropriate. It’s a News page – I would assume most people read the news somewhere, and yet answers such as ‘load testing text boxes’ are thrown in, to display a knowledge that load testing is important. Well yes, it is, but in the right context!

SEEING THE BIGGER PICTURE The challenge for the testing community is that if testers are only able to work from the confines of a user story, where is the value

of bringing in dedicated test resources into teams?

My own background was through the route of manual testing back‑end applications where we followed a waterfall process, before following the pack into web application testing using an agile process, so I have seen and experienced both approaches.

Whilst waterfall has its shortcomings, there were some benefits as a tester.

Working from large requirement documents meant that we were testing something of a reasonable size and involved having to plan out our tests to cover all the scenarios. This pretty much covered a complete piece of work from end to end, and as a tester I could see the bigger picture, which helped to plan all the tests needed, identify gaps in test coverage and plug them.

Then came agile. There are many good things about agile, not least the breaking down of barriers, and the focus on team work and sharing out tasks, but the drawback that I see is the very narrow focus just on the individual user stories within each sprint, as this has restricted the view that the team has of what is to be delivered overall.

It seems to come easier to testers who have made the ‘waterfall to agile’ transition to see where the stories fit within the overall application, as that is how they used to work.

Of course there are good agile testers who can see the bigger picture, but I am finding that they are in the minority. A greater percentage of testers who have only been through the agile route have not necessarily needed to understand the big picture, as the work is all broken down into convenient sized ‘chunks’ that can be completed within a short sprint interval of 2 or 3 weeks. And there lies the problem.

CONTEXT CAN BE CRITICAL You could argue that the testers (and developers etc.) do not need to know this information. On a car production line, if I were in charge of fitting the left hand side front and rear doors to each vehicle, I would not be concerned about where the bonnet, tailgate, seats and engine came from, as these are not relevant to my task. But I would care if the vehicle chassis was the wrong shape and only had space for one door and not two, so the doors I had would not fit. Sometimes the context around the stories is important.

There is also the possibility that different testers may cover the stories independently, and they may be in different sprints:

The challenge for the testing community is that if testers are only able to work from the confines of a user story, where is the value of bringing in dedicated test resources into teams?

STEVE WATSON TEST MANAGER ICIS

Steve Watson is an ISTQB certified tester with over 25 years of testing experience in banking, futures & options trading, vehicle leasing, automotive information and B2B publishing.

His current role is Test Manager for ICIS, a leading brand owned by Reed Business Information, where he manages a team of testers working across a number of products, and runs a ‘QA Chapter’ bringing testers from different global brands together to promote best practice and encourage knowledge sharing.

T E S T M a g a z i n e | N o v e m b e r 2 01 5

38

Where testers add value is by identifying good tests – identifying the scenarios that haven’t been covered already as well as testing for those that have been, and for that you need to have a testing mindset.

C A R E E R

• The story for the 2‑doors will pass on its own – tester A sees that they have two doors and they look ok (handles and glass exist and they are the right colour etc).

• The chassis story will also pass on its own – tester B sees that they have a chassis.

However, the value is in looking forward and seeing that when you put the doors and chassis together that it isn’t going to work!

It’s easier in sprint planning to look at one story in connection with others in the same sprint, but what about considering the impact on previously delivered stories?

EXPLORATORY TESTING I hear testers talk about exploratory testing and think they know what it is, but I don’t believe they do. Exploratory testing is the art of not constraining yourself to a story as it is written, nor to the acceptance criteria, but to think more broadly. What other tests could you do that someone else didn’t think of for you? Does the story fit with the application?

Has it covered non‑functional considerations?

Here is an example: Imagine that a user story covers a login function to a bank website. It is likely that the story will have a title ‘As a user, I want to log onto XYZ bank so that I can see a summary of my bank accounts’.

There will be a list of fields required: Username; Password; Second level password; and buttons for ‘Login’ and ‘Cancel’; a ‘Remember Me’ option; and also a ‘Forgotten password’ option; followed by a list of business rules around the error messages to be displayed if you enter incorrect credentials.

There will be a screen mock‑up to display the expected layout – colours, fonts styles, font and button sizes, corporate logo etc.

The business analyst has added the acceptance criteria and the story is ready for adding to the sprint backlog.

T E S T M a g a z i n e | N o v e m b e r 2 01 5

C O R N E R

This is a golden opportunity for the tester to look at the story and work out what is not covered.

• Does it state how long it should take for the page to actually load?

• Does it cover what browsers are in scope?

• Does it state what should happen if the user clicks the ‘Back’ button on the browser?

• For the a ‘Remember Me’ option, how is this implemented? There is a right and wrong way to do this from a security perspective.

• Will the page be https? User and password details should not be sent over plain http.

These are just some of the questions that I would expect a good tester to ask.

BRINGING VALUE TO THE TEAM Testers are uniquely placed in any development team to think of scenarios that no‑one else has even considered. Every tester should be looking to add value every day to their team by thinking more broadly than just what is written in the user story.

It isn’t just about taking the scenarios that a Business Analyst has written and then automating them. Automated tests add value, but they can be written by anyone with a development background. Where testers add value is by identifying good tests – identifying the scenarios that haven’t been covered already as well as testing for those that have been, and for that you need to have a testing mindset.

I am very fortunate to have recruited a very able team of testers who get fully involved in the agile process, adding in missing tests, asking questions, considering non‑functional tests, taking on additional roles (scrum‑master, builds etc.). The testers are highly regarded and I’m really pleased to see that they are constantly adding value.

And this is something that all testers should be aiming for. I want to challenge each tester who reads this to think outside the box. Write down for a week every test you have thought of that no‑one else did, and share them, to show where you as a tester are adding value to your team.

I leave you with this quote: “Quality is never an accident; it is always the result of intelligent effort.” – John Ruskin.

A G I L E

39

I M P L E M E N T A T I O N

FAIL BETTER Phil Comelio, Head of Development at Capita Travel and Events, argues failing better is not only possible, but imperative.

All of old. Nothing else ever. Ever tried. Ever failed. No matter. Try again. Fail again. Fail better.

Worstward Ho, 1983, Samuel Beckett

H

urdles; How many times have you heard people talk about hurdles, the clichéd ‘we mustn’t put hurdles in the way’, a management line that consistently fails to address the institutionalised hurdles that we face in delivering software systems every day? We strive to be agile, we strive to meet the maxim of ‘fail fast’, but we can only do that by learning to fail better, to eliminate those hurdles that cause us to fail badly.

Hurdles; It’s not hard to understand how Usain Bolt can run the 100 m so much faster than a 100 m hurdler, but imagine if the hurdler had to go back the start every time

they knocked a hurdle over, yet this is what we, as an industry, enforce far more often than not, and, as a result, we fail badly when we could fail better.

THE PHILOSOPHY OF AGILE Failing fast is a laudable intention, by failing fast we reduce rework resulting in reduced costs, we reduce the risks to delivery and we get our products earlier to market, but in order to fail fast successfully we must first

learn to fail better. The problem generally comes about in organisations that are moving from a waterfall to an agile environment.

Corporately agile tends to be viewed as an IT methodology, whereas it has a far broader intent. I tend to view agile much more as a philosophy than a methodology; it’s certainly not something that can be adopted by blindly applying a set of techniques from a textbook. Typically organisations cherry pick the techniques they have decided they like rather than commit to becoming a fully agile culture, but agile is so much more than the sum of its parts. These organisations are

T E S T M a g a z i n e | N o v e m b e r 2 01 5

40

A G I L E

Agile gives us a powerful concept to get the individuals interacting as a collaborative team, and that’s the user story. The trick here is to create a story team for each story to be delivered in the sprint, and that team is responsible for the story achieving your definition of ‘done’ within that sprint.

PHIL COMELIO HEAD OF DEVELOPMENT CAPITA TRAVEL AND EVENTS

Phil Comelio is Head of Software Development at Capita Travel and Events. Phil has been managing the full lifecycle of complex software developments internationally since the mid‑90s specialising in the application of rapid application development and agile delivery techniques.

afraid of failure, the waterfall model that they have worked with for years has evolved in an attempt to avoid failure, putting many checks and balances in place to pre‑empt failure, yet they still have a reputaton for failing in an expensive and spectacular fashion.

One of the key principles of agile is to embrace change, but you can take this further, once you come to terms with that fact that failure is going to happen then you embrace failure, accepting it as an intrinsic part of the software development process. If you accept that failures are going to happen then you can optimise the way you deliver in order to fail in an effective, efficient and constructive way.

WE’RE ALL IN IT TOGETHER The key impediment to this is the structure of hurdles that exist in the nature of siloed organisations, with the hand off of work and responsibility from customer to analyst to developer, to tester, back to developer (repeat as necessary), to the tester, to the customer, to the analyst... you can see where this is going. All of this extends the time and cost of delivery because we can only truly know the real requirements by the time we reach the backend of this process. Siloed organisations which decide to adopt agile continue to apply this silo mentality and we see the agile model being twisted back into the comfort zone of waterfall, albeit still with daily stand‑ups, which are little more than progress reports, and complex task boards that try to represent the numerous hand offs rather than focus on the delivery of value to the business.

This is a key indicator that the organisation hasn’t fully bought into the agile philosophy and understood that it brings with it a cultural change throughout the whole organisation. Every employee, at every level, needs to understand that when it comes to being agile, we really are all in it together. The key thing that changes here is the nature of the delivery team. No longer is IT responsible for the delivery of the project, it becomes a silo‑free composite of IT and the business. The IT team must no longer see the business as a customer from whom all must be hidden, and the business must come to realise that they are complicit in the sofware development, that they are equally

I M P L E M E N T A T I O N

responsible for its successful delivery.

This cultural change results in empowered and self‑organised cross‑functional teams, all members working together in the best interest of the whole business (rather than in what may be perceived as the best interests of their own silos). Now we can really start delivering, but how when we don’t have that waterfall discipline in which to operate? At this point remember the ‘agile manifesto’, that we value individuals and interactions over processes and tools.

Agile gives us a powerful concept to get the individuals interacting as a collaborative team, and that’s the user story. The trick here is to create a story team for each story to be delivered in the sprint, and that team is responsible for the story achieving your definition of ‘done’ within that sprint. In order to do that the team must learn to fail fast, identifying and resolving issues (be they requirement clarifications or defects) as early as possible, and no later than that. To do this the team must include a blend of analysis, development and test skills, along with empowered customer representation. By close collaboration within the story team, the details of the story will emerge, enabling test driven development within a paired delivery model where development and test take place concurrently. The objective here is to prove that the story works before it is delivered for final business acceptance testing.

By failing fast through collaborative requirements elaboration and the continuous testing of work in progress the team will have proven that the story works, reducing formal acceptance testing to a regression exercise which simply proves that the story still works in a production candidate build.

LOOKING AHEAD Once we get better at failing faster, then we begin to see the benefits of failing better; enabling us to deliver value to our customers whilst reducing our rework costs and fostering a working environment which, by embracing failure, empowers our teams to succeed.

S U P P L E M E N T

HEADLINE SPONSOR

SPONSORS

P R E-AWA R D S E X EC U T I V E D E BAT E S P O N S O RS

42

T H E

E U R O P E A N

S O F T W A R E

After months of preparation, of receiving, reading, sorting and judging excellent entries for 22 categories, The European Software Testing Awards 2015 saw the industry come together for a glittering Awards gala celebration in central London.

This year saw a record 20% more entries than previous years and with attendees from across the globe the event was an astounding success.

In addition to celebrating the finalists and winners, the Awards offered an opportunity to raise money and awareness for registered charity Great Ormond Street Hospital (GOSH).

T E S T M a g a z i n e | N o v e m b e r 2 01 5

T E S T I N G

A W A R D S

T H E

E U R O P E A N

S O F T W A R E

T E S T I N G

A W A R D S

43

For more photos visit: www.softwaretestingawards.com

T E S T M a g a z i n e | N o v e m b e r 2 01 5

44

T H E

E U R O P E A N

S O F T W A R E

T E S T I N G

A W A R D S

FINALISTS Finalists for The European Software Testing Awards 2015 include end users, vendors, individuals who have made a difference to their testing departments as well as ambitious graduates with promising careers ahead of them.

RANOREX BEST TEST AUTOMATION PROJECT

The award for the best use of automation in a software testing project.

Highly commended: Lloyds Banking Group plc.

SQS BEST AGILE PROJECT

Category Winner: William Hill Online Finalists: ★ Tyto Software Pvt. Ltd.

(On behalf of ArisGlobal Software Pvt Ltd) ★ AvivaSA in partnership with Keytorc ★ Sage UK ★ Bluefruit Software (Absolute Software Ltd) ★ LogiGear Corporation ★ Box UK ★ SQS & BNP Paribas Fortis ★ LogiGear Corporation ★ Laya Healthcare Ltd ★ Infomentum ★ Virtusa ★ Lloyds Banking Group plc. ★ Accenture ★ Cognizant Technology Solutions ★ William Hill Online

BEST USE OF TOOLS This award is for the best use of software testing tools in a project.

Category Winner: Lloyds Bank in partnership with Cognizant Technology Solutions

Awarded for the best use of an agile approach in a software testing project.

Category Winner: Aviva and Tata Consultancy Services (TCS) Finalists: ★ Credit Suisse ★ Brickendon Consulting ★ Lloyds Banking Group ★ Mastek Ltd ★ InfoStretch Corporation ★ Aviva and Tata Consultancy Services

(TCS)

TESTING MANAGER OF THE YEAR Awarded to the most outstanding individual test manager or team leader over the last 12 months.

Category Winner: Bhuvaneswari Gangadharan, HARMAN Connected Services Finalists: ★ Piyal Mukherjee,

T E S T M a g a z i n e | N o v e m b e r 2 01 5

Cognizant Technology Solutions ★ Justin Hossain, Sogeti UK ★ Beverley Wells, Virgin Media ★ Bhuvaneswari Gangadharan,

Finalists: ★ Credit Suisse ★ Kainos ★ Infrasoft Technologies Ltd

★ Lloyds Bank in partnership with Cognizant Technology Solutions ★ William Hill in association with Test Plant ★ Ladbrokes in partnership with HARMAN Connected Services

HARMAN Connected Services ★ Santhosh Reddy Gujja, Amdocs

TESTING TEAM OF THE YEAR Awarded to the most outstanding overall testing team of the year.

Category Winner: Tech Mahindra Finalists: ★ Sage UK ★ Virgin Media with Accenture ★ Camelot Global ★ Credit Suisse ★ Virtusa Corporation ★ Tech Mahindra

TESTING MANAGEMENT TEAM OF THE YEAR Awarded to the test management team that has shown consistently outstanding leadership.

Category Winner: Lloyds Banking Group plc.

Finalists: ★ Cognizant Technology Solutions ★ Ciklum ★ Virgin Media with Accenture ★ Lloyds Banking Group plc. ★ Credit Suisse ★ Direct Line

SQS TESTING INNOVATOR OF THE YEAR The individual who has pushed the boundaries, in terms of innovation and employing new methods and tools in the testing field.

Category Winner: Debapriya Jena, Tata Consultancy Services (TCS)

FINALISTS AT THE 2015 TESTA AWARDS FINALIST

FINALIST

Test Manager of the Year

Best Overall Testing Project

Justin Hossain

Communication

FINALIST

Test Champion of the Year Andrew Fullen

Sogeti - a leading provider of Software Testing services: Winner of Testing Innovator of the Year Award, TESTA 2013 Recognized in the Magic Quadrant for Application Testing Services - Gartner

“High-touch, intimate testing services.”

“We have partnered with Sogeti since 2006.

During this time they have successfully enabled us to consistently deliver better quality applications, faster, without impacting staffing costs.”

“ Key to the Group’s success is: “its testing and process

David Joice, IT Director, XL Catlin

expertise and levels of customer intimacy.”

WANT TO FIND OUT MORE? Call: +44 (0) 20 7014 8900

Email: enquiries.uk@sogeti.com

www.uk.sogeti.com

46

T H E

E U R O P E A N

Finalists: ★ Auto Snipper tool, Mastek Limited ★ Sharon Elgarat, Amdocs ★ Oleksandr Maidaniuk, Ciklum ★ useMango™ tool, Infuse ★ Sibi Luke, Allianz Insurance PLC ★ Debapriya Jena, Tata Consultancy Services (TCS)

GRADUATE TESTER OF THE YEAR A recent graduate who has shown outstanding commitment and development in the testing field.

Category Winner: Jordan Kerr, William Hill Online

★ Elevate Credit International Limited

Awarded to the most outstanding testing project in the retail sector.

Category Winner: John Lewis Finalists: ★ Cognizant Technology Solutions

Finalists:

Awarded to the vendor who receives top marks for their product/service and customer service.

Highly commended: Tata Consultancy Services (TCS) Category Winner: Cognizant Technology Solutions Finalists: ★ TIBCO Jaspersoft Product Group ★ Ciklum ★ Cognizant Technology Solutions ★ Tech Mahindra ★ HARMAN Connected Services ★ Perfecto Mobile ★ QAMentor, Inc. ★ Tata Consultancy Services (TCS)

BEST OVERALL TESTING PROJECT – FINANCE SECTOR Awarded to the most outstanding testing project in the finance sector.

Category Winner: Cognizant Technology Solutions Finalists: ★ Barclays Investment Bank ★ Tech Mahindra

T E S T M a g a z i n e | N o v e m b e r 2 01 5

and SQS Group ★ Cognizant Technology Solutions ★ Lloyds Banking Group plc. ★ Aviva and Tata Consultancy Services (TCS)

BEST OVERALL TESTING PROJECT – RETAIL SECTOR

★ Priyanka Khurana, Testing Solutions Group ★ Onur Celep, ERICSSON ★ Jordan Kerr, William Hill Online

LEADING VENDOR

S O F T W A R E

in partnership with Arcadia ★ John Lewis ★ Tech Mahindra ★ Sainsbury’s and Tata Consultancy Services (TCS) ★ Adactus Ltd and Pizza Hut Delivery UK

T E S T I N G

A W A R D S

BEST OVERALL TESTING PROJECT – PUBLIC SECTOR Awarded to the most outstanding testing project in the public sector.

Category Winner: BJSS Finalists:

★ BJSS ★ Mastek Limited

THE BEST MOBILE PROJECT Awarded for the best use of technology and testing in a mobile application project.

Category Winner: Avis Budget Group and Tata Consultancy Services (TCS) Finalists: ★ Opera Software ASA in partnership

BEST OVERALL TESTING PROJECT – GAMING SECTOR Awarded to the most outstanding testing project in the gaming sector.

Category Winner: Amsphere Finalists: ★ Inspired Gaming in partnership

with HARMAN Connected Services ★ Amsphere ★ SEGA Europe

AMDOCS BEST OVERALL TESTING PROJECT – COMMUNICATION SECTOR Awarded to the most outstanding testing project in the communication sector.

Category Winner: EE Finalists: ★ EE ★ Amdocs Netherlands B.V

and Vodafone Netherlands ★ Sogeti UK ★ Metaswitch Networks

with Cigniti Technologies ★ Accenture Services Pvt Ltd ★ SEGA Europe ★ Lloyds Banking Group ★ Intelligent Environments Europe Ltd ★ Avis Budget Group and Tata

Consultancy Services (TCS)

BEST OVERALL USE OF TECHNOLOGY Awarded for outstanding application of technology in a testing project.

Category Winner: Cognizant Technology Solutions Finalists: ★ Centrica in partnership

with Cognizant ★ Sandhata Technologies ★ Kainos ★ TIBCO Jaspersoft Product Group ★ Lloyds Banking Group plc. ★ Cognizant Technology Solutions

THE BEST OVERALL PROJECT The project, that when compared to all others, ranked higher for innovation, approach, method, best practice, collaboration, and outcome.

T H E

E U R O P E A N

S O F T W A R E

T E S T I N G

A W A R D S

47

Category Winner: Ladbrokes in partnership with HARMAN Connected Services

Finalists: ★ Sage UK ★ Virgin Media ★ Lloyds Banking Group plc. ★ Tech Mahindra ★ Ladbrokes in partnership with HARMAN Connected Services ★ DHL Global Forwarding and Tata Consultancy Services (TCS)

TEST CHAMPION OF THE YEAR Awarded to the individual who has championed the cause of software testing above all others.

Category Winner: Subhadeep Chakraborty, Tech Mahindra Finalists: ★ Andrew Fullen, Sogeti UK ★ Ruslan Desyatnikov, QAMentor, Inc. ★ Subhadeep Chakraborty, Tech Mahindra

SOGETI GREEN TESTING TEAM OF THE YEAR Awarded to the testing team that has shown the biggest commitment to environmental and corporate social initiatives.

Category Winner: Cognizant Technology Solutions Finalists: ★ Cognizant Technology Solutions ★ Tech Mahindra

SAGE MOST INNOVATIVE PROJECT Awarded for the project that has significantly advanced the methods and practices of software testing.

Category Winner: Accenture Services Pvt Ltd Finalists: ★ AstraZeneca plc in partnership

with Cigniti Technologies ★ BT Plc. ★ Tech Mahindra Ltd. ★ Accenture Services Pvt Ltd ★ Lloyds Banking Group plc. ★ iTrinegy

FROM THE EDITOR

T

he build-up to the third annual European Software Testing Awards has taken months. From our entrants selecting projects to showcase and polishing entry forms; to our judges taking up their positions, ready to read, debate and judge the entries; and much work and set-up at the venue on the day itself.

As the evening’s host, I had the pleasure of welcoming members from the judging panel, our sponsors and of course, all our winners on stage! The hall was filled with infectious joy. It truly was a privilege to celebrate the talented teams and individuals in the software testing and QA community.

I can only hope the friendly, exciting and celebratory spirit will continue to help push us all to strive for betterment in ourselves, our teams and in the industry as a whole.

We welcomed attendees and finalists from all over the United Kingdom, the Middle East, Europe, the United States and Asia, highlighting the global influence of the Awards programme and the growing awareness of our community’s importance.

With 22 Awards handed out, the evening centred on commending projects that demonstrated best use of tools, innovative approaches, strong team leadership skills, as well as agile thinking.

The competition was fierce and all the shortlisted finalists were deserving of this honour.

I want to extend another thank you to our sponsors, without whom it would have been impossible to put together such a large, celebratory programme.

Micro Focus Borland have been unwavering in their support of the awards since inception, recognising that our community is one of the corner stones of today’s technological age.

Thank you to our category sponsors Amdocs, Ranorex, Sage, Sogeti, and SQS. As well as Cognizant, Cigniti and ROQ IT who sponsored the Pre Awards Executive Debates.

Thank you to all who attended this year, and I hope to see you again in 2016!

If you’re interested in being involved with the Awards next year, don’t hesitate to reach out to the team.

Cecilia Rehn

info@softwaretestingawards.com

T E S T M a g a z i n e | N o v e m b e r 2 01 5

48

T H E

E U R O P E A N

S O F T W A R E

T E S T I N G

A W A R D S

CHARITY 31 Media is really proud to have worked with some great charities to make a positive difference over the years.

“Supporting and donating to good causes, through our events, publications, and platforms is extremely important for us, as it creates increased awareness and exposure for the charitable efforts of many organisations,” says Grant Farrell, Managing Director at 31 Media.

At The European Software Testing Awards 2015, a special prize draw was held at the end of the evening with all proceeds

benefitting GOSH. The lucky winner picked up a brand new iPad Air 2 128GB, and a total of £1500 was raised.

GOSH is one of the top five paediatric research hospitals in the world, treating children from all over the UK and abroad who are diagnosed with the most complex, life‑threatening conditions. GOSH has over 50 specialities under one roof and receives over 240,000 patient visits every year from children all over the UK and abroad, who are suffering from the most complex, life‑threatening conditions.

JUDGING PANEL ROD ARMSTRONG

Programme Quality Manager EasyJet

RIEL CAROL

Head of Test YouView TV Limited

MARK CONWAY

Director of Development Borland

PAULA COPE

Global Head of QA Tullett Prebon

KIERAN CORNWALL

Former Head of Testing ITV

ANDY FLYNN

GPSIT Testing John Lewis Partnership

MARK GALVIN

Systems Assurance Manager University of Cambridge

SHANE KELLY

Head of QA and Test William Hill

MYRON KIRK

Head of Test CoE Boots

SIMON PICKARD

Head of QA Gazprom

KASHIF SALEEM

Director QA Hotels.com

MALCOLM SAUNDERS

Head of QA and Testing Services Close Brothers

HEADLINE SPONSOR

SPONSORS

P R E-AWA R D S E X EC U T I V E D E BAT E S P O N S O RS

T E S T M a g a z i n e | N o v e m b e r 2 01 5

INÈS SMITH

Head of QA Telegraph Media Group

OLIVER SMITH

QA Director kgb – 118 118

GWEN STEWART

QA Manager Amazon

MIKE VERNER

Director IM & TC Liberty Global

MARTIN WRIGLEY

Executive Director AQuA

CECILIA REHN

Chair of Judging Panel TEST Magazine

Thank you TEST Magazine for supporting Great Ormond Street Hospital Children’s Charity Great Ormond Street Hospital is one of the top five paediatric research hospitals in the world. We treat children from all over the UK and abroad who are diagnosed with the most complex, life-threatening

conditions. But it’s only thanks to people like you that we can provide our patients with the specialist care they need.

Your donations will make a huge difference.

Great Ormond Street Hospital Children’s Charity. Registered charity no. 1160024.

For more information an d to make a don ation, please visit

www.gosh.org

SUCCESSFUL SOFTWARE DELIVERY

BUILD ON YOUR MANUAL TESTING QUALITY WITH AUTOMATION Manual testing is great, so far as it goes. The problem is, it can’t scale to keep pace with development and rapid Agile release cycles. So what if you extend the reach of human testing through automation?

With a hybrid environment from Borland, you integrate requirements, manual and automated testing. So when manual testing struggles to keep up, you get a helping hand to increase testing speed while maintaining quality – across all of your different devices, platforms and OS versions. It’s automation at its best, with Borland’s human touch.

‘The human side of test automation’– download your FREE White Paper at www.borland.com/testautomation

CONNECT YOUR PEOPLE FOR SUCCESSFUL SOFTWARE DELIVERY Copyright© 2015 Micro Focus. All Rights Reserved.

Portions Copyright © 1994-2009 Borland Software Corporation (a Micro Focus company).