TEST – September 2017

SEPTEMBER 2017

20 LEADING TESTING PROVIDERS SPECIAL

CONNECTING

THE CITY

PROGRAMME

INSIDE

All-in-One

Test Automation

RECORDING_1

RECORD MODULES

ex.

RecordItemIndex(1));

equence 'admin'.", new RecordItemIndex(2));

Any Technology

BROWSER

OPEN

MOUSE

CLICK

KEY

SEQUENCE

VALIDATE

ATTRIBUTE EQUAL

USER CODE

MY_METHOD

Recording_1.cs RECORDING_1

void ITestModule.Run { Report.Log(ReportLev "Website", "Opening w

Report.Log(ReportLev "Mouse", "Mouse Left C

Report.Log(ReportLev

Seamless Integration Broad Acceptance Robust Automation

1

Quick ROI

Licen

All Te

se

chno logi All U pdat es. es.

egration t In m iu n le e S Now with • Access Selenium with the Ranorex automation framework • Address Selenium pain points • Web testing across all major platforms and browsers • For testers and developers T E S T M a g a z i n e | S e p t e m b e r 2 01 7

www.ranorex.com/try-now

n()

1

C O N T E N T S

T E S T C O V E R

M A G A Z I N E

S T O R Y:

P R O T E C T I N G

|

S E P T E M B E R

A N D

C O N N E C T I N G

2 0 1 7 T H E

M O N E Y

NEWS

Does AI Replace Or Assist? .............................. 5

32

THOUGHT LEADERSHIP

The Race Against Time ..................................... 8 Risk Management .......................................... 10 Appium And Mobile Test Automation .......... 12 Test Automation: The Right Way is Up .......... 14

vel.Info, web site 'http://www.ranorex.

SUPPLIER PROFILE

vel.Info, Click at {X=10,Y=20}.", new RecordItemIndex(1));

Digital Transformation ................................... 16

vel.Info, "Keyboard", "Key sequence 'admin'.", new RecordItemIndex(2));

TEST AUTOMATION

Machine Learning And Software Testing ...... 20 ERPS And Automation Test Data .................... 24 EXPLORATORY TESTING

Exploratory, Agile and Automated ................. 28 FINANCE

Protecting And Connecting The Money ........... 32

20

IT ACADEMY

Part of The Academy ..................................... 38 CROWDTESTING

Testing: The Key to Agile Success ................. 44

28

SOFTWARE TESTING CONFERENCE NORTH

Welcome ....................................................... 49 Programme.................................................... 50 Sponsors......................................................... 52.

Floor Plan ...................................................... 54 Speakers ........................................................ 55 20 LEADING TESTING PROVIDERS

12

Welcome ....................................................... 59

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

2

TESTING FOR £2 500 PER MONTH REALLY? Really.

You can have your cake AND eat it.

DVT’s Global Testing Centre offers heavyweight testing solutions, backed up by 17 years’ experience and high level expertise. The DVT GTC Lite® Package is designed for small to medium size enterprises, providing an affordable, quality outsourced testing solution.

An exceptionally sweet deal.

FREE two-week Proof of Concept on Non-Functional Testing

Functional Testing

FREE access to a range of REAL mobile devices

Automation Regression Testing

FREE Automation Tool

Device Testing

FREE Performance Testing Tool (limited to 50 users)

Performance Testing*

Activation in FIVE working days

Total Delivery** * Performance Testing limited to 20 hours per month ** Total Delivery limited to 160 hours per month

INTERESTED? Contact DVT to find out more.

Tel: 0203 696 2440 | Email gtclite@dvtsoftware.co.uk

Terms and conditions apply.

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

E D I T O R ' S

C O M M E N T

3

THE TALK OF THE TOWN CECILIA REHN CONTRIBUTING EDITOR

A

s you walk, drive or commute around a major metropolitan city, you can be amazed by several things at once.

The planning – or lack thereof – in certain areas.

Impressive cultural landmarks. The blend of modern and classic.

Successful cities do not spring up overnight.

They must evolve and change to suit different generations. This adjustment does not happen organically. Investment is planned and chosen areas revitalised. There are upgrades to be made and end users to think about. Flow of traffic, people, goods and services to consider.

I think you can see where I’m headed: you can draw a parallel from the world of urban planning to digital transformation.

Heavy traffic – and the bottleneck that it is – can hamper a city’s flow and progress.

Mammoth efforts are required to revamp ageing infrastructure to keep apace with new demands and population growth. Long-term, expensive urban projects, such as Boston’s Big Dig (US$15 billion – the US most expensive highway project finished in 2007) or the ongoing Crossrail project in the UK (£14.8_billion railway project due to be finished in December 2018), represent cohesive strategy and decisionmaking from a multitude of stakeholders.

All such projects, of course, tend to change over time. Budgets go over. Priorities shift.

Unforeseen problems arise. In Boston, sunken ships were found underneath the city by diggers and needed archeological attention.

Even though costs ballooned and it underwent several delays, the Big Dig is considered a success.1 The UK’s Crossrail is expected to bring the biggest increase to the capital’s transport system in a generation.

Businesses striving to become more agile by removing old, existing bottlenecks can take inspiration from these urban infrastructure projects. It takes real vision to put together a business case for extensive digital transformation, and perseverance and strength

SEPTEMBER 2017 | VOLUME 9 | ISSUE 4

of conviction to see it through all unexpected twists and turns. But the end result can be spectacular.

Investment in show-stopping landmarks can also be a good strategy to update and improve a city, as they can help encourage and propel further investment into broader revitalisation projects. As an example, Dubai’s Emirates Office Tower was one of the first skyscrapers that showcased the United Arab Emirates’ transition to a modern, global nation. Today, the city prides itself on having the world’s tallest building, several artificial islands and ultramodern developments. The appearance of success helps foster more success.

In terms of digital transformation, investment into high-profile projects can prove beneficial to a company’s brand, and increase confidence in digital transformation as a whole.

Visible, fresh digital offerings, such as a new app or solution can improve public perception of a brand and put pressure on rivals. This positive impact takes place even if legacy systems slow progress in other areas. The recent investment into new technology by UK banks, such as Barclays integrating with Apple’s Siri2 and TSB’s iris scanning solution3, fit into this trend. And each showcased digital product from the British finance sector has received much favourable press.

A digital transformation strategy that first focuses on small, tangible, and visible goals can kickstart more financial and organisational backing, and inspire stakeholders and end users alike.

A digital transformation project must be approached as a complex, multi-layered being.

Whether focusing on smaller, more symbolic programmes or tackling a behemoth of legacy systems, patience and thinking like an urban developer can be key. After all, Rome was not built in a day.

© 2017 31 Media Limited. All rights reserved.

TEST Magazine is edited, designed, and published by 31 Media Limited. No part of TEST Magazine may be reproduced, transmitted, stored electronically, distributed, or copied, in whole or part without the prior written consent of the publisher. A reprint service is available. Opinions expressed in this journal do not necessarily reflect those of the editor of TEST Magazine or its publisher, 31 Media Limited.

ISSN 2040‑01‑60 EDITORIAL DEPARTMENT editor@31media.co.uk +44 (0)203 056 4599 REPORTER Leah Alger leah.alger@31media.co.uk +44 (0)203 668 6948 ADVERTISING ENQUIRIES Shivanni Sohal shivanni.sohal@31media.co.uk +44 (0)203 668 6945 PRODUCTION & DESIGN Ivan Boyanov ivan.boyanov@31media.co.uk 31 Media Ltd, 41‑42 Daisy Business Park 19‑35 Sylvan Grove London, SE15 1PD +44 (0)870 863 6930 info@31media.co.uk www.testingmagazine.com PRINTED BY Pensord, Tram Road, Pontllanfraith, Blackwood, NP12 2YA

softwaretestingnews @testmagazine TEST Magazine Group

References 1.

‘10 years later, did the Big Dig deliver?’, The Boston Globe, https://www.bostonglobe.com/magazine/2015/12/29/years-later-did-big-digdeliver/tSb8PIMS4QJUETsMpA7SpI/story.html

2.

‘Barclays launches first voice-activated banking’, The Telegraph, http://www.telegraph.co.uk/money/consumer-affairs/hey-siri-pay-auntmaud-150-barclays-launches-first-voice-activated/

3.

‘TSB to roll out iris scanning tech’, BBC, http://www.bbc.com/news/technology-40663365

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

4

www.isqi.org

for information on training providers and examinations

STAND OUT FROM THE CROWD, GET CERTIFIED WITH iSQI! iSQI – Your global certification partner and authorised provider for ISTQB ® exams

ilovesoftwaretesting T E S T M a g a z i n e | S e p t e m b e r 2 01 7 ©g-stockstudio_shutterstock.com

5

N E W S

DOES AI REPLACE OR ASSIST?

Does artifical intelligence (AI) replace software testers, or assist? To find out what effect AI has on tech-savvy individuals, Test Magazine reporter, Leah Alger, receives vitalizing views from an array of software testers.

W

e have witnessed the mobile and computer revolution – now similarly – artificial intelligence (AI) is unrolling its potential; not only by the way we live, but also within the majority of industries, including software testing firms.

Technology is being implemented within development teams and tools to catch bugs earlier, automatically assessing and correcting code. According to Cognifide, we need to accept that, within the next two years, AI techniques will fully dominate IT, with Amazon, Google, Facebook, IBM and Microsoft being one of many signs.

“IBM and Google aren’t the only companies applying AI techniques. Within the past year, AI in software testing has also become feasible. Software testing must evolve in response to the shift to agile and DevOps. No matter how many testers you employ, it’s simply not possible for manual testing to provide agile developers immediate feedback on whether any of their

constant changes impacted the existing user experience,” wrote Tricentis in a blog post.

‘CATCHING BUGS EARLIER’ According to QA Financial, Test Plant believes there is “a lot of talk” when using artificial intelligence to test, yet no one is using it in a “useful way”, because of testers involving it straight into testing processes, instead of using it to plan beforehand.

Senior test manager at Allied Testing, Stas Milec, agreed: “Tests do not make software any better. Your product might be working well, but if it's not something a customer expects, then it's the wrong product. It's important to get people involved in the validation process throughout.

“My strong believe is that AI will start playing a significant role in testing – we

should see AI dynamically generating test cases, while building knowledge about the software developer ability, whilst learning how customers use a particular type of software.”

Analytics India revealed that AI has made its way through software testing through a lack of infrastructure, and the need for faster deployments, involving three main aspects – testing in real-world customer environments, user acceptance testing and manual testing.

“A lot of things are going to change in the testing field with the entry of AI. Almost 70% of testing is repetitive and AI can quickly occupy that space. The 30% left is questioning the system, and that's what testers need to focus on. AI is the next big thing in testing, but it won't replace humans. The testers working alongside AI can quickly revolutionise the way we test today,” added Vijay Shinde, founder of Software Testing Help.

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

6

‘HELPING EXPERTS TEST EFFECTIVELY’ Mark Roberts, head of test at Capita IT, added: “In the software testing industry, automation and AI provide support to the individual in the workplace, with automation allowing individuals to focus on more meaningful, higher value activities.

“However, companies such as Blue Prism have produced automation tools that go one step further and successfully automate full testing business transaction across multiple platforms, allowing for successful automation of back office business processes, which will effectively lead to a reduction in the administrative work force.”

A report by Transparency Market Research found many companies are operating in the fields of IT robotic automation tools and IT robotic automation services. It appears the market is rising in competition, and witnessing wider changes, especially those firms with development budgets, tight training and cost-effective solutions, such as: Blue Prism, Be Informed B.V, Appian, IPSoft, Tata Consultancy Services Limited, Infosys Limited, Sutherland Global Services and UiPath Srl. Robert added: “AI is relatively new; it is

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

N E W S

starting to aid defect diagnosis, and is the root cause analysis, which very much relies on existing test data. However, I feel that as we get to grips with this technology, it will become more powerful and likely start to replace much of the analysis that is done in testing.”

‘A ROBOT IS NOT COMING TO TAKE YOUR JOB’ CEO of TestPlant, George Mackintosh, said to the SD Times: “There’s a lot of talk currently about test automation. However, in reality, we’ve only automated one key element: test execution. AI and analytics will be the catalysts to deliver true test automation that recommends the tests to carry out, learns continuously, enabling it to predict business impacts enabling DevOps teams to fix issues before they occur. This will help teams to keep up with user expectations and the pace of DevOps something they are struggling with today.”

During test execution the test manager should monitor the progress as per the test plan, and if required, he/she needs to take control in terms of objective and strategy.

Mackintosh added: “Software testers have a very key role. It is absolutely important

to understand AI isn’t a robot coming to take your job, but is a smart assistant instead.

The software tester is still responsible for modelling the workflow that you have, and setting up the environment and tooling.”

“They are just the ones reviewing the results and looking at the interface to provide recommendations from the systems and the feedback to the development team and to the business.”

Johan Steyn, senior manager for enterprise software quality at Nedbank, agreed: “From the point of view of software testing and quality assurance, I see a definite trend and a push to bring automation into the software testing trade. Technologies around cognitive tools, artificial intelligence and machine learning enable testing teams to test smarter and faster.”

It appears that AI assists, and does not replace, and software can’t be released quick enough, playing a large role in innovation and technology. Although consumers need to assess how AI can help achieve larger outcomes, by involving AI straight into the testing process.

7

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

THE 8

RACE AGAINST TIME: GDPR AND ITS IMPACT ON TEST DATA MANAGEMENT

Santhosh CV, practice manager, Quality Engineering and Transformation (QET), Nordics , TCS, explores why GDPR is an important piece of EU law making which will have a major impact on the software testing world

I

n April 2016, the EU parliament approved the General Data Protection Regulation (GDPR), which will become effective from 25 May 2018. The regulation will impact all enterprises that control or process EU consumer data, irrespective of their geographical location.

WHY NOW? Data breaches have become increasingly frequent at large enterprises in recent years, adversely affecting their customers. Moreover, the promulgation comes at a time when the Digital Single Market, envisioned to make trading easier between EU countries by removing procedural bottlenecks, will result in significantly more data movement across the EU. This will necessitate stringent and unified data privacy measures for all EU nations to prevent and better manage future incidents of data breach.

TEST ENVIRONMENTS: THE WEAK LINK While the impact of GDPR on overall

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

organisation security policies have been welldocumented, an aspect that could be easily overlooked are test environments.

The data for testing is often directly sourced from production without proper anonymisation of sensitive fields and without appropriate traceability mechanisms in test environments.

With clauses like the ‘Right to be Forgotten’ in GDPR, it will be a nightmare for quality engineering (QE) and testing teams managing test environments to respond to such requests.

One approach, suggested by GDPR for organisations to get compliant, is to ensure irreversible anonymisation of sensitive data fields (data masking) in test environments.

Obtaining consent from consumers, to use their real data for testing purposes is not, for various reasons, practical.

However, before attempting to devise a solution, assessing the test environment for GDPR-readiness is key, which helps to baseline the as-is state of the enterprise.

THE JOURNEY TOWARDS COMPLIANCE An absolutely critical initial phase, assessment begins with identifying potential

T H O U G H T

9

L E A D E R S H I P

non-compliance areas. Enterprises need to review their current test data management (TDM) processes and security policies governing their test environments.

A key activity in assessment is data classification and discovery – identifying sensitive PII fields and running a complete data scan to create a heat map showing compliance/ non-compliance risk levels across environments.

Creating a heat map can be an onerous task even for organisations who have mature TDM processes.

This is because classification of sensitive data, as per GDPR, is complex.

Four areas need attention – staffing, test data sourcing, tooling and agile adoption:

RECRUITMENT OF RIGHTSKILLED PERSONNEL Enterprises need specialist QE teams with blended expertise – a sound understanding of GDPR, industry domain knowledge and technical prowess.

Understanding GDPR helps with data classification, while domain knowledge provides insights into application flows and corresponding test data requirements, to ensure ‘fit for the purpose’ solutions. And technical expertise on test data management is essential for performing data scans, creating masking rules, etc.

DATA MASKING VS. SYNTHETIC DATA CREATION GDPR compliance mandates data masking.

But that’s easier said than done. A key challenge is the classification of sensitive data.

How does one devise a robust data masking strategy that caters to heterogeneous databases and meets test data requirements for end-to-end (E2E) user flows?

An alternative to the masking route would be synthetic data creation. Indeed, performance testing and other forms of nonfunctional testing benefit considerably from it.

But synthetic data does not satisfy all test data requirements for functional testing, for example, the testing of end-to-end user flows needs data that mimics real production data.

So use instead a blend of masked and synthetic data.

STRATEGIC TOOL SELECTION There are many tools in the market for data classification, E2E data traceability, data masking, data management, synthetic data creation and so on. Which does one choose?

The right choice involves tools that are not only cost-effective but also the right fit.

For instance, if the solution requires using a blend of masked data and synthetic data creation, one needs to be selective because not all tools in the market provide for both.

However, a far-sighted choice would mean selecting tools that are also scalable in the long term for purposes beyond GDPR, for example, meeting enterprise test data management and data governance needs as well.

The data for testing is often directly sourced from production without proper anonymisation of sensitive fields and without appropriate traceability mechanisms in test environments

GETTING IT ‘RIGHT FIRST TIME’ GDPR compliance solution implementation allows no window for rework. Enterprises are thus under immense pressure to get it right the first time. A few key pitfalls that enterprises must be alert to are: • Agility. Enterprises need to steer clear of the waterfall mode and run the implementation in a completely Agile fashion, with multiple sprints to avoid last-minute surprises.

• Governance. With multiple stakeholders involved, governance must be transparent, accountable and efficient, and responsibilities, clearly segregated, to avoid ambiguity.

• QA. The importance of thoroughly testing the entire solution before go-live cannot be overemphasised. So invest care, diligence and effort in this last phase.

TIME TO ACT Time is short. Regardless of how mature the TDM processes of an enterprise are, at the very least, two quarters are required to secure test environments against GDPR non-compliance.

Enterprises just cannot leave test environments on the back burner because that could very well turn out to be the Achilles heel in the GDPR journey.

A meta-approach that future-proofs enterprises and makes them complianceready – which has been tested already with organisations in Europe – should prove worthwhile.

SANTHOSH CV PRACTICE MANAGER, QUALITY ENGINEERING AND TRANSFORMATION (QET), NORDICS TCS

Santhosh CV is Practice Manager, Quality Engineering and Transformation (QET), Nordics, TCS. His primary responsibilities include building QE-focused offerings and solutions.

Also the TCS champion for GDPR compliance & TDM, Santhosh helps enterprises across Europe in their journey towards GDPR-readiness.

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

10

RISK MANAGEMENT:

IS YOUR SOFTWARE PROTECTED?

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

Nadine du Toit, Head of Managed and Professional Services at DVT Global Testing Solutions, argues that software risk management is vital and that traditional insurance morphs into a different paradigm when applied to software

T H O U G H T

11

L E A D E R S H I P

I

n today’s technological world, where the assets of a business are protected by short-term insurance, there is a vital aspect of risk management that is often neglected: protecting our software. In the 21st century, businesses are increasingly innovating their technology in response to the massive wave of digital transformation that is taking place.

There is also more financial and skills investment into software than ever before.

Software is part and parcel of our business environment, but how are we protecting one of our most valuable assets?

The traditional concept of short-term insurance is difficult to apply to software.

As a consumer, I am able to take out a short-term insurance policy with an accredited insurance company through any number of channels. After a few formalities, I can get myself insured, probably with a single phone call. As with most financial services, I can then assess or review my monthly premiums based on my current and future needs and adjust them when needed.

Over time I can also compare different pricing options between insurance companies, and pick and choose the product that’s right for me. As a policy holder, my financial behaviour is typically tracked by my service provider as I start new applications, submit claims, amend existing policies or cancel and switch.

I can earn benefits and loyalty rewards based on these habits and be more exposed to other offerings from my insurance company and its affiliates.

It becomes clear that traditional insurance morphs into a different paradigm when it is applied to software. From my perspective, software insurance has a duality of meaning: 1. Quality assurance and software testing Your software is an asset. It may not be tangible, like a building, machinery or electronic equipment, but it still exists.

Stringent quality assurance and software testing provides a form of insurance against poor service delivery, below par customer experience and customer churn. However, QA and testing should not only take place just before release, but at various stages in the lifecycle of the software, both pre-release and post-release.

In addition, testing should take place across functionality, user experience, business processes and through various layers of security. Testing criteria should be outlined and outcomes developed accordingly. Testdriven development is also a key component of QA and testing. Ongoing development of

our software allows us to keep our customer offer fresh and relevant.

2. Risk Management For a business, software development can be fraught with risk. For example, when a business invests in quality assurance and testing, it bases its decisions on anticipated targets and assessments of customer behaviour. Then using these assessments, the organisation seeks to deliver rapid results via automation.

And with today’s digital transformation strategies, IoT and cloud-based solutions speeding up change and release cycles, our QA approach often delivers right the first time which is exactly how it should be – except when the goal post changes and renders our speedy solution redundant.

Moreover, testing a business’ critical software for security and data integrity – be it customer-facing apps, web services or back-end financial systems – can also be considered a component of risk management.

In other words, we are already continuously investing in insurance upfront by improving the quality of our software. In fact, according to the 2016/17 World Quality Report, 39% of a software developers’ budget is forecast to be allocated to QA by 2018.

Testing a business’ critical software for security and data integrity – be it customer-facing apps, web services or back-end financial systems – can also be considered a component of risk management

UNWANTED OUTCOMES The consequences of inadequate QA and testing can be dire, with massive financial losses. It could result in a security breach where information is leaked to a third party. It can lead to poor performance quality.

Buying concert tickets online becomes a headache because the software takes too long to populate the data, or takes too long to send the one-time PIN. Month-end transactions become an exercise in frustration because the software cannot handle the increased demand.

A DIFFERENT PERSPECTIVE When we start to view our software as an asset, we start to think differently about how to protect it, just as we would the other assets of our business. Then, QA, testing and ongoing development no longer feels like a grudge purchase, but an intelligent, sensible decision that protects our business and invests in our people and our customers.

NADINE DU TOIT HEAD OF MANAGED AND PROFESSIONAL SERVICES DVT GLOBAL TESTING SOLUTIONS

Nadine du Toit is Head of Managed and Professional services at DVT Global Testing Solutions, and has more than 14 years' experience in business development, strategic software engineering and quality assurance consulting in the financial and telecommunications industries.

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

12

APPIUM AND MOBILE TEST AUTOMATION: 3 THINGS TO CONSIDER WHEN GETTING STARTED

With all the industry buzz around Appium, what do enterprise mobility teams need to consider when getting started using this test automation framework? Dan McFall, President of Mobile Labs reveals the top three areas that mobility professionals must consider to be Appium ready

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

T

hese days it is hard to have a conversation about enterprise mobility and test automation without hearing the word Appium. What is it about this opensource test automation framework that has captured the attention of mobile testers and quality assurance professionals worldwide?

Based on my conversations with industry professionals and customers, it is the inherent freedom and flexibility of using an open-source framework that is exciting. By leveraging Appium for automated testing, enterprise mobility teams can break free from the restrictions imposed by commercial testing frameworks and tools.

But, with this freedom comes a unique set of challenges that mobile development,

testing and quality assurance teams must conquer to truly be successful using Appium.

Oddly enough these same conversations revealed another interesting finding regarding Appium. Although many enterprise mobility teams have Appium installed, they are challenged in leveraging it to its full potential.

Many of the professionals that I encountered, are struggling as to how to get started using Appium. There seemed to be some confusion as to how Appium would impact overall test automation strategy and mobile demand.

Therefore, I would like to offer my assistance to help enterprise mobility teams navigate the world of Appium. Here are three key areas that your team must address

T H O U G H T

13

L E A D E R S H I P

when incorporating Appium into your overall strategy around automated testing and continuous delivery for mobile. #1: Consider the benefits of open-source As an open-source testing framework, Appium gives developers and testers a certain amount of freedom to customise and leverage the framework to best suit the testing needs of the team.

But, because Appium is open-source, there is no official product support. Issues are logged in GitHub and are fixed by Appium developers or contributors. Luckily, the Appium community of contributors and developers is comprised of talented, passionate developers that are committed to Appium’s success. Therefore, this group is a great resource for your developers and testers to utilise when using Appium. In fact, some of your enterprise mobility team members may want to become involved in the community to further Appium’s mission. Consider the benefits of having an enterprise mobility team that is excited to learn new skills and to be a part of growing Appium’s capabilities.

If you are testing iOS apps or mobile websites with Safari, you are going to need an Apple computer. Your team will need to procure an Apple computer because Appium does not want to modify an app for testing purposes.

Therefore, the technology used to interact with iOS comes from Apple provided services, which are required by license to run on an Apple computer.

Android devices can run on any computer, if its services are not tied to an operating system.

Planning is always an important part of testing, but with Appium you will need to ensure that your team has the proper computer hardware for testing.

With the freedom of leveraging an open-source framework like Appium, true innovation can occur

MAKING APPIUM WORK FOR YOUR TEAM

#2: Consider Appium requirements for your mobile apps When incorporating any automated testing framework, you must know what exactly you are testing. What are the unique needs of each mobile app? Remember that apps may require specific provisioning that require you to register a device with your development program.

In the case of iOS, the applications must be built for development, meaning that the device must be registered to your Apple Developer programmer and signed with development profiles and certificates. If your team is currently testing an app signed with an Enterprise profile and certificate, you must make sure that your development team knows the new requirements for Appium.

Also, if your team is testing apps or mobile websites using real mobile devices or simulators, you may need different builds of your apps. For example, iOS requires an app that runs on a simulator to be compiled differently than an app that runs on an actual iPhone or iPad. #3: Consider which type of computer can run real devices When leveraging Appium, it is important to note that Appium can be used for both iOS and Android.

By taking the points outlined above into account, your enterprise mobility team can leverage Appium to great advantage. Its opensource nature can be a source of excitement for mobile developers, testers and quality assurance teams that are not afraid of a challenge. With the freedom of leveraging an open-source framework like Appium, true innovation can occur.

One final piece of advice that I have for Appium users is to find a vendor partner who can help your team leverage Appium to its fullest potential.

Today, many vendors are offering various services to help enterprise mobility teams with Appium, including offering expansive Appium support and built-in Appium servers in device clouds and mobile testing platforms.

Having a knowledgeable vendor partner to help navigate mobile testing using Appium can make the experience easier for your enterprise mobility team by boosting DevOps, agility and performance.

DAN MCFALL PRESIDENT MOBILE LABS

Dan McFall, President of Mobile Labs, has more than 19 years of experience in software and analytics, including strategy and implementation for customer support solutions, solution design, customer acquisition and industry growth.

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

14

TEST AUTOMATION: THE RIGHT

WAY UP DevOps is driving the adoption of wider test automation, argues Jonny Fletcher, Test Architect, ROQ

A

cross the IT industry, the demand to deliver solutions ever faster whilst satisfying the expectation of improving quality continues to gather pace, as users become ever more sophisticated and demanding. Against this background, the DevOps approach has become a hot topic and is becoming widely adopted as established companies seek not to be left behind by the so called ‘disruptors.’

A healthy DevOps workflow allows all members of the IT organisation to collaborate seamlessly. The DevOps adoption has become synonymous with continuous integration/ continuous delivery (CI/CD) methods, but whilst they may have common goals and are often used in conjunction, there is a subtle difference.

Whilst CI/CD is focused on automating the processes in software delivery, DevOps also focuses on the organisation change to support

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

great collaboration between the many functions involved.

TEST AUTOMATION IS ESSENTIAL TO DEVOPS SUCCESS With the widespread adoption of agile development methodologies and the need for more frequent deployments, the volume of testing required has increased significantly, especially around regression testing. With this increased volume of development, and the associated testing this brings, comes the need for test automation. Put simply, DevOps cannot succeed if it still requires lots of test cases to be executed manually.

Implementing an effective test automation solution isn’t easy and a lot of the hard work comes at the beginning, not only in defining good test cases with sufficient acceptance criteria, but ensuring that your application has been developed with test automation in mind – especially true with testing at the user interface (UI) layer.

ESTABLISHING A TEST AUTOMATION STRATEGY DevOps requires developers and test engineers to work together to help create test scripts and maximise test coverage. These scripts and code, supported by CI/CD tools, are used to generate builds, deploy them and test them automatically. A more strategic role for test

T H O U G H T

15

L E A D E R S H I P

automation is possible where people have the technical and management experience. This architect role means defining a quality strategy aligning with the DevOps culture, which include assisting in defining requirements; creating strategies where 100% automation coverage is not possible; defining quality metrics and measuring and analysing those metrics. In this role, it is more a case of not just finding the bugs but becoming responsible for preventing bugs.

run means an unwarranted delay to delivery or release. The emphasis on portability can also become more important, as the automated build process may spin up and deploy to an entirely new environment or set of devices and invoke our scripts to run on them.

WHEN AND WHERE TO

DEVOPS CONTEXT

AUTOMATE

So how then do you get started when you need to derive the essential benefits of test automation in an emerging DevOps culture? At ROQ we apply our Understand>Define>Deliver approach:

In order to be truly effective your test automation efforts need to address at least three different levels across the solution. In non-DevOps software development, many people end up inadvertently falling into the_ ‘ice cream cone anti-pattern’ for testing by putting more emphasis on automating at the UI level. However, a more practical approach is one that flips that ice cream cone upside down. This approach, the ‘ideal test automation pyramid’ made popular by Mike Cohn, improves the ROI of automation and guarantees that you will receive the most benefits from your testing effort.

Devoting effort to automate low level unit tests that prove the functionality works, perhaps using a test driven development (TDD) and/or behaviour driven development (BDD) approach can prove a robust and effective approach.

Equally automated tests are also very effective at the integration/API layer, proving that our components (particularly with a microservices design) and external solutions work together as intended. However, as user experience is key for our discerning users and compatibility with more platforms/devices/operating systems is also a challenge, automated tests to be truly comprehensive also need to validate that our solutions behave correctly at the front-end (GUI layer) unless our solution is ‘head-less.’

DEVOPS DEMANDS OF TEST AUTOMATION Simply – the automated scripts in this world must provide instant and accurate feedback to support decision making. In the CI/CD context we need to make decisions up-front around what to do in the event of failure, i.e., which failures should prevent deployment or halt the process. Whilst there has also been a requirement for automated scripts to be reliable, the emphasis becomes even more acute when any false negative/failed

ESTABLISHING TEST

Put simply, DevOps cannot succeed if it still requires lots of test cases to be executed manually

AUTOMATION IN THE

Understand Initially you need to understand key stakeholders, where the organisation is testing capability wise, the technology stack that may require testing, what tools/environments/data provision are already in place, along with what are the key business and technical risks facing you.

Figure 1. The ideal test automation pyramid.

Define Once the background is established you can then define what your approach/strategy for automated testing should be. What skills, environments, data, tools will be required? You should conduct proof of concept exercises for additional tools as necessary.

Deliver After this is agreed you can start to on-board resource; develop skills; build frameworks; develop scripts to test all layers; secure test data; build environments; integrate with the orchestration and other tools; and start to deliver the desired benefits from test automation.

JONNY FLETCHER TEST ARCHITECT ROQ

SUMMARY In a fast-changing world, which demands more frequent deployments to keep up with the pace of change, test automation can help drive huge efficiencies in a DevOps environment. It is imperative that all parties engage as early in the process to maximise test coverage and to provide feedback to support the decisionmaking process. However, test automation isn’t a silver bullet and doesn’t replace the need for great manual testers; it purely enables them to expand the scope of the overall test strategy and to concentrate on exploratory testing and the overall user experience.

Jonny is responsible for the design, development, implementation and governance of test automation services at ROQ. As a Test Architect he has worked with global organisations across numerous sectors, ensuring their test automation frameworks are fit for purpose and maximise the return on investment. Before joining ROQ, Jonny worked in the financial services sector in a number of business areas.

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

16

DIGITAL TRANSFORMATION REQUIRES A HOLISTIC APPROACH TO QUALITY AND FOCUS ON THE USER EXPERIENCE

Yoav Ziv, Amdocs Head of Testing, perhaps more than most, knows about the importance of software quality. A 20+ year veteran of the telecom business, Yoav now leads the team that helps guide Amdocs’ customers through the challenges of adapting quality processes to the demands of the digital age. TEST magazine have caught up with Yoav to discuss what digital technologies mean to modern businesses and how they are changing the game for service providers like Amdocs.

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

S U P P L I E R

17

P R O F I L E

A

mdocs has been in the business of providing testing services in the telecommunications industry for over 15 years. What’s happening now that makes organisations renew their focus on application quality and testing?

The biggest change is that modern technologies such as mobile and cloud have exposed IT systems directly to the customer, elevating the importance of quality and performance to a whole new level. When IT systems were used mostly by internal users, occasional failures typically did not result in disastrous consequences for the business. Today, tolerance for bad user experience, application errors and poor performance is near zero – and that spells a big shift in how quality and testing are perceived in the organisation.

In addition, the move towards modern software development methods such as DevOps, highlights the need to approach testing differently than how it was done traditionally. This too adds to the focus on the testing domain.

What are some of the biggest changes in the way service providers approach testing in the digital age?

The focus of testing is shifting from verifying the application functionality and performance to validating the quality of the user experience. Software testing professionals need to focus on the enduser experience along every step of the application lifecycle. They need to think like end-users, behave like end-users, and ultimately test with the end-user in mind.

What’s more, testing has been gradually shifting from being a 'phase' at the end of the development cycle, to becoming an integral part of it. Instead of waiting for code from different development teams to be delivered for validation, testing has 'shifted left' to begin even before the development starts. Today, testing works in parallel with development, and accompanies the software application all the way through deployment, resulting in mostly automated testing, which is essentially part of the code itself. These and many other changes have caused an array of transformations in testing groups. The skills required, the processes used and the technologies used have shifted significantly in recent times and will continue to evolve.

Does Amdocs have a specialised offering to help service providers refocus their testing strategy on the end-user experience?

As a premier testing services provider for the telecom industry, it is part of Amdocs mission to help our clients navigate the changes and adapt their testing strategy to meet market demands. Amdocs Digital User Experience Testing (DUET) is our latest offering designed to provide our customers with a comprehensive portfolio of testing strategies and services that cover all aspects of the user experience, certifying back-end systems and front-end channels. In addition to traditional quality activities, such as functional, performance and security testing, DUET includes testing services specifically focused on validating the user experience aspect, including compatibility testing (to ensure that applications work on a variety of devices and platforms), accessibility testing (to provide the best experience for users with disabilities), usability and visual testing (to validate the application’s ease of use), and a number of additional services, such as in the wild exploratory testing. To save costs and accelerate delivery timelines, the DUET offering includes test automation tools and frameworks and stresses reuse of testing assets. Amdocs DUET is a result of strategic partnerships with some of the most innovative testing solutions vendors, and is integrated with our end-to-end digital testing platform Amdocs BEATTM and our automation framework, Amdocs BEATTM.

The focus of testing is shifting from verifying the application functionality and performance to validating the quality of the user experience. Software testing professionals need to focus on the end-user experience along every step of the application lifecycle

Testing is often seen as a bottleneck in the software development lifecycle. How is Amdocs addressing the need for quality while helping service providers accelerate delivery cycles?

Long release cycles and sequential phases in application development are quickly becoming the thing of the past. Concepts like agile and DevOps have been born out of necessity to dramatically increase time to market. Continuous delivery, with quality engineered into the very fabric of the application, is what enables today’s businesses to keep up with market demands.

Amdocs has developed a unique DevOps methodology that’s ideally suited for the complex environment of the Communications Service Providers. We essentially create an assembly line for each value-add scope item, which is synchronised across all development teams, software technologies and locations.

This assembly line uses standard DevOps technology, but in a much more co-ordinated

YOAV ZIV HEAD OF TESTING AMDOCS

Yoav Ziv is a 20+ year veteran of the telecom business, Yoav now leads the team that helps guide Amdocs’ customers through the challenges of adapting quality processes to the demands of the digital age.

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

18

Digital technology changes everything: to win the hearts and minds of today’s discriminating consumers, providers race to deliver new applications and functionality to the market as quickly as possible

S U P P L I E R

manner. There are many elements that enable this approach to work smoothly, including test environment management and test data management. These are not always viewed as essential components of quality, but they are in fact crucial mechanisms for implementing shift left and DevOps. You can’t test early in the lifecycle or continuously monitor your application in production if you don’t have ready access to accurate and realistic test environments and data.

Amdocs Testing has helped many companies around the globe succeed on their journey toward DevOps and modern delivery practices.

As we all know, transition to DevOps doesn’t happen overnight and most companies continue to operate in a bimodal reality, where some applications are DevOps ready, and others, often legacy systems, are still being developed in a waterfall fashion. Our testing professionals take the time to understand each customer’s individual needs and the current state of their development environments, and help them adapt their practices and methods to meet business demands. We are proud of our results, with many companies achieving a 50% decrease in release times and dramatic improvements in test coverage, quality and overall synergy between different groups involved in application delivery.

P R O F I L E

So far, we have been focusing mostly on application testing, yet there’s more to quality than just software. Does Amdocs have offerings in other areas besides application quality?

You are absolutely correct – holistic approach to quality and focus on the user experience can’t be application-centric. Communications Service Providers specifically rely on their networks to deliver reliable, scalable and secure services to their customers. The use of virtual networks is rapidly growing among CSPs, and Amdocs has developed a series of specialized services in the areas of network functions virtualisation (NFV) and softwaredefined networking (SDN) testing to support this trend. The complex and ever-changing nature of these networks requires a different set of testing techniques, methods and skills that are software-related, rather than specific to hardware. With many interconnected layers – application, orchestration, virtualised functions and supporting infrastructure – often coming from different vendors, it is essential to have testing strategies and tools designed specifically for NFV and SDN testing and monitoring. Amdocs brings to the table extensive knowledge of NFV and SDN technologies, along with our own and partner-built test automation solutions that can emulate virtual networks, generate NFV traffic, analyse results and response times and test communication between different components. We understand the importance of having a reliable, scalable, secure and resilient network, and we are helping our customers achieve this goal.

Do you have any advice for a company who might be just embarking on a digital transformation voyage?

Digital technology changes everything: to win the hearts and minds of today’s discriminating consumers, providers race to deliver new applications and functionality to the market as quickly as possible.

Interestingly enough, this does not have to come with a price tag of bad quality. A good and synchronized DevOps approach will actually increase Time to Market AND will improve quality. Also – Remember that quality today doesn’t end with the number of defects found before production or percentage of tests executed. Quality in the digital age is directly connected to a customer experience. The winners in this race are the providers who can delight their users with the best experience; and Amdocs is here to help service providers get there.

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

19

17 OCTOBER 2017

DON’T MISS THE NETWORKING AND ENLIGHTENING EVENT OF THE YEAR  CONVENIENT LONDON LOCATION |  OVER 100 SENIOR DELEGATES  12 THOUGHT LEADING DEBATES |  ALL LUNCH & REFRESHMENTS  NETWORKING AND KNOWLEDGE SHARING Want to attend? info@devopsfocusgroups.com +44 (0)870 863 6930

www.DevOpsFocusGroups.com T E S T M a g a z i n e | S e p t e m b e r 2 01 7

MACHINE LEARNING AND SOFTWARE TESTING:

20

PARALLEL LINES OR DESTINED TO MEET?

Assaf Dudai is the content marketing manager at TestCraft a pure SaaS continuous test automation solution for QA professionals. Assaf is mostly looking into the future of testing, how automation, codeless and machine learning can and will help testing evolve.

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

T E S T

21

A U T O M A T I O N

M

achine learning (ML), the leading implementation of AI, is a term thrown around a lot these days. The promise behind AI is incredible – computer programs that not only execute a command, or a series of commands, but can decide which commands to execute based on human-like sets of considerations.

Autonomous vehicles are the prime example

eventually.

If we look at driving, it’s easy to see this phenomenon manifests in the exceedingly high number of traffic accidents. Each time we get behind the wheel, we repeat the same process, and sooner or later we lose focus. It’s the human thing to do. This lays the perfect stage for automation and machines to save people from fatal accidents – and from software testing errors. Automation thrives on repetition.

of ML capabilities at the moment.

Medicine is another field that’s making impressive strides with ML for the purpose of diagnosing and redicting diseases based on patients’ data.

Software testing seems ripe for the ML treatment as well, as the growing reliance on automation suggests. But, is it written in the stars, or just a passing fad?

The ‘problem’ with automated testing is that it needs to be programmed. It might sound trivial, as any automation needs to first be programmed, but with testing it’s a bit more complex.

More tests need to be created every time a new feature is implemented or a new version is released, so automation in testing comes with a heavy coding burden, unless you turn to codeless automated testing solutions.

It’s simple to imagine how ML can simplify software testing. The ‘machine’ is presented with a software to be tested. It needs to analyse the software’s internal structure, and run all possible paths that can be taken through a site or app, verifying the validity of each path.

Since a machine (computer program) is able to digest information horizontally, it can instantaneously create a test plan, no matter how many steps or test suites are needed.

Now that we understand the ideal implementation, let’s take a few steps back to practicality.

INTERCONNECTION BETWEEN TEST AUTOMATION AND MACHINE LEARNING Software testing is heavy on repetition and error-prone execution: two pillars with a cause and effect relationship. When we need to repeat the same action over and over again, it’s inevitable that we’ll make an error

ASSAF DUDAI HEAD OF CONTENT TESTCRAFT

Assaf Dudai is the content marketing manager at TestCraft a pure SaaS continuous test automation solution for QA professionals. Assaf is mostly looking into the future of testing, how automation, codeless and machine learning can and will help testing evolve.

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

22

The first phase to involve ML in software testing is the diagnosis phase.

As in medicine, machine learning excels in determining the reason for what went wrong, or extrapolating what could go wrong THE 3 TYPES OF MACHINE LEARNING SUPERVISED LEARNING In supervised learning, the software agent is presented with labeled data sets and needs to glean information from it in order to label new data sets. Quantities of data are provided so the agent to learn how to pass judgment.

For example, images of cats and dogs are provided (input data) with their category of either ‘cats’ or ‘dogs’ (output data). The agent, after scanning a quantity of data, has been trained to pass judgment on categorised images, labeling them as either ‘cat’ or ‘dog’.

So, we can say that supervised learning is about approximation.

UNSUPERVISED LEARNING In unsupervised learning, the software agent is tasked with taking data sets and figuring out how to divide them, or how to find structure.

No labels are given. The judgment passed here is based on grouping together data sets

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

that look or act in the same way. There are endless ways by which you can divide these sets of data, but no one is telling the agent how to divide the data.

If we look at the cats & dogs example, an unsupervised agent might learn on its own how to distinguish between cats and dogs according to our perception of the two breeds. On the other hand, it might find other differentiating output factors.

So, we can say that unsupervised learning is about description.

REINFORCEMENT LEARNING In reinforcement learning the software agent is to deduce based on delayed rewards; it’s like playing a game without knowing the rules and only occasionally being told if you’re doing well or not. Rewards are set for different goals, and the software agent learns by trial and error how to achieve the various goals. It is as elusive as it sounds.

No cats & dogs example here. A reinforced agent would probably unleash the dogs just to see what they’ll do to the cats.

So, we can say that reinforcement learning is about exploration.

Of the three types, reinforcement is the most suitable for software testing. The exploratory nature of reinforcement learning is ideal for the complex requirements of testing, that go beyond the capabilities of both supervised and unsupervised learning.

Defining the rewards would be pretty straightforward – completing a registration process, adding an item to a cart (it must be noted that ‘rewards’ for a machine are different than what us humans would expect from a reward).

But defining the goals would be the most challenging part. With software, or a web application, an 'intuitive UI' is a worthy goal – would software agents ever be able to pass judgement on a goal like that? These are lofty questions that remain unanswered for now.

HOW CAN AUTOMATION TESTING TOOLS BENEFIT FROM ML? The first phase to involve ML in software testing is the diagnosis phase. As in medicine, machine learning excels in determining the reason for what went wrong, or extrapolating what could go wrong. With software, finding the bug is only half the battle, since a major component of development is debugging – understanding what causes the bug, and how to fix it.

The nice thing about testing as far as the machine is concerned is that the desired outcome of the process being tested, is already known. If we test the process for adding an item to a cart, the desired outcome is that an item would be added to a cart.

When a failure occurs, the machine needs to not only ‘report’ the bug, but also come up with a set of possible diagnoses and offer new tests to check each one of the possibilities.

This shouldn't be confused with static code analysis, which scans code and looks for problems without actually executing it.

Facebook made a splash a couple of years ago with Infer, its in-house developed AI code analysis that it later open-sourced.

But code analysis and testing are two very different things – one analyses code, the other runs actual tests on software or web applications.

Now, if a machine can suggests tests to verify its diagnosis, why not take it one step further and have it diagnose the entire software (not just bugs found) and compile tests to verify that all outcomes are achieved?

And, if it can offer tests to check its own diagnosis, why not have it run the tests instead of merely suggesting them to human testers?

As mentioned earlier, the realm of possibilities is endless as to how ML can simplify software testing.

T E S T

A U T O M A T I O N

DON’T

23

Isn’t AI an overkill for software testing? At the moment, when AI is in its infancy, it is channeled towards grand, lifechanging projects, like self-driving cars, recognising early signs of cancer and yes, creating a Twitter persona that turns nasty in 24 hours

KILL THE INTELLIGENT MESSENGER One thing to consider: isn’t AI an overkill for software testing? At the moment, when AI is in its infancy, it is channeled towards grand, lifechanging projects, like self-driving cars, recognising early signs of cancer and yes, creating a Twitter persona that turns nasty in 24 hours. So is it indeed necessary to pull out such big guns for the mere sake of software testing?

Another thing to consider: Many people refer to the current state of automation as AI. These kinds of misnomers are not uncommon with new technologies; extremely simplified implementations are often labeled as the real thing. In actuality, there are very few true AI implementations out there, though everywhere you look it seems there is a “machine learning something”.

It’s easy to understand why this confusion between AI and automation happens. You can automate a response to a specific query – or automate thousands of responses to thousands of specific queries - and present it as AI. But it’s not really (sorry Siri). To some savvy marketers, if it looks like AI, and users will think it’s AI, they seem to have no problem slapping the AI label on its back.

So with all humility, we’ll probably have

to hold off with AI in software testing for now while enjoying the fantastic progress being made with automation in testing. Without a doubt, codeless automation will one day be enhanced by machine learning and this day is not far off.

Once AI technology has matured a bit, and we succumb to the extremely high costs of implementation, the floodgates will open and we’ll soon see it integrated into, well, basically everything.

We’ll sign off with a calming message for QA testers: the human kind. Since software is made for humans (and defined, programmed and designed by humans), human feedback will always be needed.

We cherish the human factors of good software design and value the user experience. We might concede regression testing to the machines – who wants it anyway?

And save some manual testing for us.

Exploratory instincts will forever be in vogue, which is something Uber can’t say about human drivers, now can it?

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

24

ERPS AND AUTOMATION TEST DATA

Automating an ERP product has a profound impact on the automation architecture. In his experience, Ali Khalid, Automation Lead at Asset Works, says one of the most challenging aspects is test data. The highly complex and integrated business logic of ERPs coupled with configurable work-flows makes ERP’s test data very difficult to manage. Here, Khalid explains different methods of handling test data, some of which would work well for less complex applications, and methods suitable for ERPs only T E S T M a g a z i n e | S e p t e m b e r 2 01 7

T E S T

25

A U T O M A T I O N

E

veryone in the team had different and diverse backgrounds in automation, giving us a wide array of expertise. We never thought our test data management experiences would not be enough and therefore did not give the problem it’s due share in the start.

In time, we realised the test data needed was enormous, and more importantly highly inter-dependent. The use cases were long and complex, and could not be done in one test. For multiple tests we could not isolate dependencies of one test from the other.

Our pre-script data creation methods were not efficient and more importantly were not reusable since we could not delete created data at the end. Over the course of few months we tried many approaches.

While most of these methods would have worked for less complex applications, they were not compatible for an ERP. The evolution of our solution for test data management would provide insight to diverse techniques that could be used for lesser complex products, while giving an effective solution for ERPs towards the end.

COMPONENTS OF DATA MANAGEMENT A good UI automation script would have a pre-script and post-script methods. The former for setting the application state from where the actual test can begin, including creating any test data needed.

Post-script methods are used to clean up the environment while returning the application to a workable state for the next test. That means, we would need a data creation strategy to be used in pre-script, a method of saving data to be used within the script itself and then a deletion strategy to be utilised in the post script part.

SAVING DATA IN THE APPLICATION Our first attempt was to create and save seed data needed for tests in the application under different users/record names permanently.

The test would go in and use the data saved for it and generate the needed results. For a SaaS application, I had used this method before which saved us tremendous time and resources.

Nature of the ERP application was quite different. Any data used in a scenario was not

useable again since it would progress the user flow with that record and we couldn’t reverse the change for the next run. For some tests, it did work but where transactions were being performed, we realised something else was needed.

CREATE AND REMOVE After brainstorming we figured let’s try the basic approach. Although our gut told us it wouldn’t work, but was worth a shot. Data can be created in many ways, from the UI, from API level calls or adding data directly to the database. Similar permutations go for deletion.

On the UI level, the data creation steps for our product were three times lengthier than the actual test! With the additional drawback of accumulated probability of failure during data creation. Deletion was even trickier.

For many cases the data could not be deleted since the user flow was not completed yet, the application wouldn’t allow to remove it. Eventually the associated cost with creation and then not able to delete it made the whole plan non-practical.

Next we tried adding data to the database directly. At first seemed simple to just update one respective table. However, for slightly complex cases the number of changes in different tables were so complex, it was hard to figure out the database updates needed without going through the code. Finding out how to add each record through backend took considerable time and still we could not be sure if it’s done properly. Working with compromised data was not an option therefore we had to shelf this method too.

Creating data at the start and removing it once the test is complete is a very common and good practice. For most applications, this method should work smoothly. Generating data from UI level is most expensive to create and maintain, next from on the service/API level and the most efficient is through queries updating the database itself.

CREATE RECORDS: DON’T DELETE THEM Since we cannot delete data, how about if we create data that does not have to be deleted.

Every time we create a new random record, use it and then leave it in the application.

Next time create new data with a different ID. “What about long steps of data creation three times the actual test” someone said.

We never thought our test data management experiences would not be enough and therefore did not give the problem it’s due share in the start

ALI KHALID AUTOMATION LEAD ASSETWORKS

Ali Khalid is Automation Lead at AssetWorks. Ali has been working in the testing industry for the past 10 years. Having a background in electrical engineering, he has worked in varied industries, most of which are safety critical and require rigorous software quality processes and testing techniques. For the past 5 years Ali’s passion has been learning and advancing test automation and driving testing teams towards continuous testing.

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

26

We all agreed there was no way of reliably creating data at the start of each test, and almost impossible to delete that created data. A technique one of us had used before was to create a separate automation database with all the seed data added in there manually

T E S T

“We can align tests in such a way, the output from the previous test can be used as seed data for the next test”, this was the cleverest answer we could think of. This way we don’t spend time creating records which are already being created within tests. All seemed well until these tests started to run in the daily batch run.

About the random numbers, firstly not all fields allowed a huge enough permutations list due to limited number of characters supported by that field. Secondly, even the ones that did, after a short subset of random numbers, the number sequence would occasionally start repeating itself.

Generally, using random numbers would work fine for limited number of tests and wider character length fields. The drawback however, eventually you would need to clean up those records at some point in time.

As for the linked tests, most of the time some test would fail in between. Since the following tests needed data from our failed test to run, all subsequent tests would fail too.

Creating tests dependent on one another is generally a bad idea. The probability of failure gets multiplied and murphy’s law comes into play at its best most of the time.

DB RESTORE We all agreed there was no way of reliably creating data at the start of each test, and almost impossible to delete that created data.

A technique one of us had used before was to create a separate automation database with all the seed data added in there manually.

Copy over this baseline DB to another environment’s database and execute the automation batch there. Next time before running, restore the baseline again.

A backup is taken of the baseline database

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

A U T O M A T I O N

every day at a certain time. The restore process, which runs every time before the batch run, copies over the baseline DB’s back-up copy and writes back any unique environment variables different from the baseline DB. This way we don’t spend time creating data nor cleaning it at the end and at the next run, we have fresh records which can be used again.

The drawbacks of this approach we saw was firstly the overhead. Instead of one, two dedicated automation environments are needed. You can use the same app/web server; however, the databases would be separate instances.

Secondly, like us if your product has multiple versions (which is most likely the case), you might need multiple dedicated automation environments to cater to all application versions.

However, the price tag is worth the benefit. In our current automation suite, we create and delete around 20,000 fields worth seed data within 5 minutes (the time taken by the restore process). Done the conventional way, this would take hours.

SAVING DATA Along with finding the right way for creating and deleting data, we were also trying out the best possible way to save test data to be used in the scripts. The common methods used by automation folks are .xls files, .csv files, database tables and xml files. In our case the data was huge (17,000 fields and counting), and reusability was very important.

We store our data in JSON objects instead.

A separate JSON object for each screen. This format is readable, very easy to pass along in the code and has the least conversion and storage over-head.

FINDING WHAT WORKS FOR YOU Managing your test data is vital. Without ensuring integrity of the data being used, validity of the tests is at stake. More importantly, if the method is not efficient enough, it will make maintaining the scripts difficult and the test scripts brittle causing to fail randomly during batch runs.

There is no set guideline to figure out what might work best for your project.

However by knowing the different ways test data management can be done, the evolution of your test data strategy becomes easier.

27

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

28

EXPLORATORY, AGILE AND AUTOMATED:

THE TESTERS CHALLENGE

Exploratory testing has been around since 1984. But how does it fit into today's world of software development? Is it an alternative to agile testing? Or does it complement it? Where does automation fit in? Mark Thomas, CEO and experienced developer of Coderus, gives us his thoughts

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

E X P L O R A T O R Y

29

T E S T I N G

I

t was over 30 years ago that software testing guru, Cem Kaner, coined the term ‘exploratory testing’. Yet there still seems to be some debate among testers about the nature and merits of the process, and how its benefits stack up in comparison with the more commonly used agile method.

In reality they are two sides of the same coin. Both are based on continuous testing throughout the development project. There are, however, subtle yet significant differences in the methodology, giving them their own unique advantages and disadvantages.

The exploratory testing process tends to be more personalised and responsive, with tests written to resolve identified issues and bugs. As a result, the tester is given a greater degree of freedom and responsibility on their approach and structure of the project.

Agile testing takes a more team-based approach, treating testing as an integral part of development and often using automated testing tools to solve particular challenges.

Almost all developers follow the agile SDLC, which, among other things, can help reduce feedback latency.

A SHORT HISTORY Exploratory testing evolved from 'ad hoc' testing, which had acquired negative connotations – its name implying that it was carried out without due care and attention.

Kaner’s vision was to establish a “style of software testing that emphasises the personal freedom and responsibility of the individual tester to continually optimise the quality of his/her work by treating test-related learning, test design, test execution and test result interpretation as mutually supportive activities that run in parallel throughout the project.”

Agile testing follows the ideology of the Agile Manifesto, mainly its guiding principle of continuous delivery and verification.

It's a more effective alternative to the 'waterfall' approach, in which the design and implementation stages of the project happen before and separately from the verification and maintenance stages. In the agile method, developers and testers continually co-operate throughout the project, following a more thorough testing regime while saving time and money.

SPEED ADVANTAGES AND THOROUGH DISADVANTAGES So, agile takes a more organised, collaborative approach, while exploratory is based more on established testing protocols and guidelines.

But what are the pros and cons of the two methods?

In the case of exploratory testing, the advantages are mainly about speed and efficiency. Less preparation time is needed and critical issues can be found more quickly.

In addition, testing can be targeted at a specific area of the software, for example, a feature that has been patched or updated.

As for the disadvantages, exploratory testing is not really thorough enough to be used as a stand-alone solution. Firstly, it's impossible to determine if all components of the software have been tested rigorously.

Secondly, the testing process may not always align with the requirements and specifications for the product, as defined by the client.

Agile testing, as its name suggests, is generally more responsive and dynamic.

Developers gain immediate feedback on present and potential issues so that they can

MARK THOMAS CEO CODERUS

Mark Thomas is CEO of Coderus, a software development and consultancy company based in the UK, but working across the globe. Mark is a proven and respected developer and software designer. His early interest was in Mac OS X and iPhone development.

However, with the industry moving in so many directions he needed to fully understand the opportunities provided by other mobile, operating system and server platforms and as well as keeping up with the latest tools and supporting technologies.

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

30

Software testing has come a long way in 30 years. When users bought apps in boxes, a waterfallstyle production process did the job, quality assuring the final product before it was shipped

E X P L O R A T O R Y

be addressed earlier in the development cycle.

The result is a faster route to a higher quality product with fewer bugs in the final version. What's more, testing resources are generally used more efficiently. In-depth testing can be concentrated on specific facets of the app, for example, those that have been amended or further developed.

The development team plans and carries out development sprints, usually between one and two weeks in duration. At the end of each sprint, these deliver a testable product to the test team. The team can then concentrate single-mindedly on testing tasks completed during the sprint.

One of the key plus points of agile testing – a collaborative, team-based approach – can also be one its negatives. If a member of the testing team leaves the project during development this can impact significantly on the testing process, slowing it down or compromising its overall integrity.

The dynamic nature of agile testing also

T E S T I N G

has its drawbacks. It's common for new features to be implemented at various stages of the project and timelines may not be adjusted accordingly.

As a result, benchmarks can be missed and KPIs can be put under pressure.

Alternatively, things can be rushed and bugs may slip through the net. Agile testing also lacks the same coverage as other types of testing, for example, the regression method and specification-based techniques.

MAKING A CHOICE Which is best – exploratory or agile?

Because of their individual advantages and disadvantages, the simple answer is both – if timescales and available resources make it possible. If you're under time pressure, exploratory testing is the best option.

However, it could miss critical bugs that would be picked up by an agile approach.

Exploratory testing really comes into its own when there is no robust specification from which to derive specification-based test cases, or when it's unnecessary to use a fully structured set of tests. A classic example is a development sprint that consists of bug fixing rather than implementation of new features.

At Coderus we use exploratory testing when completing regression testing after a bug has been 'fixed'. Our testers will use existing knowledge of the app to test other components upon which the fix could have a direct influence.

THE ROLE OF AUTOMATION Although automated testing tools are more usually associated with agile testing, they also have their place in Exploratory Testing. In the agile process they're particularly effective at speeding up any regression testing that needs to be carried out. In exploratory testing they can be used to monitor the user flows of the app in the initial test run, saving time in subsequent runs.

Software testing has come a long way in 30 years. When users bought apps in boxes, a waterfall-style production process did the job, quality assuring the final product before it was shipped. Now that we have downloads, periodic bug fixes and updates, the idea of continuous development – and, consequently, continuous testing – has become firmly established. Exploratory and agile testing both have their roles to play. That's why both methods will probably survive well into the next decade, and probably beyond.

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

31

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

32

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

33

PROTECTING AND CONNECTING THE MONEY

Andrew Holt looks at the important relationship between software testing and financial services companies

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

34

Software testing is critical for financial services. Working with some of the leading financial services companies across the world, we have seen that the industry has increasingly recognised the essential function of software testing.

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

F I N A N C E

T

he recent and on-going spate of cyber attacks on financial services organisations brought into sharper focus the necessity of having effective cyber security. Indeed, more than any other industry, financial services operate in a vibrant and complex environment in which operations are driven by high volume events conducted in real time.

Whether an insurance company processing claims, a capital markets firm performing trades or a credit card company checking millions of transactions for fraudulent activity, the financial services business is riding on the accuracy and availability of good IT systems.

In turn, every systems user: from a business, regulator, customers and vendors bring growing requirements for speed, transparency, and accountability.

Financial institutions must continually use innovative solutions, which are enabled by modifications to existing applications or new developments. As financial services companies compete in a global business environment in which speed and

differentiation must be achieved while driving costs down but at the same time are secure.

Market shifts, technological developments, and increasing regulatory and investor scrutiny are affecting the revenue growth and operating efficiencies of financial services industry. To perform efficiently in this persistently challenging competitive environment, financial institutions need to increasingly harness new technologies to provide better services and customer offerings.

But increased requirements for data security, privacy, regulations, and customer expectations combine to add new pressures to overloaded IT teams.

EVOLVING WORLD OF FINANCE Therefore it not surprising that finance is an exciting area within testing and quality assurance, partly due to the many changes occurring across the sector. Examples of these

35

F I N A N C E

changes include shifts in the way customers now interact with their banking, insurance and financial providers, the mass adoption of contactless payment and regulatory changes aimed at ensuring a competitive environment.

This context creates a unique range of challenges and opportunities for testing teams and vendor partners. While quality remains vital in such a tightly regulated industry sector, the approach to testing has to be adapted to respond to the industry changes.

Is this happening? Yes, says Ryan Lloyd, VP of Test and Development Products at software testing group SmartBear.

“Software testing is critical for financial services. Working with some of the leading financial services companies across the world, we have seen that the industry has increasingly recognised the essential function of software testing.

“Our customers are able to ensure their applications work flawlessly and transactions are executed correctly one hundred per cent of the time, and they are also able to meet the increasing demand for speed of delivery and cost effectiveness they need compete in the marketplace.”

One such approach could be referred to as distributed agile as revealed in a Software Testing News online study. In years gone by it may have proved difficult to link multiple dispersed teams into a collaborative agile process, many of the barriers to distributed agile have now been overcome by technological advances in communication.

In addition, organisations within the financial services sector are looking to adopt and deploy agile test delivery on a more widespread basis. Importantly, it is also evident that they are now looking to apply agile in a global context.

Major financial services organisations operate within a global setting, with multiple locations across different countries and continents. As such many organisations now have global capabilities, expertise and partnerships that have now been established over several years.

Moreover, if more evidence was needed, the Software Fail Watch report analysed all software bugs reported in 2016, highlighting an extraordinary reminder of why effective software testing is so crucial to every business. The report, commissioned by Tricentis, identified 548 recorded software fails affecting some 4.4 billion people and US$1.1 trillion in assets. A significant amount of time, people and wonga.

Highlighting year-on-year software fail statistics across finance and other sectors,

with accumulated time lost due to software failures was 315 years, 6 months, 2 weeks, 6 days, 16 hours and 26 minutes. Finance numbers decreased by nearly 35%, indicating that the industry is embracing testing in some major way.

For software testing companies, it is worth noting the amount of potential business on offer: as banks command huge resources. For example, last year JPMorgan Chase spent over US$9.5 billion on technology, including US$3 billionn on new initiatives. Even getting a small share of that is big business.

CEMENTING CYBER SECURITY Inevitably, the place to begin with is cyber security, which can cost financial service companies millions and more in reputational credibility. According to one study, cyber security is now firmly cemented as a key issue in the boardroom of FTSE 350 companies. On the face of it, this sounds a comforting find.

But, this isn’t reflected in the training that management receive in dealing with cyber incidents.

The survey carried out by KPMG as part of the government’s Cyber Governance Health Check, over half (54%) of businesses place cyber risk as a top group risk when compared with other threats that a company faces – up from the 29% who did so in 2014.

Boards are also more likely to debate and agree their tolerance for cyber risk than in previous years. More than half (53%) have this “clearly set and understood”, an improvement on the 33% from 2016. This is all positive.

However, it’s clear that even though cyber as an issue is recognised and understood more than ever by boards, the training in how to deal with the issue is still lagging.

Over two-thirds (68%) said they have not received any training to deal with a cyber incident, and only 2% stated they have received comprehensive training. More worryingly, 10% of businesses revealed that they do not have a plan in place to respond to a cyber incident.

Paul Taylor, UK Head of Cyber Security at KPMG, comments: “Cyber attacks continue to pose a growing threat to business. While cyber security has cemented itself onto the board’s agenda, they often lack the training to deal with incidents.

“This is hugely important as knowing how to deal confidently with an incident in

According to one study, cyber security is now firmly cemented as a key issue in the boardroom of FTSE350 companies.

On the face of it, this sounds a comforting find. But, this isn’t reflected in the training that management receive in dealing with cyber incidents.

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

36

the heat of the moment can save time and money. The aftermath of a cyber-attack, without the appropriate training in managing the issue, can result in reputational damage, litigation and blunt competitive edge.”

GDPR IMPACT

Another factor is that with the General Data Protection Regulation (GDPR) less than a year

away, 46% of boards still do not review and challenge reports on the security of their customer’s data – even though this figure has decreased by 15% from last year.

Yet, 71% of businesses describe themselves as somewhat prepared to meet the requirements of the GDPR, but only 6% say they are completely prepared.

When asked which GDPR requirements were causing businesses the greatest concern in terms of meeting compliance, 45% of respondents cited an individual’s rights to personal data deletion.

Paul Taylor adds: “It’s worrying that with less than a year to go, many organisations still have a lot to do. GDPR will affect organisations in the UK and worldwide that have any dealings with consumers and businesses in EU member states.

“The regulation sets a new bar for customer and client privacy expectations, but the truth is that many just don’t understand what they have to do and how to deal with it.”

“Boards need to take GDPR as a warning to rethink how they collect, store, use and disclose personal information. Done right this can transform their business model helping match services to client needs, done wrongly then they run a growing risk of data breaches and subsequent enforcement action with the prospect of fines up to 4% of global turnover.”

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

F I N A N C E

Other key findings from the report include: • 31% of boards receive comprehensive and informative management information on cyber risk: an increase of 10% from the 2016 Health Check.

• 57% of businesses have a clear understanding of the potential impacts resulting from a loss of, or disruption to, key information or data assets: an increase of 8% from 2016.

• 50% of boards review and challenge reports on the security of their customer’s data: an increase of 11% from 2016.

• 53% of boards receive only some information on cyber risk: a decrease of 4 per cent from 20116.

“Board members need to take collective responsibility for cyber security and consider it in every aspect of the business. If they can do that, then perhaps cyber security will become mainstream and a vital component of doing business in our digital world,” concludes Taylor.

TESTERS AND KNOWLEDGE Getting more specific, the majority of respondents from the Capgemini Financial Services World Quality Report acknowledge that while testers in their organisation may have some domain knowledge, less than 10% are considered domain experts.

Therefore, within the industry, post the financial crisis, the competition for financial services testing talent is high.

Testers with underlying knowledge of domain processes are sought after to help create effective testing for critical applications. In addition, the report found automation which can significantly reduce testing time and costs is severely underutilised in financial services.

In addition, The World Quality Report found that financial services companies are increasingly partnering with financial

technology providers to create new services and to make better use of data to improve customer services. Another apparent opportunity for software testing companies.

Key areas of development include products relating to Blockchain technologies, digital payments and smart, standardised contracts.

Financial services have unique challenges associated with very data rich applications, a diverse technology environment of differing devices, platforms and browsers and demands for an ever faster pace of development.

And testing in financial services must achieve all of this with best of breed of security. These pressures mean an unusually high proportion of companies, 57%, are making use of managed application testing services to help maintain standards.

In addition, The World FinTech Report 2017 from Capgemini explains that FinTech has emerged as one of the most promising trends of the recent past. Increasing customer expectations, the expansion of venture capital funding, reduced entry barriers, and technology innovations have allowed the FinTech industry to flourish in every sector.

The methods of offering services through traditional firms have not changed much over the years.

The fast-changing industry and busy schedules of individuals have highlighted many loopholes in traditional financial systems: on one hand, traditional financial firms believe that their business is at risk due to FinTech startups, on the other, various traditional firms are partnering with innovative FinTech firms to tap the market.

The onward march of corporate governance within the financial world is another factor, as corporate governance regulations in financial services have heightened the need for dependable 24/7 IT solutions, and, in turn, a need for dependable financial software testing.

Successful solutions that protect shareholders and customers require initiative in several areas: • Broadening platform support and going mobile to increase accessibility • Transforming regulatory changes into business opportunities • Retaining customers through useroriented applications • Implementing SaaS business models to capture new clients Financial organisations that embrace these will be on the right track. It is up to the software testing world to effectively keep putting its case.

37

DevOps [a fusion of Development and Operations]

DevOps Online [a match between curious minds and information]

www.devopsonline.co.uk T E S T M a g a z i n e | S e p t e m b e r 2 01 7

38

PART OF THE ACADEMY

Being a test professional means having to be creative, distil fact from fiction, articulate impacts and analyse patterns. In an attempt to focus and expand on these qualities, Mark Galvin, Systems Assurance Manager, developed and created the University of Cambridge IT Academy. He shares his thinking and the benefits for the industry here T E S T M a g a z i n e | S e p t e m b e r 2 01 7

I T

39

A C A D E M Y

I

n another life, under a different name, I am the founder of a global writing community, published storyteller and debut novelist.

For my next novel, a near-future dystopia, based in a city much like your own, I am thinking of basing the lead character on a test professional.

The protagonist, a dedicated, but slightly flawed individual, is under immense pressure.

Not the normal heroics of swooping in and saving the day against seemingly impossible odds. This is more than defeating the antagonist of flawed process. It is greater than the eradication of bugs that hide lurking in plain sight waiting to devour the unsuspecting.

No, these events are just a normal day for some test professionals. Sadly, the promises of a better world or more agile, DevOps universe were not quite as utopian as promised.

No, the hero/heroine of the story is under pressure to assimilate themselves into a ruthless automation machine, or… they will become obsolete.

Sound familiar?

Perhaps, this is no work of fiction after all.

WHAT IS BEING ASKED OF US?

AS TEST PROFESSIONALS

In an age where testing and QA professionals are under scrutiny to reskill, test faster and provide greater value, it might seem counterproductive for me to advocate the benefits of branching into new areas.

Why should the people responsible for testing or quality care about nurturing future generations of IT professionals, especially if it is in disciplines that are seemingly unrelated?

You might believe that testing and QA are independent professions, with unique needs, values and career paths. Or you might believe that testing is a set of skills that should be utilised by a multidisciplinary team where roles are not static. Either way, quality should be the responsibility of everyone — few would argue against this.

If you have an academy, or are thinking of creating one, then it is my view that involving those professionals with the highest levels of experience in testing or those most passionate about quality are highly desirable aims.

Testing and quality should be embedded into the development of each new IT academy member from day one. From there it becomes a foundation stone for all IT academy members, regardless of the roles they find themselves in later.

WHAT DO WE KNOW ABOUT RUNNING AN ACADEMY? Having a career based on testing and QA is not as one-dimensional as many people want to believe. Indeed, reading some articles, posts or messages — you would feel it is the personal mission of many to degrade the role of tester, to state that it’s a job that can be done by anyone. That unless you learn to be a developer the apocalypse is coming with your name on it.

“The robots are coming. The robots are coming.”

Dystopian thinking, but it’s not as simple as that.

Of course, we all need to evolve and adapt – but hasn’t the profession been doing that since the very beginning?

Testing is not simply about coding skills.

It is a severe disservice to testing and QA professionals to say anything less.

To be a tester requires tremendous skill, guile, determination, an understanding of the technology, a grasp of the way users will confound even the greatest of other IT_professions, the ability to adapt and deliver against the odds — not just today but time

MARK GALVIN SYSTEMS ASSURANCE MANAGER AND HEAD OF ACADEMY INFORMATION UNIVERSITY OF CAMBRIDGE

Mark Galvin is Systems Assurance Manager and Head of Academy Information Services at the University of Cambridge.

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

40

It is far more than just promoting excellence in a niche skill, individual technology or IT discipline.

We don’t just want the Academy to be useful or effective.

We want it to be transformational

I T

after time —without fail.

Being a test professional means having to be creative, distil fact from fiction, articulate impacts and analyse patterns. A good test professional needs the skills of a subterranean navigator and the diplomacy of a UN peacekeeper.

They need to dispense persuasion with ease and understand the business and technical impacts of a change and adapt information and language accordingly. It means knowing which battles to fight, and which to leave.

These abilities are highly valued by me and I suspect I am very far from being alone.

If you look at what I have described, in many ways our skills have never been more needed or relevant.

But these skills are not unique to testing.

Shouldn’t we be promoting these skills for most of our IT professions?

This is, perhaps, why the leader of my division, Chris Edwards, asked me create, develop, promote and implement our internal IT Academy.

A C A D E M Y

WHAT IS THE FOCUS OF OUR INTERNAL IT ACADEMY? It is far more than just promoting excellence in a niche skill, individual technology or IT_discipline.

We don’t just want the Academy to be useful or effective. We want it to be transformational.

In this respect, it is as much about attitude and behaviour as it is about developing skills and experience.

We want our academy staff to work collaboratively, to work across the boundaries of roles, teams and technologies. To be true ambassadors for user focus. To constructively and positively challenge — with a view to advancing our aims and ambitions.

Those in the academy will not only spend time with their respective teams, but they will experience the roles of others. Regular time is set aside to spend time with test professionals, business analysts, developers, operational support, end users, sponsors, architecture, high performance computing and administration.

We track progress and provide transcripts and encourage our academy members to gain professional certifications where they are relevant and advance the needs of the individual and the organisation. However, we also recognise that development, growth, the ability to positively adapt to change and realise future potential are far more than just a certificate.

NEAR-FUTURE For us, we wanted to look to the future. The academy will help reduce single points of failure by injecting new resources to help learn skills that others (for whatever reason) have not been able to pass on.

It should promote the growth of existing staff, as new members develop and take on responsibility, it will enable them to develop themselves. It creates new ideas and thinking.

It encourages flexibility and movement between established teams.

Our first academy has five members, from different IT disciplines. However, there is a strong demand to grow the academy.

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

I T

41

A C A D E M Y

WHAT ARE THE NEXT STEPS? The senior leadership team were very supportive of the design and aims of the academy and they are as keen to see it thrive as I am.

As the academy matures, we hope it will act as a bridge between the IT central roles and IT supporting roles in the academic communities. There is further work to assess the feasibility of apprenticeships and outreach schemes to schools and colleges.

Creating a sustainable yet flexible academy will be a key requirement for me.

Using it to help existing staff and promote career or assignment mobility (where desirable), are key aspirations as the academy develops.

These are ambitious plans, and some of this will be complex to deliver, but I feel strongly that investing in the wider careers of IT professionals will help the ultimate aims of the testing and quality profession.

Evidence points to employees staying with organisations for shorter periods of time.

The IT industry has a retention challenge greater than most. Change and new ideas are necessary but many valuable employees

might feel that they have to move to grow and develop.

If we have assets in place to help train and develop new staff, and we demonstrate that we are passionate about quality then we should be using these assets to help develop existing staff too.

TESTING IS NOT EASY.

Testing is not easy, nor should it be.

Good testing can be fun and exciting (honestly), but often it is hard work

ACHIEVING QUALITY IS EVEN HARDER. If you are experienced in testing or QA then your input could make a difference to the next generation of IT talent. Testing is not easy, nor should it be. Good testing can be fun and exciting (honestly), but often it is hard work.

Automation and intelligent testing tools will become more commonplace than they are today and this certainty requires a significant change in our focus. However, it will be a very long time before machines can fully test with the complete understanding or empathy that a human has for a fellow user of a system or service.

Understanding how people feel, their

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

42

If you are just starting your career in testing, QA or other IT discipline, then you are the future we are discussing. Changes in technology over the next twenty years will be even more transformational than the radical advances we have already seen

I T

emotions and senses, their desired outcomes and aspirations, the decisions they make and how they use technology will remain just as important as testing code coverage and hundreds of device/browser combinations fired at the exact millisecond of code commit.

LESSONS FROM HISTORY It’s important to build the academy that suits your organisation and where you want to go.

Taking a standard model will get you part of the way, but not all of it.

From computing pioneers such as Charles Babbage, through to Bill Gates, Steve Jobs and more recently Mark Zuckerberg, they followed clear visions, challenged educational methods and brought their people along on the journey with them. An academy can help deliver the mission of an organisation and focus new, and existing staff, into delivering it.

WHO WILL HELP THE

A C A D E M Y

quality outcomes then you are the perfect person to help shape the future career of an IT professional. Very few organisations will say that quality is not important to them.

If you are just starting your career in testing, QA or other IT discipline, then you are the future we are discussing. Changes in technology over the next 20 years will be even more transformational than the radical advances we have already seen.

Your enthusiasm, dedication and talent are what will make these world-changing ideas a reality. Academies are a great place to use your energy and creativity and to challenge the norms.

Academies facilitate and encourage collaboration across disciplines, but they are often the also genesis of new ideas and act as a nerve-centre for the changing how organisations think about users, products and services.

Start by creating, building and defining your IT academies because it’s not just you, your organisations or your colleagues that will benefit — it is the wider world.

NEXT GENERATION OF IT

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

PROFESSIONALS?

EPILOGUE

I suggest that you are the answer. If you are experienced in testing methods or delivering

So, our protagonist has triumphed over adversity.

The prophets of doom were not all wrong in the end, neither were they completely right.

There was a difficult and challenging adventure along the way, where new skills were required, and tools had to be deployed in creative and innovative ways.

The robots came. Some good people hunkered down in their rusty tin shelters, but many perished. Some test professionals assimilated into the great coding hive-mind and they thrived.

Others used their skills, knowledge and experience to help shape their products, improve their services, enhance the user experience, develop their people and ready their organisations; and those were the people that made the greatest difference.

In every story the central character needs a strong sense of purpose. They also need allies. You should not be a lone voice on this journey. With your knowledge, passion and skills you should be leading the way for others.

Those that are starting their careers need to be inspired to help the cause. Whatever role they are in they should be passionate about quality, and who better to guide, nurture and influence them than you?

I T

43

A C A D E M Y

CI/CD

Experience the Best Test Management Tool and only Pay for What you Use

Real Time Reports & Test Analytics

Customizable Architecture with add on Apps

Enterprise grade Security & Scalability

For Agile and DevOps Teams info@qmetry.com

www.qmetry.com

+1 408 727 1101

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

44

TESTING: THE KEY TO AGILE SUCCESS

Dean Vittum, Senior Technical Sales Engineer at Applause, argues that a commitment to agile is an investment worth the effort

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

45

C R O W D T E S T I N G

T

raditional competitive advantages like offering lower prices are no longer sustainable. What is sustainable is the ability to learn quickly – quicker than competitors – and use the information gathered to meet customer demands and deliver superior customer experiences.

This shifting competitive landscape is why so many businesses have embraced agile methodologies.

Agile is the ability to pivot quickly based on customer need, and is spurred by tight feedback loops where developers, product owners and UX specialists, can understand what they are doing, how the are being successful, and where they can improve.

Agile frameworks enable teams to chunk up work into smaller pieces, thus creating less work, and release updates in a matter of weeks or even days, as opposed to the months it takes with traditional approaches.

DEALING WITH BOTTLENECKS In software development, moving and reacting quickly is easier said than done.

There always seems to be a bottleneck slowing down development, and consequently slowing down innovation.

Too often, that bottleneck is testing and quality assurance. The main reason testing and QA has slowed down the development process is that the traditional model had a few professionals testing every aspect of a digital property, so nowadays, many organisations look to other functions to pick up the slack.

More often than not, developers are tapped to take on QA responsibilities. This is problematic for a number of reasons. The first being that developers are too close to the software (having worked on it in the first place), potentially leading to some bugs being overlooked.

Second, developers are transferring their time, effort, and resources into working on QA when they otherwise would be spending it on software development. Clearly, this can slow progress to a crawl as time spent on testing is time spent away from development.

It’s not that companies haven’t tried to solve this conundrum. They have. It’s just been a fruitless exercise to this point.

Automation is a key enabler of agile, but many companies aren’t utilising it correctly, or are relying on it too much as opposed to human feedback in tandem with automation.

And, offshoring QA responsibilities can be

slow, pricey, and inadequate substitutes for real-world personas and experiences.

RETHINK QA QA and testing can be aligned with agile.

Companies need to rethink QA and understand it from a new perspective, one that takes advantage of real-world situations and real people to deliver real results. That’s where crowdtesting comes in.

Crowdtesting takes the community-based concept of crowdsourcing and applies it to digital experiences – things like websites, mobile apps, and connected devices – to understand how audiences are engaging with products and find out what software bugs exist. The aim of agile is to fulfill the high expectations and demands of users, and do so quickly, and crowdtesting is the best way to achieve this.

With crowdtesting, products are not tested in a remote laboratory, but on real devices in target markets. They’re tested by QA experts and digital natives that represent the target demographics of the app, website, etc. The results reflect the actual or possible user experience much better than would be possible in a lab or in-house.

Additionally, this method is compatible with the speed of agile because it allows many testers to test an application at the same time. Each tester providing just a few hours of testing can add up to hundreds of hours of testing activity delivered in a matter of a single day. Plus, the global nature of crowdtesting allows for testing at anytime, anywhere.

In software development, moving and reacting quickly is easier said than done. There always seems to be a bottleneck slowing down development, and consequently slowing down innovation

TESTING AT THE SPEED OF AGILE To align with agile, companies need to test early and they need to test often, throughout all parts of the software development lifecycle (SDLC). If a company is planning sprints every two weeks, it needs to be testing almost constantly to keep up with its rapid delivery plans.

Crowdtesting makes this easy. Instead of waiting on slow offshore methods, or taking time away from developers, crowdtesting allows companies to test their products on their target market, in the locations that are important to them, and on the devices they anticipate will be used.

It is essential that every relevant test procedure is planned early on and

DEAN VITTUM SENIOR TECHNICAL SALES ENGINEER APPLAUSE

Dean specialises in startup environments. As a Senior Technical Solutions Consultant at Applause, Dean works closely with prospects, reviewing testing requirements and developing proposals to meet prospects’ needs.

Prior to his role at Applause, Dean worked as a Senior Quality Assurance Manager for a number of companies in various high tech industries, building all aspects of the SDLC and QA teams.

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

46

Testers provide valuable information and insights, and this should all be recorded so that companies can visibly understand what problems arise when testers are using their products, and how they can recreate them perfectly so that they can be fixed in a timely manner

C R O W D T E S T I N G

implemented at all levels. This helps to identify and resolve problems as quickly as possible, and regression tests ensure that any new software enhancements do not interfere with existing functions.

Software is subject to frequent and often profound changes and so the advantages of regular evaluation and feedback are obvious: short development cycles with fast implementation sequences.

LEVERAGE AUTOMATION WHERE IT MAKES SENSE For products that are changing rapidly with each innovation and iteration, maintaining automation scripts can take as much effort as the product programming itself. When an automated script fails, they need to be investigated, fixed, and bug reports need to be written every single time, so choosing the right time and approach to test automation is critical.

In addition to test case execution, automated or not, it is important to run exploratory tests, in which the software is tested without a predetermined test plan.

Human instinct and creativity are needed to find both superficial and critical errors that cannot be planned for in advance. The quality of these test activities ultimately determines the quality of the final product.

This isn’t to say automation shouldn’t play a role in QA. It absolutely should. But, it shouldn’t take up all of a company’s time.

Crowdtesting vendors can curate a group of automation experts that can handle all of the toughest and time-constraining parts of automation, taking the strain off of internal teams to find, investigate, and fix broken scripts.

MANAGE YOUR DATA Test data management is underestimated, but essential. Testers provide valuable information and insights, and this should all be recorded so that companies can visibly understand what problems arise when testers are using their products, and how they can recreate them perfectly so that they can be fixed in a timely manner.

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

At its core, agile exists to optimise workflows – if data is collected and shared with other departments, companies have the tools to improve on existing practices and work as efficiently as possible.

While it requires a rethink of operations, the change to an agile methodology holds enormous potential for software development and frees up resources for other areas. In order to minimise obstacles in the implementation stages, companies need to understand agile as a holistic approach which includes testing.

The close cooperation between developers and testers is an absolute prerequisite – developers need full technical support and testers need to be involved in the respective processes as part of the sprint planning. Therefore, reacting to changes in the short-term ends up being more important than following a long-term plan.

Ultimately, if a digital product – no matter how shiny and new – doesn’t work, users will drop it immediately for an alternative. By investing in agile, companies are investing in the quality, security, and user-friendliness of their products to safeguard against potential user disappointment and create a much faster, better performance.

47

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

WHEN IT COMES TO SOFTWARE TESTING...

48

...DO YOU HAVE AN EYE FOR DETAIL? To support our continued expansion and client wins, ROQ is expanding our Test Automation Team and actively looking for experienced Test Engineers and Senior Test Engineers.

If you are looking for your next career move, have experience in Java or C# and knowledge in test automation concepts and approaches, ROQ wants to hear from you.

Permanent and Contract opportunities available.

Interested in finding out more?

Please get in touch with Katie Franklin on 01257 208890 or email careers@roq.co.uk.

www.roq.co.uk

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

MAKING SOFTWARE WORK

49

Welcome

to the Software Testing Conference NORTH

T

wo years ago we launched the ‘Northern’ conference in the beautiful city of York, to help those within the community that were located further afield, and what a success it has been. With a venue as striking as the Principal York Hotel and such cutting edge content, that’s all supported by leading industry vendors, Software Testing Conference NORTH is now firmly established on the industry calendar.

Our main aim throughout the conference is to bring together the software testing and QA community, and professionals that recognise the crucial importance of quality within the software development lifecycle will attend this wonderful event.

Delegates will find themselves debating at roundtables, listening to senior industry speakers, watching inspirational presentations, networking with new contacts, and sourcing the latest products on the market. All this to ensure you with practical advice and solutions to share with your teams and organisations.

We are aware that everyone has their own ways of doing things, which is why we are supplying you with a diverse group of speakers and Executive Workshop hosts, from

different business backgrounds, but all with a little something in common...a passion for software testing and QA! The conference will consist of experienced professionals from the likes of Aviva, BBC, Eurostar Trains, Merseyside Fire & Rescue, JP Morgan, Reed Business Information, Hotels.com and more.

In order to maximise your time at the conference you will of course, need to take part. So don’t shy away, get stuck in and share your questions, concerns and comments with other attendees at the Executive Workshops and during the lively questions and answers panel debates at the end of each day.

GRANT FARRELL MANAGING DIRECTOR

I hope you enjoy the Software Testing Conference NORTH as much me and my team have organising it.

Finally, I would like to say a massive thank you to our sponsors, speakers and roundtable hosts, without whom it would be impossible to put together such an event, and of course thank you for coming.

Please share your thoughts and feedback with the team and online using #NationalTestConf!

#TestingConfNORTH

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

D A Y

50

DAY ONE

Track 1

Track 2

Workshops

Time

Events Centre (lower ground floor)

Crown Suite (first floor)

Minister Room (ground floor)

08:45 – 09:20

Registration

09:25 – 09:30

Welcome from Grant Farrell

09:30 – 10:30

Leigh Rathbone,Test Principal, Shop Direct: Take the blue pill, or the red pill, you decide

10:30 – 11:00

Coffee, networking and exhibition

11:00 – 11:35

Bill Watson, Test Manager, BBC: Outsourced software

Sriram Angajala, QA Lead, Eurostar Trains: Role of automated rests in the continuous delivery pipeline

11:35 – 12:05

Cassy Calvert, Head of Testing, BJSS: Good or Evil? Will AI’s Killer Instinct End Testing as we Know It?

Gary Hallam, Director of Sales Engineering, EMEA at Delphix: DataOps - how to test faster with greater confidence

12:05 – 13:20

Lunch and exhibition

13:20 – 14:20

Keynote: Tom Berry, Test Delivery Manager, Aviva: When worlds collide – how test can succeed in a mixed methodology world

14:20 – 15:00

Mahesh Tanneeru, QA Engineering Lead at Reed Business Information: Getting Test Automation Right To Deliver Quality Software

Suresh Chandra Head of Testing & Quality AssuranceNewDay Ltd: Automated Testing in Agile Projects & Behaviour Driven Development

15:00 – 15:20

Refreshments

15:20 – 16:20

Q&A Panel:Measuring software testing; what are the new and best practices?

16:20 – 17:30

Drinks Reception

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

O N E

Keith Yorkston, Head of SQS Academy: Everything you wanted to know about security testing but were afraid to ask (90min)

D A Y

T W O

51

DAY TWO

Track 1

Track 2

Workshops

Time

Events Centre (lower ground floor)

Crown Suite (first floor)

Minister Room (ground floor)

08:45 – 09:20

Registration

09:25 – 09:30

Welcome from Grant Farrell

09:30 – 10:30

Keynote: Lee Crossley, QA manager, JP Morgan: The Rise of the Machines

10:30 – 11:00

Coffee, networking and exhibition

11:00 – 11:35

Mark Rice, Business Relationship Manager,Merseyside Fire and Rescue Service: Localisation testing

11:35 – 12:05

Keith Yorkston, Head of SQS Academy: The problems with Risk and Risk-Based Testing

Bernard Jauregui, Cloud Consultant, BJSS: How I Learned to Stop Worrying and Love the Cloud

12:05 – 13:20

Lunch and exhibition

13:20 – 14:20

Keynote: Al Lines, QA Manager / Senior Software Engineer, Moodys Investors Services: 21st Century Test Management: A Retrospective

14:20 – 15:00

Leigh Rathbone,Test Principal, Shop Direct: Agile tester - you are not alone

15:00 – 16:00

Q&A Panel: Is Quality Testing: a debate; the future of testing revisited

16:00 – 16:30

Refreshments

Lee Crossley QA Manager, JP Morgan Rage Against The Machines (90min)

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

S P O N S O R S

52

EVENT PARTNER

SII Sii – top nearshore testing services provider in Central Europe.

The leading IT and engineering expert in Poland, providing solutions for business partners from all around the world. The company has the biggest testing resources in Poland and its own mobile testing lab. Read more www.sii.pl/en/offer/testing/

EXHIBITORS

JAMO SOLUTIONS

INFUSE

Jamo Solutions markets the tool Jamo Automator. The tool disrupts the test automation market by the fact that no technical skills are needed to create re-usable automated test cases.

Infuse provides modern software testing delivery, consulting, and training services. We also offer a test automation tool, useMango™.

Our specialities are test automation and performance engineering.

TESTING CIRCLE / SPARTA GLOBAL

SCOTT LOGIC

Testing Circle is a leading Software Testing Services organisation; providing tailored, flexible and cost effective solutions to Finance, Government, Retail, Telco, Gaming, Media and FTSE 100 clients throughout the UK.

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

Scott Logic is a bespoke software consultancy with offices in Newcastle, Edinburgh, Bristol and London.With a number of longstanding clients from a range of sectors, the company provides a broad range of services, including User Experience (UX) design, project delivery and testing.

S P O N S O R S

53

EXHIBITORS

DELPHIX Delphix’s mission is to connect people to data and accelerate innovation. Fortune 100 companies use the Delphix Dynamic Data Platform to connect, virtualize, secure and manage data in the cloud and in on-premise environments. For more information visit www.delphix.com

BAG SPONSOR

LANYARDS AND BADGES

BJSS

SCOTT LOGIC

BJSS is an award-winning deliveryfocused IT Consultancy. With over 20 years' software development and IT advisory experience, we are renowned for technical excellence, cost-effective delivery and our proven BJSS Enterprise Agile approach.

Scott Logic is a bespoke software consultancy with offices in Newcastle, Edinburgh, Bristol and London.With a number of longstanding clients from a range of sectors, the company provides a broad range of services, including User Experience (UX) design, project delivery and testing.

MICRO-EXHIBITORS

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

F L O O R

54

F L O O R

P L A N

7.

8.

6.

1.

pillar

5.

pillar catering

main entrance

3.

4.

O

A

2.

K

R

O

O

M

1

Delphix

5

Jamo Solutions

2

Sparta Global

6

Apica

3

Infuse

7

Neotys

4

Sii Polska

8

Scott Logic

Free WiFi is avaialble

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

P L A N

S P E A K E R S

SRIRAM ANGAJALA QA LEAD EUROSTAR TRAINS

55

TOM BERRY

TEST DELIVERY MANAGER AVIVA

CASSY CALVERT

LEE CROSSLEY

HEAD OF TESTING BJSS

QA MANAGER (MOBILE) J.P. MORGAN

Role of automated rests in the continuous delivery pipeline

When worlds collide – how test can succeed in a mixed methodology world

Good or Evil? Will AI’s Killer Instinct End Testing as we Know It?

Once we have continuous integration and test automation in place, we create a deployment pipeline: the key pattern in continuous delivery. In the deployment pipeline pattern, every change runs a build that creates packages that can be deployed to any environment and runs unit tests and possibly other tasks such as static analysis, giving feedback to developers in the space of a few minutes.

Across many organisations, the workplace isn’t a straight forward split of agile or waterfall projects; instead different teams and projects deliver in different ways at different speeds. In this session, Tom Berry, Aviva Test Delivery Manager, will look at what testers need to bring to the table to deliver quality at pace in a complex environment

What possibilities does AI hold for Testers, and what should you be asking? How can you be certain that AI is doing the right thing and where will AI testing falter? Which industries will readily embrace it, and how can AI be tested alongside Agile techniques?

Sriram has worked with BBC, TFL, VISA, DELL, COOP and is passionate about transforming agile development process into quality driven process.

He is a technologist with diverse experience in multiple automation development stack from days of Silk Test, Winrunner till Webdriver IO. Sriram believes that advanced development approaches like BDD and continues delivery are only possible with right quality gates in delivery stack. He also enjoy training sessions and solving complex challenges and is the founder of East London Testers Meetup group.

Tom is currently Test Delivery Manager for Life Finance and Protection within Aviva. Having worked for Aviva for nearly fifteen years, his experience covers Finance Operations, Change Management and Assurance, specialising in Actuarial change and Data Warehousing. Tom believes that as part of the IT change community, Assurance Professionals are consumer champions. Tom will be discussing how, in an environment with multiple change models and methods, testers can deliver value to the business and more importantly our customers by focussing on outcomes and choosing the right tool for the job

In July 2017, BJSS engaged with Test professionals from across all industries to answer these questions. In this interactive session, Cassy Calvert, BJSS Testing Services Manager shares the results from this exercise and will provide a practitioner’s voice to these burning questions.

Dedicating her career to the art of Testing since 1999, Cassy joined BJSS in 2012 where she is responsible for ensuring quality test delivery across multiple on-shore and off-shore work streams in a high-pressure, Agile, delivery-focused environment.

Outside of IT, Cassy’s spare time is filled with enjoying life with her two daughters and coaching at her local gymnastics club.

Rise of the Machines

A look at the current and future state of mobile delivery technology, covering the following topics: • Context Driven Testing Mind Map Generation Defect Management across multiple Jira instances Dynamic Reporting across multiple Jira instances UI Automation/ Performance Testing using distributed mac infrastructure Service Virtualisation • Continuous Integration • Staying close to source (Apple/Google) • Mobile Device Farms • The future of UI Automation

A test professional with extensive experience in testing within multiple business sectors including Finance and Energy.

Specialising in building, mentoring and leading test teams and creating/shaping the right test strategies to best fit those teams and the applications under test.

In conjunction with his team and the wider technology community he created a test approach called Iterative Rapid Testing (IRT) to help test a multi award winning voice biometrics offering at Barclays.

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

S P E A K E R S

56

GARY HALLAM

#TestingConfNORTH

DIRECTOR OF SALES ENGINEERING, EMEA DELPHIX

DataOps - how to test faster with greater confidence

Reduce wait-time for environments and speed up application releases with fast, secure data distribution.

Learn how companies all over the world use the Delphix Data Platform to reduce the time to provision any type of test data—including masked or unmasked production data, subsets, and synthetic data sets—to just minutes, enabling organizations to collapse cycle times.

Gary runs the Delphix Sales Engineering team in EMEA. He has an application development and consulting background and prior to joining Delphix he worked for Oracle in Spain, managing an EMEA-wide Sales Consulting team focused on Information and Database Security solutions. Gary has a long experience in IT, leading and implementing integration and CRM solutions across the world. Gary is passionate about delivering real business value through IT innovation.

AL LINES

QA MANAGER / SENIOR SOFTWARE ENGINEER ANALYTIC AND TECHNOLOGY SOLUTIONS GROUP MOODYS INVESTORS SERVICES

21st Century Test Management: A Retrospective How has test management changed now we are in the 21st century compared to testing of old.

Are we (still!) agile?

A light hearted view of how things have changed over the years for test managers and where this looks to be going, along some of the skills needed to “keep on testing”!

Al is still passionate about testing – even though he has been working in the field for almost 2 decades. He has been involved in testing systems of varying sizes, such as safety critical ones running the national railways, to a landmine…

Most of his work has been in the finance sector, in different asset classes, working on trading systems ensuring they are fit for purpose.

All of his testing can be summed up under the motto “Life is too short for poor quality wine”

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

MARK RICE

SENIOR BUSINESS TEST RELATIONSHIP OPERATIONS MANAGER MANAGER MERSEYSIDE VIRGIN MEDIA FIRE AND RESCUE SERVICE Pan European Test The Challenges Facing Transformation Localisation Testing Without the right focus, investment, strategy and What is software localisation? support, test automation How do fail we to testdeliver it? What projects the challengesresults. face the industry promised With today?

Despite beingbeing a the future of testing criticaltowards part of software driven cognitive, testing, Mark that predictive andargues intelligent significant challenges exist, test automation techniques, notcritical least the tasksthe of defining it’s to have right and standardising software foundations and culture in localisation testing. Mark place. presents his efforts to raise awareness of the challenges and then summarises the Heather overISTQB. 20 years’ responsehas of the test management experience in utilities, telecoms and Mark isentertainment. the Business Her digital Relationship for current role isManager Senior Test MerseysideManager Fire and Rescue Operations at Virgin Service. He has previously Media. worked in multiple roles for an international videogames company as QA manager and volunteers for the ISTQB. Mark has a PhD in psychology and is qualified in Advanced ISTQB (Test Manager/Agile), Scrum, ITIL, PRINCE2, TMMi, BRM and Six Sigma.

S P E A K E R S

KASHIF SALEEM DIRECTOR, QA HOTELS.COM

What is Agile Performance Testing?

Taking you through a journey talking about: Who is responsible for agile performance testing?

Where can the performance tests be run?

What performance tests to run?

How frequently should the performance tests be run?

How to automatically validate your performance test results?

What tools help you with this entire journey?

Cloud – enabler or disabler?

Kash started his QA career straight out of university in 2001 and has never looked back! He has worked in the defence, medical, education, psychology and travel industries for start-ups and multi-national corporations, covering all aspects of QA. He is currently Director, QA for Hotels.com, part of the worlds biggest travel company, Expedia Inc., where he is responsible for team management, functional/ non-functional testing processes and strategies. He still tries to find time to write performance scripts and review automation code!

57

MAHESH TANNEERU

LEAD DEVELOPER IN TEST REED BUSINESS INFORMATION

Getting Test Automation Right to Deliver Quality Software Over the past few years Test Automation has evolved as a big asset for software testing and it is not just appropriate for regression testing anymore. If we have a good test automation pack, it can help to improve lots of other things like team performance, sprint velocity, shorter release cycles, and provide confidence in releases.

It also gives testers the opportunity to spend more time on exploratory testing (where they can find more bugs) and teams can create quality products. Now we are in a position to automate most of the functional and non-functional aspects of applications with the different automation tools and technologies available.

Mahesh has 6 years of experience in software testing and currently works as QA Engineering Lead in Reed Business information.

KEITH YORKSTON

BILL WATSON

HEAD SQS ACADEMY

TEST MANAGER BBC

The problems with Risk and Risk-Based Testing

Most testers recognise the terms “Risk” and “Risk-based testing”. In a world of agile projects and tight deadlines, risk is becoming the go-to in terms of the testing and quality-related activities. But, in IT, do we really understand the nature of risk? Maybe…

With 20 years of IT experience, Keith has been involved with a wide range of project methodologies, industires and “levels of project quality”. Keith currently works for SQS as the Head of the SQS UK Academy, specialising in test management, preformance and security testing.

Outsourced software

An often overlooked area of software testing is the acceptance testing of outsourced software. The BBC primarily outsources its large IT systems and as such has a multi-million pound IT infrastructure spend; when these software and systems are delivered to the BBC we need to ensure it meets the needs of the business users who requested it.

This has traditionally been done by the business users (SMEs) having a test expert help them to create test suites mapped to their business requirements or even by the test expert writing the tests for the SMEs. The SMEs then run the tests on the software when it is delivered.

Bill Watson started in IT with a Dragon 32 computer in the early 1980s. Having a computer with a less than popular OS meant adapting his first programmes from a ZX Spectrum magazine and lengthy debugging sessions ensued. After diversions into a number of non-tech fields, he returned to the fold and now has over twenty years’ experience in the IT industry.

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

58

industry news

to talk

about BOOKMARK IT TODAY!

www.softwaretestingnews.co.uk T E S T M a g a z i n e | S e p t e m b e r 2 01 7

59

WELCOME TO THE 2017 EDITION

We hope that you’ll find this guide outlining different, selected software testing and quality assurance products and services useful. The software testing landscape changes rapidly and we find that an annual update on the marketplace is a good starting place as you consider purchase decisions going forward.

Sponsored by

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

60

2 0

L E A D I N G

T E S T I N G

P R O V I D E R S

2 0 1 7

Tata Consultancy Services (TCS) A part of the Tata group, India’s largest industrial conglomerate, Tata Consultancy Services Ltd (TCS) has over 385,000 of the world’s best-trained consultants in 46 countries.

The company generated consolidated revenues of US$17.58 billion for the year ended March 31, 2017.

TCS is an IT services, consulting and business solutions organisation that delivers real results to global businesses, ensuring a level of certainty no other firm can match. It offers a consulting-led, integrated portfolio of IT, BPS, infrastructure, engineering and quality engineering services.

TCS offers Quality Engineering & Transformation services (QET) across the testing value cycle, including test consulting and advisory, test services implementation and managed services for test environments and test data management.

TCS is a major employer in the UK and the rest of Europe. Its employees cover 50 locations and 350-plus customers across Europe, including 44 of the FT Europe Top 100.

TCS QET has been helping enterprises in the UK and the rest of Europe in their digital transformation journey, accelerating speedto-market, assuring superior end-customer experience, reducing cost of quality and mitigating risk. For this, TCS QET has invested in Digital Assurance capabilities, recent methodologies such as Agile, DevOps and CI/ CD, a world-class Customer Experience Center (CEC), QA Innovation Labs, and partnerships with leading product vendors.

Leading analyst firms – the Everest Group, Gartner, HfS, IDC, NelsonHall and Ovum – recognise TCS QET as a leader in the QA and testing space.

TCS QET Differentiators 1. Digital assurance capabilities: TCS QET has invested extensively in digital assurance capabilities, creating award-winning tools and accelerators that assure digital transformation of enterprises.

2. Next-gen QA offerings: Focusing on accelerated speed-to-market for enterprises and assuring superior end-user experience, TCS QET has built next-generation offerings and taken the lead in the market.

Independent industry analysts acknowledge this. A case in point is the AI-powered TCS 360 Degree Assurance, a solution that leverages advanced AI methods like machine learning, natural language processing, artificial neural networks and linear regression for problem analysis and root-cause identification, test suite

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

optimisation and defect prediction.

3. State-of-the-art labs: TCS QET has invested in the TCS Digital Transformation Studio and Customer Collaboration Center, San Francisco, the TCS Mobility CEC in India and the TCS Innovation Lab, Cincinnati, which showcase the latest devices, tools, infrastructure and accelerators.

4. Quality Engineering expertise: TCS QET’s decades of experience in Quality Engineering has attracted innovative and uniquely talented consultants with deep domain and QA knowledge and experience and enabled it to have direct positive impact on customer’s businesses.

Business heads across Europe have appreciated TCS’s contribution to the continent. 2016/17 Whitelane Research findings report: • 1,500 C-level executives from Europe’s top companies across 13 countries have given TCS the industry’s highest general satisfaction rating (79%), ahead of 24 other top IT services companies • For the fourth consecutive time, TCS has emerged ahead of its competitors: − Scored 7%-plus, on average, in all 11

KPIs polled − #1 in 6 KPIs, including delivery,

account management, proactivity,

trustworthiness, change

management and business value − #1 in application services and

infrastructure services; retained pole

position in the UK, the Nordics,

Germany and The Netherlands • TCS is #1 in UK, Germany, Austria, Switzerland, The Netherlands and the Nordic region • TCS is one of the top three in France TCS partners with over 200 schools and academic institutions to develop cutting-edge digital skills benefitting over 51,000 students.

Recent Awards and Recognitions • The world’s most powerful brand in IT Services (Brand Finance, the world’s leading brand valuation firm) • The top employer in North America, Europe, UK, India, Latin America and Australia for five years running, till 2017 • Britain’s Top Employer for five consecutive years, till 2017 (The Top Employers Institute, Amsterdam) • A Business Superbrand for 2015-17 • One of UK’s leading employers for women (2016 Times Top 50 Employers for Women) • Winner of the Best Education Project for its IT Futures initiative (The 2016 National CSR Awards)

+44 020 3155 2100 uk.info@tcs.com www.tcs.com TCS office 5th Floor, 2 Derry Street London W8 5TT United Kingdom

www.tcs.com • Winner of the Social Responsibility award for its IT Futures initiative (The 2016 UK Employee Engagement Awards) Commitment to the Community The premium TCS places on commitment to the community may be gauged by the following: CSR Initiatives: TCS invests millions of pounds every year in corporate social responsibility. It supports over 40 charities across Europe. TCS employees volunteer 100,000-plus hours annually, supporting social causes and community projects.

TCS IT Futures: TCS’s multi award winning community engagement programme and a part of the TCS Purpose4Life initiative, IT Futures, aims to inspire a generation of young people to work at the forefront of technological change.

Since 2013, TCS has dedicated over 8,000 hours to this initiative, engaging over 170,000 students aged 7-25 years.

TCS Eco Futures: This initiative, focusing currently on energy efficiency, waste, recycling and business travel, engages TCS employees across UK to reduce their environmental footprint.

2 0

L E A D I N G

T E S T I N G

P R O V I D E R S

2 0 1 7

61

QASource +1 925 271 5555 info@qasource.com 73 Ray Street Pleasanton

California 94566 USA

www.qasource.com

The Leader in Pure Play Software QA and Testing By offering innovation, strategies and processes that eliminate inefficiencies and produce quality software products — combined with test automation expertise, industry-leading visibility, strategic planning, and metrics — QASource makes the decision to outsource easy. We invite you to learn more at qasource.com.

WHAT is QASource?

Headquartered in Silicon Valley, and with offices in Mexico and India, QASource is one of the world’s leading software testing providers: • QASource is an organization of 700 professionals servicing clients worldwide, ranging from VC-funded startups to Fortune 500’s such as IBM, Oracle, and Cisco • We deliver international-standard quality, strict adherence to deadlines, and total satisfaction WHO is QASource?

Our highly skilled engineers take your quality assurance from the initial design phase to completion, providing quality assurance every step on the way: • They are empowered to make product and procedural recommendations to clients and serve them at the highest level. Testers are encouraged to focus on creative solutions and challenge the status quo • They are hired for their depth of test and

software expertise, and are continually trained in new testing methodologies, effective communication, and best-practice processes • Our CEO is personally involved with every one of our partners What can we do for YOU? We focus exclusively on how we can best serve our partners, and only execute on strategies and tactics that drive value: • We can extend your QA department in less than 24 hours, without the associated setup and hassle • We are relationship-oriented — so much so, that we boast a 94% client retention rate over 15 years • With an emphasis on time-bound delivery and customised solutions, we help our partners manage the quality of their deliverables while keeping costs low • Our 91% employee retention rate means our engineers provide clients with stability and continuity, and therefore an institutional depth of knowledge not found in most engineering teams • Our hybrid “onsite-offshore-nearshore” service model combines with U.S. management — enabling organisations to mitigate the risks that can accompany a remote testing team • Our portfolio of products includes our core QASource offerings; MyCrowd, a crowdtesting option; and QAOnDemand, a pay-as-you-go model

Amdocs With over 30 years’ experience in the communications and media industry, Amdocs has a proven track record of ensuring high quality software, network and entertainment deployments. As a leading testing provider to the world’s largest service providers, Amdocs’ team of testing experts ensure go-live success, industry-low defects and cost-efficient production.

Amdocs’ innovative testing approach includes a patented, award-winning framework, Amdocs BEAT™, which standardises, optimizes and automates testing processes. Using a sophisticated analytical model, repository of best practices and professional know-how, Amdocs BEAT™ ensures testing excellence, enabling service providers to deliver superior customer experiences.

Our in-house tools such as a no-code automation platform and an automated, graphical test design console allows Amdocs Testing to automate the entire testing lifecycle for reduced testing effort, improved test case reusability and faster time to market.

We have shifted our culture to a DevOps mindset, strengthening Amdocs Testing’s consistently reliable methodologies, exceptional

technology and professional practical experience.

By transforming project deployment processes from agile development to agile-to-production, Amdocs evolves cross-vendor delivery processes into one that is test-driven and automated.

In support of our customers’ digital transformation, Amdocs’ shift-left testing offers agile/DevOps methodologies, enabling service providers to delivery key features faster and achieve value and revenue at the speed of digital.

Amdocs Testing recently introduced Amdocs Digital User Experience Testing (DUET), a state-ofart, unique offering that certifies back- and frontend systems, as well as other channels covering all the aspects required for seamless and flawless digital user experiences.

Amdocs is a leading software and services provider to the world’s most successful communications and media companies. As our customers reinvent themselves, we enable their digital and network transformation through innovative solutions, delivery expertise and intelligent operations. Amdocs and its 25,000 employees serve customers in over 85 countries.

Listed on the NASDAQ Global Select Market, Amdocs had revenue of US$3.7 billion in fiscal 2016.

+1 314 212 7000 testing@amdocs.com Missouri 1390 Timberlake

Manor Parkway

Chesterfield, MO 63017 USA

www.amdocs.com

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

2 0

62

L E A D I N G

T E S T I N G

P R O V I D E R S

2 0 1 7

DVT 

Bruce Zaayman +44 (0) 203 696 2440 bzaayman@dvtsoftware.co.uk Dawson House 5 Jewry Street,London EC3N 2EX United Kingdom

www.dvtsoftware.co.uk

DVT, founded in 1999, is a global software and solutions company providing bespoke business software solutions for some of the largest and most dynamic organisations.

Our core focus is delivering value-added automated regression testing, functional and performance testing and agile software development services that help our clients get their solutions to market faster and at a lower cost.

We employ more than 600 staff globally with our UK office situated in Jewry Street, London. Additionally, DVT’s Global Testing Centre (GTC) offers heavyweight testing solutions, backed up by 17 years’ experience and high-level expertise.

With multiple engagement options and a full suite of outsourced and onsite testing and development services to choose from, our clients are guaranteed a solution that best suits their needs, time frames, budgets and strategies.

Based on proven agile methodologies and frameworks, our software testing services include: • Test automation for web, desktop, mobile, web services and mainframe software solutions.

• Performance testing as a certified

Neoload partner using Neotys to test the performance of web, mobile and web service-based solutions.

• Functional testing is performed when required in order to validate the behaviour of an application.

• Mobile testing with an extensive mobile device lab and our framework to automate the testing of mobile apps.

• Manual testing of mobile, web, desktop and gaming software.

All our clients, at every level of engagement, get access to a dedicated management team and permanent project staff, using our home grown open-source testing and development framework – although we are able to use any third-party licensed framework where it is appropriate and available. Clients also get access to our extensive mobile devices lab, ensuring their software is tested on multiple current and legacy devices without the burden or cost of sourcing these devices independently.

Many of our engagements are based on a rapid proof-of-concept at no initial cost, ensuring only the right fit is found for your project before any longer-term commitment is made.

froglogic froglogic is a software company based in Hamburg, Germany with additional offices in the USA and Poland with over 3,000 customers worldwide.

Bamboo, TeamCity, HP ALM, MS ALM and more, froglogic’s customers save tremendous amounts of time testing their GUI and HMI applications.

froglogic is well known for its automated testing suite Squish with its flagship product Squish GUI Tester and the code coverage analysis tool Squish Coco.

Squish Coco is a professional, cross-platform C, C++, C#, QML and Tcl code coverage tool which allows to analyze how much of an application’s source code is being tested, whether there are redundant tests and which tests need to be added.

Squish GUI Tester is the market-leading, cross-platform automated testing tool for GUI applications based on Qt, QML and QtQuick, Java AWT/Swing, SWT/RCP and JavaFX, macOS Carbon/ Cocoa, Windows MFC, .NET and WPF, iOS, Android and for Web and HTML5 applications running in a variety of web browsers.

The Squish GUI Tester is an object based GUI test automation tool and works on desktop PCs, mobile and embedded devices running Windows, Linux, macOS, Solaris, iOS, Android and several embedded platforms such as embedded Linux, QNX and WinCE. Due to Squish GUI Tester’s advanced features including powerful scripting, remote testing, robust object and image verifications and integrations into frameworks such as Jenkins,

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

+49 40 78897137 sales@froglogic.com Gasstr. 18

Haus 1

22761 Hamburg Germany

Squish Coco works with several compiler tool chains and supports all coverage levels from basic line and branch coverage up to MC/DC and MCC. Advanced reporting such as comparing executions, patch analysis and automatic bug location detection makes Squish Coco a unique tool.

Squish Coco can be easily integrated with all kinds of test frameworks such as GoogleTest, CppUnit, NUnit, Jenkins, SonarCube and more.

Both, Squish GUI Tester and Squish Coco, are fit for use in safety critical environments complying to IEC 61508, ISO 26262, EN 50128 or DO 178C. To find out more about froglogic and its products, please get in touch!

www.froglogic.com

2 0

L E A D I N G

T E S T I N G

P R O V I D E R S

2 0 1 7

63

INDIUM +1 646 583 0243 info@indiumsoft.com Suite 210, 1250 Oakmead

Parkway Sunnyvale

CA ­94085 USA

www.indiumsoft.com

Indium Software is a Global Independent Software Testing Services Company with a multi-domain focus, fostered by IP-led innovation.

Highlights: • 17+ years in software testing • 6 global locations in 3 continents • 4 test accelerators • 240+ happy clients from Fortune 100 to start-ups • 450+ career testers Indium offers software testing to both Enterprise and ISV clients across US, Europe, Asia Pacific, South Africa and India. There is a unique way through which our team integrates itself into the culture of our client's IT or QA teams and strives to Assure our client's Business through Quality Assurance.

Indium's objective to servicing our global clients can be explained with this simple line: "We're small enough to care, large enough to deliver". To add credence to our motto, Indium was featured in Top Software Application Testing Firms-2017 by Clutch.co, an independent research agency in the USA. Adding to our pride, Indium was featured as "Pure-Play Multi-Domain Testing Services Firm" in Gartner Vendor Guide.

Key strategy Critical to Indium's strategy for accelerated testing is its frameworks, which are branded under iAccelerate suite of frameworks & tools.

In the past decade, these frameworks (both functional and non-functional) have evolved drastically to improve the quality. They are built based on best practices delivered over many years to customers across various industries that include retail/eCommerce, gaming, media, banking, healthcare, education and life sciences. All these clients have one objective in common, which is a need for the reliable and robust software they deliver to their user base.

This is what Indium brings to the table.

Service lines Functional Testing, Compatibility Testing, Mobile Apps Testing, Test Automation, Game Testing, Performance/Load Testing, Security Testing, Compliance Testing and Test Advisory Services.

Game Testing Division Indium unlocks the value by rebranding Game Testing into a separate division called iXie Gaming. iXie has experience in testing more than 150 titles covering various genres from pre-alpha phase to release candidates. iXie's Game Testing competency includes PC/Web Games, Mobile Games, Console Games and Handheld Games.

Keytorc Software Testing Services “Tested by Keytorc®” Since 2005, Keytorc Software Testing Services has been assisting its clients to manage their critical software testing processes to reduce the total cost of producing high quality systems. The company has the best references from many domains as; banking, insurance, telecommunications, defense, IT-High-Tech, and e-commerce.

Besides being the regional leading testing services company, Keytorc also provides international software trainings by being an ISTQB Accredited Training Provider®.

Innovations and Achievements After the establishment of its R&D centre in the largest technology campus in Istanbul, Keytorc has started providing significant improvements and innovations in software testing area, especially on test management, test automation and performance testing. Very recently, Keytorc has opened its 3rd branch in Brussels, Belgium.

Keytorc R&D Team has developed and initiated “Test Capability Rating® Model (TCR)” which brings new perspectives and insights for test process improvement.

The framework constitutes of 4 main areas: Test Process, Test Organisation, Test Technology and Test Related Streams. TCR® has a proven record in improving test capabilities for a broad target audience.

Keytorc has brought many innovations on test automation technologies, which mainly result in improvements on test coverage, test effectiveness and productivity. These advancements are ensured by

constituting solid integrations between open-source and commercial test automation solutions.

Keytorc also has valuable achievements in reputable international testing competitions and events. The company holds, attends and supports prestigious testing events and conferences in national and international scale.

Keytorc is the organiser of the first international online and offline combined software test automation hackathon: TestHackathon. The event has an international Judging Panel in which there exist experts from various countries and domains and the contest offers many benefits and prizes to test automation people from all around the world.

Services Provided by Keytorc: • Test Outsourcing • Test Automation • Performance Testing • Mobile Application Testing • Test Process Improvement & Test Maturity Assessment • Testing Center of Excellence (TCoE), Test Management • Test Data Management • ISTQB® Accredited Software Testing Trainings Overview of Places, Projects and Clients • Services provided in 4 Continents • Projects in 25 Countries • More than 800 Clients

+90 212 290 76 60 info@keytorc.com Giz 2000 Plaza, Maslak-Istanbul, 34398 Turkey

www.keytorc.com

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

2 0

64

L E A D I N G

T E S T I N G

P R O V I D E R S

2 0 1 7

Maveric Systems +91 44 4344 2500 info@maveric-systems.com Lords Tower, Block 1 2nd Floor, Plot No. 1 & 2

NP, Jawaharlal Nehru Road

Thiru Vi Ka Industrial Estate Ekkaduthangal

Chennai - 600 032, India

www.maveric-systems.com

Maveric Systems is a leading Technology Assurance Provider offering integrated assurance services to global banking leaders across UK, Europe, Middle East, Mexico, US, and Singapore. Our services are aimed at eliminating quality, cost, and time-to-market risks associated with large IT transformation programs across key banking and telecom domains including retail, corporate, transaction banking, payments, multi-channel, wealth management, and regulatory & compliance.

To enable customers to succeed in the digital services landscape, we provide a wide spectrum of services including application development across web and mobile (Greenfield and existing platforms), automated QA, UI/UX, and Omni channel. Our data services comprising of services in Data Migration/ ETL, Data Management (including Test Data Management and Synthetic Data Creation), Data Masking, Data Cleansing, and Database Virtualisation help manage and analyse data to bring about superior customer experience and business differentiation. We also support our clients’ shift to an Agile/DevOps model through Agile coaches, scrum masters, product owners, developers, and testers.

We have invested in offering platform-led strategic solutions across technologies like digital, data, API, middleware, virtualisation, and automation, along with capabilities across leading industry as well as open tools such as Appium, CA Lisa, Eclipse, JIRA Portfolio, Bamboo, Docker, HP LoadRunner, Maven, Splunk, Apache, Sonar, etc. Maveric also has to its credit multiple engagements around a large number of banking and telecom products including Temenos T24, Flex Cube, Misys Equation, Finacle, Eximbills, Fidessa, Actimize, TIBCO, etc.

Maveric’s expertise has been recognised across various international awards, the latest among which include Best Automation (Functional) Award at the 2016 TESTA Awards, and the Banker Middle East Industry Awards for the ‘Best Banking Technology Partner’ for 5 years in a row from 2013 - 2017.

We have a dedicated offshore delivery and research center in Chennai, India. Our 1000+ assurance professionals operate across centers in UK, US, Europe, Middle East, APAC and, Mexico, and provide services to more than 50 financial services organisations across the globe.

Mobile Labs At Mobile Labs, we know that mobility can be a challenge throughout the enterprise. For software developers and testers in particular, the growing complexity of mobility and its impact on mobile app testing is caused by many factors. These factors include a growing variety of device types, form factors, operating systems with compressed release, along with more sophisticated mobile app users. Managing all of these factors while rapidly delivering quality, reliable apps doesn’t have to be chaotic. Mobile Labs helps bring order to the chaos with enterprisegrade, end-to-end mobile testing solutions.

Our mobile app testing solutions are secure, agile, scalable and available at a fraction of the cost of public cloud and first – generation test tools.

We know that mobile security is a serious concern for the mobile-first enterprise. Yet, many organisations are considering using devices in a public cloud for their mobile application and mobile website testing. That’s why Mobile Labs offers deviceConnect™, a private mobile device cloud that provides secure, comprehensive management of mobile devices, mobile apps, tester access

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

and tester activity. Designed for enterprise testing teams, deviceConnect's powerful management console enables remote management, tracking and monitoring of enterprise testing assets by individual device name, manufacturer, model number,

OS version, availability and apps currently loaded for test. The solution's patented technology lets testers – regardless of location – remotely access and share AndroidTM and Apple® iOS smart phones and tablets. In addition, deviceConnect facilitates both manual and automated mobile app testing. Unlike public device clouds, deviceConnect is highly secure and uses a very light hardware footprint. Its installation behind the corporate firewall makes it easy to install and manage, while protecting the security and integrity of private information that may be part of enterprise application data.

Welcome to the future of mobile application testing, mobile website testing, and mobile device access management. It’s cost-effective.

It’s accurate. It’s secure. For more information please visit www.mobilelabsinc.com

404-214-5804 info@mobilelabsinc.com

www.mobilelabsinc.com

2 0

L E A D I N G

T E S T I N G

P R O V I D E R S

nFocus Testing +44 (0) 370 242 6235 info@nfocus.co.uk e-Innovation Centre,

Shifnal Road, Telford,

Shropshire, TF2 9FT

United Kingdom

www.nfocus.co.uk

As the longest established independent testing consultancy in the UK, and winners of the prestigious Leading Vendor award at the European Software Testing Awards 2016, nFocus are ideally placed to provide your software testing services.

Read our 17 year history and evolution of nFocus infographic on page 69 of the magazine.

By winning the Leading Vendor award, we beat twelve of the world’s largest Service Integrators and pure play Test Consultancies. The success criteria the award was judged on: • Evidence of value for money • Meeting and delivering against business objectives • Evidence of a commitment to high quality and standards to customers • Commitment to customer satisfaction • Evidence of reliability, flexibility and speed of installation • Innovation and being a thought leader Cecilia Rehn, Chair of the judging panel and former Editor of Test Magazine said, "Congratulations to nFocus Testing for winning The Leading Vendor category. The judging panel was in agreement that this entry stood out from the rest by demonstrating thought leadership, innovative DevOps practices, as well as by promoting best practices for the future.”

2 0 1 7

65

nFocus specialise in: • Fast, reliable and strategic resourcing of specialist testing roles • Testing Healthchecks • Functional Testing • Test Automation • Agile and DevOps • Performance Testing • Mobile Testing As well as our core specialisms, we understand that quality delivery is not just about testing and encompasses roles across the whole delivery lifecycle.

We can provide roles from Developers to Programme Managers all with a track record of getting quality right.

Experts in our field: Specialist quality and testing consultancy for over 17 years managing a large team of UK based consultants.

Vast domain experience including technology, media, financial services, public sector, retail, construction, law, utilities/energy, healthcare, insurance and leisure.

Multi-award winning testing consultancy including Microsoft’s prestigious 2014 Worldwide ALM Partner of the Year for delivering services across the entire application lifecycle.

To discuss your situation or requirement with one of our team, or to find out more about nFocus, please contact us using the details provided.

Panaya Bringing the Promise of Agility to Enterprise IT Panaya is the leader in continuous delivery of innovation to the world of enterprise IT applications. With Panaya, IT and Business leaders can accelerate and promote change so that enterprise applications meet the rising pace of today’s business.

We provide a cloud based Enterprise Agile Delivery platform that includes test management, test automation and application lifecycle management solutions.

Ensuring visibility and collaboration between Business and IT, Panaya enables faster release velocity while ensuring quality.

Enabling the Agile Enterprise • Enterprise Agile Delivery Platform - Panaya provides a unique process centric approach to application life cycle management with Panaya Release Dynamix (RDx), a cloudbased enterprise agile delivery platform.

From initial business change request through to testing and production, Panaya delivers the value of agile, continuous delivery to the world of enterprise applications.

• End-to-End Testing - Panaya Test Center enables a process centric approach to test management delivering real time visibility into the status of requirements as they are being delivered. Powered by Machine Learning technologies Panaya Autonomous Testing provides automation and acceleration of regression testing and user acceptance testing.

• Delivery Impact and Risk Mitigation Panaya utilises real-time code analysis of your SAP, Oracle and SFDC applications and provides actionable insights into all other impacted transactions to ensure those impacted areas have test coverage and to predict the full impact of change, before changes are made.



Ronit Eliav Director of Product Marketing +972 9 7618000 reliav@panaya.com

Customers Panaya customer base spans over 1600 enterprises worldwide, including over 220 of the Fortune 500 companies. There are more than 300 Panaya Test Center customers and over 50 ALM replacements across various markets including Utilities, Oil & Gas, Automotive, Manufacturing, Pharmaceuticals, Infrastructure and Services.

www.panaya.com

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

66

2 0

L E A D I N G

T E S T I N G

P R O V I D E R S

2 0 1 7

Plutora +1 408 601 4792 sales@plutora.com 3945 Freedom Circle

Suite 730, Santa Clara

CA 95054

Plutora is the market leader in enterprise IT software delivery, encompassing release, environment and test management solutions.

The Plutora Platform transforms IT release processes by correlating data from existing toolchains and automating manual processes providing a single view of releases and associated metrics, such as testing quality. This results in predictability in the software release process, improving the speed and frequency of releases and better aligning IT software development with business strategy. Plutora has helped the world’s largest companies and Fortune 500 brands, including eBay, Merck and Dell.

Plutora provides visibility and control to application delivery which has suffered due to increasing complexity in the IT landscape including: geographically dispersed teams, adoption of SaaS-based services and the emergence of agile, DevOps processes.

www.plutora.com

We Help Enterprises Succeed Plutora uses a single repository that includes information from various development and operations systems. By incorporating information from tools such as application release automation, IT service management and project and portfolio management, we are able to

provide insight, analytics and reporting across the entire development process and the complete release portfolio. The resulting single source of truth gives IT executives full visibility into every aspect of release and provides collaboration across all associated development, operations and testing teams.

Unparalleled Views and Collaboration A key component of the Plutora Platform is Plutora Test, a modern, enterprise test management tool that supports the complete software testing process across all types of development approaches from Continuous Delivery to traditional waterfall approaches.

Testing design, execution, defect tracking and progress reporting are all supported and offer collaboration every step of the way. Integration is provided for all related tools and systems, such as JIRA and Selenium. Test progress is reported back into the overall release dashboards providing the critical quality information required.

It's Never Been Easier to Switch Organisations using legacy systems can easily upgrade to our Saas Platform in just a few clicks using the Bulk Upload feature in Plutora Test.

This allows users to easily incorporate existing test requirements and use cases.

QMetry QMetry Digital Quality Platform is designed for Agile and DevOps teams to build, manage and deploy quality software faster and better. QMetry is complete software testing platform with test management, automation and powerful quality analytics for digital enterprises. It is trusted by 600+ customers globally across Finance, Healthcare, ODMs, Hospitality, Retail, and High-tech verticals.

QMetry is made up of following independent yet fully integrated products.

1. QMetry Test Management is a flagship Cloud / On-Premise Test Management product for enterprises to manage the Quality Lifecycle. It is used by leading Fortune 1000 customers and it has evolved into a contemporary agile ready test management tool of choice.

2. QMetry Test Management for JIRA is the best test management add-on for JIRA in Atlassian Marketplace.

It enables QA teams to get involved earlier in development cycle in Agile environment. Manage manual and automated test cases, execute and analyse.

3. QMetry Automation Studio is the unified test automation product for web, mobile & web services with BDD/TDD support. Using Selenium and Appium for scripting; it adds structure, efficiency, and reusability

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

to automated testing. It supports coded/scripted automation and enables manual teams to transition into automated testing seamlessly with script-less automation.

4. QMetry Wisdom is the pioneering AI and Machine Learning based, predictive analytics product for QA teams. It highlights major bottlenecks, major errors by categories and productivity metrics through well-defined reporting dashboards. Through AI and machine learning, Wisdom self-discovers and provides smart suggestions to improve the software quality.

+1 408 727 1101 marketing@qmetry.com 3200 Patrick Henry Drive

Suite 250 Santa Clara

CA 95054 USA

5. QMetry Voyager is the full feature and free exploratory testing product. The Screen Capture software allows to add assertions, annotations and voice memos to recorded screen captures for your Web Apps. These can then be linked to test cases in JIRA. QMetry has strategic integrations with leading CI/CD platforms Jenkins (QMetry for Jenkins) and Bamboo (QMetry for Bamboo).

QMetry is the best platform for QA teams covering end to end needs of managing test projects, exploratory testing and test automation. It adds a unique Analytics layer through Wisdom. It is a scalable and future ready digital quality platform.

www.qmetry.com

2 0

L E A D I N G

T E S T I N G

P R O V I D E R S

2 0 1 7

67

Ranorex +43 316 281328-300 info@ranorex.com Strassganger Strasse 289

8053 Graz Austria

www.ranorex.com

Test automation for desktop, web and mobile applications What kind of application do you want to test?

Is it installed on a Windows desktop? Does it run in a browser? Is it used on a smartphone or tablet? It does not matter which platform your software is developed for. You only need one Ranorex license to test any type of mobile, desktop or web application.

The powerful UI object recognition, strong technology support, broad variety of flexible tools and quick ROI make it the ideal choice for any team and development environment, whether you are using a traditional or agile approach. This is why over 3000 companies worldwide trust in the award-winning Ranorex GUI test automation software.

The ideal software for your team and development environment Finding a test automation software that not only fits your development environment and budget, but also supports working in teams can be tricky. This is where Ranorex excels. You can seamlessly integrate it into your development environment, using continuous integration and version control systems, enabling you to generate quick test automation results. At the same time,

Ranorex offers tools that suit different skills in a team. While you do not need any programming skills to create robust and easily maintainable automated tests with the Ranorex Recorder, test automation experts and developers can add further functionalities, or create new test automation projects entirely in C# and VB.NET. Committed to excellent test automation software The Ranorex test automation software is continuously evolving to provide innovative solutions to help you deal with new testing requirements. Recently, it has released its Selenium WebDriver integration, enabling teams to address common Selenium pain points with the powerful Ranorex automation framework, as well as create web tests based on the Selenium WebDriver standard.

All major and minor software updates are already included in a license at no additional costs. Whenever you do need help with test automation, Ranorex will provide you with the in-depth answers you are looking for – whether it is with the outstanding Ranorex support team, monitored online forum, comprehensive user guide, free webinars or training courses.

ROQ ROQ delivers strategic software testing solutions that make a genuine difference to our clients. We’re trusted to get the job done and support world-leading companies in realising the business benefits of high-functioning, high-quality software applications. Our mantra is ‘Making Software Work’ and this permeates through all aspects of our engagements and UK operations.

Our client focus and independence reduces surprises, provides new perspectives, and increases the likelihood of programme success. Collaboration, responsiveness, flexibility and delivering real value to our clients are core to our business model and key to our continued success and the success of our clients.

A Flexible and Responsive Approach Recognising the need for a more flexible and responsive approach to software testing that improved upon traditional service delivery models; ROQ established its UK-based Test Lab, creating a market leading environment and knowledge centre to deliver innovative testing services. Our

central UK location means we are ideally located for easy engagement and often incorporate an element of on-site support to facilitate stakeholder management.

Our resourcing models provide the flexibility to quickly ramp up/down resources to allow for cost effective and timely response to peaks and troughs in demand. Core team members remain in place to ensure continuity of knowledge of the project. In addition, tightly defined processes allow for prompt knowledge transfer and upskilling of additional new team members as the team grows or if specialists are brought on-board for a defined time period.

Industry Recognition and Accreditation ROQ is proud to be have recognised as the only UK-based, pure-play provider in both Gartner’s ‘Mobile Application Testing Vendor Landscape for 2017’ and Forrester’s ‘DevOps, Continuous Development, and Continuous Integration in 2017’. ROQ is also accredited as a service provider on G-Cloud 9, Digital Outcomes & Specialists II and the Home Office QA & Testing Framework.

01257 208890 ask@roq.co.uk Assurance House East Terrace, Euxton Lane Euxton, Chorley PR7 6TE United Kingdom

www.roq.co.uk

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

2 0

68

L E A D I N G

T E S T I N G

P R O V I D E R S

2 0 1 7

Soflab +48 22 211 26 02 soflab@soflab.pl 103 Ligocka Katowice 40-568 Poland

www.soflab.pl

Soflab Technology is the largest testing and QA company in Poland. With offices around major cities in Poland, our staff of over 200 employees has completed over 4000_different projects across Europe in the last 9 years. Recently, our projects have also expanded to the EMEA region as well as East Asia. Our services include but are not limited to:

As an independent company with no affiliation with any IT system suppliers, our work is objective, focusing on the success of our client’s requirements. We support our clients in ensuring the reliability of implemented software and maintaining business continuity. We provide accurate reports on the quality of inspected IT solutions.

• • • • • • •

Our clientele consists mainly of international telcos, banks, insurance companies, media, public entities, and IT integrators. Soflab is the first company in Poland that has won and implemented two large-scale full testing and QA SLA-based outsourcing projects (for an international telco group and a major national public entity).

Functional software testing Performance testing Security testing Mobile application testing Test automation Audits, advisory, quality management Test tools, data, and test environments

Our employees hold over 150 ISQTB certificates as well as several other IT-certified skill sets. This includes QAMP, Prince2, AgilePM, Professional Scrum Master, SAFe Consultant, and ScaledAgile.

Our experts are actively involved in QA events around Europe, implementing best practices in our projects in Poland and abroad.

Offering an extensive range of high quality of services with a high level of technical and analytical skills, freedom of movement within the EU, and a customer focused approach at highly competitive rates, it is no surprise why we are identified as a gazelle company in Poland.

Sogeti Sogeti is a leading provider of technology and engineering services, and part of the Capgemini Group. Globally, we deliver solutions that enable digital transformation, and with 25,000 professionals in 15 countries we are able to provide cutting-edge expertise in cloud, cybersecurity, digital manufacturing, digital assurance & testing, and emerging technologies.

Sogeti combines agility and speed of implementation with strong technology supplier partnerships, world class methodologies and a global delivery model, Rightshore®.

Our UK portfolio Businesses need to adopt new digital practices, and innovate faster to meet the demands of today’s customers. By partnering with Sogeti, we can help you to fast-track new opportunities and deliver quantifiable business value. In the UK, Sogeti’s portfolio covers a range of solutions including DevOps, digital assurance & testing, security, cloud, and consulting services. Each of these offerings can be delivered individually, or as part of a complete solution or managed service.

Flexible pricing and thought leadership We continue to evolve and innovate by creating new frameworks and accelerators and T E S T M a g a z i n e | S e p t e m b e r 2 01 7

updating our methodologies that are widely used across the industry (TMAP®, and TPI® and PointZERO®). We are able to offer a range of commercial models, from time and materials to outcome-based agreements. Our Testing as a Service offering is available, within a fixed price model, for many of our specialist services including automation, performance, and mobile testing.

Analyst recognition Sogeti, together with the Capgemini Group, is recognised as a leader in our industry by a number of industry analysts: • Positioned as a leader by Gartner in the Magic Quadrant for Application Testing Services Worldwide in November 2016 • Recognised as a leading player in NelsonHall’s Vendor Evaluation and Assessment Tool (NEAT) in both the Consultancy Focus and Transformation Focus market segments in July 2016 • We are listed as the #2 provider, in both the UK and EMEA software testing markets, based on revenue and market share according to NelsonHall’s report: ‘Transforming Software Testing Services Through DevOps and Digital Automation’ dated July 2016

+44 (0)330 588 8200 enquiries.uk@uk.sogeti.com 85 London Wall

London EC2M 7AD

United Kingdom

uk.sogeti.com

2 0

L E A D I N G

T E S T I N G

P R O V I D E R S

69

2 0 1 7

17 YEARS OF SOFTWARE TESTING

2017

Named in Test Magazine’s 20 Leading Testing Providers nFocus expand with new Norwich office

nFocus named a Microsoft Gold DevOps Partner

150

Customers to date

New services added for Digital, Big Data, Security, IoT, Social, Media, Analytics and Cloud nFocus awarded Crown Commerical Supplier Status

nFocus win Leading Vendor Award at the European Software Testing Awards and finalist for another award

2016

Triple finalists at the DevelopHER awards. Started offering DevOps

nFocus win Best Small-Medium Business & Best Marketing at the Best Business Awards

2015 120

2014

2013

Added to the Government’s G-Cloud Digital Framework

nFocus named Worldwide Application Lifecycle Management (ALM) Partner of the Year

Offering Agile services

2011

nFocus first test consultancy awarded Microsoft’s Gold Application Lifecycle Management partner status

2008

One of the first 6 companies in the UK to gain certification in the Accredit UK Std

2004

In the Sunday Times top 30 growing companies

@nfocus_ltd

Technical Director, Danny Crone, recognised with ALM Ranger status

80

2010 2007

25

1

2000

nfocus-ltd

Started offering test management, test analysis & test automation

West Midlands ICT Excellence Award

Offering functional testing and automation in the tech industry

= Customers

launch:

17 YEARS OF SOFTWARE TESTING Whatever your objectives, we have the breadth of knowledge and the depth of expertise to deliver the tailored IT quality assurance services and support your organisation demands, both today and for the future.

0370 242 6235

TESTING HEALTHCHECKS

AGILE TESTING

FUNCTIONAL TESTING

PERFORMANCE TESTING

TEST AUTOMATION

WWW.NFOCUS.CO.UK

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

INFO@NFOCUS.CO.UK

2 0

70

L E A D I N G

T E S T I N G

P R O V I D E R S

2 0 1 7

Sii +48 22 486 37 37 contact@pl.sii.eu al. Niepodległości 69

02-626 Warszawa

3rd floor,

Metron building Poland

www.sii.pl/en

Employing 3000 specialists, Sii is the leading provider of IT and engineering services in Poland, delivering solutions for business partners from all around the world. The company cooperates with more than 200 clients – from Canada, USA, Japan and Western Europe – including Great Britain. Sii specialists carry out projects for leading companies from: financial services, banking & insurance, hi-tech, electronics, telco, OEM, retail, consumer products, transportation & logistics, manufacturing, aerospace, media, communication, e-commerce and other sectors. Among our clients you will find i.a. Dollar Financial Group, Thyssen Krupp, Fresenius, Scalepoint, Berlingske Media, Widex, Sabre, T-Mobile and many more.

As an offshoring leader in Central Europe Sii has profound experience in international cooperation. It is a trusted nearshoring partner, supporting companies in the area of consulting, analysis and tests as well as software development, infrastructure management, integration, system maintenance and engineering. Projects are carried out based on the Lean and Agile methodology, which guarantees flexibility and a full compliance of the service with the Customer’s needs. Sii is first and foremost a technological expert, whose technical skills focus around 11 Competency Centers: Testing,

Digital, Application Services, AX, SharePoint, CRM, Business Intelligence, Salesforce, Engineering, Trainings and IT Operations. These are specialized units within which the most experienced specialists provide their expertise, develop their own skills and share knowledge.

Sii has the biggest testing resources in Poland and its own mobile testing lab. The Sii team counts as many as 600 testers, most of whom hold professional certificates. The core of Sii’s testing is the Testing Competency Center, which carries out project for nearly 100 clients (almost a half of them for foreign companies). For 11 years, Sii has gained the complex knowledge necessary to help clients and support them in optimising testing costs through the implementation and execution of automated tests, selecting the best testing tools, implementing efficient processes as well as the right choice of test teams and adjusting the salaries and testers’ rates to market conditions. Additionally the reliability and care for adhering to the most strict international norms are confirmed by the Common Criteria safety certificate. Sii has its 8 branches in: Warsaw, Cracow, Wroclaw, Gdansk, Poznan, Lublin, Lodz, Katowice. Read more www.sii.pl/en. For more information, please contact our Business Manager: Szymon Wroblak, swroblak@pl.sii.eu.

Vornex Vornex is a provider of enterprise software testing solutions. Our flagship product, TimeShiftX is a date and time simulation software that lets you time travel software into the future or past for temporal testing or date shift testing to validate all date and time sensitive functionality and code such as yearend, daylight savings, leap year, billing, rates, policies, etc.

• Cross Platform & Cloud - Container Time Travel TimeShiftX is compatible on all platforms & operating systems and can be run in the cloud and inside containers • Distributed Environment Time Shifting TimeShiftX allows you to easily temporal test large, distributed software stacks and environments

TimeShiftX enables time travel (even inside Active Directory & Kerberos) without code changes, manual work, or server isolation in order to perform your forward date or back date testing with features including: • Instant Time Travel No code changes, no altering server clocks, no isolating servers. Just turn on TimeShiftX & start time traveling.

• Active Directory Compatibility Safely time travel inside Active Directory, Kerberos, LDAP, and other domain authentication protocols.

• Total App & DB Compatibility TimeShiftX enables time travel for all applications & databases such as SAP, SQL, Oracle, WAS, .NET, and others.

Vornex services worldwide clients with date shifting who extend many industries including Banking, Finance, Healthcare, State & Local Government, Utility, and Insurance. In addition, Vornex works with industry leading IT consultants, 3rd party integrators, and VARs such as Accenture, CapGemini, Wipro, IBM Global Services, etc.

All these industries and enterprises rely on TimeShiftX to validate their date sensitive applications and accelerate their time of delivery. Vornex enables organisations to increase efficiency, reduce resources & cost, and improve corporate bottom line by providing TimeShiftX as an affordable solution that can fit any company’s IT budget.

TimeShiftX +1 408 713 1400 info@vornexinc.com 43575 Mission Blvd. Suite 613

Fremont, CA 94539 USA

www.vornexinc.com

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

2 0

L E A D I N G

T E S T I N G

P R O V I D E R S

2 0 1 7

froglogic Squish

71

GUI Test Automation Code Coverage Analysis

cross platform. multi language. cross design.

Learn more and get in touch: www.froglogic.com T E S T M a g a z i n e | S e p t e m b e r 2 01 7

72

2 0

L E A D I N G

T E S T I N G

P R O V I D E R S

2 0 1 7

Validata Group Validata Group is the leader in the Enterprise Software market for DevOps and Application Lifecycle Management (ALM), helping its clients accelerate application delivery, optimise business risks and reduce costs.

While the focus of most testing companies has been to test the systems of engagements and mobile technologies, our focus is on testing the back end of these systems which is the most difficult task.

Validata provides the first unified Test Automation and DevOps platform, built from the ground up with Agile and Cloud in mind, to bring together requirements, testing, defects, planning, resources, development and deployment, capable of integrating with external systems such as HP QC, CA Clarity, IBM RTM, Atlassian Jira etc, delivering a ‘single version of the truth’ and enabling continuous delivery, integration, testing and monitoring.

Validata’s platform drives quality and velocity for DevOps-oriented or traditional development teams showing 10x overall productivity improvements over siloed QA tools. Its

T E S T M a g a z i n e | S e p t e m b e r 2 01 7

model-based approach enables the automatic generation of test cases and eliminates the high maintenance costs of traditional script-based test automation.

It bridges DevOps with Continuous Testing, involving Service Virtualisation, model – based test automation, real-time Test Data Management, Test Analytics and scriptless automated test case design.

+44 (0)20 7698 2731 info@validata-software.com 52 Brook Street

London W1K 5DS UK

Automated testing using Validata’s platform makes it possible to test the complete software stack, run more tests during the development phase and provide the ability to manage the list of releases.

Validata enables its clients establish a wellplanned QA strategy, to align Dev, Ops and QA/ Testing as part of the true DevOps philosophy.

Validata solutions are also available as softwareas-a-service (SaaS), enabling customers to establish their testing environment with minimum upfront capital investment on a unique pay-as-you-go model.

www.validata-software.com

BU

In harmony

End User

IT operations

Amdocs DUET – Digital User Experience Testing Spanning all aspects of the digital user experience, our state-of-the-art digital testing portfolio covers your entire front and back-end operations – ensuring your customers receive the seamless and consistent omni-channel experience they expect.

For more information, access https://www.amdocs.com/dt T E S T M a g a z i n e | S e p t e m b e r 2 01 7

BV

QUE SERA SERA (Said no test professional ever)

We know you can’t take reckless risks; that you are expected to work swiftly and deliver quality software, while staying safe. That’s why Panaya offers solutions to help you introduce change fast and risk-free.

Experience smart testing

www.panaya.com

T E S T M a g a z i n e | S e p t e m b e r 2 01 7