TEST – September 2018

SEPTEMBER 2018

transportation SPECIAL

20 LEADING TESTING PROVIDERS

MODERN MANUAL TESTING LOCALISATION TESTING DEVOPS & DEVSECOPS TEST AUTOMATION DATA SCIENCE

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

C O N T E N T S

1

ENTERING THE AUTOMATION AGE

4

10

HOW TO AVOID 70,000 ANGRY PASSENGERS

14

22

TRANSPORTATION

18

THE IMPORTANCE OF LOCAL

32

FROM DEVOPS TO DEVSECOPS

How automated testing can help companies avoid costly outages

04

Maintaining an end-to-end test team with offshore & onshore components

08

Using automation to keep your QA teams lean & productive

10

Can we ever move away from software testing using humans?

14

Localisation testing is paramount for international customer UX

18

The changing role of the developer in the software testing world

22

It's time to move from DevOps to DevSecOps and make security key

32

Top 20 Leading Providers

38

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

2 upcoming

INDUSTRY EVENTS

16 -17

16

30

Oct

Oct

Oct

The National DevOps Conference will be held at the Millennium Gloucester Hotel, Kensington, London, on the 16-17 October 2018. This is a two-day conference designed to connect to a wide range of stakeholders and engage not only existing DevOps pros, but also other senior professionals keen to learn about implementing this useful practice. At The National DevOps Conference, you will have the chance to listen to peers who have successfully begun their DevOps journey; receive advice and knowledge from industry practitioners; as well as join in and debate at executive workshops.

The DevOps Industry Awards will be held at the Millennium Gloucester Hotel, Kensington, London, on the 16 October 2018, to celebrate companies and individuals who have accomplished significant achievements when incorporating and adopting DevOps practices.

This glittering awards gala has been launched to recognise the tremendous efforts of individuals and teams when undergoing digital transformation projects – whether they are small and bespoke, or large complex initiatives.

DevTEST Summit Scotland is a one-day event, which will be held on the 30 October 2018 at the iconic Principal Grand Hotel, Glasgow, Scotland.

The full days programme will see a panel of key professionals speak, before hosting interactive sessions on recent issues related to software testing and DevOps. DevTEST Summit is open to all organisations and individuals within the software testing and DevOps community who are keen to increase their knowledge and harvest workable solutions towards various issued faced in complex, burgeoning sectors.

devopsevent.com

devopsindustryawards.com

devtestsummit.com

21

DEC

Nov

2018

THE EUROPEAN SOFTWARE TESTING

For the sixth year running, The European Software Testing Awards will celebrate companies and individuals who have accomplished significant achievements in the software testing and quality assurance market on the 21 November 2018 at Old Billingsgate, London. Enter The European Software Testing Awards and start on a journey of anticipation and excitement leading up to the awards night – who knows, it could be you and your team collecting one of the highly coveted awards.

The European Software Testing Summit is a one-day event, which will be held on the 21 November 2018 at The Hatton, Farringdon, London. The European Software Testing summit will consist of up to 100 senior software testing and QA professionals, who are eager to network and participate in targeted workshops. All delegates will receive printed research literature, have the chance to interact with The European Software Testing Awards’ experienced Judging Panel, as well receive practical advice and actionable intelligence from dedicated workshops.

softwaretestingawards.com

softwaretestingsummit.com

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

E D I T O R 'S

SEPTEMBER 2018 JULY 2018

transportation SPECIAL

20 LEADING TESTING PROVIDERS

TRANSPORTATION

T

MODERN MANUAL TESTING LOCALISATION TESTING DEVOPS & DEVSECOPS TEST AUTOMATION DATA SCIENCE

WWW.TESTINGMAGAZINE.COM

BARNABY DRACUP EDITOR

here can’t be many industries where a simple software glitch can affect quite so many people. The software that allows for the organisation and transportation of travel and goods, on a scale never seen before, generally ticks away, behind the scenes, unnoticed.

Until it all goes wrong of course, and the fallout rapidly accumulates a direct cost in human terms; stuck and stranded passengers and their cargo, delays, lost revenues and general bad sentiment towards the affected companies. When software fails it is humans who have to pick up the pieces, being drafted in to solve problems and produce work-arounds manually, often with pen and paper, time and physical labour.

Last summer, when the most devastating piece of malware to date, NotPetya (est.

US$10billion total global damage), paralysed the global shipping giant Maersk (among others), it was the efforts of a plucky team of global IT staffers who managed to find the company's lone surviving domain controller in a remote office in Africa and begin patching the system back together again (for Andy Greenberg’s fascinating article on this, check out wired.com).

However, the point is that perhaps there will always be an inescapable link between humans and the software we create – no matter how fully-automated we can make our testing. The human touch may always be a prerequisite in any system (see p.14), at any stage of the process; be it recovery, development or maintenance.

Another consideration is will a machine or piece of code ever be able to determine the aesthetic appeal of a product? Will they ever be able to tell if something is a pleasure to interact with or just a pain in the arse? Maybe by the stage they can, it will no longer matter to us!

3

TEST MAGAZINE SEPTEMBER MARCH 2018VOLUME 2018 VOLUME VOLUME ISSUE 10 2ISSUE 1 4 MAY 2018 10 10ISSUE

GETTING FROM A TO B

C O M M E N T

This is an ongoing debate among testers, with many espousing the virtues of automation for bullet-proofing the software used by high-stakes industries – global airlines in particular (see p.06) – and, when considering the global transportation industry, it is easy to agree with this position when one considers the mammoth task to which the software testing industry is being increasingly put in terms of faster releases of higher quality from smaller teams (see p.10), with an ever increasing scope of enterprise extending to every corner of the earth and beyond (if the likes of Branson & Musk continue to forge ahead successfully with their spacefaring plans).

So, will full AI test automation be a thing of the future? It's hard to say just yet, but if we consider author and broadcaster, Christian Wolmar's, brief gedankenerfahrung called ‘the Holborn problem’: If driverless vehicles are programmed to stop when they detect a pedestrian, what happens when they are confronted with a crowd of people milling across a busy road? Will they wait all day? Or will we be asked to accept a lower safety standard?

From this we, as humans, can instantly see the problem and assess the situation, see what’s going on and effect a positive result with a (potentially constantly changing) course of safe action. However, what would a self-driving vehicle do? Could it account for this element of humaninduced chaos? Even if so, what kind of delays would be incurred to the cargo while the issue is resolved?

Despite this, it all points to an interesting, challenging and exciting future for the transportation industry and the software testers that will be increasingly required to ensure quality, consistency and above all, safety. Whether they are driving those tests forward on automatic or manual.

SEPT 2018 | VOLUME 10 | ISSUE 4 © 2018 31 Media Limited. All rights reserved.

TEST Magazine is edited, designed, and published by 31 Media Limited. No part of TEST Magazine may be reproduced, transmitted, stored electronically, distributed, or copied, in whole or part without the prior written consent of the publisher. A reprint service is available. Opinions expressed in this journal do not necessarily reflect those of the editor of TEST Magazine or its publisher, 31 Media Limited.

ISSN 2040‑01‑60 EDITOR Barnaby Dracup editor@31media.co.uk +44 (0)203 056 4599 STAFF WRITER Leah Alger leah.alger@31media.co.uk +44 (0)203 668 6948 ADVERTISING ENQUIRIES Shivanni Sohal shivanni.sohal@31media.co.uk +44 (0)203 668 6945 PRODUCTION & DESIGN Ivan Boyanov ivan.boyanov@31media.co.uk Roselyne Sechel roselyne.sechel@31media.co.uk 31 Media Ltd, 41‑42 Daisy Business Park 19‑35 Sylvan Grove London, SE15 1PD +44 (0)870 863 6930 info@31media.co.uk testingmagazine.com PRINTED BY Pensord, Tram Road, Pontllanfraith, Blackwood, NP12 2YA  softwaretestingnews  @testmagazine  TEST Magazine Group

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

4

HOW TO AVOID 70,000 ANGRY PASSENGERS There is a growing need for automated testing platforms in the transportation industry and companies should turn their manual testing processes into modern ones he following news is the worst nightmare for QA managers, especially ones working in the airline industry. Around a year ago, a leading software system supplier to British Airways, Air France, and Lufthansa experienced an awful glitch in their system (The Telegraph, September 28, 2017). Their check-in system was down for 15 minutes during the holiday season and experienced a power failure to top it off. This created absolute mayhem.

Thousands of customers filed complaints, six hundred flights were cancelled and over 70,000 angry passengers were created. However, the software glitch didn't just create upset customers – this failure ultimately led to an international news frenzy. The airline brands were negatively affected and lead

T

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

to a loss in profit.

There’s no question of how critical customer loyalty is these days.

Throughout the past decade there have been many instances where a small software glitch has turned a company upside down – cases where one small software bug creates one big fiasco. So, how can big airline companies avoid this?

COMPETITIVE EDGE – CONTINUOUS DELIVERY

Nowadays, a bumpy digital interaction is a deal breaker for both the business and the customer. Brand loyalty can weaken in an instant when a customer becomes unsatisfied. Marketplace competition is much stronger, so there is a greater chance these customers will move on

YARIN PODOLER CO-FOUNDER & CHIEF PRODUCT OFFICER TESTCRAFT Yarin has been coding since the age of 12 and previously worked as an R&D developer at StartApp, Perion, and SweetIM.

T E S T

to a competitor. If I enter Expedia to purchase a plane ticket, for example, and experience even a minor bug, I would then instantly move to Skyscanner or any other comparison website to move forward with my purchase. This not only applies to travel-fair websites, but to all transportation companies where competition relies not only on product quality, but on customer service and user experience.

The digital revolution, thus, turns customers into omni-powered beings who expect nothing less than a seamless and effortless experience. Software development, inevitably, moves hand-inhand with a customer’s expectation, and the industry has seen huge developments towards a smart and automated environment that provides continuous delivery. Nonetheless, software testing is still heavily manual and is lagging behind the expeditious agile dev process.

Previous QA efforts are no longer effective in a fast-paced industry, which brings us to a new era. The era of the modern manual automation tester in the agile world.

WHEN DID AGILE BREAK THROUGH?

Companies have started to adopt agile through the dev process over the past decade. However, the agile framework has picked up momentum over the past few years. The motivations from shifting from waterfall to agile were pretty clear from the get-go: bring software faster to the market by dividing the production process into short stacks, thus, increasing both the level of software quality and customer satisfaction in organisations.

Is agile the norm today? Definitely yes – a recent HPE survey shows that 67% of companies are either pure agile or leaning towards it.

A U T O M A T I O N

With agile taking over the software development process, many challenges rose to the surface that companies were not yet aware of. Teams that had a lack of communication typically failed to pursue deadlines and deliveries for new features. Plus, the teams weren’t appropriately equipped with the right tools. It is important for any company or team to have tools made especially for agile delivery. Agile-based tools may include types for continuous integration that allow teams to continuously develop and deploy robust test code.

Still early on in agile, all the focus was on the dev process and the tools behind it, while the testing part was not dealt with. Fortunately, the DevOps movement was introduced to better align and integrate QA with every aspect of the development process. I won't go deep into it, however, it is important to highlight the realisation that, from ten years ago to recent days, there has been a latent need for better integration of teams and automated processes. Which brings us to question: why are companies still relying on old-fashioned manual testing, even 10 years after the introduction of DevOps and automated testing?

SOFTWARE TESTING & DEVELOPMENT GAPS

Regardless of major upgrades in the development process, testing was still left behind. The focus was on the dev team, the processes of the dev team, and the tools to enhance their productivity.

QA, in contrast, could not catch up and testing soon became a bottleneck. The challenges are ever growing – software release cycles are getting shorter and more frequent. From releasing once a year in the waterfall method, to even a few times a week now, means that regressions are being rapidly built up.

Plus, the introduction of many different platforms multiplied by several test

5

cases have created huge matrices for companies to cope with. With all of that to consider, everybody would agree that test automation is a must.

INDUSTRY STANDARD SELENIUM

The need for test automation led to the quick adoption of Selenium, which started simultaneously with agile and has become the de-facto standard in the industry. It caught momentum by being an opensource platform with a vibrant community, by having integrations with different APIs and plugins, supporting several devices and browsers, and other reasons most of us are already familiar with.

However, Selenium has some considerable hidden costs. For example, the Selenium integrated development environment (IDE) stores element locators in each step of the test, so testers need to interact with each element every time the locator is defined in a different place — if it's 20, that's 20 locations.

This produces serious maintainability issues because of the duplications involved. In addition to that, Selenium requires substantial technical expertise and the support of third-party tools to get completely functional. Plus, manual testers are not able to easily familiarise themselves with the tool as it requires knowledge in coding.

BRINGING THE MANUAL TESTER INTO DEVOPS

Manual testers bring significant value to the company – they deeply understand business processes and the applications that they test. In many cases, they come from testing related disciplines, so they know how to structure testing and what to test. In fact, the testing world still heavily depends on manual testing - 70% of testing is still done manually, according to Capgemini. If manual testers are already present in the business, then it might be easier to keep the team, rather than hiring a whole new team of test engineers. Manual testers bring significant value, but manual testing as the core practice is inefficient.

Of course, all companies should find a way to automate their tests and to work at a continuous and fast pace.

For a transportation company, time effectiveness is extremely important.

This means that every second a software

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

6

T E S T

A U T O M A T I O N

application is delayed, it's a second longer to take one item (or person) from X to Y, and a second later for check-in or worse, another customer lost to the competition.

Transportation companies, such as airlines, online ticket resellers, deliveries, etc., should look for ways to bridge the gap between the clear need for automation, and to support digital transformation and the value brought by manual testers. The way to do it is to find a way for manual testers to start creating test automation, without having to learn how to code, thus being connected with the DevOps and agile methodology.

Automated tests should be intended to serve manual testers, therefore should be easy to create and, even more importantly, easy to maintain. With this in mind, QA managers should look for codeless test automation that will enable manual testers to turn their manual test scenarios into automated ones, without coding knowledge. Even further, managers can choose codeless Selenium test automation platforms, that will enable their team to enjoy the advantages of Selenium without the hidden costs.

Codeless platforms often have a mechanism to reuse test elements or tests so that when a change is made, it will be automatically applied to all other tests that use the same element.

Additionally, you can choose platforms that use artificial intelligence or machine learning to automatically overcome changes in the app and adapt the tests accordingly. This will allow a smooth transition to continuous testing (CT).

The need for CT is quite clear when it comes to the transportation industry. This industry can have massive financial losses just from single software bug instances. If your team correctly bridges this gap it can lead to fewer glitches and bugs, which in turn leads to less money lost, no negative media coverage and an edge over the competition.

MANUAL TRAINING FOR AUTOMATION

The next step to achieving a more efficient QA automation operation is to better train your team. The training implementation has been far too toolcentric. Testers are often incorrectly taught how to use a specific automation tool rather than learning about its underlying features and patterns. Testing experts believe the right automation

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

training should be focused on how to build an automation strategy and how to manage certain expectations around automation. Previously, QA managers and testers were stuck with legacy tools and had to adapt themselves to them. Now, tools such as codeless test automation platforms are created and adapted for the tester.

Another focus of efficient automation training is to better understand when and what to automate: • When: automation should be done at the beginning of the dev lifecycle. It is paramount to test early and to test often. Part of CT is to allow testers to begin creating tests early in the release cycle ('shift left') – an issue already addressed by some of the new-age automation tools • What: when it comes to what tests to automate you will gain a lot of benefit from automating repetitive tests, multi-data sets, functionality tests, etc., but you definitely don’t want to automate everything or to reach 100% test coverage. Teams should have all of this in mind when training and learning to be successful automation testers.

COMPARING MANUAL VS. AUTOMATION TESTING Testing execution at scale is much faster when done by a machine. Software development is a repetitive process by nature, thus being able to execute new test code alongside regression is an integral part of the release criteria.

Performing such activities manually is completely inefficient and hinders the entire release process. In fact, most cases of tech disasters such as the one described in this article are consequences of unidentified software bugs and inadequate QA. Can companies really blame manual testers for it, or should they take it as a lesson and introduce tools to automate and improve the reliability of the process?

Manual testing alone is by definition error-prone and indeterministic compared to consistent machine-based test automation. Companies need the ability to consistently get the same results when the expected criteria are met (or not) to release high-quality products. Such reliability at scale, as the product grows, is a critical enabler for continuous testing.

Most organisations of the digital age are exposed to fast market releases of new operating systems, devices, gadgets and more. Therefore, they are expected to readjust their product and try new features and operations at the same pace.

Fortunately, companies that work with agile and DevOps have this mentality as a core practice. Dealing with these changes in an efficient way is only possible by having a robust test automation foundation that can be adjusted and triggered on-demand as changes occur.

The testing framework and network for transportation companies can be a bit more complex than any other industry's network. A transportation company's network needs to be adaptable and able to perform multiple different functions like monitoring vehicles and cargo, finalising delivery payments, checking supplier operations and last-mile deliveries, tickets purchase, check-ins, delays notifications, reminders, etc. Those functions should always be available, otherwise there may be consequences throughout the entire supply chain.

AUDIT TRAIL AND EVIDENCE

Test automation leaves trails and can be retained for as long as needed, while manual testing is by nature harder to record and store. Specifically, having the test data for ad-hoc audits and compliance processes is a clear business requirement for verticals like transportation, logistics, and airlines.

The case for having test automation platforms as key enablers for the entire DevOps and business' success was proven years ago and, in reality, this case has grown in significance due to advancing technologies. It is now all about expanding test automation capabilities through best practices, increased value of the QA tester, robust test labs, and highvalue test frameworks to support the digital changes.

Nevertheless, it is not sufficient to simply have a great test automation tool. QA managers also need to push themselves more thoroughly to cope with testing developments. DevOps methodology is a powerful enabler for testing teams to be integrated with the development process. The benefits of DevOps and test automation are clear, but how should the modern manual tester change to fit with them?

7

SHAPING QA MANAGERS & THEIR TEAM

To enable DevOps, teams are focusing on rapidly delivering high value and quality.

This translates into efficient continuous integration (CI) that builds new software, and continuous testing (CT) that validates its quality in each step of the development process.

Only by having reliable CI and CT, teams can effectively release software as part of their continuous delivery/ deployment (CD). The QA manager's role in assuring the fluent process of these three pillars is fundamental. QA management should have on-demand data in each stage starting from design, implementation, build acceptance, integration testing, and production testing.

CONTINUOUS TEST MEASUREMENT

DevOps and software development are continuous processes. Innovation does not stop at the end of a release, nor does test management and measurement.

Hence, QA managers need to define metrics to help them measure efficiency from one to the other. Such metrics would include: • efficiency of test scenarios • pass/fail rate between cycles • amounts of defects on the defect backlog • progress of the CI cycle • test coverage alignment with market and business requirements & product analytics/usage • ratio of test automation/manual.

TEAM PRODUCTIVITY & SKILL SETS

Continuous improvement in the software quality process (as mentioned earlier) is great, but we must not forget that the process is led by people. The team must continue to sharpen their skills, to learn new tools, techniques, and to measure their efficiency against the management

objectives. For example, if issues like a test stability problem due to flakiness, or too many escaped defects to production appear, it is time for the management and the team to come together and build a plan to evaluate what needs to happen to overcome these issues.

Fortunately, the world of testing is evolving fast, with new open-source frameworks, machine learning, and deeplearning based tools. Such innovations open doors for practitioners and managers to improve and develop to a more mature state.

QA managers can be the steering wheel for a mature DevOps practice. It involves leadership, cross-team communication, measurements and improvement based on metrics, and a matching set of technologies to support their agenda. If, in the past, QA was about how many tests one can develop in a given time, these days the reality is not about amounts of tests, rather about the reliability, value, stability, and the speed at which tests can run.

"The implementation of test automation tools serves as a catalyst to bring testers into the DevOps era – an era where we will no longer see worldwide chaos in airports caused by system failures"

CONCLUSION

The transportation industry is going through a massive digital transformation.

Customers are expecting nothing less than complete digitisation from businesses, hence, if a company wants to survive or even beat the competition, its internal processes have to keep up. Experience has become pivotal in customers' journeys through brands, meaning if interactions are not smooth and fluid, they will immediately shift to a competitor.

The software industry has come a long way since the introduction of waterfall.

We have seen major changes and new practices in software development, but still, QA testing as a manual practice is still heavily used. Modern manual testing, as a combination of test automation tools and an integrated team, are the keys to keeping companies en route on their digital roadmap. The implementation of test automation tools serves as a catalyst to bring testers into the DevOps era – an era where we will no longer see worldwide chaos in airports caused by system failures.

The question here is, not why corporations should work with automated software testing platforms, but how soon will all of them use them.

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

8

MANAGING OFFSHORE & ONSHORE RESOURCES Test Manager at Three, Victoria Bayes, exclusively reveals to TEST Magazine how she builds, manages and maintains the end-to-end test team comprising of both onshore and offshore resources

B

eginning her testing career seven years ago as a user expert, Bayes now manages a dual location team and is responsible for the test and automation strategy across the digital programme. Accustomed to managing multiple projects and priorities in fastpaced environments, she is a firm believer in embedding quality assurance into deliveries as early as possible, with collaboration and honesty with stakeholders being key attributes to allow her team to be successful.

"I have been at Three for three years; working in the digital area and managing all test phases including integration systems, UAT, SIT, API, functional, live support, performance testing and taking control of the entire backend system which sits behind our Wuntu app," says Bayes.

"I don't test the app itself, that's done by a separate scrum team. However, once it leaves the app and comes into the backend system I am accountable for the quality."

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

A TYPICAL DAY IN THE OFFICE

For Bayes, a typical day in the office consists of managing 10 projects in various phases of the software delivery lifecycle (SDLC) and around six ad hoc change requests. She has a daily standup with all leads – checking where they are and if they need anything from her that day. Project meetings, stakeholder management, resource allocation and admin days also take place.

"I have a team of around 20 people, seven onshore in the UK office while the others are offshore in India.

"My key tip as a manager is to always be open and honest by being upfront about what can and can't be achieved, with project managers and stakeholders, which are all onshore. As long as you're open and honest about the risks that may occur, you will always get (at least) the quality you're expecting," says Bayes.

"Testers will obviously push to ensure they produce the best product they can but there are risks such as financial budgets and

LEAH ALGER JOURNALIST TEST MAGAZINE For industry news and features submissions please contact leah.alger@31media.co.uk

O N S H O R I N G

timescales which can get in the way.

"Nevertheless, I am really proud of my team. All problems that have been aligned we have been aware of unless it was outside of our control – such as a problem with the back-end server."

IMPORTANCE OF TEAM ENGAGEMENT

At Three, the software teams can be moved around. This allows them to be agile and flexible in terms of what they are working on. According to Bayes, it's important to advocate the shift-left testing approach by getting her team engaged as soon as possible.

"Everyone knows defects are expensive to fix once in production," adds Bayes.

"It's important to get my guys engaged in concept and not leave anything until the last minute – when things are too late to be fixed. This is why we moved away from waterfall to an agile method."

At Three, they have weekly video meetings with all onshore and offshore colleagues. It’s important for colleagues to communicate face-to-face with each other to ensure a healthy work culture.

OFFSHORE & ONSHORE BARRIERS

"Once a week, the offshore and onshore colleagues do PechaKucha, which is

&

a Japanese presentation style where they make 20 slides about themselves on whatever they want – such as what they do outside of work and their favourite things. It's important for them to know about one another, to get an understanding of what they're all about.

I do this to ensure those offshore are involved as much as possible," says Bayes.

"Nevertheless, it can be difficult to build a team mentality with offshore clients. Confidence and language barriers can get in the way. Sometimes, you can't beat being in the office listening to conversations. But, then again, it's a plus that offshore clients don't get disturbed and can crack on with their work. Offshore is also a large cost benefit, of course."

Three have recently launched a large project, updating the backend for the Wuntu app. It has been a great, successful project for Three, despite it entailing a lot of back-end changes.

"Thankfully, I only needed to write one strategy for the Wuntu app," says Bayes.

"This allowed me to create an automation strategy and to review the team's overall performance, as well as taking the business drivers and their change in ways of working – including industry and business changes, and customer needs," she concludes, before adding a whimsical aside, noting that 'onshore guys are typically more senior, while offshore guys are more tech-savvy'.

O F F S H O R I N G 99

"It can be difficult to build a team mentality with offshore clients.

Confidence and language barriers can get in the way.

Sometimes, you can't beat being in the office listening to conversations"

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

10

ENTERING THE AUTOMATION AGE Keeping your QA teams lean and productive with automation is a must, especially in this age of increasing business pressures he age of automation is upon us, it is prevalent in all facets of software delivery and is growing at a colossal rate. No matter what industry, sector, or company you may be working in, businesses are catching on to the importance of automation practices. This article focuses on how automation can assist with industry-type examples, putting tester's front and centre, and therefore helping businesses usher in the 'age of the automation tester'.

I'll be focusing on the transportation and financial industries, using real life examples I’ve implemented to keep QA teams lean and productive, using automation as an enabler. My hope is that you are able to take these learnings, practices and guides, to support whichever company you're in, and to really help you push on to the next phase of your automated test career.

T

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

STEPPING BACK IN TIME

In order to understand if we are in the automation age, we must look back to where it began to understand where it can potentially lead. Testing is not a new notion to us, we've always tested in one form or another, whether it be a pair of shoes, a car, golf clubs, boats or everything else that has existed since the dawn of time; we all have an inbuilt need to test anything and everything around us. I'm getting married next month and some say that’s the ultimate test of all – although I don’t think they've gotten around to automating that just yet!

Now, understanding that testing is a function of life, and not just a 9-to-5 job, is key to seeing the whole landscape of what has come before and what is still to come in the future of automation.

Automated testing in computing isn't anything new and is at least 40 years old in

SEAN RAND LEAD QA VELOCITY RISK UNDERWRITERS Sean leads a team of QA Automation Engineers and is a Selenium .NET contributor. He loves to learn new techniques to stay on the cutting edge of QA, informing his daily decisions when leading the QA efforts of his team.

T E S T

its purest form, so why now is there a big craze surrounding automation testing?

This, I believe, is due to the knowledge, skill, and accessibility of practices, toolings and frameworks being readily shared by the testing community. For example, it takes me less than 10 minutes to open my laptop, open my development IDE, write a Selenium browser / API test, execute it, see the result, commit this to my coding repository and make a pull request into our continuous integration environment.

Eight years ago, my laptop alone might have taken 10 minutes to open, let alone accomplish all of those tasks! This cumulative effect of progression, in all facets of technology, is truly an enabler for the 'automation age'.

FINANCIAL SERVICES TESTING

Using the above example, where I create an automated test, this login feature test example below uses Selenium WebDriver to drive the web application; C# .NET Framework. The below test should be familiar to each and every one of us as we as testers can relate to having a negative

login test as part of our test suites.

[TestCategory("FAST")] [TestCategory("LOGIN-FEATURE")] [TestCategory("JIRA-5465")] [TestMethod] public void UnableToLoginWithInvalidPassword() { // Arrange LoginPage.EnterUsername("user"); LoginPage.EnterPassword("invalid"); // Act LoginPage.ClickLoginButton(); // Assert LoginPage.InvalidUsernamePassword DialoguePresent().ShouldBeTrue(); } Within the financial sector I'm part of, I deal a lot with rating algorithmic tests. As one would imagine, the financial world has a lot of tests which are data-driven centric on $ values; meaning large datasets and controlling execution times are both areas we must ensure are managed to be the best in class.

In this next example I will create a test exercising the rating algorithm, my test will

A U T O M A T I O N

11

be coded and automated. This, however, only currently gives me one test and one test run. The key to making this scenario a success is that we want to use machines for what they’re good at: • compute of calculations • repetition / reliability • execution time • parallelisation.

[TestCategory("FAST")] [TestCategory("RATING-FEATURE")] [TestCategory("JIRA-54525")] [TestMethod] public void PremiumAlgorithmExampleTest() { // Arrange Rater.SetupPreconditions(); Rater.ImportTestDataRow("EXAMPLE INPUTS"); // Act Rater.Execute(); // Assert Rater.Result().ShouldBe("$750,985"); } It's extremely rare in the automation age to want to run just one test at a time; we

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

12

T E S T

A U T O M A T I O N

want to data drive and exercise around 100,000 permutations on the insurance platform for pricing, as an example, so would not want to write 100,000 tests.

Why automating this activity makes sense vs manually testing: • average time per human execution = 180 seconds • average time per computer execution = 100ms • total time taken by a manual tester for all 100k tests: 5000hr or ~208 Days • total time taken by the automated test: ~2hr 40mins • total time saved if automated: 99.8% time saving and resource allocation freed up to be productive on the next test.

To accomplish the 100,000-permutation test as detailed above, it takes only a few, fairly simple modifications to make this work from the top side of the test.

We need add a data source attribute as highlighted in the updated data driven test shown below. Notice we use our data source as the input and our expected output so the test becomes very generic, our test is a harness for the data.

[DataSource("RatingPermutations")] [TestCategory("FAST")] [TestCategory("RATING-FEATURE")] [TestCategory("JIRA-54525")] [TestMethod] public void PremiumAlgorithmExampleTest() { // Arrange Rater.SetupPreconditions(); Rater.ImportTestDataRow(TestContext DataRow["TEST INPUT"].ToString()); // Act Rater.Execute(); // Assert Rater.Result().ShouldBe(TestContext DataRow["EXPECTED RATE"].ToString()); } This is how we accomplish large volumes of test executions, using the same test to drive, via a data source. This data source can be formed by any and/or everything that persists data, from a flat file, excel file, database or JSON file storage system in AWS or AZURE. In the financial world and others, there are many tests where large data sets are required. Within the insurance industry, insurance factors make up parts of the overall puzzle to calculate a premium for the consumer. These parts of the puzzle,

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

i.e. rating factors, must be tested heavily and continually to ensure they never change unless a rate change comes into effect to protect the consumers' rates.

Rating factors I test against are exposed through an API, using automation testing with large datasets of inputs and expected results. We can test every factor, millions of them in less than 10 minutes, therefore, automation testing is giving ultimate confidence in our pricing and products knowing our factors are exact.

If we were not able to automate the testing of such features, the QA team would increase from two/three people as it currently stands, to well over 30 people and that would still not be as effective or efficient as our automated tests. So, the importance of you being the enabler for the business to automate, give quality assurance on the product you're testing, and to also educate people, is crucial to the testing world. We must showcase some of our tests and processes just like normal business users do when they reach their goals. Why is it any different for testers, I say?

TRANSPORTATION

I'm based in Nashville Tennessee, or 'music city' as it's known. The transportation system here is under a lot of stress due to it being one of the largest growing cities in the USA. The infrastructure strains are being felt by all, so the state-level government and technology community are coming together to try and turn those strains into something fantastic. When we think of a big city in Europe, we think of rail links, great roads, cycle lanes, bus lanes and public transportation infrastructure.

However, Nashville has grown so fast it hasn't been able to keep up with the demand – so its melting pot of people, cultures and arts are all coming together to build a new Nashville. Due to this growth, the Nashville Technology Council are working with like-minded technology people using automation techniques to simulate and test permutations of projected traffic plans, new road systems and public transportation links. Talks on these matters, and the sharing of ideas, are just some of the forums via which transportation issues are being tackled head-on in Nashville.

The volume of traffic data what we produce via our phones' GPS systems, such as Google Maps or Waze, generate a

wealth of information to test simulations using machine learning techniques to predict traffic patterns. With such data, and the automation tester's knowledge, technologists now participate in helping the city to best plan the potential infrastructure in a way which is backed by data trends and predictive analysis testing.

One of these methods of testing is called a Monte Carlo simulation, whereby there are test constraints driven by the huge dataset: our test data input.

These will range from a few hundred thousand permutations to tens of millions.

Remember, we mentioned earlier the working example for the financial rating tests of 100,000 permutations, the principles again of an automated tester in this scenario remain the same, only the inputs and outputs differ.

So, you can hopefully see a picture forming of why automation can be applied in many different scenarios, no matter how large or small.

Using test automation via a Monte Carlo simulation allows for millions of outputs to be analysed to best form a result in understanding how to expand the city's infrastructure in a modern way.

These are some simple examples to get us thinking how automation can break the mould, and help us answer questions in the infrastructure and transportation links of cities around the world. If you apply your knowledge through automation, you can tackle most problems head-on and will become a valuable asset with such understanding.

CAN THE AUTOMATION AGE APPLY TO YOU?

As new technologies arise with toolings, languages and frameworks everdeveloping, those who are unable to maintain and adapt to efficiencies will feel the pressure to become faster and morenimble, much sooner than those on the edge.

As tester's we must remember our tests are our most important tool for ensuring we maintain standards throughout whatever platform we are working upon.

If we use a new framework to drive our test, which did the same as the prior, did we improve our test, did we expand our test or did we give value to the business on delivery? If the answer is 'yes' then great, but if 'no' then we must always ask ourselves: does this new tool help me or is it just a newer version of something which

13

might not add the value it suggests?

If you fail to maintain pace and move with automation practices adding value to tests, then you may find yourself falling behind in terms of delivery.

These skills, which are procured as a tester through automation and development practices, are invaluable in supporting your industry and company.

I am currently developing in-house BOTs which are using the Microsoft AI Framework. You may ask why, testers are not developers? But, aren't we all developers?

If you can develop Hello World! or Facebook, or write test automation code, aren't we all actually developers? I believe we are, and are just at different stages of that journey. Using that developer knowledge to help us implement our tests, and other areas to support your company, can turn you from a tester/QA to a valuable innovator overnight. That's a pretty amazing thought, you can become

an automation QA and also an innovator through development knowledge gained, therefore being more valuable to your company.

BUT A TESTER MUST STILL MANUAL TEST!

In a nutshell, I will use my experience to choose the path of manual vs automated as a tool to support a desired outcome. For example, if I'm working in the highly agile world – daily iterations on features, which are truly evolving between myself and a developer, I'll choose manual testing all day long. Reasons being: it's exploratory, tactile, and immediately allows me to learn while testing. In this example, automating would not make sense. As features mature, I'd then turn to automating them. My rule of thumb is, if I'm testing the same test case more than twice – automate it.

I want to always be forward-facing and staying at the forefront of delivery

whilst maintaining quality. The ability to automate a full regression suite and run it in CI is one way that has allowed me to achieve this. Until a few months ago I have been the sole QA at a leading coastal risk insurer. I've filled that position happily for over two years supporting a business which has a turnover of US$150million+ yearly and growing. It's only through precise use of automation techniques, tools and practices I have been able to achieve such a feat. Prior to my current position I headed up our parent company's QA team within the world's largest insurance-linked securities hedge fund out of Bermuda, the fund size is US$13billion and we were able to run QA effectively with three automation engineers.

The 'automation age' is here, and it's not going anywhere! It is so much easier for us, as a profession, to be able to stand out and shine now. So, go shine and prove that software testing really is today's standout discipline!

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

14

THE MANUAL APPROACH Ever shortening release cycles are pushing us further towards automation, but can we ever really move away from software testing using humans? n the modern testing landscape, we appear to be setting ourselves unreasonable targets, which has increased the speed at which testing has to be carried out, and also the amount of automation and emulation packages which are needed to ensure testing meets the 'go live, pass' criteria set. And this is true whether it is end-to-end, integration, or unit testing.

Our ability to automate testing has resulted in job losses within the field of testing, and can be compared to job losses due to production line automations, faster robotic technology and the use of improved Programmable Logic Controllers.

In reality, this is not always a bad thing, and only good when quality can be assured.

With the advent of interlocking applications, allowing the isolation of a

I

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

failure within an end-to-end process, how testing is carried out has changed and also what is considered a 'pass' or a 'fail'.

Here are some thoughts on why we can never really move away from testing using humans, despite all the modern testing aids which are available. I believe testing is dependent on listening to the judgements humans are able to make, based on something other than logic, including emotional reactions based on how something makes us feel, like or dislike.

This is something software will never be able to do, at least one hopes.

WHY HUMAN INTERACTION MATTERS

As hard as I try, I cannot think of a truly automated environment in which I have worked. An environment where the

AZIK CHOWDHURY BUSINESS ANALYST & PROJECT MANAGER JAGUAR LAND ROVER Azik specialises in managing and carrying out business analysis on large-scale change and transformation projects in the public and private sectors.

M O D E R N

entire end-to-end process had no human involvement. I have witnessed increasingly clever code procedures which mimic keystrokes and functions carried out by humans – but they only ever mimic, they are not yet capable of functioning in a 'what if?' type environment. They can only conduct a repetitive process.

My belief is, as long as the solution, software or product is going to be used by humans, there will always be a requirement for humans to test the product; for usability, navigation and essentially to ensure it does what it says on the tin.

Another important factor, which cannot be answered with technology, is that of a product's aesthetics – 'does this solution/ product appeal to the end user, and is it easy to use?' is a question that can only ever be answered by humans.

Ease of functionality is another key element. As an example, let's look at Microsoft's Excel program; for office workers this is a ubiquitous product. It can be used as an everyday spreadsheet application to carry out simple calculations, or a modelling tool with complex formulas, ratios, and the resultant outputs. Part of the attraction of Excel is the way it can take a novice and within a month turn them into an advanced user. This is possible because Excel has numerous template wizards, formula wizards, recognisable icons and help texts. Also, via the internet in the wider world, you can find an incredible number of books, forums, and websites which allow a user to get to grips with the product and its capabilities. The only drawback with having something as powerful as Excel as part of your end-to-end process is that it is not always tested as thoroughly as something like a software upgrade.

When a core spreadsheet fails the most significant problem faced is that it is not always easy to recover, and in some cases can be unrecoverable. The reason for this is that the knowledge needed to fix an Excel problem is often the intellectual property of just one person – the spreadsheet developer. This is a single point of failure.

I was once involved in creating a very complex workaround after an Excel spreadsheet left by my predecessor stopped working following a Windows upgrade. A very painful lesson!

However, the proliferation of Excel solutions in the workplace is a very different topic, and should be covered in another article.

M A N U A L

"I believe testing is dependent on listening to the judgements humans are able to make, based on something other than logic, including emotional reactions based on how something makes us feel, like or dislike.

This is something software will never be able to do, at least one hopes"

T E S T I N G

15

MANUAL TESTING ESSENTIALS

For the purpose of this article, the case being made about Excel is that it feels as if it’s been developed and tested by humans for humans, since it is such an intuitive tool. Its functionality is enhanced by icons which allow usability of the product across the globe, despite differing language formats. I feel Excel would not be the product it is without human intervention.

This is a good example of why we can never exclude humans in testing when the product or software has to be used by humans – to do so leaves huge gaps in the solution, which have to be back-filled at a later date. We can, of course, emulate network traffic or data packets, the stuff humans don't see, but data also creates many of the delays in completion of a process. The frustration felt, while an hour glass spins, is just as significant a test as is the successful completion of a process.

REWRITING TEST SCRIPTS IN AGILE

Methodologies can actually create additional work when it comes to testing.

For example, one of the adverse side effects of using agile methodology comes from one of its fundamental benefits; the ability to make fluid changes and reflect them in the product development flow, the UI, or features and functionalities.

This fluid capability has a major impact for testing and its prerequisites, test scenarios and scripts. Any changes which impact the fundamental design, function and end product require a rewrite of the automated and manual test script to reflect these changes, otherwise testing, in part, becomes invalid.

TEST AUTOMATION ISN'T ALWAYS GOOD VALUE

Let's stack up a few of the most common costs incurred when looking at testing automations: • software + license + maintenance = general costs • configuration of software = consultancy costs • writing test scripts usually by a software specialist = SME costs • reusability of scripts = wastage costs.

For large scale projects, or global change projects involving multiple sites and a

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

16

single product, solution, telephony, or repetitive testing scenarios, it is worth paying for the elements listed. However, for small or one-off projects these costs do not make sense. To fully appreciate this, you would need to carry out a return on investment (ROI) study, to gauge whether automation tests or man-hours-spent makes more sense.

Another down side of automated testing is that you need to stay on top of test script versions and ensure it matches the applications version it will test. Testing needs to reveal enough to allow the organisation to make meaningful changes to the overall solution and realign plans and/or delivery dates.

Remember, automation of testing takes a very black-and-white approach, it is either a pass or fail. Human intervention allows a test script to be stopped, and understanding that a bug fix or reconfiguration will arrive first thing in the morning, realise it is better to postpone it rather than fail the test.

The following should be understood: technology cannot do it all, there is no such thing as a self-cleaning oven – after a while it needs us to get on our hands and knees, and scrub!

FROM MANUAL TESTER TO SUPER USER

When going through a test phase, I prefer to work with in-house testers or employees.

It always amazes me how much enthusiasm an in-house tester can bring, how much they notice, and how often they have helped in shaping a product to be user friendly, easy to navigate, and right for their company's needs. Humans will often think outside the box and beyond the logic of a simple 'yes' or 'no'.

As an example, when someone who is

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

familiar with the way their organisation will use the solution or software is involved in testing, they may look at the help texts in respect of functionality, to ensure it supports the function. While this may not be a reason for failing the test, it may be reason enough for the solution to be enhanced.

The other benefit of using an in-house tester is that, by the time testing is complete and the solution is rolled out, an organisation has in their midst a Super User, and this will help to enhance how proficiently an organisation uses a solution.

THE LIMITATIONS OF AUTOMATED TESTING

Before leaping into the world of automated testing, here are some scenarios where automated testing will need the human touch: • device compatibility and interaction cannot be tested using automated scripts • switching WiFi off and then turning it on cannot be automated • switching permission settings, and retesting access cannot be automated • calls and texts can stop a test script in its entirety • running multiple applications.

Some of these bullet points apply to mobile apps, however mobile apps are becoming increasingly important and more frequently used in the corporate world, a couple of examples of this: • purchasing authorising applications, allowing managers to authorise monetary value based on their authority level • PDA's and docking information for maintenance, fork lift trucks etc.

"The following should be understood: technology cannot do it all, there is no such thing as a selfcleaning oven – after a while it needs us to get on our hands and knees, and scrub!"

17

This section is only highlighting some of the limitations of automated testing which exist today, and it is nowhere near an exhaustive list.

FASTER, FASTER I NEED IT YESTERDAY!

The current testing landscape is shaped by the desire to shorten the release cycle, and this explains the growth and demand for automated testing, since it can cut out delays caused by human hesitation when making decisions. Automated testing gives us a very definite 'yes' or 'no', and it does this within a split second of the results being available. Also, depending on the overall pass/fail percentage, it allows a systemic decision to be made purely on outcome. Try to remember that passing any testing stage gate with a 70% pass rate means you still have 30% of the functionality to fix. This reveals itself by the number of patch releases and bug fixes needed after a product has gone to market, or has been released in-house, whether it is on your mobile phone, PC or server.

Some industries may consider pushing a product to market after a 70% pass rate, and the automotive industries are often no different. My opinion on this practice of lowering testing pass rates and criteria, and removing human intervention, is that it is damaging to the reputation of the industry and, after a while, has a negative impact on the users' viewpoint. Without confidence in a product it will impact sales for external products and cause a lack of usage when it is developed for in-house solutions. This can lead to a dip in market share, profits, and ultimately it could result in a profit warning for external products, and complete rejection of the solution.

This lowering of testing standards can also

"Some industries may consider pushing a product to market after a 70% pass rate, and the automotive industries are often no different. My opinion on this practice of lowering testing pass rates and criteria, and removing human intervention, is that it is damaging to the reputation of the industry"

be seen in internal projects.

To give an example of this, ask yourself the following question: 'how many organisations are you aware of where workarounds exist, despite state-of-the-art systems?'

For me the answer is simple, every company I have ever worked for has workarounds or supplementary processes, usually via Excel, which allow the full endto-end process to be completed.

My feeling is that part of this is down to the way we test, but an automated test does not ask such questions as: • why doesn't the software do that?

• I'd expect it to complete the process?

• why do I have to go into the system and press a button?

• why can't I easily get a report from this system?

Automated testing often tests keystrokes or partial gateways which make up the end-to-end process, and based on these results it will consider it as an overall pass or fail. The major problem is, these series of tests never ask questions about usability, or convenience.

My final succinct thoughts on testing are as such: testers should clearly define the outcome of each test, clearly define what is expected if a test fails (further development), clearly define the roles played by the business and tester (particularly for internal projects if external testers are being used), clearly define automation and manual testing roles and clearly define any go/no-go criteria.

Testing and the outcomes of testing should allow decisions to be made and to allow an organisation to go forward.

Do not underestimate the value of a well thought-out and thorough testing strategy; it will avoid a lot of pain later.

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

18

THE IMPORTANCE OF LOCALISATION Localisation testing is paramount for ensuring a seamless user experience for your international customers, enhancing revenues and driving new commerce from across the globe n essence, localisation testing is ensuring that your software application appeals to a global market by adapting its regional settings, local parameters, images and languages, while still remaining personal to the end users' cultures, customs and habits. Having localised content creates a bond between end user and application; making them more accepting, and combining it with a userfriendly UI makes their experience a pleasant one.

You may well have come across the following acronyms, L18N and L10N. These basically refer to internationalisation and localisation respectively; in my experience these go hand-in-hand with one another and sometimes it is difficult to distinguish between them.

Further down in this article I will list a

I

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

few pointers which should be considered when transforming your software to adapt to the local region, but first let me briefly explain L18N and L10N and you will see that if you want to localise your software they need to be done in unison.

Let's begin with internationalisation.

You should think of this as a prerequisite for localisation to occur. This is where the developers build the software as generic as possible so that when the need arises, it is flexible enough to be adapted for a specific locale. For example, a developer won't restrict the number of characters allowed in an ID text box and will keep it open to allow for alpha-numeric characters.

On the other hand, the process of localisation is making the software understandable to the end user within a certain locale by translating all the text

SUHAIL HASSUN QA MANAGER XET GROUP SOUTH AFRICA Suhail has been in the software testing industry since 2011 and has worked on various projects both locally and internationally across many domains. He aspires to continue building on his craft and mentoring his team in delivering working, trusted solutions.

L O C A L I S A T I O N

and using relevant colours and graphics that correlate to that particular culture and region. You could almost think of L18N as the back end and L10N as the front end. I will loosely use the term localisation from this point forward which encompasses L18N and L10N.

WHEN TO CONDUCT LOCALISATION TESTING

I strongly believe the localisation thought process should begin at the very start of the project during the design phase, if the team is aware that this project will be used in different regions the system needs to be flexible from the outset. For example, when designing the database and front-end – if the team haven't considered the software's usage in different countries, limitations would have been applied to certain fields such as telephone numbers, ID numbers, currency fields etc. To fix this 'bug' once reaching the quality assurance stage will have a higher cost factor involved as well as impacting on project deadlines. Any fix would have to be tested to see if it now allows for global input, testing system-

generated reports and PDF's to ensure that changes have really propagated all the way through.

A FEW THINGS TO CONSIDER

These may seem like very trivial points in relation to localisation testing but they do have an impact further down the line which, apart from frustrating end users, could lead to revenue loss. Let's assume a piece of software works 100% from a functional standpoint but has typographical and grammatical errors that an end-user experiences before getting to interact with the software; would they trust it? I know I wouldn't!

If such obvious errors haven't been spotted and fixed, imagine how many other errors could exist throughout the software?

Based on my experiences I have jotted down some important points that should be considered by teams to help tune their minds to localisation testing. This is by no means an exhaustive list, but should prove sufficient to assist software teams going forward.

T E S T I N G

19

"Let’s assume a piece of software works 100% from a functional standpoint but has typographical and grammatical errors that an end user experiences before getting to interact with the software.

Would they trust it?

I know I wouldn’t!"

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

20

L O C A L I S A T I O N

T E S T I N G

Figure 1 (above): Tree structure of localisation tests Figure 2 (left): Generic test case example

Text structure & encoding Upon designing the software keep in mind that you may have to cater for languages that read right-to-left as well as vertical characters. As a safe option it would be best to use UTF-8 encoding standards as it's almost guaranteed to work across most browsers, with the least amount of errors/crashes.

Text boxes & text areas Consider the character limitations before applying them, this will be very frustrating for the end-user and I'm sure most of us have experienced this at some point, for example when typing out an address and the character limit is reached.

Date/time formats This is a common one and could even boil down to preference instead of purely being governed by the regions in which the software is deployed. Whatever the case, it is important to keep the formatting consistent throughout the entire application, as well as on any reports, SMS, emails or letters which are generated via the application. NB: don't forget to take into consideration the public holidays for that local market, you don't want your customers losing revenue over this.

Dropdown lists Bear in mind that when the application gets converted to the local language within a particular region, some words, which are short in English, can double in size once converted. If your dropdown lists do not scale, half the word or sentence can be cut-off. Most booking sites make strong use of dropdown lists and if a user is unable to read or see an option properly, revenue loss occurs as users abandon the booking process.

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

Colours It took me some time to fully understand this, but colours do play a very, very important role in your application. This can actually go even further than just country or region; a particular culture within a country can have certain feelings towards a colour based on their belief system. Humans are visual creatures, so be cognisant of your colour choices so as not to offend anyone. I usually select a softer, natural colour palette, instead of something like primary reds/greens etc., to be on the safe side.

Currencies Ensure that currency symbols reflect accordingly. This is by far the most obvious issue that everyone thinks of when discussing localisation, however what's overlooked is adopting the correct decimal point used in that region. If not catered for this can lead to a nasty error or crash being displayed.

Legal, T&Cs, Help These can be easily missed, yet are extremely important and possibly out of this list may be the most costly and specialised of the lot. You will need to verify layouts, spelling and most importantly, grammar. Every sentence needs to make perfect sense and be legally accurate.

APPROACH TO LOCALISATION TESTS

As established above there are many factors to consider when conducting localisation tests, with some being highly specialised and some just very basic checks. In my experience, I have found that creating the test cases in a generic

manner and then copying them across into each language test suite saves time initially and guarantees you consistency across each language test, however there will be a maintenance overhead. Below is an example that I use and has worked for me, perhaps it could point your team in the right direction and serve as a basis for your test suite and/or can be improved on? (see figures 1 & 2 above).

IS LOCALISATION TESTING IMPORTANT?

In short, yes. As we have established above, we are in a technologically driven world and some may even say a world without borders where, from the comfort of our own homes, we are able to order an item from the other side of the globe without having to worry about the language, currency, tax, or time zone implications.

As an end user we 'trust' that all the working mechanisms and logistics will function as expected and the software we are using will be clear, easy to relate to, and simple enough to allow for this.

Far too often as testers we focus most of our attention on functionality, but equally important is usability and I believe we need to dedicate sufficient time towards testing this, coupled with localisation, to give the end-user as seamless a system as possible After all, as testers it is our mandate to have the system work effortlessly and to make end-users' lives easier and more productive. I've barely scratched the surface with the amount of depth we could go into with this topic, but hopefully it's enough, so that on your next project, you will have a better understanding of what to look out for.

21

See the future of Software Testing

software testing news at one glance

softwaretestingnews.co.uk T E S T M a g a z i n e | S e p t e m b e r 2 01 8

22

THE CHANGING ROLE OF THE DEVELOPER With the role of the software developer everchanging in a fast-moving world, it’s our responsibility as an industry to help developers cope with those changes hange is the only constant: this is a well-worn phrase that has special significance for developers.

We all know that the role is constantly morphing – and as enterprises are increasingly pressured to deliver fast and frequent releases, developers are finding themselves taking on more responsibility than ever before.

So, how has the role changed over the last decade? And more importantly, how can developers cope with the pace of change they're facing now? During my fifteen years in enterprise software development, I've witnessed much of this transformation first hand; as I helped develop products and solutions for optimising software quality and application lifecycle management – all against the backdrop of a fast changing and unpredictable business environment.

C

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

THE NEED FOR SPEED

The most significant change in the last decade has been the shift from waterfall development methods to agile.

Digital transformation is accelerating the adoption of agile because digital is the main portal to customers and the customers' expectation is to get more – faster and easier.

In a world where a bad review can go viral in seconds, better apps and faster service have created a make-or-break scenario for vendors – customers are much less tolerant of bugs and issues.

In addition, fast innovation by handset manufacturers and operating system makers has created a continuous stream of innovation opportunities e.g. easier, more secure authentication using biometric support and new workflow opportunities, such as using pictures as

UZI EILON CTO AMERICA PERFECTO Uzi joined Perfecto Mobile seven years ago after a 15-year career as a software developer and manager at IDF, Netrialty, Comverse and Sundisk. His fields of expertise include managing expanding R&D teams, leading sales engineering teams, agile methodologies, mobile application testing, automation tools, and defining customer projects and on-boarding.

S O F T W A R E

part of the insurance claims process. Agile makes it possible to increase velocity whilst ensuring quality, but it brings some challenges too. One hurdle when making this move has been getting to grips with the concept of iterative development.

For waterfall organisations, the accepted way of working has been to design and layout the entire project prior to beginning development. Of course, this might entail months of requirement gathering and design sessions before a team could even begin working on code and during this lengthy process user requirements may change: what was once appropriate no longer fits, resulting in wasted efforts. Agile processes have caused a complete change in direction, helping teams move to focused, small and iterative work and, as a result, allowing them to develop and deploy services quicker than before.

Of course, the shift from waterfall to agile, which went against developers' previous training and experience, caused apprehension for many. This is normal and was expected. But the clear benefits of the agile methodology helped developers to immediately recognise its value.

And, for us, the primary value of agile

D E V E L O P M E N T

is speed. Agile creates fast feedback opportunities between developers and users, enabling faster course correction and improved quality.

Indeed, it's this need for speed and fast feedback which is a driving force behind many of the changes we've seen in software development – and it's the reason, too, for development teams' responsibilities expanding exponentially over the last decade. Where testers, developers and quality assurance (QA) teams once had distinctly different jobs to do, the functions are quickly merging and dev teams are now tasked both with making apps and services and also ensuring they are of the highest quality.

23

"The developer now wears many ‘hats’ and, of course, the problem for some is that these hats aren’t necessarily the ones they want to wear, or even signed up to wear!"

THE NEED FOR TRAINING

The developer now wears many 'hats' and, of course, the problem for some is that these hats aren't necessarily the ones they want to wear, or even signed up to wear! As well as writing code, developers find themselves focused on enhancing and, improving customer experience, boosting service efficiencies and lowering costs. All vital jobs of course, but very different from what they were initially

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

24

S O F T W A R E

D E V E L O P M E N T

trained to do.

Let's take microservices as an example.

The deployment of microservices (the loosely coupled, autonomous, fine-grained services, which many commentators see as the Holy Grail of modern development), is a sea change for many. Deploying many microservices to a shifting number of users means that app deployment is getting much more complicated, and so no longer falls under a pure-play developer's remit. Instead, it becomes the role of a wider, frequently shared, DevOps function, which has responsibility for the whole product development and production lifecycle, and which is the function where many developers now sit.

So, in this world, where job functions are increasingly fluid and complex, it's important that organisations are committed to upskilling their developers and sharpening skill sets towards the goal of automating key activities during the DevOps pipeline. As always, speed is crucial and teams must focus their efforts on improving the processes that are slowing them down.

We believe that leadership is a critical aspect of helping developers seamlessly transition to what essentially constitutes a new job role. Diverse teams and personas call for strong leadership as a unifying force, and a leader's active role in affecting change is crucial. Of course, part of leadership is to enforce stringent metrics and KPIs which help to keep everyone on track. The continuous

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

measurement of both development productivity (the number of user stories per sprint) and quality measurements (the number of defects per release) are imperative.

– commit code is not complete without commit tests alongside it, and new test tools and platforms have been developed in order to help developers deliver quality code and stable automation tests.

THE NEED FOR TESTING

EVOLUTION OF RESPONSIBILITIES

In the past, we saw developers follow a linear process when it came to testing.

Committing code without testing, or with minimal testing, wasn't a problem when QA teams were on hand to catch issues and fix them. But today the QA phase has gone (at least in mature shops), and the iterative development process that is the hallmark of the new DevOps world means that testing and quality assurance needs to happen in tandem with development.

And, just as the developer's role is changing, so too have our ways of measuring: not only our progress, but also the effectiveness and performance of our code, tools and processes. This increased visibility is a good thing and helps to catch issues early. When the developer 'owns the quality', continuous testing becomes absolutely crucial; it's key to improving quality, extending coverage, speeding up development, reducing costs, and saving man hours, and the defacto requirement to achieve the development speed objectives of modern development teams.

And it's with this plethora of reasons that we now see so much emphasis put on test automation. The development process has been fundamentally changed

So, the development world has fundamentally changed over the last decade. And there's little doubt anymore that DevOps has evolved from being the bright, shiny object that everyone talks about, to being the necessary reality. Developers, as part of the DevOps function, now have significant responsibility in an enterprise. They own the availability and sustainability of the customer experience – and that can make or break a business.

But, we recognise that, for a developer, this merging of roles we've seen over the last decade can be tricky. As a developer, it may be that you have always been spending a fair amount of your time on the Ops side, but now it's your responsibility to do both. And much more besides.

Of course, it's difficult to have at least half your job consumed by doing something you didn't necessarily go to school to learn, but this increased responsibility can be vital to the success of a business. In recognition that their role is a crucial one, it's our responsibility as an industry to help developers cope with the change.

25

NEWS | VIDEOS | PRODUCT NEWS | WHITE PAPERS

Industry Leading Portal

DevOps Awards

DevOps Online is the premium online destination for news, reports, whitepapers, research and more relating to the DevOps movement.

Covering all aspects of IT transformation you can be sure that DevOps Online will keep you informed and up to date.

www.devopsonline.co.uk T E S T M a g a z i n e | S e p t e m b e r 2 01 8

26

BRINGING QA TO DATA SCIENCE Despite the existence of best practices in software testing for operational applications, there is a remarkable lack of established QA practices for advanced analytics and data science or decades, the advanced analytics community, rooted in academia and research, has tolerated the lack of best practices for solution deployment.

Today, as the practice of data science proliferates across businesses, conducted by a broadening variety of analytics specialists and data scientists, the number of insufficiently tested solutions is growing rapidly.

F

CHALLENGES OF TESTING

Many advanced analytics practitioners and data scientists rely on code reviews by team members, because typical software testing methodologies cannot accommodate the special needs of their models and applications. As an example, simple changes in data can adversely affect the performance of analytics models. The uniqueness and size of an advanced analytics software solution can make it very challenging to test scalability and prepare for successful implementation. Regular testing of

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

production analytics is required, as models may not have been examined for many years, while the business processes and software environments evolved.

ADVANCED ANALYTICS QA METHODOLOGY

In blending best practices of software testing and analytics, we can successfully execute and institutionalise the review and validation of mathematical optimisation and predictive models. This approach uncovers new ideas for improvement, enables benchmarking of team practices, gives business leaders more confidence in solutions, and helps specialists improve development skills. As seen in several examples, my colleagues and I have verified the robustness and reliability of mathematical optimisation-based software systems, while enabling ongoing improvements to the underlying models.

At Remsoft, a global leader of optimised planning and scheduling software for landbased assets, performance issues were impacting in-house and client-side users

IRV LUSTIG, PHD OPTIMISATION PRINCIPAL PRINCETON CONSULTANTS Irv is passionate about the application of advanced analytics and data science to drive better decisions. He has more than 20 years experience with advanced analytics software and deep technical knowledge of mathematical optimisation, operations research, analytics, data science, and software development.

A D V A N C E D

of their software. Developers encountered inadequate solution times of their formulations of mixed integer programs (MIPs), especially for larger problems with many assets and time periods.

Conducting a third-party model review, my team interviewed key company personnel to understand the business problem and context, and to determine the current structure of the different models. Next, we reviewed documentation to understand the modelling platform and the data sources, and studied data sets to understand how the data and modelling platform mapped to a model's implementation, looking for differences between the understanding of the mathematical model and the actual implementation.

In another example, leaders of a US government agency required an external review of an innovative operational control system to manage a national mobile workforce. After reviewing documentation and interviewed personnel about the business problems and the current solution methodology, we tested for scalability and deployment, recommending techniques to address a variety of performance issues. The US government agency's leadership better understood the risks with its algorithmic approach and chosen solution methodology. Additional improvements to the implementation were determined to minimise risk of failure when the system was deployed.

A N A L Y T I C S

&

D A T A

The benefit of a third-party review was clear in the case of a transportation company, with longstanding and robust analytics capabilities, who required an evaluation of forecasting and optimisation models used in their operational decision making. The outputs of the forecasting models were used as inputs to a sequence of optimisation models. The review and validation uncovered that forecasts were using small amounts of historical data and simplistic techniques for outlier removal and were not tuned to account for the variability of the business in different geographies.

In one optimisation model, it was found that there was a potential for wide variability in the results that would drive future decisions. In a second optimisation model, the review uncovered that the model allowed answers that were not feasible in the business, and the data supplied to the model misrepresented the business conditions.

Based on these working examples and others, I believe the following steps can be recommended for advanced analytics QA: • interview stakeholders from business and analytics development to understand the business problem and context • review existing models and procedures • review data sources • implement models in alternative technologies to compare results

•

•

S C I E N C E

27

— languages, solvers, analytics engines experiment with models and a variety of test data sets to uncover issues and stress the model implementation suggest improvements and recommend possible further investigation.

An advanced analytics QA team requires expertise in modelling, advanced analytics algorithms, numerical computing, commercial and open source packages for analytics and data science, and deployment of systems embedding advanced analytics. Conducting a review entails vital questions about the correctness of the model, data sourcing and integration, publishing and use of solutions in the business, sensitivity of the answers to the inputs, and other issues.

These questions often can't be answered internally for a variety of reasons. An independent testing team may need to be supplemented by third-party experts.

Any organisation that relies on advanced analytics for core processes and key decisions must determine if suitable quality assurance has been conducted. A formal process should be established for testing advanced analytics, in-line with testing of other operational software. The failure to do so could reduce the potential impact of advanced analytics and data science in the business environment.

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

28

Digital Quality platform Accelerating Quality for Digital Enterprises Intelligently

TEST MANAGEMENT

TEST AUTOMATION

ADVANCED ANALYTICS

Test management that scales from Manual to Agile & DevOps Modular, Scalable, Compliance Driven and Secured Automate and manage your CI/CD pipelines with confidence Better visibility, traceability, optimized test coverage and actionable insights

Share the Success Of 100,000+ Happy Users!

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

www.qmetry.com

29

www.qmetry.com/ free-trial-tm

QMetry Test Management Cloud | Server

All the features you need to shift the quality left!

Modularity and Scalability

Cross - Project Reusability

E - Signature and SAML Support

info@qmetry.com

Multiple Integrations

Customized Reports & Smart Dashboards

+1 408 727 1101

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

30

MOBILE APP TESTING IN REAL-TIME Discover how a mobile device cloud makes manual testing happen in real-time hat is the role of manual testing today?

While the testing world pushes for increased automated testing, manual testing still has an important role to play.

And while automated testing is often faster, especially for regression testing, manual testing puts human judgement and creativity at the fore – does everything look right? Are there new or non-obvious paths? How might a user misunderstand what's being presented?

Is there something a test script might know to do that some users will not find easy or intuitive? What happens when the user makes a mistake or does the unexpected? The human tester can also make qualitative assessments of the look, feel, and appeal of the app.

Cloud-based testing is especially important to manual testing because a large variety of devices and OS versions can be checked quickly. But, such cloud access can come with internet lags that slow the work down, or even cause some

W

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

events to be missed. With smooth action, however, cloud-based manual testing can equal the speed and accuracy of having dozens of devices in-hand, something almost no one has. The answer lies in implementing real-time manual testing leveraging a high-performance mobile device cloud that excels in three distinct areas – device access, interactivity and productivity.

IMPROVED DEVICE ACCESS

Without an organised and instantaneous means to share devices, managing mobile devices is a time-consuming process, and due to the ever-increasing number of smartphones and tablets, purchasing multiple devices for testing is expensive.

For mobile developers and testers that are located in different offices worldwide, or even on different floors of the same building, sharing devices in real time is virtually impossible. Sure, devices can be shipped, but waiting to receive devices in

DAN MCFALL PRESIDENT & CEO MOBILE LABS Dan is an experienced technical executive with a passion for helping companies achieve business objectives via technology. His client relationships are almost exclusively in the Global 1000 across a variety of verticals with a heavy focus in finance and financial services, consumer product goods, retail, telecomms and transportation.

M O B I L E

transit is a huge time constraint and can lead to major delays. But it is not much better if the device you need is down the hall in the locked office of a colleague.

The goal for modern mobile testing is to be faster, and attempting physical device sharing leads to a huge strain on time, cost and productivity.

While many testing teams have attempted solutions like using personal devices, the only reasonable way to have sufficient coverage of form factors, models, and OS versions is to have a large pool of devices that everyone can use. A mobile device cloud stores such a pool in one central location where the devices are always charged, accounted for, can be reserved and scheduled, and are ready for instant access anywhere in the enterprise.

IN-HAND VS. INTERACTIVITY

What does manual testing look like if testers do not have a device in hand?

Manual testing with a mobile device cloud enables testers to login to a portal from a computer. Upon login, manual testers can see available devices and

choose the needed one. The device's display contents are shown in real time on the engineer's monitor and it appears to the tester as if the physical device is in his or her hands. All user interactions with the device are initiated with the workstation's keyboard and mouse.

Whether the team decides to implement a mobile device cloud on-premises or keep secure dedicated devices in a data centre, the devices are available 24/7 and easy to find in real time. A mobile device cloud offers the added benefits of allowing testers to reserve devices ahead of time, allowing devices to be used more efficiently.

PRODUCTIVITY GAINS

The world moves fast and so does mobile app testing. By taking devices out of testers' hands, manual testers can test in real-time thanks to not having to wait for physical devices to become available.

A smooth, high-performing device cloud can replicate the results achievable on an in-hand device and improve productivity in other key areas.

Manual testers can use the mobile device cloud to view multiple devices

A P P

T E S T I N G

31

side-by-side for comparison and can save multi-touch gestures.

Reporting bugs is faster because a mobile device cloud allows manual testers to copy and paste text and add screenshots to a defect report.

For apps that rely on geographical location to function properly, such as navigation, restaurant, or retail apps, a mobile device cloud helps manual testers rapidly test through location simulation by changing the GPS location of a device.

Furthermore, when new apps are developed or when an updated app release is ready for testing, a mobile device cloud makes installing apps across devices fast. Apps can be installed in real time if the cloud is integrated with the build system so that new builds are installed automatically.

With all the buzz around mobile automated testing, the role for manual testing might seem diminished. But, in order to release top-quality apps quickly, manual testing must be part of the mix.

The most effective enterprise mobility teams embrace strategies that erase the challenges of manual testing and transform it into real-time testing that improves speed, quality and agility.

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

32

GOING FROM DEVOPS TO DEVSECOPS It is time to make security a core component to what we do in DevOps and make the much needed move to DevSecOps evOps is the union between the development tribe and the operations tribe to create an efficient smooth delivery of services.

It has revolutionised our approach to development and support. However, it is time to consider the security tribe and to move security over to the 'left' and make it core to what we do in DevOps: we need to move to DevSecOps. Data is now more precious that gold, diamonds or shares in Apple. There are relentless and irresistible groups that are using increasingly sophisticated tools and techniques to penetrate defences and harvest data. The threat can come from state actors, criminal organisations, competitors or lone-wolf hackers determined to gain kudos and peer recognition. It seems like every week there is another major data breach, or we are sent a new credit card as the credentials have been compromised.

Attackers are creative, persistent and varied in their modes of attack.

A security policy needs to take into account the likelihood of a determined attack being carried out across several different vectors simultaneously. They can attack through all technical channels

D

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

as well as directly attempt to create social and behavioural attacks to gain access. A determined attacker can be patient, gain limited access and use that access to compile enough information to fully breach a system and steal information.

Traditionally security testing is something that happens late-stage in the development process.

With DevOps we want to test and deploy continuously – but can we do this securely?

THE THREATS

As data becomes more valuable and hacking a computer system becomes a much safer option than robbing a bank, organised crime will try and make a viable income stream from cyber theft. This means that more time and money will be invested by larger, better-funded actors.

This means that the threats will increase in sophistication and diversity and at best we can only stay one step ahead.

National states are increasingly turning to cyber weaponry to achieve their state goals. One outcome of this is the inevitable trickle-down of this cyber

WAYNE TOFTEROO QUALITY LEADER As a provider of test leadership for agile and DevOps, an innovator and a believer in change, Wayne believes there are always ways we can improve.

D E V O P S

weapon technology into the hands of cyber-criminal groups who can modify it to achieve their own goals, changing the payload to enable the stealthy delivery of malware into unsuspecting systems cross the globe.

I think we have to move away from the concept of security testing as an individual activity and start to consider the concept of security as an end-to-end process that is holistic across the entire enterprise and deeply embedded in everything we do.

SECURITY BY DESIGN

Security needs to be built into everything we engineer and for that to happen smoothly, security requirements need to be embedded into the design process in the same way that functional requirements are. This includes the infrastructure, network and all applications.

Guidelines for developers need to be produced that inform on secure engineering practices.

When developers come into a DevOps organisation, the tools and frameworks should be in place to enable them to understand the development practices and standards to use. Known vulnerabilities need to be explicitly avoided, they should be included in design guidelines and developers' coding standards.

The important key concept here is

that we are in control of managing security risks from the outset of a project and that security policy makers and developers should also be making fullyinformed decisions about security from the outset.

SCANNING, AUTOMATED TESTING, PEER REVIEWS

We need to develop and test continuously. We can integrate scanning tools into the DevOps pipeline and run these in the same way we continuously test functional code.

Cloud infrastructure tools – such as Microsoft Azure Advisor and other third-party cloud scanning tools that can be run as well as application scanners, such as OWASP ZAP, that look for vulnerabilities in application code – should be considered in development squads and teams. It isn't a total solution but will identify where known types of vulnerabilities have crept into the development process. Vulnerabilities can then be dealt with like any other defects in the code. They can then be prioritised.

Security tests can be automated and run as part of the DevOps pipeline in the same way as functional tests are. Tools such as BDD-Security or Gauntlet.io, can provide the automation test frameworks needed to run security tests alongside functional tests.

Regular peer reviews and open discussions on security will help keep

S E C U R I T Y

33

team members focused on security and emphasise the importance of keeping to build patterns. These techniques are not so much as to enforce the design guidelines but to keep everyone focused on the goal of building a secure application.

PENETRATION TESTING

Penetration testing, or ethical hacking, gives security testers the opportunity to look for vulnerabilities in the solution as a whole. Penetration testers take on the role of hackers and perform simulated attacks on the system. However, while it is a necessary component in your cyber security arsenal, it is a manual process and cannot be integrated into the traditional DevOps pipeline.

The purpose of penetration testing is to prove the design and build of the security requirements are sufficient to prevent malicious attacks. A policy decision needs to be made on where and when to use this technique. In the world of continuous deployment, our applications are changing, potentially many times a day. We need to determine what type of release we will use this technique on. Perhaps a policy of regular cycles of penetration testing might be an option.

MONITORING & ANALYSIS The answer to the question of 'what should be monitored?' should be

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

34

D E V O P S

S E C U R I T Y

'everything that can be monitored'.

All system events should be monitored and analysed, including all system and firewall activity. I actually think this is now the most important form of defence that we now have. We have to accept that, despite all our security policies and defences, an attacker will find a way through. A way of identifying an attack might be in identifying a change in system behaviour or firewall activity.

An example could be where file-less malware has gotten in and is monitoring activity internally and sending intelligence to an external entity in preparation for a more mature and dedicated attack.

File-less malware attacks have risen by 94% in the first half of 2018, (SentinalOne), and are difficult for legacy anti-virus tools to identify. Other attack types are also increasing and the ability of preventing an attack from penetrating your defences is being challenged every minute of the day by new and unknown attackers.

Having a good picture of how your system operates when uninfected and monitoring for any deviations seems to be a sensible idea and using analysis tools, such as AI, to rapidly identify any unexpected deviation from the norm can only be a good thing.

Analysis tools can provide alerts and automatically generate issues that can be

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

fed to a team's backlog for prioritisation or immediate fix. AI tools can learn and adjust to new threats as they evolve.

SOCIAL & BEHAVIOURAL SECURITY

Although the above are means of creating a DevSecOps pipeline, the problem of securing the enterprise is not just technical, it is a 'people problem' as well.

As we harden up our cyber-defences and become more security conscious, our people and processes will become relatively easier options for attackers to target.

An increasing number of attackers are collecting information on the internal workings and processes of an enterprise with the intention of developing a fullblown attack – where the attacker is making use of their internal knowledge of the organisation and exploiting the internal biases and cognitive vulnerabilities of the staff. This type of threat is now known as 'advanced persistent threat' and it is growing.

The only true defence against the infiltration of your internal communication processes is to increase the use of secondary validation techniques, a variation on secondary authentication.

It is now no longer safe to authorise the

change of payment details for a supplier on the strength of an email from a known contact, the supplier's email system may have been hijacked and the email is the result of intelligence gathering malware and a patient attacker developing the attack over time. A secondary form of authorisation needs to happen, such as a phone call, but not to the number given on the email!

These processes need to be thought through and prepared for, as part of the attack will be to increase pressure on the recipient by saying that it is an urgent request, or by playing on a friendship or existing goodwill. Persuasion techniques are well known and will be integrated into the attack when it is played out.

SUMMARY

Good security is not just about creating a DevSecOps pipeline. This will be a key component of it. It is a combination of many things that need to be thought through and understood clearly.

A clear security policy needs to be in place and security needs to be an integral part of the DNA and thinking of the organisation. It isn't just technology and it isn't just people. The system we are protecting is the people and the technology interacting together.

16 - 17 October 2018 Millenium Gloucester Hotel London

www.DevOpsEvent.com

The National DevOps Conference has its finger on the pulse as it is owned and supported by industry leading news portal DevOps Online and industry leading journal TEST Magazine, and is produced by the organisers of The DevOps Industry Awards.

This two-day programme is designed to connect a wide range of stakeholders, and engage not only existing DevOps pros, but also other senior professionals keen to learn about implementing this practice.

At The National DevOps Conference, you can hear from peers who have successfully begun their DevOps journey; from industry practitioners sharing advice and knowledge; join in executive workshops; network and much more.

 Learn from leading industry experts at cutting edge seminars. Interact at collaborative Q&A sessions

 Join in executive discussion forums  Source the latest products and services via the industry leading exhibition  Network with your peers and key industry figures  Swap and share ideas with likeminded professionals  Network and make new and influential contacts

 Take away ‘real-world’ scenarios and apply learned solutions to accelerate your own IT transformation

 Network with your peers and new acquaintances at the Evening Drinks Reception

REGISTER TODAY 2 - for - 1 End User Registration

 02039315827

2 Days**

£735

Day 1 Only*

£525

 info@devopsevent.com

Day 2 Only

£525

 http://devopsevent.com

*Includes entry to the Evening Drinks Reception on the first day of the Conference. All prices are subject to VAT. **2 for 1 offer applies

only

to 2-Days tickets. Includes 2x2 2-Days tickets to the Conference and 1 entry to the DevOps Industry Awards

36

THE NEW ERA OF TEST AUTOMATION The true cost of buggy software can amount to more than just lost revenue, so what can testers do to help mitigate such problems? hat is the true cost of a buggy software? WorldPay, the renowned payment processing firm, that processes 36 million payments, suffered major glitches for three weeks due to software failure, while British Airways faced major IT failures in 2017, impacting their call centers, website and mobile apps and affecting around 70,000 passengers as flights were cancelled. All of this, left experts predicting that BA could face a final bill of over ÂŁ100million in compensation costs.

And yet the true cost of a software failure is not restricted to loss of revenue. There is significant reputation and competitive risk associated with poor quality software. In this age of rapid digital transformation, customer experience drives business strategy and technology. To meet the rising

W

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

expectations of the digital customers, enterprises have embraced the agile and DevOps approach. Software quality and testing are critical to meet the release velocity, quality and stability required by customers' digital experiences.

Test and QA teams have adopted test automation to meet the mandate of speed, greater complexity, and error-free releases. Software test automation led the first wave of change that provided scale, efficiency and faster time-tomarket for the QA process.

Continuous integration (and continuous testing) has shrunk the testing lifecycle further, adding to the efficiency gains.

The agile mantra is 'high quality at speed'.

Yet, businesses still struggle with a sluggish test suite – slowed down by a mounting QA backlog, automating large volumes of test cases, poor visibility and inadequate coverage.

VISHAL JHALA DIRECTOR - PRODUCT DEVELOPMENT INFOSTRETCH Vishal has 15+ years of experience in software product development across quality engineering, security & access control systems and supply chain management.

He is passionate about technology and has worked across a broad range of technologies & platforms. He holds a Master of Computer Science degree from Sardar Patel University.

T E S T

THE NEW WAVE IN AGILE TESTING

'Programs mirror human logic, but they don't mirror intuitive thought' – Rich Friedrich, director of systems software at Hewlett Packard Labs.

In other words, test automation needs to be smarter and more intuitive. It should learn from trend analyses of legacy data so that you can anticipate error-patterns and avoid re-inventing the wheel. This is where disruptive technology trends like BOTs, artificial intelligence (AI) and machine learning (ML) come in to transform the entire quality lifecycle, as we know it.

Intelligent test automation heralds the third wave in the test automation or DevOps journey, with its pre-emptive, prescriptive and predictive approach to quality. Simply put, intelligent testing uses AI and ML to address the pain points that organisations face, by introducing data-driven insights, predictions and recommendations.

It can thus automate, optimise and continuously improve the software development lifecycle. This has shifted the needle from descriptive or reactive analytics to predictive and prescriptive

analytics. Some of the major challenges that organisations face today are huge amounts of test data and test results, redundant test cases, flakiness of tests and maintenance and decision-making with an overwhelming amount of information.

Intelligent testing tools can sift through high volumes of test data, analyse trends, decode patterns and forecast future trends and outcomes. The tools analyse structured and unstructured data gathered from defect management tools and automation test results and use this information to predict outcomes and suggest actionable insights.

The job is no longer simply to validate, but to automatically detect regressions and high-risk defects in apps.

Using this data-driven approach, software can predict failures, bottle-necks, error categories and productivity struggles across the project cycles. Is this enough coverage?

Are you testing more than necessary? What should you prioritise and focus on? This information is exceptionally valuable when you have a large QA backlog, or are looking at the release deadline for a complex application suite. With machine learning you can project data and take informed, proactive decisions.

A U T O M A T I O N

37

These intelligent insights help in deciding the next courses of action, improving the outcomes and ultimately creating a constant feedback loop.

QMetry's Intelligent Testing tools implement solution stacks that enable agility, efficiency and quality. Our end-toend automation led by AI and ML helps companies to: • optimise test coverage and test depth • increase reusability with data-driven testing • enhance quality of test suites with higher traceability and visibility • weed-out duplicates and dead test cases • predict outcomes and prescribe actionable changes • forecast accurate and insightful decisions for release-readiness, testing adequacy and risk index • improve the ROI while reducing time-to-market.

So, why not look into how you can leverage intelligent test automation to deliver scalable and risk-proof digital experiences faster, and with confidence?

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

38

2018 EDITION

With the software testing landscape changing so rapidly, we hope this guide outlining some of the best testing and quality assurance products & services available, will be useful to you and your business.

Supported by

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

20 LEADING TESTING PROVIDERS 2018

39

froglogic froglogic is a software company based in Hamburg, Germany with additional offices in the USA and Poland with over 3,000 customers worldwide. froglogic is well known for its automated testing suite Squish with its flagship product Squish GUI Tester and the code coverage analysis tool Squish Coco.

Squish GUI Tester is the market-leading, cross-platform automated testing tool for GUI applications based on Qt, QML and QtQuick, Java AWT/Swing, SWT/RCP and JavaFX, macOS Carbon/Cocoa, Windows MFC, .NET and WPF, iOS, Android and for Web and HTML5 applications running in a variety of web browsers.

The Squish GUI Tester is an object based GUI test automation tool and works on desktop PCs, mobile and embedded devices running Windows, Linux, macOS, Solaris, iOS, Android and several embedded platforms such as embedded Linux, QNX and WinCE. Due to Squish GUI Tester’s advanced features including powerful scripting, remote testing, robust object and image verifications and integrations into frameworks such as Jenkins, Bamboo, TeamCity, HP ALM, MS ALM and more, froglogic’s customers save tremendous

amounts of time testing their GUI and HMI applications.

Squish Coco is a professional, cross-platform C, C++, C#, QML and Tcl code coverage tool which allows to analyze how much of an application’s source code is being tested, whether there are redundant tests and which tests need to be added.

Squish Coco works with several compiler tool chains and supports all coverage levels from basic line and branch coverage up to MC/ DC and MCC. Advanced reporting such as comparing executions, patch analysis and automatic bug location detection makes Squish Coco a unique tool.

+49 40 78897137 sales@froglogic.com Gasstr. 18

Haus 1

22761 Hamburg Germany

Squish Coco can be easily integrated with all kinds of test frameworks such as GoogleTest, CppUnit, NUnit, Jenkins, SonarCube and more.

Both, Squish GUI Tester and Squish Coco, are fit for use in safety critical environments complying to IEC 61508, ISO 26262, EN 50128 or DO 178C. To find out more about froglogic and its products, please get in touch!

froglogic.com

QASource +1 925 271 5555 info@qasource.com 73 Ray Street Pleasanton

California 94566 USA

qasource.com

The Leader in Pure Play Software QA and Testing By offering innovation, strategies and processes that eliminate inefficiencies and produce quality software products — combined with test automation expertise, industry-leading visibility, strategic planning, and metrics — QASource makes the decision to outsource easy. We invite you to learn more at qasource.com.

WHAT is QASource?

Headquartered in Silicon Valley, and with offices in Mexico and India, QASource is one of the world’s leading software testing providers: • QASource is an organization of 700 professionals servicing clients worldwide, ranging from VCfunded startups to Fortune 500’s such as IBM, Oracle, and Cisco • We deliver international-standard quality, strict adherence to deadlines, and total satisfaction WHO is QASource?

Our highly skilled engineers take your quality assurance from the initial design phase to completion, providing quality assurance every step on the way: • They are empowered to make product and procedural recommendations to clients and serve them at the highest level. Testers are encouraged to focus on creative solutions and challenge the status quo • They are hired for their depth of test and software expertise, and are continually trained in new

testing methodologies, effective communication, and best-practice processes • Our CEO is personally involved with every one of our partners What can we do for YOU? We focus exclusively on how we can best serve our partners, and only execute on strategies and tactics that drive value: • We can extend your QA department in less than 24 hours, without the associated setup and hassle • We are relationship-oriented — so much so, that we boast a 94% client retention rate over 15 years • With an emphasis on time-bound delivery and customised solutions, we help our partners manage the quality of their deliverables while keeping costs low • Our 91% employee retention rate means our engineers provide clients with stability and continuity, and therefore an institutional depth of knowledge not found in most engineering teams • Our hybrid “onsite-offshore-nearshore” service model combines with U.S. management — enabling organisations to mitigate the risks that can accompany a remote testing team • Our portfolio of products includes our core QASource offerings; MyCrowd, a crowdtesting option; and QAOnDemand, a pay-as-you-go model

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

20 LEADING TESTING PROVIDERS 2018

40

Mobile Labs Truly agile mobile app deployment overcomes challenges developers and testers face sharing devices, collaborating effectively, and coping with the plethora of device types and OS versions. The right technology and expertise get teams moving faster, keeps them moving, and helps achieve both higher quality and reduced time to market.

Since 2012, Mobile Labs has been the market-leading provider of in-house device clouds that bring order and agility to mobile testing and development by connecting remote, shared smartphones and tablets to mobile web, gaming, and app engineering teams. Its patented mobile device cloud, deviceConnect™ is available on-premises or hosted in a secure data center and features dedicated devices that provide high levels of security in a costeffective manner. deviceConnect gets teams moving with world-class manual testing performance, easy to manage continuous integration and continuous testing, and an “instant-on”

Appium server and scripting productivity tools. Developers and testers can debug, check code, and test using Xcode or Android Studio. Our high-performing Appium server speeds script startup and runs more concurrent tests. Powerful realtime metrics help engineers fine-tune app performance.

With deviceConnect, enterprise mobility teams can manage cloud access and activity. Designed for both testing and development teams, deviceConnect's powerful console manages, tracks, and monitors devices and apps. Engineers – regardless of location – can remotely access and share Android™ and Apple® smartphones and tablets.

+1 (404) 214 5804 info@mobilelabsinc.com 3565 Piedmont Road NE

Suite 400

Atlanta, GA 30305

Mobile Labs wants to be a partner in your success. Try our advanced technology and get to know our world-class support team. Welcome to the future of mobile application testing, mobile website testing, and mobile device sharing in the cloud.

It’s secure, fast, cost-effective, accurate, and easy. For more information please visit mobilelabsinc.com

mobilelabsinc.com

Panaya 

Ronit Eliav Director of Product Marketing +972 9 7618000 reliav@panaya.com

panaya.com

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

Bringing the Promise of Agility to Enterprise IT Panaya is the leader in continuous delivery of innovation to the world of enterprise IT applications.

With Panaya, IT and Business leaders can accelerate and promote change so that enterprise applications meet the rising pace of today’s business.

Panaya enables organisations to accelerate application change and continuously deliver innovation with its Enterprise Agile Delivery Platform. We provide cloud-based application delivery, and testing solutions that ensure collaboration between Business and IT. Enabling enterprise agility with faster release velocity and uncompromising quality, Panaya delivers an optimized user experience with end-to-end visibility of the application lifecycle.

Enabling the Agile Enterprise • Enterprise Agile Delivery Platform - Panaya provides a unique process-centric approach to application life cycle management with Panaya Release Dynamix (RDx), a cloud-based enterprise agile delivery platform. From initial business change request through to testing and production, Panaya delivers the value of agile, continuous delivery to the world of enterprise applications • End-to-End Testing - Panaya Test Dynamix is a test acceleration platform that helps organisations accelerate and standardize

packaged application testing without compromising on scope or quality. By encompassing the needs of all stakeholders and integrating standardized testing processes for both technical IT and Business users, Panaya reduces overall testing effort by 30-50%.

Powered by machine learning capabilities, Panaya Autonomous TestingTM for SAP eliminates the pains associated with creating and maintaining manually engineered scripts and enables zero-touch test case creation and maintenance • Delivery Impact and Risk Mitigation - Panaya utilizes real-time code analysis of your SAP, Oracle and SFDC application to predict the full impact of change before implementation.

Providing actionable insights into impacted transactions, Panaya ensures that impacted areas have test coverage before the changes are made.

Customers Panaya’s customer base spans over 2,000 enterprises worldwide, including over 220 of the Fortune 500 companies. There are more than 300 Panaya Test Dynamix customers and over 50 ALM replacements across various markets including Utilities, Oil & Gas, Automotive, Manufacturing, Pharmaceuticals, Infrastructure and Services.

20 LEADING TESTING PROVIDERS 2018

41

nFocus Testing As the longest established UK-owned independent testing consultancy, and previous winners of the prestigious Leading Vendor award from the European Software Testing Awards, nFocus are ideally placed to provide your software testing services.

quickly identifying strengths, weaknesses and quick wins with a roadmap for improvement.

Health Check - We can help improve your overall software testing and quality assurance approach by reviewing the health of your practices and providing a roadmap for improvements.

Test Automation Managed Service - We can help businesses to save money and reduce the maintenance headache of test automation through a world-class managed service.

Flexible Resource – Getting products and services to market quicker by extending your internal test team with reliable and friendly testers; as and when you need them.

User Acceptance Testing - We protect our clients’ interests and investment by undertaking thorough UAT of development projects done in-house or by a third party.

Outsourced Testing - Flex, scale and manage the high peaks of your testing requirements by calling on a partner that specialises in testing.

Load and Performance Testing - We help our clients performance test their systems to ensure they meet their non-functional requirements when under high load and peak demand.

Test Automation Readiness Assessment Helping you to identify the right approach, tooling and skills needed to successfully implement test automation to enable your adoption of Agile and DevOps. Test Automation Health Check - Helping you improve your test automation approach by

Test Automation Consultancy - We help businesses to implement, improve and maintain test automation to enable Agile and DevOps.

0370 242 6235 info@nfocus.co.uk e-Innovation Centre

Shifnal Road, Telford

Shropshire, TF2 9FT

Mobile App Testing – Utilise our huge range of device and OS combinations with the nFocus device lab.

Penetration Testing – We maximise efficiency through repeatable automated security testing of your web apps and infrastructure.

nfocus.co.uk

Validata Software +44 (0)20 7698 2731 info@validata-software.com

Achieve Digital Experiences through intelligent test automation

and ‘delighting’ customers by meeting their business goals.

Validata Group is the leader in the Enterprise Software market for DevOps and Continuous Testing for Temenos Core Banking and WealthSuite , delivering on the promise of full automation and predictive analytics and ensuring our customers achieve agility @ scale to unlock and accelerate their digital transformation initiatives.

Validata is the most mature, easy to use, scriptless, no-code platform that radically accelerates the delivery of digital testing powered by industry-leading modelbased test automation, agile test data management and provisioning, service virtualisation, and more. Leveraging AI, machine learning and analytics it intelligently navigates applications, searches for bugs and increases user experience with better quality, increased productivity and lower costs.

It provides the only unified DevOps and Continuous Testing platform, capable of integrating with both open source and commercial software such as HP QC, CA Clarity, Atlassian Jira etc, and delivering a ‘single version of the truth’ across all projects. It enhances DevOps pipelines, linking business, development and operations enabling faster product delivery with visibility ensuring an optimal digital experience

It offers a unique approach, focused on the only indicator that matters: The End User. It validates, predicts and monitors the user experience both before and after each release, across multiple channels of engagement focusing on testing the true, end-to- end user journey.

validata-software.com

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

20 LEADING TESTING PROVIDERS 2018

42

Vornex Vornex is a provider of enterprise software testing solutions. Our flagship product, TimeShiftX is a date and time simulation software that lets you time travel software into the future or past for temporal testing or date shift testing to validate all date and time sensitive functionality and code such as yearend, daylight savings, leap year, billing, rates, policies, etc.

TimeShiftX enables time travel (even inside Active Directory & Kerberos) without code changes, manual work, or server isolation in order to perform your forward date or back date testing with features including: • Instant Time Travel No code changes, no altering server clocks, no isolating servers. Just turn on TimeShiftX & start time traveling • Active Directory Compatibility Safely time travel inside Active Directory, Kerberos, LDAP, and other domain authentication protocols • Total App & DB Compatibility TimeShiftX enables time travel for all applications & databases such as SAP, SQL, Oracle, WAS, .NET, and others

• Cross Platform, Cloud, Container Support TimeShiftX is compatible on all platforms & operating systems and can be run in the cloud and inside containers • Distributed Environment Time Shifting TimeShiftX allows you to easily temporal test large, distributed software stacks and environments.

TimeShiftX +1 408 713 1400 info@vornexinc.com 43575 Mission Blvd. Suite 613

Fremont, CA 94539 USA

Vornex services worldwide clients with date shifting who extend many industries including Banking, Finance, Healthcare, State & Local Government, Utility, and Insurance. In addition, Vornex works with industry leading IT consultants, 3rd party integrators, and VARs such as Accenture, CapGemini, Wipro, IBM Global Services, etc.

All these industries and enterprises rely on TimeShiftX to validate their date sensitive applications and accelerate their time of delivery. Vornex enables organisations to increase efficiency, reduce resources & cost, and improve corporate bottom line by providing TimeShiftX as an affordable solution that can fit any company’s IT budget.

vornexinc.com

SofLab

+48 22 211 26 02 soflab@soflab.pl 103 Ligocka Katowice 40-568 Poland

soflab.pl

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

Soflab Technology is the largest testing and QA company in Poland. With offices around major cities in Poland, our staff of over 200 employees has completed over 4000_different projects across Europe in the last 9 years. Recently, our projects have also expanded to the EMEA region as well as East Asia. Our services include but are not limited to:

affiliation with any IT system suppliers, our work is objective, focusing on the success of our client’s requirements. We support our clients in ensuring the reliability of implemented software and maintaining business continuity. We provide accurate reports on the quality of inspected IT solutions.

• • • • • • •

Our clientele consists mainly of international telcos, banks, insurance companies, media, public entities, and IT integrators. Soflab is the first company in Poland that has won and implemented two large-scale full testing and QA SLA-based outsourcing projects (for an international telco group and a major national public entity).

Functional software testing Performance testing Security testing Mobile application testing Test automation Audits, advisory, quality management Test tools, data, and test environments

Our employees hold over 150 ISQTB certificates as well as several other IT-certified skill sets. This includes QAMP, Prince2, AgilePM, Professional Scrum Master, SAFe Consultant, and ScaledAgile.

Our experts are actively involved in QA events around Europe, implementing best practices in our projects in Poland and abroad.

As an independent company with no

Offering an extensive range of high quality of services with a high level of technical and analytical skills, freedom of movement within the EU, and a customer focused approach at highly competitive rates, it is no surprise why we are identified as a gazelle company in Poland.

20 LEADING TESTING PROVIDERS 2018

43

Keytorc Software Testing Services Since 2005, Keytorc Software Testing Services has been assisting its clients to manage their critical software testing processes to reduce the total cost of producing high-quality systems. The company has also horizontal expertise in diverse domains as; banking, insurance, telecommunications, defense, IT-High-Tech, retail, and e-commerce.

Besides being the regional leading testing services provider, Keytorc enhances its business in Europe combining technological proficiency with competitive service rates. This year, Keytorc plans to expand by opening its 4th branch in London.

Inspired people creating 'the new' as 'a team' After the establishment of its R&D center in the largest technology campus in Istanbul, Keytorc has started providing significant improvements and innovations in software testing area, especially in continuous testing, test automation, performance testing and static code analysis fields.

From a definitive view, it becomes more crucial to support agility with solid engineering practices.

Shorter iterations and pressure of delivering a viable product for each iteration make quality and testing activities more important. At this very point, Continuous Testing makes it enable to manage end-to-end software lifecycle integration reducing cycle-time. Keytorc provides top-notch Continuous Testing services disrupting existing practices.

Keytorc also focuses on Code Quality and Analysis services leveraging high-quality software development activities and ensuring technical

maturity. By this way, it becomes possible to figure out not only defects but also potentially problematic areas contributing Technical Debt.

Besides, Keytorc has valuable achievements in reputable international testing competitions and events. The company holds, attends and supports prestigious testing events and conferences in national and international scale. Keytorc is the organizer of the first international online and offline combined software test automation hackathon: Test Hackathon. The event has an international Judging Panel in which there exist experts from various countries and domains and the contest offers many benefits and prizes to test automation people from all around the world.

Services Provided by Keytorc: • Test Outsourcing, Near-Shore Testing Center of Excellence (TCoE) • DevOps / Continuous Testing • Code Quality and Analysis • Software Test Automation • Performance Testing • Mobile Application Testing • Test Data Management • Test Process Improvement & Test Maturity Assessment • Test Lifecycle Management • ISTQB® Accredited Software Testing Trainings Overview of Places, Projects and Clients: • Services in 4 Continents • Projects in 29 Countries • More than 1000 Clients

+90 212 276 06 40 info@keytorc.com Maslak Is Merkezi

Maslak - Istanbul 34398 Turkey

keytorc.com

DVT Inspired Testing by DVT



Bruce Zaayman +44 (0) 20 3946 7503 bzaayman@dvt.co.uk City Pavilion Cannon Green 1st Floor 27 Bush Lane, London EC4R 0AA

dvt.co.uk

With over 20 years’ experience, software development and testing company DVT has created an Inspired Testing model, which turns testing around by introducing quality at the start of the software process.

Traditional testing models rush in quality at the end of the development cycle, but Inspired Testing utilises a fresh approach that sees testing as an integral aspect of the software development cycle, from inception to completion. Our model views quality as a driver of process, with compliant systems, integration points, portals and Apps forming a powerful technology interface, which ensures that software quality is never compromised.

The minimising of functional failure, in fact, becomes an end user retention strategy.

In conjunction with on-site test solution architecture and functional testing, we focus on automation and performance testing for long-term return on investment. We know how to structure, execute and automate testing, using a unique combination of experience, frameworks, techniques and a blended onshore offshore delivery model.

Our flexible, scalable resource pool with 300+ QA professionals provides our clients with the best skills and most competitive prices, to achieve measurable return on investment.

DVT’s Inspired Testing model utilises Solutions Architecture and a Test Automation approach that complements our clients’ journey of digital transformation. We use a blend of custom-built tools and leading global products to deliver significant improvements in quality, utilising an automation framework for long-term benefit. Our model provides testing and seamless integration across all platforms, devices and environments. We understand that ultimately what we do makes a difference to the lives of our clients’ customers.

DVT is a wholly owned subsidiary of Dynamic Technologies, a software and technology group with 1000+ staff and eight group companies providing a diverse range of technology solutions, digital services and related core competencies. Our group companies comprise DVT (which includes the DVT Academy), Cloudsmiths, EventSmiths, Swarm, Blue Pencil Consulting, Emerald Consulting, Dynamic DNA and DTH Services.

Visit Dynamic Technologies at dynamic-tech. com.

Inspired Testing provides a measurable return on investment and ongoing real-time results, in an environment where software must work perfectly, every time.

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

20 LEADING TESTING PROVIDERS 2018

44

Sii Sii is the leading provider of IT and engineering services in Poland, employing over 3700 specialists and delivering solutions for business partners from all around the world – including USA, Japan, Norway, Sweden, Germany and Great Britain. The company supports its clients within the scope of testing, software development, infrastructure management as well as system integration and system maintenance.

As an offshoring leader in Central Europe, Sii has profound experience in international cooperation. The company executes projects for leading worldwide companies representing various sectors, such as banking and finance, insurance, telecommunication, hi-tech, utilities, transportation & logistics, FMCG and many others. Thyssen Krupp, Leica Geosystems, mBank, Fresenius, Scalepoint, Berlingske Media, Widex, Sabre, T-Mobile are some of Sii’s many clients.

Sii is first and foremost a technological expert, whose technical skills focus around 12 Competency Centers: Testing, Digital, Business Intelligence, Dynamics 365, Office 365, Salesforce, SAP, Engineering, Embedded, Legacy Systems, Business Process Automation and IT Operations. These are specialized units, within which highly experienced specialists provide their expertise, develop their own skills and share knowledge. In Sii projects are carried out based on the Lean and Agile methodologies, which

guarantee flexibility and a full compliance of the service with the customer’s needs.

Sii is the biggest testing services provider in Poland with over 700 testers and more than 100 projects carried out for local and international clients. The Testing Competency Center offers a whole range of testing services for web applications, web services as well as mobile and desktop applications. It supports clients in the process of Digital Transformation and helps them to implement Agile and DevOps through Continuous Testing. Automated tests, that are carried out for clients, focus on UI and API layers, allowing for a test automation rate to increase even to 85% per project.

+48 22 486 37 37 contact@pl.sii.eu al. Niepodległości 69

02-626 Warszawa Poland

The Center supports its clients in planning and designing tests and test processes to cover business risk and at the same time decrease the amount of performed tests to allow companies to make frequent changes in systems without the risk of failure.

Currently Sii has 12 branches in major Polish cities: Warsaw, Gdansk, Cracow, Wroclaw, Poznan, Lublin, Lodz, Katowice, Rzeszow, Bydgoszcz, Czestochowa and Pila, and is planning to expand to other locations both in Poland and abroad. For more information please visit: www.sii.pl/en or directly contact our Business Manager Szymon Wroblak at swroblak@pl.sii.eu.

sii.pl/en

iSQI

+44 118 9000 700 exam@isqi.org iSQI Ltd.

71-75 Shelton Street London

WC2H 9JQ

The International Software Quality Institute (iSQI) is a leading provider of certification examinations all over the world with iSQI Group companies based in Germany, the UK, the Netherlands and the USA. Our capacity is further extended through a partner network and local offices.

For almost 15 years, iSQI has played a significant role in certifying the know-how of IT professionals in over 100 countries on 6 continents in 10 languages. With over 30,000 certifications per year, iSQI paves the way for successful career development.

Constantly changing job requirements generate a demand for lifelong learning.

To stay competitive in today´s job market, professionals need to possess comprehensive and comparable skills and knowledge. The iSQI ‘Certified’ Program includes certification examinations that set the global standards in Software Testing, Requirements Engineering, Agile, Business Analysis, Mobile App Testing as well as Usability and User Experience etc.

isqi.org

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

iSQI examinations are supported by a global network of accredited training providers.

Their expertise in education assists professionals in polishing their knowledge and skills for the certification exams. iSQI’s ISO 9001 compliance is the independently confirmed guarantee for its business partners that all contracts are fulfilled in a secure and high-quality manner. Its ISO 17024 compliant certification processes meet all strict requirements in the day-to-day business practice. iSQI assembles professionals by regularly hosting and attending international conferences. These platforms for experts from the industry and academia foster experience exchanges and business development as well as advancements towards software quality engineering as a matured discipline.

To find out more about iSQI visit: isqi.org

20 LEADING TESTING PROVIDERS 2018

45

Edgewords Edgewords: The Training Specialists for Automated Software Testing With over 15 years’ experience of delivering training to Software Testers across the globe, Edgewords is the trusted training company for the Software Testing industry.

Our Trainers: Edgewords has been delivering training in Automated Testing Training for over 15 years.

We are Ranorex and HP training partners. Our trainers are all professionally qualified and deliver the highest quality courses; providing you with the ideal combination of theory and practice.

Cucumber (Java and JavaScript) • Appium • JMeter • LoadRunner • Performance Testing • UFT and LeanFT (UFT Pro) • Ranorex Studio • ALM.NET and ALM Octane

+44 (0)333 939 8884 info@edgewords.co.uk Lasyard House,

Underhill St.,

Bridgnorth WV16 4BB

For Testing Teams of three or more, we can provide on-site training at your offices we provide all you need; laptops, projector, materials.

Our Delivery Methods: • Public Schedule of courses in Birmingham & London • On-site Training at your offices • Online Training; Live instructor-led training via any web enabled device, delivered internationally

This is a highly effective and cost-efficient way

Our Courses: • Selenium WebDriver Java, C# and JavaScript (all include Continuous Integration) • Behaviour-Driven Development (BDD) with SpecFlow • Behaviour-Driven Development (BDD) with

best practices and our approach is to get you up

to ensure your test team is maximizing the use of your automated testing tool of choice.

At Edgewords, we are all testers. So our philosophy is to write and deliver training that we know is useful in the real-world. All of our trainers have worked in industry, so we know and running with the automated tools as quickly as possible.

For more information or for a quote, call +44(0)333 939 8884 or visit our web site.

edgewords.co.uk

QMetry Digital Quality Platform +1 408 727 1101 marketing@qmetry.com 3200 Patrick Henry Drive

Suite 250 Santa Clara

CA 95054 USA

QMetry’s Digital Quality Platform is tailor-made for the digital transformation challenges of Agile and DevOps teams.

It enables enterprises to match the pace of agile software development with a range of modern and intelligent testing tools. Covering the full gamut of the QE lifecycle, QMetry covers test management, automation, AI and ML driven predictive and prescriptive quality analytics to optimize test coverage, reduce duplicates and shift left with quality engineering.

QMetry is trusted by 700+ customers globally across Finance, Banking, Insurance, Healthcare, ODMs, Hospitality, Retail, and High-tech verticals.

qmetry.com

QMetry Test Management is an enterprise – grade, Cloud / On-Premise Intelligent test management tool designed to manage manual, automated and CI/CD based testing projects. It offers BDD driven test case creation, cross-project reusability & functionality, test execution, and customised

reports and smart dashboards. It has integrations with Jira, Bamboo, Jenkins and other major testing tools. QMetry supports e-signature and SAML. QMetry Automation Studio is the omnichannel test automation product for web, mobile & web services with BDD/TDD support. Built on Selenium and Appium, QAS offers automation scripting in Java now and will support 4 other languages like Python soon. QAS delivers functionalities such as object spy, web recording with exploratory testing, script-less automation and detailed reports.

QMetry Digital Quality Platform is built to deliver ROI, increase efficiency and reduce time to market through better test management, effective test automation and actionable insights.

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

20 LEADING TESTING PROVIDERS 2018

46

Infuse Infuse is a UK headquartered IT Software and Services Business enabling modern software delivery that is providing a distinct alternative to its competitors, by helping organisations get the most out of their software.

Infuse is recognised as the leading provider of test automation and lifecycle transformation for the most successful and ambitious organisations.

Infuse prides itself on its ability to provide innovative solutions, especially in test automation. They have been automating mobile testing since 2003 and developed a functional script-less test automation tool, useMango™.

Infuse leverages strategic consulting partners as well as its technology partners to deliver the best solution for you. They effectively join skillsets that rarely exist within one supplier. Infuse is a Gold Partner of Micro Focus and Oracle, and has a partner network including CA, Microsoft, AWS and SAP.

+44 (0)20 3755 5135 info@infuse.it Infuse Consulting

Broad Sanctuary, Westminster,

London SW1P 3EE

Since 2002, Infuse has successfully helped over 500 customers globally to get the most out of their software.

“Infuse helped us reduce 4 weeks of manual testing to 11 minutes with automated testing.”

Nigel Summers, Test Manager University of Bristol (Student Lifecycle Support Programme)

infuse.it

KRONE | consulting services | +421 911 365 779 info@kroneit.com Suché Mýto 6,

811 03 Bratislava, Slovakia

"We don’t just test. We identify and unblock the blockers to get the projects delivered."

KRONE | consulting services | is leading technology company with focus on testing and QA. With delivery center in Bratislava, Slovakia we provide solutions primarily to clients in Austria, Germany and Switzerland.

Key domains: • Banking • Insurance • Automotive Our team skills: • 100% of our Delivery Experts are ISTQB Certified • 100% of Employees speaks English • 65% of Employees speaks also German

kroneit.com

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

Our services include but are not limited to: • Functional software testing • Performance testing • Mobile application testing • Test automation • Test management • Audits, advisory, quality management • Test tools, data, and test environments

Our experts are regularly involved in QA events around Europe and we are investing back to testing community by organizing biggest testing conference in CZ / AT / SK region – Testing United (testingunited.com) As an independent company with no affiliation with any IT system suppliers, our work is objective, focusing on always delivering added value to our clients.

Cultural closeness of Slovaks and people from DACH region, orientation on results and great technical and language skills of our Employees allows us to quickly adapt to our clients’ environment and fit into their teams.

We heavily invest into our people’s qualification and further development to cover not just today’s topics like Test Automation in Tosca or Selenium or multiplatform testing, but also to gain experience with emerging technologies like Machine Learning and IoT. This allows us to keep high quality of services and keep delivering in the future.

Not just testing, but added value.

20 LEADING TESTING PROVIDERS 2018

47

Traffic Parrot for Microservices At Traffic Parrot, we help teams working with microservices accelerate delivery, improve quality, and reduce time-to-market by providing a tool for API mocking and service virtualisation.

We also provide consulting services on test automation, API mocking and service virtualisation.

Accelerate delivery, improve quality and reduce time-to-market Traffic Parrot can help you with delivering your microservices: • Unblocking testers and developers waiting for APIs • Easier and faster to reproduce production bugs • Speed up setting up test data • Eliminate the need for scheduling time on environments • Reduce third-party transaction costs • Make your performance tests more reliable • Make automated builds run faster Enjoy developing and testing your applications Traffic Parrot is an API mocking and service virtualisation tool. It simulates APIs

and services so that you can test your microservice without having to worry about test data set up or environment availability.

Traffic Parrot is specifically designed to maximise developer and tester productivity when working in autonomous or crossfunctional product teams. Small footprint (less than 100MB of disk space), lightweight but powerful, supporting HTTP, JMS, IBM MQ, File Transfers, gRPC and more.

Exclusive offer for Test Magazine subscribers Get a free 60-minute consultation and see how we can help you deliver microservices faster and improve quality. Email testmagsep2018@trafficparrot.com Offer expires on the 30th of November 2018.

Next steps • Email testmagsep2018@trafficparrot.com and get a free 60-minute consultation • Download the free 14-day trial at https:// trafficparrot.com/download.html and follow the 3-minute quick start guide.

You can also email testmagsep2018@ trafficparrot.com for an extended 30day trial. Offer expires on the 30th of November 2018.

• Explore case studies https://trafficparrot. com/casestudy.html

TestDevLab +371 25423562 info@testdevlab.com Kr.Valdemara 8-9

Riga, LV-1010 Latvia

testdevlab.com

+44 793 88 47 800 sales@trafficparrot.com 20-22 Wenlock Road, London,

England, N1 7GU

TestDevLab was founded in 2011 by two ex-Skype engineers and today we work with product and engineering teams globally at both startups and Fortune 500 companies including Skype, WorldRemit, Orange, Microsoft, Twilio and Truecaller.

TestDevLab in a Nutshell: • 100+ ISTQB certified engineers • 500+ real devices for testing • We are based in Europe • We develop custom software testing solutions and products • 1.5 billion people use the software that we test

trafficparrot.com

Our Advanced Services Our clients value our dedication to creating advanced custom testing solutions and frameworks.

• Audio & video testing lab - audio and video testing under different network conditions with different devices to assess various metrics of media quality.

• Battery & data usage testing lab - with battery and data usage testing we can help determine leaks, compare results with your direct competition, make sure that new implementations are not introducing new leaks and more.

Our Products

Our clients can leverage TestDevLab’s extensive experience with software testing tools and a pool of over 500 devices (mobile devices, PCs, VR glasses, wearables etc.) to ship better products faster, overcoming the complexities of shipping software across a broad range of network and device environments.

Leveraging our extensive experience we've developed our own software testing products for mobile, web and backend testing.

We can work remotely and integrate in client's teams to provide software quality assurance services ranging from manual to automated testing, security testing, usability testing and test planning and management for mobile, web and backend testing projects.

• Loadero - tool that simulates traffic to web applications with tons of user configurations available (multiple browsers, region support as well as simulated media streams)

• Apimation - codeless functional and performance API test automation tool • Test48.com - cloud service for mobile (Android, iOS) application testing. Finalist of European Software Testing Awards 2017.

• Appbench.io - fully automated service that will scan your Android application for vulnerabilities and other security issues.

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

20 LEADING TESTING PROVIDERS 2018

48

A1QA At A1QA, we all come to work every day because we want to help our clients deliver reliable and robust software to their user base. To bring it to the table, everyone is searching:

+44 208 816 7320 info@a1qa.com 3900 S. Wadsworth Blvd.,

Suite 485,

Lakewood, CO, 80235

- QA managers search for the ways to deliver testing excellence and support efficient communication with the client’s team.

- Test automation engineers search and develop the best possible solution to automate QA processes for reduced testing effort, improved test case reusability, and faster time to market.

- Test engineers search for defects of all kinds and priorities, describe and report them.

The results of this continuous search performed by 600+ QA pros are always impressive: - Developed solutions meet the requirements, key features and applications are delivered faster, QA costs are optimised, and the customer opens up new markets with top-notch software solutions.

What is A1QA?

a1qa.com

• • • •

1500+ happy clients 600+ career testers Multiple engagement models 10 Centers of Excellence

Core services: • Functional Testing • Performance / Load Testing • Test Automation • Security Testing • Usability Testing • Accessibility Testing Value-added services: • Baseline Testing • Monitoring Setup • Benchmarking • Pre-Certification Audit • System / Data Migration Support • Localisation / Internationalisation Testing

A1QA is the global provider of software testing and QA services. The company is headquartered in Lakewood, Colorado with its business development office located in London, UK.

Consulting services: • Educational Services • QA Audit • Quality Strategy Development

Our Highlights • 15 years in software testing • 15 global locations across 2 continents

We invite you to learn more about our Quality Assurance solutions at a1qa.com.

Basis Technologies DevOps and Test Automation for SAP Basis Technologies provides a complete automated DevOps and testing platform engineered for SAP applications, featuring flagship products Testimony and ActiveControl. Their automation technology massively reduces the time and effort needed to execute change and testing. Users are able to maintain better competitive agility by responding more quickly to market opportunities while simultaneously reducing costs and lowering risk.

Many of the world’s largest brands, including BP, Honda, and Polo Ralph Lauren, use Basis Technologies’ automation to accelerate innovation and ensure continuous quality and agile delivery across SAP software landscapes.

Testing without scripts Testimony is a unique test automation solution that employs a revolutionary new approach – Robotic Test Automation (RTA) – to reinvent SAP regression testing.

Robotic Test Automation eliminates the pain and expense of traditional regression testing methods, whether automated or manual. It eliminates scripting and test maintenance, instead automatically learning and validating system behavior without user intervention to protect critical processes from unintended

T E S T M a g a z i n e | S e p t e m b e r 2 01 8

consequences of change, and provides the confidence your business needs to operate at high speed.

In addition to this increase in testing efficiency, Testimony reduces business risk by going beyond the UI. Server-side instrumentation of SAP systems is used to create test libraries with massively greater coverage, including interactions and processes like RFCs, BAPIs and batch jobs which cannot normally be addressed during regression testing.

+44 207 958 9025 info@basistechnologies.com

Benefits of Testimony • Eliminate manual effort: automatically create, execute and update test cases • Accelerate transformation projects, upgrades, and updates • Start testing in a matter of days for fast, sustainable ROI • Test beyond the user interface to increase confidence and reduce risk • Increase development efficiency by shifting regression testing left • Reduce the cost of testing • Liberate functional experts to focus on value-add activity • Enhance DevOps for SAP through automated continuous testing

basistechnologies.com

new for 2018 49

30th OCTOBER 2018 See the future of DevOps and Software Testing A one-day event held in Central Glasgow that attracts up to 100 senior software testing and DevOps professionals, eager to network and participate in targeted workshops, interactive Q&A sessions, and lively presentations – all housed in the impressive Glasgow Marriott Hotel.

DevTEST Summit is open to all individuals or organisations within the Software Testing and DevOps community who are keen to increase their knowledge and harvest workable solutions to the various issues faced in these burgeoning and complex sectors.

REGISTER NOW WWW.DEVTESTSUMMIT.COM T E S T M a g a z i n e | S e p t e m b e r 2 01 8

50

T E S T M a g a z i n e | S e p t e m b e r 2 01 8