TEST – May 2018 by 31 Media

MAY 2018

THE NATIONAL SOFTWARE TESTING CONFERENCE EVENT HIGHLIGHTS

PAGE

SMART CITIES FUNCTIONAL TESTING DEVOPS/AGILE INNOVATIONS EMBEDDED TESTING

C O N T E N T S

T E S T COVER

STORY:

M A G A Z I N E REMAINING

M A Y

COMPETITIVE

2 0 1 8 A

NEW

WORLD

SMART CITIES/DEVICES

Remaining Competitive In A New World ......... 4 Evolving From Smart To Smart ......................... 8

EVOLVING FROM SMART TO SMART

The Smart Outsourcing Option ...................... 12 NSTC Speakers Test Smart .............................. 16 Speak Up To Technological Advancements ... 22 The Future Is Smart ........................................ 24 FUNCTIONAL TESTING

Test Soon And Fix Faster ................................ 28 Ideal Testing Approach For IoT Ecosystem .... 34 The Functional Testing Conundrum............... 38 DEVOPS/AGILE INNOVATIONS

Driving The Digital Change ............................ 42 Innovation: The Holy Grail Of Business ......... 46

TEST SOON AND FIX FASTER

Reimagining Quality Assurance ..................... 50 The Phoenix Of Software Testing ................... 54 EMBEDDED TESTING

Climbing The Pyramid .................................... 56 Building CX Assurance Into Enterprise .......... 60

INNOVATION: THE HOLY GRAIL OF BUSINESS

From Traditional Qa To Agile .......................... 62 Future Testing Challenges .............................. 64 Delivering Consistent Value ........................... 66 EVENTS

National Software Testing Conference .......... 70

THE FUNCTIONAL TESTING CONUNDRUM

T E S T M a g a z i n e | M a y 2 01 8

2 upcoming

INDUSTRY EVENTS

12-13

18-19

16 -17

June

Sept

Oct

The DevOps Conference NORTH will be taking place for the first time ever at the luxurious Principal York Hotel, York, on the 12-13 June 2018. This unmissable event will provide the IT community with invaluable content from revered industry speakers who are IT Heads, Managers and Directors from the likes of Sky Betting and Gaming, The Royal Bank of Scotland, Chestnut Hill Consulting, Allan Kelly Associates, NHS Digital, The Independent, etc.; practical presentations; roundtable discussion forums that are facilitated and led by key figures; as well as a market leading exhibition, enabling delegates to view the latest products and services available to them. http://north.devopsevent.com

The Software Testing Conference NORTH will be held at The Principal York Hotel, York, on the 1819 September 2018. This northern conference will provide the software testing community with practical presentations, where the winners and finalists of The European Software Testing Awards will touch upon pressing industry topics; executive workshops led by industry assets; and will give delegates the chance to check out the latest products and services within the industry via an array of exhibition stands.

The National DevOps Conference will be held at the Millennium Gloucester Hotel, Kensington, London, on the 16-17 October 2018. This is a two-day conference designed to connect to a wide range of stakeholders and engage not only existing DevOps pros, but also other senior professionals keen to learn about implementing this useful practice. At The National DevOps Conference, you will have the chance to listen to peers who have successfully begun their DevOps journey; receive advice and knowledge from industry practitioners; as well as join in and debate at executive workshops.

http://north.softwaretestingconference.com http://devopsevent.com

Oct

Nov

The DevOps Industry Awards will be held at the Millennium Gloucester Hotel, Kensington, London, on the 16 October 2018, to celebrate companies and individuals who have accomplished significant achievements when incorporating and adopting DevOps practices. This glittering awards gala has been launched to recognise the tremendous efforts of individuals and teams when undergoing digital transformation projects – whether they are small and bespoke, or large complex initiatives.

DevTEST Summit Scotland is a one-day event, which will be held on the 30 October 2018 at the iconic Principal Grand Hotel, Glasgow, Scotland. The full days programme will see a panel of key professionals speak, before hosting interactive sessions on recent issues related to software testing and DevOps. DevTEST Summit is open to all organisations and individuals within the software testing and DevOps community who are keen to increase their knowledge and harvest workable solutions towards various issued faced in complex, burgeoning sectors.

For the sixth year running, The European Software Testing Awards will celebrate companies and individuals who have accomplished significant achievements in the software testing and quality assurance market on the 21 November 2018 at Old Billingsgate, London. Enter The European Software Testing Awards and start on a journey of anticipation and excitement leading up to the awards night – who knows, it could be you and your team collecting one of the highly coveted awards.

http://devopsindustryawards.com

http://devtestsummit.com

http://softwaretestingawards.com

THE EUROPEAN SOFTWARE TESTING

Nov The European Software Testing Summit is a one-day event, which will be held on the 21 November 2018 at The Hatton, Farringdon, London. The European Software Testing summit will consist of up to 100 senior software testing and QA professionals, who are eager to network and participate in targeted workshops. All delegates will receive printed research literature, have the chance to interact with The European Software Testing Awards’ experienced Judging Panel, as well receive practical advice and actionable intelligence from dedicated workshops. http://www.softwaretestingsummit.com

T E S T M a g a z i n e | M a y 2 01 8

E D I T O R ' S

AN INCREASINGLY CONNECTED WORLD LEAH ALGER JOURNALIST

echnology has the power to make cities around the world more efficient and tech-savvy, so it’s no wonder that an array of civil societies want to be a part of this technological revolution. Throughout the years, I’m sure we can all say we’ve witnessed technology taking over the way the world works – now it’s even managing transportation systems, power plants, traffic, law enforcements, schools and other community services. Aspects of cities are beginning to connect with Internet of Things (IoT) devices, because of its potential to completely change how a city works by enhancing the performance, quality and interactivity of urban services, as well as simplifying a person’s typical day by improving the quality of life. IoT gives organisations across private, public and non-profit sectors the opportunity to implement connected sensors, lights, and meters to collect and analyse data; which will help service quality, automate processes, and provide feedback to users. This, in the long run, helps create a smart city through smarter decision-making. Despite this, it depends on national governments and whether or not they want to support certain development processes – not forgetting the financial aspect – £10million should be enough to see good results of a smart city. But, even so, the budget would have to be spent wisely. Privacy and security in an increasingly connected world is a concern for many, bringing in an array of different debates. Interestingly, for a smart city to be as

effective as possible, it needs all data points as connected as possible. In other words, data silos need to be avoided. For example, if one system is used to store data about our waste container levels and doesn’t pool that data with footfall, we wouldn’t be able to predict a full bin based on foot traffic. (page 6) Another issue raised is the confidentiality of data, and, of course, potential cyber security challenges. Locational services and sensor analytics could create unexpected risks. The reason breaches occur is because functionality and customer orientation are still the highest priority for vendors, but even in times of increased connectedness, security and data protection is still neglected (hopefully the new GDPR will fix this). Nevertheless, I believe we should embrace this interesting time of life where we can focus more on broader human needs by helping to design an environment to suit our personal needs and unique ways of thinking. I personally can’t wait for the day when I get to sit in a car without having to drive it, my windows open when it knows I’m too hot, and the kettle turns itself on because of knowing I’m thirsty and in need of caffeine. Not forgetting, it would be great to have a robot to do my hoovering – or is that just pure laziness?

C O M M E N T

MAY 2018

THE NATIONAL SOFTWARE TESTING CONFERENCE EVENT HIGHLIGHTS

PAGE

SMART CITIES FUNCTIONAL TESTING DEVOPS/AGILE INNOVATIONS EMBEDDED TESTING

MAY 2018 | VOLUME 10 | ISSUE 2 © 2018 31 Media Limited. All rights reserved. TEST Magazine is edited, designed, and published by 31 Media Limited. No part of TEST Magazine may be reproduced, transmitted, stored electronically, distributed, or copied, in whole or part without the prior written consent of the publisher. A reprint service is available. Opinions expressed in this journal do not necessarily reflect those of the editor of TEST Magazine or its publisher, 31 Media Limited. ISSN 2040‑01‑60 EDITORIAL DEPARTMENT editor@31media.co.uk +44 (0)203 056 4599 JOURNALIST Leah Alger leah.alger@31media.co.uk +44 (0)203 668 6948 ADVERTISING ENQUIRIES Shivanni Sohal shivanni.sohal@31media.co.uk +44 (0)203 668 6945 PRODUCTION & DESIGN Ivan Boyanov ivan.boyanov@31media.co.uk 31 Media Ltd, 41‑42 Daisy Business Park 19‑35 Sylvan Grove London, SE15 1PD +44 (0)870 863 6930 info@31media.co.uk www.testingmagazine.com PRINTED BY Pensord, Tram Road, Pontllanfraith, Blackwood, NP12 2YA  softwaretestingnews  @testmagazine  TEST Magazine Group

T E S T M a g a z i n e | M a y 2 01 8

REMAINING COMPETITIVE IN A NEW WORLD Regional Technical Services Manager, Amanpreet Singh, reveals a narrative approach of what he thinks life might be like in a smartÂ city

T E S T M a g a z i n e | M a y 2 01 8

S M A R T

s a species, we have come an extremely long way with our ability to communicate. From the earliest cave paintings dated to 62,000 BC, the Yam route developed by Genghis Khan, and of course, the holiest of resources today, the internet. The commoditisation of sensors allows us to wirelessly connect any device to the internet; known as the internet of Things (IoT). Using electronic sensors, which are: uniquely identifiable, addressable, able to collect and store data, and then communicate, is a huge enabler in forming a “smart city". This can give us a real-time picture of our assets and resources (Things). The PAS 180:2014 smart cities. Vocabulary defines smart cities as “The effective integration of physical, digital and human systems in the built environment to deliver a sustainable, prosperous and inclusive future for its citizens”. With a large enough data set, a computer can quickly identify trends and patterns and correlate them with the factors that cause them. With this capability, we can more accurately predict events before they occur and drive autonomous actions to react. Using data in this way is called big data and we can see the effectiveness of this within our smartphones. For example, your Google app automatically “learns” what you tend to search for and will push topics that it believes interest you. Having a processor seamlessly make decisions for us can add convenience, increase productivity, and remove boring tasks from our everyday lives.

A DAY IN A SMART CITY It’s 05:15 in the morning, your alarm buzzes and as you rise for the day it appears your home wakes up with you. The blinds retract, the lights come on and you receive a readout of today’s schedule. As you pour out the last of the milk into your breakfast, an order for a new bottle is automatically placed. At 05:45 it’s time to leave your house and your home assistant alerts you that your shuttle has arrived. You unlock the vehicle with your fingerprint and slide back into a plush armchair, sitting in what looks like a mobile lounge. In this space, you catch up on sleep during your 3-hour commute and ask to be woken up an hour before your meeting. There’s no awkward silence and you don’t ask the obligatory “when did you start?” and “how long are you on till?” because you are alone. You’re in an autonomous fleet vehicle, which you pay a monthly fee for, rather than owning

C I T I E S / D E V I C E S

a car. The hour after you wake gives you time to work at the desk in front of you, equipped with regular office amenities: plug-in sockets, a desk lamp, stationery etc. Street lighting only activates when someone is in its vicinity and you notice the road is dark behind you. As you get closer to your destination, your vehicle is alerted of an available parking space outside of your office. You leave the vehicle and walk up to your client’s office building. The doors are locked, however, you were provided with a virtual guest pass gives you access to the building. This is contained in an app on your phone, which also guides you through the building to your meeting room. Your client is waiting in the room; of course, they received a notification of your arrival as you swiped in. The gracious host hands you a cup of coffee made specifically to your liking (the same app allows you to set your preferences which are then relayed to the coffee machines). During the meeting, air quality sensors detect the room is becoming stuffy and warm. Window actuators automatically open to allow fresh air to enter the room an. After your meeting, you leave the building, and your guest pass becomes invalid. The shuttle is waiting outside after having completed a few local drops for other users while you were inside. It drives you to your local hospital for an afternoon appointment. A health-monitoring platform throws you an alert and arranged an appointment with you and your regular doctor. This platform is fed data in real time from smart devices such as your weighing scales, blood pressure machine and smart watch. Your health profile is moving towards someone with hidden obesity, which can cause issues such as diabetes or heart disease. Luckily, it’s been addressed before any major ailment was contracted, and you’ve been prescribed preventative lifestyle changes to diet and exercise, rather than reactively taking expensive medicine. On the way back to your house, you check your “Smart Home” app on your phone, which shows that your daughter got home at the usual time and her school has left a reminder in the diary that its parent’s evening is next week. A virtual billboard has appeared on the cars head up display, alerting you that you’re nearby a food outlet. You check the stock in the pantry from your phone and decide to order food from the establishment instead of cooking. As you retire to bed, you reflect back on what was a productive, convenient and connected day. Amazing, right?

Having a processor seamlessly make decisions for us can add convenience, increase productivity, and remove boring tasks from our everyday lives

AMANPREET JALIF REGIONAL TECHNICAL SERVICES MANAGER GOV FACILITY SERVICES LTD

Aman graduated in Electronic Engineering at Coventry University in 2014. After a gap year spent working and travelling, he joined a Graduate Development Scheme with Carillion's Engineering Centre of Excellence. He now provides Senior Engineering support to a large government department and explores new technologies available in the built environment.

T E S T M a g a z i n e | M a y 2 01 8

S M A R T

C I T I E S / D E V I C E S

Privacy and security in an increasingly connected world is a huge topic and concern for many people when looking at smart cities, bringing interesting debates

T E S T M a g a z i n e | M a y 2 01 8

SMARTER DECISIONS Using data to make smarter decisions is an obvious choice, and the aggregation of small efficiencies across the range of activities carried out by a city means smart cities use less energy, are more productive and make life more convenient. I can barely scratch the surface on how new technology will affect the various aspects of our communal living, but I will cover a few. Transport in the future will look very different, particularly car ownership. In the current model, the majority of a car’s life is spent switched off. I believe that we will move to an “on demand” driverless system that we pay a subscription for and these vehicles will be designed for our comfort and needs, rather than for us to drive. More carpooling options will be available in an environment where fine grain data can be taken advantage of. This reduces the number of cars on the road per number of journeys, meaning less traffic and potential for accidents. Furthermore, and this is a particular issue for my local council, we’ll have fewer cars spilling onto the roads in residential areas. Therefore, our suburbs will be greener, as the requirement for parking spaces will be reduced. Occupancy health and wellbeing is a phrase heard often in the built environment. It’s an interesting time to be in the industry where we are focussing more on broader human needs and designing the environment to suit. Ambient lighting, workspace music, sit-stand desks and air quality are just a few of the things now considered when designing the modern office. Businesses recognise that a healthier workforce is a happier one; meaning more productive and reduced absenteeism. Companies that focus on the varying and individual aspects of psychological and physiological human need will attract better staff that will stay long-term. A good example of this is the mobile phone manufacturer OnePlus, where customer service staff enjoy a lunchtime nap. The office goes dark and the office chairs recline to accommodate a midday snooze. I support this idea. Not just because I love a good nap, but because it’s a smart thing to do. Recent research suggests that humans evolved on a bi-phasic sleep pattern i.e. slept twice a day. Therefore, a business appealing to its employee’s intrinsic animal need makes perfect sense. That lull you feel just after midday is not just because of a heavy lunch!

POTENTIAL DRAWBACKS

Whilst embracing new technology is an obvious choice for societies that want to remain competitive in a new world, there are some potential drawbacks. A study by the University of Texas at Austin shows that our cognitive capacity is significantly reduced when our smartphone is nearby. This negatively affects our ability to complete tasks that require focus. I’ve noticed this impact myself and now take steps to ensure my phone is on silent and in another room whilst I’m working. Try it yourself and see the difference it makes. I have the feeling that the more data we make use of, the more our world will be able to push notifications at us, which doesn’t bode well for our cognitive capability. A smart city should be designed smartly, with ergonomic factors like this taken into consideration. For a smart city to be as effective as possible, it needs all data points as connected as possible. In other words, data silos need to be avoided. For example, if one system is used to store data about our waste container levels and doesn’t pool that data with footfall, we wouldn’t be able to predict a full bin based on foot traffic. A smart city requires cohesive design from a local authority, rather than plugging and playing as many new technology systems as possible that don’t “talk” to each other.

PRIVACY AND SECURITY Privacy and security in an increasingly connected world is a huge topic and concern for many people when looking at smart cities, bringing interesting debates. If I download an app on my phone, I have to agree to terms and conditions before I can use it. These T’s and C’s cover things such as what data is available to the app, how my data is used, and where it is stored. Do we have to agree to the terms and conditions of a smart city, and what happens if we don't? Our personal data will be anonymised I’m sure, but anonymised data can be de-anonymised as proven by a contest hosted by Netflix. Specific subscribers were identified from a Netflix’s anonymised data. Does this mean very descript pictures of personal lives could get into the hands of people we’d rather didn’t? As these considerations come to the fore in a smart city, it will engage more consumers in discussions about technology and data. This can only be good for the industry as a whole as people become more in tune with how Engineers have designed the infrastructure around them.

registrations are now open KEY DATES 24 August – Booking & Submission Deadline 12 September – Finalists announced 16 October – Awards Gala Dinner

he DevOps Industry Awards celebrate companies and individuals who have accomplished significant achievements when incorporating and adopting DevOps practices. The DevOps Industry Awards have been launched to recognise the tremendous efforts of individuals and teams when undergoing digital transformation projects – whether they are small and bespoke, or large complex initiatives. Enter The DevOps Industry Awards and start on a journey of anticipation and excitement leading up to the awards night – it could be you and your team collecting one of the highly coveted awards.

Be a winner - REGISTER TODAY

 www.devopsindustryawards.com T E S T M a g a z i n e | M a y 2 01 8

EVOLVING FROM SMART TO SMART Amparo Marin, CIO Technology and Regulatory Affairs at Santander EspaĂąa, explains how technology and collaboration is transforming our world at an unprecedented pace, from how we work to how we shop

T E S T M a g a z i n e | M a y 2 01 8

S M A R T

lthough the “smart city” concept is not something new, since it emerged more than a decade ago, it has been subject to continuous evolution. Back in 2004, the movement started with a worldwide programme to create a digital community. Companies from diverse sectors, such as telecommunications, security, construction, audio-visual, consumer electronics, utilities, computer science, healthcare and education, joint with governments and public administrations set the building blocks of the first digital cities that soon evolved to smart cities. There is not one definition for the smart city label, but one of the most accepted ones entails the following factors: 1. Complex interconnected systems that apply electronic and digital technologies to communities and citizen services (transport, energy resources, civil protection, communication...). 2. The use of ICT to transform life and working environments. 3. Identification and reaction to consumer needs, embedding such information in systems, knowledge management and practices that bring people together. Whether it is large capitals or smaller cities, there is a worldwide trend towards attaining the qualifying of smart. There are different non-excluding models that can support the development of smart cities, based on 4 fundamental dimensions: • Sustainability • Connectivity • Innovation • Social Cohesion The true metropolis should embrace all the

C I T I E S / D E V I C E S

dimensions above, but successful players have proven that focus on selected categories rather than tackling all of them at the same time is an effective strategy. New York, London, Paris, Barcelona, Singapore or Stockholm are real examples of advanced cities. Any tips that others can follow to progress and qualify as smarter? Here are 9 tips that can help to create the smart city roadmap: 1. Meet the needs of the present without compromising the ability of future generations to meet their own needs, through the improvement of environmental sustainability. This encompasses antipollution plans, support for green buildings and alternative energy, efficient water management, and policies that help counter the effects of climate change. 2. Adequate urban planning to improve the habitability of a territory, avoid over-costs and incentivise investment. Design local master plans, including compact, wellconnected and accessible public services, green areas and spaces for smart growth. 3. Improve the administration’s efficiency, redefining organisational and management models. There is a high correlation between public management and the state of public finances. And public accounts decisively affect people’s quality of life and a city’s sustainability, as they determine the level of taxes that must be supported by residents and the production system, the expected growth of the general level of prices, the possibilities of investment. 4. Improve the city brand and seek international outreach. Maintaining a global impact and having international recognition attracts foreign investment and contributes to developing representation abroad and strategic tourism plans. 5. Facilitate movement through cities and access to public services. Mobility and transportation affect the quality of life and can be vital to the sustainability of cities over time. The workforce’s need to commute as well as the need for an outlet for production generates externalities in the production system that need to be addressed. 6. Promote economic development of a territory: local economic development plans, transition plans, strategic industrial plans, and cluster generation, innovation and entrepreneurial initiatives. 7. Make technology an integral part of the society’s backbone. Technology must be

In order to execute strategic plans, it is necessary to acknowledge that cities cannot do it all alone. The transformation of a city is not an individual undertaking but instead a collective endeavour, so collaboration is essential

AMPARO MARIN DE LA BARCENA DIRECTOR T&O - CIO TECHNOLOGY AND REGULATORY AFFAIRS AT SANTANDER ESPAÑA

Telecom Engineer & PhD Candidate in quantitative models applied to the strategic and financial environments. With a clear focus on quality and execution, Amparo has spent a large part of her career understanding processes in order to challenge existing frameworks and design models driven by systemic innovation and continuous improvement.

T E S T M a g a z i n e | M a y 2 01 8

S M A R T

C I T I E S / D E V I C E S

Connectivity Sustainability

International Presence

Mobility & Transport

Environment Public Management Urban Planning Governance & Civil Participation Human Resources

Social cohesion

Economics Technology

Innovation

Development of smart cities should drive clear benefits but public sector leaders face the imperative to protect their cities from rising new threats and challenges

understood as the key enabler to gain and extend competitive advantages of a city’s productive system and quality of employment. A technologically developed Conditions for city prosperity city has proven significant comparative advantages in terms of security, education and health, which are basic to secure sustainability and the city’s capacity to optimise investment, consumption and labour productivity. 8. Set effective, quality and sound guidance of state intervention encourage citizens’ participation and involve business leaders and local stakeholders in the application of government plans. The citizen is the meeting point for solving all the challenges facing cities. Authorities’ ability to engage all parties towards the same goal is a key success factor. 9. Attract and retain talent: creating plans to improve education and promote creativity and research.

MOVING ONE STEP FORWARD Development of smart cities should drive clear benefits, but public sector leaders face the imperative to protect their cities from rising new threats and challenges, such as cyber attacks, multimillionaire investments in advanced infrastructure, easy access to public data, changing consumption patterns shifting to the “phygital” world. In the midst of the Digital Transformation wave, which most companies are undergoing, it is time to understand and design the future customer’s journey, reimagine the business and create a new user experience that will foster user adoption. The model for success relies on three core elements: • Strategy • Collaboration • Sound principles All of them are sourced by the power of innovation. Strategy: Cities need to define their Identity and establish a strategic plan. One of the most important questions a city has to ask itself is: What kind of city do you want to be? The answer will not only define the identity but also the path of transformation. Environment

T E S T M a g a z i n e | M a y 2 01 8

Urban Planning

Public International Management Presence

COLLABORATION “In order to execute the strategic plans, it is necessary to acknowledge that cities cannot do it all alone. The transformation of a city is not an individual undertaking but instead a collective endeavour, so collaboration is essential.” (Pascual Berrone and Joan Enric Ricart) The majority of leaders are clear on the imperative to evolve from the idealistic vision of a smart city to the realistic one. The objective of making “cities inclusive, safe, resilient and sustainable” is among the United Nations’ “17 goals to transform our world”. How to make this happen? Collaboration is the cornerstone. • On one hand, governments need to bring together key stakeholders and experts to enable that the right insights are available to inform decision-making. • It is essential to agree on a unified strategic vision, which will drive efforts from the different parties in an aligned way. • With a clear and aligned vision in place, stakeholders can collaborate to achieve prosperous and future development for their cities leveraging on technology and operational enablers.

SOUND PRINCIPLES S = Security M = Metrics A = Adaptation and inclusion R = Resilience T = Technology as an enabler of innovation and sustainable growth A SMART city must be able to detect people’s needs and react to citizens’ demands transforming human and systems’ interactions into knowledge. Delivery of secure and robust public services, ideally in real time and flexible enough to adapt and anticipate upcoming trends should be the key drivers of sustainable progress and continuous improvement.

Mobility & Transport

Economics

Technolog y

Governance & Human Civil Resources Participation

TEST Magazine is widely regarded as the leading journal for software testing and quality assurance in Europe and beyond with a Global reach over 20,000+ readers and followers.

ď&#x201A;&#x2022; +44(0) 203 931 5827

T E S T M a g a z i n e | M a y 2 01 8

THE SMART OUTSOURCING OPTION CEO of Future Processing says outsourcing is one of the only methods that can provide all-round support for implementing smart cities and IoT projects

T E S T M a g a z i n e | M a y 2 01 8

S M A R T

mart cities are no longer concepts that belong in the future. Today, there are various smart city initiatives around the globe, with IoT infrastructures leading the way in a world that is more connected than ever before. In Sau Paulo, for example, IBM used smart city technology to provide citizens with an app that could photograph problems with the infrastructure of the city (including sewage or waste problems) and to submit problems to city administrators for review. By doing this, IBM could map four square kilometres of land in a matter of hours; tracking waste, sewage, and traffic inconveniences. Alongside Sau Paulo, in Barcelona, sensors preserve the city’s water resources, saving the city both time and money. The city now has improved public transport services, as it has real-time information at bus stops and a simplified ticketing process. It is predicted that by 2020, the smart city data analytics market will be worth over US$400million, while the global market for smart city solutions is anticipated to climb from US$40.1billion to US$97.9billion by 2026. With clear smart city benefits, the demand for intelligent IT solutions to urban problems is on the rise. However, while the demand for smart city technology is on the rise, there is a talent shortage within the IT industry. While this may cause hiring problems for smart city developers and councils, outsourcing and nearshoring IT experts can provide much-needed talent to progress smart city development. But, why is outsourcing needed for smart cities – including the development of testing phases – and what specific benefits will outsourcing or nearshoring bring to smart city projects around the world?

INNOVATION AT A FASTER PACE With 65% of the world’s population expected to live in cities by 2040, city administrators need to prepare for massive city population growth by installing smart technology efficiently. Today, an outsourcing provider is an engine that drives efficiency through agility, ensured through unique creative ideas resulting from specific technical know-how. This drives companies

C I T I E S / D E V I C E S

towards innovation at a faster pace than they would otherwise and can be equally applied to smart cities looking to progress efficiently. As city administrators continue to transform their cities into the smart urban ecosystems of tomorrow, collaboration is becoming more and more essential for efficient, secure and effective implementation. To cope with the sheer demand for smart services, combined with a shortage of skills that is only growing, outsourcing can provide the skills that are crucial for smart city technology. For example, if a council or developer is struggling to hire or locate machine learning scientists, geospatial and mapping scientists, software testers, integration engineers, or urban informatics analysts (among others), they can outsource or nearshore IT teams to provide the expertise. With access to suitable staff, outsourcing providers can fill IT roles efficiently, preserving the product’s quality by allowing team members to focus on tasks that require their exclusive expertise.

Today, there are various smart city initiatives around the globe, with IoT infrastructures leading the way in a world that is more connected than ever before

OUTSOURCING’S COMMUNICATION CHALLENGES Although outsourcing talent from a distant country brings numerous benefits to the hiring company, it may also bring communication difficulties. With time zones, culture differences, and language barriers in place, outsourcing may not be the right choice for the smart city project. Instead, organisations should look to nearshore. The proximity of nearshoring typically ensures that two organisations share a

JAROSLAW CZAJA FOUNDER AND CEO FUTURE PROCESSING

Jaroslaw is the Founder and CEO of the leading outsourcing company, Future Processing. He currently plays an instrumental role in adapting the outsourcing industry to current demands and practices where the service provider now assumes a role similar to that of a business partner.

T E S T M a g a z i n e | M a y 2 01 8

S M A R T

C I T I E S / D E V I C E S

After the security of the smart city has been achieved, outsourcing IoT services can provide considerable help with optimising and integrating smart devices

common language and culture. Alongside this, outsourcing teams from Europe are likely to have proficient language skills in English, French and German at both a written and verbal level, simplifying the process for smart city developers. If the IT team implementing the smart city technology is nearshored, then the close proximity means that both organisations are also likely to share a time zone. This boosts the efficiency and ease of communication between organisations, and also ensures that both teams can communicate quickly if required. If an emergency does occur, the teams can even travel to the other organisation for meetings or planning sessions. Meetings, where outsourced teams can oversee the implementation of smart technology, can be arranged merely days in advance. If smart city initiatives are set to tight deadlines, this prevents delays.

COLLABORATION IS KEY Because outsourcing and nearshoring experts are brought in to fill the IT engineering positions that organisations struggle to locate internally, they bring a unique expertise that councils otherwise wouldn’t have. Outsourced and nearshored teams can consequently collaborate with other teams to share knowledge, including how to develop, test, and maintain the technology. Some outsourced and nearshored developers can take an advisory role to the project, and can, in return, learn from the teams they join. By collaborating and sharing knowledge and expertise, the project is likely to progress at a faster rate.

THE FUTURE IS SMART AND SECURE Despite smart cities often being associated with better traffic control, pollution monitoring and other processes driven through public citizen data, factories and the manufacturing process play an equally important role in defining the smart city of the future. With the movement towards factories that are fitted with smart cyber-physical systems, outsourcing companies are increasingly being employed to help the production industry take advantage of the benefits that Industry 4.0 – including the IoT, cloud computing and cognitive

T E S T M a g a z i n e | M a y 2 01 8

computing in manufacturing technologies – can bring to smart cities. This is because outsourcing can provide expertise in algorithms, IoT, big data, and security. Security is perhaps one of the biggest concerns in the transition from current manufacturing systems to modern smart manufacturing systems. By applying a combination of Ethernet, IoT, fog computing, virtualisation, and cloud computing to the process, the resulting modern system can often be left with numerous security vulnerabilities, threats, and risks. To combat security issues, outsourcing can support mitigation strategies, adjust security procedures to the administrator’s requirements and quality standards, apply new or adjusted security policies, and protect against security threats which minimises the risk of production downtime. With numerous risks around building smart cities, outsourcing can help develop security architecture design, conduct sufficient and careful testing, source code audits, assess system vulnerabilities, test application penetration, and much more. After the security of the smart city has been achieved, outsourcing IoT services can provide considerable help with optimising and integrating smart devices. This includes implementing industry 4.0 with an effective retrieval of valuable data collected by sensors and other internetconnected devices, adjusting processes according to data collected from smart devices that are fitted with sensors, and ensuring that the fully connected IoT environment remains intact. Outsourcing is one of the only methods that can provide all-round support for IoT projects and for implementing smart cities, with the possibility of being implemented at a variety of stages. Smart cities are an important part of the future of urban life, so it is important for organisations to anticipate skills shortages and source the best industry talent from within the industry. In today’s increasingly federated environment, looking outside the organisation and indeed a country’s borders to fix these challenges can be surprisingly efficient, flexible and secure.

NEWs Videos Product News Jobs white papers

bookmark now

www.softwaretestingnews.co.uk T E S T M a g a z i n e | M a y 2 01 8

NSTC SPEAKERS TEST SMART The National Software Testing Conference (NSTC) speakers talk exclusively with TEST Magazine Journalist, Leah Alger, about testing smart

T E S T M a g a z i n e | M a y 2 01 8

S M A R T

EST Magazine Journalist, Leah Alger, exclusively catches up with The National Software Testing Conference Speakers – Steve Watson, Interim Project Manager (GDPR) at Reed Business; Alan Richardson, Independent Consultant at EvilTester.com; Wojciech Bulaty, CTO at Traffic Parrot; Felicity Lord, Quality Assurance Manager at Wyndham Vacation Rentals; Thomas Noe, Product Quality Manager at Collibra; and Chekib Ayed, Global Head of Testing Practices at Société Générale Corporate and Investment Banking (SHCIB) about testing tools, smart cities, and smart homes. TELL ME ABOUT YOURSELF AND YOUR JOB ROLE: Steve Watson: I've been in testing for a number of years, working my way from a Tester to Senior Lead Manager. My last role had the grand title of 'Director of Quality Engineering' - essentially a Senior Test Manager role, but the role was made redundant in February. My current role is Interim Project Manager GDPR at Reed Business, which sees me through to the end of June. It's an interesting piece of legislation, and I find I am using a lot of my management skills here. Alan Richardson: I’m an Independent Test Consultant and Trainer who works directly with teams to help them improve their software development testing and programming approaches. I train people online and face-to-face using Java, automating and testing in a technical perspective. Wojciech Bulaty: I have been developing and testing software, and leading teams doing so, for more than 12 years’ across many industries from e-commerce and airlines to retail and investment banking. Recently, I founded a company called Traffic Parrot that helps software testing teams deliver faster and lower the cost of testing. We do this by implementing API mocking and service virtualisation for autonomous teams working with microservices. Felicity Lord: I have worked in IT for around 16 years, with 12 years in testing over a diverse range of industry sectors including finance, retail and leisure rentals. In my role as Quality Assurance Manager at Wyndham Vacation Rentals UK, I am in charge of quality assurance, change, release and environment management. I own the whole quality, release and deployment process through non-production environments into Live. I am now responsible for leading the

C I T I E S / D E V I C E S

UK quality assurance team by developing and continuously improving QA practices, processes, and methodologies. Thomas Noe: I am currently the Product Quality Manager for Collibra, where I am playing a central role between the engineering team and the field team (support, customer account manager, customers, etc.). I see a lot of customer bugs, escalations, quality issues and so forth. We are currently setting up an escalation process with all the stakeholders in our company. This will bring a lot of visibility and efficiency, and reduce communication threads. The output will drive our priorities, as well as the priorities of the engineering team. Chekib Ayed: I have over 18 years of experience in QA teams' management with different roles related to a mixed sourcing and worldwide context. I have been acting for the three last years on structuring programs for IT transformation related to continuous delivery and Agile@Scale. Currently, I am heading testing practices for Société Générale GBIS with the ambition to scale up both the global testing maturity and the staff autonomy.

Smart homes can mean so many great opportunities for an interconnected world

WHAT TESTS ARE YOU AND YOUR TEAM CURRENTLY WORKING ON? Steve Watson: Mainly around GDPR at the moment - ensuring we are compliant. Other than that, we, at Reed, are working on a new front-end website for our subscription users, to offer something new and fresh to the market. Alan Richardson: At the moment I’m experimenting with different ways of creating maintainable abstraction layers for automating and finding better ways to train people in technical and programming skills. Wojciech Bulaty: We automate all of our regression testing, and allow people to contribute manually to what they are best at finding new issues and ensuring the quality of new features. We also try to find a balance between reducing the risk of production issues and having too much testing that reduces the speed of delivery. Felicity Lord: My team are currently working on a DevOps initiative to move our organisation to a DevOps culture. This is a really exciting project for us as it allows us to ensure that automated deployments, continuous integration and monitoring are all built in with a quality first mindset. By designing the right DevOps culture for our company we are able to focus on ensuring that we can provide innovative software

STEVE WATSON INTERIM PROJECT MANAGER REED BUSINESS INFORMATION

Steve is an ISTQB certified Quality Engineering Professional with more than 30 years’ experience in banking, futures and options trading, vehicle leasing, automotive information and B2B publishing. His last role was as a Senior Quality Engineering Manager at Reed Business Information, as part of RELX, a FTSE 100 company, where he managed a team of testers and directed the overall testing strategy and approach across a number of products.

T E S T M a g a z i n e | M a y 2 01 8

S M A R T

C I T I E S / D E V I C E S

If we don't keep an eye on the future it might hit us hard in the face when it becomes the present

solutions in a more agile manner. Thomas Noe: We are currently implementing performance browser page render tests, which we want to add in the pipeline. For now, we use Puppeteer, but perhaps we will use other tools if needed to improve the coverage of other browsers. These automated performance tests will assure that we don’t regress during our new development activities, and if we do regress, we block the pipeline so that we can address this issue. Chekib Ayed: The team mission has been to focus on automated regression testing including functional testing. Recently, this mission is transformed to take the ownership of testing practices. The team is now acting as a centre of excellence to scale up the maturity of our internal testing practices while increasing the autonomy of testers inside feature teams. We are providing support for feature teams members to essentially sustain all kind of testing practices from unit to end-to-end tests. HOW DO YOU AND YOUR TEAM ENSURE TOOLS AND ENVIRONMENTS ARE FIT FOR PURPOSE, UP-TO-DATE AND EASILY ACCESSIBLE?

ALAN RICHARDSON INDEPENDENT CONSULTANT EVILTESTER.COM

Alan has more than 20 years’ of professional IT experience, working as a Developer and at every level of the testing hierarchy from Tester through to Head of Testing. Now he works as an Independent Consultant, helping companies improve their agile software development and testing processes. He works directly with teams, coaching them to ensure an effective use of automation and exploratory technical testing is being applied.

T E S T M a g a z i n e | M a y 2 01 8

Steve Watson: Test environments always used to be a pain to manage – they were never up to date with the right data or code changes, and there was always a cost element to requesting more environments, so often environment clashes would ensue. With the advent of cloud computing, we can spin up a test environment when needed, and tear it down once finished. The data we need could be deployed, although now, with the advent of GDPR, we have to be careful about obfuscating any personal data that we may have in pre-live environments. In terms of tools, Selenium Webdriver has almost become the defacto standard for web application testing, and you can configure it to work with other tools such as Specflow or Gauge to add in business case definitions, and then code in whichever language you choose. Alan Richardson: It can be very hard to stay up-to-date, particularly when you are few in numbers. But if we don’t keep an eye on the future then it might hit us hard in the face when it becomes the present. Very often we put too many constraints in place, which prevents people from experimenting. We do need to make sure that strategic work doesn’t have a free for all approach to tooling, but for tactical work, making it easy

to experiment provides a low-risk way to learn new approaches. Wojciech Bulaty: We ask ourselves why our current environments are not fit for purpose, up-to-date or accessible. When you understand why we change the root cause or the problem. Typically, it’s because of failing to articulate the on-going costs of keeping things as they are, compared to the cost of change. Sometimes it’s best to leave things as they are, more often fix the issue by upgrading software or architecture ASAP. Felicity Lord: Ensuring tools and environments are fit for purpose, upto-date and easily accessible is a major part of my job! I undertook a substantial restructure and investment over last year, and this, to make sure that the tools and environments we use are suiting our needs and future desires. This has ultimately led to the DevOps initiative that my team are now working on. Thomas Noe: Recently we hired a dedicated continuous delivery expert which will keep those tools up to date so we can benefit from the latest features we also are inventing all our tools we have and documenting many parameters in our own data governance solution. Via this way, way we are drinking our own champagne and can reduce the number of tools we have because in a scale-up each and every department needs tools and connections between all those tools. Chekib Ayed: This is a good question, especially since I do not have yet the right full recipe. Let’s answer it by taking each piece apart. In my context, we used to recommend Microfocus QC/UFT whatever the teams’ needs. Since agile@scale transformation, we extended our tools’ recommendations to include more adapted tools to the agile methodology. We have currently 10+ proposed tools packaged and customised for our internal use. Teams are autonomous to choose the most convenient tool to fit their purpose. A set of guidelines, training and bots are available to help teams onboard tools with the assistance of testing experts. The environment management is the responsibility of the feature teams. Each team is autonomous to setup, update to define access to its own environments as it wishes. HOW HAS THE SCOPE OF SOFTWARE TESTING CHANGED SINCE YOU JOINED THE INDUSTRY? Steve Watson: Wow - hugely! Everything was mainframe-based, manually tested,

S M A R T

and there were no tools! We had no training and there were no certificates, so we learned on the job. This was before the internet, so, of course, there was no interconnectivity and no need to be so security aware as there were fewer ways for people to access the bank's computer network. Now, of course, not only have we had e-commerce for a number of years, we have the 'Internet of Things', and all the challenges it brings with interconnected devices. Alan Richardson: Testing is much more integrated with the development approach than it was when I started. We used to fight to see code speak to development teams. Now, collaboration is encouraged, leading to new pressure for testers since they have to talk and act as a more technical level, as well as work with requirements and GUIs. I think is a good thing since I always approached testing from a technical perspective and work as closely as possible with development teams. Wojciech Bulaty: The two main areas of change in my daily practice since I joined the industry 12 years ago are more automated testing and the new microservice architectures. Automated testing has been around for a while, but now the majority of the market is adopting it. Microservice architectures are relatively new and but are becoming widely adopted in the industry. Felicity Lord: There are a number of challenges that I have seen appear within software testing over recent years. Particularly in a web-driven market, emerging markets, new contenders and start-ups can lead to business model disruption. From a testing perspective, that means an ever greater push for pace and agility to meet changing priorities and to provide timely information to make quick decisions regarding risk acceptance. Thomas Noe: At the beginning of my career, we were mostly working with the waterfall model and manually testing. We lacked proper test data, stable environments, time and we were testing mostly at the end (no incremental testing). We used many standalone tools, which were not integrated with one another. In recent years, QA engineers have been increasingly integrated into the engineering teams and have been more involved at the start of the engineering process. These days, QA engineers can test something in a couple of minutes and control their own testing instance, locally (via Docker) or in the cloud. This way, incremental testing can be carried

C I T I E S / D E V I C E S

out, and due to fast-paced (in parallel) automated testing, regression bugs can be found and fixed. Chekib Ayed: Without a doubt, software testing has changed since I joined the industry 20 years ago. When I started, I remember when I received a dozen of guides on my first day of work, with hundreds of pages each, describing the product to test. Tests were manual, fully functional and based on structured and formal reports. Today, tests are presenting differences in scale and timing. They are much more free and informal with light documentation not centred on process and governance, as well as are more imaginative and exploratory. In the same time, they are also less repeatable and less measurable.

Smart cities are built on new types of hardware but also a lot of highly distributed software. Software testers will have to step up to the challenge and contribute to ensuring the quality of new solutions

HOW ARE SOFTWARE TESTERS GOING TO CONTINUE TO REVOLUTIONISE SMART CITIES? Steve Watson: The most obvious things will be to coordinate how we live and interact with each other more - traffic flow, refuse collection, deliveries, and maintenance of utilities. Rather than our lives happening as a series of non-connected events, the way in which we organise ourselves, and in which our towns and cities are governed and run will be more intertwined. How will this affect us? It goes back to the themes of data security and privacy. The sad thing for me is that whenever something new is invented, it doesn’t take long for others to find a way to use it for criminal purposes. It will be down to us all to ensure that we design (rather than test in) security, and failsafe to protect people from those who would want to disrupt services and hack into our interconnected networks for their own gain. Alan Richardson: As long as software is involved, we will need people who can think through the risks as well as the functional paths. Privacy will become increasingly important, particularly when the smart city is trying to collate and utilise as much data as possible. I think it will be interesting to see how compliant smart cities are with the new GDPR. Wojciech Bulaty: Smart cities are built on new types of hardware but also a lot of highly distributed software. Software testers will have to step up to the challenge and contribute to ensuring the quality of the new solutions so that we all as a society can benefit from safe and frustration free use of the new technologies. In my opinion software testers will continue to work on

WOJCIECH BULATY SOFTWARE CONSULTANT TRAFFIC PARROT

Wojciech is the Founder of Traffic Parrot, and a Software Consultant with more than 12 years’ of international experience in multiple projects. He is also as an agile, XP, TDD, pair programming and clean coding enthusiast.

T E S T M a g a z i n e | M a y 2 01 8

S M A R T

C I T I E S / D E V I C E S

FELICITY LORD QUALITY ASSURANCE MANAGER WYNDHAM VACATION RENTALS UK

Felicity is an experienced Quality Assurance Manager with a demonstrated history of working in the leisure, travel and tourism industry. She is also skilled in test management, system integration testing, systems thinking, management and test planning.

THOMAS NOË PRODUCT QUALITY MANAGER COLLIBRA

Thomas is an energetic and passionate Test Professional with more than 10 years’ of experience ranging from hands-on operational testing to test management positions. He graduated as a Bio-Engineer in 2006, but was given a chance to taste the testing world. The following years he saw different approaches across several industries such as utilities, telecom, financial world, online payments and data governance. Step by step, as he moved from Test Engineer eventually to QA manager, he improved from both good and bad experiences.

T E S T M a g a z i n e | M a y 2 01 8

ensuring the quality of the smart cities software is up to expectations. For more details, we have to wait and see where the market moves. Felicity Lord: There are three areas that I believe are key to smart cities: data reliability, ICT interactions and their integration, and performance. Software testing can play a powerful role in ensuring that the data is accurate and connected in order to enhance the quality and interactivity of our urban environments and allowing for real-time responses. The global market for smart urban services is predicted to be massive over the next couple of years so I can definitely see the benefits of expending effort on getting this right. Thomas Noe: In my opinion, data governance is the key to smart cities. You can collect as much data as you want, but if you don’t govern your data and trace the source of the data, you will not benefit from it. Once you have the governance set, you can start the process and interpret the data you are harvesting from all the sensors or applications you have implemented. In light of this, software testers will need to work more with real-life data collected from many use cases around the world. Software testers often use dummy data, but real-life data is much richer and it can help you to adapt your test strategy, based on those data sets. The software tester needs to continuously adapt the testing strategy or risk running too far behind. Chekib Ayed: Smart cities are all about optimising the organisation; the well being of people, and operating costs. The key to smart cities is basically to ensure resiliency, robustness, sustainability, security and the data relevancy of the exposed city services. As trust is the key to this kind of emerging concepts, it is fundamental to promote adoption in order to guarantee the expected value. In this context, the role of testers is crucial. They will bring the high-level view by framing what should be tested as well as the needed technics to do it. Especially since testing smart cities devices and sensors will need in addition to technical skills, the ability to explore an unlimited combination of use cases. Testers know how to establish a sharp and optimised testing plan. HOW CAN SOFTWARE QUALITY ASSURANCE HELP BUILD SMART HOMES? Steve Watson: We need to ensure that when designing smart homes, we as testers

are involved in helping to highlight the many different types of scenarios that could happen and that they are properly covered. Smart homes need to be secure against hacking, and manipulation for criminal gain. They need to be safe and trusted by homeowners. Alan Richardson: I can see many risks in IOT smart homes. As soon as we start putting more devices around the home, the possibility of emergent behaviour, that is hard to test for in isolation, becomes an issue. Testers really need to think in terms of systems with all of the ‘smart’ features of the house as a web of systems that might impact each other in unexpected ways which can possibly interact with other systems outside the home. Wojciech Bulaty: Smart homes are about helping us lead easier and happier lives. Bugs in software cause frustration and safety hazards. We need to make sure that smart home software is reducing the frustrations, not adding to them. There are also big challenges with smart home technology safety. Software testers should address both of those risks. Felicity Lord: Smart homes can mean so many great opportunities for an interconnected world: from better care for individuals at home to more efficient use of energy, to better security; you can connect the devices and appliances in your home so they can communicate with each other and with you, and in the near future they will also begin to learn from us and adapt. This can seem like a scary concept for a lot of people so software quality assurance plays a powerful role in ensuring that smart homes connect in an expected way and do the things we want them to do, and not go wrong. It sounds simple, but the success and growth of smart homes lie in peoples’ reassurance and peace of mind, which can only come from knowing that a product has been tested and works well. Thomas Noe: Software quality assurance brings high business value to enterprises and housing businesses that are contributing to the creation of smart homes. The testing strategy needs to focus on different areas, such as connectivity, data security, user device communication and so forth. Software testers can address these. We not only ensure the functional system but also verify security and accessibility under varying conditions. Chekib Ayed: Software quality assurance is an enabler for businesses to validate what they are meant for. This applies to smart homes, which are expanding with a variety

S M A R T

of impressive solutions. The value of these solutions depends mainly on their quality. The quality difference could be observed through the price that varies, sometimes, from simple to quadruple, for the same sensor. It is obvious that users would prefer secured, plug and play, and cheap solutions that require the minimum installation and maintenance effort. As a user and fan of home automation, I myself faced different issues. Most of them occurred when my home use cases were not managed by the sensors’ plug-ins. WHAT DO YOU THINK THE SOFTWARE TESTING INDUSTRY IS YET TO WITNESS? Steve Watson: AI is very much a talking point, and the creation of artificially intelligent machines will be a turning point for us humans. A lot of what we do uses code – but most of what we produce is used by us human beings. This will change, with less human actions, and more machine-tomachine types of devices; with machines making the decisions that humans would have done. We may care less about usability and how things look on a screen, as the end user is not a person, in which case our focus will be much more on data and analytics. Alan Richardson: We haven’t yet seen model-based testing as a ‘normal’ approach to automated execution. It might seem like we are experiencing a strong push to become even more technical and increase our skill overlap with programmers. I think that we will become even stronger and that testers need to put in the time to keep their skills and knowledge up-to-date. I also believe we will see more of an overlap between software testing and security testing. Wojciech Bulaty: I think we need to trust more in applying human cognitive skills where they are required and automating everything else. Many companies are doing that, but it looks like the majority of the market is still not there yet. We need software testers to do what they are best at not regression testing but finding unexpected issues and innovating. I look forward to the day when most of the industry's regression testing is automated. Also, as automated testing grows in complexity, testers need to take into consideration long-term maintenance issues of their suites of tests. We will need to adopt practices already widely present in software development. Felicity Lord: I think we have only seen the tip of the iceberg on what AI and machine

C I T I E S / D E V I C E S 21

learning might mean in the way it impacts not only software testing but also our lives as a whole. I can see that advances in this area with big data sets and analytics could bring us particular challenges as well as the new developments in context-aware devices. Augmented reality is an exciting area and one that I could see directly having an effect on the leisure and tourism industry. Security and data protection is a particularly hot topic at the moment and it will be interesting to see what impact quantum computing has on how we cope with encryption and security in the future. Thomas Noe: Although the focus is on automation, I think we should not forget the importance of validation. Automation is only part of the coverage, so you will still need to manually validate or manually break the feature or application. The time spent manually testing will decrease, so we will need to be more efficient during that phase. Teams or QA engineers cannot neglect this. It is tempting to put all your money into automation. Upper management does not always know the details of thorough testing so always try to explain or show that with clear objective numbers. Many QA managers need to do a better job there. Something that will help us a lot is AI technology. More and more start-ups are investing in AI testing tools, which will help manual testers in the area of user behaviour. I am very curious about this evolution and how it could potentially help us. Chekib Ayed: Moving forward with agile and continuous “things”, the testing industry has already taken the benefit of time to market. Tests are thus adapted to development cycles allowing rapid executions and analysis thanks to DevOps culture. This works even better when automation is excessive, fully integrated with the software factory and respecting the right ratios of a testing pyramid. However, automation is still not effective in many cases. Automation is not the only challenge; many teams face often the matter to implement end-to-end tests since they need to set up a full test environment with the whole eco-system, as well as set the balance between the right level of test proofs that might be requested by regulatory bodies and the right level of reactivity needed for agile methodology.

Although the main focus is on automation, I think we should not forget the importance of validation

Find out more about The National Software Testing Conference: www.softwaretestingconference.com

programmes for IT transformation

CHEKIB AYED HEAD OF TESTING PRACTICES SOCIÉTÉ GÉNÉRALE CORPORATE & INVESTMENT BANKING

Chekib has more than 18 years’ experience in the management of QA teams. He joined Société Générale Corporate and Investment Banking for 13 years where he held different management roles related to mixed sourcing and worldwide team context. Taking benefits of a dynamic and constantly changing environment with regular challenges of time to market, costs pressure and production quality, Chekib has been acting on structuring related to continuous delivery and Agile@Scale.

T E S T M a g a z i n e | M a y 2 01 8

SPEAK UP TO TECHNOLOGICAL ADVANCEMENTS The digital marketing landscape is constantly evolving, so itâ&#x20AC;&#x2122;s important to keep up-to-date with the latest technological advancements

T E S T M a g a z i n e | M a y 2 01 8

S M A R T

ne development that is beginning to revolutionise search marketing is voice search. Google recently announced that 20% of searches on its mobile app and Android services are now carried out by voice, which is more than likely to increase in the coming years. The statistics about voice search today are proof of just how important this method of search is becoming. Microsoft Cortana has over 133 million active users, more than 40 million people use Siri on their Apple devices every month, and Amazon has already sold over 20 million Echo devices. These figures are not only staggering but also prove the growing consumer desire for voice search technology. The reason for this growth has a lot to do with increased voice search accuracy. Early users of voice search will appreciate how you do not always get what you asked for. Trying to ask your device to ‘Call X’ would often lead to it starting a call with someone completely different and then ignoring your requests to cancel. Luckily, things have moved on. In 2013 the word accuracy rate of voice search devices was just 76%. By the end of 2017, this had risen to an accuracy rate of 95%, which is incredible when you consider the difference in accents and word pronunciation. Further improvements will be made this year. The fact that when you use voice search now you generally receive relevant information which demonstrates that more and more people are using voice search, as this has enabled devices to learn search intent and provide accurate results. Voice search can be a much faster and convenient way of searching the web. It’s much easier to speak than type, demonstrated by the fact that, on average, we can type 40 words a minute, but can speak 150 words in the same amount of time. Now technology is in place to support searches made by voice, it is important to adapt your businesses marketing strategy.

SIX IMPACTS OF VOICE SEARCH: • FIRST RESULTS IN SEARCH ENGINES ARE NOW MORE IMPORTANT: Voice search will have an impact on search engine optimisation. One of the main changes voice search introduces is the fact that only the first result is read out. Suddenly optimising pages to be on the first page of Google is not good enough. Even being second in Google results is not impressive. This means it is even more important to be the first result. • WHAT QUESTIONS ARE YOUR AUDIENCE ASKING? Search queries from voice are often longer so it is, therefore, a good idea to start optimising for longer search queries rather than purely keywords. Start to pay attention to what questions your customers are asking your business and then create a page on your

C I T I E S / D E V I C E S

website that answers that question. If you want to find out what questions consumers are asking around the subject of the product or service you provide, then a good website to use is 'AnswerThePublic', which generates title ideas around the theme you input. • DISINTERMEDIATION: The main reason voice search is growing is because of its convenience factor. People want results, and they want them fast and efficient. However, this also eliminates choice. For example, if you use Amazon Echo to order a pizza then it will order it from Domino’s because the pizza company has a deal with Amazon. • LOCAL SEARCH AND MOBILE: Voice search is becoming increasingly used in local searches, such as queries like ‘coffee shop near me’ or ‘where can I buy a newspaper?’ If your business relies on local customers, then it is important to be found when people are asking for the product or service you sell. Localised search results come from 'Google My Business' and 'Bing Places for Business', ensuring your business has these listings set up, otherwise, you could be missing out on local search traffic and customers. • MISSPELLING AND MISPRONUNCIATION: This is a challenge for artificial intelligence (AI). The word accuracy rate is now at 95%, but it is still not 100% – this means there is still a margin for error. There are certain brands that are almost impossible to find via voice search. Just try asking for Yves Saint Laurent handbags. If your brand name is difficult to say then changing it just to work better for voice search probably isn’t an option, but is worth bearing in mind when developing new products. Is the new product easy to pronounce? If it is then it will be easier to be found via voice search. • ORG IS VITAL: For a search engine to understand what your website page is showing it must be able to understand the page data. This data is delivered in a large part by structured data and this is defined by the schema.org standard. This is the system used to mark-up content on a web page, which then enables search engines to understand the text in the code. By implementing schema.org you will help AI to identify information on your website pages, increasing your chances of ranking for voice search results.

The main reason voice search is growing is because of its convenience factor. People want results, and they want them fast and efficient

It is worth investing time and money in optimising your business’ website for voice search as the growth of voice search shows no signs of slowing down. By 2020, it is estimated that 50% of searches will be voice searches. If your business is not prepared for the shift in the way people search online, then you could miss out on search traffic and ultimately lose sales to competitors who have optimised their sites effectively.

visibility, lead generation and customer

TIM BUTLER FOUNDER INNOVATION VISUAL

Tim Butler is the Founder of Innovation Visual, which is a digital marketing consultancy that specialises in digital effectiveness and increases website acquisition. Recognised by Google as one of the UK's top 30 digital agencies, the firm uses the latest techniques in SEO, PPC and online content creation, and is an award-winning Google, Bing and Hubspot partner.

T E S T M a g a z i n e | M a y 2 01 8

THE FUTURE IS SMART The Fast Future team explores some of the critical trends, visions, ideas, and disruptions shaping the rise of smart cities

T E S T M a g a z i n e | M a y 2 01 8

S M A R T

hilst smart city visions differ, at their heart is the notion that in the coming decades, the planet’s most heavily concentrated populations will occupy city environments where a digital blanket of sensors, devices and cloud-connected data are being brought together to enhance the city living experience for all. Smart concepts encompass all of the key elements that enable city ecosystems to function effectively – from traffic control and environmental protection to the management of energy, sanitation, healthcare, security, and buildings.

PURPOSE AND VISION The evidence to date from smart city initiatives around the world is that the best results come when we have a clear sense of what the end goal is. In a fast-changing world, it can be hard to develop a clear future vision and strategy – when every sector is being disrupted and all our assumptions are being challenged. However, that’s exactly what we have to do. City governments must create inclusive processes that inform citizens about the forces shaping the future and the possibilities and challenges on the horizon and then engage the population in dialogue about the kind of future we want to create. We have to explore what a livable city means to its people and be clear on how we will attract and support a constant flow of industries of the future. Alongside this, we need to articulate a clear vision and direction around education, environment, public services, access to justice, city administration, and civic engagement. These pillars then provide the guiding requirements, which will in turn influence the design of the physical, digital and human elements of a smart city infrastructure.

SURVEILLANCE AND PRIVACY Smart cities are designed to inform decisions by capturing massive amounts of data about the population and its patterns, such as water use and traffic flows. This information gathering results in what is called big data, and it is essentially gathered via surveillance. There can also be voluntary efforts to collect information, but the ease and affordability of sensors, AI and advanced analytics in the future will mean this function can be completely automated. The data can be

C I T I E S / D E V I C E S

collated from a constantly evolving and expanding IoT – encompassing traffic lights and cameras, pollution sensors, building control systems, and personal devices – all literally feeding giant data stores held in the cloud. The ability to crunch all this data is becoming easier due to rampant growth in the use of devices algorithms, AI, and predictive software – all running on networks of high-performance computing and storage devices. Singapore is a leading example of a smart city, and is constantly evolving its “city brain”, a backbone of technologies used to help control pollution, monitor traffic, allocate parking, communicate with citizens, and even issue traffic fines. The behavioural aspect is not to be overlooked. Singapore’s “brain” is attempting to modify human behaviour; for example, one system rewards drivers for using recommended mapped routes and punishes those who do not. Ultimately, Singapore’s planners hope to discourage driving and guide most commuters to making greater use of public transportation. The city is planning for 100 million “smart objects” including smart traffic lights, lampposts, sensors, and cameras on its roadways, which will be used to monitor and enforce laws.

In a fast-changing world, it can be hard to develop a clear future vision and strategy – when every sector is being disrupted and all our assumptions are being challenged

DATA BEACONS Smart cities rest on advanced technology to make sense of massive collections of information. Indeed, the amount of information on the internet is expected to grow exponentially as a result of the Internet of Things (IoT). Essentially IoT means that everything (“things”) – and potentially everyone – will become beacons and data collection devices, gathering data on ambient and behavioural patterns from its surroundings and from the information it is fed and networking all this data via the cloud. Hence, after data, the IoT is the second driving force behind the rise of smart infrastructure; for everything from air conditioning to parking meters to function in a smart city, the use of microphones, sensors, voice recognition, and all sorts of other hightech gadgetry must be hooked up to the IoT. Companies and planners are already beginning to explore the possibilities. For example, a case study from India suggests that light poles along the highways can offer both smart city and connectivity solutions. In addition to helping monitor road conditions, the light poles could be fitted as high-speed data connections. Data is a critical element of the smart city/smart road future. However, because this option will further expand

ROHIT TALWAR CEO FAST FUTURE

Rohit is a Global Futurist, Strategic Advisor, Award-Winning Speaker, Author, and the CEO of Fast Future. He focuses on the future of nations, cities, businesses, and citizens, talking regularly about the opportunities to pursue step change market growth, and the challenges of preparing society for the disruptive impacts of exponential technologies. He advises nations, cities, global businesses and professional services around the world.

T E S T M a g a z i n e | M a y 2 01 8

S M A R T

C I T I E S / D E V I C E S

the relationship between internet service providers, surveillance, and private business including advertisers, there are several issues around privacy to be considered.

ROADS: THE BIGGEST OPPORTUNITY STEVE WELLS COO FAST FUTURE

Steve is a Global Futurist and Keynote Speaker. He advises global corporations on the transformative potential of disruptive technologies and how brands can leverage them to create distinctive identities in a fast changing world and drive exponential growth and improvement. He has a particular focus on the new models of leadership, collaboration, and innovation required to sustain and differentiate brands in a rapidly changing reality.

Within and between the smart cities of the future, smart roads are where planners can put into effect the ultra-efficient mechanisms that will characterize the modern smart city. In general, the concepts around smart cities, smart roads, and smart infrastructure are becoming less visionary and more strategic and sustainable by the day. As a smart infrastructure future unfolds, three important new technologies – big data, the IoT and renewable energy – are working in tandem to transform the day-to-day. For example, South Korea is planning an entire network of smart roads by 2020, including battery-charging stations for electric vehicles (EVs) as well as infrastructure to handle autonomous vehicles. Autonomous vehicles will require roads to transform into information superhighways; the vehicles will need to communicate not only with each other but also with the city infrastructure. The need for ubiquitous data is a critical step in the evolution toward self-driving cars. Mapping, traffic signals, and safety regulations, for instance, are all parts of the physical and digital infrastructure that will have to become highly coordinated for autonomous vehicles to function.

A SUSTAINABLE FUTURE

ALEXENDRA WHITTINGTON FUTURIST WRITER FAST FUTURE

Alexandra is a Futurist, Writer, Foresight Director of Fast Future, and a Faculty Member on the Futures programme at the University of Houston. She has a particular expertise in future visioning and scenario planning. She is also a Contributor to The Future of Business, the recently published ‘Beyond Genuine Stupidity’ – ensuring AI serves humanity and the future reinvented – reimagining life, society, and business.

T E S T M a g a z i n e | M a y 2 01 8

All this data and awareness will enable decisions that make the best possible use of space, fuel, energy, water, electricity, and all resources, with an emphasis on sustainability. For example, a clear priority is being able to anticipate big traffic jams and provide alternate routes to save time, fuel, and reduce the impact on the city infrastructure itself. Limiting waste is a very logical outcome and benefit of the merging of big data, AI and IoT, which feeds into the rise of smart infrastructure. Electric vehicles are growing their market foothold; hence the charging concerns related to EVs are gaining urgency in the eyes of many policy-makers and planners. One of the biggest hurdles to electric vehicle adoption is keeping a charge across long distances. To help address this challenge, UK researchers are testing out smart road technology that charges electric cars while

they are being driven. The achievement of smart infrastructures could drive the required policy and behavioural tweaks to enable wide adoption of today’s renewable resources. Another smart technology involves pavement surfaces prepared with panels that capture solar energy allowing streets to power themselves, which has been tested in the U.S. and the Netherlands. Meanwhile, a special powder that gathers sunlight and glows at night has shown promise, but not in the wet climates of northern Europe – it washes off with excessive rain. A futuristic concept that would not require sunshine is the capture of renewable energy with piezoelectric crystals installed beneath asphalt on busy highways. Theoretically, the crystals could capture the energy released by vehicles driving over the asphalt, which could then be stored or used for powering highway infrastructure. Piezoelectric crystal pilot programmes, though so far unsuccessful, provide a fresh perspective on renewable options for gathering energy to power our infrastructure and roads.

A HUMAN FEATURE The smart city movement now afoot has the potential to transform the organisation of people and physical objects in a way that transcends urban development, as we know it. The shift to smart infrastructure is not simply fashionable or aspirational; in many ways, it appears to be a critical enabler of the future sustainability of cities. It can be argued that the future of human life on the planet rests on a smooth transition to cities that are more efficient, less wasteful, and more conscious of the impacts of the individual upon the greater good. This may include a range of new negotiations along the boundaries of freedom and privacy; for example, allowing self-driving cars to replace human drivers in the hope of preventing death and injury in auto accidents, increasing traffic efficiency and removing environmental impacts. Similarly, we might have to agree to invasive monitoring of waste generation, energy and water use in the home to reach municipal conservation goals. These are the kinds of tensions that future planners will need to wrestle with on a continuous basis. A well thought through vision, enabled by a robust and well-executed smart city model could provide a foundation stone for the next stage of our development, where science and technology are genuinely harnessed in service of creating a very human future.

16-17 October 2018 27

Millenium Gloucester Hotel London

www.DevOpsEvent.com

National DevOps Conference 2018 Practical Presentations | Workshops | Networking | Exhibition

The National DevOps Conference has its finger on the pulse as it is owned and supported by industry leading news portal DevOps Online and industry leading journal TEST Magazine, and is produced by the organisers of The DevOps Industry Awards.

 Learn from leading industry experts at cutting edge seminars. Interact at collaborative Q&A sessions  Join in executive discussion forums  Source the latest products and services via the industry leading exhibition  Network with your peers and key industry figures  Swap and share ideas with likeminded professionals

This two-day programme is designed to connect a wide range of stakeholders, and engage not only existing DevOps pros, but also other senior professionals keen to learn about implementing this practice. At The National DevOps Conference, you can hear from peers who have successfully begun their DevOps journey; from industry practitioners sharing advice and knowledge; join in executive workshops; network and much more.

 Network and make new and influential contacts  Take away ‘real-world’ scenarios and apply learned solutions to accelerate your own IT transformation  Network with your peers and new acquaintances at the Evening Drinks Reception

£735

Day 1 Only*

£525

Day 2 Only*

£525

 0203 931 5827  info@devopsevent.com  http://devopsevent.com

*Includes entry to the Evening Drinks Reception on the first day of to VAT. T the E SConference. T M a g a zAlli nprices e | are M subject a y 2 01 8

TEST SOON AND FIX FASTER Mobile and Third-Party Products Director at The Telegraph gives tips to having the right checkpoints and alerts in place in order to test soon and fix faster T E S T M a g a z i n e | M a y 2 01 8

F U N C T I O N A L

veryone has been through this scenario: You just released a new version of your app and something is going wrong. Something your team should have spotted during testing. But there you are, facing a crisis and having to act quickly to fix the problem. What can you do to avoid this in future? Where is the process broken, and how can it be improved?

NO MAGIC TOOL First, the bad news: There's no 'one size fits all' solution. The problem you are having for your product is different to the one I am having for my product; you may find a way to avoid the same problem from happening, but you are going to have a different problem tomorrow. So what can you do? You have to be sure you have the right checkpoints and alerts in place so you can test soon and fix faster.

EXERCISE YOUR CODE Tests should exercise your code, looking for both expected success and expected failures, to cover all of the boundary conditions. One of the mistakes that myself and my team have made in the past is that we only tested when we wanted to release a new version of our apps live: Our technical landscape is complex and yours most likely is as well, so it can happen when a ring of the chain is broken and when we don’t realise before our customers; this is the reason why we need to be able to verify on a daily basis (and sometimes more often) to ensure that the production versions of our apps are working fine. We have a number of tools/processes that helps us to achieve this goal. • AUTOMATED MONITORING ALERTS: If one of the KPIs we identified as essential to help evaluate the health status of our apps is above/below a specific threshold, the Dev and Ops team automatically receives an alert via various channels (Slack, email, etc.). We have two different threshold levels, orange and red (red meaning ‘take immediate action’) and everybody knows what to do if we receive one of those alerts. • DAILY SANITY CHECKS: Our QA team runs a set of standard tests to ensure the apps are working as expected on a daily basis. Those sets include analysis like memory usage, CPU usage,

simulated usage for a specific amount of time, and automated testing against a mock server – this allows us to verify specific edge cases and manual sanity check. • WEEKLY AUTOMATION TESTS: Fragmentation is a challenge on the Android platform, and we have a very differentiated device base on both platforms that force us to maintain support for old versions of operating systems. To manage this, we are running all our automation tests using a third-party test farm where we can use hundreds of devices with different configurations for OS, network connections, device types, etc. The final report is analysed at the start of the week to identify any major trends that could potentially be an issue that is yet to be detected. • PRE-RELEASE OS VERSIONS: As developers, we know that a new version of an operating system can depreciate or change something we use or rely on in our code. This is why we run our tests on any ‘developer preview’ version of the OS that we support. In doing so, we may not only catch a problem way before our customers, but we can also verify that app performances are not changing between different versions of the same operating system.

T E S T I N G

You need a good amount of unit tests. The tools are complementary and if your team is able to use them for the right scope they will prove invaluable

TDD/BDD/UT/ AUTOMATION Furthermore, this can work to check the status of production versions of our apps. But what about the tools, frameworks and best practices we use during the ‘delivery’ of changes, updates or new features? We do not do anything fancy here other than rely on standard best practices and tools that are well known and mature, such as TDD, BDD and unit tests/automation. More than often I see people discussing (sometimes fiercely) about TDD and BDD: Some people say that TDD is the way to go, others believe BDD is the only thing that should be implemented. My view is that you need a good amount of unit tests. The tools are complementary, and if your team is able to use them for the right scope, they will prove invaluable. Now let’s start with unit tests. As a quick search on Wikipedia will tell you, unit tests focus on a ‘single unit of code’ that is a function of an object or module. This highlights the basic characteristics (and the

EMILIO VACCA DIRECTOR OF MOBILE THE TELEGRAPH

Emilio is a Technology Enthusiast who has spent the last 25 years working as an Interactive Designer, Software Developer and Architect for Enterprise Software. Now he is the Director of Mobile at The Telegraph. His career path has provided him with unique advantages and visions, allowing him to move quickly between different technologies and sectors including telecommunications, healthcare, fashion and media.

T E S T M a g a z i n e | M a y 2 01 8

F U N C T I O N A L

The fact that you end up with a very extensive coverage is the power of TDD: Once you implement TDD, you will be able to trust that your code works

T E S T I N G

resulting scope) of a unit test: • It is specific to a very small piece of code. • It should be simple and quick to write • It should be quick to run. • It should be isolated from dependencies (a.k.a. network operations, database access, etc.). Now the implicit simplicity of a unit test is the reason they are so powerful in protecting code from a ‘unintended consequences’ of change: If you have a set of unit tests verifying how your code works, then you can safely change your code and your programme should not break. If it is not easy to write a unit test then most

probably it is not a unit test so you should reduce its scope. TDD stands for ‘Test-DrivenDevelopment’ and the process to write/ execute a test should include the following steps: 1. Start writing a test. 2. Run the test and any other tests that already exist - the new test you added should fail. 3. Write only the code that would allow the test to pass. 4. Run the test again and any other test that already exists (all tests should pass). 5. If needed refactor your code. 6. Go back to point 1 and iterate. This requires great ability from engineers, because they will have to write the tests before writing any code, so it is quite a dramatic shift in mindset. Despite this, the reward can be huge – especially if test coverage goes up from 90% to 100%. The fact that you end up with a very extensive coverage is the power of TDD: Once you implement TDD, you will be able to trust that your code works and the changes you apply won’t break the rest of the code in your programme. Finally BDD, which gives many (including myself) a headache: BDD is a set of best practices that, when used with unit tests and TDD, can give great results. BDD helps to shift the focus from implementation to behaviour, and, to do this, we need to spend some time thinking about what the scenario is. This is facilitated by the fact that usually, you write a BDD test with something like ‘it should do something’. If the requirements change, as long as the scenario does not, you should not modify your test. You should use unit tests, TDD and BDD together, and in combination, because: • Unit tests gives you what to test. • TDD gives you when to test. • BDD gives you how to test.

MANUAL TESTING: THE MACHINES CAN’T DO IT When I see my QA testers following a rigid Scenario->Case->Steps schema while manually testing our products I shudder: They should not spend their time on that. This is food for a machine; for fully automated testing that we can run on hundreds of devices in one night. I usually define my fellow QA colleagues as our 'last

T E S T M a g a z i n e | M a y 2 01 8

F U N C T I O N A L

line of defence before a disaster strikes': I need their brain, skills and experiences, not a robot that repeats tasks until its energy runs out. A manual testing session can focus on specific areas, which allow testers to find more defects and use their skills at their best. You cannot define all the possible ways a human can use a product, especially a mobile app â&#x20AC;&#x201C; or maybe you can, but it would be terribly expensive and time-consuming. Get the basic scenarios in order, focus on the path that is most relevant for your customers/users, automate that path and its scenarios, and then ask your QA team to spend their time thinking about the product freeing them from the repetitive tasks. Their brain and their judgment is something that a computer doesn't have (yet)!

FUNCTIONAL VS NONFUNCTIONAL I won't go into the details of why you should care about functional and non-functional testing, because you should! They are both incredibly important and no product should be considered ready for release if it has not been tested in one of those two 'areas'. I often see engineers, testers, product owners, stakeholders and so on debating fiercely about those two 'areas'. If the person speaking is an engineer, an architect, or someone deeply involved in technical discussions then they will highlight how fundamental the non-functional tests are. If a PO or stakeholder is speaking, they will ask you to be sure that the product you want to release is aligned with the business requirements and objectives. Which one should you start with? Which one is more important? Well, that is something like the 'customer dilemma' well known and understood in the world of customer rewards. For example, you get customers who purchase more expensive airline tickets to collect miles for future vacations, fashionistas that get 30% off coupon that arrives in his/her mailbox sent by the nearby boutique, and those who get a free cup of coffee offered in response to the ten they have already paid for and purchased. The world of rewards usually serves two ends: To build customer loyalty and to entice new business. But is one more important than the other? And to whom should firms offer a better deal? We care about our customers, the way they use and engage with our products, as well as (hopefully) re-engage. A first time customer will probably be attracted

by the look and feel, the navigation, and the content offered by your apps. But the second time the use the apps, the app will have to be fast (the emotion of discovering your product features is gone at that point), reliable, and the user flow should be simple. Usability will also play a big role, as well as security. When I hear that functional testing ensures there are no major bugs and that non-functional testing stands up to customer expectations, I wonder if we are just considering our customers' trained animals, which hopefully will behave correctly so we won't have a problem with our apps! This should not a problem with how your app stores user data (security, so non-functional) but what about bugs? And what if an app slows down because the API we designed to communicate with the backend is not performing at its best (again, non-functional)? Those two tests are both extremely important and should have equal importance within your testing strategy. We want new and existing customers to be happy, as well as stakeholders, engineers, architects and the company CTO, because our apps are designed, implemented and tested to respect the high level of nonfunctional requirements that are set for the entire company. So next time you design a feature, ask yourself: Am I considering the two different tests? Do they have equal importance in my testing plan? Am I considering customer satisfaction, APIs and product reliability?

T E S T I N G

QA testers must break traditional working practices, develop new skills, be interested in software design and development, and be involved in many different stages of the delivery process

SET THE BASELINE Now that we understand that functional and non-functional tests are two equal citizens we can focus on non-functional testing. I remember a time when I was working for a major fashion company and we released a new feature to present an image gallery in one of our apps that apparently created a problem that some stakeholders reported: The app was not just slow, it was painfully slow. I remember how the PO went into panic mode asking everyone to "fully focus on the issue right now!" We rushed to our keyboards; typed as fast as we could to test the feature, monitored the status in production to help understand what went wrong between our tests, and looked at when we released it into production. It turned out after 30/40 minutes that we tested the features using JPG images, which were sized between 500Kb and 1024Kb, the design team decided

T E S T M a g a z i n e | M a y 2 01 8

F U N C T I O N A L

The fact that you end up with a very extensive coverage is the power of TDD: Once you implement TDD, you will be able to trust that your code works and the changes you apply won’t break the rest of the code in your programme

T E S T I N G

to publish ‘better-looking images’ the night before via the CMS that was powering the apps. The images were super sharp, beautiful, and showcased the products to their best. Unfortunately, their size was less beautiful and was in the range between 3 and 5Mb each. That, for obvious reasons, slowed down the entire gallery that had to load between 5 and 8 images every time. To triage this issue, we spent extremely valuable time with our engineers and testers, as well as a very stressed PO. But what could we have done to prevent that? Enter the magic world of defining a ‘baseline’: In performance testing it is important to understand what the current status is, so it is possible to verify at any given time if a change is affecting the product performances. If we had defined a baseline for the gallery feature we would have verified immediately that there was nothing in the app that was responsible for the degraded performances. The baseline alone won’t be enough: You have to check that your app performances are in a predefined range every time you apply a change in DevOps or every day in your production environment. Mobile apps are typically at the end of a long chain of systems and each one of them could, potentially, affect how the apps work. Using the predefined baseline as a way to compare the daily status of the chain will help identify any issue sooner, and reduce the impact it could have on your customers'. My team, for example, is able to check the performances every single day (we usually do this in the morning, but nothing would stop us running the same tests multiple times a day) so we can be sure to investigate and, if need be, fix the problem as soon as possible. If we need to introduce a new feature that has no impact on the performance (a simple UI change, for example) of our apps then we only run a regression test. On the opposite, any feature that may affect the performance of the app must be verified carefully.

GET INVOLVED One of the best practices that are generating good results is to have the QA team to actively participate in design, architecture and implementation discussions: They will be able to highlight any major flaws with the visual and architectural designs before anyone spends any time writing code. This, in turn, will reduce the development cycle

T E S T M a g a z i n e | M a y 2 01 8

and helps to achieve delivery goals.

CHOOSE WELL Every release is different, simply because in every release cycle we add a specific feature or refactor specific areas of the apps. When we work with our product owner to prioritise the stories for the next sprint we also prioritise our testing effort, focusing on those parts of the app that are touched by the changes we agreed on with our stakeholders.

PRIORITISE YOUR BUGS We use the analytics tools to understand the impact of a bug, using two kinds of information – crashes and usage data. If the app crashes we always want to fix the problem as soon as possible, but, sometimes, it is not possible to do this straight away; so we assess the severity of the issue by checking how many customers/ devices are affected by the crash. Similarly, if the bug is not causing the app to crash then we verify usage data: Is the bug appearing in an area that is often used by our customer? The priority of the fix will be high and the opposite if the bug is happening in a view that is rarely used by our customers.

KEEP SOME SANITY When you go into production do not forget to run your post-release process: Once a release is live, run a sanity check on the apps using production accounts so you know that your customers can see what it is expected and that the apps are not crashing on the multitude of devices supported. You should also ensure third-party systems that interact with apps are correctly integrated.

THE GHOSTS WILL HAUNT YOU Maintaining product quality while delivering new features and changes is a difficult task: QA testers must break traditional working practices, develop new skills, be interested in software design and development, and be involved in many different stages of the delivery process. Do it right: The quality of your products will go up and the time you need to develop will shorten. Do it wrong: The ghosts of your quality process will haunt you at some point.

froglogic Squish GUI Test Automation Code Coverage Analysis

cross platform. multi language. cross design. Learn more and get in touch: www.froglogic.com T E S T M a g a z i n e | M a y 2 01 8

IDEAL TESTING APPROACH FOR IOT ECOSYSTEM FUNCTIONAL OR AUTOMATED?

Project Manager, QA at ValueLabs explains the value of IoT ecosystems by demonstrating ‘a grid of assorted and interconnected layers’ T E S T M a g a z i n e | M a y 2 01 8

F U N C T I O N A L

oT (Internet of Things) is an organised which connect with each other as part ecosystem, encompassing entire gamut of initiating an interaction/transaction of interconnected objects like computing of data. But, to enable this among the devices, mechanical and digital machines, wide spectrum of devices, it demands which are accessed through the internet for an implementation of huge set of and have the ability to collect, transfer and/ communication protocols (both regular or exchange data. In today’s digitalisation and custom) prominent among them is world, IoT has a significant role to play Modbus, LLRP etc. Importantly, just like in helping businesses grow, empowering anywhere, the protocols are essential organisations to make informed decisions to transform data, bringing them to a for the fact that IoT facilitates a framework standard format. to rope in critical categories of data (which 2. DATA INGEST/TRANSFORMATION/ is of transactional and interactional nature) EXCHANGE LAYER: This layer is intended IDEAL TESTING APPROACH FOR IOT ECOSYSTEM - from multiple places and analyse the same to process the data received from the thoroughly. Importantly, in recent times, devices such as sensors, actuators, RFID FUNCTIONAL OR AUTOMATED? growing organisations are more inclined to tags etc. The data from the devices can bs explain adopt the value of IoT ecosystems by demonstrating IoT for achieving optimisation in on- ‘a grid of beassorted ingestedand through any of the available nected layers’ going operations, cost reduction, scaling and data facilitation mechanism such as TCP/ efficiency. the two fundamental IP Communication, Web-services (SOAP, ternet of Things) is However, an organised ecosystem, encompassing entire gamut of concerns and/or open challenges with IoT REST API) etc. However, the prerequisite nnected objects like computing devices, mechanical and digital machines, which are assuring qualityand andhave overcoming security for data and/or to get ingested, transformed or essed through the internet the ability to collect, transfer risks, since predominantly, IoT is all about exchanged seamlessly is that it should be e data. In today’s digitalisation world, IoT has a signiBicant role to play in helping the unprecedented volume of data. It is in a standard format. ses grow, empowering organisations to make informed decisions for the fact that imperative to give thoughtful consideration 3. DATA PROCESSING LAYER: this layer itates a framework to rope in critical categories of data (which is of transactional ensuring quality through a systematic and the same forms the crux of the entire IoT ecosystem eractional innature) from multiple places and analyse thoroughly. ntly, in recent times, growing organisations are more inclined to adopt IoT for critical modules/ well-defined IoT testing. since it contains ng optimisation in on-going operations, cost reduction, scaling components and efBiciency. that work together to enable r, the two fundamental concerns and/or open challenges with IoT are assuring IOT ECOSYSTEM the analysis of the transformed data, by and overcoming security risks, since predominantly, IoT is executing all about certain the real-time programmes dented volume of data. is imperative to give thoughtful consideration In current times,It many eminent technology and analytics.in g quality through a systematic and well-deBined IoT testing. enthusiasts, researchers and expert groups 4. APPLICATIONS/BUSINESS LAYER: This have come together and contributed their layer outlines the wide range of available osystem thought process about IoT and its ecosystem. applications/products that can readily Thankfully, it has given a scope for having make use of the processed data sets nt times, many eminent technology enthusiasts, researchers and expert groups constructive definitions, perceptions and and leverage by catering most promising me together and contributed their thought process about IoT and its ecosystem. ully, it has interpretations given a scope about for having constructive perceptions and services across different IoT. However, fromdeBinitions, a and optimising tations about IoT. However, from a naïve standpoint, the best way to explain an or segments. naïve standpoint, the best way to explain business domains system is by putting it as ‘a grid of assorted and interconnected layers’ – IoT is a an IoT ecosystem is by putting it as ‘a grid of ecosystem comprising of following: assorted and interconnected layers’ – IoT is a layered ecosystem comprising of following:

ices Layer • Devices Layer

• D ata Ingest / Transformation / a Ingest / Transformation / Exchange Layer Exchange Layer • Data Processing Layer a Processing Layer • Applications / Business Layer

lications / Business Layer

* 04 layers together form the IoT Ecosystem. * Chronological order - top to bottom

The key thing about the ecosystem is that all of the aforementioned layers are arranged IS IOT DATA SETS CENTRIC OR DATA-SETS thing about the ecosystem is that all of the aforementioned layers are arranged in proper chronological order and can be DRIVEN? er chronological order and can be read and/or interpreted either in bottom-top read and/or interpreted either in bottomh or vice-versa, since each layer is unique in itself in terms of objectives and top approach or vice-versa, since each layer As of today, it’s still an argumentative debate nce. So, let’s take a holistic view of the individual layers to understand what each asses and its functions. is unique in itself in terms of objectives and to understand and appreciate the fact IoT importance. So, let’s take a holistic view of is data-set centric or data-set driven. But, VICES LAYER: As the name indicates, this layer typically encompasses variety of the individual layers to understand what each taking the aforementioned IoT ecosystem ces (like sensors, RFID tags, smart meters, smart phones, drones etc.), which encompasses and its functions. explanation into consideration, it is very nect with each other as part of initiating an interaction/transaction of data. But, 1. DEVICES LAYER: As the name indicates, evident that there is a co-existence of number enable this among the wide spectrum of devices, it demands for an this layer typically encompasses variety of heterogeneous entities which are logically lementation of huge set of communication protocols (both regular and custom) of devices (like sensors, RFID tags, smart tied together and completely driven and meters, smart phones, drones etc.), controlled by rich data-sets i.e. without the

T E S T I N G

Growing organisations are more inclined to adopt IoT for achieving optimisation in ongoing operations, cost reduction, scaling and efficiency

NARAYANA PROJECT MANAGER, QA VALUELABS

Narayana is a Computer Science and Engineering Graduate, currently working as Project Manager, QA. He has more than 11 years’ of experience in validating the applications and products that are built on open-source technology stacks, and shows a keen interest in assessing the system’s functional and non-functional attributes, as well as investigating and implementing new testing approaches, tools, techniques and methodologies.

T E S T M a g a z i n e | M a y 2 01 8

4. APPLICATIONS/BUSINESS LAYER: This layer outlines the wide range of available applications/products that can readily make use of the processed data sets and leverage by catering most promising and optimising services across different business domains or segments.

F U N C T I O N A L

T E S T I N G

IS IOT DATA SETS CENTRIC OR DATA-SETS DRIVEN?

As of today, it’s still an argumentative debate to understand and appreciate the fact IoT is data-set centric or data-set driven. But, taking the aforementioned IoT ecosystem explanation into consideration, it is very evident that there is a co-existence of number of heterogeneous entities which are logically tied together and completely driven and processed data-sets flowing from one layer to controlled by rich data-sets i.e. without the processed data-sets Blowing from one layer another in the ecosystem, it is definitely hard to another in the ecosystem, it is deBinitely hard to imagine the existence of the IoT.

to imagine the existence of the IoT.

It is learnt that IoT is data (or data-sets) driven, being it an interactional or transactional in nature. Now, surprisingly, the very driving factor itself has turned out to be a biggest challenge for IoT because per Gartner, by 2020, there will be over 26 billion connected devices, and on similar lines one other research (Morgan Stanley, Hammersmith Group) study forecasts that the count could go up to 100 billion, which scientiBically or statistically means there will be at least three times more IoT devices in comparison to world population. This straight away technically infers that the underlying data that is ingested, transformed and exchanged is going to be increased in unprecedented volume, which gives rise to following two fundamental challenges. •

Sr. No

Processing Layer

Data-sets Blowing across layers

Data Ingestion / Transform Layer

Devices Layer

IOT KEY CHALLENGES It is learnt that IoT is data (or data-sets) driven, being it an interactional or transactional in nature. Now, surprisingly, the very driving factor itself has turned out to be a biggest challenge for IoT because per Gartner, by 2020, there will be over 26 billion connected devices, and on similar lines one other research (Morgan Stanley, Hammersmith Group) study forecasts that the count could go up to 100 billion, which scientifically or statistically means there will be at least three times more IoT devices in comparison to world population. This straight away technically infers that the underlying data that is ingested, transformed and exchanged is going to be increased in unprecedented volume, which gives rise to following two fundamental challenges. • Storing, administering and handling such huge volume of data-sets. • Ascertaining the quality characteristics (both functional & non-functional) to ensure that IoT devices perform as intended both at a component level as well as a data-set level.

Test Scope

Sensors

▪ ▪ ▪ ▪ ▪

Interfacing Application

▪ ▪ ▪ ▪

Network Communication

Underlying / Supporting Database

▪ ▪ ▪ ▪ ▪

Device Hardware Embedded Software Performance Device Response Time Fault Control / Handling Mechanism Functionality UI Capabilities Role and Access controls Request handling Connectivity Data transmission

Data consistency Data transmission/loss Data encryption/decryption

Importantly, it is learnt that a combination of functional + non-functional testing is more effective and result oriented when compared to implementing test automation, for the following major reasons: • Primarily, IoT is not for a specific business domain, and likewise, the corresponding field of application is never the same. • Diverse business domains are driven by bundles of complex functionalities, which demands expert intervention and validation. • Above all, it is imperative to have embedded testing both considered and included on top of regular testing for IoT, which is not a correct grouping for test automation.

TESTING TYPES MATRIX FOR IOT Importantly, it is learnt that a combination of functional + non-functional testing is more effective and result oriented when compared to implementing test automation, for the

following major reasons: Now since it is learnt that IoT is data-set • Primarily, IoT is not for a speciBic business domain, and likewise, the corresponding driven, with data being assimilated from Bield of application is never the same. different sources indriven an by IoT platform, it is which • Diverse business domains are bundles of complex functionalities, demands expert intervention and validation. critical to have testing conducted across • Above all, it is imperative to have embedded testing both considered and included on following testing types. The below grid top of regular testing for IoT, which is not a correct grouping for test automation. TESTING TYPES MATRIX FOR IOT outlines the holistic view of the different, Now since it is learnt that IoT is data-set driven, with data being assimilated from major (and recommended) testing types for different sources in an IoT platform, it is critical to have testing conducted across following testing types. The below grid outlines the holistic view of the different, major validating IoT platform: (and recommended) testing types for validating IoT platform: Sr. No

Objectives

Functional Testing

Validate the functionality of the application against requirements

Usability Testing

Primarily to assess user friendliness of the application

Database Testing

Performance Testing

Security Testing

Compatibility Testing

Testing Type

1 2

IOT From a testing standpoint, knowing the IoT ecosystem alone will not help test specialists formulate a comprehensive test strategy or define a test approach, but it is equally critical to know and attain detailed understanding about the essential key components that

IoT Component

TESTING APPROACH FOR

T E S T M a g a z i n e | M a y 2 01 8

Storing, administering and handling such huge volume of data-sets.

• Ascertaining the quality characteristics (both functional & non-functional) to ensure function together and collaborate to have an that IoT devices perform as intended both at a component level as well as a data-set IoTlevel. work as intended. So, once we have the TESTING APPROACH FOR IOT list of components, one can easily ascertain From a testing standpoint, knowing the IoT ecosystem alone will not help test specialists the test scope. The below grid outlines formulate a comprehensive test strategy or deBine a test approach, but it is equally critical to know and attain detailed understanding about the essential key components the details of the key IoT components and that function together and collaborate to have an IoT work as intended. So, once we have corresponding test scope details: the list of components, one can easily ascertain the test scope. The below grid outlines the details of the key IoT components and corresponding test scope details:

Applications / Business Layer

IOT KEY CHALLENGES

Network Testing

Validate data types, values, integrity and data consistency

Validate the response time of the components (sensors etc.) Validate the data reading, writing and retrieval ability of component. Validating data privacy, network security, user roles and corresponding access.

Validate connectivity and underlying protocols.

Validate functioning of components across varied device hardware, communication protocols, operating systems etc.

registrations are now open KEY DATES 14 September – Submission Deadline 4 October – Finalists announced 21 November – Awards Gala Dinner

Now in its sixth year The European Software Testing Awards celebrate companies and individuals who have accomplished significant achievements in the software testing and quality assurance market. Enter The European Software Testing Awards and start on a journey of anticipation and excitement leading up to the awards night – it could be you and your team collecting one of the highly coveted awards.

Be a winner - REGISTER TODAY

 www.softwaretestingawards.com T E S T M a g a z i n e | M a y 2 01 8

THE FUNCTIONAL TESTING CONUNDRUM Functional testing is the empowerment of testers and developers to guarantee that functionalities of an application are in a state of harmony, according to QA Manager at The Economist

T E S T M a g a z i n e | M a y 2 01 8

F U N C T I O N A L

o me, testing is more than just making sure your application works as you expect. It also functions as a form of communication across a team and application, meaning another developer can confidently begin contributing to an application, using the tests as a guide. The greater complexity of software applications and its increasing competitiveness has created the need for more efficient and exhaustive testing processes. When you have a complex piece of software the importance of functional testing is key, it de-risks going live with a poor product which does not meet customer expectations.

EMPOWERMENT OF TESTERS AND DEVELOPERS Functional testing (FT) is the empowerment of testers and developers to guarantee that the highlights and functionalities of an application are in a state of harmony. It answers questions like ‘can the user do this’ or ‘does the particular feature work’. Without these, a user could be impacted in a huge way. A black box testing approach where test cases are prepared, keeping compliance with client requirements in mind. Take a new phone, for instance, if you cannot surf the internet it may be deemed worthless, FT would have picked up this flaw with consummate ease if done efficiently. Non-functional tests look at its overall performance (e.g. testing scalability, reliability, security and compatibility). If you asked me in the early stages of my career, functional testing would have primarily focused on manual testing and more efficient ways of working. 5 years into my career the big buzzwords were around “automation”. Big advancements in the next few years will be “machine learning” and “AI”. WHAT FUNCTIONAL TESTING CAN INVOLVE • UNIT TESTING: Focuses on a single “unit of code”, such as a class method. • INTEGRATION TESTING: Builds on the unit test by testing multiple pieces of your application together. • SMOKE TESTING: addresses the most basic functionalities crucial to the working of the product. • REGRESSION TESTING: Ensures that previously working functionality is still functional. • USER ACCEPTANCE TESTING: End-users review the functioning of the product.

A key concept is to have a solid set of unit tests, issues can be found early and refactoring can take place with confidence. With FTs there is a safety net rather than the first line of defence. Unit tests should be the car engine and FT’s should be the paintwork. Because of the high scope of tests, this essentially means that there is a low level of abstraction. In an ideal situation, the tests should be defined before the development starts and after the product requirements have been defined. From experience, tests should not be driven by software development processes but should be driven by what the user expects the function of the software to actually be.

T E S T I N G

The greater complexity of software applications and its increasing competitiveness has created the need for more efficient and exhaustive testing processes

CONDITIONS FOR A GREAT FUNCTIONAL TEST • Derived from the product requirements. • Clearly defined and understood by not just a QA person but also the whole team. • Visible to stakeholders. • Gives actionable information to developers. • Audited frequently to ensure they are updated with product requirements. • QA clear on the functionality of the system. • The correct set of data. • Test cases cover all possible scenarios.

CHALLENGES The risks of seeing the functional test as a repeatable unit mean that they will not be adaptable and issues can be missed. The functional test should be seen as an interactive component of a QA process. 1. Definition of clear and complete test requirements. 2. Managing changes in requirements. 3. Dealing with functional gaps in test plans. 4. The difficulty faced by the development team whilst reviewing large test plans. 5. Ensuring that you have optimum test coverage. 6. Dealing with blocking issues. TDD APPROACH TO FT Refers to a process by which you develop and test your code. Under a Test Data Driven (TDD) paradigm, your development process looks something like this: 1. Before coding, you write a test. 2. Run the tests along with any other tests. 3. Write the minimum amount of code necessary to make the tests pass. 4. Refactor. 5. Repeat the process. With TDD, you can get broad test coverage of your application. This gives you the confidence

DILEEP MARWAY QA MANAGER THE ECONOMIST

Dileep has more than 10 years’ experience as a Software Testing Professional and is extremely passionate about quality assurance, with his key principle being “putting customers first”.

T E S T M a g a z i n e | M a y 2 01 8

F U N C T I O N A L

Advancements in testing processes now mean that more efficient ways of working are now available

T E S T I N G

to make changes to your application as it grows, as you can see older tests passing and trust that your changes have not regressed. BDD / BEHAVIOUR-DRIVEN DEVELOPMENT APPROACH TO FT

AGILE AND CONTINUOUS

Behaviour Driven Development is a process by which you can approach development and testing. BDD is more focused on how you test than when you test. With BDD, you focus your tests on behaviour, rather than implementation, ideally starting from your customer's/user's expected experience. In my view leads itself more so to functional testing. Advancements in testing processes now mean that more efficient ways of working are now available. For instance, the use of the Cucumber tool (BDD) for automating browser tests and Gherkin for human-readable syntax. This allows you to provide a business user with detailed scenarios which outline how someone uses the system. Also, the use of user stories early in the life cycle gives clear product expectations, making it easier to test. e.g. User story – Companies are using the job search application and they should be able to create a job application. FT: • Open application • Log in to the company • Create a job posting

TESTING

Based on my experience, a user story should have explicit acceptance criteria. Good acceptance criteria are ones that “must” resolve to either a “pass” or “fail” after observations are made.

In the future, machine learning driven test automation will be the key driver. In most companies, user interface (UI) elements are always changing, which means that test scripts fail as the UI has changed and tests are brittle. Artificial intelligence and machine learning mean that test adaptation can be done easily, this provides a workaround and the user is asked to ‘accept’ or ‘deny’ the workaround. To which the piece of test will adapt accordingly.

AUTOMATION VS MANUAL TESTING Automated test processes have helped to make processes more efficient, especially tasks such as regression testing. These processes have meant that functional testing can be done in a quicker and efficient way, leaving manual testers to work on edge cases. UI automation is the act of conducting specific tasks via automation. Automation can help to run tasks more quickly and do so automatically based on conditions being met. Not all tests can be automated; we still need manual testing in a variety of cases. It makes sense to take a manual testing approach when you’re dealing with legacy systems that don’t easily support automation when you need to adhere to strict regulations and

T E S T M a g a z i n e | M a y 2 01 8

require documentation as a result. Automation is more about checking or confirming that something is true than it is for testing or recognising unanticipated problems.

API testing creates more reliable code. Historically, testing would take place at the graphical user interface (GUI) level. When a developer would finish their work, they would hand it off to the QA engineer. The engineers had limited time so they would test the code at the highest level - the GUI. This would cover both the frontend and the backend development. This worked for manual testing and for the beginning of automation testing but isn’t right for the age of agile and continuous testing. GUI testing is brittle and automated scripts break easily. In addition, teams can’t wait for the entire system to be updated and the GUI to be ready before testing occurs. In the age of agile, testing must take place at a lower level of abstraction (API level). API tests can be created before development is complete. This means developers can validate their code based on pre-written tests (Test Driven Development).

MACHINE LEARNING

SUMMARY Today’s users are tech savvy, they use software in nearly every aspect of their daily lives and they have high expectations for how this software should work in order to help them go about their everyday needs better, smarter and faster. As a result, software testing has become increasingly important to and intertwined with the entire software development process. Whether your team is using hardcore testdriven development (TDD) or simply wants to automate otherwise manual QA, FT benefits from clear, explicit user stories have little room for ambiguity.

T E S T M a g a z i n e | M a y 2 01 8

5 KEY DRIVERS FOR SOFTWARE TESTING IN 2018

AI DEVOPS

PREDICTIVE AND PRESCRIPTIVE QUALITY ANALYTICS

AND MACHINE LEARNINGÂ

TEST

UNIFIED

AUTOMATION

TESTING PLATFORMS

DRIVING THE DIGITAL CHANGE QMetry explores some major drivers that are becoming extremely important in 2018

T E S T M a g a z i n e | M a y 2 01 8

D E V O P S / A G I L E

oftware testing has seen two major transitions in the past decade. The advent of agile heralded the shift from manual to automation and then in recent times, DevOps is driving the 2nd wave of change for testing teams with continuous integration and delivery. Although, in many ways, we are seeing some of the old practices coming back to life again in testing. Let us explore some major drivers that are becoming extremely important in 2018 and will remain in the future.

DEVOPS The DevOps paradigm and continuous delivery have led to a digital shake-up for enterprise IT. With DevOps comes a new kind of agility, compressed release cycles, enhanced application quality and above all a renewed synergy between development and operations. This breaking of barriers is essential to match the transformation journey. Yet, it is not just speed and quality; the business value of DevOps is quite remarkable. It evolves the paradigm of continuous integration to continuous development. Simply put, the code that is written and committed to version control is now being built, developed, tested and installed in the production environment for consumption. This helps the entire ecosystem because environments and processes become standardised and every action in this chain becomes automated. Software testing role becomes crucial here as ‘test early, test frequently’ is the key to achieving better quality software for DevOps teams. This requires the effective use of key enablers like test automation, continuous integration, capability for continuous feedback and the right mix of tools and process for DevOps.

TEST AUTOMATION But with great power comes great responsibility. Testing is one of the key pieces in the continuous delivery puzzle and organisations are now scaling up their agile and DevOps services toward continuous testing and continuous delivery. There is high pressure to fix defects urgently, in order to reduce the technical debt and achieve the pace that is set by agile and DevOps thinking. Thus, as companies shorten their

I N N O V A T I O N S

sprint cycles and increase their release velocity, testing must match both the high frequency and volume needs. Manual testing is simply not enough to accomplish this pace. Test Automation is one of the key drivers for DevOps and digital transformation. greatly increases your coverage and accuracy. Test automation is at the heart of your CI/CD pipeline to achieve the coverage and accuracy. From faster feedback loops, reduced expenses, reusability, organisation and faster time to market, automated testing vastly improves your efficiency and overall quality.

With DevOps comes a new kind of agility, compressed release cycles, enhanced application quality and above all a renewed synergy between development and operations

AI AND MACHINE LEARNING Modern test management has the advantage of information. The vast amounts of test data and results produced by their test automation suites hold valuable insights and intelligence. And while it is manually impossible to wade through all this data, BOTs can quickly analyse terabytes of information. BOT-enabled tools can then produce actionable insights and optimisation recommendations. These analytics are useful to detect and reduce/remove performance bottlenecks and also identify areas of most failed scenarios or tips/ insights on critical areas. Artificial intelligence, machine learning and the use of BOTs have emerged as the major disruptive forces in the current digital landscape. By leveraging the power of AI and ML in their test automation efforts, businesses can significantly increase their ability to fulfil time to market pressures while also meeting the stringent quality needed for desired business outcomes. These AI-and ML-led smarter test automation solutions – also known as Intelligent Testing will be the core differentiators in achieving frictionless automation and continuous feedback.

RUTESH SHAH CEO, INFOSTRETCH CORPORATION ADVISOR, QMETRY

Rutesh is a highly reputed Quality and Management Consultant in Silicon Valley, CA. He founded InfoStretch Corporation with a vision to provide

UNIFIED TESTING

value-added solutions that leverage

PLATFORMS

technology and innovation to

The organic progression of software testing organisations has led to the use of various legacy tools and processes, and teams with a wide variety of scripting languages and methods. This weighs down the test

significantly improve software quality. Prior to founding InfoStretch, he was Vice President of Reliability Services at Arsin Corporation.

T E S T M a g a z i n e | M a y 2 01 8

D E V O P S / A G I L E

DevOps and digital transformation have set the pace from a highly proactive, fast and integrated delivery culture

I N N O V A T I O N S

automation efforts as it leads to duplicity of efforts, lack of reusability and high maintenance overheads. The cumulative outcome is increased the time to market and non-optimisation of software quality. DevOps and digital transformation have set the pace for a highly proactive, fast and integrated delivery culture. The confidence in the quality and timeliness of releases requires a unified approach that transcends the siloes and barriers. This has led to the era of tightly integrated, unified testing platforms that address various testing needs and challenges under one umbrella. Modern, unified solutions provide a holistic test management solution that optimizes the testing cycle with end-to-end coverage and integrates with your existing

automation, CI/CD tools and project management tools.

PREDICTIVE AND PRESCRIPTIVE QUALITY ANALYTICS Delivering quality at the speed of business is a digital imperative. This needs businesses to define the requirements clearly, use the right infrastructure, and use established processes. AI and ML algorithms have made it possible to mine large volumes of data from automation suites and test management tools to gain insights and intelligence. Predictive analytics can help you shorten the testing cycles by optimising your process, anticipate defects early and set the priorities for testing that will have the greatest impact. Prescriptive analytics goes one step further and makes automation smarter by providing a holistic view of root causes and failures, heat map-visualisations and providing recommendations that are both actionable and accurate.

CONCLUSION Delivering software quality continuously is the need of the day in the era of digital transformation. As the lines continue to blur with DevOps and agile practices enabling more fluid and transparent work practices, integration and visibility are essential for effective test management. Software test automation holds the power to increase your coverage, depth and scope and solve many of the prevalent challenges. But it is intelligent test automation that is the real game-changer, with its power of data, AI and machine learning. Intelligent testing takes quality engineering to the next level in the DevOps context by giving you an unparalleled competitive advantage. That of optimising testing activities, instant feedback, faster data-driven decisions and self-adaptive software testing for your quality lifecycle. Smarter testing led by AI and BOTs has beckoned the third wave in the software testing ecosystem. If you would like to read more about how intelligent test automation is changing the software quality landscape, download our white paper on the subject.

T E S T M a g a z i n e | M a y 2 01 8

registrations are now open KEY DATES 24 August – Booking & Submission Deadline 12 September – Finalists announced 16 October – Awards Gala Dinner

Be a winner - REGISTER TODAY

 www.devopsindustryawards.com T E S T M a g a z i n e | M a y 2 01 8

INNOVATION: THE HOLY GRAIL OF BUSINESS Head of Test Common Services at DVLA, Wayne Tofteroo, believes innovation through agile methodologies is the 'holy grail of business'

T E S T M a g a z i n e | M a y 2 01 8

D E V O P S / A G I L E

nnovation is the holy grail of business. In the current business world you either innovate, or die, and this is clear from the changing face of retail and banking – to name two sectors experiencing rapid change. Yet, innovation is not something that can be taken out of the box and rolled out – you can’t just buy it off the shelf. It can’t be turned on when it’s needed. In fact, it is probably one of the most misunderstood concepts in business. Relabeling an organisation will not, on its own, suddenly spark innovation. Creating an office space with skateboard walkways, on its own, will merely create a fun place to work, but will not guarantee innovation. Innovation in an existing organisation is much harder to perform than the creation of an innovative start-up. Can a large organisation innovate? Amazon and Google seem to exist purely to innovate. Will the rate of innovation at Amazon and Google slow down as they mature or have those companies been designed with a corporate mentality of constant innovation? It is not money or the size of the corporation that will dictate this. “Innovation has nothing to do with how many R&D dollars you have. When Apple came up with the Mac, IBM was spending at least 100 times more on R&D. It’s not about money. It’s about the people you have, how you’re led, and how much you get it!” – (Steve Jobs) I think the answer to the riddle is in the nature of innovation itself. Innovation is a state of mind; it is not something that is turned on by request. From the quote above, you have ‘to get it’, as well as the people and the leadership within the workplace. Innovation often happens in your favourite coffee shop or on your journey home, not necessarily in your skateboard equipped office space. In my experience, it comes from a feeling that you can do better, a desire to make a change, the feeling of freedom to be able to experiment and attempt something new, as well as to see a problem and know you can do better. Innovation is personal and a 'team thing', as one person will rarely be able to turn an idea into reality.

WHAT CREATES INNOVATION? So, what is it that creates innovation and what stifles it? Innovation is clearly

I N N O V A T I O N S

about people, attitude and the freedom to experiment without the fear of failure. What can an organisation do to stimulate and encourage innovation? One factor is leadership, and how you lead your team or organisation will directly affect the level of innovation. Another factor is how you structure teams and manage them. The growing number of companies will give employees time and resources to experiment and to innovate. Waterfall software development has been copying methods used successfully for large high-risk engineering projects that often take years to complete. However, engineering projects require predictable outcomes. If you are building a three-lane bridge, you can’t suddenly change this to a four-lane bridge half way through! These outcomes are guaranteed by the use of highly structured and formalised processes. The innovation stage is at the front-end of the process before the idea is then locked down and built. Once the idea has been agreed, the waterfall process will relentlessly work its way to delivering exactly what was agreed on at the start of the process. A change will be costly, difficult, and time-consuming. While this is the only way you can build a bridge or a skyscraper, it is not the only way to build software. For an existing business or a startup, software will be the key enabler of growth. We need to maintain our software so we can react to sudden changes such as the need make a quick response to a competitor or a change in the marketplace. If we build software exclusively like we build skyscrapers or bridges, then we will be tied into long periods of development with little opportunity for change. How does that work in today’s rapidly changing business world? It doesn’t.

WHY AGILE? In the early 1990’s, a number of competing development methodologies started to appear to try and address the issue of long development cycles and the difficulty of changing existing software to keep up with business changes such as scrum, extreme programming (XP); crystal; adaptive software development (ASD); feature-driven development (FDD); and the dynamic-systems-development method (DSDM). It was clear that software development needed to break away from

Innovation is not something that can be taken out of the box and rolled out – you can't just buy it off the shelf. It can't be turned on when it's needed. In fact, it is probably one of the most misunderstood concepts in business

WAYNE TOFTEROO THE HEAD OF TEST, COMMON SERVICES DVLA

Wayne has provided Test Leadership for a wide range of strategic programmes and enterprise-wide transformations. He is a proven Innovator and Founder of several innovative start-ups, including the leading provider of drug discovery software, IDBS.

T E S T M a g a z i n e | M a y 2 01 8

D E V O P S / A G I L E

The rapid expansion of consumer computing in the 90’s, the internet, e-commerce, and later the first smartphones which enabled mobile computing, forced the software industry into creating agile

I N N V A T I O N S

development methodologies designed to support large ongoing timescale engineering projects. In 2001, 17 developers who called themselves “organisational anarchists” met in Snowbird, Utah’, and from this meeting a single approach arrived. The term agile was coined for the new approach. It was adopted from the title of a book, “Agile Competitors and Virtual Organizations: Strategies for Enriching the Customer”. The book gave examples of companies that were creating new ways of adapting to more turbulent markets. Being able to quickly react to turbulence was the defining quality that they were looking for. This was swiftly followed by the “Manifesto for Agile Software Development”, which spelt out four key values. They then developed 12 operating principles, called “Principles Behind the Agile Manifesto”. From 2001 onwards, all development frameworks that aligned with these values and principles were then known as agile techniques.

FOUR VALUES OF AGILE DOCUMENTED IN 2001

• Individuals and interactions over processes and tools • Working software over comprehensive documentation • Customer collaboration over contract negotiation • Responding to change over following a plan

HOW DOES AGILE SUPPORT INNOVATION? How can the agile approach affect innovation? It’s clear from what we know of innovation that a focus on individuals responding to change, an environment

T E S T M a g a z i n e | M a y 2 01 8

which favours interactions over documentation, and a close collaboration with customers will create an atmosphere fosters innovation. Agile has short cycles of development, a review after each cycle, full-time involvement from the product owner (or the owner of the idea), the opportunity for experiment, and innovation is built into the methodology. A concept in agile is ‘fail fast’; an idea can be turned into a working solution quickly and it can then be tested to see if it meets the hopes of the developers. If it fails, then this will leave the team to be able to move onto another idea with little time or money wasted. This also enables teams to experiment and develop new ideas with little risk. The rapid expansion of consumer computing in the 90’s, the internet, e-commerce, and later the first smartphones which enabled mobile computing, forced the software industry into creating agile. When the Snowbird meeting happened, the world was waiting for agile, so the Snowbird attendees delivered.

AN AGILE WORLD? In July 2008, the Apple App Store opened with 7.7k games and 15.3k apps. As of July 2017, those figures had risen to 783k games and 2,338k apps. In that time period, 180 billion apps had been downloaded from the App Store. There are a number of factors that have enabled this growth. However, this growth would not have been possible without the widespread adoption of agile methods. Agile has also arguably been the trigger for a wave of new development tools that support agile development and support an environment where high-quality software can be developed quickly and changes to software can be done quickly and efficiently. The ultimate culmination of nearly 20 years of method and tool improvements is the DevOps revolution. The breakaway from innovation stifling waterfall methods to innovation enabling agile methods is now possible for most business scenarios. The increasing number of start-ups, new apps, and the digital transformation occurring in existing businesses is a testament that innovation is now fully supported by the agile world.

new for 2018 49

30th OCTOBER 2018 See the future of DevOps and Software Testing A one-day event held in Central Glasgow that attracts up to 100 senior software testing and DevOps professionals, eager to network and participate in targeted workshops, interactive Q&A sessions, and lively presentations â&#x20AC;&#x201C; all housed in the impressive Principal Grand Hotel. DevTEST Summit is open to all individuals or organisations within the Software Testing and DevOps community who are keen to increase their knowledge and harvest workable solutions to the various issues faced in these burgeoning and complex sectors.

REIMAGINING QUALITY ASSURANCE FOR DIGITAL IT Infosys and Delphix held a roundtable in April 2018 in London to discuss the differences between QA and agile

T E S T M a g a z i n e | M a y 2 01 8

D E V O P S / A G I L E

he roundtable attendees included Directors and Heads of QA/testing from various industries. The session speakers were Tom Reuner, Managing Partner, HfS Research, Shishank Gupta, VP and Delivery Head, Infosys, and Gary Hollam, Director of Sales Engineering, Delphix. Continuous delivery is becoming an important business priority for organisations competing in today’s fastpaced economy. With time-to-market being a key differentiator, many IT organisations are looking for new technologies and tools that accelerate their delivery cycles through automation. However, uncovering inefficiencies within existing processes entails analysing copious amounts of data to gain accurate and actionable insights. The ultimate goal here is to improve the quality of the final deliverable whether it is software, mobile or web to enhance the customer experience. The various compliances and regulations (including the European Union’s General Data Protection Regulation (GDPR)) pose several challenges for managing data in an agile environment, which in turn affects how testing is done.

I N N O V A T I O N S

THE CHANGING NATURE

Shishank further explained the changing nature of QA: “Previously, development teams relied on marketing teams to understand customer sentiment and accordingly make changes in the back-end systems. Today, the roles are changing and QA teams themselves are using cuttingedge tools.” He highlighted how Sentiment Analytics tools can be used by such teams to understand customer sentiment and identify defects that lead to poor experiences. For instance, QA teams can use application logs in shift-left testing to predict defects. Alternatively, these teams can use customer sentiment analytics in shift-right testing to gauge whether the problem is performance or experiencerelated. Fixing these issues in real-time will enhance customer experience and improve app ratings. Additionally, with customer experience tools, one can evaluate accessibility, security and usability of the final product, such as the website or the mobile app. These parameters are tracked across every release to determine customer satisfaction with new releases, as well as provide companies with a direct correlation (through actionable insights and metrics) on how effective software delivery and quality improve customer sentiment.

OF QA

'AI IS MORE THAN A

In the opening roundtable session, Shishank Gupta spoke about the nature of agile QA. Elucidating the differences between traditional QA and agile QA, he said: “Traditional QA is linked to the waterfall methodology and involves semi-annual and annual releases. Requirements are formally noted, reviewed several times and finalised by the business. Then these are sent for execution across downstream activities such as architecture, design, testing, etc. In such a scenario, delays of nearly a week don’t matter much. However, in a digital enterprise, even a day’s delay can cost the business significantly.” He stressed the need to reduce the time taken between generating and executing the requirements to mere seconds instead of days or weeks. Further, requirements gathering is often not limited to just the business; it involves customer feedback as well. Shishank warned that, to move forward, organisations should sense what the market wants and respond dynamically rather than depend on a set of static requirements. This is where technologies such as artificial intelligence and machine learning (AI and ML) come in handy to help make QA agile.

CONCEPT IN QA' “This is where deep system knowledge and deep learning become important. AI is more than a concept,” says Shishank. Companies should invest in building a knowledge repository based on past trends. Unlocking the value in such data allows companies to use advanced techniques such as intelligent pattern matching, defect prediction and autonomous testing to increase agility. Tools such as application crawlers and continuous learning feed greater data inputs into the repository, making it more powerful. Shishank ended the session by emphasising that using AI/ML tools in QA puts the focus on customer-centricity through delightful end-user experiences. He said, “as is the case with automation, AI/ML are means to an end, the end being shorter time-to-market. It goes beyond the views of business analysts and experts to understand real-time market demands, new trends, competitor products, and customer feedback.” Once the process of gathering customer feedback is automated, companies can instantly analyse huge

The various compliances and regulations including the GDPR pose several challenges from managing data in an agile environment

COMPILED BY INFOSYS

T E S T M a g a z i n e | M a y 2 01 8

D E V O P S / A G I L E

I N N O V A T I O N S

with high volumes of unstructured data being generated, it creates challenges for enabling mobile automation.” Further, many companies want to link their investments in digital transformation and AI with the outcomes achieved. According to Tom, this represents a shift from a technology-centric and cost-saving mind-set to an outcomebased and value-creation one where testing is of prime importance. It mandates integrating front-end, middle-office and back-end systems. Thus, organisations need a clear AI strategy that highlights how value is created by focusing on data.

THE PROBLEM OF DATA MASKING

Digital transformation is forcing organisations to create omnichannel experiences for product and service delivery

amounts of qualitative data that are broken down to clear action points. Organisations can then choose whether to act on these or not, improving decision-making. The ability to leverage such technologies and capabilities becomes a key success driver for companies. In Shishank’s point of view, this is how technology enables the business. He said: “There are several companies that may be asset-poor but have the right technology, which is what makes them formidable competition for assetheavy and product-rich companies.”

AGILE QA NEEDS A NEW ENTERPRISE MINDSET In the following session, Tom Reuner, Managing Partner, HfS Research spoke about the current situation of AI. He said: “As the number of devices is varied,

T E S T M a g a z i n e | M a y 2 01 8

In the next session, Gary Hollam, Director of Sales Engineering, Delphix highlighted why data masking was important – and challenging – with the GDPR deadline coming up. In a nutshell, his viewpoint is that achieving continuous delivery through continuous automatic testing requires continuous data. But, enabling this poses several challenges owing to data friction arising from large amounts of data, issues of data privacy, need for data masking, etc. He spoke about the Delphix Dynamic Data Platform, which is an API-first tool for companies to enable integrated agile masking where data is virtualised for endusers. The platform uses APIs to automate processes and has AI algorithms to search for and mask sensitive data or even replicate it using selective data distribution. Replicated data is made available through a self-service interface and the copied data mimics the format of the initial database whether Oracle, Seiko, etc. A database version control system allows teams to refresh production data and run transactions, data definitions, etc., through APIs. This ensures the data isn’t tampered with. “Through DevOps, the right testing, production and development environments can be created. Through automation, copies of production data can be continuously delivered to respective teams faster to accelerate the software delivery lifecycle,” Gary said.

DEALING WITH UNSTRUCTURED DATA Gary also fielded questions on storage, security and masking of unstructured data within the Delphix platform. The platform

D E V O P S / A G I L E

securely handles data storage using virtual archive machines either on-premises or in the cloud. APIs help to automate data delivery and integration in a secure manner. Reduction algorithms can be used to handle unstructured data. He alluded to an innovative company in Turkey that is using a new method of tokenization to address the problem of doing credit checks on masked data. Here, a separate tokenization environment is created that uses tokenization algorithms to tokenise values from sensitive data sets. Using a web service, the tokenized data is identified and gathered from a third party that tokenizes it again before transfer. Gary summed it up saying, “The key benefit of products such as these is that it containerises data through data pods making it easier for teams to identify and bookmark data and create subsets. It enhances the agility for clients by enabling continuous data delivery and reducing the lifecycle times.”

EXTREME AUTOMATION FOR QA During the panel discussion following the sessions, attendees shared their viewpoints on the specific challenges they face and the value they see at the convergence of AI/ML, QA and DevOps. The Chief Digital Officer of a UK-based universities admissions service succinctly explained how AI/ML can deliver value for QA using an example of how customers today want to “‘try before they buy”. In fact, many consumer goods and retailers leverage this as a way to increase stickiness. But, the applications can be so much more. He further explained, “AI can be used to provide student applicants with snippets of virtual lectures, campus tours, faculty interaction, etc., across universities so they can make informed choices about where they want to study for the next 4 years.” On the one hand, such insight will reduce the student dropout rate. On the other hand, data generated through such interactions will provide universities with clear information on what attracts students. While strategic value can be driven where social media, AI and data meet, he admitted that non-functional and functional testing is inadequate to address the challenges faced by universities. The Testing Team Manager of a large European telecommunications company said, “Most transformations are geared within (for the enterprise) or outside (for

the customers). QA departments also require transformative approaches.” He shared how the transformation strategy in his company focused on enhancing quality, improving time-to-market and managing cost. They used automation, offshoring, shift-left testing, shift-right testing, etc., which has strengthened systems testing. Today, they are already reaping the benefits through a better user experience, effective cost management, compliance with GDPR, and more. The Director of IT Services in a large container and shipping company confided that despite the benefits of QA, synchronising legacy systems with agile ways of working remains a challenge. They have addressed this using strong change management and AI tools to upgrade technical architecture and applications through digital innovation, he added. While the industry may appear divided on the value of AI tools, all attendees agreed that each company must decide which tools work for their specific goals and business needs. This approach is particularly true for QA departments. They should have the freedom to choose the right tools and technologies to improve efficiency. They also need to start showcasing the value and benefits achieved through QA to get stronger support from top management and stakeholders.

I N N O V A T I O N S 53

While the industry may appear divided on the value of AI tools, all attendees agreed that each company must decide which tools work for their specific goals and business needs

CONCLUSION Digital transformation is forcing organisations to create omnichannel experiences for product and service delivery. Ensuring high-quality digital experiences requires faster releases. This poses challenges for QA teams who must capture user experiences in real-time across the web and mobile platforms and then test and fix defects in evershrinking timelines. Since speed-to-market is a vital requirement, organisations are naturally looking towards automation to accelerate processes. AI and ML can play a vital role in enabling such efficiencies. More importantly, it allows companies to streamline data-related processes and unlock value faster and in a secure manner. Data-driven insights can help companies gauge customer sentiment in realtime, remedy defects and enhance user experiences, thereby generating positive business outcomes. Infosys helps companies by delivering enterprise digital assurance and you can reach out to them at askus@infosys.com

T E S T M a g a z i n e | M a y 2 01 8

THE PHOENIX OF SOFTWARE TESTING When more non-technology leaders start to understand and appreciate the work done by the testing community the phoenix of software testers will rise again

oftware testing is undergoing a hard phase with a constant focus on its valuecreation within the modern software delivery life cycle (SDLC) built on agile and DevOps, and software testers need to spend more time building automated scripts. Software testing changing is an understatement, and unfortunately, isn’t a lucrative career path for software engineers that have recently finished University. This is because the career of a software tester is not clear in the industry, unlike developers

T E S T M a g a z i n e | M a y 2 01 8

who have clearer roadmaps and a lot of role models where programme directors, CIO’ and CTO’s from development backgrounds can help. While software testing is a discipline, it has limited growth provided within organisations and is kept at par with other technology functions, such as application support and project management offices (PMO). Testers now measure quality and developers build quality. So, at what point of the modern software development

D E V O P S / A G I L E

lifecycle does the maturity point come? Is it when developers start to build first time right products? An additional pair of eyes in the form of testers are no longer required prior to production releases, so have we just resigned to the fact that modern software architecture is so complicated and intertwinned with other applications so developers can’t build and check his/her own work comprehensively?

REDUCING DEPENDANCY From a senior technology management perspective, Utopia’s self-fulfilling teams are created (like scrums) and are assigned parts of a project, to build independently before the final product is integrated together and checked against acceptance criteria. If no issues are found, then the product is released to production. The question is, do these self-fulfilling teams need a business analyst, developer and QA (3 amigos model) or can true agile be reached when we do not need to label individuals in a team? Software testing has had a good run for the past 10-15 years, with large TCoE’s (Testing Center of Excellence) and testing consulting organisations bagging big contracts for independent verification and validations. On average, 40% of project budgets are allocated for testing and organisations are spending more and more to train their test professionals on both their application domains, as well as test automation products to speed up testing. Like the elephant in the room, no one wants to accept that software can be built and managed without human concept, or at least for the need to reduce dependency. Another challenge for software testing is: how can we effectively measure the ROI of large testing investment? Reduction of production incidents are often regarded as one of the many tools, but is the reduction due to increase issues found in QA cycle, or to improve development processes for first-time ethos of developers?

DOOM AND GLOOM FOR TESTERS? Defects within a QA cycle is another measure to monitor effectiveness, but again, organisations rushing to embrace enterprisewide agile methodologies and the agile gurus/ coaches are preaching not to raise defects within sprint cycles; this can take away the camaraderie of developers and testers, and build walls between them. Focussing on the

I N N O V A T I O N S

testing automation pyramid and pressure to build more BDD frameworks, QA members are now even less focussed on test execution and are asked to focus on building feature files in Gherkin; only which are then automated by developers in BDD frameworks and executed in the CI/CD pipeline. The role of testers in an agile and DevOps world is often debated, and the argument comes back to testing vs checking. By using automated scripts or bots we can check and argue that we need tests to achieve endto-end testing and exploratory testing. But, is this really true, or just an argument from testing professionals to fight back for its own existence? So, is it all doom and gloom for software testing as a profession? Testing is now truly just an extension of build processes, which are ultimately developers’ accountability, as well as responsibility – they can no longer abdicate this responsibility to another team of testers like they have been doing.

Software testing is undergoing a hard phase with a constant focus on its valuecreation within the modern delivery lifecycle built on agile and DevOps

STEP 1: TESTERS TO TECH RISK ANALYSTS First and foremost, software test professionals in this new world will not be called testers. Instead, they will need to think and act as technology risk analysts who manage technology risks. As soon as the job title changes from being a test analyst to risk analyst, the mindset will shift from checking to testing. STEP 2: TECHNOLOGY ORGANISATION The new age technology risk analyst will no longer be reporting to CIO’s, as technology organisations are heavily led by development and project delivery mindsets. Technology risk analysts will work alongside other risk professionals like operational risk, credit risk, market risk etc., and report to risk and compliance organisations. STEP 3: PERMIT TO DEVELOP New age technology risk analysts will no longer be required to constantly prove their worth or even produce test reports to highlight delivery risks, which often get ignored by programme managers. Instead, they will work alongside technology teams like audit and compliance members, and if they find any risks in production they can be empowered to stop the development process – if need be. As you see from the points above, neither software testing as an engineering discipline or as a profession is diminishing or dying. Instead, it’s the mindset that needs to be changed.

SUDEEP CHATTERJEE PROGRAMME TEST MANAGER BANK OF AMERICA MERRILL LYNCH

Sudeep is a Senior Technology Leader with 20 years’ experience in top tier investment banks, fintech and consulting firms, as well as manages testing globally for enterprise-wide change programmes. Currently, Sudeep is working as Head of QA – FX Horizontal at Bank Of America Merrill Lynch within FICC - Global FX Technology group. Prior to Bank Of America Merrill Lynch, he was the Head of QA at Lombard Risk, Barclays, UBS, GE and Accenture.

T E S T M a g a z i n e | M a y 2 01 8

CLIMBING THE

By adopting the â&#x20AC;&#x2DC;Test Pyramid Modelâ&#x20AC;&#x2122; approach, Senior Test Automation Consultant at Home Office, Gabriel NG, manages to implement a smoother, cheaper and faster test lifecycle T E S T M a g a z i n e | M a y 2 01 8

E M B E D D E D

vast number of development projects and teams are now facing the challenges of having a sacred number of QA resources available, averaging at one tester to five developers per team. As projects are being delivered at a rapid pace, along with changing project requirements, QA resources are becoming more and more valuable. Today, there are still many delivery teams that are far too focused on the strict manual process of test requirements, planning, analysis, test case design, implementation, execution and closure with many manual and approval processes in place. Over time, costs increase and the quality of delivery decreases. Modern test lifecycles should now follow a leaner and rapid approach to accommodate the growth of automation test capabilities, to be embedded into agile and continuous delivery teams. Here are a few insights into implementing a smoother, cheaper and faster test lifecycle:

Credit https://www.testingexcellence.com/agile-test-strategy-example-template/

Credit https://www.symbio.com/solutions/quality-assurance/test-automation/

UI testing. These tests are brittle as any UI enchantments can easily break test packs, therefore, should ideally be automated on building pipelines, allowing maintainability and further exploratory test efforts as iterative builds progress. Through the test pyramid model, teams are able to save cost and time in their test cycle, by increasing test coverage at the lower end of the pyramid, as well as eliminating any bugs exposed as we move upwards into business-centric UI test layers. BDD TEST LIFECYCLE Following on from the evolution of the ‘TDD’ test-driven methodology, the newly born ‘BDD’ behaviour driven development has transformed agile teams and is almost becoming an industry standard for many organisations. The methodology complements well and replaces the traditional way of writing test requirements and the analysis phase of the test cycle where business analysts, project managers, architects, developers, and testers are able to provide direct inputs and structure test acceptance criteria in ‘Gherkin’ language, as well as the ‘given’ ‘when’ ‘then’ format for agile user stories. In essence, having BDD acceptance criteria/test scripts act as a replacement to the gathering test requirements and analysis phase, eliminating any misunderstanding between team members, as well as removes any unnecessary sign-off for approval Credit https://confluence.atlassian.com/adminjiracloud/using-jira-applications-withprocesses which creates an efficient test cycle. confluence-868983272.html Further to this, by leveraging automation

T E S T I N G

The test pyramid follows an approach as a model in modern test lifecycles, while emphasising on finding bugs early, cheap and fast within the test lifecycle

TEST PYRAMID MODEL APPROACH The test pyramid follows an approach as a model in modern test lifecycles, while emphasising on finding bugs early, cheap and fast within the test lifecycle.(fig.3) The biggest chunk is that 70% of tests written should be unit tests, which are primarily written by developers to test their own code and catch bugs early before reaching QA teams. These tests are crucial as they provide a fundamental of test coverage that reduces the number of major bugs found later in the test cycle. When climbing up the pyramid, the next 20% should be allocated to integration tests, which is typically written as systems are integrated with other dependency and third-party systems, where mock services and automated virtualised environments could be utilised to fully fill in coverage where unit tests may be missed. As we reach the top of the pyramid, the last 10% should focus primarily on functional

GABRIEL NG SENIOR TEST AUTOMATION CONSULTANT, UK HOME OFFICE

Gabriel Ng came from a software No credit required, own diagram

test capabilities such as ‘Cucumber’ and ‘JBehave’, the potential of the methodology can be maximised, especially when writing technical test scripts. This ties in greatly with test reporting, where there are BDD reporting engines such as ‘Cucumber HTML Reports’ and ‘Serenity BDD’, which takes the pain away from traditional reporting and test closure as it creates manual test reports towards each test cycle. These reporting engines are user-centric professional test reports of test scenarios, steps, coverage, and execution, which should be shared with teams and stakeholders.

developer background and has eased into being a Test Automation Consultant with more than 6 years’ experience. He began his career in Accenture, then worked across retail, finance, government and other industries. Recently he started his own consultancy, which works with multiple clients on a contract basis to provide bespoke test automation consultancy services.

T E S T M a g a z i n e | M a y 2 01 8

E M B E D D E D

T E S T I N G

When diving into the traditional process of test lifecycles, we can see there are many test artefacts and documents created to support test activities and provide assurance to stakeholders. Some may think moving to a leaner approach may require eliminating test artefacts and documentation – well, not exactly. LIVING DOCUMENTATION

As you can now gather, using BDD Credit https://confluence.atlassian.com/adminjiracloud/using-jira-applications-withmethodology in the test lifecycle greatly confluence-868983272.html improves the communication bridge between

business and technical teams to improve test implementation, execution, and reporting processes through its simple common business language concept.

CONTINUOUS INTEGRATION IN TEST LIFECYCLES

No credit required, own diagram

As development teams integrate into the continuous integration/continuous development space (CI/CD), this has now become the new catalyst to maintaining a productive and efficient test cycle. CI/CD heavily focuses on the implementation of automated build pipelines, compared to a traditional test cycle, where most of the test and build pipelines requires tons of manual processes in place. CI/CD does not only tackle the challenges of QA resources but also frees up the time to enable functional testers to work on other activities such as exploratory testing to deliver and produce bugfree deliverables.

Creating a leaner test cycle does involve reducing unnecessary documentation and approvals, however, rather than removing these practices, we are creating a new breed of ‘living documents’. One where the traditional test documentation approaches of ‘word’ and ‘excel’ tables are impractical, which no one else reads. The goal is to achieve sets of living documentation and artefacts that would utilise toolsets such as JIRA ‘Confluence’ to create dynamic test documents that, perhaps, can be linked to daily scrum backlogs with the latest status and updates from the team; along with intuitive macros and user tagging/ commenting capabilities. Essentially, creating your own ‘Facebook’ page where anyone could contribute. Through the living documentation, it enforces teams and stakeholders to take part throughout the creation and modification of documenting the process in each iterative delivery milestones with the latest reporting status coming directly from the team, which could save hours of unproductive back and forth email communications. Each team will be made up of different resources, goals, and processes. However, by following some of these principals, while leveraging market automation capabilities, there will be significant improvements in the long term, once a leaner test lifecycle has been shaped, transformed and achieved.

Credits

fig.1 https://www.testingexcellence.com/agile-test-

strategy-example-template

fig.2 https://www.symbio.com/solutions/quality-

assurance/test-automation/

fig.3 https://confluence.atlassian.com/adminjiracloud/

using-jira-applications-with-confluence-868983272.

html

Credit https://www.testingexcellence.com/agile-test-strategy-example-template/

T E S T M a g a z i n e | M a y 2 01 8

12-13 June 2018 The Principal York, York

The first ever DevOps Conference North will take place at the luxurious Principal York Hotel on 12th and 13th June 2018. The DevOps Conference North is produced by the same organisers as The National DevOps Conference, which has been running in London since 2016, after it became apparent that a replica event was required in the north of England to cater for those in our northern cities, as well as our friends and colleagues in Ireland and Scotland. The DevOps Conference North is a UK-based conference that provides the IT community with invaluable content from revered industry speakers; practical presentations; roundtable discussion forums that are facilitated and led by key figures; as well as a market leading exhibition, which will enable delegates to view the latest products and services available to them. In 2017, the National DevOps Conference featured speakers from many well-known companies, including NHS Digital, ITV, Tesco and HSBC to name but a few; as well as a wide selection of presentations and a major exhibition, which saw companies such as Sogeti, Contino, Interoute and CA Technologies showcasing their latest products and services. DevOps Conference NORTH has its finger on the pulse as it is owned and supported by the industry leading journal TEST Magazine and is produced by the organisers of The DevOps Industry Awards. No other DevOps conference in the UK or Europe can boast these credentials. It’s here that you get access to the authoritative analyses and ground-breaking research that will enable you to keep one step ahead of market trends.

£525.00

Day 1 Only*

£325.00

Day 2 Only*

£325.00

*Includes entry to the Evening Drinks Reception on the first day of the Conference. All prices are subject to VAT.

 0203 931 5827  info@devopsevent.com  http://north.devopsevent.com T E S T M a g a z i n e | M a y 2 01 8

BUILDING CX ASSURANCE INTO DIGITAL APPLICATIONS Sandeep G.N., Quality Engineering and Transformation Practice Manager at Tata Consultancy Services (TCS), explains why a holistic four-stage, spherical shift-based practice of software assurance, delivered through automated, on-demand testing for compatibility, usability, performance, accessibility and security ensures delightful and engaging digital customer experience

recent trend in the behaviour of digital customers of large enterprises is the rising demand for delightful customer experience (CX). This has made a significant difference in the way key verticals such as banking, insurance, retail and utilities conduct business today. To remain competitive and relevant, therefore, organisations look for innovative ways to continuously engage their customers across various digital channels. For many of those organisations, CX is a core objective of digitisation or digital transformation. But ensuring superior-quality digital CX needs a shift in gears. It calls for engineering quality processes and procedures in the context of digital transformation and adopting the customer centricity mind-set to drive high-velocity, ‘first time right’ quality for delightful CX. Specifically, this means ensuring software assurance has comprehensive coverage and comprises not merely functional testing. This comprehensiveness translates into taking

T E S T M a g a z i n e | M a y 2 01 8

a holistic view of the various dimensions of user experience through compatibility testing, usability testing, performance testing, accessibility testing and security testing because traditional functional testing can only verify and validate the specifications/design of digital channels.

WHY IS HOLISTIC CX ASSURANCE IMPORTANT? To answer the question, let’s take the example of a well-known bank, which recently revamped their personal banking web pages only to find that the number of users dipped to a record low! What had happened? A number of the bank’s users were browsing using Chrome. An incompatible client-side image loader script had slowed response times, and those users had understandably dropped off. Performance testing and compatibility testing could have helped unearth the issue before go-live. Or take another faux pas from real life –

E M B E D D E D

defective airbags in cars, at an automotive major. The airbags were not possible to deploy under crash conditions because of a performance issue in the vehicles’ embedded software. As a result, the automotive manufacturer involved had to recall over four million vehicles! To guarantee seamless and high-quality CX, therefore, the test value chain must cover the five types of non-functional testing named above and, in an age of continuous integration, adopt a spherical shift approach. Only then can CX assurance deliver defect-free results and at a shorter time-to-market. While comprehensiveness in testing would make Omnichannel CX successful, Shift Left would ensure that effective user feedback mechanisms are in place. As Vivek Gupta, Corporate Vice President and Head of Quality Management, COE and Testing Shared Services at New York Life, one of America’s largest mutual life insurance company, put it recently: “A comprehensive digital assurance strategy should ensure zero negative sentiments of end-users. By addressing these aspects in the test value chain, there will be an automatic dip in the number of post-production issues reported on user experience. As a consequence, the cost associated with re-testing defects goes down.”

THE COMPONENTS OF HOLISTIC CX ASSURANCE While page specifications, CSS and aspects like user journeys can be tested for their functionality against a standard set of requirements, non-functional aspects provide insights on factors impacting CX: • COMPATIBILITY: Tests ensure that the application behaves uniformly across all digital mediums. So, for example, a life insurance policy quotation can be started on a tablet and completed via a mobile phone. • USABILITY: Testing addresses quantitative validations, like task times, error rates against industry standards and guidelines on aspects such as human-computer interaction, which addresses navigation, presentation, content and interaction. It also covers a user’s difficulty in accessing a particular link or finding information or a needed message, and visual inconsistency across screens. • ACCESSIBILITY: Testing ensures “inclusive” assurance. It verifies digital assets against the 'World Wide Web Consortium’s Web Content Accessibility Guidelines'. • PERFORMANCE AND SECURITY: With the advent of social media and the exposure to multiple online communication options, developing an engaging digital conversation is a must-have for organisations. This is where performance testing and security testing come in – to ensure information is made available at the right time without compromising customer privacy.



T E S T I N G

A comprehensive digital assurance strategy should ensure zero negative sentiments of end-users. By addressing hese aspects in the test value chain, there will be an automatic dip in the number of post-production issues reported on user experience. As a consequence, the cost associated with re-testing defects goes down



Vivek Gupta, New York Life

IMPLEMENTING CONTINUOUS CX ASSURANCE While testing for compatibility, usability, accessibility, performance and security makes for a comprehensive view of CX, it is important also to ingrain these within the release life cycle to gain crucial insights into quality continuously. The tests should therefore not only be automated but modelled for on-demand usage through provisioning of appropriate environments, test data, automated execution, defect tracking and logging. This will culminate in high-velocity quality. It will also, in the process, help testing teams to understand the impediments for superior CX through the correlation of observations with critical key process indicators (KPIs). For example, examining the combination of high bounce rate and reduced user time spent on a page could indicate a performance and/or browser compatibility issue. Improved online traffic and conversions, reduced exit rates and consistent and seamless experience across multiple channels are some of the many other benefits that can be derived from correlation. To conclude, the following four key stages may be considered, when building cost-effective, high-quality, ‘first time right’ CX assurance: • CX QUALITY EVALUATION: To assess and remediate enterprise applications, correlating results with business KPIs. • PROVISION OF AN ON-DEMAND ECOSYSTEM: Enabling Capex optimisation against devices, browsers and OSs. • QUANTIFICATION OF CX MATURITY: Based on industry best practices, with remediation for improvement. • AN OUTCOME-BASED STRUCTURE: That assumes responsibility for verifying compliance against regulations and enhances CX. Combining these aspects should go a long way in building delightful CX and developing powerful digital brands for organisations. With digital adoption and transformation moving at an unprecedented pace, quality engineering of CX assurance is all the more relevant now, in ensuring delightful and engaging CX.

SANDEEP G.N. QUALITY ENGINEERING AND TRANSFORMATION PRACTICE MANAGER TCS

As Quality Engineering and Transformation Practice Manager at Tata Consultancy Services Ltd (TCS), Sandeep G.N. leads TCS’ quality engineering and transformation practice for insurance engagements in the North America geography. Sandeep’s areas of expertise include design and development of agile practices for quality engineering, automation frameworks, continuous integration/ continuous development and DevOps. With over 12 years’ of experience, he has worked with various insurers in injecting high-velocity quality into their DevOps, digital and cloud journeys.

T E S T M a g a z i n e | M a y 2 01 8

FROM TRADITIONAL QA TO AGILE SMART METRICS TO LEAD THE TRANSFORMATION

Head of QA Office at Thomson Reuters, Hussein Jibaoui, believes that in order to accelerate business growth and delight customers, business models have to be moved to one unique platform-based product

he journey started a few years ago within our large organisation in charge of building a world-class product largely used by professional users of the financial markets. To accelerate our business growth and delight our customers, the business model has moved to having one unique platform-based product. This model aims at reducing costs and improving the time to market by getting fast and short

T E S T M a g a z i n e | M a y 2 01 8

release cycles with much more features. To support this change with the quality assurance (QA) group, many challenges were coming up quickly: e.g. for a largescale system which is evolving fast, how to keep a very high quality product, how to maintain very high performance and stability of the platform, how to remove the testing bottlenecks, how to test as early as possible and quickly catch bugs impacting

our customers, etc. The previous traditional QA model (e.g. independent QA team, Iterative waterfall cycle, main regression was manual, test management, etc.) became outdated and didnâ&#x20AC;&#x2122;t satisfy the new challenges. We decided to change our QA model by getting rid of the central group and putting in place agile approaches. Below are 4 main principles we focused on by leading the

E M B E D D E D

related changes in terms of organisation, methods of work and processes: • THE QUALITY IS FIRST IN LINES OF CODE: We moved the quality ownership to development. • TESTING AND CODING AT THE SAME TIME: We moved testers onto same “Delivery” teams as developers and product owners. The testers sit in the middle of the developers. • REGRESSION TESTS ARE AUTOMATED: Few manual and much more automation testing, we reduced the manual testers and hired more automation engineers and implemented the continuous integration testing. • LEAD THE TRANSFORMATION BY THE METRICS: Use metrics to bring transparency and drive efficiency: Define clear quality goals to share across the delivery organisation and provide the related metrics in an accessible and transparent way. Different challenges appeared, some are related to the method of work and process (e.g. how to ensure the right test coverage without test management, how to leverage the automaton testing, how to put in place the continuous integration and deployment, etc.) and some are organisational (how to drive the developer quality mindset, how to assign quality accountability and keep the transparency, how to manage the transformation of traditional QA roles in an agile world, etc.). But one of the most important and exciting item was with introducing the culture of metrics and leading the transformation by the metrics. This paper gives insights on how we managed to build smart quality metrics. Indeed, talking about the quality metrics in general, not specific to the Software Engineers, it is still tricky, questionable, contentious and even some times a boring topic despite its importance and its impact. This is not because people lack interest or they don’t commit on the objectives behind the metrics, but often the metrics implementation is not so SMART (i.e. no purpose, not actionable, not meaningful, not intuitive, or even counterproductive). How many times we hear such feedback as “building metrics, just for having metrics”, “nice metrics but I don’t need them, or they can’t help me”, etc. Unfortunately these statements might be true, if metrics are not SMART. On top of that, and beyond the

implementation challenges, measuring the Software Quality is one of the most heated and fascinating debates in the software industry. There is no single way to measure an absolute quality for software. Indeed, there are different angles, meanings and drivers to assess the quality. Often, many assumptions and interpretations are made to explain standard metrics. Currently, with the digitalisation era; cloud and open platforms, IoT, etc. it is easier to bring more transparency as the data is recorded, structured and consumed. Indeed, this pushes the software engineers to design their products with more check points, counters, usage tracking, performance monitoring, etc. This is offering new ways to build and consume effective metrics and significantly empowering the quality analytics approach. Below are 5 tips to help us building smart quality metrics: • INTRODUCING THE CULTURE OF METRICS: Accept metrics to measure our performance, take benefits of the digitalisation and consume our own data to bring transparency and improve process, etc. • DEFINING OUR OWN METRICS: The new data sources are a huge opportunity to define and tailor our own metrics beyond the standards. • BUILDING TRUSTWORTHY METRICS: Make metrics easy to explore, to access, to drill down, to get access to the underlying data, etc. For instance, from a defect chart summary report, navigate to the defect and open the details. • VISUALISING SELF-EXPLANATORY METRICS: Make metrics based on data that easy to collect, easy to generate, to visualise, informative, self-explanatory, meaningful for both executives and team members • BUILDING PREDICTIVE METRICS: Use Big Data approaches to build metrics, use Data Mining tools, build risk analysis tools to get predictive metrics.

T E S T I N G

There is no single way to measure an absolute quality for software. Indeed, there are different angles, meanings and drivers to assess the quality

HUSSEIN JIBAOUI, HEAD OF QA OFFICE THOMSON REUTERS

With over 16 years' of experience in software testing and quality assurance, Hussein is passionate about software testing and convinced that testing

In conclusion, “If you can not measure it, you can not improve it”, that is a quote from the physicist Kelvin. The power of metrics in improving and leading transformation was a key. The smart way we did produce metrics, was the main driver to get people to trust and consume them.

is scientific research, which requires continuous improvement and a curious mindset. Prior to joining Thomson Reuters, he worked as a Consultant for ALTRAN CIS covering various industry sectors including insurance, finance, telecoms and pharmaceuticals.

T E S T M a g a z i n e | M a y 2 01 8

FUTURE TESTING CHALLENGES Lead Automation Engineer at Gov, Varun Sharma, reveals how to incorporate data security and system vulnerability as part of the continuous delivery pipeline

T E S T M a g a z i n e | M a y 2 01 8

E M B E D D E D

t can be tricky to test offline, as well as upcoming service worker applications from an end-user journey perspective; especially if you are using non-JVM based test frameworks. Frameworks like Puppeteer seem to be a good alternative, but before making decisions, it is an area to watch for if you only intend to support recent versions of Chrome for your endusers. With the concept of frontend and backend narrowing in the future, there has to be a carefully evaluated decision on which automation test stack to use based on team dynamics i.e. JVM or java-script based frameworks. With the global concern around GDPR regulations, there is a need for incorporating data security and system vulnerability testing as a part of the continuous delivery pipeline; to check for the vulnerability of open source artefacts prior to their use in implementation.

Alternatively, to reduce dependency on environments, we should be able to dynamically auto-generate the test data for testing using modern container technologies, which will be your baseline to create your test infrastructure and even purge it.

START EMBRACING

WHAT TO AVOID

PRACTICES

Avoid using the old world test reporting monolithic products or putting effort into heavy manually prepared excel sheets to show business test execution progress. Instead, adapt to the concept of live documentation by using the open source reporting library i.e. Serenity, which helps you write better structured, more maintainable acceptance criteria; and also produces rich meaningful test reports. Refrain from practices having separate test case copies in multiple places for manual and automated tests. Instead, use the common way of having all tests in a single source of truth as live test cases documentation i.e. BDD format and tagged as appropriate i.e. @manual etc. It is wasted effort writing an automation utility just to analyse the historical test execution stability matrix. Instead, use existing open source custom CI plugins like 'Test Analyser Report', which gives you a historical comparative view of the tests execution status i.e. pass/fail and execution time which in turn will help you to iron out unstable/flaky tests. End focus for any delivery should be on delivering faster. The end-user journey scenarios should sit at top of the test pyramid and should be least. We must have appropriate alignment with the test pyramid with most of the tests at unit test level followed by integration test or tests at an API level.

Start using the Google Lighthouse score in your continuous integration builds and as part of your code reviews, as well as for your user-facing applications; it helps to measure your application scorecard based on a combined calculation of your web page load time, flaky connection, accessibility and SEO audits. Start automatic testing on canary release versions, as no business wishes to be shamed on Twitter feeds by users due to their website homepage not working after a new auto-update of chrome version. For front-end developers, start adding the automatic check on your performance impact due to an addition of an npm package, preferably in CI which will, in turn, force to optimise your npm dependencies. Make sure your project dependencies are saved and stored in company corporate repositories after pilot builds and that you are able to build your project even if GitHub or the hosting site is down for few minutes. Automation testers should start taking part in code reviews for both frontend and backend as an observer, to begin with. There are often complaints about environmental issues. In order to highlight the overall health factors, it is a must to have the Environment Health Check and Heart Beat Monitors’ dashboard.

T E S T I N G

Refrain from practices having separate test case copies in multiple places for manual and automated tests

VARUN SHARMA LEAD AUTOMATION ENGINEER GOV

Varun recently finished as a QA Architect at TicketMaster before joining the government sector as an Automation Lead. After working as a Software Developer for big players like IBM and Ericsson, he thought to explore the world of automation testing. After working in the gaming and entertainment industry, he then learnt how true agile teams have mastered the art of relaxed production delivery.

T E S T M a g a z i n e | M a y 2 01 8

DELIVERING CONSISTENT VALUE Head of Test at Global Stress Index, Riel Carol, proposes a few ideas on how to apply the principles of agile and DevOps to embedded development and testing

T E S T M a g a z i n e | M a y 2 01 8

E M B E D D E D

hen people talk about testing, it generally applies to all types of software testing. However, if you look at a bit more in detail, there are some areas of testing that claim their own space and embedded testing is one of them. It is an area that deserves to be treated as a specific case due to the particularities of the requirements and of the skills needed. With the evolution of the development and testing practices towards agile and DevOps, this column aims to propose a few ideas on how to try to apply the principles of agile and DevOps to embedded development and testing. Embedded testing is the practice of testing the software stack developed to run on a device or hardware for which the software has been designed. For example, you can think of the software running on a sports watch (Fitbit, Garmin), a set-top box, medical equipment, automated production line devices, etc. In order to understand a bit more let’s start by describing the system under test. It is safe to assume that we will have the following: • Some custom hardware in which the software will have to run. Very often, as it is the case in consumer devices, the resources of the hardware will be reduced to the minimum in order to keep the cost as low as possible. There will be a certain amount of future proofing, but in the 3 to 4 years following the launch of the device, it is common for the hardware to be struggling to keep with the demands put on it. • It is likely that there will be some kind of firmware or driver layer that will do the interface with the hardware and a middleware layer to implement some of the functionality of the device. • There will be a user interface to drive the device; to send commands to the hardware, to configure it, or to make it perform the device purpose. • Finally, there will probably some kind of connectivity, either with another device or with some back-end services. This is a very diverse ecosystem and the task of testing the code running in the device is to ensure that the software and by extension the device and its connected services are able to perform their purpose functionally and with quality.

T E S T I N G

CUSTOMER VALUE

It is important to not

Historically, the main device manufacturers have got release cadences of about 2 to 3 years for a new consumer device. They will produce software updates more often, sometimes quite regularly (up to a release every 1 to 2 months) but more likely it is 3 to 4 releases a year at the beginning, and then after a few releases, the devices become obsolete or the cost of developing and testing is too high to carry on the specific device and it is left to stagnate and to become stale. So what can be done to try to ensure a regular and consistent delivery of value to customers? First, the description of the system already provides some part of the answer. It is essential to decompose your system and apply the same rules as normal testing. Test each layer or component as independently as possible. It is good to design the system in a way that it is possible to release the different components independently. The more hard dependencies and coupling in between the different parts, the more difficult it will be to have a predictable release cadence, as a single issue in a part of the system, will stop the full release. It is highly likely that there won’t be any off-the-shelf automation solution to test the full system, as it is very specific to the device. Therefore a good idea is to try to use the tools that are available for the subparts of the system that is standard. For example, is it possible to test the UI layer outside the device using some standard UI automation tools? The other parts of the system can be tested at unit test level and full component level. It then becomes very important to have very well defined interfaces that are thoroughly tested to try to minimise the amount of integration testing. The above steps will ensure that the integration-testing phase is then fulfilling its purpose, which is to ensure that all the components work well together, and not to test functionally some of the components. This can be difficult as some of the functionality might only be exposed and properly testable once the full system is put together. This is where if there isn’t a tool available, it will be required to develop a custom tool. The effort required would be substantial as it is a fully integrated tool that will be domain specific.

leave functional testing

as the last phase of the development and testing cycle. Non-functional testing has got to be performed as often and as thoroughly as functional testing

RIEL CAROL HEAD OF TEST FELIX.COM (GLOBAL STRESS INDEX)

Riel has 20 years’ experience in quality assurance and testing applied to telecoms, TV and internet industries. From an original background in software development he has become a highly technical Senior Manager who has worked in successful start-ups, as well as very well established companies. He also has extensive experience in creating quality assurance and testing strategies, and building teams and quality processes that focus on automation.

T E S T M a g a z i n e | M a y 2 01 8

E M B E D D E D

Again, by having reduced the dependencies between components, it should be possible to adapt the release cadences to the characteristics of each component

T E S T M a g a z i n e | M a y 2 01 8

T E S T I N G

EMBEDDED TESTING One of the complexities of embedded testing is that the system is heavily optimised, meaning that they operate very little spare capacity and changes can easily have unexpected consequences on other parts of the system as the execution timings are slightly modified. A second complexity is that it is difficult to debug the system by running some instrumented code or attaching a debugger will also change the way the software behaves. The final challenge is that some defects can be very difficult to reproduce and are given in some very special conditions that are not covered by testing. In the case of a consumer device, the space to cover is virtually infinite, so having the ability to gather system information from devices that are out in the field with customers will help a lot in understanding the configuration and environments required for testing. The principles above do not only apply to functional testing but also to non-functional testing, in this case, performance, load, volume, security, energy consumption, stress, reliability and any other non-functional characteristic. It is important to not leave functional testing as the last phase of the development and testing cycle. Nonfunctional testing has got to be performed as often and as thoroughly as functional testing. Great care has to be put in tracking the performance indicators on a build per build or throughout the intermediary releases, to try to correct the cumulative degradation of performance gradually introduced with new code and features. Fixing non-functional issues at the end of the development cycle can be very expensive and add an unacceptable delay to the release schedule. The overall aim has got to test each component as much as possible and as independently as possible. This is not just a matter of test execution, but it is also a matter of design and implementation of the functionality. The flexibility provided by the reduced interdependence will be a major step forward in being able to release regularly and predictably. The last aspect to consider is the overall cadence and delivery cycles for the embedded system. Again, by having reduced the dependencies between components, it should be possible to adapt the release cadences to the characteristics

of each component. For example, there is absolutely no reason not to go full dev ops for the back-end services with micro releases happening transparently.

DELIVERY CONSISTENCY Depending on how much decoupling there is inside the different layers of the device, it should be possible to release components independently and frequently. If there is little separation then it is better to try to establish parallel delivery cadences, letâ&#x20AC;&#x2122;s say a fast cadence that could be weekly and a slower one that could be monthly and a very slow one, quarterly and then try to fit the different functional changes according to their complexity and dependencies in the different parallel delivery slots. The more coupled is the system the less agility and flexibility of delivery will be possible. This will be a mirror of the development and testing processes, if the system and processes are not designed for fast turnaround from design to feature tested, integrated and available for release to customers, it will not be possible to engineer it once the code is developed. Evidently, the fundamental basis underpinning this strategy is test automation. It is essential in order to have sufficient test coverage (and this is a very subjective measure), fast and reliable execution. Yet this will not be enough without some good manual testing to ensure that the good test cases are specified and automated and especially to ensure some good exploratory testing. As we have described, embedded testing present some specific challenges. So as to be able to reduce the resulting risks, a good system design that reduces dependencies between components, a good test strategy based on testing components independently as much as possible, a strong automation strategy (for functional and non-functional testing), and a multi-speed delivery cadence adapted to the complexity of the deliverables, will go a long way towards achieving regular and predictable releases of functionality to customers.

NEWS | VIDEOS | PRODUCT NEWS | WHITE PAPERS

Industry Leading Portal

DevOps Awards

DevOps Online is the premium online destination for news, reports, whitepapers, research and more relating to the DevOps movement. Covering all aspects of IT transformation you can be sure that DevOps Online will keep you informed and up to date.

www.devopsonline.co.uk T E S T M a g a z i n e | M a y 2 01 8

2018 #NationalTestConf

EVENT HIGHLIGHTS

www.SoftwareTestingConference.com T E S T M a g a z i n e | M a y 2 01 8

The National Software Testing Conference

#NationalTestConf

Welcome to The National Software Testing Conference

elcome to The National Software Testing Conference! Yet again, it’s wonderful to be here alongside an array of software testing assets as this conference returns to the heart of London for the fifth year running. With this event being held at the spectacular Millennium Gloucester Hotel, we are sure to have an enjoyable couple of days which will give you the opportunity to listen to inspiring speakers and attend useful workshops. Bringing together the software testing and QA community, you will find yourself surrounded by professionals who recognise the importance of quality within the software development lifecycle when debating at roundtables, watching presentations from inspirational speakers, and networking with clients and potential contacts – ensuring you return to work with solutions and practical advice to share with your fellow colleagues. As we all know, everyone has their own unique ways of doing things, which is why we are supplying you with a diverse group of speakers from different business backgrounds, who, of course, have the know-how to test sufficiently at the highest of standards. The first conference day will embrace skilled speakers from the likes of Reed Business Information, Elsevier, Wyndham Vacation Rentals UK, Guardian News & Media, Collibra, DVLA and Credit Suisse. On the second day, you will have the opportunity to listen to testers from TP ICAP, Third Bridge, Traffic Parrot, Jaguar Land Rover, Badoo, EvilTester. com and Société Générale Corporate & Investment Banking.

Giving you the chance to share questions, concerns and comments, our first workshop will be held by Ian Molyneaux, Senior Consultant at Infuse, who will touch upon “scoping performance testing requirements in response to changing application design choice and emerging technologies”, followed by Sogeti and Neotys on the second day. If you are here to source new products and meet potential clients, check out our exhibition and networking area where you can find out new information and build up those all-important contacts. More informally, there will be a drinks reception at the end of day 1 which will give you the chance to network with a variety of people from a range of different sectors, while enjoying a tipple or two. I thank you all for attending The National Software Testing Conference and hope you enjoy the next couple of days as much as we, at 31 Media, enjoy hosting it. Last but not least, I would like to thank all sponsors, speakers and workshop hosts for their on-going support and for helping us put together this wonderful, educational event – without them, this event would be non-existent! Don't forget to share your feedback and thoughts on Twitter via #NationalTestConf and with the 31 Media team who will guide you throughout the course of the day and answer any questions.

LEAH ALGER JOURNALIST

Follow National Software Testing Conference 2018 #NationalTestConf  softwaretestingconference.com  @testmagazine  @softwaretestingnews  TEST Magazine  @testmaguk

T E S T M a g a z i n e | M a y 2 01 8

#NationalTestConf