NOVEMBER 2017
TACTICAL TESTING TOOLS
THE EUROPEAN SOFTWARE TESTING AWARDS SPECIAL
ww.ranorex.
All-in-One
Test Automation
RECORDING_1
RECORD MODULES
0}.", new RecordItemIndex(1));
d", "Key sequence 'admin'.", new RecordItemIndex(2));
Any Technology
BROWSER
OPEN
MOUSE
CLICK
KEY
SEQUENCE
VALIDATE
ATTRIBUTE EQUAL
USER CODE
MY_METHOD
Recording_1.cs RECORDING_1
void ITestModule.Run { Report.Log(ReportLev "Website", "Opening w
Report.Log(ReportLev "Mouse", "Mouse Left C
Report.Log(ReportLev
Seamless Integration Broad Acceptance Robust Automation
1
Quick ROI
Licen
All Te
se
chno logi All U pdat es. es.
egration t In m iu n le e S Now with • Access Selenium with the Ranorex automation framework • Address Selenium pain points • Web testing across all major platforms and browsers • For testers and developers T E S T M a g a z i n e | N o v e m b e r 2 01 7
www.ranorex.com/try-now
n()
1
C O N T E N T S
T E S T C O V E R
M A G A Z I N E S T O R Y:
|
I N T E R N A L
N O V E M B E R T E S T I N G
2 0 1 7
S T R U C T U R E S
NEWS
Blockchain To ‘Reshape’ Digital Businesses ..... 5
38
Appleby Hack ................................................... 7 IoT: The Treasure Trove For Hackers ................ 8 Quality Assurance In Digital Banking ............... 9 THOUGHT LEADERSHIP
Standard Life-TCS Partnership ....................... 10
vel.Info, web site 'http://www.ranorex.
Solve Development Challenges ..................... 15
vel.Info, Click at {X=10,Y=20}.", new RecordItemIndex(1));
The World Around Us Is Getting Smarter ..... 18
vel.Info, "Keyboard", "Key sequence 'admin'.", new RecordItemIndex(2));
Manual Testing vs Automation ...................... 20 Avoiding The Worst IoT Scenarios ................. 22 Choosing Testing Tools ................................... 26 AGILE TESTING
How To Conduct Transformative Agile Testing In A Risk-Averse Industry ............................... 28 TEST MANAGEMENT
30
A Brave New World ....................................... 30 ORGANISATIONS AND STANDARDS
Creditable Of Cultural Transformation .......... 34 TESTING TOOLS
Internal Testing Structures ............................... 38 The Key To Low-Code and Innovation .............. 40
A BRAVE NEW WORLD
46
INNOVATIONS IN SOFTWARE TESTING
Reflecting On A Transforming Industry .......... 42 THE EUROPEAN SOFTWARE TESTING AWARDS
Finalists .......................................................... 46 Judges ............................................................ 51
THE WORLD AROUND US IS GETTING
18
SMARTER
T E S T M a g a z i n e | N o v e m b e r 2 01 7
2
You can have your cake AND eat it. REALLY? Really.
GTC LiteÂŽ Enterprise Testing Solution, Tailormade for SMMEs. DVT's Global Testing Centre offers heavyweight testing solutions, backed up by 17 years' experience and high level expertise. The DVT GTC LiteÂŽ Package is designed for small - medium - micro size enterprises, providing an affordable, quality outsourced testing solution.
An exceptionally sweet deal. Analytical and technical testing skills.
Lean software testing.
Delivery managed by a single point of contact.
Business driven test management approach.
Support from Test Consultants.
Basic to executive level reporting.
Tool selection approved by SQA Architects.
Software testing on real devices.
Dedicated engagement/account management.
Testing aimed at digital transformation.
Short notice for activation.
Automation of regression testing.
Scrum/Kanban used for script development projects.
Performance testing with top tier tools.
INTERESTED? CALL US TODAY Tel: 0203 696 2440 | Email gtclite@dvt.co.uk Terms and conditions apply. T E S T M a g a z i n e | N o v e m b e r 2 01 7
E D I T O R ' S
C O M M E N T
HUNTING DOWN ISSUES, WHILE REPLACING THE HUMAN ELEMENT
A
ssisting the most adequate tests possible, software testing tools help to defeat repetitive, vulnerable operations – replacing the human element – by searching, automating and managing testing activities; accommodating software and test development organisations to hunt down issues in a product before a customer. For further assistance in the big wide world of testing, this issue of TEST Magazine primarily focuses on testing tools, agile testing, test management, lowcode applications, and organisations and standards. Firstly, after some astonishing industry news, Standard Life and Tata Consultancy Services embark on a partnership-based engagement (page 10), along with the transformation of DevOps, digital services and delivery assurance. Similarly, Exabeam explain how a partnering approach can help solve development challenges. (page 14) Nir Polak, Co-Founder and CEO of Exabeam, examines issues of developing new software in today’s fast-paced business environment and discusses how utilising design partners can be an effective way to limit research and development headaches. Senior Manager for Enterprise Software Quality Assurance at Nedbank, Johan Steyn, focuses on software development and quality, but differently shares the first chapter of his new book The Business of Software Testing (page 30). “Will we find you swimming or sinking as the DevOps Tsunami hits?” Taking on a goody bag of internal testing structures, Veracode’s Consultant Solution Architect, Colin Domoney,
3
LEAH ALGER JOURNALIST
discusses different ways to enhance and improve security testing (page 38). But “what happens to the millions of increasingly vulnerable devices that could remain online for decades?”External Relations Officer at RIPE NCC, Marco Hogewoning, reveals ways to avoid the worst IoT scenarios (page 22). Furthermore, Head of Testing at Scott Logic, Laurence Pisani, touches upon faster product releases and better customer satisfaction (page 28), because “testing with agility is not only suitable for the finance industry, it's critical”. After Barclays won the Overall DevOps Industry Award for Cultural Innovation (page 34), it was essential to get some tips regarding a good work culture from Nick Funnell, Barclays GTIS Vice-President. And with 2018 around the corner, I felt it was important to reflect upon the year by catching up with a variety of assets in the software testing field, who spoke about their careers and their future ambitions within the industry (page 42). Last but not least, I am TEST Magazine’s Journalist, Leah Alger, who supplies you daily with industry trends and news, online and on paper. Please email me: What you like about the magazine, what you would like to see more of, what you want to see less of, what topics are covered well and any additional comments.
NOVEMBER 2017 | VOLUME 9 | ISSUE 5 © 2017 31 Media Limited. All rights reserved. TEST Magazine is edited, designed, and published by 31 Media Limited. No part of TEST Magazine may be reproduced, transmitted, stored electronically, distributed, or copied, in whole or part without the prior written consent of the publisher. A reprint service is available. Opinions expressed in this journal do not necessarily reflect those of the editor of TEST Magazine or its publisher, 31 Media Limited. ISSN 2040‑01‑60 EDITORIAL DEPARTMENT editor@31media.co.uk +44 (0)203 056 4599 JOURNALIST Leah Alger leah.alger@31media.co.uk +44 (0)203 668 6948 ADVERTISING ENQUIRIES Shivanni Sohal shivanni.sohal@31media.co.uk +44 (0)203 668 6945 PRODUCTION & DESIGN Ivan Boyanov ivan.boyanov@31media.co.uk 31 Media Ltd, 41‑42 Daisy Business Park 19‑35 Sylvan Grove London, SE15 1PD +44 (0)870 863 6930 info@31media.co.uk www.testingmagazine.com PRINTED BY Pensord, Tram Road, Pontllanfraith, Blackwood, NP12 2YA
softwaretestingnews @testmagazine TEST Magazine Group
T E S T M a g a z i n e | N o v e m b e r 2 01 7
USTGlobal
®
4
Quality Engineering as a Business Advantage 17
3000+
Years of Quality Engineering Experience
50+
Career Testers
Global 1000 Customers
24/7
10+
25+
Dedicated Test Labs
24/7
Test Accelerators
24/7 Coverage
WHAT WE OFFER: Test Consulting
Test Automation
Enterprise Security Testing
Performance Engineering
Test Environment Management
Agile & DevOps Test Enablement Test Data Management
Digital Assurance
Product Assurance
KEY DIFFERENTIATORS: Zero impact transition framework to minimize business disruption during transition End to End & Scriptless Automation Platforms for rapid implementation of test automation Self-Service data provisioning frameworks to accelerate identification and provisioning of test data Intelligent & Continuous Monitoring of quality indicators for proactive remediation Automated user experience validation using Artificial Intelligence to improve consumer experience
India Testing Awards 2017
European Testing Awards 2017
UST Global’s NoSkript Test Automation
Finalists in the leading testing vendor and
Tool was the Finalist in the Testing Tool Category
Non-functional testing category for the services provided for 2 Large UK Customers
QA Financials – European Software & App Awards 2017
National Quality Excellence Awards 2014
Winner of the Test Transformation Project
Recognition of successfully setting up a TCoE in
for the Contributions towards Agile test transformation
North America for a leading Health insurance Provider
services provided for a Large UK based Investment Company
T E S T M a g a z i n e | N o v e m b e r 2 01 7
For more details, contact us at Assurance@ust-global.com
5
N E W S
BLOCKCHAIN TO ‘RESHAPE’ DIGITAL BUSINESSES TEST Magazine Journalist, Leah Alger, studies blockchain and its tremendous potential to disrupt and transform the world of money, business and society
B
lockchain used with robotics, machine learning, artificial intelligence and virtual reality will “reshape” digital businesses with “disruptive” outcomes by 2018, according to a report by Dimension Data. The report found companies that have not begun the digital investment cycle are at high risk of being disrupted, with the most popular trend coming this year being the adoption of blockchain, and its potential to disrupt and transform the world of money, business and society, using a variety of applications. Ettienne Reinecke, CTO at Dimension Data, said in a press release: “Last year when we looked at the top digital business trends for 2017, we predicted that centralised transaction models would come under attack. We were spot on. "In the financial services sector, we have seen the United States and European capital markets moving onto blockchain platforms, and similar activity in markets such as Japan. Considering how conservative and compliance-focused this
sector is, that’s quite remarkable. “It’s ironic that the cyber criminals who perpetrated the recent WannaCry ransomware attack could hold a federal government to ransom and demand to be paid in bitcoin. "Bitcoin might be a cryptocurrency, but it’s based on blockchain, and if cyber criminals are confident that bitcoin provides a safe mechanism for the payment of ransoms, it indicates just how secure the distributed ledger approach is." Companies that will accept bitcoin by 2019 are Airbnb, Amazon, Amtrak, AT&T, Costco Wholesale, Tesla, Google, Goldman Sachs, Greyhound Lines, FedEx, Lyft, PayPal, Hyperloop One, Starbucks, Burger King Corporation, McDonald’s, T-Mobile, Uber, USPS Office of Inspector General, Whole Foods Market, Waymo and Walmart, according to the CEO of Bitcoin Inc, Morgan Rockwell. The entire Blockchain market has reached over US$100billion, and CoinDesk Bitcoin has accomplished its new all-time high at US$6,306.
The price of Bitcoin has grown by more than 500% this year, climbing from just below US$1,000 on 1 January 2017, to its new price at the end of October 2017. Despite this, The State Bank of Vietman has issued “information related to the use of virtual currency” that bans paying with cryptocurrency, according to The Register. The State Bank of Vietnam said in a statement: “Bitcoin virtual currency and other similar is not lawful means of payment in Vietnam; The issuance, supply, use of bitcoin and other similar virtual currency as a means of payment is prohibited in Vietnam.” Vietnamese citizens who decide to use bitcoin or other cryptocurrencies are at risk of a US$9,000 fine. Rieneckle also noted that she believes "Blockchain has the potential to totally reengineer cyber security, but the industry has yet to come to terms with it."
T E S T M a g a z i n e | N o v e m b e r 2 01 7
6
t s e T t r Sta w o N n o i Automat
in
! s d n o c e S 0 1 Testinium minimizes the time you need for testing with
SIMPLE, EFFECTIVE and SCALABLE solutions Selenium & Appium based Real device testing Cross-browser testing BDD support No need to configure Java, Dotnet or Selenium No need to install IDE
To learn more about Testinium’s other superior features, go to:
www.testinium.com T E S T M a g a z i n e | N o v e m b e r 2 01 7
7
N E W S
APPLEBY HACK LEAKS ‘SUPER-RICH’ CLIENTS PRIVATE INFORMATION ‘Super-rich’ clients of Bermuda-based law firm Appleby were recently warned they might be implicated in a “data security incident” which occurred last year; leaking 11.5 million documents of highly sensitive information
W
ealthy tax avoiders who are members of the world’s fourth biggest offshore law firm, Appleby, have been instructing public relation firms and lawyers to ensure their reputations are not damaged regarding a data leak which affected 11.5 million documents, according to the Daily Mail. Appleby said in a statement: “We are an offshore law firm who advises clients on legitimate and lawful ways to conduct their business. “We do not tolerate illegal behaviour. It is true that we are not infallible.
authorities. “We are committed to protecting our clients data and we have reviewed our cyber security and data access arrangements following a data security incident last year, which involved some of our data being compromised. “These arrangements were reviewed and tested by a leading IT forensics team and we are confident that our data integrity is secure.” Among the British politicians who are members of Appleby, Lord Ashcroft, Baroness Pamela Sharples and Ex-Tory MP, Michael Mates, were recognised.
DATA ACCESS
‘LEAKED LOANS WORTH
ARANGEMENTS
£2BILLION’
“Where we find that mistakes have happened, we act quickly to put things right and we make the necessary notifications to the relevant
The files also referred to secret offshore companies linked to Egypt’s Ex-President, Hosni Mubarak; Libya’s Ex-Leader, Muammar
Gaddafi; and Syria’s President, Bashar al-Assad; showing deals and loans worth up to £2billion. Ilia Kolochenko, CEO of web security company, High-Tech Bridge, added: “Currently available facts give a clear indicator that the law firm may have been breached. Many of the allegedly compromised documents are extremely sensitive and normally should exist only on paper. “If the law firm digitalised them without the highest degree of care – it may be found liable for negligence and have to compensate the victims of the breach. On their side, the law firm can try to implead the IT company who installed the digital system, especially if they had an indemnity clause in their contract. “However, I would abstain from blaming any party before the investigation is duly finished." "We cannot exclude a human factor and insider activity – a risk that virtually any large law firm cannot entirely mitigate today.”
T E S T M a g a z i n e | N o v e m b e r 2 01 7
8
N E W S
IOT: THE TREASURE TROVE FOR HACKERS Gemalto reveals that 90% of consumers lack confidence in the security of Internet of Things (IoT) devices, showing that education regarding IoT security is needed among the majority of consumers and businesses
A
ccording to a survey conducted by the international security company, consumers’ main fear is hackers taking control of their IoT devices. Despite 54% of respondents owning an IoT device, only 14% of survey respondents believe they are knowledgeable when it comes to knowing about the security of their tech devices. In terms of the level of investment in security, the survey found IoT device manufacturers and service providers only spend 11% of their total IoT budget on securing their devices. Organisations reported encryption as their main method of securing IoT assets (67%), with 62% of survey respondents admitting they encrypt their data as soon as it reaches their IoT device. 92% of companies also said they see an increase in sales or product usage after implementing IoT security measures. ‘IOT WON’T SEE MAINSTREAM ADOPTION’ Jason Hart, CTO for Data Protection at Gemalto,
T E S T M a g a z i n e | N o v e m b e r 2 01 7
said: “It’s clear that both consumers and businesses have serious concerns around IoT security and little confidence that IoT service providers and device manufacturers will be able to protect IoT devices. “With legislation like the General Data Protection Regulation (GDPR) showing that governments are beginning to recognise the threats and long-lasting damage cyber attacks can have on everyday lives, they now need to step up when it comes to IoT security. Until there is confidence in IoT amongst businesses and consumers, it won’t see mainstream adoption.” According to the survey, 61% of businesses are in favor of regulations, making it clear who is responsible for securing IoT devices and data at each stage of its journey. In fact, almost every organisation (96%) and consumer (90%) is looking for government-enforced IoT security regulations.
GOVERNMENT-ENFORCED IOT REGULATIONS Hart continued: “The lack of knowledge among both the business and consumer world is quite worrying and it’s leading to gaps in the IoT ecosystem that hackers will exploit. “Within this ecosystem, there are four groups involved – consumers, manufacturers, cloud service providers and third parties – all of which have a responsibility to protect data. “Security by design is the most effective approach to mitigate against a breach. IoT devices are a portal to the wider network and failing to protect them is like leaving your door wide open for hackers to walk in. “Until both sides increase their knowledge of how to protect themselves and adopt industry standard approaches, IoT will continue to be a treasure trove of opportunity for hackers.” The report concludes that businesses are realising they need support in understanding IoT technology, so are turning to partners for help.
9
N E W S
THE NEED FOR QUALITY ASSURANCE IN DIGITAL BANKING Bank account holders embrace new banking technologies, favouring it over traditional methods, despite not having patience if a technical issue occurs
A
ccording to a report launched by SQS, 86% of UK consumers trust their banking providers with their personal information and to manage their money “effectively”. Despite this, the report found that consumers are quick to look for another provider if a technical issue occurs. Out of the survey respondents, 62% of account holders said that if their bank suffered a data breach their trust would be broken, with 55% admitting they would consider becoming a victim of fraud grounds for loss of trust. Those who would lose faith in their provider if a website or mobile app stopped working properly accounted for 37% of respondents.
BANKING TECHNOLOGY CONVINEINCE However, 95% of those who now bank online agree that it makes banking quicker, valuing the convenience banking technology brings. The report also found 18–24 year olds are more forgiving as a generation, with 55% of respondents saying they would lose trust in their bank as a result of a data breach, compared to 71% of 65-74 year olds. Dik Vos, CEO of SQS, said: “With technology a key differentiator among providers, banks face a challenge to drive market share from mature technologies whilst exploring and implementing new service options effectively.
“As the digital banking ecosystem continues to evolve, the need for digital quality assurance becomes a top priority to keep customer trust and generate positive feeling.” The report concludes that customers are embracing banking technology and favouring it over traditional methods, with only 2% of customers saying they would check their balance by telephone.
T E S T M a g a z i n e | N o v e m b e r 2 01 7
10
STANDARD LIFE-TCS PARTNERSHIP AN EPITOME OF AGILE AND DEVOPS TRANSFORMATION
Edinburgh-based global pensions, savings and investment firm, Standard Life and Tata Consultancy Services (TCS), embarked on a partnership-based engagement at the start of 2016. Both organisations have worked collaboratively, leading change to weave transformative quality assurance (QA) into Standard Life’s fabric
T E S T M a g a z i n e | N o v e m b e r 2 01 7
T H O U G H T
11
L E A D E R S H I P
A
long with the transformation in DevOps, digital services and delivery assurance, the partnership has seeded trust, respect and cultural inclusiveness. This was made possible by embracing change, deep professional commitment, a strong performance focus, management support and a spirit of innovation. Standard Life’s Workplace Benefits Platform (WBP), a web-based B2B corporate management service, is one of the programmes benefitting from this transformative QA service.
THE CHALLENGES OF GROWTH Rapid expansion of the WBP customer base necessitated frequent and swift customisation of numerous WBP applications. Implementing code changes and providing good Customer Experience (CX) demanded quick turnarounds and technical expertise. Development and testing teams were stretched and keeping pace with the WBP customer requirements became difficult to manage. Customers were experiencing application problems resulting in poor CX. Standard Life and TCS identified that the WBP team (consisting of 60 Development and Test operations personnel) required a robust QA strategy. This would be achieved by an organisational transformation to distributed agile and implementing DevOps. Three major objectives were identified: • Improve time-to-market to acquire new customers faster • Improve quality, eliminate production issues and improve CX • Achieve cost efficiency, reduce IT costs and improve profitability To meet these objectives, Standard Life wanted to build competency in next-gen Quality Assurance (QA) practices, improve its Agile skill map, enhance WBP performance, improve non-functional testing (NFT), ensure uniform Cx across the WBP journey and adopt bespoke automation practices. Standard Life and TCS decided to target four areas: • Transform to an agile/DevOps culture • Reskill the WBP teams to incorporate mature QA practices • Introduce automation to swiftly and efficiently scale up testing • Boost SLA compliance to the 80%-plus band even at peak system loads
THE STANDARD LIFE-TCS PARTNERSHIP: A UNIQUE MODEL Both parties implemented a strategic partnership model that involved: • Closely collaborating with Standard Life during the initial period, improving automation, agile, QA practices, ensuring continuous integration/continuous delivery (CI/CD), security, digital, NFT services, WBP’s CX and user experience (UX) • Leading by example the transformation, in the process embedding organisational change management in Standard Life • Judiciously choosing industry-standard tools, in-house accelerators and related processes and deploying them at the right time in the programme
DAVID BAYLEY HEAD OF DELIVERY INTEGRATION STANDARD LIFE
Responsible for implementing delivery assurance, improving agility and innovation across the organisation. David has pioneered tech and cultural innovations which have contributed towards the service’s success.
The duo worked as one team, embedding a range of software development and QA tools, processes, methodologies, practices and skills in Standard Life’s organisation, adopting a comprehensive QA-driven approach.
PREPARING FOR SUCCESS Standard Life and TCS formed an apex testing management team consisting of TCS’s consultants (QA experts from various specialties) who championed initiatives within the programme, TCS quality engineers, embedded within the WBP team and Standard Life SMEs. While TCS came with expertise in driving transformation, market/industry insights and account management, Standard Life supplemented this by providing strong organisational knowledge, experience and management support. This made a formidable team. A five-pronged QA transformation strategy was adopted: Transforming to Agile - TCS and Standard Life customised the agile framework to build a lean, highly skilled distributed agile team that achieved: • Tighter regression testing windows (reduced from 3 weeks to 3 days) • Embedding Behaviour Driven Development (BDD) • Close tester-developer-business collaboration, impromptu interactions for greater clarity on requirements and bug prevention from the requirements phase
RYAN MCMANUS SOURCING INTEGRATION MANAGER STANDARD LIFE
Responsible for managing strategic partner relationships and developing the partnership principles.
SAIKIRAN NAZRE RELATIONSHIP MANAGER TATA CONSULTANCY SERVICES (TCS)
Sai works with Standard Life to transform service delivery into an innovative and highly agile unit. He is a DevOps practitioner and a digital enthusiast.
T E S T M a g a z i n e | N o v e m b e r 2 01 7
12
Standard Life and TCS formed an apex testing management team consisting of TCS’s consultants (QA experts from various specialties) who championed initiatives within the programme
T H O U G H T
• Maximum team utilisation through task rotation, best practice propagation and optimal onshore-offshore collaboration • Real-time, cross-location onshore-offshore defect triaging and ownership • Advanced skill-building across the board • Continuous assessment and planned process and practice improvements Inducting T-shaped people - Experienced quality engineers were inducted. This improved the team’s skills and capabilities, drove transformation activities, introduced new services and delivered more releases with a 10% smaller team. Introducing a new tech paradigm - The duo selected and introduced the following tools, processes and practices, with inputs from TCS’s Centres of Excellence (CoE) and consultants: • Innovative processes - A bespoke, comprehensive BDD automation framework, to bring the business, development and testing teams together to drive efficiencies - Performance scrums, to introduce NFT in sprints to achieve high quality the first time around • Time-tested tools - Dynatrace implementation within the WBP landscape, to monitor performance, improve CX and weed out performance inconsistencies - TCS Omni and TCS iAccess, tools for browser compatibility testing and accessibility analysis and testing - Fortify, to enhance application security • Proven techniques - Performance assessment and engineering, to improve production performance - CI/CD and automated deployment, to enable DevOps - CX and UX practices, to improve core application design, raise business effectiveness
T E S T M a g a z i n e | N o v e m b e r 2 01 7
L E A D E R S H I P
and reduce effort Strengthening team ethos. Team-building activities were organised. Standard Life managers visited TCS’s offshore locations. A Standard Life training academy was set up for team capability enhancement. Instituting continuous improvement. Both organisations measured progress and set targets for continuous improvement, inculcating an incremental transformation culture. TCS iAccess, TCS Omni and TCS QuLoc and other assets helped build high-quality PoCs, feasibility analyses and implementation that enabled Standard Life achieve targets faster.
ALL PLANNED OUTCOMES ACHIEVED This relationship has championed the cause of comprehensive, transformative QA into Standard Life’s landscape, institutionalising a QA paragon. The outcomes have been inspiring: • 125% increase in release cycles frequency (cycle reduced from 9 weeks to 4 weeks) • 35% improvement in customer experience SLA • 200% increase in new customer hosting capability • 35% increase in automation with integrated tooling • 60% more releases delivered with 20% reduction in team size • Cost savings achieved with a sharper focus on innovation Beyond WBP and during 12 months, TCS and Standard Life have delivered and transformed major programmes in legacy modernisation, digital technology and compliance. To date, 500-plus Standard Life staff have enjoyed and benefitted from customised training in Agile, automation, security, digital and cultural alignment. Standard Life now has a comprehensive QA team, which consistently delivers value, drives transformation and is an epitome of leadership and relationship building. “Every Promise Delivered” encapsulates the partnership.
13
27 February 2018 th
Park Inn by Radisson London Heathrow
2FOR1 OFFER! ÂŁ349 (Non-vendor rate)
Delegates David Leakey david.leakey@31media.co.uk +44 (0)203 668 6946 Sponsorship Kadi Diallo kadi.diallo@31media.co.uk +44 (0)203 668 6942
Don’t miss the networking and enlightening event of the year 1 day event Convenient London location Over 100 senior delegates 12 thought leading debates All lunch & refreshments Networking and knowledge sharing www.TestFocusGroups.com T E S T M a g a z i n e | N o v e m b e r 2 01 7
14
HOW A PARTNERING APPROACH CAN HELP SOLVE DEVELOPMENT CHALLENGES
Nir Polak, Co-Founder and CEO at Exabeam, examines the challenges of developing new software in today’s fast-paced business environment and discusses how utilising design partners can be an effective way to limit research and development headaches T E S T M a g a z i n e | N o v e m b e r 2 01 7
T H O U G H T
15
L E A D E R S H I P
W
e live in a world where software underpins innovation in almost every industry, from retail, to finance, to IT. And yet, software development itself still faces well-known challenges. Today, you’d be hard-pressed to find an IT professional who doesn’t agree that new software is often hard to deploy and hard to use. We hear all too often of cases where it has taken up to a year to get software rolled out and then months to get users fully up and running. These problems are common across the software development sphere, but are exacerbated in certain sectors. One such problem that immediately comes to mind is cyber security product development. Typically, security products are built and tested in a lab. This is a controlled space without any external factors that could adversely impact development. But this also means that the software is not subjected to the “real-world” factors that it will face as soon as it goes live at a customer. The result? The software is installed at a customer site and faces hurdles that it had not previously encountered and there are countless issues with the practicality of its design. Most of these issues stem from developers making assumptions in the lab that do not translate over to the real-world use cases and customer environments. For example, a developer may work on a brand-new tool to manage the installation of a software agent on 70,000 company computers. In theory, the new tool will help to save the company’s IT staff the time and effort. But what if the company, with 70,000 installed agents, has to work with regular install and update failures? Or, what if the conditions that allowed the product to work well in the lab suddenly make it ineffective and prone to errors once it’s used on customers’ dirty data? Something that appears to be relatively straightforward in the lab setting can often become a headache in reality. Unfortunately, the pace at which business runs today is not helping the software development situation. We end up with the following scenario: a developer has a brilliant idea for a new piece of technology that can make millions of pounds. Business leaders want to capitalise on the competitive advantage of this new, moneymaking technology, and ask developers to go off and build it with the help of an internal
team. As soon as it is developed, the company markets and sells the tech to customers, where it falls at the first hurdle. Thankfully, the business world is starting to change. According to Trustwave’s 2017 report, 65 per cent of respondents feel pressure from the business to roll out IT projects before they have undergone the necessary security checks and repairs. This number, while still significant, has dropped from 77 per cent in the previous two versions of the report. This speaks to improvements in software and application development itself, as well as a broader realisation that catching vulnerabilities early is less costly than dealing with them later. However, even when developers do make sure to run through the necessary checks and repairs with a new product, the software itself is still often very complicated to use and takes a long time to roll out. Many enterprise software companies run into this exact predicament. So, how can they avoid the disconnection between the lab versus real-world R&D, and mitigate any problems before the roll out? There are three software development rules that all companies should live by: 1. Aspire to make software easy to
NIR POLAK CEO EXABEAM
Nir is the Co-founder and CEO at Exabeam, a start-up in the big data analytics and security intelligence space. He has a strong history in information security, having worked at Imperva for 10 years where he helped grow the business from a start-up into a NASDAQ-listed company. In 2013, Nir co-founded Exabeam, which has since been included as a 2015 Gartner Cool Vendor and has, raising US$65million in funding.
T E S T M a g a z i n e | N o v e m b e r 2 01 7
16
When it comes to new software, both users and developers face a level of risk, be it a company worried about its product suddenly becoming ineffective, or a customer constantly having to deal with installation and update failures
T E S T M a g a z i n e | N o v e m b e r 2 01 7
T H O U G H T
deploy at a large company 2. Enable employees of varying skill levels to be able to use it 3. Make sure (i.e. allow time to test) that the product is effective in the real world Put in these terms, these rules seem so obvious that many people are often surprised to learn that the majority of software isn’t already developed this way. One technique that can solve many of the software development challenges is the use of design partners. This involves a software developer working alongside several existing or potential customers from the earliest stages of product research and development. The design partnership process enables both the developers and the customers to see what works and what doesn’t in real-time, as well as which features and functions are easy for non-developers to use and vice versa. Working with design partners often results in a product that’s not only far simpler and faster to deploy and use, but also better meets customers’ needs. There are of course some key considerations to keep in mind in order to have a successful customer-developer relationship. For starters, companies need to accept that working with design partners is not easy. Great care should be taken to select these partners, as the customers that companies choose to involve can either make or break how successful the strategy is. Trust
L E A D E R S H I P
underpins this approach. The process requires customers to trust the software developers, as they will need to access both their data and people onsite. Some may see this openness as vulnerability, but it’s a necessary risk that customers need to take for the partnership to work. This trust element is also critical for the developer, as they need to be willing to show customers their intellectual property long before they would normally start to do so. This could be a concern for companies that aren’t typically comfortable with customers seeing a partially finished product. Many people argue that both of these considerations are leaps of faith, but in reality, creating enterprise software is in itself a leap of faith. When it comes to new software, both user and developer face a level of risk, be it a company worried about its product suddenly becoming ineffective, or a customer constantly having to deal with install and update failures. Communication in this process is key, as the ability to troubleshoot issues with customers can help ensure that the software is doing its job effectively. While not all firms will be open to taking such a radical approach, working with design partners is an idea worth considering if they have reliable customers who are willing to take the risk with them. After all, in this age of innovation, surely software development approaches should be subjected to some innovation too.
17
T E S T M a g a z i n e | N o v e m b e r 2 01 7
18
THE WORLD AROUND US IS GETTING
SMARTER
W
e live in a time of several revolutions/shifts in the testing approach: in the 90s to mid 2000s, most testing activities were concentrated on desktop applications and most tools were designed to fit this purpose. The next period could be called web era, where more and more tasks could be performed online, generating demand in skills for web app testers and automation tools for web based solutions. With the parallel growth of mobile application industry and the number of Android/iOS based devices, testing skills and automation tools became demanded. What all these “eras” have in common is the absence of success recipe due to lack of experience and de-facto industry standards. Something like that still can be observed when you have to work on very specific tasks where there is no known way of doing it right. One of the ongoing revolutions in QA is the testing of smart device based solutions, that will force QA specialists to adopt new skills, technologies and change the way how test processes are built. To understand how “smart” the world
T E S T M a g a z i n e | N o v e m b e r 2 01 7
around us is we can compare the number of MCUs per person manufactured each year. According to last years IEEE report, there are 20 B MCUs produced annually, which is about 3 per person. The number of smart devices connected to the internet almost achieved 7 B, which is close to 1 device per person; this number is expected to grow to 30 B by 2020, creating demand for tools, approaches and skilled QA specialists.
WHAT IS IOT TESTING AND CAN IT BE AUTOMATED? If someone in the industry is not convinced that longterm QA automation will help reduce costs, the era of smart electronics will definitely change their opinion. Although there are multiple diagrams describing smart device network architecture, lets focus on narrow examples of a smart meteostation that sends temperature and
T H O U G H T
humidity data to remote storage, and display warnings sent to metsation. If we split it into modules and data processing layers, such system could look like this: Image 1.
Temperature sensor
SoC MSU SoC MSU Display
Humidity sensor Humidity sensor
If we split it in layers, it will be obvious that smart device based solution requires a different approach, since it has to be tested on hardware, firmware and software level: Image 2. SW SW
FW FW
RESULTS
agrees on message format.
IoT gateway
Data ingestion IoT module
gateway Temperature sensor
19
L E A D E R S H I P
Data Data storage
ingestion module
Data storage
Before choosing simulator architecture and functionality, teams should outline challenges that are seen in smart device testing: 1. Interactivity tests: Simulated device Display should talk both ways and will reply to command in matter of milliseconds, that cannot be achieved by manual tests 2. Variety of models: With predicted growth of smart device market to 30 B devices, it is safe to assume that there will be new manufacturers, new device types, new functions of devices. Manual tests of each model for each release are not wise, so Simulator should be able to emulate all supported devices. 3. Non functional tests: Getting data from a device and putting it to DB does not sound like rocket since, but it becomes more complicated when the number of connected devices increased. Meaning simulation should be scalable.
HW HW
STIMULATOR
Hardware tests are typically performed using the bed of nails approach, and in some cases, manual quality control and smoke tests (literally – smoke tests). In most cases it is easier to adapt existing platforms or outsource its creation to third party and concentrate on FW and SW creations. FW creation can also be done in different ways that could be divided by hardware abstraction level. But in this article we will concentrate on testing SW, which includes simulating behavior of a smart device network.
STRUCTURE:
EXPECTED BENEFITS
As mentioned before, simulator does not emulate FW and HW modules, so this should be taken into account when building processes for a smart device solution QA. Another possible limitation is that proposed simulator architecture is not designed to emulate any outages on FW or HW side, as well as data corruption during transmission and other negative scenarios. This leaves less than 100 per cent availability, but as a first step to QA process automation, it is wise to prioritise work on positive scenarios.
Although simulator can never fully emulate real life scenarios, it is not uncommon to use them in critical industries such as healthcare (e.g. EEG and ECG baseline signal generators). The same approach could be used for smart device testing as well: code should simulate baseline device and behavior for all the device models supported by platform. Sure, they does not test HW and FW, but it is cheaper and allows to add support of devices, even before they hit the market which simply
Use of simulator in process with automated data tests allows team to: • Make sure that SW supports data processing from all smart device models • Automate data processing tests since raw data generated by simulator is available for automation scripts and can be compared against data processing result • Simulator could emulate millions of devices • Adding support of new model is a matter of minutes.
With parallel growth of mobile application industry and the number of Android/ iOS based devices, testing skills and automation tools became demanded
There are multiple mocking, simulation solutions available, but after a brief analysis of available tools, it obvious that, as a long-term solution, it would be easier to create your own tool, specifically designed for needs of the project. First steps could be hard, but later adding support of new models in a matter of minutes if it requires just adding new device data and metdata templates.
LIMITATIONS RAMA R ANEM IT PROFESSIONAL SUNPOWER
Rama R Anem is an IT professional who has managed teams and complex technology-driven projects. She has 13 years of experience, working with the likes of IBM, AMD and led global teams. Rama has contributed in technical software international conferences, workshops and conferences.
T E S T M a g a z i n e | N o v e m b e r 2 01 7
20
MANUAL TESTING VS AUTOMATION WHICH IS BETTER AND WHICH IS THE FUTURE?
Artificial intelligence is a modern phenomenon that has, and will continue, to drastically transform both formats. But does this mean the end of manual testing?
T E S T M a g a z i n e | N o v e m b e r 2 01 7
T H O U G H T
21
L E A D E R S H I P
I
n recent times, manual and automated testing have sat on an even keel. Although often laborious, manual testing provides a realistic insight into the functionality and usability of a specific platform for everyday human use. Contrastingly, automation prioritises convenience and speed in delivering the best results. In short, the answer to that question is no. Manual testing has many benefits that means it will always have a pivotal role in the testing of software and applications. The best example of the essentiality of manual testing is simply due to the weight of human input available. Apps and software are, after all, being tested for human use and getting real people to test them is always going to be very important. The usability of a particular platform requires immense subjectivity and this cannot be provided to the same degree by automated and artificial testing platforms. Furthermore, if you are operating on a tight budget manual testing is definitely preferable. The costs involved in automation can be endless and if your priority is keeping costs at a minimum, then manual testing is certainly the way to go. Thus, in terms of pricing and real life results of usability manual testing appears to have the edge over automation. So I hear you cry, why would you ever need to use automated testing if manual testing is cheaper and far more realistic? Well, in the context of finding bugs there is really no rival to automation. The factor of human error is removed entirely when you are operating through automated systems. Automation will never overlook bugs and system errors because of factors such as tiredness and fatigue. They can operate for as long as you want them to, consistently delivering results of the highest standard. Also, manual testing may be cheaper in the short run but the speed at which automation can run might mean it is actually cheaper in the long run. Automation allows you to reuse test codes and quickly add updates to your existing systems without having to write the scripts from scratch. This not only saves time but it can save you a lot of money. Furthermore, if the purpose of your study is strongly centered on data and analytics then automation may also be preferable. Automation gives you greater transparency into the quantitative elements of your study and subsequently a greater transparency into the numbers side of testing. Ultimately, there is a place for both methods in the world of testing. The desired outcome for any tests varies on a case-bycase basis and thus there is not a ‘one size
fits all’ approach that can be taken. If speed and data processing is your priority then automation should be the preferred option. However, anything relating to the humancentric importance of usability should always pose more of an emphasis on a manual approach. The relationship between the two is not a dichotomy, but rather a symbiosis where both can be used in harmony to gain the best results. It is evident in today’s market that there is a growing tendency from testing companies to prioritise automation over manual. This is not only due to the benefits of automation, but also because of the ‘AI boom’ we have seen over the past few years. However, a logical approach for testers in the future is to incorporate both into their offered services. A company like Testbirds is a good example of this, tailoring the needs of their clients to get the best results irrespective if it involves a manual or automated approach. Manual testing will always have an integral role in the testing community. However, in the future it is necessary to integrate it alongside the growing importance of automation. The relationship will not be one of mutual exclusivity However, we as the testing community need to be responsible that these two disparate methods become complimentary and not destructive to the future of testing.
DANIEL OWEN PUBLIC RELATIONS AND MARKETING TESTBIRDS
As part of the Public Relations and Marketing team at Testbirds Daniel Owen currently works for market leaders in the crowd testing of software and apps.
T E S T M a g a z i n e | N o v e m b e r 2 01 7
22
AVOIDING
THE WORST
IOT SCENARIOS
External Relations Officer at RIPE NCC, Marco Hogewoning, questions: "What happens to the millions of increasingly vulnerable devices that could remain online for decades?"
T E S T M a g a z i n e | N o v e m b e r 2 01 7
T H O U G H T
23
L E A D E R S H I P
“HELP! MY TOASTER IS TRYING TO KILL ME!” The Internet of Things (IoT) is here. With it comes some pretty cool sci-fi sounding applications, along with the more humdrum talk of connected toasters and thermostats. But the rapid pace of change also packs some hefty security and privacy risks, which could lead to serious reputational and liability issues for manufacturers. Compromised IoT devices have even been used to attack the underlying infrastructure of the internet. Informed action from a range of different stakeholders and standards development organisations is needed if we want to prevent the worst scenarios from becoming a reality.
WAIT, WE’RE AN INTERNET COMPANY NOW? Light bulb manufacturers used to produce light bulbs. Today they’ve become internet companies and most don’t even realise it yet. This shift has introduced a whole host of new requirements in terms of privacy, security and technical interoperability that need to be met quickly. Unfortunately, many companies haven’t yet woken up to the fact that these are even things they need to be thinking about. This shift introduces massive new liability risks for manufacturers. Imagine an internet-enabled thermostat that relies on a connection with the manufacturer’s network to operate. If a network outage coincides with severe low temperatures – suddenly there could be a lot of people stuck in the cold. Many IoT manufacturers appear to be sleepwalking into a crisis here. And legislation is quickly catching up, with the European Commission currently drafting new cybers ecurity requirements and the recent reform of EU data protection rules. Internet companies have different business models than manufacturers. What does it mean to sell a product that will need security patches for its entire lifetime? Software as a service may work for many technology companies, but it remains to be seen whether a subscription model can be applied to home appliances. And if a company goes bankrupt or a product line is no longer supported – what happens to the millions of increasingly vulnerable devices that could remain online for decades? To
the average user, a web-enabled CCTV camera that’s part of a giant botnet looks and mostly acts the same as one that is not. The ISPs, content providers, large academic and enterprise networks at the core of the internet have a long-established tradition of cooperation on areas affecting the digital commons. This is driven by the demand for interoperability that is needed to make the internet work. The requirement that networks peer to exchange packets has over time created a myriad of both formal and informal interpersonal networks that in many ways mirror the physical infrastructure. Network operators know one another personally and interact as part of a community to share ideas and experiences. This dynamic is lacking between IoT manufacturers who are typically in direct competition and working in isolation from one another.
WILL STANDARDS KEEP PACE? Shared standards will no doubt prove to be an important part of any solution. However, there are a few issues here as well. The broad scope of the IoT industry results in a complex landscape of different standards development organisations (SDOs). Where a number of different areas intersect, such as smart cities or networked cars, cooperation between different SDOs can prove to be a challenge. SDOs are used to drawing on their expertise and authority for a specific field – suddenly they’re faced with externalities that require expertise in fields ranging from ICT security to privacy and internet Protocol-based communications. Similar issues are faced on the regulation side. When an IoT smart meter is developed that communicates over the internet – is it the electrical regulator or the telecommunications regulator that is responsible? Meanwhile, driven by demand and a need to quickly enter these emerging markets, manufacturers and service providers can’t afford to wait for standardisation efforts to catch-up. Instead, they are often choosing to develop their own proprietary solutions. In this context, building-in security and privacy costs time and money. Solutions that are developed are often not well tested, and can present significant risks to the IoT devices and their users.
Light bulb manufacturers used to produce light bulbs. Today they’ve become internet companies and most don’t even realise it yet. This shift has introduced a whole host of new requirements in terms of privacy, security and technical interoperability
MARCO HOGEWONING EXTERNAL RELATIONS OFFICER TECHNICAL ADVISOR RIPE NCC
Marco Hogewoning is the External Relations Officer and Technical Advisor for RIPE NCC. As part of the External Relations team, he helps lead RIPE NCC's engagement with membership, the RIPE community, government, law enforcement and other internet stakeholders.
T E S T M a g a z i n e | N o v e m b e r 2 01 7
24
Through necessity, the internet industry has developed novel approaches to setting standards and addressing stakeholder concerns
T E S T M a g a z i n e | N o v e m b e r 2 01 7
T H O U G H T
WHAT DOES A SOLUTION LOOK LIKE? There are no silver bullets here, and probably there will never be a point where we can say that all of the issues with IoT have been “solved”. There will always be issues that need to be addressed. If there is a solution, it will likely come in the form of a process and a mode of working that allows the IoT industry to quickly adapt and address the many different concerns of customers, regulators and other stakeholders in a timely fashion. Here it may be instructive to look at how the traditional Internet community has worked over the years. Through necessity, the internet industry has developed novel approaches to setting standards and addressing stakeholder concerns. This has resulted in the so-called “multistakeholder model” that is built on openness, transparency, and participation from all stakeholders. There is also a kind of flexibility inherent in this model. While a perfect solution may be elusive, there is usually a compromise that can be found. “We believe in rough consensus and running code,” states one infamous internet Engineering Task Force document, which underscores the fact that many internet standards were developed “on the go” with consensus decisions based on the results of working prototypes.
L E A D E R S H I P
When there is a security incident, there is an expectation that it will be followed by appropriate and transparent disclosure. This is key not only to maintaining trust but also to safeguard other network operators who may share the vulnerability. No doubt there are similar approaches to the responsible disclosure of security and safety incidents in the airline industry, for example, and these working examples could be applied to parts of the IoT landscape, where small design mistakes or manufacturing errors can have serious and far reaching consequences. All of this is in stark contrast to today’s reality, where attempting to reveal a security vulnerability to IoT manufacturers will often generate legal threats. Finally, in as much as the IoT is about things that are connected to the internet – many of the associated issues may not be quite as novel as they first appear. There are established, open communities that have been working on network and ICT security, privacy, network abuse and related issues for decades, including RIPE, the IETF, IEEE and W3C. These communities have developed a base of standards, documentation and knowledge that will help developers who are working at the intersection at these issues. They also welcome the unique perspective of people working in the IoT field to inform their policy and standards development discussions.
25
industry news
to talk
about BOOKMARK IT TODAY!
www.softwaretestingnews.co.uk T E S T M a g a z i n e | N o v e m b e r 2 01 7
26
CHOOSING TESTING TOOLS Don’t be afraid of new arenas Elon Musk
Don’t be afraid to explore new and alternative solutions to the tools you currently use for testing
T E S T M a g a z i n e | N o v e m b e r 2 01 7
T H O U G H T
27
L E A D E R S H I P
B
efore we actually select our testing tools, however, we need to take a step back and examine the cognitive and analytical processes that occur before a testing tool is chosen or purchased. Ten years ago, testers didn’t have the multitude of choices that are now available. The decision was both quicker and easier. Today, the tester has a range of testing tools and features from which to choose. The testing arena has expanded and evolved, much akin to the evolution of ‘horseless carriages’ – automobiles that were built prior to 1 January 1916 – into self-driving cars that will become the next revolution in automotive technology. Along the way, thousands of types of motor vehicles have been developed with different functionality, safety features, maximum speeds, engine capacity and price tags. Similarly, a testing tool will have specific functionality and should meet the tester’s requirements for the job at hand. It also needs to be appropriate for the client’s budget, which in many instances is the most challenging requirement. Its capability and features have to be understood. It needs to solve a problem, and provide a solution. Do we require a 1.0-litre small family car for pottering between villages, or are we wanting the racetrack performance of a 5.0-litre V8? This is where the tester with an enquiring mind comes into his or her own. Asking the right questions will lead to the right answers – and to the right testing tools. WHAT TESTING TECHNIQUES AM I USING? Is my focus on automation testing or does my client require high-level security? Will I require regression testing? What level of testing detail is necessary to ensure the functionality of an application is free from defects? AM I TESTING THIRD PARTY PRODUCTS, CUSTOM SOFTWARE OR IN-HOUSE PRODUCTS? If I’m testing third party products, I’ll be testing the integration between them. Custom software and in-house software will require detailed testing of every individual module and component. Will my testing tool give me what I need to do the job? WHAT IS MY BUSINESS SECTOR? If it’s retail, your testing tool will need to ensure that the point of sale system is stable and functional. If it’s banking, security will be a priority and so will the capability to handle thousands of transactions concurrently.
DO I SEE NEW TECHNOLOGY AS THREATENING OR EXCITING? The advantage of today’s testing tools is that they come complete with technological advancements such as self-remediation scripts and data management. While such advancements are exciting, they can also be overwhelming or even threatening. When making your testing tools selection, be aware of your own bias and strive to keep your choices based on objective criteria. IS OUR QUEST FOR SPEED GOING TO IMPACT ON OUR QUALITY? How perfect does your product need to be before you go live? Testing an innovation where speed to market is more important than ensuring every single bug is detected will require a different approach to launching a new banking app.
Today, the tester has a range of testing tools and features from which to choose. The testing arena has expanded and evolved, much akin to the evolution of ‘horseless carriages’
IS RESEARCH PART OF OUR PREDECISION INVESTIGATION? While research plays an important role in furthering our knowledge, perhaps a sensible approach would be to limit your research time and approach it with a particular goal in mind. I know a company that appointed a tester to conduct research and development into their own test automation tool. After 12 months, the tester was unable to present a working product. Your research should enable you to present your findings to your team, project manager or CEO as a faster, better or more cost-effective solution. AM I USING A TOOL BECAUSE OF A LEGACY RELATIONSHIP? What will be the cost of moving away from the existing relationship and can I get a better solution elsewhere? The thought of moving away from an existing tool that has been in use for some years can be daunting for both tester and financial director. The good news is that the investment is not lost, as many of the new tools include the capability to migrate your existing scripts from a current system to a new system. Like choosing a new car, your testing tools have to meet your specific requirements. The analysis and evaluation process leading up to the final decision is as important as the actual testing tools you choose. Today’s testing world is opening up and presenting new options. Like the automotive industry evolving into the self-driving car, testing – and by association, how we choose testing tools - is going into uncharted territory. It remains to be seen who are the early adopters and who will be left behind.
MARIO MATTHEE HEAD OF SQA ARCHITECTURE AND R&D DVT GLOBAL TESTING SOLUTIONS
Mario Matthee is the Head of SQA Architecture and R&D at DVT and has been instrumental in building the biggest test automation centre in South Africa, the DVT Global Testing Centre. His client-centric approach is underpinned by combining the experience of older warhorses with the talent of bright youngsters, to provide quality test automation solutions for clients globally.
T E S T M a g a z i n e | N o v e m b e r 2 01 7
28
HOW TO CONDUCT TRANSFORMATIVE AGILE TESTING IN A RISKAVERSE INDUSTRY
Testing with agility is not only suitable for the finance industry, it's critical
T E S T M a g a z i n e | N o v e m b e r 2 01 7
A G I L E
P
29
T E S T I N G
rojects which run within an agile framework typically have faster product releases and better customer satisfaction; teams work more efficiently, and projects comfortably meet deadlines and budget targets. But does it suit every industry? Can risk-averse sectors such as finance also benefit from bringing testing into an agile methodology, or is there simply too much red tape and regulation to allow it to work successfully? At first glance, agile testing and finance seem incompatible, with the greatest risks appearing to be incomplete testing, and slower development. The fear that agile testing will be incomplete testing seems logical at first glance: how could you possibly test everything if you’re working iteratively in short sprints? Agile testing can seem haphazard, but of course that’s not the case. In a waterfall approach a tester will have a list of requirements, which they use to define test cases. Yet, bugs are found with regular frequency in live systems even after these ‘comprehensive’ test cases have passed. Little thought is given to how and where applications will fail - the interest is just that it has passed a particular check. There is a fundamental difference when testing with agility. Testing with agility makes it possible to use modern tooling, gain quick feedback, and execute the same level of checks without relying on heavily documented and timeintensive test cases. The second fear, that testing will slow development, can be true – in a waterfall approach. Managers who only know testing from waterfall will think of testing as a long process. Testers write a test plan and approach based on the requirements, followed by test suites and cases to meet the coverage demand. It’s a very labour intensive process. Bottlenecks are created when the development and testing are not done in parallel, and then at the end of a lengthy test period the development team are presented with a long list of problems to resolve. This process is repeated and repeated as code changes introduce more, new and different bugs, leading to increased costs and missed delivery dates. But agile testing takes the opposite approach. Testing with agility produces frequent feedback based on shared models and information, and constant questioning as development is ongoing. Even simple changes to the development cycle, such as testing on development branches, allows testers to give frequent feedback, which developers can immediately incorporate into their own work
to produce effective results. Mind maps can be an excellent aide memoire, and extremely useful alternatives to writing out full test cases. Recording exploratory journeys can provide useful evidence and the ability to replay this execution can give a greater scrutiny to spot problems. Where possible, replacing manual test cases with automation will reduce the need for manual testing for regression. Testers can also ensure good unit and integration tests are used and new ones added when code is changed and bugs are fixed, to avoid future regression. A noticeable factor in the finance industry is its reliance on reports – which is at odds with the agile manifesto’s ethos of ‘working software over comprehensive documentation’. Bringing clients into the agile process is the best way to help them realise that they don’t necessarily need or want highly detailed documentation, and that sharing knowledge with minimal documentation actually speeds up development and creates a culture of action and accountability. By being involved in the project from the beginning, finance clients see the efficacy of our testing, they understand how and why we’ve tested, and they have the confidence to be able to replicate similar tests if they do any more development in the future. We want to avoid over-documenting and we need to feel confident that we can share the right knowledge with clients and other testers who may work on the project in the future. Projects need to uncover what information is valuable or useful. Bug counts will tell you little about the state of the application, but providing usable software to demo and use can display progress - and also build confidence that you are building the right thing. The traditional view of testing is that it follows on from development: it’s a final quality check before the code is further developed, or released. In a regulation-heavy industry such as finance, this presents a real risk for development. Bringing testing in early gives your project a high level of quality and confidence from the start. If you’re working in two-week sprints, you’ll immediately see the benefits of starting testing in the first week, instead of in the middle of week two. Bugs will be found and driven out early in the cycle, smoothing the delivery towards the end of the sprint. Testing with agility is not only suitable for the finance industry, it's critical. It's best way to deliver robust, reliable software, and an essential part of any financial software development.
At first glance, agile testing and finance seem incompatible, with the greatest risks appearing to be incomplete testing, and slower development
LAURENCE PISANI HEAD OF TESTING SCOTT LOGIC
Laurence is Head of Testing at Scott Logic, a bespoke software and services consultancy delivering intuitive solutions into complex domains. With 20 years' experience in the software industry, and a keen interest in how humans interact with technology, Laurence is responsible for delivering innovative testing and DevOps solutions through the Scott Logic test practice.
T E S T M a g a z i n e | N o v e m b e r 2 01 7
30
A BRAVE NEW WORLD
Senior Manager for Enterprise Software Quality Assurance at Nedbank, Johan Steyn, shares the first chapter of his new book The Business of Software Testing. Will we find you swimming or sinking as the DevOps Tsunami hits?
T E S T M a g a z i n e | N o v e m b e r 2 01 7
T E S T
31
M A N A G E M E N T
WHAT “TESTING SCHOOL” FAILED TO TEACH YOU Software quality management and testing is an exciting career. Our peers in software development often see it as a secondary career choice, but the New World cliff that we are careening towards is forcing change at a pace that few appreciate. Technological advances such as test automation, cognitive and artificial intelligence, DevOps, the digitalisation of industries that were previously slow to adopt new technologies, and the Internet of Things are forcing changes that are breathing new life into the notion of testing as a career choice. Many test professionals are not equipped with the technical know-how to embrace New World tools and frameworks, and few are ready to grow in a career where the lines between business, development, and testing are continually blurring. Few test professionals have been trained or exposed to skills that are needed to navigate the map of the New World. We are required to be able to work with our peers in business, speak their language, and explain the process and benefits of testing with business acumen. Fewer in number still are the test professionals who effectively plan their career paths and who are enabling themselves for the next step in their careers. Over time, many land in leadership positions where they find themselves doing less testing, and dealing more with team issues, recruiting for and building their teams, budgeting, forecasting, working with external vendors, and navigating the various pitfalls of politics in corporate life. Most test professionals have never learned how to “sell themselves”. How do you build your brand and promote yourself? Are you seen as a thought leader? Have you been able to establish a network among your peers in the industry? Many testers dream about launching a test consultancy firm of their own. But, many who do so fail within the first year, as they had no clue about the difficulties this choice will introduce. So, how do you start and manage a business? How do you secure funding and control your cash flow? How do you propose your firm's offerings to new customers? The Business of Software Testing is a book that introduces these concepts to test professionals. Whether you plan to start your own company, or whether you want to climb
the corporate ladder, this book will enable you with the knowledge that is essential to prepare yourself for the next step. We are racing toward the New World cliff. You can be ready to jump with confidence and to fly to new heights.
Few test professionals have been trained or exposed to skills that are needed to navigate the map of the New World. We are required to be able to work with our peers in business, speak their language, and explain the process and benefits of testing with business acumen
THE BUSSINESS OF SOFTWARE TESTING Chapter One
THE SHIFTING SAND UNDER OUR FEET There is a momentous shift-taking place in the world of digital technology. Industries and careers that offered sanctuary to many professionals for many decades are disrupted in ways that we may never be able to grasp. Although the news, media and industry forums have been shouting this news into our ears for a long time, many of us are oblivious to the dramatic impact and speed at which we are approaching the cliff of innovation. We are entering a new technological world, a world where only the brave will survive. Who are those brave souls? They
JOHAN STEYN SENIOR MANAGER: ENTERPRISE SOFTWARE QUALITY ASSURANCE AT NEDBANK
Johan is responsible for the Business Management Office in EQA, which includes the management of the Bank’s Testing/QA spend and budget, enterprise-wide QA demand management, EQA vendor management, contractual and procurement management, and change management.
T E S T M a g a z i n e | N o v e m b e r 2 01 7
32
We are entering a new technological world; a world where only the brave will survive. Who are those brave souls? They have the foresight to understand the massive impact of what is already happening to our world, and have taken the needed steps to survive the coming tsunami
T E S T
have the foresight to understand the massive impact of what is already happening to our world, and have taken the needed steps to survive the coming tsunami. Tsunami is the right word to use here. When a tsunami approaches, we cannot do much to stop the destruction about to hit our homes. But we can heed the warnings from scientists and prepare accordingly. A tsunami moves with great speed and is usually unexpected. As meteorological technology advances, we will have more time to organise when the warning bell sounds. But we will never have enough time. A tsunami wave moves faster than we can imagine.
THE DEVOPS TSUNAMI Tsunami is the word I have been using for a long time to describe the changes in our digital world and technical careers. Some months back, I published an article on LinkedIn called The DevOps Tsunami, which caused quite a stir among my peers. The article was also picked up by an influential British Software Testing publication. Resultantly, many software quality professionals from a global spectrum contacted me to express their views. My sincere belief was that my description of the tsunami would echo what many
M A N A G E M E N T
others in our industry already knew and experienced. But I was surprised by the amount of resistance and criticism that filled by Inbox. Many who made contact expressed a belief that DevOps and the resultant impact on Software Quality Management were just a fad – another buzzword like agile or scrum – and that it would soon disappear like the sound of a jet plane passing by. They expressed a “been there – done that” view: they have seen the many changes hitting our technological world but have experienced little change in their daily lives as testing practitioners. There are always new tools at our disposal, new buzzwords and new trends. But many are still conducting software testing in a manual way, and they seem to be quite happy with that.
THE STATUS QUO This comfort zone of the status quo was built on personality cults and empires that were carefully manufactured in our corporate environments over the years. These cult leaders may have been good testing professionals in their hay-day. But over time, have they climbed the corporate ladder, nestled in a comfortable career where change and innovation was the enemy, and where like-minded minions filled the ranks of the teams they managed. They have managed to become the go-to software guys in their corporate divisions and are the holders of the keys to quality. But to justify their existence, they keep their stakeholders – especially those with the funding on which their kingdoms depend – at ransom. Concepts like automotive innovation, cognitive technology and even the expertise of vendor partners are avoided at all costs. Innovation, the reuse of assets and the employment of disruptive thinkers are not welcomed. These things will make their houses built on sand to crumble.
THE TESTERS OF TOMORROW (TODAY) The clarion call goes out to the Software Quality and Testing community. What we desperately need TODAY is an army of the “Testers of Tomorrow”. The call goes out to those testing professionals who embrace the coming tsunami with all the change and uncertainty it brings. Nothing would have prepared you for this. What does the Tester of Tomorrow look
T E S T M a g a z i n e | N o v e m b e r 2 01 7
T E S T
M A N A G E M E N T
like? First of all, it is a testing professional with good technical skills. This is not someone who is bound to a specific tool, framework or methodology. This adaptable tester allowed himself to be exposed to a variety of the tools of his trade. Exploration, hunger for growth and innovation is the name of his game. The Tester of Tomorrow is a real leader. Where many in her trade like to work in the shadows, she operates in the trenches with her team. She drives by her example of commitment and dedication and she sees the strengths in her team not as threats, but as those essential elements that will make her successful, too. She is always keen to promote others and to give praise where it is due. The Tester of Tomorrow is a commercially savvy leader. He understands that software quality management and testing is a means to an end. He always and foremost takes into account the business objectives of his customers and stakeholders. He spends time and effort with his team to ensure all are aligned with the business goals of their organisation, and aligns their testing approach and planning to these. He is measured and measures his team on the successful realisation of business goals through software quality management. The tester of tomorrow is a shrewd political navigator. She knows that both her and her team’s success rely on her political capital within her organisation. She makes sure that she is connected to the relevant influencers and that she has their ear. She knows that gossip and secondhand information within the corridors of the workplace can scuttle her success. She knows how to promote herself with skilled manoeuvring, and she always ensures that the achievements of her team and the credit due to them are visible to her stakeholders. She recovers from failures gracefully, knowing how to dust herself off and tackle the failure with ownership to exceed expectations. The tester of tomorrow is a reader and a learner. Learning never stops for this leader. He is on the cutting-edge with technological advances and innovation because he attends conferences, participates in webinars and spends time reading. He is not a lazy information gatherer. He is also well connected with his peers in the world of Software Quality. He is a voice worth listening to, a thought-leader. The tester of tomorrow lives and breathes software quality management. She is not merely a tester at the end of the cycle. She is not seen as the “stepchild of the SDLC”. Her voice and influence are heard from the very outset of a new project or feature being planned. Her
33
peers welcome her opinion and shape their planning around her guidance. She embodies “shift left” as she skillfully practices her craft throughout the software development and release process.
THE IMPOSSIBLE DREAM? What I have just described may seem like a far cry from the reality that most quality professionals experience. Its environment restricts the growth of a plant growing in a pot. Most organisations – whether end-users of software services such as banks, or even the supposed experts like global vendors – are not aware of and prepared for the tsunami. Your career ambitions as a tester of tomorrow may not realise where you currently work. Many organisations still see software testing as a necessary evil to be avoided at all costs, or at least as a grudge purchase like shortterm insurance. Traditionally, our peers in the software world looked at testers as second-hand citizens. Testing was seen for those who did not “make the cut” to become developers. One would never be able to entice a hardcore developer into a career of software testing. The tsunami will force a change here. As we wake up to the tsunami-hit world around us, and as the actual role of software quality is recognised in a world moving at a fast pace that introduces massive risk, the tester of tomorrow will find her real place. I see a world where those hardcore, weirdo ponytail developers can be enticed to focus on a career in software quality management. In this world, their technical and development skills will make them the ideal candidates to test software.
Software testing has come a long way in 30 years. When users bought apps in boxes, a waterfallstyle production process did the job, quality assuring the final product before being shipped
T E S T M a g a z i n e | N o v e m b e r 2 01 7
34
CREDITABLE OF CULTURAL
TRANSFORMATION
The Winner for Most Successful Cultural Transformation at The DevOps Industry Awards 2017 reveals how to create an exceptional culture within the work place
T E S T M a g a z i n e | N o v e m b e r 2 01 7
O R G A N I S A T I O N S
F
ollowing the Overall Winner for Most Successful Cultural Transformation at The DevOps Industry Awards 2017 on 18 October 2017, Vice-President of Cloud Technology at Barclays Investment Bank, Nick Funnell, explains to TEST Magazine Journalist, Leah Alger, the importance of creating a significant culture within the work place
CAN YOU TELL US ABOUT YOURSELF AND YOUR JOURNEY TO YOUR CURRENT ROLE? My background is as a financial software developer – my most recent role before I joined Barclays was within Credit Risk Analytics. I joined Barclays with a background in high-performance computing and analytics, but I quickly transitioned into cloud technology and infrastructure. In my current role I spend most of my time enabling others, and working out how we can most effectively engage with the rest of the bank. Partly, I spend time guiding our development methodology and the tools we employ, and work around how we’re organised and structured, and the ‘cultural’ aspects. We have many very smart technologists, and it’s essential that we ensure the needs of the organisation are met while letting them concentrate on what they’re best at. WHY DO YOU THINK BARCLAYS DESERVED TO WIN THE ‘MOST CULTURAL TRANSFORMATION AWARD?’ Our team of 70 or so people is part of Barclays GTIS (Global Technology Infrastructure Services). We have moved from a functionally-silo’d infrastructure organisation to a flat, cross-functional cloud delivery team. We have – in effect – gone from ‘racks and cables’ to near the cutting edge of softwaredriven infrastructure. While we’ve employed many of the latest techniques and tools that help to enable ‘DevOps’, fundamentally, this has been about the people: How we work as an organisation, how we communicate with each other, how we’ve challenged accepted conventions. Top-down transformation rarely works, and our success has been due to the efforts of every single person at every level within our team.
A N D
35
S T A N D A R D S
DO YOU PREFER TO WORK INDEPENDENTLY OR AS A TEAM? Personally, a mixture of both. I don’t have as much dedicated time as I did in my previous role to focus on writing code, although there are others in the team that are far better at that than me, these days. Ultimately, everything we deliver is, in some way, a team effort, so I take satisfaction from that. WHAT TYPE OF WORK ENVIRONMENT DO YOU PREFER?
I joined Barclays with a background in high-performance computing and analytics, but I quickly transitioned into cloud technology and infrastructure
This is a tricky one. There’s a lot been written about context switching, and how hard it is to focus in an open-plan office, yet we also have many useful face-to-face conversations, and there’s huge benefit in teams spending time together. Ideally, I think you’d want a mixture of open-plan desks collaboration spaces, breakout areas, but with small rooms available for focused work. We do our best with what’s available, and we have many collaboration tools, and extensive video conferencing in order to ensure we can work effectively across our locations. It gives a great sense of camaraderie and team work in all of our offices, and helps us feel like one team – it works really well. WHAT PROBLEMS HAVE YOU ENCOUNTERED WHILST AT WORK, AND HOW DID YOU HANDLE THEM? A lot of the challenges we’ve faced have been due to the complexity of the organisation. Infrastructure – as you’d expect – underpins the technology that keep the bank running, and thus the whole bank is waiting for the services we’re delivering. We have a wide array of stakeholders, many with differing priorities. We’re also engaged with many different parts of the organisation, so the main challenges have been around getting everybody to understand the vision of what we’re doing, and thus to push in the same direction – the success around our cultural transformation is partly due to how we’ve formed crossfunctional teams, and I’d like to see that culture spread further across the organisation.
NICK FUNNELL LEAD DEVELOPMENT AND PRACTICES BARCLAYS
Nick lead development and practices within the GTIS Hosting CTO organisation, focusing on the delivery of cloud platforms and services within Barclays. His background is in software development within financial services. Before his role at Barclays, he worked in credit risk analytics and high-performance and distributed computing.
T E S T M a g a z i n e | N o v e m b e r 2 01 7
36
O R G A N I S A T I O N S
A N D
I’m a big fan of the findings from the DORA DevOps Survey, which identifies lead time, mean time to recovery (MTTR), deployment frequency, and deployment failure rate as indicators of high-performing teams – measuring all of these ‘equally’ ensures positive behaviour
WHAT PERSONALITY TRAITS DO YOU LOOK FOR IN IDEAL CANDIDATES?
HOW DO YOU MEASURE SUCCESS AND OVER WHAT TIME FRAME? HOW ARE THESE METRICS DETERMINED?
There’s not really a standard set of traits – it’s down to how each candidate comes across. Tenacity is important, and a willingness to be flexible: Developing and deploying software in a large financial institution has its own challenges, and requires a level of rigour and patience – in essence, it’s a lot harder than simply working on a project at home! Personally, I think it’s important to question and challenge the status quo: The way things have been done, traditionally, often need to change. This needs to be done with integrity – helping the organisation to evolve, rather than trying to ‘go around the side’. Also, it’s not really a personality trait, but we need engineers who possess skills at the intersection of infrastructure and software development: We have infrastructure engineers, and we have software developers – those who are strong in both areas are increasingly valuable. HOW ARE EMPLOYEES RECOGNISED FOR THEIR EFFORTS? Within our wider team, we make sure we highlight and celebrate successes, and noteworthy efforts from both teams and individuals. Within the wider GTIS, we also have a mechanism (known as ‘Thank-You Tuesday’), allowing everyone to highlight and recognise when colleagues have gone over and above their role to help others. Having said that, it’s important that we don’t celebrate ‘heroics’ too much. Sometimes it’s necessary, of course, but our work should flow in a sustainable fashion, and we don’t want people working crazy hours simply to keep the lights on. Barclays also has a number of internal awards and recognition mechanisms both for technical excellence, and our ‘Better Products, Faster’ awards, which are for teams finding better (and more agile) ways of working within Barclays.
T E S T M a g a z i n e | N o v e m b e r 2 01 7
S T A N D A R D S
This is critical, in my opinion. I talk a lot about ‘eliminating gut feel’, and being able to show in concrete terms when things are improving (or not!). We use tools like JIRA and Agile Central to track work, and we can pull out metrics like lead-time and cycle time from this. I’m a big fan of the findings from the DORA DevOps Survey, which identifies lead time, mean time to recovery (MTTR), deployment frequency, and deployment failure rate as indicators of highperforming teams – measuring all of these ‘equally’ ensures positive behaviour (e.g. simply measuring deployment frequency might result in frequent but low-quality releases – measuring deployment failure rate as well corrects this). We try to track the team and product level (never at the individual level), and I’m very conscious that measuring the wrong things can drive the wrong behaviours, so it’s an art. WHAT CONTINUING LEARNING OPPORTUNITIES ARE AVAILABLE FOR BARCLAYS EMPLOYEES? We had a lot of internal training available, and various mechanisms (classroom, online, etc.). We also encourage all our team members to spend some time educating themselves – a lot of the things we’re doing have come out of ‘side projects’, and things people have been looking at in their spare time. So we’re keen on self-learning, and experimenting, although we’re also clear on when an ‘experiment’ needs to be productionised, and more rigour applied.
37
MARCH 2018 Park Inn by Radisson London Heathrow
2FOR1 OFFER! £349 (Non-vendor rate)
DON’T MISS THE NETWORKING AND ENLIGHTENING EVENT OF THE YEAR CONVENIENT LONDON LOCATION | OVER 100 SENIOR DELEGATES 12 THOUGHT LEADING DEBATES | ALL LUNCH & REFRESHMENTS NETWORKING AND KNOWLEDGE SHARING Delegates
Sponsorship
David Leakey david.leakey@31media.co.uk +44 (0)203 668 6946
Kadi Diallo kadi.diallo@31media.co.uk +44 (0)203 668 6942
www.DevOpsFocusGroups.com T E S T M a g a z i n e | N o v e m b e r 2 01 7
38
Colin Domoney, Consultant Solution Architect at Veracode, discusses different products and tools used to enhance and improve security testing
A
ccording to Domoney, Veracode secures the software that “powers your world” - no matter if you are a developer writing a new application, a business owner with a portfolio of apps, or a CISO responsible for major enterprise. Veracode has a solution to help you produce, build and operate securely.
WHITE-BOX AND BLACK-BOX Domoney said: “We inspect the source code, how it’s built, then what goes through applications through white-box testing; a method of testing software that tests internal structures or workings, instead of functionality (black-box testing). “White-box testing is an inside-out view of
T E S T M a g a z i n e | N o v e m b e r 2 01 7
looking into the internals of an application. We also offer software composition analysis which creates an inventory of open source and 3rd party components used in applications. Whilst the use of such components increases the speed of software delivery the problem with these components is that they may contain flaws.” As well as white-box testing, Veracode uses dynamic analysis (a type of black-box testing), which can be applied virtually to every level of software testing (unit, integration, system and acceptance); a method that examines the functionality of an application without peering into its internal structures or workings. “Dynamic analysis is black-box testing, but instead operates an application the way an attacker would – the complete opposite of white-box testing,” added Domoney.
T E S T I N G
39
T O O L S
GREENLIGHT Veracode’s primary product is static analysis, both at application level and individual file level with its new Greenlight product. Domoney advised: “Greenlight testing is useful. If you are testing an application it can take a long time to test. We want to encourage testers to test quickly. It gives you instant testing directly, with instant results.”
SHIFT-LEFT Different developers, of course, have different approaches towards testing. Since the 1950s, programmers knew it was better to start testing earlier, which is when ‘shift-left’ testing began, according to smartbear. Despite this, application security was a “latecomer” to this way of testing. Traditionally, security testing has been a manually driven process such as pentests. The remediation cycles associated with such tests would be measured in weeks or months due to the review and triage processes required. “Security testing is normally a manual process. You create a statement of work, and then sign a contract. It can take days, weeks or months to test software. People have expectations that it is going to be a long running process (usually around three weeks). It doesn’t suit well in the “left-shift” development side. We try to reposition the way people think, and what is capable, but it doesn’t always work like that,” revealed Domoney.
DEVOPS Veracode has to address three perceptions around the way they do testing: that security testing is difficult to use (they do this be integrating and automating with ease), that security tests are full of false positives (they have a demonstrably low false positive rate) and that security testing takes a long time (their scan times have reduced dramatically). Domoney admitted: “Three of the main problem switching to the ‘DevOps way of doing things’ was security testing – testing tools are difficult to use; tests take longer, run slowly, and are full of false positives. “At Veracode, the changes we had to make to adapt was to create easy products user-friendly. We had to make sure we integrated well with environment developers. The false-positive problem means that testing a piece of code can create extra pieces of work; although our scan times have reduced dramatically to 15 minutes or less since the
DevOps transition – a massive turnaround from three years ago.”
DOCKER Domoney believes the way developers are building applications is changing dramatically. Domoney clarified: “We’re seeing the breakdown of the monolith and a tremendous adoption of microservices, container technology and of course a move to the cloud.” This has meant that developers want to test ever increasingly smaller chunks of code, as well as test more and more frequently. The large scale adoption of automation (driven largely by the adoption of DevOps). “Traditionally, people used to use a selfcontained single-tiered software application, which combined user interface and data access code in a single programme from a single platform. “The way people build applications is changing. At first, testers would use the monolith application, which is hard to change without the whole thing crashing down. Organisations need to be more adaptive and responsive, which is challenging from a security point-of-view, because of the changes. The adoption of Docker is also getting used a lot, so developers are testing smaller and smaller pieces of code.
The way people build applications is changing. At first, testers would use the monolith application, which is hard to change without the whole thing crashing down
‘PEOPLE PROBLEM’ "It’s a people problem, not a technology problem. Most of the time problems aren’t about the technology, but instead, changing the status-quo and the way people think they should face a problem. I want people to build more secure code." Veracode will continue to address the demands from the industry, to ensure that the balance between security and speed is balanced and addressed adequately. Growth will be driven by the overwhelming groundswell in the industry, as it is no longer acceptable to produce software that hasn’t been tested particularly for security, according to Domoney. Domoney also noted Veracode’s challenge is to ensure that solutions are constantly evolving and improving – specifically by better integrations and quicker scans. Veracode believes its huge user base will start to become pivotal to success and growth – driving adoption and embedding in more and more places.
COLIN DOMONEY CONSULTANT SOLUTION ARCHITECT EMEA VERACODE
Colin Domoney is the Consultant Solution Architect, EMEA at Veracode. Colin has over 20 years of development and security expertise and joined Veracode in 2016. In his role, he helps evangelise application security and the securing of DevOps. He also works with Veracode’s largest customers, helping them secure their software estate.
T E S T M a g a z i n e | N o v e m b e r 2 01 7
40
THE KEY TO LOW-CODE AND INNOVATION TEST Magazine Journalist, Leah Alger, speaks exclusively with OutSystems, the leading low-code application development platform that helps organisations build and deliver enterprise-grade apps, rapidly
G
rowing in popularity amongst busy IT teams, “low-code” rapid application development platforms appear to be the way organisations of all sizes are finding new ways to increase efficiencies, saving time and money. Antonio Barros, Product Manager at OutSystems, revealed: “In 2016 there were 12 low code competitors, in 2017 there were 55. We are investing to make sure that we continue to support our customers’ needs for the new types of digital systems and
modern architectures that lie ahead.” Dan Juengst, Director of Global Product Marketing at OutSystems, added: “Building software and driving innovation is a critical factor for any organisation. It is the key ingredient to success in today’s marketplace. OutSystems enables this by providing a powerful platform that allows business and IT to turn ideas into applications in days or weeks.”
T E S T M a g a z i n e | N o v e m b e r 2 01 7
T E S T I N G
41
T O O L S
DRIVING INNOVATION INTO ORGANISATIONS Last month, at NextStep 2017 conference, OutSystems announced that, in order to simplify calling external artificial intelligence (AI) and Internet of Things (IoT) services delivered through the cloud, it is creating free plug-ins for the OutSystems low-code application development environment. For customers to support the change required in their businesses, OutSystems believes it needs to enable innovation by providing access to the latest technologies and best practices well ahead of its competitors. “Staying ahead of competitors and driving innovation is a lot of work for everyone. When you accomplish that, it’s a really good feeling. We work hard, then we celebrate!” revealed Juengst. Barros agreed: “Staying ahead of competitors is the key to success, as well as a strong culture of innovation. OutSystems is extremely unique in that way.”
BOOSTING SECURITY CREDENTIALS At NextStep 2017 in Lisbon, OutSystems also announced a bevy of security certifications for its low-code application platform. In addition, as part of its commitment to security best practices in cloud computing, the company recently joined Cloud Security Alliance (CSA). “Through joining the Cloud Security Alliance, OutSystems will be able to work with other industry cloud providers, end-user organisations and governmental agencies. We will work with the best and brightest, to make sure we drive the industry forward together,” added Juengst. OutSystems completed the following certifications to “boost” its security credentials: • ISO 27001: This certification, from auditor BSI, approves the OutSystems systematic approach to managing sensitive company information so it remains secure. • ISO 22301: This certification, also from BSI, approves the management system the company has put in place for business continuity arrangements. • SOC (Service Organisation Controls) 2 Type II: This attestation, from KirkpatrickPrice, demonstrates that OutSystems has selected and implemented a specific and well-defined set of security controls. The framework validates that the service provider’s data management systems are
secure, available and setup to maintain the confidentiality of data. Barros added: “Working with ISO 27001, ISO 22301 and SOC certifications meant that we had to focus on all systems security, and ensure every feature and process was adapted to the highest of standards.” OutSystems provides security best practices throughout the development lifecycle. Apps or systems created in the platform, automatically include code protection for high-risk threats, including the Open Web Application Security Project’s (OWASP) top ten: SQL Injection, CrossSite Scripting, Sensitive Data Exposure, etc. In addition, OutSystems implements security requirements, including application security checks, identity management, access control, single sign-on, encryption and auditing. “Many people aren’t aware of how much security is involved in their development. The process does not always make people happy, because it may imply additional work. But, while driving innovation, it is important to work securely,” announced Barros. “OutSystems has 196 security features, every one as important as the next. Any vulnerability is a risk, which needs to be protected – whatever the issue. That’s why we’ve worked hard to enable as much security as possible in our platform” said Juengst.
DAN JUENGST GLOBAL PRODUCT MARKETING OUTSYSTEMS
Juengst has 20+ years of experience in IT automation, DevOps, cloud computing, virtualisation, application performance management, enterprise systems management, datacentre management and automation, and enterprise computing platforms in a variety of roles, including: marketing, product marketing, sales engineering, technical marketing, systems administration, software development, and rocket engineering.
A STRONG CULTURE OF INNOVATION “We focus on hiring the best people, we train them and keep them focused on what is important: the success of our customers” admitted Barros. Juengst added: “Low-code is really taking off. We attract top people, because of our low code technology and our innovation culture, including from tier 1 engineering companies. We are growing rapidly and this infusion of smart developers is helping us drive even more innovation for our customers. “One of my greatest experiences was last month at our annual user conference, NextStep. Everyone was very enthusiastic about the platform. We had 1400 conference customers attending, and the culture of the company showed up in the culture of customers. Altogether, it was so rewarding. “We want to continue being the leading player in that perspective – top product, top marketers, top customer success driven organisation.” Barros also noted OutSystems focuses on adding value to its customers, through transforming businesses needs into realities, using a tool that no one else has.
ANTONIO BARROS PRODUCT MANAGER OUTSYSTEMS
Antonio Barros is the ‘leader in low-code’ and Product Manager at OutSystems, with 20 years’ experience of strong technical knowledge. He has managed multidisciplinary teams to achieve common goals in different areas, such as: software quality assurance, construction, customer support or software engineering.
T E S T M a g a z i n e | N o v e m b e r 2 01 7
42
REFLECTING ON AN EVOLVING INDUSTRY With 2018 around the corner, a mixture of software testers came together to reflect upon how the industry has changed dramatically, and what innovations they would like to further witness
T E S T M a g a z i n e | N o v e m b e r 2 01 7
I N N O V A T I O N S
I N
S O F T W A R E
T E S T I N G
43
A
s 2017 comes to a close, Leah Alger, Journalist of TEST Magazine, interviewed Arun Jayabalan, Head of Testing and Release Management at Metropolitan Police; Lee Crossley, VicePresident and QA Manager at JP Morgan; and Anthony Kalu-Uma, Senior Software Engineer in Test at BBC Radio/Music, to talk about changes that have happened, and changes they would like to see within the software testing industry. TO BEGIN, HOW LONG HAVE YOU WORKED IN THE SOFTWARE TESTING INDUSTRY? Arun Jayabalan: I have over 20 years' experience. Lee Crossley: I also have over 20 years' experience in the software testing industry. Anthony Kalu-Uma: I have 5 years' experience as a tester. WHAT HAS CHANGED THE MOST SINCE YOU HAVE WORKED WITHIN THE INDUSTRY? Arun Jayabalan: The infrastructure building, testing and release cycle duration. Lee Crossley: Testing education and understanding has improved greatly; although some of the education is questionable. Anthony Kalu-Uma: Testers have evolved into issue preventers, and problem solvers adopting new skills like BDD, TDD; enabling more collaboration with development teams and business stakeholders. Testers are now deeply embedded in the development process! WHAT HAS STAYED THE SAME (WHICH IS STILL IMPORTANT TO USE) SINCE YOU STARTED WORKING WITHIN THE INDUSTRY? Arun Jayabalan: Misunderstanding the requirements and bugs on a developed code. Lee Crossley: The core principles around manual testing have remained, e.g. you have only ever needed a pen and paper to test a system. Anthony Kalu-Uma: Ensuring that the quality of the software tested meets the business expectations and requirements. WHAT DO YOU ENJOY MOST ABOUT YOUR JOB ROLE? Arun Jayabalan: Every day learning new things. Implementing new open source software. Lee Crossley: The people have a great impact on my job. When you hire the right
team and align them to the right tasks, allowing a little time for innovation… then great things can happen. Anthony Kalu-Uma: I enjoy making sure that the software I'm testing works as expected. It's well known that testers get more excited than they probably should do when they find a bug – I'm definitely no different. I really enjoy the pressure and the excitement of ensuring that the BBC News Website functions as it should when we roll out extensive features during high profile events, such as the EU referendum, general elections, and the Olympics kept me on my toes and made finding a bug even more rewarding. WHAT DO YOU DISLIKE MOST ABOUT YOUR JOB ROLE? Arun Jayabalan: Sometimes you will face a different kind of attitude from people in departments like DevOps, UAT, and development. Lee Crossley: Projects using archaic and non-agile toolsets while claiming to be agile. Anthony Kalu-Uma: Funnily enough – the
ARUN JAYABALAN HEAD OF TESTING AND RELEASE MANAGEMENT METROPOLITAN POLICE
Arun Jayabalan is a speaker and software consultant with more than 21 years of experience in the software industry. Specialising in test management, automation, release management and agile coaching, he works for a public sector in London, as a Head of Testing and Release Manager.
T E S T M a g a z i n e | N o v e m b e r 2 01 7
44
I N N O V A T I O N S
I N
S O F T W A R E
T E S T I N G
LEE CROSSLEY VICE-PRESIDENT AND QA MANAGER JP MORGAN
A test professional with extensive experience in testing within multiple business sectors including finance and energy, Lee specialises in building, mentoring and leading test teams, creating/shaping the right test strategies to best fit those teams and the applications under test.
pressure. Although there are elements of it that keep me excited, managing the time constraints of any project can be quite stressful. WHAT DO YOU THINK THE THREE BIGGEST INNOVATIONS IN SOFTWARE TESTING ARE? Arun Jayabalan: Automation testing framework for multiple mobile/web platforms, open-source cross-platform servers and crowd testing. Lee Crossley: Open source technology, the automation of automation, and data virtualisation. Anthony Kalu-Uma: The adoption of agile, BDD and Gherkin. WHAT EXCITES YOU ABOUT THE FUTURE OF SOFTWARE TESTING?
ANTHONY KALU-UMA SENIOR SOFTWARE ENGINEER IN TEST BBC
Senior Software Engineer In Test at the BBC, and Founder of cinema app tech start up CineGo, fascinated with how technology can be used to solve some of the most pressing problems in the developing world.
T E S T M a g a z i n e | N o v e m b e r 2 01 7
Arun Jayabalan: More tools and frameworks for mobile testing. Lee Crossley: Aligning far closer to the ‘Dev' efforts and tool sets. Anthony Kalu-Uma: The demand for software testing will only begin to increase. Software is everywhere; in our cars, phone, homes and even our clothes. With the rise of IoT it will only continue to penetrate every aspect of our lives. Who will test IoT? The rise of IoT will lead to a conscious allocation of higher IT budgets both for software testing and quality assurance (supply and demand).
WHAT PARTICULAR TECHNOLOGIES MAKE YOUR WORK DUTIES EASIER? Arun Jayabalan: Cross-platform tools like Node and different building tools. Lee Crossley: Web services automation; it's the tie that binds all good test projects. Anthony Kalu-Uma: JIRA - for issue management Capybara - this helps test web applications by simulating real user interaction Docker; enabling me to use multiple test environments to execute scripts I have created, and to scale up overall test coverage Cucumber. This helps me collaborate effectively with non-technical people on projects I work on. IS THERE A TOOL/SERVICE YOU WOULD LIKE TO SEE INVENTED, IF SO, WHAT DOES IT DO? Arun Jayabalan: We need a more single, generic and lightweight mobile app testing tool with simulated network traffic. I feel mobile app testing with network and security features should be a must in the future. Lee Crossley: I'm not convinced revolution is required, just a healthy dose of evolution in the existing toolsets. Anthony Kalu-Uma: Adaptive intelligent testing (artificial intelligence/machine learning testing) and test scripts that learn every time they are executed – I'm not sure if this already exists, but I think it would be really effective and will increase productivity.
WHEN IT COMES TO SOFTWARE TESTING...
45
...DO YOU HAVE AN EYE FOR DETAIL? To support our continued expansion and client wins, ROQ is expanding our test team and is actively looking for experienced Test Consultants across all levels. If you are looking for your next career move, relish new challenges and are eager to collaborate and develop alongside like-minded test professionals, ROQ wants to hear from you. Permanent and Contract opportunities available across the North West and London. Ready to take your next step? Please get in touch with Katie Franklin on 01257 208890 or email careers@roq.co.uk.
www.roq.co.uk
MAKING SOFTWARE WORK T E S T M a g a z i n e | N o v e m b e r 2 01 7
46
F I N A L I S T S
Following an extensive, fair judging process, we are pleased to announce the finalists in The European Software Testing Awards 2017
C A T E G O R Y
T E S T M a g a z i n e | N o v e m b e r 2 01 7
S P O N S O R S
THE
EUROPEAN
BEST AGILE PROJECT Awarded for the best software testing project in an agile environment.
FINALISTS ★★ AEVI ★★ Brickendon ★★ Thomas Cook In Partnership With Ciklum
★★ ★★ ★★ ★★ ★★ ★★
Cognizant Harman Infosys Schroders Scott Logic Tata Consultancy Services Ltd (TCS)
SOFTWARE
TESTING
A W A R D47 S
BEST MOBILE TESTING PROJECT
BEST TEST AUTOMATION PROJECT - FUNCTIONAL
Awarded for the best use of technology and testing in a mobile application project.
The award for the best use of automation in a functional software testing project.
FINALISTS ★★ ★★ ★★ ★★ ★★ ★★
ABN Amro Accenture Digital Atom Bank Ciklum Cognizant Metro Bank in Partnership with Maveric ★★ Performance Lab ★★ QArea ★★ TestDevLab
FINALISTS ★★ A1QA ★★ British Telecom TSO In Partnership With Accenture ★★ BT In Partnership With Virtusa ★★ Bilyoner ★★ Credit Suisse ★★ Direct Line ★★ DST Systems ★★ Lloyds Banking Group Digital In Partnership With Wipro ★★ Mastek ★★ BT In Partnership with Accenture ★★ Saha Information Technologies ★★ Siskon ★★ Tricentis
Saha Information Technologies We believe in turning bugs into wisdom, failures into success. We believe that innovation is only possible when faced with hard challenges. For this reason, we welcome every bump in the road and see every obstacle as an opportunity. We work hard with confidence and devote ourselves to business excellence because we know that this is the only way that a rough road can lead us to the heights of greatness. We started our journey in 2009 by establishing Saha Information Technologies as an IT company. Later we specialised in software test automation solutions for mobile, web and desktop applications to ensure great digital experiences with the highest quality. Testinium is our flagship software test automation product offered on cloud
and on-premises deployment. Testinium utilizes Selenium and Appium libraries and is optimized for Mobile, Web and Desktop testing. Loadium is our load testing solution which is based on Apache's JMeter and SBox is our image transmission automation solution which helps to automate tests for set-top boxes including any image media player. With our highly skilled expert team of developers, we additionally provide a wide range of services including software lifecycle consulting, test outsourcing and training to meet specific business requirements of distinguished firms to take their businesses to a superior level of success. We strive to deliver the highest performance, and we are committed to meeting and exceeding the expectations of our customers.
+ 90 (216) 504 56 55 info@testinium.com Akasya Acıbem Ofis Kuleleri A-3 Blok Kat:12 Istanbul/Turkey
www.testinium.com
T E S T M a g a z i n e | N o v e m b e r 2 01 7
T 48H E
EUROPEAN
SOFTWARE
BEST TEST AUTOMATION PROJECT NON-FUNCTIONAL
TESTING
AWARDS
GRADUATE TESTER OF THE YEAR
TESTING MANAGER OF THE YEAR
The award for the best use of automation in a non-functional software testing project.
A recent graduate, who has completed university or an apprenticeship scheme in the last 2 years, who has shown outstanding commitment and development in the testing field.
Awarded to the most outstanding test manager or team leader over the last 12 months.
FINALISTS
FINALISTS
★★ Accenture Digital ★★ BT In Partnership With Accenture ★★ Dow Jones In Partnership With Infosys Limited ★★ Infosys Limited ★★ Keytorc ★★ sahibinden.com ★★ Santander Bank ★★ Sunpower ★★ UST Global
★★ Tim Beaston – Capgemini Group (Capgemini & Sogeti) ★★ Raul Khemlani – KPMG ★★ Francis McGowan – Norton Rose – In Partnership With Sparta Global ★★ Harvtar Rathore – Cognizant ★★ Rodnev Ruswa – Channel 4 – In Partnership With Sparta Global
★★ ★★ ★★ ★★
FINALISTS
★★ ★★ ★★ ★★ ★★
Andrew Baker – Lloyds Banking Group Guy Bastiaensen – Panaya Phil Edwards – nFocus Alan Hughes – Capgemini Group (Capgemini & Sogeti) Tina Kelly – EG Solutions Amy Munn – Three Sib Roy – Tech Mahindra Tamoor Sajad – KPMG Meltem Serhatlı – Hitit Computer Services
UST Global +44 (20) 7487 4818 assurance@ust-global.com 5th Floor, 40 Portman Square London W1H 6LT United Kingdom
www.ust-global.com
T E S T M a g a z i n e | N o v e m b e r 2 01 7
UST Global is a leading digital technology company that provides advanced computing and digital services to large private and public enterprises around the world. Driven by a larger purpose of transforming lives and the philosophy of “fewer clients, more attention”, we bring in the entrepreneurial spirit that seeks the fastest path to value in today’s digital economy. Our innovative technology services and pioneering social programmes make us unique. UST Global is headquartered in Aliso Viejo, California, and operates in 21 countries. Our clients include Fortune 500 companies in banking and financial services, healthcare, insurance, retail, high technology, manufacturing, shipping, and telecom. UST Global believes in building long-lasting, strategic business relationships through agile and clientcentric global engagement models that combines local experts and resources with cost, scale, and quality advantages of global operations. UST Global’s award winning and IP Driven quality engineering services are built to provide several compelling outcomes that can radically change the quality of your products and services.
We have inevitably changed baseline quality by several folds for many of our Global 1000 customers. UST Global has had enormous success with scale in QA Operations, and has consistently delivered exponentially more value in terms of savings, efficiencies, and speed. UST Global owns 25+ testing and domain accelerators that are bundled with steep expertise and strategies to increase your savings from day one. The key differentiators of our quality engineering services are: • Rapid implementation of test automation by leveraging end-to-end and scriptless test automation platforms • Accelerate the identification and provisioning of test data by enabling self service capabilities using our test data management framework • Early validation of user emotions using emotion detection technologies to enhance the end consumer experience • Zero impact transition framework to minimize business disruption during transition • Continuous Monitoring of quality indicators usingAIandanalyticsforproactiveremediation
49
April at Great Ormond Street Hospital undergoing a number of life-saving surgeries to treat her heart condition.
THE CHILDREN AT GREAT ORMOND STREET NEED YOUR HELP. Great Ormond Street Hospital is a place where, every day, seriously ill children from across the UK come for life-saving treatments. The hospital has always relied on charitable support to deliver the extraordinary care these patients need.
Donations help to fund the most up-to-date equipment, rebuild and refurbish wards and medical facilities, support groundbreaking research, and fund patient and family support services.
TO FIND OUT HOW YOU CAN TRANSFORM THE LIVES OF CHILDREN LIKE APRIL visit gosh.org Great Ormond Street Hospital Children’s Charity. Registered charity no. 1160024.
T E S T M a g a z i n e | N o v e m b e r 2 01 7
THE 50
EUROPEAN
SOFTWARE
BEST OVERALL TESTING PROJECT RETAIL Awarded to the most outstanding testing project in the retail sector.
FINALISTS ★★ ★★ ★★ ★★
Accenture Digital Cognizant Godel Technologies Europe Ltd Qualitest Group
TESTING
BEST OVERALL TESTING PROJECT - FINANCE Awarded to the most outstanding testing project in the finance sector.
FINALISTS ★★ ★★ ★★ ★★ ★★ ★★ ★★ ★★
AEVI Atom Bank Brickendon Close Brothers In Partnership With Cognizant Infosys Limited KPMG Lloyds Banking Group – Retail Business Assurance QA Mentor
AMDOCS BEST OVERALL TESTING PROJECT COMMUNICATION
BEST USE OF TECHNOLOGY IN A PROJECT
Awarded to the most outstanding testing project in the communication sector.
Awarded for outstanding application of technology in a testing project.
FINALISTS
FINALISTS
★★ Accenture Digital ★★ Deutsche Telekom In Partnership With Tech Mahindra ★★ BT In Partnership With Accenture ★★ Sunrise In Partnership With Tech Mahindra ★★ Vodafone Enterprise IT.UK ★★ Tech Mahindra ★★ Vodafone In Partnership With Accenture
★★ ★★ ★★ ★★ ★★ ★★
T E S T M a g a z i n e | N o v e m b e r 2 01 7
★★ ★★ ★★ ★★
Accenture Digital Philips In Partnership With Applause Centrica Hughes Systique KPMG Lloyds Banking Group Digital In Partnership With Sapient BT In Partnership With Accenture Cognizant BugFinders Tata Consultancy Services Ltd. (TCS) in partnership with British Gas
AWARDS
BEST OVERALL TESTING PROJECT GAMING Awarded to the most outstanding testing project in the gaming sector.
FINALISTS ★★ Harman Connected Services
THE
EUROPEAN
SOFTWARE
J U D G E S
TESTING
A W A R D51 S
2 0 1 7
DELIA BROWN HEAD OF TEST HOME OFFICE TECHNOLOGY
RIEL CAROL HEAD OF TEST YOUVIEW TV LIMITED
SUDEEP CHATTERJEE GLOBAL HEAD OF QA AND TESTING LOMBARD RISK
JASON EMERY QA MANAGER - MOBILITY HUB TUI GROUP
MYRON KIRK HEAD OF TEST COE BOOTS
AMPARO MARIN CERTIFICATION GOVERNANCE DIRECTOR BANCO SANTANDER
COLIN O’BRIEN HEAD OF QA & CHANGE CONTROL RYANAIR LABS
NIRANJALEE RAJARATNE HEAD OF QUALITY ASSURANCE THIRD BRIDGE
MALCOLM SAUNDERS HEAD OF QA AND TESTING SERVICES CLOSE BROTHERS
PAULA COPE GLOBAL HEAD OF QA TULLETT PREBON
NADINE MARTIN SENIOR MANAGER, TEST SERVICES SONY COMPUTER ENTERTAINMENT EUROPE
OLIVER SMITH QA DIRECTOR KGB – 118 118
FOLLOW THE EUROPEAN SOFTWARE TESTING AWARDS 2017 ON TWITTER PAULA THOMSEN HEAD OF QUALITY ASSURANCE AVIVA
#SoftwareTestingAwards
T E S T M a g a z i n e | N o v e m b e r 2 01 7
T 52H E
EUROPEAN
SOFTWARE
INFOSYS TESTING TEAM OF THE YEAR Awarded to the most outstanding overall testing team of the year.
FINALISTS ★★ Sky Betting & Gaming ★★ Softengi ★★ Aviva in Partnership with Tata Consultancy Services Ltd (TCS) ★★ Third Bridge ★★ Atom Bank ★★ Close Brothers ★★ Lloyds Banking Group – Insurance ★★ IBS Software ★★ Infopulse ★★ Tech Mahindra ★★ Lloyds Banking Group Digital in Partnership with Wipro
BEST USER EXPERIENCE (UX) TESTING IN A PROJECT The award for the best use of user experience testing in a project.
FINALISTS ★★ Accenture Digital ★★ UXservices ★★ Tech Mahindra
T E S T M a g a z i n e | N o v e m b e r 2 01 7
TESTING
PRODAPT TESTING MANAGEMENT TEAM OF THE YEAR Awarded to the testing management team that has shown consistently outstanding leadership.
FINALISTS ★★ Atom Bank ★★ Boots In Partnership With Cognizant ★★ Dow Jones In Partnership With News Corp ★★ IBS Software ★★ ING Bank ★★ Lloyds Banking Group – Retail Business Assurance ★★ Schroders ★★ Standard Life In Partnership With Tata Consultancy Services Ltd (TCS) ★★ Yapi Kredi Bank
LEADING VENDOR Awarded to the vendor who receives top marks for their product/service and customer service.
FINALISTS ★★ ★★ ★★ ★★ ★★ ★★ ★★ ★★ ★★ ★★
Apica Applause CA Technologies Cognizant Mobile Labs Plutora QA Mentor QASymphony UST Global Private nFocus
AWARDS
MOST INNOVATIVE PROJECT Awarded for the project that has significantly advanced the methods and practices of software testing.
FINALISTS ★★ ★★ ★★ ★★ ★★
Accenture Digital Aviva Infosys Lloyds Banking Group TestingXperts
53
T E S T M a g a z i n e | N o v e m b e r 2 01 7
54
Walking a Tight Rope with SMEs and Business User Testers? Panaya Test Center is an easy-to-use, collaborative test management platform for true IT and Business convergence.
Experience Smart Testing www.panaya.com
T E S T M a g a z i n e | N o v e m b e r 2 01 7