39 minute read
Progressive release management
by d2emerge
DEVOPSDEVOPS WATCHWATCH
Plutora’s insights dashboard lets users focus on problem areas.
Progressive release management
Plutora updates platform with Insights Dashboard
BY DAVID RUBINSTEIN
Plutora today is releasing updates to its platform with an emphasis on empowering progressive release management, a new practice that relies on analytics to focus on things that are problems.
According to Plutora, progressive release management helps organizations scale and de-risk their release processes as they transition from traditional, project-based ways of working to current Agile and DevOps approaches. This role is also part of Value Stream Management, a practice that improves the flow of work from idea to realization by showing the progress of work across an organization to identify impediments to its flow, surfacing insights on how to improve it, and providing the control to guide continuous improvement.
Where progressive release management is in place, it has been shown to double the number of features organizations can deliver, even as they release software with a third fewer incidents, the company said.
Today ’ s release includes a reworked Insights Dashboard that can direct teams to problem areas and hot spots, enabling release managers to manage by exception and free up their time and energy to improve process and efficiency, according to the company ’ s blog announcing the updates. The platform upgrade also introduces new build analytics to provide stability, quality and risk information for every pipeline. Further, there ’ s an initiative tracking analytics dashboard that matches initiatives to progress, utilizing flow metrics to predict delivery progress and enabling release managers to redirect resources to align with business priorities — something release managers today don ’t actually take into consideration.
And, given that many of the traditional release management duties are being automated, release managers increasingly feel that their role is in question in a DevOps environment, Plutora said. To stay relevant and facilitate the evolution of software development, release managers know they need to be doing more, but they also need the tools and data to accomplish that.
“The job of the release manager is going away in a DevOps world, ” said Jeff Keyes, director of product marketing at Plutora, noting that current release management practices are likened to “hair on fire, reactive ” situations. That’ s because release managers are spending their time reactively gathering data or communicating.
The key to progressive release management, he said, is to find the exceptions and work to fix those, using data from the insights dashboard and the build analytics. This practice, according to Keyes, says, “I’ m only going to focus on those things that are problems. Everything else that’ s following the process, and I have tooling in place, I never have to worry about. I’ m freeing up my time, and I’ m using data to do it. ”
The Insights Dashboard allows users to see into all the releases across the portfolio and see what’ s behind them, what’ s being followed or not being followed. “You can see all the metrics in one place, ” Keyes said, which is important because “the further along this journey a team is, the more efficient they are, and the lower risk they are. And they can help ‘ unstick’ DevOps processes because they can identify precisely what the problem is. As one release manager said, ‘It was like finding a needle in the haystack. It just popped out. I saw my problem child right away. ’ ”
“I think modern release management, in fact, maybe should start being called Value Stream Management, ” Keyes added cheekily. So, should there be a specific role of a value stream manager? There has been a lot of debate about that in the industry, and Keyes asked, “Who [in the organization] is it going to be? Are you going to hire somebody new? Will they have the knowledge to adjust the pipeline, manage the risk? That’ s what we ’ re calling progressive or modern release managers. That’ s what they ’ re doing. ” z
BY JENNA SARGENT
While we ’ re not quite fully there yet, we may not be too far away from AI being a major part of the development process, helping developers eliminate some of the more mundane tasks of coding by suggesting code, autocompleting code, and making other useful suggestions.
According to Chandra Kalle, VP of engineering at LeanTaaS, a company that provides software to healthcare organizations, much of software development is actually spent doing these mundane tasks, so handing those tasks over to a tool that can do it for them can free up an engineer to spend more time innovating. These mundane tasks Kalle is referring to can include installing third-party frameworks, writing test scripts, and other setup tasks required before coding.
“We don ’t need AI that will replace the developer, ” said Kalle. “We don ’t need AI that can really solve all the complex business logic problems that they go out to solve. But we could have AI that can keep the developer focused on working on the most important things. That will be huge. ”
According to Kryon ’ s 2021 Automation at Work survey, 32% of employees say they waste more than half of their day doing repetitive or mundane tasks. Seventy-nine percent report that those tasks take up to 30% of their time.
Almost all employees (96%) report that these tasks negatively affect their productivity. Sixty-two percent wished they had more time for professional development, 52% wished they had more time for creative thinking, and 38% wished they had more time for strategizing. Kryon believes that this disconnect might be having a negative impact on job satisfaction.
A number of IDEs have already been using AI to do autocompletion for some time, and new solutions, such as GitHub Copilot, are even able to suggest entire snippets of code.
“That is one manifestation of what’ s possible with the new generation of AI, which is quite exciting, even though it’ s very nascent. I expect more companies to leverage this approach to solve all the repetitive and mundane things that developers do, ” said Kalle.
But in their current state, these more hefty solutions aren ’t quite ready for mainstream use in production just yet. However, that doesn ’t mean they never will be.
According to Ryan Jones, VP of software engineering at Jobber, which provides business management tools, incorporating AI into the development process may be one of the most important development changes that we see over the next decade. It will allow the developer workflow to stay more in the code editor, which will allow for greater focus and innovation.
Currently, when a developer gets stuck and has a question, they have to leave their browser, open up Google, and come up with a search that might answer their question. Then they have to browse through those search results and pull out something helpful, then go back into the editor and make changes.
New tools eliminate that step and accomplish the same thing all within the editor. GitHub Copilot is one such tool, and it allows a developer to type what they need as a comment in the editor and it will create a code snippet that might help. For example, typing “Get average runtime of successful runs in seconds ” would result in a snippet of code with a function that accomplishes that.
“If you think of all the steps that it just removed, you ’ re no longer having to go to Google, you no longer have to go to Stack Overflow, ” said Jones. “I had mentioned that sometimes you ’ll take an answer from Stack Overflow, and you ’ll make changes to it. And within your code, tools like GitHub Copilot will actually be context-aware and pull variables from your code right into the solution. ”
Jones predicts that within 10 years all developers will be using a tool like that. He likens the current evolution that is going on with these tools to the changes that Gmail has gone through over the years. Google has been applying more AI to Gmail and now it’ s gotten to the point where it can fairly accurately predict what you might type.
“At the beginning it was kind of, it was kind of hokey, it didn ’t give you the exact solution, ” said Jones. “And then over time, it starts to get better and we use the tooling a lot more. And now when I use Gmail, it finishes whole sentences for me and I’ m just totally taken aback where it’ s like, holy cow, 10 years ago, this wasn ’t even a thing. You know, spellcheck was good enough. ”
One question that often comes up is the quality of suggestions, but Jones believes that’ s not a concern if true quality assurance practices are in place. Ultimately, the developer is still responsible for actually accepting the solution from the tool.
“In self-driving cars or assistive driving, at this point, you still have to keep both hands on the wheel as you ’ re driving. And I’d liken GitHub Copilot to a similar way where you still have to keep your hands on the wheel as you ’ re in your editor, and are making sure you understand what GitHub Copilot is giving you, ” Jones said.
The solution was only released over the summer, but over time Jones predicts eventually a developer might be working on a file in their editor and Copilot will say
“hey, you ’ ve done this a couple of times - do you want to do it again here?” It will become more context aware, which will make it more useful.
Horizontal learning required to uncover true power
According to Kalle, before these tools can be useful in a development environment, they need to do a lot of horizontal learning. That is, these AI tools need to learn from a wide range of similar applications in order to come up with better predictions and suggestions.
For example, there are a bunch of services that make it easy to do email marketing, such as MailChimp and SendGrid. If a developer wants to create a wrapper within their codebase to send email, that piece of code could be autogenerated because there is already a wide range of public use cases to learn from, Kalle explained.
But there are a number of vertical use cases in which the AI needs to first get better at understanding your product, the problem you ’ re trying to solve, and putting things into perspective.
“We haven ’t seen any product in both dimensions that we would use today and say, ‘Oh, wow, this has really made my life easier. ’ I think the excitement for us, for me, is the art of what’ s possible, ” said Kalle.
Cost also plays a role in adoption
Another barrier to entry for many of these tools is cost, especially for smallto medium-sized businesses. “The issue is that, at least from my perspective, for fast growing companies like us, every project, every initiative we undertake, is very costly, ” said Kalle.
According to Kalle, at his company they have a very laser sharp focus on where they invest their time in engineering effort.
“Now if I come across something like Copilot, which is still in very early nascent stages, you know, people play around with it. It’ s pretty cool. Can that be productized in any shape or form today? Absolutely not. If there is a tool that will help us, you know, write secure code, I’ll definitely take the time to take a look into it because it’ s something that keeps me up at night. But if there ’ s a tool that will make it easy for us to write code by automatically generating code, like a glorified autocomplete, if you will, we ’ re gonna take a very hard look at it and ask ourselves really what is this buying us and I can ’t think of any tools out there today that meet any of these use cases,
Testing in DevOps
BY JAKUB LEWKOWICZ
Testing in DevOps is as much about the people that are behind the tools as it is about the tools themselves. When they work in synchrony, organizations can see major benefits in the quality of their applications and SDLC process.
However some organizations still struggle with how to advance their DevOps testing initiatives because they are also implementing containerization, microservices, and other cloud-native methods that can sometimes complicate the environment.
In some organizations, those responsible for testing need to keep up with changes forced onto them by other teams, third-party applications, and platforms and also keep up with the growing list of regulatory compliance.
Since most of the applications rest on the cloud, businesses also must quickly react when cloud-based platforms receive updates.
The demand for speed and quality has prompted organizations to look towards a way to automate many of the facets of testing and changing the way that they define value.
“DevOps requires that testing is fast, accurate, meaning low false positive and low false negative rates, and runs without human intervention. Fast can be achieved with more compute power but for the tests to be accurate they need to handle the dynamic and evolving nature of modern applications, ” said Gil Sever, co-founder and CEO of Applitools.
Traditional test automation requires frequent and human intervention to update the tests through assertions and navigation, but AI has the ability to learn how the application behaves and respond appropriately, reducing the human intervention. “This makes AI essential for modern software development teams to keep pace with increased release frequency, ” Sever added.
But shifting everything to the DevOps mentality of automation is not an overnight process and in some cases, the ideal delivery story won ’t even apply to every company or any project, according to Marcus Merrell, Senior Director of Technology Strategy at Sauce Labs.
“Not all systems can do true DevOps, ” said Gareth Smith, general manager of Keysight Technologies. “If I am building a retail website, and it just requires a simple thing, then that’ s fine. But if I’ m rolling out something that needs to work with various IoT connectors, then not all platforms are able to automate all that. ”
QA brings all hands on deck for testing
Quality engineering is being elevated because the C-level sees quality engineering as a key enabler. While developers used to throw things over the wall to QA, they ’ re bringing QA into the con-
< continued from page 25 versation and the industry is seeing much more collaborative DevOps teams, where quality is a shared responsibility between developers and QA and even product owners, according to Dan Belcher, co-founder of mabl.
The interweaving of the maintenance and automation aspects of testing with the speed of DevOps has led to the new term QAOps.
“Much in the same way that we would think of shifting left as looking at those defects early on because they are then cheaper to fix, now it’ s a much greater level of having the whole structure of QA early on and throughout the DevOps cycle, ” Belcher said.
Belcher added that now the CTOs are driving the transformations. “Now it’ s a mandate coming from the C-level, to make investments in quality engineering to enable these transformations, whether it’ s digital, or DevOps, or UX. ”
While many large organizations keep a central QA department, we ’ re seeing more and more of a shift to automation developers and manual testers being assigned to individual Squads, with a Center of Excellence to support the tools. This allows testers to remain focused on business needs and not worry so much about test infrastructure or tooling, according to Merrell.
While there are still people in the organizations who are responsible for testing as part of their job title, it has also become much more of an all-hands on deck approach in DevOps.
In leading organizations, software quality has become everyone ’ s responsibility and has expanded beyond “does it work” to “is it the best customer experience ” . Developers are increasingly involved, as well as others such as UI/UX designers and domain experts, to ensure the digital experience is not only working but that it is delivering on the goals of the business, according to Applitools ’ Sever.
“This approach of having all hands on deck is beneficial because with the fast feedback cycles of DevOps, it’ s much easier for a developer to understand the impact of a change that they through a dedicated QA cycle, ” said Chris Haggan, product management lead at HCL OneTest.
AI and automation are key components of testing in DevOps
AI automation tools are necessary to provide insight by ingesting data from a plethora of data sources.
“Once you move to automated testing and a more integrated process, it enables you to check on things every step of the way and see whether you ’ re still on the right track, ” said Joachim Herschmann, senior director and analyst on the Application Design and Development team at Gartner. “I can see the direct impact that my development, bug fixing and enhancements have whether they improve or make it worse. ”
The more data that can be thrown at AI, the better the result is because it includes all of the subtle variants and different data from all the different sites that one connects it to.
“You can also use it right now to auto generate the test asset universe, what we refer to as the digital twin, ” Keysight’ s Smith said. Users of the ‘digital twin ’ can define what type of test they want and the AI will work out what the best test scenario for that situation is.
Execution speed can be increased by assigning more resources to the problem, and the key benefit to AI is its ability to learn and improve the tests over time with minimal human intervention, Applitools ’ Sever said.
There are several areas where AI has the potential to help with testing: smart crawling, although it is still in its infancy; self-healing, which is already well established and understood; and visual validation.
“For visual validation to be effective, it must be accurate to ensure the team is not overwhelmed with false positives — a problem with the traditional pixel-based approach. It needs to handle dynamic content, shifting elements, responsive designs across different screen sizes and device/browser combinations — as well as provide developers and testers ways to optimize the review and maintenance of regressions, ” Sever said.
Automation can also help with typically manual-centric types of tests such as UX testing. UX testing still requires manual input because here the outcomes of a test are subjective. However, testers don ’t need to run the tests manually for every device because they can watch tests being run on a desktop app and then decide whether the quality is acceptable or not in an assisted manual testing fashion, mabl’ s Belcher explained.
“A real simple example is if I’ m halfway through entering my credit card details, and I talk to somebody, I roll forward my device, my device goes flat, it rotates and then I come back. Now with that accidental rotation of the device and back, does that still work, ” Keysight’ s Smith said. “And in many cases, that particular use case or between those, between filling in field six and field seven on a form, then you rotate the device; no one will test that particular combination, but those happen in the real world. That’ s where AI can help look at those different combinations as you ’ re going through the usual continuous tests. ”
DevSecOps now a top priority
One of the biggest trends of 2021 is that security became a top priority for testing in the wake of massive breaches that resulted in tremendous costs.
The Executive Order on cybersecurity that the Biden administration signed in May helped to put security awareness in the spotlight, according to Jeff Williams, the co-founder and CTO of Contrast Security.
Disrupting the Economics of Software Testing Through AI: An interview with Torsten Volk
EMA (Enterprise Management Associates) recently released a report titled “Disrupting the Economics of Software Testing Through AI.” In this report, author Torsten Volk, Managing Research Director at EMA, discusses the reasons why traditional approaches to software quality cannot scale to meet the needs of modern software delivery. He highlights 5 key categories of AI and 6 critical pain points of test automation that AI addresses.
What’s wrong with the current state of testing?
Many software development teams are struggling to deliver on the promise of Agile / DevOps and are faced with numerous technical challenges, such as rising application complexity and explosion of browsers / devices. Multiply this by the constant drive for faster releases to deliver increased customer value — without the ability to hire additional quality control staff — and you can quickly see why the traditional approach to software testing can no longer scale to keep up. We need to optimize the process with AI to eliminate the mundane and repetitive tasks and ultimately ensure business success.
How does AI help with software testing?
There are five key capabilities AI provides to help improve human effi ciency: smart test creation, self healing, coverage detection, anomaly detection, and visual inspection. In the report, I discuss the six critical pain points where these capabilities are delivering ROI today: false positives, test maintenance, ineffi cient feedback loops, rising application complexity, device sprawl, and tool chain complexity. Of the capabilities available, AI-driven visual inspection has the broadest reach and highest impact. This discipline aims to provide test engineers with an additional “pair of eyes,” leaving the engineer to focus on activities that really need human intelligence. It provides humans with the contextual information needed to accelerate their test and remediation efforts, recommending solutions wherever necessary and remembering human decisions.
You talk about visual inspection having the highest impact. What pain points does it address?
Traditional pixel-based comparison a empts to perform visual inspection but is plagued with false positives. Training deep learning models to inspect an application through the eyes of the end-user removes a lot of the mundane repetitive tasks that cause humans to be ineffi cient. Due to its accuracy, AI-driven visual inspection can enable teams to create a stable suite of automated tests with reduced false positives. Visual inspection can be further leveraged to drive cross-browser / cross-device validation at scale.
How are people adopting AI? Do I need to hire AI experts or develop an AI practice?
The world’s top brands are already adopting AI to transform their software testing, but they’re focusing on the ROI gained from the speed and scalability of leveraging AI in the development process — not developing the AI itself. There are solutions on the market today that give you the ability to take advantage of fully trained models and several can be layered on top of your existing test automation. One of the solutions that seems widely adopted for visual inspection is Applitools, which is already trained on +1B images and delivers extremely high levels of accuracy.
How does the role of the developer / tester change with the introduction of AI?
Developers and testers still need to make a decision about what and how something should be automated. AI minimizes the mundane and repetitive tasks, freeing the engineers to do more creative, interesting and valuable work. For example, what happens when a tester is reviewing thousands of webpages for accuracy? Some pages have diff erences, but they don’t ma er. AI can fi lter those out and highlight only the subset that needs review. Auto-classification, grouping issues together, is another great example of being able to reduce your work and improve accuracy. If I’m looking at what looks like the same problem over and over again, I might miss a defect. Whereas if the AI categorizes and groups similar issues together it greatly reduces the possibility of a missed defect.
What’s the future state for software testing, and where do we start?
Autonomous testing is the vision for the future, but we have to ask ourselves, why don’t we have an autonomous car yet? It’s because today, we’re still chaining together models, and models of models. We’re striving to get to the point where AI is taking care of all of the tactical and repetitive decisions and humans are thinking more strategically at the end of the process, where they are more valuable from a businessfocused perspective. I would recommend starting with the mature solutions available today in the areas of visual inspection and self healing — keeping an eye on the future and ongoing evolution of AI.
If you are interested in reading the full report you can download it here:
Gil Sever, co-founder and CEO of Applitools
Applitools is helping over 400 of the world’s top digital brands accelerate the delivery of visually perfect digital experiences across all browsers, devices and screens through AI-powered test automation.
Trained on 1B+ images to deliver 99.9999% accuracy, Applitools’ Visual AI mimics the human eye and brain to deliver reliable full page validation that integrates into your existing test automation — with 50+ SDKs supporting open source frameworks (such as Selenium, Cypress, Playwright, Appium, etc.) and integrations with commercial test automation tools.
Applitools Eyes provides users with the ability to perform complete validation of the end user experience with a single line of code. Tests utilizing Applitools are 5.8x faster to create, 3.8x more stable and catch 45% more defects.
The Applitools Ultrafast Test Cloud combines Applitools Eyes with the Applitools Ultrafast Grid to deliver a modern approach to cross-browser/cross-device testing that executes tests 18.2x faster than legacy cloud execution grids or device farms.
Applitools’ Visual AI modernizes critical test automation use cases — functional and visual regression testing, web and mobile UI/UX testing, cross browser/cross device testing, localization testing, PDF testing, digital accessibility and legal/compliance testing — to transform the way businesses deliver innovation at the speed of DevOps without jeopardizing their brand.
Jeff Williams, co-founder and CTO of Contrast Security
We provide a platform of products that are designed to help companies become good at building secure code, doing it fast and reliably. And we do it by giving instant feedback to the folks that need it through the tools they’re already using.
Unlike scanners that plow through your whole application portfolio, Contrast runs in the background, a lot like an APM tool. It gathers a ton of telemetry across all your applications in parallel — APIs, cloud-native, and serverless functions — brings that all together, and gives you dashboards to show you exactly what you need.
Most developers don’t really want another dashboard, what they’d really like, is their security results, right in JIRA or they’d like to fail a build or get Slack alerts or in their IDE. There’s a million ways to consume the data that we generate, but I think the most important thing is that we have super accurate data based on observing the actual application run. We’re not guessing about vulnerabilities.
We offer Contrast Assess, which runs within the application and uses instrumentation to find vulnerabilities in your custom code and in your libraries.
We also have Contrast OSS, which finds the known vulnerabilities in all your open source so that you’re not using libraries that have known vulnerabilities.
Then we added Contrast Protect. It’s the same instrumentation approach, but now we applied it in production so that it’s super high performance and it prevents vulnerabilities from being exploited. We also added Contrast Scan, which is a static analysis tool with a new algorithm called demand-driven static analysis, making it much more efficient at finding vulnerabilities and you can run it in your pipeline. As a result of the tremendous uptake in serverless, we launched our first security for serverless offering for AWS Lambda.
Gareth Smith, general manager of Keysight Technologies
Using artificial intelligence (AI), machine learning (ML) and real user data, Keysight’s Eggplant solution automates test creation and execution. The Eggplant Digital Automation Intelligence (DAI) platform tests and monitors user interface (UI) performance to improve software development, enhance quality, and elevate the customer experience at DevOps speed.
Instead of testing the code, the DAI platform focuses on the end-to-end customer experience. It provides teams with unparalleled intelligence on where problems lie, significantly reducing the time to resolve these issues. This means organizations can meet customer experience demands and continuously deliver innovation faster while devising strategies to expand DevOps.
Customers across aerospace and defense, automotive, education, financial services, healthcare, retail, and telecoms rely on the intelligent automation. The DAI platform automates over 95% of activities, including test-case design, test execution, and results analysis. This allows teams to rapidly accelerate testing and integrate with DevOps at speed.
As environments grow more complex and interconnected and with workers distributed, organizations need continuous intelligent test automation that is easy to integrate and scale. Keysight Technologies’ Eggplant automation helps businesses rapidly create products that delight users, test the entire customer experience across any technology, and predict the quantified impact of new product versions on the user before release.
By partnering with Keysight Technologies, enterprises can deliver better software at a faster pace that delights users.
< continued from page 29
Chris Haggan, Product Management Lead, HCL OneTest
A product that is rushed to market with little time for quality assurance can massively damage the reputation of even well-established organizations. Adopting new technologies, and the fastpaced work environment driven by users who expect more from the applications they work with, will not change. It is time to find testing solutions that evolve with changing landscapes.
HCL OneTest supports UI, performance and API testing along with synthetic data generation and service virtualization to help meet the challenge of testing highly-integrated and complex applications. It features a script-less, wizard-driven, test authoring environment, and supports more than 100 technologies and protocols. HCL OneTest helps with the connections and dependencies between services and components to plan integration test strategies, and generates coverage reports to help identify which processes and services require further testing. Together, these HCL OneTest components help automate and run tests earlier and more frequently to discover errors sooner when they are less costly to fix.
To achieve a successful DevOps strategy, software testing teams must automate regression testing to reduce the risk of deploying poor quality software into production. Effective test automation includes application programming interface (API) testing, user interface testing, and overall system testing. Employing service virtualization in conjunction with test automation allows these tests to be executed earlier, while covering a wider range of scenarios.
Dan Belcher, co-founder of Mabl
We’ve seen a profound shift in how organizations view software testing and quality assurance. Historically QA received less focus and investment than other functions, but that is changing: CTOs and engineering leaders are looking to quality engineering as a key enabler of DevOps and digital transformation, which require a broader mandate to ensure that quality is embedded deeply throughout the software delivery pipeline. Mabl is the only test automation platform designed to fulfill this new mandate in the enterprise.
Mabl features a low-code UI and framework that allows everyone, regardless of coding experience, to create automated tests with 80% less effort, spanning web UIs, APIs, and mobile browsers. Using artificial intelligence, mabl reduces test maintenance with autohealing, which detects and adapts to changes automatically. With functional test creation and maintenance streamlined, QE can spend time on broader quality attributes — including performance, accessibility, and UX — while keeping pace with DevOps.
Mabl also integrates with popular tools such as Microsoft Teams, Slack, and Jira, so that users can incorporate testing information seamlessly into their workflows and benefit from rich diagnostic data from mabl. Rich reporting supports continuous improvement and improved collaboration across the software development pipeline by addressing one of the biggest inhibitors to DevOps: process changes. 1-in-3 development professionals cite the slow pace of change as their biggest DevOps challenge, making easy-to-adopt tools essential for success.
Marcus Merrell, Senior Director of Technology Strategy at Sauce Labs
Continuous testing is a key enabler of digital confidence — the knowledge that you’re delivering the best possible user experience to your customers. Digitally confident organizations know that their web and mobile applications look, function and perform exactly as intended, every single time they’re used.
Sauce Labs gives companies the confidence to deliver a flawless digital brand experience to their customers. The Sauce Labs Continuous Testing Cloud is designed to quickly identify code errors, accelerating the ability to release and update web and mobile applications that look, function and perform exactly as they should on every browser, operating system and device, every single time. Sauce Labs dramatically reduces the time and effort required to discover and fix errors using automated or manual tests, multiple frameworks, leading operating systems, and on real or virtual devices for faster, cleaner releases and more successful, trusted customer experiences. z
< continued from page 26
“I think it’ s a real harbinger of better security for apps in the future that they require a minimum standard for AppSec testing, much-improved visibility into what you ’ ve done to secure your code, including things like security labels, ” Williams said. “I look forward to a day when you can go to your online bank, insurance company, social media, or your election system and if you want to know a little bit about how that software was built, and how it was tested for security, it should be available to you; that should actually be a fundamental right. If you
’ re trusting your life, or your healthcare, or your finances, or your government to a piece of software, I think you have the right to know a little bit about how it was tested for security. ”
However, security isn ’t always handled with the utmost care at organizations. A lot of this comes down to a lack of security expertise, according to Williams.
There ’ s never enough attention being paid to security, in testing, or in development. As hard as test/security vendors work to keep up, the bad actors always seem to be one step ahead — aided by the fact that they ’ ve been every bit as institutionalized as the products they ’ re subverting, according to Sauce Labs ’ Merrell.
Security testing has traditionally required a lot of expertise to run tools such as SaaS or desktop scanners, or even SCA scanning tools.
“You can ’t just take tools designed for security experts and hand them to developers early in the process and just say ‘Go, ’” Williams said. “They ’ll end up with tons of false alarms and tons of wasted
NEW
NEW
NEW
NEW
A guide to DevOps testing tools
n FEATURED PROVIDERS n
n Applitools is built to test all the elements that appear on a screen with a single line of code. Using Applitools’ Visual AI, you can automatically verify that your web or mobile app both functions correctly and that the digital experience is visually perfect across all devices, all browsers and all screen sizes. Applitools is designed to integrate with your existing test automation rather than requiring you to adopt a new tool and supports all major test automation frameworks and programming languages covering web, mobile, and desktop apps. n Contrast Security is the industry’s most comprehensive Application Security Platform, removing inefficiencies and empowering enterprises to write and release secure code faster. The Contrast platform automatically detects vulnerabilities while developers write code, eliminates false positives, and guides fast vulnerability remediation, which enables application and development teams to collaborate more effectively. This is why many of the world’s largest organizations rely on Contrast to secure their applications in development and in production. n Keysight Technologies Eggplant Digital Automation Intelligence (DAI) platform is the first AIdriven test automation solution with unique capabilities that make the testing process faster and easier. With DAI, you can automate 95% of activities, including test-case design, test execution, and results analysis. This enables teams to rapidly accelerate testing and integrate with DevOps at speed.
Applitools: n HCL OneTest provides UI, API, and performance testing, as well as service virtualization and synthetic data fabrication, to support testers throughout the project lifecycle. It features a script-less, wizard-driven test authoring environment and support for more than 100 technologies and protocols. HCL OneTest belongs to the Secure DevOps portfolio of HCL Software, which is a division of HCL Technologies (HCL). HCL Software develops, markets, sells and supports more than 20 product families in the areas of Customer Experience, Digital Experience, Digital Solutions, Secure DevOps, Security, and Automation. n mabl is the intelligent test automation company that empowers high-velocity software development teams to integrate automated end-to-end testing into the entire development lifecycle. Mabl users benefit from a unified platform for easily creating, executing, and maintaining reliable tests that result in faster delivery of high quality, business critical applications. Learn more at https://www.mabl.com; follow @mablhq on Twitter and @mabl on LinkedIn. n Sauce Labs is the leading provider of continuous testing solutions that enable customers to deliver digital confidence. The Sauce Labs Continuous Testing Cloud delivers a 360-degree view of a customer’s application experience, ensuring that web and mobile applications look, function, and perform exactly as they should on every browser, OS, and device, every single time.
Contrast Security: HCL OneTest:
mabl:
Keysight Technologies: Sauce Labs:
n Appvance is the inventor of AI-driven autonomous testing, which is revolutionizing the $120B software QA industry. The company’s patented platform, Appvance IQ, can generate its own tests, surfacing critical bugs in minutes with limited human involvement in web and mobile applications.
n Digital.ai Continuous Testing (formerly Experitest) enables organizations to reduce risk and provide their customers satisfying, error-free experiences — across all devices and browsers. Digital.ai Continuous Testing provides expansive test coverage across 2000+ real mobile devices and web browsers, and seamlessly integrates with best-in-class tools throughout the DevOps/DevSecOps pipeline so developers can get test results faster and fix defects earlier in the process, allowing them to deliver secure, high-quality applications at-speed and atscale. Learn more at www.digital.ai/continuous-testing. n HPE Software’s automated testing solutions simplify software testing within fast-moving agile teams and for continuous integration scenarios. Integrated with DevOps tools and ALM solutions, HPE automated testing solutions keep quality at the center of today’s modern applications and hybrid infrastructures.
n IBM: Quality is essential and the combination of automated testing and service virtualization from IBM Rational Test Workbench allows teams to assess their software throughout their delivery life cycle. IBM has a market leading solution for the continuous testing of end-to-end scenarios covering mobile, cloud, cognitive, mainframe and more.
n Micro Focus: Accelerate test automation with one intelligent functional testing tool for web, mobile, API and enterprise apps. AI-powered intelligent test automation reduces functional test creation time and maintenance while boosting test coverage and resiliency. Users can test both the front-end functionality and back-end service parts of an application to increase test coverage across the UI and API.
n Microsoft’s Visual Studio helps developers create, manage, and run unit tests by offering the Microsoft unit test framework or one of several third-party and opensource frameworks. The company provides a specialized tool set for testers that delivers an integrated experience starting from Agile planning to test and release management, on-premises or in the cloud.
n Mobile Labs (acquired by Kobiton): Mobile Labs remains the leading supplier of in-house mobile device clouds that connect remote, shared devices to Global 2000 mobile web, gaming, and app engineering teams. Its patented GigaFox solves mobile device sharing and management challenges during development,
< continued from page 33 debugging, manual testing, and automated testing.
n NowSecure: Through the industry’s most advanced static, dynamic, behavioral and interactive mobile app security testing on real Android and iOS devices, NowSecure identifies the broadest array of security threats, compliance gaps and privacy issues in custom-developed, commercial, and business-critical mobile apps. NowSecure customers can choose automated software on-premises or in the cloud, expert professional penetration testing and managed services, or a combination of all as needed.
n Orasi is a leading provider of software testing services, utilizing test management, test automation, enterprise testing, Continuous Delivery, monitoring, and mobile testing technology.
n Parasoft helps organizations continuously deliver quality software with its market-proven, integrated suite of automated software testing tools. Supporting the embedded, enterprise, and IoT markets, Parasoft’s technologies reduce the time, effort, and cost of delivering secure, reliable, and compliant software by integrating everything from deep code analysis and unit testing to web UI and API testing, plus service virtualization and complete code coverage, into the delivery pipeline.
n Perfecto: Users can pair their favorite frameworks with Perfecto to automate advanced testing capabilities, like GPS, device conditions, audio injection, and more. It also includes full integration into the CI/CD pipeline, and continuous testing improves efficiencies across all of DevOps. With Perfecto’s cloud-based solution, you can boost test coverage for fewer escaped defects while accelerating testing.
n ProdPerfect is an autonomous, end-toend (E2E) regression testing solution that continuously identifies, builds and evolves E2E test suites via data-driven, machineled analysis of live user behavior data. It addresses critical test coverage gaps, eliminates long test suite runtimes and costly bugs in production, and removes the QA burden that consumes massive engineering resources. automation solution that helps teams be more efficient in functional, performance and load testing, improving test coverage and reducing the number of bugs that slip into production.
n SmartBear focuses on your one priority that never changes: quality. Whether it’s TestComplete, Swagger, Cucumber, ReadyAPI, Zephyr, or one of its other tools, SmartBear spans test automation, API life cycle, collaboration, performance testing, test management, and more.
n SOASTA’s Digital Performance Management (DPM) Platform enables measurement, testing and improvement of digital performance. It includes five technologies: TouchTest mobile functional test automation; mPulse real user monitoring (RUM); the CloudTest platform for continuous load testing; Digital Operation Center (DOC) for a unified view of contextual intelligence accessible from any device; and Data Science Workbench, simplifying analysis of current and historical web and mobile user performance data.
n Synopsys: A powerful and highly configurable test automation flow provides seamless integration of all Synopsys TestMAX capabilities. Early validation of complex DFT logic is supported through full RTL integration while maintaining physical, timing and power awareness through direct links into the Synopsys Fusion Design Platform.
n testRigor helps organizations dramatically reduce time spent on test maintenance, improve test stability, and dramatically improve the speed of test creation. This is achieved through its support of “plain English” language that allows users to describe how to find elements on the screen and what to do with those elements from the end-user’s perspective. On top of it, testRigor helps teams deploy their analytics library in production that will make systems automatically produce tests reflecting the most frequently used end-to-end flows from production.
n Tricentis Tosca, the #1 continuous test automation platform, accelerates testing with a script-less, AI-based, no-code approach for end-to-end test automation. With support for over 160+ technologies and enterprise applications, Tosca provides resilient test automation for any use case. z
< continued from page 30 time, they won ’t be able to tailor the tools properly, and they ’ll end up really frustrated with security. ”
This has created a need for tools that can be packaged in a way and in the right place for developers to use.
“There still is a role for expert-based pentesting and expert threat modeling and things like that. But they should work at the margin. Instead of trying to do everything with a pen test, including the stuff that your tools already did a great job at, have your pen testers focus on the things that are hard and difficult for tools, ” Williams said.
Evolving testing in DevOps is primarily a people process
Although tooling is necessary, testing in DevOps is also about a mindset shift on the part of the people in an organization and on making the process easier. After all, they will still have a major part to play in testing in the near future.
Organizations are showing a strong preference for low code and test automation solutions as opposed to script-based solutions. They are also looking for unified quality engineering platforms, rather than best-of-breed point solutions for various aspects of testing, according to mabl’ s Belcher.
Although AI is being applied to a growing number of use cases as part of testing in DevOps, some experts agree that there will always be humans in the loop and that the purpose of those underlying frameworks is to supercharge those people.
The next leap in the field is going to be autonomous testing where the team will steer the AI at a high level, review if the AI did the right thing and then spend most of their time focused on more strategic work, such as the usability of the application, according to Sever.
“AI is still an emerging technology, and its role in testing is evolving constantly. The most visible type of AI tooling we see is around AI-assisted automated test creation, ” Merrell said. “These tools, while extremely useful, are still no substitute for the human mind of a tester, nor do they take the place of a skilled test automation developer. ” z
Intelligent Test Automation for Agile Teams
TODAY, software development teams across the globe are facing the challenge of delivering high-quality web applications while keeping pace with business and customer demands. The risk of releasing bugs into production, impeded product velocity, and a diminished customer experience is too great.
Built for CI/CD, mabl integrates automated end-to-end testing into the entire development lifecycle. Creating, executing, and maintaining reliable tests has never been easier. With mabl, teams can:
Easily create automated UI tests - and save on test maintenance with the help of AI Create automated end-to-end tests through the UI, capturing a true end-user perspective
Increase test coverage across applications and browsers with a single platform Gain actionable insights from mabl’s rich application data for quicker issue resolution
Jira, and Gitlab
90%
Increase in test coverage
3x
Faster test creation
40%
Fewer bugs in production
Modern software development needs a modern testing solution. Try mabl free to see how easy it is to start testing.
