Testing in DevOps A GUIDE FOR BUYERS
Full Page Ads_SDT034.qxp_Layout 1 3/20/20 3:04 PM Page 32
033-38_BuyersGuide_SDT034.qxp_Layout 1 3/23/20 12:56 PM Page 33
www.sdtimes.com
April 2020
SD Times
Creating a clear testing path to DevOps takeoff BY CHRISTINA CARDOZA
D
evOps has transformed the way businesses think and software development teams work, but the power of DevOps is still limited. According to Shamim Ahmed, CTO for DevOps solutions at the global technology company Broadcom, testing still stands in the way of achieving true DevOps and continuous delivery. Testing is a time-consuming process that requires many moving parts to happen in the right way, he explained. Testing in general is just more complex, according to Maya Ber Lerner, CTO of Quali, a cloud automation and digital transformation company. “Testing is not as lightweight as development. You need to have you test automation in place, you need to have your applications in place, third-party components in place, and you need to have the right infrastructure and data set in place. Each one of those things can easily fail a test,” she said. In addition, there are just a number of different questions and scenarios you have to ask yourself when it comes to testing, Matt Davis, managing director for QA Systems, a software quality company said. “Testing, of course, is at the heart of something that you develop.
You test it to determine whether it's going to be released, but that testing can be on the basis of what should I test? Should I test everything? Should I be looking at impact analysis and changebased testing? Should I be looking at auto-test case generation?” he said. And these are just a small set of questions and thinking that is required for testing. Luckily, there are some ways teams can start to break down the barriers of testing in DevOps:
Don’t treat testing like a phase in the life cycle. Despite efforts to test early and to test often, Broadcom’s Ahmed said testing is still looked at as a particular phase in the life cycle, “when in fact it should be continuous and embedded throughout the entire life cycle,” he said. To do this, testing needs to shift left and shift right. “The more testing, especially around automation, that you can build into the application from day one is key,” said Dan McFall, CEO of Mobile Labs, an enterprise mobile app testing company. McFall explained techniques like test-driven development or behavior-driven development help developers become more involved in testing and really taking the time to look at the features and solutions, how
those are going to be tested, and how those tests are going to validate business requirements. Shifting right enables developers to work better with operations to understand what is going on in production and take advantage of that to improve tests and test conditions, according to Ahmed.
Acquire the right skills. “As we shift left and shift right with continuous testing, we need to start to bring in additional skills to the traditional QA testing mindset,” said Ahmed. One trend Ahmed is seeing is the introduction of the software development engineer in test, which are individuals who participate in development and also know testing techniques like white box testing. “These guys are able to participate with developers for example in code reviews, understanding what is going to be built, and participating in the technical debt of the code,” he said. Make testing automated and continuous. “DevOps as a process doesn’t work unless it’s automated and continuous,” said QA Systems’ Davis. He explained that in order to shift from doing nightly builds to continuously inte-
033-38_BuyersGuide_SDT034.qxp_Layout 1 3/23/20 12:56 PM Page 34
SD Times
April 2020
www.sdtimes.com
grating testing throughout the entire life cycle, things need to be 100% automated. “You can’t do things continuously if you have to manually intervene,” he noted. “You need to be able to trigger things and react to activities or react to results and outcomes all the way along your pipeline. One of the biggest door openers for automation is the connection on a pipeline, according to Davis. “There is a major advantage to being able to set up a series of quality gates and triggers, and automating different types of activities throughout the testing and development life cycle,” he said. “The more that can be automated in these pipeline stages via decorative pipeline scripts, the better it can be.” Test environments should also be automated, according to Quali’s Ber Lerner. “If you have a testing organization that’s trying to write tests real fast, and they are automated tests, but it takes two weeks or even three days to set up a test environment, are you really Agile?” she asked. Additionally, test environments should be set up in a way that separates environment issues from test issues so there is no confusion. Broadcom’s Ahmed explained another way to speed up testing is through model-based testing, a technique where tests are automatically generated by models or a description of the system’s behavior. According to Ahmed, model-based testing helps auto generate test assets on the fly to improve test productivity and make sure all the assets are available as soon as the requirements are ready. Impact analysis can also be used to speed up testing. Impact analysis, or change-based testing, helps run only the tests that were impacted by code changes, limiting the amount of tests that are actually run, according to Davis. “Quality counts for the bottom line. The way you make that bottom line more efficient is through automation and integration,” said Davis. “If software doesn’t work as well as it should, or the quality level is just not there,
companies are going to lose ground to their competitors.”
Leverage manual testing. Despite the increased need to automate things, there is still a need for manual testing in DevOps. Automated test assets free up testers to be able to do more valueadded pieces of testing such as testing from a real-user customer experience perspective or doing exploratory testing on a new piece of functionality, according to Broadcom’s Ahmed. He explained that not everything can be automated and testing things that have a human element to it is hard to automate because they can be very subjective to tests. Testers need to be able to go manually into features to actually evaluate the quality. “You want to free up testers to go find your edge cases and try to break them,” said Mobile Labs’ McFall. “There are a lot of contextual things automated testing won’t cover.” For example, he explained from a user experience perspective, if something like a drop-down menu or search field doesn’t work the way it should, test automation won’t be able to catch that. It will only tell you it is there, but it can’t tell you if it isn’t user friendly. McFall believes it is always good to have a person available to validate the test cases before they are automated just to make sure it is actually worthy of automation. Automate feedback. According to QA Systems’ Davis, there needs to be a way to share information, results and test analysis in a timely manner. Even if testing is being done manually, that doesn’t mean the feedback loop can’t be automated. Mobile Labs’ McFall explained as you manually test things like user experience and capture interactions, you can send your findings back to a tool in an automated fashion such as a real user monitoring system or customer experience system. “I am running through a manual test case, but what is automated is the inter-
action of the manual verification with something like my ticketing system so that I know in Jira, it is done, passed and automatically sending back logs and other types of information. That to me is how you can have manual testing in a DevOps environment. The feedback loop itself is automated,” said McFall.
Don’t treat all applications the same. For instance, mobile devices and applications are different from browsers and the web architecture. Mobile devices such as computers are very fragile, according to McFall, and there are many challenges around it. You need to understand the programmatic pieces around mobile apps as they relate to their infrastructure and environment. “People expect the mobile web to be similar to a desk web architecture. Things can look similar, but under the hood they are very different. It is hard to automate that if you are trying to use common frameworks,” said McFall. Enable self-service environments. According to Quali’s Ber Lerner, another main barrier of DevOps is infrastructure provisioning and application provisioning. “Making sure that everyone gets access to the infrastructure and the applications they need, whether it is people trying to run tests or systems that are trying to do automated tasks becomes a big bottleneck,” she explained. In order to overcome this, Ber Lerner explained teams need access to cloud-agnostic environments, which they can get through self-service portals APIs or different plugins. “It gives people self-service access to their test DevOps environments while it is still possible for IT Ops to govern the way that it is done. It makes it possible for people to be fast but still be in control.” “At the end of the day, it is about breaking down the barriers you have within the four walls of your organization and making sure people have access to the environments they need, systems they need, and tools they need to be successful,” said Mobile Labs’ McFall. “Let’s make sure we focus on testing the right things, and the more important things.” z
Full Page Ads_SDT034.qxp_Layout 1 3/20/20 9:36 AM Page 35
Full Page Ads_SDT034.qxp_Layout 1 3/20/20 9:35 AM Page 36
033-38_BuyersGuide_SDT034.qxp_Layout 1 3/23/20 12:56 PM Page 37
www.sdtimes.com
April 2020
SD Times
How do you help test in DevOps? Shamim Ahmed, CTO for DevOps Solutions at Broadcom, a global technology company: The promise of DevOps is that we could deliver more, faster, with no sacrifice in quality. In reality — we see some common blocks to DevOps success. At Broadcom, we address those challenges: we help eliminate the testing bottleneck and bring teams together in a single platform that lets everyone work the way they want to work. Agile teams want to work in their IDEs and command lines. They want to use open source, and they want tools that are seamlessly embedded into the CI/CD pipeline. Traditional testers want to use a UI, and features like scriptless testing. Broadcom makes this simple with BlazeMeter Continuous Testing Platform, a single application that delivers all the functionality you need to make continuous testing a reality. BlazeMeter Continuous Testing Platform is designed for every team across the SDLC. It can be used “as code” in the IDE or with the easy UI. All teams can share assets and align around common metrics and AI-driven insights. AI is also used to optimize test cycles, predict defects and highlight areas for continuous improvement. Most organizations know that DevOps success depends on the ability to shift left and right, and deliver new capabilities with volume and velocity. BlazeMeter really helps them do that — all the way from aligning the business and dev around model-based requirements to using data from production to drive continuous improvement. And best of all — we make it easy. It’s literally click to start and there’s a free version so you can get started today. Dan McFall, CEO of Mobile Labs, an enterprise mobile app testing company For Mobile Labs, we really tackle the problem of mobile devices as enterprise infrastructure. What that means is answering the questions of: Where are my devices? Who has them? What state are they in? What is on them? What application versions are loaded? What can they see? All of the things you need to basically have mobile devices be available at the development and test environment. We solve that problem, and then make them essentially act just like virtual machines. You can call them via API layers. You can build a seamless, headless process around our infrastructure component into your DevOps process. You can have a broad and deep testing space that gives you the confidence that you have covered your bases. We are also looking into more scripting as well, such as low code or no code scripting environments, more behavioral-driven environments. We are seeing that a lot of people are resource challenged, and don’t have folks who can write mobile automation. We are going to make it easier for people to do mobile automation from a scripting perspective this year. Those are the areas where we are continuing to help, which is just the right people with the right skills with the access to
the right environments at the right time. That is going to be a really key aspect to having a successful DevOps strategy. Matt Davis, managing director for QA Systems, a software quality company QA Systems helps DevOps engineers overcome the challenges of test automation and tool integration by focusing on repeatable steps and command line interfaces. Not everything in testing can be automated. However, by removing tedious manual steps from the process, we help engineers focus on building the right tests and solving problems. Automating checks on software quality metrics, architectural relationships, hierarchy and dependencies in your code, ensures that you don’t deviate from your intended design or your code becomes less maintainable as it evolves. By combining automatic test case generation, integrated code coverage, a change-based test build system, plugging testing gaps automatically and linking your tests directly to your requirements, engineers can now access unprecedented test capabilities. Code level analysis and testing should be at the heart of DevOps, where developers can use them efficiently every time code is checked in. QA Systems have found that fully automating these capabilities on the basis of open standards and integrated solutions, significantly enhances the functionality of the verification CI/CD pipeline. Maya Ber Lerner, CTO of Quali, a cloud automation and digital transformation company Test automation is great, but it only solves one part of the DevOps testing problem. To ensure the quality of your application, your developers and testers need instant access to dynamic, production-like environments throughout the value-stream to develop applications and run automated tests effectively. However, time-consuming, error-prone manual processes for setting-up and tearing down these environments creates a huge bottleneck—leading to multiple teams struggling to share static environments, or skirting around ITOps and implementing shadow-IT practices, which can greatly drive up costs and bypass security best practices. Environment as a Service solutions, like Quali’s CloudShell Colony, make it possible for developers and testers to gain immediate access to dynamic, production-like environments ondemand with one click, or automatically by connecting your CI/CD tools to accelerate the value stream. We even have a customer that set up a Slack-bot to provision environment requests. With CloudShell Colony, you can bridge the gap between Dev, Sec, and ITOps leveraging the speed of self-service, automated set-up and tear-down of dynamic environments across the value stream coupled with policy-based configurations ensuring security, compliance, infrastructure utilization, and costs control all from one tool. z
033-38_BuyersGuide_SDT034.qxp_Layout 1 3/20/20 3:07 PM Page 38
38
SD Times
April 2020
www.sdtimes.com
A guide to DevOps testing tools n BMC AMI DevOps for Db2 accelerates the delivery of new and updated applications to the market. It comes with out-ofthe box integration with Jenkins, an application development orchestration tool. n Cobalt.io is modernizing penetration testing by building hacker-like testing into development cycles. Pentests are performed by a global team of vetted, highlyskilled professionals with deep domain expertise. n Eggplant enables companies to view their technology through the eyes of their users. The continuous, intelligent approach tests the end-to-end customer experience and investigates every possible user journey, providing unparalleled test coverage essential to DevOps success. . n GitLab helps delivery teams fully embrace continuous integration to automate building, packaging, and testing their code. GitLab’s industry-leading CI capabilities enable automated testing, Static Application Security Testing, Dynamic Application Security testing, and code quality analysis to provide fast feedback to developers and testers. n HCL: AppScan is an automated application security testing and management tool. The company recently released version 10 of the solution, which features on securing DevOps. New features here include interactive application security testing capabilities out-of-the-box integrations with DevOps toolchains, and a new plugin to help developers identify vulnerabilities in their dev environments. n IBM: Continuous Testing provides an end-to-end picture of how products react to new code. It does this early in the development lifecycle which gives Product teams confidence to push incremental code changes more frequently. n Micro Focus: Minimize risk and maximize user satisfaction by testing early, often, and at scale with Micro Focus’ industry-leading, integrated portfolio for continuous and comprehensive testing of web, mobile, and enterprise applications. n Progress: Telerik Test Studio enables QA and SDET professionals to create func-
n
FEATURED PROVIDERS n
n Broadcom: The BlazeMeter Continuous Testing Platform is a complete solution for shift-left continuous testing. The platform includes UI functional testing, user experience testing, API testing and monitoring, performance testing, and virtual services. All capabilities are deeply integrated in an intuitive workflow designed for agile teams and provide robust support for popular open source tools. Delivered in SaaS with support for multiple clouds or private cloud, it is a powerful tool for delivering innovation with quality and speed. n Mobile Labs: The company’s patented GigaFox is offered on-premises or hosted, and solves mobile device sharing and management challenges that arise during development, debugging, manual testing, and automated testing. A pre-installed and pre-configured Appium server with custom tools provides “instant on” Appium test automation. GigaFox enables scheduling, collaboration, user management, security, mobile DevOps, and continuous automated testing for mobility teams spread across the globe and can connect cloud devices to an industry-leading number of third-party tools such as XCode, Android Studio, and many commercial test automation tools. n QA Systems: Cantata from QA Systems is a certified standards compliant automated unit and integration testing tool for embedded C/C++ code. Highly automated test case generation, code coverage, static metrics and requirements tracing are supplemented by architectural analysis and test status management with Test Architect and Team Reporting add-ons. Cantata is integrated with an extensive set of development toolchains, from cross-compilers and debuggers to ALM and continuous integration tools. n Quali: Quali’s CloudShell Colony helps organizations streamline effective application testing by providing development and testing teams with self-service access to automated test environments while delivering security, governance, and cost control. By removing error-prone manual inefficiencies and conflict-ridden static test environments, it creates a solid foundation for Continuous Testing and DevOps. Founded in 2007, Quali helps businesses accelerate innovation, improve quality, and control costs with on-demand access to automated application and infrastructure environment provisioning across any cloud. tional, performance and load tests that work immediately. Patent-pending multisense discovery eliminates broken tests and technical debt that plague other testing solutions. n QASymphony’s qTest is a Test Case Management solution that integrates with popular development tools. QASymphony offers qTest eXplorer for teams doing exploratory testing. n Sauce Labs: With more than 3 billion tests run and counting, the Sauce Labs Continuous Testing Cloud is the only continuous testing platform that delivers a 360degree view of your customers’ application experience.
n ShiftLeft Inspect is a next-generation static code analysis solution, purpose-built to insert security into developer workflows without slowing them down. n At SmartBear, we focus on your one priority that never changes: quality. We know delivering quality software over and over is complicated. So our tools are built to streamline your process while seamlessly working with all the tools you use — and will use. n Testlio: With robust client services, a global network of validated testers, and a comprehensive software platform, we provide a suite of flexible, scalable, and ondemand testing solutions. z
Full Page Ads_SDT034.qxp_Layout 1 3/20/20 10:27 AM Page 39