DevSecOps A GUIDE FOR BUYERS
For effective DevSecOps, shift left AND extend right
BY DAVID RUBINSTEIN
D
evSecOps has come to be known by many as the shifting left of security, making it a key part of software development while code is being written, as opposed to trying to put security onto the application after it’s completed. This follows the trends of DevOps, which moved operational considerations for applications into development, as well as software testing — though the term DevTestOps hasn’t really caught on. And DevSecOps, like many initiatives in their early stages, has awareness but often is not well understood. “People recognize the term DevSecOps and have a general notion of what it means,” said Jeff Williams, co-founder and CTO at Contrast Security. “It means shifting left and automating security somehow, but in practice we’re really just at the very early stages of this. I think most folks don’t have a very well-formed idea of exactly what they need to do [for] DevSecOps. In fact, I think most people get it dramatically wrong.” Shifting left, of course, puts even more responsibility on developers, who have been trained to write code, and
even find and correct errors in code, but who largely have not been trained on security best practices. “People have this idea that shifting left is taking the things you’re currently doing and pushing them on to the developers,” Williams said. “I call it shitting left. It doesn’t work. The way that security works today is largely built around experts, using expert tools. You can’t just take those same tools and shove them onto developers who don’t have the skills or background to be effective using them, and expect great results. All you’re going to do is create a lot of alienated developers who don’t do good security. You’re probably going to end up hurting security overall.” That could be problematic for organizations where security is a priority, as 73 percent responding to a recent Forrester study said it is. Trusting developers alone with security is the wrong approach, as developers are often the ones introducing insecure code into their applications through the use of open-source components. The Forrester report found there were 17,308 vulnerabilities published in 2018, up 23 percent from a year earlier. It also indicated that a best security practice would be to have developers
Buyers Guide Just moving everything onto developers, who aren’t trained in security, could actually hurt your efforts
reduce the attack surface in their applications as they code. “Today’s reality is that developers don’t code securely,” the report stated. “When measured against major industry vulnerability standards, 70 percent of applications fail security testing on the first scan.” None of this is to say developers are at fault here. Forrester noted that the top 40 computer science programs in the United States do not require secure coding or application design in their curricula. Yet Williams cautioned against taking too much of a developer-centric view of DevSecOps, and noted that many people trying to do it correctly simply forget about the ‘extending right’ part of DevSecOps. Particularly in application security, Williams said most organizations don’t have any idea of who’s attacking them, or what attack vectors they’re using to go after them, or what systems they’re targeting. Large organizations, he said, are blind at that level and they don’t have a way of stopping those attacks or using intelligence of how they’re being attacked to drive their security strategy. “It’s that kind of feedback loop that’s one of the real characteristics of DevOps,” he explained. “Someone attacks you
using a new attack, that should instantly drive changes in the product. But we don’t have that feedback loop, so really, when I think about DevSecOps, it’s about continuing to do what you do now — generate assurance — but extend left and extend right. This idea of shifting left is dumb, and dangerous. It’s unfortunate, but those [advocating shift left] are people who haven’t thought it through very well.”
Doing DevSecOps effectively Today, many organizations are just taking the DevSecOps name and pinning it on trivial modifications of what they’ve been doing, and it’s not really that, Williams said. “DevSecOps is a fundamental transformation of security the way that DevOps is a fundamental transformation of the way we build software,” he explained. “My friend at Comcast runs their security program and says vendors are putting DevSecOps lipstick on a traditional security pig, because they’re not fundamentally changing how their products work; they’re just kind of taping them onto a DevOps pipeline and going, “Yep, we’re DevSecOps! Look!” Williams went on to say the organizations that are doing DevSecOps effectively are being smart about security across the entire software life cycle. “In dev, what that means is you empower developers to find and fix their own vulnerabilities, fix their own code, and check in clean code,” Williams said. “Seems pretty straightforward, and automation is a big part of that. It’s got to be accurate, because developers don’t
have [the] time or skills to deal with inaccuracies. If we can achieve that in dev, there are some really good downstream benefits from that. In CI/CD, both traditional and what we might call QA, in that stage, I think the goal has to be to generate assurance; that what you’re pushing into production has been thoroughly tested and is free of vulnerabilities.” Traditionally, this assurance came from a big test after the application was complete. So by pushing all that to the left, that final assurance is lost. But, if security has been factored in earlier in the process, the big test should find nothing because tests were done along the way and any found vulnerabilities would have been remediated. There is no assurance, though, that an effort to do DevSecOps effectively will succeed, because — like Agile, DevOps and Value Stream — the methodologies are not prescriptive. Organizations are usually left to their own devices to determine how they are going to realize the benefits. “There’s some real value in DevSecOps and I don’t want to see the term get watered down to apply to anything that’s security,” he said. “I think it really does mean something. When I go back to the fundamental principles of DevOps, things like breaking down the work into small pieces to create flow, creating tight feedback loops and creating a culture of innovation and learning, those three things, if you interpret them for security, that’s DevSecOps. So that means breaking security work down to small pieces to create flow; it
A self-protecting prophecy Cybersecurity expert Ed Amoroso talks about a model he calls Explode-Offload-Reload. Contrast Security’s Jeff Williams explained: “What that means is as you move from the traditional internal monolithic applications, you need to explode them into pieces, and move each of those workloads into the cloud, that’s off-loading, and then reload means adding those protections back to the stack that runs that code, creating a secure, self-protecting instance in the cloud. Instead of having one big wall, now you’ve got a whole bunch of little walls. It’s not even good to think about walls; it’s really just to secure applications that are able to protect themselves. But I like that description because he’s talking about how organizations can move from a very sort of traditional outside-in approach to security to the future, which is this self— David Rubinstein protecting way of doing things.” z
means creating tight security feedback loops, and it means creating a culture of security innovation and learning. “Those are the three ways of DevOps and I think they’re essentially the same for security,” he added. “But very few organizations are really focused on that. They’re focused on let’s buy some new tool and plug it into our CI/CD pipeline, and bam! We’re DevSecOps. But that is not how it works. You’re not going to achieve a transition overnight. You’re gonna have to do it piece by piece over the course of years.” DevSecOps, by its very definition, encompasses the entire stack, from coding, to UI, to the infrastructure it’s running on, and Williams added that the whole stack is turning into software. “If you’re deploying into the cloud, you’ve got a container on top of that, maybe you’ve got an app server running in the container, you’ve got libraries in the app server in the container, and you’ve got trusted code running on top of the app server... but it’s all really software.”
Inside out, outside in, perpetual change To have an effective DevSecOps practice, you have to approach security at each layer of the stack. If you’re running containers, you’ll need to create rules to ensure that the container has no vulnerabilities, is built with the proper defenses, and that it’s being monitored at runtime. “The old way we used to do that is with what I’ll call an outside-in approach,” Williams explained. “We used to put a firewall around it and scan the shit out of it, and try to see if the whole thing is secure. The problem is, modern architectures are much too complicated for that. I think the effective approach today is to get inside the thing we’re trying to secure. If you’re trying to secure a container, you need to be inside the container asking those questions about security. If you’re trying to secure an app server, you need to be inside the app server. If you’re trying to secure custom code, you need to be inside that custom code. That’s where you have all the information to make a smart decision about whether something is secure or not.”
Why do the same vulnerabilities keep showing up? Jeff Williams, co-founder and CTO of Contrast Security, created the OWASP Top Ten list, first published in 2003. While he’s proud of the work done, he’s a little disappointed that the list has not changed all that much in 16 years. “My thought at the time was, we’ll put this Top Ten out, we’ll solve some of these issues and we’ll raise the bar over time to get to a place where application
security is a lot better,” Williams said. “It’s hard to believe that it’s almost 20 years later. Part of me is like, they’re difficult to solve because they’re pervasive across so much code everywhere, and some of them are tricky to find. But at the same time they’re also [doing] basic blocking and tackling, like solving SQL injection is not particularly hard. We’ve taken this approach of mostly chasing
Top Ten 2003
1 2 3 4 5 6 7 8 9 10
2017
Unvalidated Input
Injection
Broken Access Control
Broken Authentication
Broken Authentication and Session Management
Sensitive Data Exposure
Cross-Site Scripting (XSS) Flaws
XML External Entities (XXE)
Buffer Overflows
Broken Access Control
Injection Flaws
Security Misconfiguration
Improper Error Handling
Cross Site Scripting (XSS) Flaws
Insecure Storage
Insecure Deserialization
Denial-of-Service
Using Components with Known Vulnerabilities
Insecure Configuration Management
Insufficient Logging & Monitoring
What Williams described is an instrumentation-based approach to security. Contrast Security, he said, doesn’t do container or cloud security. What Contrast does is instrument the application layer so vulnerabilities can be found and so the team can prevent vulnerabilities from being exploited at runtime. “If you zoom out and look at that, you can imagine instrumenting each layer of the stack with the right products, and then that stack is secure. It secures itself. And then you can put that stack wherever you want. If you want to put it internally, in an internal data center, great. If you want to put it in the cloud, great. The security goes with the code. For me, we’re talking about securing everything, and that’s a very DevOps/Cloud/Container kind of way
of doing security. It doesn’t matter if you’re rolling out tons of elastic servers or you’re spinning up containers all over the place, because the security goes with the code. Trying to do that kind of protection with an outside-in approach is impossible, because you can never keep the walls up around everything, and you can never scan everything from the outside, because what’s in there keeps changing, moving.” As Williams said, automation must play a big role in DevSecOps, because automation is what creates the guardrails around your development pipeline, to ensure no bugs or vulnerabilities sneak into the code and gets pushed into production. “So you have this automated pipeline that does all that work; that optimizes for the devel-
vulnerabilities and trying to remediate them as opposed to changing the way that we interact with databases. If everyone used prepared statements everywhere, we’d be a lot closer to solving SQL injections. It’s when people write custom queries and concatenate in untrusted data that we get into trouble.” He said he believes the right path forward is to give developers great automation so they just get alerted whenever they step outside the guardrails that DevSecOps provides. “For me, we’re not going to train our way out of this, we’re not going to pen test our way out of this, we’re not going to static analysis our way out of this. We’re going to have to get really good accurate automation that works instantly if we want to solve this, because the scale of the problem is just staggering.“ WIlliams went on to note that on average, applications have 27.6 serious vulnerabilities.” If we were an airline, and on average every time you did a safety check there were 27.6 safety problems, nobody would ever leave the ground,” he said. “But we don’t treat it like airline safety. People are a lot... we don’t take it as seriously as we should, as a country or a world. We just don’t. We could do better. We just need the commitment.” z — David Rubinstein
oper the ability to create new functionality and push it into production quickly,” Williams said. “All security, especially application security, has massive scale problems. There are just not enough people to do the work the old way, so you have to automate. Most big organizations, they’re really only doing effective application security on 10 percent of their applications. They only secure the public-facing stuff, or the ones they deem to be critical. They’re not securing all their applications, and it’s a huge risk. The only way to fix that problem is we’ve got to change the economics. We’ve got to figure out a force multiplier, and I believe that is DevSecOps. By empowering developers, we can use the big machinery of software development to do the security work.” z
029-33_SDT024.qxp_Layout 1 5/22/19 6:18 PM Page 32
A guide to DevSecOps tools n Aqua Security enables enterprises to secure their container and cloud-native applications. The Aqua Container Security Platform protects applications running onpremises or in the cloud, across a broad range of platform technologies, orchestrators and cloud providers. Aqua performs image scanning for known vulnerabilities during the build process, image assurance to enforce policies for production code as it is deployed, and run-time controls for visibility into application activity. n CA Veracode creates software that fuels modern transformation for companies across the globe. DevSecOps enables the build, test, security and rollout of software quickly and efficiently, providing software that’s more resistant to hacker attacks. Through automation, CA Technologies helps teams work collaboratively earlier in the DevSecOps process to detect security vulnerabilities in every phase, from design to deployment. n CodeAI is a smart automated secure coding application for DevOps that fixes security vulnerabilities in computer source code to prevent hacking. Its unique user-centric interface provides developers with a list of solutions to review instead of a list of problems to resolve. Teams that use CodeAI will experience a 30-50 percent increase in overall development velocity. n Synopsys helps development teams build secure, high-quality software, minimizing risks while maximizing speed and productivity. Synopsys provides static analysis, software composition analysis, and dynamic analysis solutions that enable teams to quickly find and fix vulnerabilities and defects in proprietary code, open-source components, and application behavior. n Checkmarx provides application security at the speed of DevOps, enabling organizations to deliver secure software faster. It easily integrates with developers’ existing work environments, allowing them to stay in their comfort zone while still addressing secure coding practices. n Chef Automate is a continuous delivery platform that allows developers, opera-
n
FEATURED PROVIDER n
n Contrast Security: Contrast Security’s Contrast Assess produces accurate results without dependence on application security experts, using deep security instrumentation to analyze code in real time from within the application. It scales because it instruments application security into each application, delivering vulnerability assessment across an entire application portfolio. Contrast Assess integrates seamlessly into the software lifecycle and into the toolsets that development & operations teams are already using. Contrast Protect provides actionable and timely application layer threat intelligence across the entire application portfolio. Once instrumented, applications will self-report the following about an attack at a minimum: the attacker, method of attack, which applications were attacked, frequency, volume, and level of compromise. Protect provides specific guidance to engineering teams on where applications were attacked and how threats can be remediated. Contrast doesn’t require any changes to applications or the runtime environment, and no network configuration or learning mode is necessary. tions, and security engineers to collaborate effortlessly on delivering application and infrastructure changes at the speed of business. Chef Automate provides actionable insights into the state of your compliance, configurations, with an auditable history of every change that’s been applied to your environments. n CloudPassage has been a leading innovator in cloud security automation and compliance monitoring for high-performance application development and deployment environments. Its on-demand security solution, Halo, is a workload security automation platform that provides visibility and protection in any combination of data centers, private/public clouds, and containers. Delivered as a service, Halo integrates with infrastructure automation and orchestration tools along within CI/CD tools. n CollabNet offers solutions across the DevOps toolchain. Its solutions provide the ability to measure and improve endto-end continuous delivery, orchestrate delivery pipelines and value streams, standardize and automate deployments and DevOps tasks, and ensure traceability and compliance across workflows, applications, and environments. n CyberArk Conjur is a secrets management solution that secures and manages secrets used by machine identities (including applications, microservices, CI/CD tools and APIs) and users throughout the
DevOps pipeline to mitigate risk without impacting velocity. Conjur is the only platform-independent secrets management solution specifically architected for containerized environments. n IBM provides a set of industry-leading solutions that work with your existing environment. And of course they work fantastically together: Change is delivered from dev to production with the IBM UrbanCode continuous delivery suite. Changes are tested with Rational Test Workbench, and security tested with IBM AppScan or Application Security on Cloud. IBM helps you build your production safety net with application management, Netcool Operations Insight and IBM QRadar for security intelligence and events. n Imperva offers many different solutions to help you secure your applications. Imperva WAF protects against the most critical web application security risks: SQL injection, cross-site scripting, illegal resource access, remote file inclusion, and other OWASP Top 10 and Automated Top 20 threats. Imperva security researchers continually monitor the threat landscape and update Imperva WAF with the latest threat data. n JFrog Xray is a continuous security and universal artifact analysis tool, providing multilayer analysis of containers and software artifacts for vulnerabilities, license compliance, and quality assurance. Deep recursive scanning provides insight into
your components graph and shows the impact that any issue has on all your software artifacts. n Nosprawl integrates with software development platforms to check for security vulnerabilities throughout the entire software development life cycle to deliver verified secure software before it goes into production. n Parasoft: Harden your software with a comprehensive security testing solution, with support for important standards like CERT C, CWE, and MISRA. To help you understand and prioritize risk, Parasoft’s static analysis violation metadata includes likelihood of exploit, difficulty to exploit/remediate, and inherent risk, so you can focus on what’s most important in your C and C++ code. Parasoft provides flexible, intelligent dashboards and reports specifically designed for each standard to provide necessary information for reporting and compliance auditing. n Qualys is a leading provider of information security and compliance cloud solutions. The Qualys Cloud Platform and apps integrated with it help businesses simplify security operations and automates the auditing, compliance, and protection for IT systems and web applications. n Redgate SQL Provision supports database DevSecOps, keeping compliance central to the process. It enables multiple clones of masked databases to be created in seconds, allowing them to be used safely within the development and test process. n Perforce helps thousands of global enterprise customers tackle the hardest and most complex issues in building, connecting, and securing applications. Our Klocwork static code analysis tool helps DevSecOps professionals, from developers to test automation engineers to compliance leaders, create more secure code with on-the-fly security analysis at the desktop and integrated into large-scale continuous integration workflows. n Signal Sciences secures the most important applications, APIs, and microservices of the world's leading companies. Our next-gen WAF and RASP help you increase security and maintain site reliability without sacrificing velocity, all at the lowest total cost of ownership.
What does Contrast bring to the table to address DevSecOps? Jeff Williams, co-founder and CTO, Contrast Security Contrast is an integration platform for application security. We use an instrumentation-based approach, so we work from inside the running application layer. From there, we support the entire software life cycle with three things. The first thing is, we help identify vulnerabilities. Typically you want them to be discovered really early in the life cycle, so that’s what we do. As developers are writing their code, they can get instant feedback on the code that they’re writing, they can fix those problems the way they normally could, and they can check in clean code without breaking stride. There is no scanning. I want you to imagine all of your applications — there could be thousands of applications in an enterprise — I want you to imagine them all testing themselves simultaneously, as opposed to having to go to each one and scan it, serially. It’s a very scalable approach to application security, finding vulnerabilities. The second thing that we do is we analyze open-source libraries for both known and unknown vulnerabilities. So this is really a big deal. There have been a bunch of big breaches related to the use of open-source libraries. Contrast is an effective way of doing that at scale in real-time across the organization, and our big differentiator there is that we can tell you exactly how each of those libraries is being used. Instead of just saying, ‘you’re using that library, therefore you have to replace it,’ we tell you ‘whoa, whoa, that has a vulnerability, but you’re never actually invoking that library, so you’re really not insecure,’ and that can cut the amount of work dramatically. LIke, three-quarters of the vulnerabilities those other tools report are false positives, so it really cuts the work. The last thing that we do is extending right into production. We work there as well. In production, Contrast prevents vulnerabilities from being exploited. We do this from inside the running application, but we can prevent SQL injection, cross-site scripting and express language injection, and a whole bunch of other classes of vulnerabilities, because we can actually observe them inside the running application. We’re not trying to guess whether they’re being attacked by looking at network traffic or HTTP requests or whatever. We’re actually watching the code run, seeing an exploit attempted, and preventing it from harming the application. So when you zoom out, we’re protecting the whole application process, from the first line of code all the way through production, all at the application layer. You still need to secure your operating system, your containers and your cloud environment. We don’t do that. We take care of the application layer. z — David Rubinstein n Sonatype Nexus IQ enables Nexus Firewall, which stops risky components from entering the development environment. From there, trusted components are stored in Nexus Repository, and can be easily distributed into the development process. Then, Nexus Lifecycle uses Nexus IQ to automatically and continuously identify and remediate OSS risks in all areas of an environment, including applications in production. n Sumo Logic simplifies DevSecOps implementation at the code level, enabling customers to build infrastructure to scale securely and quickly. This approach is required to maintain speed, agility and innovation while staying alert
for malicious cyber threats. n WhiteHat Security has been in the business of securing applications for 17 years. In that time, applications evolved and became the driving force of the digital business, but they’ve also remained the primary target of malicious hacks. The WhiteHat Application Security Platform is a cloud service that allows organizations to bridge the gap between security and development to deliver secure applications at the speed of business. Its software security solutions work across departments to provide fast turnaround times for Agile environments, near-zero false positives and precise remediation plans while reducing wasted time verifying vulnerabilities, threats and costs for faster deployment. z