THE FIRST OF THREE PARTS
CI/CD Pipelines are Expanding
SPONSORED BY
www.sdtimes.com
January 2020
SD Times
BY LISA MORGAN
CI/CD pipelines are evolving as organizations identify opportunities to improve release velocity and as the industry considers what CI/CD pipelines should look like in the first place.
A
malgam Insights recently released “The 2020 Guide to Continuous Integration and Continuous Delivery: Process, Projects and Products.” In it, report author and research fellow Tom Petrocelli explains what basic and extended CI/CD pipelines involve. The “basic” CI/CD pipeline includes five processes, which are: merge, build, test, package and deploy. All of these are individually defined so readers have a common reference point. The basic pipeline includes sub-pipelines associated with each step, such as moving artifacts from a build into a repository. An extended pipeline has twice as many processes (10 total). It includes plan, code, merge, build, test, security scan, package, artifact repository, deploy, and monitor. Although the flow in both diagrams appears linear (and the report notes that the term “pipeline” suggests a linear process), loopbacks to earlier points in the workflow can occur when failures happen or a security scan indicates a threat. “The standard pipeline is just the basic stuff. Once my code has been written, how do I get it to release? There are other non-tooling stops along the way because some pipelines will deposit you in UAT or some kind of production staging area but overall the idea is to get from point A to point B,” said Petrocelli.
“When you think about an extended tool chain — which is where a lot of the vendors are going — you start to bring in other things like monitoring.” By extending the pipeline, teams are able to connect and automate more of the total software development process, though there may still be manual steps along the way. “Right now, CI/CD really addresses one small piece of the total software development process, but there’s other stuff that needs to be integrated into it,” said Petrocelli. Hasan Yasar has a similar view. He’s technical manager of the Secure Lifecycle Solutions group in the CERT Division of the Software Engineering Institute at Carnegie Mellon University. “The point of a pipeline is basically to help build an application, test the application, deliver the application, and deploy it into production. A pipeline is not just having continuous integration servers or deployment tools and techniques, but covering the full life cycle,” said Yasar. By full life cycle he means from the inception phase through build, test, delivery and deployment, as well as monitoring the production environment.
As far as tooling, Yasar said a pipeline should include an issue tracking system, a build system, an integration environment, a communications system such as chat, and monitoring systems that can monitor the progress of various development environments. In the center of all that should be a repository management system. That system should be connected to all the other systems and include all artifacts including infrastructure as code. The central repository helps facilitate communication and collaboration and ensures that all stakeholders have access to the artifacts. All the tools should be integrated, of course.
Whose tools? End-to-end solutions are emerging alongside the point solutions. The Amalgam Insights report includes a representative sample of some of the vendors moving toward end-to-end solutions. They include Anaxi, Atlassian, AWS CodePipeline, CircleCI, CloudBees, GitHub, GitLab, Google, Oracle, Pivotal and Red Hat. The end-to-end solutions are integrated, which saves customers the hassle of cobbling together point solutions. “Complexity works against you when continued on page 5 >
3
4
SD Times
January 2020
www.sdtimes.com
The Continuous Delivery Foundation advances CI/CD BY LISA MORGAN
More organizations have matured from CI to CI/CD, but their paths differ as do their pipelines and results. Most enterprises are implementing a mix of open source, commercial and even home-grown tools, and they’re looking for answers. One place to look is the Continuous Delivery Foundation (CDF) which is home to many of the fastestgrowing CI/CD open-source projects. The CDF fosters vendor-neutral collaboration among developers, end users and vendors to further best practices and industry specifications. DeployHub CEO and co-founder Tracy Ragan Tracy Ragan, who serves as the CDF general membership board representative, provides additional insight in now for the CI/CD community to have a this Q&A. conversation about what CI/CD looks like today (terms and definitions) and SD Times: Why was the CD where it is headed tomorrow (KuberFoundation formed? netes pipelines, AI/MLOps). Ragan: We are on the top of a swell that is turning into a massive tsunami What does the organization hope when it comes to software development. to accomplish? According to “The Future of EmployWe have 9 strategic goals: ment: How Susceptible Are Jobs to 1) Drive adoption Computerisation?” by Carl Benedikt 2) Cultivate growth of open-source projFrey and Michael A Osborne, 47% of ects U.S. jobs may be replaced by AI. If this is 3) Foster tool interoperability true, we have lots of software to develop 4) Champion diversity and inclusion over the course of the next 20 years and 5) Foster community relationships our current CI/CD process will not be 6) Grow the member base able to sustain this massive amount of software development and management. 7) Create value for all members For most organizations, CI/CD is a work- 8) Promote security as a first-class CI/CD citizen flow orchestration tool that calls scripts. In most cases, the scripts do the heavy 9) Expand into emerging tech areas. lifting of the movement of code through the process. In order to build the softIn what ways will the CD Foundaware of tomorrow, CI/CD must do bet- tion help organizations improve the ter, be more automated and include efficiencies and effectiveness of their more than just “check-in and build.” CI/CD practices? And we are moving from monolithic softThe CD Foundation provides a platware to microservices. This directly form and thought leadership communiimpacts the CI/CD process. The time is ty for driving CI/CD to the next level.
The CD Foundation’s job is not to produce a CI/CD “stack” or best practices guidelines. CD is too broad to have a single solution for all. The CD Foundation’s job is to bring members together to achieve our 9 strategic goals and provide a vendorneutral platform for open-source tools that fit into the CI/CD landscape. What impact might the initial and future projects have on CI/CD tools and tool chains going forward? While it’s hard to predict the impact today, we are beginning to see more productive discussions between projects. Jenkins and Spinnaker are working to define how they interoperate. There is community discussion around Tekton and JenkinsX. Most organizations will have a variety of different pipeline orchestration tools (Jenkins, Tekton, CircleCI, Bamboo, Spinnaker, etc). And most companies will allow individual teams to decide on what those orchestration tools will call. Not all teams will look the same. A mix of open source and commercial will continue to be the way new tools are adopted and implemented. What the CD Foundation can offer is a platform for managing the open-source tools and foster discussion between the teams. Our third most important goal is to “foster tool interoperability.” If we are successful, we will have achieved the ability for one orchestration tool to call another. Developers might use Jenkins while Production Control is using Spinnaker. What does the future of CI/CD look like? In the future, CI/CD will grow in importance as developers are pushed to create more software. We now talk about moving software through the lifecycle faster, tomorrow’s discussion will
www.sdtimes.com
be around sustaining the number of new applications hitting the market. So fast is important and CI/CD is a key player in ‘faster’ and ‘more’. From a lower level, if we look at microservices, our CI/CD process changes. First off, version control becomes less critical. We will not have source code that is several thousands of lines of code long that requires branching and merging to allow multiple developers to work on it at the same time. We will instead have code that is hundreds of lines long at max. When you think of microservices, you must think functions. Second, compiling code may not be required. While Go is compiled, Python is not. And even if it is compiled, it will be quick — no more long builds and no linking. Third, microservices are “loosely coupled.” This means no linking at build time. APIs at runtime replace a build “link” step. The concept of an “application” built all at once goes away. Our CI/CD pipelines will be managing thousands of individual microservice workflows. Today we generally have one workflow per application; tomorrow we will have one workflow per microservice. Deployments will always be incremental. Deploying a full application will no longer happen. A new microservice will be updated, creating a new version of the “logical” application. An application is just a collection of services. And finally, configuration management will be needed to map a collection of microservice versions back to a logical view of an application version and to track which applications are using which service. So yes, lots of changes are coming our way, and the CD Foundation is here just in time to help lead the discussion, manage new open-source projects and inspire tool interoperability. z
< continued from page 3
you want to achieve higher velocity,” said Amalgam Insights’ Petrocelli. “But the broader effect is without these tools, it’s going to be hard to have a microservices architecture. You’re not going to get what you want to get out of it, which is faster and more frequent updates. You’re not going to be able to do those continuous deployments without automated tools.” “Microservices” is the operative word driving the need for CI/CD. Without that, high levels of automation may not be necessary, particularly if the application doesn’t require frequent changes. The age-old choice between point solutions and end-to-end solutions is the same for CI/CD as it has been for other types of tools: depth of features versus breadth of features. The latter is easier to implement because all of the pieces are designed to work together. However, depending on the complexity of the application and the environment, it may be preferable to implement “best of breed” point solutions.
January 2020
SD Times
chain. Many of them aren’t running integrated DevOps teams,” said Petrocelli. “If you’re not doing the management piece, you’re not doing DevOps.” The Amalgam Insights report includes a section entitled “DevOps vs CI/CD” to help alleviate some of the confusion between the two. DevOps is deemed a project and organizational strategy. Since it’s conceptual, there’s no fixed implementation. CI/CD is defined as process automation, specifically “the process of taking raw developer code and other build artifacts and turning them into running applications and services.” “When it comes down to implementation, DevOps is management and organization. It’s not tools. There’s no such thing as a DevOps tool chain,” said Petrocelli. “You can do DevOps and not have CI/CD tool chains in place and you can have CI/CD tool chains in place and not be doing real DevOps or DevOps at all.” The report also explains the individual elements of CI/CD: continuous integration, continuous delivery and continuous
“Right now, CI/CD really addresses one small piece of the total software development process, but there’s other stuff that needs to be integrated into it.” —Tom Petrocelli Whatever the application mix is in an organization, it’s wise to procure CI/CD tooling with the future in mind so it can support the breadth of what exists, including traditional and new types of applications. “What you don’t want to do is create tensions in your own organization because some people are getting the advantages of automated tools and others aren’t,” said Petrocelli.
DevOps and CI/CD: Real or not? “DevOps” and “CI/CD” mean different things to different people. Many organizations claim to be doing one, the other, or both, which is not always the case. “A lot of teams call themselves ‘DevOps’ when they’re really doing a tool
deployment. Most people associate continuous deployment with the digital native disrupters that are doing hundreds or thousands of releases per day. Continuous delivery is what’s practical for everyone else. Two things that distinguish continuous delivery from continuous deployment are release velocity and the level of automation, which go hand-in-hand. According to the Amalgam Insights report, continuous delivery involves manual while continuous deployment automates deployment. What organizations call their practices matters less than the results they’re deriving from it. However, it may be harder to achieve the results desired when it’s unclear what the organization is trying to achieve in the first place. z
5
6
SD Times
January 2020
www.sdtimes.com
How some organizations are implementing CI/CD
BY LISA MORGAN
CI/CD implementations and the impetus for them varies among companies, but everyone wants to ensure faster delivery of high-quality software. Following are three examples of companies that have adopted CI/CD in their own way.
Lucidchart improves productivity About four years ago, diagramming solution provider Lucidchart had 30 developers. The team was “running into issues” because not everyone was aware of the changes going into production. “We often spent a lot of time tracking down the individual who made code changes that went out with a given release so that we could try to piece together what happened and how best to fix it,” said David Torgerson, director of DevOps at Lucidchart. “Our QA team was having trouble scaling the testing that needed to be done because of all of the different combinations of manual tests that needed to be done.” The goal was to achieve a regular release cycle, but because they were unable to track down which change caused a problem, releases were “terrifying.” So, they had a meeting following one release to discuss what when wrong. Out of that meeting came the idea of “emergency releases” (aka bug fixes). However, the result was meetings about the emergency releases they had to do because of the problems with the original release. Worse, because there was no classification for a businessrelated change, they created a third category of release called
“business release,” which led to meetings before release meetings just to determine whether the release was an emergency release or a business release. Eventually, Torgerson and other technical leads realized that the time spent in meetings would be better spent on iterations, so they started moving toward a CI/CD process, albeit slowly because everyone wasn’t ready for the “mindset shift” required. That first meeting devolved from a discussion to four concurrent arguments. At the second meeting, they agreed that their definition of continuous deployment was “as frequently as possible.” “If we were looking strictly to be a purist CI/CD shop and appear as a DevOps shop, we would probably do things differently, but our approach has been to improve developer productivity and increase production stability.” The QA team hated the idea of CI/CD because they didn’t want the manual regression tests they were doing automated. CI/CD seemed like an existential threat. Torgerson said they’re now the biggest CI/CD proponents because they’re having more fun trying to break the product than they ever had doing manual regression tests. z
Negotiatus perfects manual processes, then automates Spend management software provider Negotiatus has grown from 2 to 45 employees since it was founded in 2016. It wanted to improve developer speed by streamlining the onboarding process and improving the reliability of code. They also wanted to improve the speed and accuracy of the deployment process because it had become clear that when the deployment process is manual, it’s easy to forget to run a specific command. What they didn’t want to do was automate processes that weren’t optimized to begin with. “We try to solve as much as we can manually first, but in an extremely structured way. We basically document a process, run the same exact steps every single time, and ensure people are following them so we can identify any gaps in the process and fill them in,” said Negotiatus CTO Tom Jaklitsch. “The whole thing would run for a month or two and we’d iterate on it.” Another area targeted for improvement was test coverage because it was less than 30% on a monolithic application at the
time. Now, it’s more than 75%. They also made sure all new services were above 95%. Developers are now required to run their own QA, which has improved code quality and reduced the amount of back-andforth between developers and QA. Deployment now happens about 15 times a day, with each deployment taking about 25 minutes. Of the 25 minutes, developers spend one or two minutes at the halfway point making sure everything looks fine. Then, the developer clicks a button and the rest is automated. “It’s better to be safe than sorry versus automating every step of the process,” said Jaklitsch. “I’d love to get the process down to 10 to 15 minutes, but as of right now, once you kick off and deploy, it kicks off all these automatic processes.” The deploy ticket queue is populated and managed in Slack with the last person adding their release to the end of the list. That process is being optimized now, Jaklitsch said. z
Stealth Communications modernizes its application and processes Internet service provider Stealth Communications was saddled with a legacy business application built in 2009 that was written in C++ and PHP. It had 1.6 million lines of code and took two years to build. The application’s architecture made it extremely difficult to implement CI/CD, so about a year ago, the development team started rewriting it as a microservices application so they could implement CI/CD. More importantly, the new architecture would enable easier upgrades to the UI and easier implementa-
tion of new features without massive, infrequent overhauls. “We want to release features in a real-time fashion without having the user reinstall it or do these upgrade paths that might be intensive or error prone,” said Shrihari Pandit, CEO of Stealth Communications. “Instead of a single stack application that controls everything, we have mini applications that plug in so we can easily take advantage of them. CI/CD is extremely crucial because that’s how you’re going to react to moving markets.” z
www.sdtimes.com
CI/CD success requires a sound approach There’s considerable confusion about “the best way” to approach CI/CD when no single path exists. There are important considerations organizations should contemplate to avoid wasting time and money that could have been spent making progress, however. “One of the first things an organization should do is understand what their needs are [in terms of] the business, their application and the application environment,” said Alan Zucker, founding principal of training and consulting firm Project Management Essentials. “The other thing you should do when looking at your CI/CD pipeline needs is figure out what you’re going to get the biggest benefit from, your automation or integration work or where you pain points are. Then start stitching things together.” Clearly, progress is always the goal, but sometimes organizations come up short. Following are some of the “gotchas” that can get in the way. n COMPLEXITY CI/CD tool chains tend to be complex because CI/CD involves so many processes and associated types of tools. The resulting complexity can be difficult, time-consuming and expensive to maintain over time. “Now you’ve got a whole group of people whose only job is to maintain the tool chain, which was supposed to keep you from having all these people in your organization,” said Tom Petrocelli, research fellow for DevOps and Collaboration at Amalgam Insights. “If you’re a midsize company, you’re not going to have the budget to hire all the engineers to do that and maintain those engineers over time. The result is modest tool chains that don’t have the impact the organization expected or the cost prevents them from doing anything.” The emerging end-to-end solutions alleviate the tool management overhead since all elements are integrated. However, highly complex applications running in highly complex environments may require point solutions that have a greater depth of features. “I think a successful pipeline really simplifies complex operations. Your pipeline may grow, so it becomes more complex in all of the different aspects of what it’s doing, but it needs to be simple, straightfor-
ward and easy to work with,” said Daniel Ritchie, distinguished engineer at Broadridge Financial Solutions. n SECURITY Security needs to be part of a CI/CD pipeline; however, a lot of times “security” is limited to just a quick CVE scan. Alternatively, a security scan may be left until the end. If a vulnerability is found, developers may be more inclined to file an exception request so they can fix it in the next release rather than spending the time to fix it in the current release. “Security should be everywhere, not just before or after integration. A lot of organizations think of security as just static analysis or dynamic analysis,” said Hasan Yasar, technical manager of the Secure
January 2020
SD Times
testing or they don’t understand the value of the testing they’ve done. Security should have threat modeling or understanding of security requirements at the beginning of feature development.” Don’t think of security as a step, but rather a process that should be integrated throughout the pipeline. n CULTURE Some people are really excited about CI/CD because they want to get more done faster, but not everyone. CI/CD automates a lot of manual tasks that have been associated with job descriptions. “Organization and culture are the biggest barriers. You should have accountability at the lowest responsible level so people are taking ownership for their work and people have that end-to-end responsibility,” said Project Management Essentials’ Zucker. “[Companies are] trying to acquire new methodologies and processes and spend a lot of money training people in the same old tools, but then they still behave the same old way and they don’t get the
‘A lot of organizations think of security as just static analysis or dynamic analysis.’ —Hasan Yasar
Lifecycle Solutions group in the CERT Division of the Software Engineering Institute at Carnegie Mellon University. “Everywhere” means various types of security testing throughout the SDLC, beginning with defining security requirements along with functional requirements. Yasar recommends including tools like Fortify or SonarQube as part of the CI server. Depending on the application, other security considerations may include authentication, network segmentations, regulatory compliance (such as PII usage) and the like, so security testing is happening at the code level and the architectural level. “Almost every organization is struggling with false positives or how much security testing they need to do because static and dynamic analysis such as pen testing take time,” said Yasar. “If the organization is only doing static and dynamic analysis without tying them into the requirements in the beginning, they don’t know what they’re
results they want.” Having the people affected by CI/CD involved in its planning, problem-solving and implementation can help break down barriers to adoption. n UNREALISTIC EXPECTATIONS It’s easy to expect too much too soon. Buying tools alone won’t help, nor will automating processes that weren’t designed well in the first place. The space itself is evolving, and as a process, organizations are wise to approach CI/CD from the perspective of continuous improvement, since they’ll discover many things that could along the way anyway. Trying to do too much too quickly can backfire. “I would not counsel anybody to go from a well-controlled annual process to pushbutton full release management. You’re talking about a several-year journey with a lot of fits and starts,” said Zucker. z — Lisa Morgan
7