Comcast – March 2020

Introducing DevSecOps at scale

02

Comcast: Introducing DevSecOps at scale

03

w w w.c o mc a stc o rpo ra t i on . com

COMCAST

Larry Maccherone, Distinguished Engineer of Comcast, discusses DevSecOps’ growing influence on one of the world’s biggest telecommunications companies

A

s a global leader in media and technology, Comcast is the parent organisation

of three primary businesses: Comcast Cable, NBCUniversal, and Sky. Comcast has more than 04

55 million subscribers, with Sky renowned as one of Europe’s leading entertainment companies operating in seven territories and Comcast Cable recognised as one of the biggest cable TV, high-speed internet, and phone providers in the United States. Sitting down in the new Comcast Technology Centre at its headquarters in Philadelphia, Pennsylvania, Larry Maccherone, Distinguished Engineer of Comcast Cable, shared how the company is uniquely positioned for success in their agile approach to achieving a DevSecOps cultural transformation.

Maccherone’s professional background heavily revolves around data analytics and Lean-Agile, and he started his first business while still an undergraduate at university. “I’ve been a serial

05

w w w.c o mc a stc o rpo ra t i on . com

Stop managing.

Start fixing.

The vulnerability management platform built for remediation

Handing you a solution Vulcan Cyber curates the best solution for any given vulnerability, whether it be a patch, configuration change or workaround

Initiating and orchestrating cross-team remediation workflows Vulcan Cyber helps establish cross-team alignment by managing and tracking all the steps in workflow from initial discovery to a successful remediation

Highlighting the organization’s biggest risks By prioritizing the vulnerabilities in your organization according to the true business-risk they pose to your environment, the platform minimizes workloads and lowers costs

Automating the grunt work

By orchestrating the cyber hygiene lifecycle, Vulcan Cyber makes the repetitive, most time consuming tasks automatic

Learn More

COMCAST

“ I believe that if you’re doing DevOps right, then the security part is just automatically included” entrepreneur throughout my entire — Larry Maccherone, Distinguished Engineer, Comcast

career. My first business had 80 employees and made US$20mn annually in sales,” explains Maccherone. “We were writing software that controlled a large portion of the world’s power generation,

08

and it meant that if hackers exploited a vulnerability in the software, then it potentially brought down the world’s power grid. We got really skilled at writing software that didn’t have exploitable vulnerabilities.”

DevSec CLICK TO WATCH

|

2:07

09 Upon joining Comcast in June

as empowered engineering teams

2016, Maccherone became responsi-

taking ownership of how their prod-

ble for overseeing the company’s

ucts perform in production, including

DevSecOps transformation. “I have

security. When you get development

a love/hate relationship with the

teams owning the problem, you

term DevSecOps. I believe that if

get a fundamental difference in

you’re doing DevOps right, then

decision making.”

the security part is automatically

Since its creation over a decade

included,” he explains. “You don’t

ago, DevOps has become a vital

call it DevTestOps or DevPlanningOps,

component of how companies oper-

it’s just DevOps. However, what

ate. Building upon the foundations

I do like about DevSecOps is the

of the agile movement, DevOps lev-

emphasis on security. My definition

erages automation, for quality and

of DevOps and DevSecOps is

security testing as well as for for-

essentially the same. I define both

merly manual deployment and w w w.c o mc a stc o rpo ra t i on . com

You change the world, we’ll secure it.

Secure sof tware is crucial in our digital world. And we believe in focusing on your security.

So you can conf idently create the sof tware that helps move our planet for ward.

So you can focus on your business.

So you can be the hero of your stor y.

Cox Automotive, one of the world’s largest automotive service providers, is pushing its industry forward with software.

Cox Automotive’s vision -- to transform the way the world buys, sells, owns and uses cars -- relies

Learn More

on software, and DevSecOps provides speed and security for its applications. By working with Veracode, Cox Automotive is realizing its vision, without being held back by security.

COMCAST

operations activities, in a bid to intro-

to achieve that level of trust,

duce software into production at

Maccherone introduced a trust algo-

speed. The primary goal of any

rithm. “The trust formula has three

DevSecOps initiative is to enable

terms combined in the numerator:

development teams to change their

credibility + reliability + empathy

mindset and adopt security practices

which are all divided by apparent

into their daily activities.

self-interest,” he explains. “It’s impor-

However, Maccherone believes

tant that the apparent self-interest

it’s impossible without healthy collab-

is as small as possible, with an

oration and mutual trust. In order

emphasis on shared interests.”

E X ECU T I VE P RO FI LE 12

Larry Maccherone Larry Maccherone is a Distinguished Engineer at Comcast where he currently leads the DevSecOps transformation initiative. Previously, Larry served as the Insights Product Line Director at Rally, where he published the largest ever study correlating development team practices with performance. Before that, Larry worked at Carnegie Mellon with the Software Engineering Institute (SEI) and CyLab conducting research on cybersecurity and software engineering. While there, he co-led the launch of the BuildSecurity-In initiative. He has also served as Principal Investigator for the NSA’s Code Assessment Methodology Project, on the Advisory Board for IARPA’s STONESOUP program, and as the Department of Energy’s Los Alamos National Labs Fellow. He speaks and publishes regularly on DevSecOps, Lean/Agile, and analytics and he is the primary author of a dozen open source projects, one of which gets 400,000 downloads per month.

13

Maccherone believes that under-

in its early stages back then and

standing and embracing each pillar

it’s fundamentally different now.

of the trust algorithm is vital to suc-

If you come into a meeting with those

cess in DevSecOps. “Credibility

old mindsets, make assumptions

means that you know what you’re

and use outdated terminology, then

talking about and it’s important that

the development team will pick up

you’re not just saying things for the

on that and you’ll lose credibility.

sake of it or repeating something you’ve read,” explains Maccherone.

“Writing code has

Reliability is the same regardless of the context; it’s the old business expectation of making and meeting commitments. It’s important to follow

changed a lot

through and do what you say you

in five years.

are going to do. Finally, empathy

DevOps was

is all about how much compassion w w w.c o mc a stc o rpo ra t i on . com

Time is precious.

Automate Vulnerability Remediation

IDENTIFY COMPONENT

DETECT

VULNERABILITY

ENFORCE POLICY

AUTOREMEDIATE

Learn more about us at whitesourcesoftware.com

“ If you come into a meeting with those old mindsets, make assumptions and use outdated terminology, then the development team will pick up on that and you’ll lose credibility” — Larry Maccherone, Distinguished Engineer, Comcast

supersedes all of those challenges and give the reasons why. It’s vital that you aren’t dictating them.” The

you show, and the awareness of

importance of coaching rather than

how challenging something is.”

policing is a key aspect of Comcast’s

Following the foundation of the

strategy. The company also has

trust algorithm, Maccherone believes that it has successfully allowed for increased efficiency and has ultimately meant better decisions.

“Lots of security groups at other large companies spend an inordinate amount of time cajoling development teams to do things,” he says. “The reason they have to spend such a considerable amount of time policing is due to a lack of trust. Showing empathy is crucial and it’s important to acknowledge how difficult something is to do. However, it’s also fundamental to explain why you’re trying to make the case that this risk w w w.c o mc a stc o rpo ra t i on . com

15

50m lines of code driving the kids home The more powerful you make software, the more inseparable DevOps and security need to be.

See why Checkmarx leads the industry in delivering automated security scanning for DevOps.

Checkmarx.com

WORKING@COMCAST CLICK TO WATCH

|

1:08

17 a programme in place that provides

what we want; we want them to reach

immediate feedback to the develop-

out and partner with us.”

ment team while also providing

Change management is a key

aggregated metrics to guide coach-

driver to Maccherone and Comcast’s

ing efforts. “We created a workshop

strategy. “The traditional way of

where we sit down with the develop-

gathering a response was to produce

ment team, walk through the trust

surveys. However, we found that

formula and the company’s

the behaviour didn’t change,” he says.

DevSecOps practices and give

“We decided on a framework that

them a chance to internalise what

we can coach from and enable the

that practice means,” explains

developers to reflect on whether

Maccherone. “When someone feels

or not they meet the criteria. If we

like they’re being forced into out-of-

send an email to them then we get

context practices, their natural

almost no response. However, if

reaction is to avoid them. That isn’t

we sit with them and allow them w w w.c o mc a stc o rpo ra t i on . com

to ask questions directly then they instantly start changing their behaviour.”

With any successful transformation comes the challenge of recruiting and retaining top talent, and Maccherone believes it’s the most challenging part of any business.

“It’s the key to any tech company,”

1963

Year founded

$108.9bn Revenue in US dollars (2019)

affirms Maccherone. “The HR department that we have at Comcast is fantastic. They really understand the importance of exceptional talent.

Candidates want to have work that is interesting, fun and challenging,

190,000 Number of employees

w w w.c o mc a stc o rpo ra t i on . com

19

COMCAST

20

“ Three years ago, I started a Google alert on DevSecOps and would get one hit a week or even a month.

Now, I get 10-20 every day” — Larry Maccherone, Distinguished Engineer, Comcast

21

w w w.c o mc a stc o rpo ra t i on . com

LEARN MORE

Comcast Partners CLICK TO WATCH

|

1:01

23 in addition to working with peers

are trying to design their products

they respect.”

to fit in with the direction we’re going.”

In a bid to achieve mutual success,

Maccherone believes that without

Comcast Cable has established

developing such robust and long-

a number of key partnerships, such

standing partnerships, the challenge

as with WhiteSource, Vulcan Cyber,

of reaching the level of success

Checkmarx, Go2Group, Contrast

Comcast has achieved would have

Security, Synopsys, Bugcrowd and

been significantly harder. “Our vendors

Veracode. Maccherone recognises

are a key to our success and we’re

the value of forming strategic, business

extremely excited and happy with

relationships in order to realise long-

the current set we have,” beams

term success. “We’re at the forefront

Maccherone. “They align well with

of DevSecOps, and lots of our vendors

our values and that’s been the differen-

see that,” says Maccherone. “We’re

tiator to finding ways to reduce our

constantly searching for vendors that

security risk.” w w w.c o mc a stc o rpo ra t i on . com

Integrate security into your DevOps environment code

deploy

test

se ea re l

Ops

operate

build

an

pl

Dev

m o n ito r

Automate application security testing with the development and deployment tools you use today.

Read why Gartner selected us as a leader for Application Security Testing

“ When someone feels like they’re being policed, their natural reaction is to avoid the police. That isn’t what we want; we want them to reach out and partner with us” — Larry Maccherone, Distinguished Engineer, Comcast DevSecOps has become a 25

hot topic in the technology space in recent years and Maccherone has observed its rapid rise first-hand. “Three years ago, I started a Google alert on DevSecOps and would get one hit a week or even a month,” he says. “Now, I get 10-20 every day and we’re not even at the steepest part of the adoption curve for DevSecOps yet.” In 2019, Comcast’s goal was to scale the DevSecOps programme, the tech giant achieved that by tripling the number w w w.c o mc a stc o rpo ra t i on . com

27

of teams onboarded to the pro-

we reach that saturation point,

gramme. “By the end of 2020,

I anticipate that we’ll add more

we aim to double that number again,

capability, tools and practices over

and I expect that will get us close

the next few years.”

to the saturation point of all the teams at Comcast. We’ve gone from essentially launching the programme to evolving, optimising and scaling it to the point of saturation. After w w w.c o mc a stc o rpo ra t i on . com

1701 John F Kennedy Blvd.

Philadelphia PA USA ComcastCorporation.com