The Colo and Cloud supplement by DCD Magazine

Sponsored by

Contents 4. T he rebirth of colocation Cloud adopters are returning to colo - with a new set of demands 8. A dvertorial: How to tackle environmental impacts and meet sustainability goals 10. Why colos need to get serious about security Colo providers need to adapt to new threats in a changing landscape 14. H ow Dropbox pulled off its hybrid cloud transition We explore Magic Pocket, and whether others could do the same

4 34 10

Born-again colocation

he public cloud is a very good idea, but it's been hugely oversold. That's why we're now seeing a mass movement, as early adopters return to colocation.

But they don't want traditional colocation services. They've been changed by their cloud journey, and want more.

Onramps and metal There are many reasons why public cloud has lost its luster for early adopters. Among others, it turns out that renting virtual instances by the hour can work out expensive. Ten years ago, we expected it would be possible to do everything in AWS, Azure, or Google Cloud. Now it's obvious that there are many things that don't make sense in the public cloud. But there are still some things that are perfect for AWS or the others. Those things that lured us into public cloud are still good. So this isn't an exodus from public cloud. It's a movement to combine colocation with those services we want to keep in public cloud. That means two things: onramps that can connect the new in-house instances with those in the cloud, and bare metal servers, which offer some of the flexibility of cloud resources, with the privacy and ownership which some applications need. All of which means that colocation companies must step up and invest to meet the new demands. That changes the dynamics, and could prompt yet more consolidation amongst colocation

providers, because not everyone can add all the necessary features to a small local portfolio of data centers space. The future of colocation is not fixed, and we expect a lot more changes (p4).

Security blanket With new technology comes new security risks. It has always been that way, and the new colocation world is no exception. As new kinds of ransomware strike, colocation providers will be well outside their comfort zone. The security arms race will always continue, and we look at the next steps in keeping up (p10).

Magic Pocket One widely-publicized example of a company where the cloud lost its initial charms is Dropbox. Fast growth in its online file sharing business left the company addicted to Amazon's S3 storage service, with costs potentially out of control. The answer was Magic Pocket, one of the world's biggest data migrations, that put Dropbox back in control of its storage, and allowed it to introduce better technologies at a pace it wanted (p14). But the story really is a prime example of the New Colo, because Dropbox has not made a total, oneway migration. If times change, Dropbox can move data in the opposite direction. If Amazon ever offers a better deal, lead developer Preslav Le told us, Dropbox's data can go right back there. As everyone eventually discovers, there is no single right answer.

Colo to Cloud Supplement 23

The rebirth of colocation Early cloud adopters are coming back to colocation services. But the born-again colo customers are very different, and providers face completely new challenges

or some years, there’s been a school of thought that colocation is out of date, and will eventually wither away in favor of the cloud. But that idea runs counter to the facts. The colo market is stubbornly growing. But it’s not the same market. Early cloud adopters are partially returning to colocation - and these born-again colo users are very different to the old school. It’s been fashionable to see the cloud as an all-consuming future. The cloud can handle massive workloads, services are easy to buy, and are scalable. So why would anyone go to the trouble of buying racks and servers and installing them in retail colocation space? Surely you should let the cloud handle the grunt work, and get on with your real job! Market figures tell a different story. Averaging out forecasts from a bunch of providers, it seems the colocation market as a whole is growing massively, at around 16 percent per year. Over the next ten years, that adds up to a market that will quadruple in size, going from roughly $46 million in 2020, to $200 billion in 2030. Market researchers say the retail colocation sector is bigger than wholesale colocation, where whole data centers are rented by large operators - and retail colo will keep its lead at least till 2030. What’s going on? Cloud is massive - and efficient First off, it’s more complicated than that. Cloud data centers really are massive because, alongside the ones leased in wholesale colo deals, hyperscalers own a massive number of sites, which they’ve built themselves. These are huge beasts, with power demands up to 1,000MW. “They’re dominating the market today,” says Yuval Bachar, a hyperscale veteran with stints at Microsoft Azure, Facebook, Cisco, and LinkedIn. “These mega data centers actually account for about 70 percent of the data center business in the world - from the power consumption as

Peter Judge Global Editor

“The public cloud as we know it has been around for 12 years, right? Everyone sees the growth, everybody sees people going pure cloud, and just running to the cloud, drinking the Kool-Aid" well as from a floor space perspective.” But hyperscale includes some behemoths which are actually giant in-house IT services, like Facebook, Bachar points out: “Facebook is probably one of the biggest data center operators in the world nowadays. But they're serving their own enterprise needs. They're not a public cloud service - they're running their own internal cloud.” Bachar says hyperscale cloud data centers do indeed have a big advantage over other sectors, in their ability to deliver cheap IT power: “These sites are usually located in remote areas where the land is inexpensive, and power is available from multiple green sources.” If those sites don’t have connectivity, the hyperscalers have the muscle to provide it: “The large companies who are building those mega data centers need to bring connectivity into those sites and be creative to create the network backbone. And each and every one of them is creating their own backbone.” On these sites, hyperscalers “start with one or two buildings, and then expand in a replication mode, on the same site,” Bachar says. “They create a very high level of efficiency operating the data center with a PUE of 1.06 to 1.1.” In his view, the hyperscalers are “creating a very, very significant level of green data centers.” Colocation has challenges Smaller colocation sites are very different, he says. They were set up to host physical servers owned by enterprises which “decided not to actually build their own data center but actually to put part of their

4 DCD Supplement • datacenterdynamics.com

IT load into a colocation site. “These are small sites between 50 and 75MW, and in some cases can be even smaller than 15MW. They are closer to urban areas - because historically those sites actually have been put closer to the headquarters of their customers.” These colo providers have big challenges, says Bachar: “These buildings are not scalable. Because they're sitting in urban areas, the size they have been built to this the size they're actually going to operate under for the remainder of their life. They don't have expansion space.“ A second challenge is, “they are heavily regulated - because the closer you get to the middle of the city, the heavier you are regulated for emissions, power availability and every aspect that impacts the environment around you.” So the odds are stacked against smaller colocation companies. But their market share resolutely refuses to decrease and there’s a surprising reason for this. According to Greg Moss, a partner at cloud advisory firm Upstack, large numbers of early cloud adopters are moving capacity out of the cloud. Cloud defectors come back to colo “The public cloud as we know it has been around for 12 years, right? I mean, the big three - GCP, Azure, and AWS. Everyone sees the growth, everybody sees people going pure cloud, and just running to the cloud kind of drinking the Kool-Aid. What they don’t realize is there's two sides to that coin.” According to Moss, the early adopters, the “sexy, innovative” companies who

Colocation Reborn went all-in on the cloud twelve years ago, “are now at a point where they're pulling out at least a portion of their environment, it could be 20 percent, it could be 80 percent, and hybridizing, because what they've realized over the last 12 years, that cloud isn't perfect. To really get the efficiencies from an economic and technical perspective, you really need to be in some sort of hybrid environment.” Companies started with a “knee jerk reaction” to put everything in AWS, he says: “Why? Because some board member mandated it, or because our competitors are doing it, or because it's the rage right now.” Later on it goes sour, because in a lot of cases, renting capacity on demand costs a lot more than owning the hardware: “Someone's losing their job, because they realize they're spending 30 percent more than they were - and the whole exercise was around cost reduction and innovation!”

would have liked to pull out a portion of their environment six years ago, but they can't because they have no headcount. There's a big deficit in the industry for talent.” And there’s company politics: “There’s a person who's been there 15 years, who just doesn't want to do more than what he's doing. He picks up his kid every day at school at three, and he knows that if the IT sits in AWS, he can continue to do his job and leave at three and pick up his kid. He could be the gatekeeper. “I've seen large companies dismiss $50 million a year savings because the gatekeeper, a $150,000 employee, just

doesn't let the management know that there's an opportunity.” Sooner or later, those early adopters can get past the gatekeepers, and start shifting the balance of their IT provision towards a hybrid model with some loads returning to colocation. But these customers are a new generation, and they will want more than just the resilient racks with power and networking, that were good enough in days gone by. Born-again colo needs: bare metal and cloud onramp “You can't just have great resiliency, you have to have a total solution. That

The trouble with cloud It turns out that going to the cloud isn’t a simple answer to all questions: “It doesn't solve anything. It just hands your data center environment to a different company. If the data center just went away, and is miraculously living in the ozone, then fine. But it's not. You're just shifting infrastructure around in a different billable model. It makes sense: some people want to consume hardware in a day to day or hour by hour function.” The hyperscale cloud operators can afford to lose some custom, says Moss, because they still have massive growth due to the late adopters: “AWS, GCP, and Azure are still seeing so much growth right now, because of healthcare, because of not-for-profit, because of legal, because of all the non-sexy companies that are just now getting comfortable enough to move to the cloud.” But the early adopters really aren’t happy - and they have problems: “They're stuck for five to 10 years, because no one's going to pull out of a massive migration or massive decision after just doing it regardless of the outcome. So that's why the early adopters are now exiting. Finally! After 10 or 12 years.” But it’s still not easy: “They probably

“I've seen large companies dismiss $50 million a year savings because the gatekeeper, a $150,000 employee, just doesn't let the management know that there's an opportunity" Colo to Cloud Supplement 5

means big buckets - a data center that's resilient. And some sort of bare metal or custom managed component, like Equinix Metal for instance. And then there's the connectivity to the large public clouds through a partner like Megaport or a direct onramp. Those are the three components that make up hybridization.” The capacity speaks for itself, while bare metal is a way to own dedicated capacity in someone else’s infrastructure. Customers can need this to meet privacy rules which require customer data to have a specific location away from shared hardware. And the need for on-ramps to the public cloud is obvious. If customers are building hybrid clouds that include public cloud services as well as their own colocated servers, there should be easy to use links between the two. Unlike the early cloud enthusiasts, the born-again colocation customers are thinking ahead, says Moss. Privacy rules might force some loads onto bare metal in future. Or they might open up a new commerce branch which would have seasonal peaks - and that could require a quick link to the cloud. They’re thinking ahead because of the trouble they’re experiencing coming off their cloud addiction, but also because, if they pick the wrong colo, they could have to move all their IT. And, as Moss says, “nobody wants to move a data center. It's the biggest pain in the ass.” There are companies that will physically move racks of servers from one facility to another, but Moss says: “They charge $5,000 in insurance for every million dollars in hardware, even if you're moving three blocks away. If you move $10 million worth of hardware, your insurance cost is going to be upwards of $50,000. And will they even turn back on?” Power and networking According to Bachar, the new colo customers have another demand: they are much more power-hungry: “If we look at the technologies in the mega data centers and the colos, 80 percent of the IT load is compute and storage servers now. We're

“Nobody wants to move a data center. It's the biggest pain in the ass”

"If we look at the technologies in the mega data centers and the colos, 80 percent of the IT load is compute and storage servers now" starting to see the emergence of AI and GPU servers, which are growing at a much faster pace than the compute and storage servers, and specialty storage servers going hand in hand with the GPUs and AI. “And the reason for that is that we're starting to deal with very large data sets. And to process those very large data sets, we need a server, which is beyond the standard compute server.” But GPU servers, and GPUs integrated standard compute servers demand more power: “Those high power servers are challenging our infrastructure. If you look at a typical high-end GPU server, like the ones from Nvidia, these servers are running between 6000W and 8000W watts for every six rack units (RU). That is very difficult to fit into a standard colocation where the average power per rack is 6kW to 8kW.” On those figures, a standard rack is 42 RU, so a full rack of GPU servers could demand a sevenfold increase in power. One thing which would help is more flexibility: “Am I taking a high power rack or a low power rack? Can I actually mix technology within the rack. We need a very flexible capability in the data centers.” New apps also need more network bandwidth, says Bachar: “Networking today is 100 and 400 Gigabit Ethernet as a baseline. We will continue to grow this to 800G and the 1.2Tbits in the future.” Can small colos cope? All these changes are placing huge demands on small colocation firms, while there’s a surge in demand for what they provide, and that is a big factor driving the current surge in colocation mergers and acquisitions, says Moss. Smaller colos realize that they can’t actually fund all the changes they need to be truly successful: “So you see a lot of these smaller data centers selling off to the larger guys.” Meanwhile, he says: “The larger guys are buying them because it speeds their go-to-market - because the infrastructure is already in place. It takes a long time to build a data center. You could probably get away with a brownfield build in the US within 18 months. If it's Greenfield, it's more likely in three years. A lot of requests are on a shorter

DCD Supplement • datacenterdynamics.com

timescale than that: “Imagine you are Equinix, you have three data centers in a market and they're all bursting at the seams. You have very little inventory left. But one of your largest customers, or an RFP from a new customer, says ‘In 12 months, we're going to need a megawatt and a half.’ But you can't build in that time.” In that situation, the large player can buy a smaller regional player, whose data center is only 30 percent full, and put that customer in there. “You invest some money in upgrades, you bring it up to standards, and you get certain certificates that aren't there, and you now have an anchor tenant, and maybe the facility is 60 percent full,” says Moss. “The bank loves it, because the bank takes on the existing customer leases to finance, and they also take the new signature tenant lease, that's probably 10 years long.” The other customers are happy too, as the data center gets a perhaps-overdue facelift, along with the addition of those new must-have features, bare metal services and on-ramps. The odds are on big colo players Small colo players often rail against giants like Equinix or Digital Realty (DRT), claiming they overcharge for basics like power and cooling, as well as services like cross-connects - links between two servers in the network. It’s very cheap for a large colo to activate a network link between two of its customers, who may even be in the same building - and yet customers are charged a high price for those crossconnects. Multinationals don’t see that as a problem, says Moss: “A company like Equinix or DRT has everything that you would need to be successful. You are going to pay a premium, but that premium, if utilized properly, isn't really a premium. If I'm using Equinix in three countries, I may be paying 30 percent more in space and power, but I'm saving a hell of a lot of money in my replication costs across those three data centers because I'm riding on their fabric. “A local 200 person business in Pennsylvania, whose network engineer wants to touch every part of the hardware, is going to TierPoint, because it's two miles down the road,” he says. “He doesn't have

Colocation Reborn this three country deployment, he has just, 10 racks in a cage and wants to make sure he's there if something fails. There's still plenty of that going on in the country, but most of the money's being spent with companies like Equinix and DRT.” Bigger issues on the horizon But there are more issues to come, which will have even the largest players struggling. Bachar sums these up as Edge and Climate. Colocation providers are going to have to keep providing their services, offering increasing power capacity, from a grid which is having to shift to renewable energy to avert climate catastrophe. “Our power system is in transition,” says Bachar. “We're trying to move the grids into a green grid. And that transformation is creating instability. Grids are unstable in a lot of places in the world right now, because of that transition into a green environment.” At the same time, capacity is needed in the urban locations where grids are facing the biggest crisis. At present, all Internet data has to go through exchange points. “In the United States, there are 28 exchange points covering the whole country. If you're sending a WhatsApp message from, from your phone to another phone, and you’re both in Austin, Texas, the traffic has to go through Chicago.” The next stage of colo will need localized networks, says Bachar: “In the next three to five years, we're going to have to either find solutions to process at the Edge, or create stronger and better backbone networks. We're having a problem with Edge cloud. It's not growing fast enough.” The colocation data centers of the future will have to be in urban areas: “They will have to complement and live in those areas without conflict,” says Bachar. That means they must be designed with climate change in mind - meeting capacity needs without raising emissions. “We cannot continue to build data centers like we used to build them 15 years ago, it doesn't work. It doesn't help us to move society forward and create an environment for our children or grandchildren.”

"If I'm using Equinix in three countries, I may be paying 30 percent more in space and power, but I'm saving a hell of a lot of money in my replication costs across those three data centers" Colo to Cloud Supplement 7

Colocation Providers – How to Tackle Environmental Impacts and Meet Sustainability Goals Data center need to take sustainability seriously - setting clear goals, and working hard to beat them, Schneider Electric’s Greg Jones says

loud and colocation

On the one hand, there is tremendous

uptime. But there are ways that colocation

providers have an important

pressure coming from regulatory bodies, from

providers can do both – build reliable and

role to play in the global

new standards, and from shareholders to

cost-efficient data centers while also making

efforts toward sustainability.

swap out their current infrastructure for more

them sustainable.

As critical leaders in the data

efficient solutions and more environmentally

center industry, they can

friendly business models.

influence others by shifting infrastructure

At the same time, customers don’t typically

Sustainability roadmap for colocation providers

towards more energy efficient and renewable

put sustainability high on their priority list

A 451 Research survey that polled 800+ data

energy sources. Yet, there are pressures to

when they are shopping for a colocation

center professionals said sustainability is a

achieving these sustainability goals.

provider – it’s more about SLAs, reliability, and

competitive differentiator, but only 43 percent

DCD Supplement • datacenterdynamics.com

Schneider Electric | Advertorial

“You can start that journey by setting sciencebased targets around carbon neutrality, net-zero emissions, and climate neutrality” and climate neutrality. This concept of ‘green’ colocation or data center facilities is widely popular. I also spoke with Datacenter Dynamics CEO George Rockett in a recent podcast. In our chat, we recognize the global impact and variance in maturity models across colocation companies and their sustainable practices. Hint: Some colocation providers are further along than others. One key takeaway is that most colocation providers want to move forward but need to develop actionable sustainability strategies. There are ways colocation data centers can get started by setting specific goals, developing metrics, and putting procedures in place to measure and monitor progress. Getting started: A framework for achieving sustainability The climate and business case are clear colocation providers are challenged to ramp up sustainability efforts. A critical first step is to figure out how to develop or define their organization’s sustainability strategy. Wherever you are · I mportance of sustainability: it’s not just

on your journey towards carbon neutrality,

improvement plans for their infrastructure.

a feel-good concept; why sustainability

net zero, or sustainable and energy efficient

Many questions still exist on how to balance

is a competitive differentiator driven

design, I encourage you to check out these

meeting the day-to-day demands of running

by shareholder and evolving consumer

two sessions to ascertain additional actionable

a colocation facility along with ramping up

demands.

insights: Innovation Talk: How Colocation

have developed sustainability initiative

sustainability efforts. So, in a recent Innovation Talk Webinar,

·M omentum and urgency throughout the

Facilities Can Tackle Environmental Impacts

data center industry is building: there’s

with Energy Targets, and DCD>Podcast:

I spoke with colleagues from Schneider’s

a mass movement among corporations

Green Colocation with Greg Jones.

Energy Sustainability Services organization

toward climate action. Meanwhile, 74

about how to tackle environmental impacts

percent of colocation providers say their

feel free to get in touch. Let’s keep the

and meet energy targets. Here are some key

customers expect contractually binding

discussion going.

takeaways from our discussion:

efficiency and sustainability commitment,

·T hree megatrends directing the

If you have additional input or questions,

Greg Jones is vice president of Strategy &

but only 43 percent of colocation providers

Offer Management - Cloud & Service Provider

have a comprehensive sustainability

Segment, Schneider Electric

new energy landscape: digitization,

program.

decarbonization, and decentralization are

So, there’s work to be done.

reshaping market and energy demands. ·E nergy transition challenges require new

Setting sustainability targets: a greater

ways of thinking: evolving technology

focus needs to be placed on setting climate

and climate change is shifting the C-suite

goals and energy targets. You can start that

perspective and driving new investment

journey by setting science-based targets

strategies around sustainability.

around carbon neutrality, net-zero emissions,

Colo to Cloud Supplement 9

Why colos need to get serious about security Dan Swinhoe News Editor

Simple colo services are increasingly making way for a hybrid mix of cloud models and platforms. Colo providers need to adapt to new security threats amid this changing landscape

raditional colocation providers are well-versed in physical security. The norms of constructing a resilient building and restricting access both to the building and individual customer cages are well established. But as facilities get smarter and operators evolve to become hybrid or private cloud providers, the security landscape changes. As those cybersecurity risks change, the relationship and responsibilities around security between the operator and the customer also need to change. More IT, more risk for colo providers As security becomes more of a concern for organizations of all shapes and sizes, colocation providers sit in the unenviable position of needing to not only manage physical security of a data center portfolio alongside the core IT of their own organization, but also secure a growing selection of software and services being offered to customers. Many companies might have a large remit in terms of what needs securing, but few CISOs and security leaders will be bound to as many customers in terms of uptime requirements and SLAs as colo CISOs. In September 2020, Equinix suffered a ransomware attack that didn't affect customers. But other colo and hosting providers haven’t been so lucky in recent years. A ransomware attack on CyrusOne in 2019 led to a number of customers – mostly serviced by the company’s New York Data Center – being affected. The same year, QuickBooks cloud hosting firm iNSYNQ was also hit with a MegaCortex ransomware attack in July. The company said it was a “carefully planned ransomware attack” on one of its primary data centers, affecting more than 50 percent of its customer base. The malware entered its network via a phishing email and spread rapidly through its

network, including some backups. 2019 also saw hosting firm A2 Hosting go down for more than two weeks after a ransomware attack encrypted some of their Windows hosting servers and virtual private servers. A compromised RDP connection infected A2’s data center in Singapore before spreading to its US facilities as well as some customer backups. Full service wasn’t resumed for more than a month. A bad year for ransomware attacks on hosting providers, 2019 also saw ASP.NET hosting provider SmarterASP.NET as well as cloud hosting provider Dataresolution. net hit. In late 2020, Managed.com suffered an attack that bought customer sites offline. Montreal-based service provider Web Hosting Canada suffered a lengthy outage in August 2021 it blamed on unauthorized activity by an undisclosed third-party service provider. “No organization from a CSO perspective is there to eliminate all risks,” explains Michael Montoya, CSO, Equinix. “But our role is to help balance risk for the company; understand our risk and mitigate that risk as much as possible.” “From a data center perspective and product perspective, we drive security across protecting the physical elements of our HVACs, our PDUs, our UPS devices, all of our power distribution, access control into our IBX facilities,” he adds. “Then we have to protect our core critical IT assets that run our financial systems and core business infrastructure, and we have to identify our key suppliers and make sure that our data is protected within those suppliers.”

IoT and OT: increasingly integrated, increasingly targeted The broad collection of industrial control systems – often grouped together as what’s known as Operational Technology (OT) – are relatively simple in operation, but key to ensuring systems such as HVACs function normally. Their simplicity, though, can often be an advantage for attackers. OT is often viewed separately to traditional IT systems, meaning it can lack the same controls, maintenance, and security as more standard hardware and applications despite sitting on the same network. This means they can be both easy targets to compromise if connected to the Internet and vulnerable to attack if connected to a compromised IT network. “Hackers attacking back-office systems, such as building automation and building management systems are common,” says William Schultz, VP of technology at Netrality Data Centers. “Hackers will use monitoring systems as a backdoor to access the broader network in order to circumvent front-end layer security. These back-office systems are generally not as well protected.” Recent years have seen large-scale OTbased attacks increase. A 2020 survey of OT executives by Fortinet found just eight percent of respondents had seen no intrusions in the previous 12 months, yet 65 percent had seen three or more incidents. Another survey from Honeywell noted that three-quarters of facility managers were worried about the security of OT systems and improving security posture

“Hackers will use monitoring systems as a backdoor to access the broader network in order to circumvent front-end layer security. These backoffice systems are generally not as well protected”

10 DCD Supplement • datacenterdynamics.com

Secure Thinking

was a priority over the next 12-18 months. Montoya notes that there are around 13 threat actor groups that are actively building tools and technology responsible for OT-related attacks. “Unfortunately in the industry there's been this perception with OT environments that it's air-gapped,” he says. “There's been this, in my opinion, very false sense of security that’s put OT environments years behind IT security. “But if you look at the latest breaches that just happened with Colonial Pipeline in the United States, with the large meat provider GBS, or with the Florida Water System recently, a lot of organizations are finally waking up with some of these more visible breaches that are happening. “That's been a big focus for us for years; we’ve spent tremendous efforts on doing the right level of segmentation on a physical side as well as to control access to those systems and facilities, and then ensuring that that is very well tied into our data lake so if we do see some anomaly, we can triangulate that against some of our IT assets that they may touch and how do we sort of understand more if there's a threat environment happening inside of our facility space.” At the same time, as more Internet of Things (IoT) devices make their way into data centers, a new playground for potential

attackers to compromise opens up. New sensors might make data centers much smarter when it comes to monitoring operations, but it creates added complexity and potential vulnerability as each device can potentially be a new point of failure or route in for an attacker. “When it comes to those IoT types of things, you want to try and isolate those things as much as possible,” notes Flexential’s VP of cyber security Will Bass. “You don't want those devices being on the same network as customer data traffic for instance.” Managing those Industrial IoT (IIoT) systems starts to look increasingly similar to managing a traditional IT stack, requiring constant security monitoring, regular patching cycles, restricted access controls, and the ability to respond to any unusual activity quickly. “IoT devices, such as CCTV cameras and HVAC systems, are often the targeted entry point due to vulnerable security within deployed systems,” explains Michael Carr, head of strategic development at UK IT firm Six Degrees. “This often leads to access into the corporate networking environment.” Separation of IoT and building systems from both core IT and customer environments – no matter what kinds of services an operator might be providing – is key, as is robust monitoring and access management. Regular penetration testing and patch management processes should also be adopted. “At our data centers, all supporting infrastructure is both physically and logically

separated from customer environments,” Carr says. “Physical security controls – including door access, CCTV, and HVAC systems – operate on separate networks within facilities, and all segmented control networks and systems are monitored through event collection into a SIEM platform analyzed 24x7 by our SOC facility.” New services mean new security challenges for colo providers Colo companies are increasingly offering software and service solutions that blur the lines between traditional colocation and cloud. “You still have customers that come in and just want to buy data center space,” says Bass. “But we're also having more customers come in and want some colo space, some private cloud, some help with disaster recovery. “You definitely see that merging and changing for data center companies,” Bass continues. “Protecting the HVAC is definitely much different than having a VMware stack that has customer data on it, and we have to have the right processes and alerting and monitoring in place.” As colos evolve their offerings, the cybersecurity focus has to change too. Software development requires constant consideration around security, but even more so when applications and services being developed are being consumed externally. Colo providers need to ensure they are adopting the latest advice and methodologies around securely developing applications, such

Colo to Cloud Supplement 11

as OWASP top ten or NIST’s Secure Software Development Framework, to ensure they offer resilient products. “As we move more to the software element, we have put a lot of focus into ensuring that we've got the right security controls around our software fabric or metal service, starting with how we do development overall of our fabric solutions,” says Montoya. “We're running a very strict automated CI/CD pipeline; we work very closely with our product organization to control that instrumentation and ensure that we have visibility across that pipeline so that before it hits production we are able to sign off and ensure that all of the right security gates are made. “Starting from the threat modeling, all the way to the build, into the actual scanning of code as well as anything in production that we need to manage once it gets into our production facilities.” Colos becoming clouds means new security responsibilities Major vulnerabilities in IaaS providers’ cloud stacks are rare, while companies leaving themselves accidentally exposed due to configuration errors are nearly daily occurrences. Exposed AWS S3 Buckets leaking information have been a common configuration faux pas for a number of years, but AWS will always reaffirm its platform is secure. Such cloud compromises are usually rooted in human errors; something that cloud providers often offer a service to help with, but would never take the blame for. Cloud providers have spent years informing customers about the cloud security shared responsibility model and the notion that they will secure the hardware and underlying software, but everything to do with configuration, access, and monitoring of data and applications remain firmly in the customers’ hands. Where the traditional roles and responsibilities of colo operator and customer have long been well understood, those old lines have become blurred as more colo providers offer cloud services. And as yet there isn’t an equivalent shared responsibility model for the new cloudy colo firms for who owns what risks. “As organizations seek to take advantage of colocation services, we find that there isn’t always a clear delineation for which entity is responsible for network security,” says Mike O’Malley, SVP of technical advisory firm SenecaGlobal. “Companies often incorrectly assume that the colocation provider is handling all aspects of cybersecurity, protecting their servers, applications, and digital assets in a sort of electronic vault. “Colocation providers that clearly communicate to clients how they protect the

“There's a lot of work we need to do as colo providers to really help people understand where those demarcations are, and how we play in the overall shared inherited risk model" physical colocation premise and network infrastructure – and what security protections for applications and data need to be handled by the client – are in a better position to protect the entire ecosystem.” Equinix’s Montoya acknowledges that no such shared responsibility model exists for the new world of cloud and service-based colocation, and that the industry as a whole probably has to get better at educating both customers and operators on who owns what risks. “There's a lot of work we need to do as colo providers to really help people understand where those demarcations are, and how we play in the overall shared inherited risk model,” he says. “I think as a community there's a lot more dialog that needs to happen and collaboration around thinking about inherited risk and shared security overall. “This is an incredible opportunity for us as a community to create more standardization, so that we all are speaking the same language, and we're all able to build support around a very sort of common approach to how we're dealing with shared security.” Quite what that shared responsibility model between colo and customers could look like in the future hybrid world is still up for debate, but for now the onus is still very much on the customer to do their homework. “That responsibility aspect is definitely different from someone that's in our colo than it is someone that's in our private cloud,” adds Flexential’s Bass, “But it's figuring out where do those responsibilities stop [that is difficult]; every company needs to do that risk assessment.” Changing customers, changing risks As much as colo providers need to ensure attackers don’t use compromised company IT or building systems to attack customers, at the same time they have to be vigilant that their customers aren’t posing a potential risk to the company or its other customers. “Are our customers a risk? Absolutely,” acknowledges Montoya, “We have to understand our customer base to understand what risks they may bring us.” Montoya notes that there are many threat actors that are interested in using colo companies to perform what he calls upstreaming - also known as island hopping or supply chain attacks - where a provider is

12 DCD Supplement • datacenterdynamics.com

compromised in order to disrupt or pivot to customers. “They're not necessarily interested in us, but maybe they're interested in just disrupting our customers,” he says. “Our concern is how they would use our facilities or our services to try to disrupt the services of our customers. “You think about some of the big system integrator and telecom breaches that have happened over recent years; it was less about going after those companies and more about going after their customers.” As a result, Montoya says Equinix does a lot of analysis on who would be interested in its customers, whether that’s through disrupting a facility or compromising a network and attempting to pivot into a customer’s environment (which he unsurprisingly says would be ‘incredibly hard to do’). He does note, however, that companies hosting problematic content on Equinix infrastructure are higher on that risk register than an actor hopping from their cage into its interconnection environment. “Our concerns with customers are less around can they pivot to one of our physical services and probably more who are our customers and are they bringing other concerns to us, such as the events of January in the US.” Bass agrees that customers can attract added interest from unwanted eyes, saying Flexential has a number of clients it doesn’t talk about to avoid becoming more of a potential target for sophisticated actors. For now, however, the industry standard of colos protecting the building and leaving customer hardware well alone remains in place, leaving operators forced to remain vigilant but hands-off. “We do see, in some cases, activities that come to us because customers may have poor hygiene in their environment,” says Montoya. “We will alert them and help them understand the potential risk in their environment. But we don't have control over how our customers perform their own hygiene.” Flexential offers incident response services, but can only help if requested by the customer. Like other colo firms, it needs to make sure customer incidents aren’t in danger of bleeding out while remaining hands largely hands-off. He notes it is often the smaller ‘mom and pop’ businesses that end up having security challenges.

Secure Thinking “We want to make sure that we understand exactly what's happening on the edges of all customer environments so that we can see if they're having some sort of security incident or issue. We want to ensure that that is not it's not getting out and going to anyone else.” “But on the flip side, it is their environment. We're not going to go in and make changes to it without them and working with them on those issues.” Supply chain security gets a new focus A number of companies DCD spoke to noted that the recent SolarWinds breach – where attackers compromised the company’s Orion IT monitoring and management software to gain highly privileged access to its customer’s networks – has driven growing interest and focus on supply chain security. Enterprise customers are now wanting to make sure the supply chains of their own supply chain are secure. Audits from those customers about controls, compliance, security are growing in number and detail, leading to their suppliers asking the same from colo providers. “[Our customers] are making sure that they are secure so that they can prove to their customers that they are secure; that customer

data is secure in their environment, which could also be part of our environment,” says Bass. As a result, merely being compliant with any given compliance requirement or standard – whether NIST, ISO, Cyber Essentials, SOC, HIPAA, PCI, or any number of others – is no longer good enough. Montoya notes that not only are the number of audit requests increasing significantly, but the intensity of those audits have also increased. Where in previous years customers would be happy with a copy of the desired compliance certificate, they are becoming more knowledgeable and creating their own audits with customized controls. “A lot of customers now like to create their own control view and bring increased inspection on controls,” he says. “Where they might have previously had 20 additional controls, suddenly we see in some of these audits they’re doing 100 additional customized controls for review.” At the same time, colo providers must take closer looks at their own supply chains. Every vendor employed – whether to help the company operate its own business or provide a service to customers – creates a potential risk for both the colo and its customers. Target’s 2014 data breach via a

“We will alert them and help them understand the potential risk in their environment. But we don't have control over how our customers perform their own hygiene”

compromised HVAC provider remains one of the most notorious examples of supply chain breach, and one that’s very relevant for a data center industry reliant on air conditioning. But risks can come from almost any supplier. “You really have to understand the supply chain that you're relying on to deliver your services, whether those services are to protect your core data, protect your core business or products, or protect your customers,” explains Montoya. “We've implemented a third-party audit process as well as what we call continuous assurance which helps us take our key suppliers and evaluate them for their cyber risk in a much more real-time basis.” The future of colo The reality is the colo landscape is both changing rapidly while still remaining the same; some companies will always want some standard hosting services, and providers will still need to protect their core IT and their buildings, just with the added complexity of multi-cloud. “I think traditional colo is always going to be around,” says Bass. “Even if it's not the small company coming by and buying colo from you, data has to live somewhere, all these SaaS applications have to live somewhere. “The customer profile might change and I certainly think we're going to see a more mixed hybrid type of approach coming,” concludes Bass. Some customers will always only ever want you to be a landlord to host their cages, while others will want much more. It’s up to colo providers to be ready to offer what customers need, but do it securely.

Colo to Cloud Supplement 13

How Dropbox pulled off its hybrid cloud transition

Sebastian Moss Editor

We explore Magic Pocket, and whether others could do the same

hen file hosting service Dropbox first announced its hybrid cloud effort Magic Pocket in 2016, many saw it as a sign that the company was done with Amazon Web Services and was betting on an on-premise future. But the reality is more nuanced, lead developer Preslav Le told DCD.

The company has always had its own data center presence, but Dropbox needed more capacity and soon grew to become a major customer of Amazon S3 (Amazon Simple Storage Service) after joining in 2013. It didn’t take long for the company to wonder whether it made more sense to do it themselves. "We used AWS S3 because storage at scale was an extremely hard problem to solve," Le said. "It was only a few years later,

14 DCD Supplement • datacenterdynamics.com

when we really believed we could tackle this problem better for our needs, that we even tried." The result was Magic Pocket, one of the largest data migrations off the cloud in web history. This, Le said, has allowed for significant cost savings and more control - but is not something that most other companies could easily replicate. Over a two-and-a-half-year period, the company built its own massive on-

Sync to Colo

"The cloud have to solve really broad problems just imagine all the different usage patterns for S3. Our usage patterns are much simpler"

premises platform, officially launching it in 2015. This involved a huge amount of software work - including switching from programming language Go to Rust midway through to reduce memory use - and getting deeply involved with the hardware to ensure that every ounce of possible storage was squeezed out of a rack. "It's not only the language we changed," Le said. "We also significantly improved the architecture. We moved from using a file system to just managing the drive directly we literally open the drive as a block device and then we have our own formats. This allowed us to gain a lot of efficiencies from avoiding the file system, but also move quite a bit faster." For example, the company could adopt shingled magnetic recording (SMR) hard disk drives without waiting for drivers to support them. SMD disks can be much denser by writing new tracks that overlap part of the previously written magnetic track, somewhat

like overlapping roof shingles. "This is one of the examples where we were able to work closely with hard drive companies and were able to move much faster than some other companies," Le said. "They need to build a new file system, etc. Some of the big players still don't use SMR." The company helps design its own custom servers, cramming more and more storage into its data centers. "We replace our hardware every four years, but have at least a couple of new generations in those four years," Le said. "Back when we started, we worked with four terabyte drives. Now we have 20 terabytes... but we also increased the number of drives per chassis so we really increased the density quite a bit." By 2016, the company said that it had moved around 90 percent of its files over to on-prem, comprising four data centers. "What we've seen in the last couple of years is that we tend to move more things onprem than towards the cloud for our core storage production," Le said, but declined to share the exact percentage. The initial move was a big risk. "Looking back, it really turned out to be a great investment for both our velocity and the business," Le said. "Amazon and the cloud have to solve really broad problems - just imagine all the different usage patterns for S3. Our usage patterns are much simpler, and we understand them, so we can [build for them]." So does this mean Dropbox has dropped the cloud, and is essentially an on-premises business now? Not so, Le argues. "Magic Pocket is this very famous system, and often people say 'what's the Magic Pocket team?' We don't have one, we have the Storage Team. The reason we call it Storage is because their job is not to do Magic Pocket. "Their job is to provide the best, most reliable and cost-efficient storage for Dropbox. So if ever Amazon can innovate and they're better than us, and they're cheaper, or we can secure better deals wherever makes sense, their job is to advocate us moving the data back." Indeed, in places where Dropbox doesn't have the scale, or prices differ, it still relies on S3 - including the UK, mainland Europe, Japan, and much of the non-American world. It does, however, operate its own Point of Presence network. It’s all about keeping one’s options open,

Le said. "For the initial migration out of S3 to Magic Pocket, we built the ability to move data back and forth between the two locations. Over the years, we decided that it's worth retaining that capability. "So if we ever decide because of supply chain issues, Covid, or whatever, that you want to spin over some capacity to S3, we can just do it with a click of a button - we don't need to write code, you don't need to deploy, you can literally click a button and then some data can go back." He added: "If adopting other cloud providers made sense, we'd do that too.” There are other areas where the cloud comes first, too. "Some workloads from our analytics and dev box and other auxiliary things, we've moved to the cloud, where we can allow people to move faster and the cost is acceptable." The cloud still makes sense for most businesses, Le said. "I think if you're starting a company, just go use the cloud. Operating your own infrastructure comes with a cost. “And the only way to justify it is if A) You have a very good understanding of the problem. B) You have the right scale usually, that means a huge scale: with Magic Pocket we store exabytes of data. And then there’s C) Do you have the right talent?" Dropbox is also fortunate that it is primarily a storage-focused company, so it's hard to get locked into the cloud. Users of more specialized cloud services or databases are increasingly finding themselves trapped on platforms that are hard to extricate their workloads from. "Sometimes vendor lock-in is ok when building a prototype. It's a small scale, it's not expensive, just go use AWS. But if you're building something where your business margins are seriously affected, then you should seriously think of vendor lock-in." That's why, if you have the scale and the team, "you should try to really embrace hybrid cloud," he said. The cost of R&D on Magic Pocket "has not been hard to sustain" since the initial flurry of investment in the shift. "There are all these other costs like hardware and data center operations but whenever we compare costs, we take all those things into account. "Magic Pocket was a really sound investment that really paid off multiple times over."

Colo to Cloud Supplement 15

IT professionals manage with

at the edge using EcoStruxure™ IT.

Gain the flexibility you need to optimize uptime at the edge. • Gain visibility and actionable insights into the health of your IT sites to assure continuity of your operations. • Instead of reacting to IT issues, take advantage of analytics and data-driven recommendations for proactive management. • Keep costs under control by managing your IT sites more efficiently. Choose to outsource to Schneider Electric’s experts, address issues yourself, or leverage a mix of both approaches.

ecostruxureit.com

©2021 Schneider Electric. All Rights Reserved. Schneider Electric | Life Is On and EcoStruxure are trademarks and the property of Schneider Electric SE, its subsidiaries, and affiliated companies. 998-21556505_GMA-US

EcoStruxure IT Expert