DCD Issue 42 - Life as a Google contractor

Issue 42 • October 2021 datacenterdynamics.com

Submarine cables are changing Hyperscalers are upending submarine cable economics

Sunevision's CEO

What it takes to build a data center company in Hong Kong

Inside Google’s data centers Our investigation into the mistreatment and mismanagement of Google contractors

Data center water a time of in center usage Data water droughts

usage in a time of We just don't know how centers are much datadroughts Weconsu just don't ming know how much data centers are consuming

RELIABLE BATTERY SOLUTIONS

MPL - HIGH POWER SERIES HIGH RATE LITHIUM UPS BATTERY SYSTEM

YOUR BEST CHOICE FOR LITHIUM BATTERY SOLUTIONS UL9540A Tested UL1642, UL1973, UN38.3 Certified Compliant LFP Safe Lithium Technology 15-Year Design Life Solution Delivering Up to 300 kW CBC/IBC Seismic Zone 4 Certified

www.mpinarada.com | ups@mpinarada.com | Newton, MA, USA 800-982-4339

ISSN 2058-4946

Contents October 2021

6 News Facebook goes down, Google underpays contractors, and more 12 Google’s dark secret Our six month investigation into the poor working conditions at Google’s data centers

18 The CEO interview

18

37

“ We are not for sale. That’s exactly why hyperscalers want to work with us,” Sunevision CEO Raymond Tong on building a growing data center business in Hong Kong

21 Colo & Cloud supplement How colocation has adapted to the age of the hyperscale cloud, what that means for cyber security, and what lessons we can learn from Dropbox 37 D ata center water usage in a time of droughts We know its growing, but we just don’t know how much the industry is consuming

53 21

42 Revisiting Equinix’s ransomware How the company prevented a devastating attack 45 Submarine cables are changing Hyperscalers are upending submarine cable economics

50

50 Processing the universe Exploring the compute and networking needs of the world’s largest radio telescope, the Square Kilometer Array

57

53 Take a bath Why one small English council turned a listed building into a data center 57 Satellite attacks Charting the history of physical attacks of satellite infrastructure - and what it means for space networks

50

62 Back page: Don’t be stupid Facebook is a prime example of how not to build resiliency into your systems

Issue 42 • October 2021 3

CUSTOMIZE YOUR CONTAINMENT SWING DOORS · SLIDING DOORS · RIGID WALLS · CURTAINS AVAILABLE FOR NEW BUILDS OR EXISTING ROOMS · THREE STANDARD COLORS

amcoenclosures.com/containment

an IMS Engineered Products Brand

847-391-8100

MADE IN THE USA

From the Editor

Meet the team

W

hen Internet giants have troubles, you can rely on them to pretend nothing is wrong.

Facebook just experienced a sevenhour outage (p62) which also killed Instagram and WhatsApp - caused by inept network configuration.

The cockup comes as the US Congress hears testimony that Facebook harms children and democracy. Social media is being compared with tobacco and oil firms for knowingly selling deadly products.

There's an exodus from the cloud to colo, as early adopters change direction As usual, Mark Zuckerberg has shrugged off criticism, while users shuffled back onto the platform, joking weakly about how good their enforced afternoon break was.

Let's cut emissions - someday The industry's response to the climate emergency is more positive, as long as it doesn't affect business too much.

For instance, Sunevision CEO Raymond Tong (p18) told us he wants to cut emissions, and keep growing.

That's a contradiction in Hong Kong, where there's a dearth of renewable power. And it's a common position for data center leaders everywhere.

$100m

Whatever happened to Don't Be Evil?

The rebirth of colo

Amount Google underpaid workers in 16 countries

Our supplement this issue looks at colocation - and might surprise you.

For the last ten years, the industry has been expecting the cloud to swallow all the capacity currently in colocation.

It turns out there's an exodus in the opposite direction, as early cloud adopters change direction (p21).

One of the leading examples of this: Dropbox's Magic Pocket, which replaced a lot of its early cloud storage.

But these born-again buyers have learnt from the cloud, and now want more than old-school colocation.

Colo now needs bare metal, cloud onramps, and a sustainability story that can keep capacity in urban areas.

Global Editor Peter Judge @Judgecorp Editor Sebastian Moss @SebMoss News Editor Dan Swinhoe @DanSwinhoe Partner Content Editor Claire Fletcher Head of Partner Content Graeme Burton @graemeburton SEA Correspondent Paul Mah @PaulMah Brazil Correspondent Tatiane Aquim @DCDFocuspt Designer Dot McHugh Head of Sales Erica Baeta Conference Director, Global Rebecca Davison Conference Director, NAM Kisandka Moses Conference Producer, APAC Chris Davison Chief Marketing Officer Dan Loosemore

Head Office DatacenterDynamics 22 York Buildings, John Adam Street, London, WC2N 6JU

Google in denial Google suffers from the same conviction that it's always right, no matter what the evidence says.

For our cover feature, Sebastian Moss spoke to multiple Google staff and contractors, and found a disturbing pattern of exploitative employment practices (p12).

Google's response to our queries was to offer off-the-record commentary at cross-purposes, effectively denying there's a problem.

Water and space Elsewhere, we look at water - cables which cross the ocean (p45), data centers' ongoing drinking problem (p37), and a cool refit in old baths (p53).

And we look to space - the dangers of space wars there (p57) and the science of stargazing (p50).

Thanks for joining us. Without readers, we're just talking to ourselves.

Let us know what else you want to read here and on the website.

PEFC Certified This product is from sustainably managed forests and controlled sources PEFC/16-33-254

Peter Judge DCD Global Editor

Dive even deeper Follow the story and find out more about DCD products that can further expand your knowledge.

Each product is represented with a different icon and color, shown below.

Events

Intelligence

Debates

Training

Awards

CEEDA

www.pefc.org

© 2021 Data Centre Dynamics Limited All rights reserved. No part of this publication may be reproduced or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, or be stored in any retrieval system of any nature, without prior written permission of Data Centre Dynamics Limited. Applications for written permission should be directed to the editorial team at editorial@ datacenterdynamics.com. Any views or opinions expressed do not necessarily represent the views or opinions of Data Centre Dynamics Limited or its affiliates.

Disclaimer of liability: Whilst every effort has been made to ensure the quality and accuracy of the information contained in this publication at the time of going to press, Data Centre Dynamics Limited and its affiliates assume no responsibility as to the accuracy or completeness of and, to the extent permitted by law, shall not be liable for any errors or omissions or any loss, damage or expense incurred by reliance on information or any statement contained in this publication. Advertisers are solely responsible for the content of the advertising material which they submit to us and for ensuring that the material complies with applicable laws.

Data Centre Dynamics Limited and its affiliates are not responsible for any error, omission or material. Inclusion of any advertisement is not intended to endorse any views expressed, nor products or services offered, nor the organisations sponsoring the advertisement.

Whitespace

News

NEWS IN BRIEF

The biggest data center news stories of the last three months

CyrusOne board changes the CEO yet again Interim CEO David Ferdman takes over from Bruce Duncan, CEO since June 2020. Gary Wojtaszek was CEO till February 2020, and Tesh Durvasula had a spell as interim CEO.

Fire could cost OVHcloud €105 million On March 10, a fire on OVHcloud’s Strasbourg campus destroyed its SBG2 data center, and disabled three other buildings, one of them permanently.

The company plans to launch an IPO.

Supreme Court declines to hear Oracle’s JEDI challenge The company claimed it was unfairly locked out of a contract that no longer exists. The government disagreed, saying it never would have been able to compete.

Facebook disconnects Facebook Ushering in seven hours of global peace and prosperity Facebook went offline for more than seven hours on Monday, October 4, in a devastating outage that also took down Instagram, WhatsApp, and Oculus.

During routine maintenance of Facebook’s global backbone network, “a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network, effectively disconnecting Facebook data centers globally,” the company’s VP of infrastructure Santosh Janardhan said.

“Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool prevented it from properly stopping [it].”

The command caused a complete disconnection of Facebook’s server connections between its data centers and the Internet. “And that total loss of connection caused a second issue that made things worse,” Janardhan said. “One of the jobs performed by our smaller facilities is to respond to DNS queries. DNS is the address book of the Internet, enabling the simple web names we type into browsers to be translated into specific server IP addresses.

Those translation queries are answered by our authoritative name servers that occupy

well known IP addresses themselves, which in turn are advertised to the rest of the Internet via another protocol called the border gateway protocol (BGP).”

Facebook’s DNS servers disable BGP advertisements if they themselves can not speak to the data centers, since this is an indication of an unhealthy network connection. With this outage, the entire backbone appeared unhealthy causing the BGP advertisements to be withdrawn.

“The end result was that our DNS servers became unreachable even though they were still operational,” Janardhan said. “This made it impossible for the rest of the Internet to find our servers.”

Engineers were unable to access Facebook’s data centers through normal means because their networks were down, as were many of the internal tools they would normally use.

The company sent engineers to data centers, but it took time to get into the secure facilities with access systems down.

It took extra time to activate the secure access protocols needed to get people onsite and able to work on the servers. Only then could we confirm the issue and bring our backbone back online.” bit.ly/WhatsDownn

6 DCD Magazine • datacenterdynamics.com

Australia arrests ship captain accused of damaging Vocus’ ASC Cable The police claim that the ship Maersk Surabaya was behind the submarine cable break on 1 August, which caused about AU$1.5 million (US$1.1m) in damages. They blame the capitan for negligent conduct.

Rolls-Royce pitching small nuclear reactors to power data centers Small modular reactors (SMRs) are under development by a consortium led by Rolls-Royce, and could potentially power data centers or other infrastructure that needs a steady supply of low-carbon energy. They will not be available until at least 2030.

Arctic World Archive adds latest data deposit in Svalbard facility The data center stores a proprietary film which it claims can last as long as 1,000 years. The northernmost settlement in the world with a permanent civilian population, Svalbard has been declared demilitarized by 42 nations, and boasts a network of disused mines, one of which is home to the Arctic World Archive.

HPE awarded $2bn HPC GreenLake contract with NSA, hosted out of QTS The National Security Agency has awarded a 10-year, $2 billion contract to Hewlett Packard Enterprise for high-performance computing technology services.

The NSA will use the HPE GreenLake platform, the company’s on-premises fully managed cloud services platform.

The deal will include a combination of HPE Apollo systems and ProLiant servers, built and managed by HPE out of an undisclosed QTS data center, beginning in 2022.

“Implementing artificial intelligence, machine learning and analytics capabilities on massive sets of data increasingly requires High Performance Computing (HPC) systems,” said Justin Hotard, SVP and GM, HPC and Mission Critical Solutions (MCS) at HPE.

“Customers are demanding HPC capabilities on their most data-intensive projects combined with easy, simple, and agile management.”

The NSA also operates a huge data center in Utah, launched in 2014 for a cost of $1.5 billion. It has other data centers across the US and around the world, including in the UK and Denmark, and plans a large cloud contract with AWS (see below).

The NSA has huge storage needs thanks to its mass surveillance programs that continue years after they were exposed by contractor Edward Snowden in 2013. After a five-year investigation, the Privacy and Civil Liberties Oversight Board (PCLOB), an independent government watchdog, said in July that the

NSA’s XKeyscore program operated with no oversight. “What most concerned me was that we have a very powerful surveillance program that, eight years or so after exposure, still has no judicial oversight, and what I consider to be inadequate legal analysis and serious compliance infractions,” PCLOB board member Travis LeBlanc said in a partially redacted statement.

XKeyscore is just one of the NSA’s known programs, but the agency has expanded to other forms of surveillance since the 2013 disclosures.

That mass data storage requires HPC systems to trawl through the lives of US citizens and those in foreign nations.

“Modern electronic surveillance touches on pioneering uses of artificial intelligence/ machine learning systems, including autonomous collection of massive datasets, analysis of those massive datasets through algorithmic decision-making, and many other AI/ML issues,” LeBlanc said.

“Whether the public wants it or not, these systems are almost certainly here to stay.”

The head of the NSA when it first built its post-9/11 mass surveillance network was General Keith Alexander, who pushed for the agency to tap the networks of Google, Microsoft, Yahoo, and Facebook, as well as all major submarine cable landing stations.

At the time, he misled the House Intelligence Committee when asked whether his agency was involved in warrantless wiretapping.

Last year, Alexander joined Amazon’s board of directors. bit.ly/HPENSAQTS

NSA awards secret $10bn cloud contract to AWS; Microsoft files challenge The NSA has awarded a secret $10 billion contract to Amazon Web Services.

But Microsoft has already filed a protest against the contract between the two mass surveillance organizations.

The NSA contract, first reported by NextGov, is code-named “WildandStormy.” Details of the “Hybrid Compute Initiative” are limited, but could be part of the NSA’s overhaul of its Intelligence Community GovCloud primary classified data repository.

Microsoft has officially contested WildandStormy’s award to Amazon, but has not detailed why. “Based on the decision we are filing an administrative protest via the Government Accountability Office. We are exercising our legal rights and will do so carefully and responsibly,” the company said in a statement. A GAO decision is due by October 29th.

Microsoft’s attempt to block the deal comes after AWS successfully managed to scupper the $10 billion JEDI military cloud contract which initially went to Microsoft. The Central Intelligence Agency is trying a different route - its C2E contract, which could be worth tens of billions, was awarded jointly to AWS, Microsoft, Google, Oracle, and IBM. bit.ly/SurveillancePrime

Whitespace

Morgan Stanley accused of “ignoring industry standards” during decommissioning data loss incident Morgan Stanley is accused of “ignoring industry standards” over its 2016 data loss incident.

The company lost customer data during the decommissioning of two data centers in 2016. Having already paid a large fine to the regulator as a result of the incident, the company is also facing a potential class-action lawsuit which it is trying to have dismissed.

Lawyers representing consumers in a class action complaint against Morgan Stanley filed a response to the company’s recent request that the lawsuit be dismissed - providing more details and allegations of negligence.

As reported by Resource Recycling, the plaintiff’s lawyers said the bank had dismissed IBM in favor of an “unknown and unqualified

vendor” to decommission its computer equipment as part of “profit-driven decisions” in order to save $100,000 (roughly 0.0017 percent of its 2016 revenue, notes the filing).

Morgan Stanley is accused of “reckless behavior,” having a “reckless disregard of privacy,” and “failing to ensure and verify that its vendors followed proper sanitization and disposal practice.” The company also reportedly failed to locate an overwhelming majority of the lost devices, one of which was located by the plaintiff’s counsel, who said it had client Personally Identifiable Information (PII) accessible and readable in plain text.

There were also alleged lapses in Morgan Stanley’s internal record-keeping tracking retired assets. The plaintiff lawyers alleged

the Morgan Stanley vice president who was fired “admitted to his colleagues that Morgan Stanley had used asset inventory control software to track decommissioned devices… early in the project, but then stopped doing so.”

The bank is yet to file a response, but claims no harm has come to customers as a result of the data loss.

“We have continuously monitored the situation and have not detected any unauthorized access to, or misuse of, personal client information,” a Morgan Stanely spokesperson noted in a written statement to RR. “We continue to vigorously defend against these claims.”

In its August request to dismiss the case, Morgan Stanley laid the blame on a vendor known as Triple Crown, saying the company secretly sold on devices with customer data instead of wiping and recycling them as it was contracted to do.

In its filing, the bank says Triple Crown was reportedly contracted to remove, wipe, and recycle the devices. But, instead, it sold the devices to another ITAD firm, AnythingIT and told Morgan Stanley the devices had been destroyed as requested.

AnythingIT then also failed to wipe the devices, and sold them to another ITAD company, known as KruseCom, which either destroyed or sold on the devices. A year later an IT consultant in Oklahoma informed the company he had found some of its data on a storage device he had purchased from KruseCom. The company said it then investigated, took steps to recover devices, and found ‘no evidence’ of data misuse. bit.ly/SoMuchForACleanRecord

Right wing terrorist gets 10 years for plotting to blow up AWS data center Seth Aaron Pendley has been sentenced to 10 years in federal prison for planning to blow up an Amazon Web Services data center in Virginia.

Pendley, 28, was arrested in April after trying to acquire an explosive device from an undercover FBI employee in Fort Worth. He pled guilty in June.

“As this case shows, radicals are lurking on the Internet, looking for ways to lash out - and far too often, they move their plans off of the web and into the real world,” Acting US Attorney Prerak Shah said. “We are indebted to the FBI employee who put his life on the line to disrupt Mr. Pendley’s plot before he could inflict real harm on data center workers.”

Pendley told a confidential source of his plan in January. He said hoped to “kill off about 70 percent of the Internet.” Prosecutors previously said that he had hoped to blow up three data halls, and painted his gun bright colors so that it might be mistaken for a toy.

The 70 percent figure seems to be based off of misinterpreting the oftshared statistic that 70 percent of Internet traffic flows through Northern Virginia, the world’s largest data center hotspot. It is not found in one AWS data center. bit.ly/ADumbPlan

8 8 DCD DCD Magazine Magazine • datacenterdynamics.com • datacenterdynamics.com

Google underpaid temps by more than $100m, attempted cover up Failed to follow laws, and failed to fix the problem

Google contractors tell us their experience, p12

Google illegally underpaid thousands of temporary workers in 16 countries, with more than $100 million thought to be owed. The company realized it was breaking the law, but tried to cover up the failure rather than pay its workers the correct wage.

The Guardian reports that executives at the cloud and search giant were aware that Google was failing to comply with pay parity laws in the UK, Europe, and Asia since at least May 2019. In those regions, corporations are required to pay temporary workers the same as they pay full-time employees, and in some they are required to provide the same benefits. The US does not have such pay parity protections.

A whistleblower represented by Whistleblower Aid has filed a complaint about the alleged violations with the US Securities and Exchange Commission, claiming at least $100m is owed.

“It’s clear that this process has not been handled consistent with the high standards to which we hold ourselves as a company,” Google‘s chief compliance officer Spyro Karetsos said. bit.ly/DoSomeEvil

Hundreds of AWS staffers claim race and gender discrimination and retaliation And female employees launch lawsuits against the company Five female current and former Amazon employees have separately sued the company over claims of discrimination.

In addition, more than 550 Amazon Web Services staffers have signed a petition alleging “an underlying culture of systemic discrimination, harassment, bullying, and bias against women and under-represented groups.”

The five suits cover workers both at Amazon e-commerce and the AWS cloud division.

In one, Black human resources employee Pearl Thomas alleges she was punished for accusing her manager of using the n-word after believing he had ended a call with her.

Fellow Black human resources staffer Tiffany Gordwin claims she lost out on several promotions after complaining about racial bias by a White supervisor.

AWS executive Cindy Warner alleges that her manager made homophobic comments, and said that she was fired in retaliation, possibly due to her hiring a lawyer.

“We are conducting thorough investigations for each of these unrelated cases, as we do with any reported incidents, and we have found no evidence to support the allegations,” Amazon spokesperson Jaci Anderson said in a statement that provided

few concrete details.

But after mounting media attention, the company said it had opened a wider investigation into claims of discrimination at AWS, following a petition signed by more than 550 employees.

The petition said that Amazon’s discrimination investigation process is “not fair, objective or transparent,” and is “set up to protect the company and the status quo, rather than the employees filing the complaints.”

Seen by The Washington Post, the petition also references a lengthy 2020 LinkedIn post “Why I Left AWS.” In the widely shared piece, former employee Laudon Williams, claims that she had “personally heard an L8 leader using homophobic language.”

She added: “I have had people confide in me that they wanted to file HR complaints but feared for their job, as they had seen other people pushed out after airing issues. There are managers that I know have had multiple HR complaints against them substantiated and are still with the company.”

Williams also criticized the leadership at AWS and said that the division’s success masked deeper issues with how the company was operated. bit.ly/ALargeNumberOfComplaints

Report: Amazon CEO Andy Jassy prevented AWS firing employee accused of discrimination Amazon’s new CEO overruled an HR report that recommended firing an employee for alleged discriminatory behavior, Protocol reports.

The publication claims that Andy Jassy, who was CEO of AWS at the time, decided to keep the employee and did not attend a meeting about the incident.

In 2019, a Black female AWS employee told HR that Joshua Burgin had made what she felt were discriminatory comments to her. Burgin was at the time the chief of staff to senior leadership member Charlie Bell.

Following the complaint, an internal AWS team investigated the allegations, creating a report that recommended Burgin be fired. A meeting of senior staffers concluded they should have another meeting, this time with Jassy - but it never happened. Instead, Bell allegedly had a private conversation with Jassy, and the division head decided to keep Burgin on. bit.ly/AmazonsNewBoss

Whitespace Computacenter: “We are starting to see major problems in the supply of data center components” British IT reseller Computacenter has warned that supply chain issues are impacting its data center component sales.

In its latest quarterly earnings report, the company said that sales rose 29.2 percent to £3.18 billion ($4.4bn) and profit before tax jumped 59.1 percent to £115.2m ($159m).

“In the US business, the mid-market customers who materially reduced spend during H1 2020 returned during the period and, coupled with the continuing success in hyperscale data center-based customers, drove good overall organic revenue and profit performance,” Mike Norris, Computacenter’s CEO, said.

However, he added: “What has certainly influenced our business in the first half, and will continue to affect us until at least the end of the year, is the shortage of many Technology Sourcing products. This particularly affects network and workplace lines of business, but we are also starting to see major problems in the supply of data center components. The current order backlog has been around 70 - 80 percent above the average of recent years for several months.” bit.ly/NowTryGettingaPS5

Arista CEO says semiconductor supply chain is “the worst I’ve ever seen it” Shortages are “going to be prolonged,” she warns The CEO of cloud networking company Arista Networks has warned that the global semiconductor shortage is impacting much of the supply chain, and said it could still last some years.

The comments come as companies struggle to fulfill orders, and Intel’s CEO says that the worst is yet to come.

“This is the worst I’ve seen it,” Jayshree Ullal said in an earnings call. “And there have been some pretty big ups and downs in my career of several decades... and I think it’s also going to be prolonged.”

She added: “Everything from copper shortages to wafer starts to assembly to manpower, people, logistics, freight. Just about every aspect of it is challenged, too.”

Company SVP John McCool said that the

shortages are “expected to remain for the foreseeable future,” continuing: “Component lead times are the highest we’ve seen and have roughly doubled from pre-pandemic norms. Most notable are semiconductor lead times, which have extended in the range of 40-60 weeks.”

With factories operating at near full capacity, the company expects “extended lead times and escalating product costs due to expedites and elevated component increases in 2021 and 2022,” McCool said.

Intel’s new CEO said that the chip crisis will get worse in the “second half of this year.” Pat Gelsinger told the BBC that “it’s going to be a year - to two years - until we’re back to some reasonable supply-demand balance.” bit.ly/HunkerDownForTheLongHaul

Peter’s semiconductor factoid A wayward balloon nestled between two live conductors caused a major power outage in Dresden, Germany. Also brought offline were the chip fabs of Infineon and Robert Bosch.

Chip lead times hit 20.2 weeks as shortage drags on The average time it takes for a chip to be delivered has increased to 20.2 weeks, Susquehanna Financial Group found.

The increase of eight days over the previous month is the longest wait time since the company began tracking the time in 2017.

Since Covid-19 began, the world has grappled with a lengthy semiconductor shortage.

The pandemic itself was first to blame as factory shutdowns played havoc on supply chains. Then came the sudden shift in consumer consumption habits - demand for home IT boomed, as did that for cloud tech.

10 DCD Magazine • datacenterdynamics.com

At the same time, some companies pulled back from buying chips, presuming the national shutdowns would dampen demand most notably the automotive industry, which wildly misjudged how demand for cars would change, and has been playing catchup since.

That sector alone forecasts $100bn in lost sales for the year due to missing out on chips.

Adding further pressure was a series of unrelated crises - a fire at a Japanese semiconductor fab, a climate change-induced storm in Texas, an accidental cable cut, and the specter of a growing drought in Taiwan. bit.ly/SupplyShortagesContinue

Vertiv to buy E&I Engineering Group for $2 billion Bringing busways to Vertiv Vertiv is buying E&I Engineering Group, a maker of switchgear, busway, and modular power solutions.

The purchase “completes” Vertiv’s data center offering and expands its addressable market by $7 billion, according to Vertiv’s announcement.

Vertiv is paying $1.8 billion upfront, consisting of $1.17 billion in cash, and $630 million in Vertiv stock - with the potential for a further $200 million in cash based on achieving 2022 profit targets.

Founded in 1986 by Philip O’Doherty, who still serves as CEO, E&I has become known for its in-house integrated power designs and tailored technology offerings, with annual sales of around $460 million and 2,100 employees.

“The acquisition of E&I represents a key milestone in Vertiv’s strategy, completing our portfolio of in-building power train offerings for data centers and vital commercial and industrial markets,” said Vertiv CEO Rob Johnson.

“The combination will amplify Vertiv’s growth opportunities and profitability, while enabling Vertiv to deliver differentiated solutions that manage a customer’s entire power infrastructure as an integrated system.”

He added: “We look forward to adding E&I’s highly skilled team members to the Vertiv family.

“Our companies share a strong culture of engineering excellence and innovation and a passion for serving our customers with differentiated products and service.”

It’s Vertiv’s first acquisition since it became a public company, via a merger with GS Holdings.

“Our team has thoughtfully followed acquisition best practices during the process of identification, valuation, due diligence, and integration planning,” said Vertiv chairman Dave Cote. “E&I represents a unique opportunity for Vertiv and it fits well in the Vertiv portfolio.

I am excited about the potential of these two great businesses coming together as one.”

O’Doherty welcomed the move: “This transaction brings together two highly complementary businesses and represents a great outcome for E&I’s employees and customers.

“We are excited to join the Vertiv team and to continue to grow our business through Vertiv’s global reach, strong channel presence and great customer positioning in critical digital infrastructures.” bit.ly/TheGreatDataCenterConsolidation

Digital 9 Infrastructure acquires Icelandic data center company Verne Global Digital 9 Infrastructure has acquired the Icelandic data center operator Verne Global for £231 million ($320m).

Verne operates a 24MW data center campus on a former NATO site near Keflavik, offering colocation and high-performance computing services. Another 8MW of capacity is currently under development.

Verne hosts some local colocation customers, but primarily hosts HPC workloads, with an increasing focus on AI applications.

The deal is D9’s first data center investment, after the company launched earlier this year. Managed

by Triple Point Investment Management, D9 raised £300 million ($422.9m) in a March IPO, and then acquired submarine cable owner Aqua Comms for £160 million ($215m). In June, it raised another £175 million ($246.7m).

“Data centers form a key part of the digital infrastructure backbone,” D9’s Thor Johnsen said. bit.ly/BuyingUpIceland

Vantage acquires Agile Data Centers, will take over PCCW’s data center business Vantage Data Centers has announced a major expansion into Asia Pacific through two acquisitions.

The company has acquired Agile Data Centers, a data center provider in the Asia-Pacific region developing greenfield hyperscale campuses totaling almost 170MW across facilities in Tokyo, Osaka, and Melbourne. AgileDC was founded in 2020 by DigitalBridge, and in February 2021 formed a joint venture with Goldman Sachs through its Koguma DC Holding entity.

At the same time, Vantage will be taking on the data center portfolio of PCCW Ltd. Vantage’s majority shareholder DigitalBridge acquired PCCW Ltd’s 100MW data center business in July for $750 million.

Giles Proctor, formerly president and co-founder of Agile Data Centers will serve as president of Vantage’s APAC business, while PCCW DC’s senior vice president of data centers, Brian Groen, will join Vantage as senior vice president, APAC. bit.ly/VantageGetsAgile

Sebastian Moss Editor

Underpaid and overworked: Behind the scenes with Google’s data center contractors Contractors and Google employees speak out against a broken culture 12 DCD Magazine • datacenterdynamics.com

How Google Treats Its Workers

I

t had been another long day in the Google data center, and Shannon Wait was tired. It was late December, but the mood was far from festive. Someone was crying. It wasn’t the first time.

"I asked my friend what was wrong, and she said that her contract was set to expire on January 1st,” Wait told DCD. “She hadn’t heard anything about an extension, so she didn’t buy her three kids any Christmas presents. Just hearing her say that broke my heart.

“This is happening at Google? How is this necessary?"

Google’s data centers are staffed by growing numbers of temps, vendors, and contractors (TVCs), many of whom do the same work as the Googlers employed alongside them. But they are paid far less, kept on short contracts, and stripped of basic benefits.

Over the past six months, we spoke to multiple current and former TVCs, Google employees, and labor rights researchers about the hidden workers that run the Internet. This is their story.

We gave Google several weeks to respond to a list of detailed questions. After initially promising an on-the-record reply, the company offered unsolicited on-background commentary that did not appear to match the reality of what we learned about conditions at its data centers. IT outsourcing firm Modis did not reply to requests for comment.

Wait was never officially employed by Google, despite working at the company’s data center in Moncks Corner, South Carolina, every day for two years.

"When I first got a call about the job from the recruiter, I was under the impression that I would be getting a Google job - and I'm not the first person to say that. But by the time I got the paperwork for onboarding, it was pretty clear that I was working for a contractor and not Google," she said.

Wait was employed by Modis, part of contracting giant Adecco. Still, she wasn't too perturbed: sure, it was through a contractor, but after all she was going to work at a Google site. "It seemed like a reputable company that treats its workers well,” she recalled.

"But the disparity between being a TVC and being a full-time employee (FTE) was like night and day, and you pick up on that very quickly. It makes you feel like an outcast.

“And it only gets worse when you realize that you're actually doing more of the dirty work, or the scut work, than the actual Google employees who get paid twice as much as you and have more freedom to slack off or to do personal projects, while we're back at the data center, in hot working conditions, lifting heavy machines without lifts, because most of the lifts are broken, or

not available" (see box out for more).

She added: “I must have torn a ligament in my shoulder because I was lifting something so heavy, but I didn't go see a doctor about it because I don't have health insurance.”

This disparity was confirmed by every TVC and Google employee DCD spoke to.

One Google employee, who asked not to be named for fear of reprisal, said that the difference was stark. “Attendance policies are different, the work expectations are definitely quite different. I work with plenty of Google FTEs who do nothing most of the day, and that's acceptable for them. But TVCs would have a talking-to if they didn't close X number of cases in a day."

For all this, TVCs are paid less. In the US, where they usually get $15 an hour, it is perfectly legal for companies to pay TVCs less than FTEs doing the same work (unless legislation like HR 7638; Restoring Worker Power Act of 2020 is passed). In some countries, this is against the law.

In 16 of those countries, Google underpaid thousands of workers by at least $100 million. Reporting by The New York Times and The Guardian found that Google knew it was breaking the law for several years, but opted not to do anything for fear of bad publicity.

Not only are the TVCs paid less, they also don’t get access to the same benefits, like bereavement days or easily accessible paid time off.

And they are often kept on a short leash.

US-based Modis employees are only given three month contracts, and are usually told at the very end of the contract whether they are getting another three month extension.

“It was very common for contract extension decisions to come through at the last minute,” Wait said of her time at the company. “Google typically would alert Modis of an extension well in advance, but it was Modis who failed its workers when alerting employees of extensions.”

One day, Wait was having lunch with a coworker who told her it was his last day because no one had said anything about an extension. “That same day, it was found out that his contract was extended another three months, and that they just forgot to tell him,” she said.

If Modis contractors manage to cling

on through months of uncertainty, their contracts are terminated after two years, without fail. They are not allowed to reapply for a position for at least six months. Many turn to unemployment benefits to get by, relying on what little savings they had managed to accrue on $15 an hour contracts with few benefits.

Having the state subsidize Google’s workforce is a cruel irony, when the company often gets generous tax benefits for coming to struggling communities across the US, offering the promise of a large investment.

Jobs have always been a poor incentive for communities to accept data center proposals - the facilities simply don’t require many people compared to, say, a manufacturing plant. The vast majority of jobs come in the first year or so, during the construction phase, with a large number of those being out-of-state contractors.

Plus, DCD understands, since switching to a partially prefabricated model, Google has managed to roughly halve the number of job hours at new data center construction sites.

Still, of those few jobs that are more permanent, Google tells communities that they are high-skilled, well-paid, and bring the benefits of the search engine’s brand reputation.

In exchange, cash-strapped city councils, schools, and county boards vote for expansive tax breaks. This includes $97.5 million in tax cuts for Google in Ohio, $25.2m in Nevada, $16m in Arizona, and $15m in Minnesota, to name but a few. Many similar deals are not disclosed, including how many millions Google receives in abatements for its Moncks Corner data center (nor, indeed, is it known how much water that facility uses, see page 37).

A company professing to organize and share the world’s knowledge, Google negotiates these deals in secret, even going so far as to obfuscate its identity from local planning groups using shell companies until incentives are secured. "We want to make sure that we're getting fair market pricing and we're not getting special Google pricing,” a company exec told DCD in 2018.

At the same time, Google takes part in very public goodwill gestures - installing Wi-Fi on school buses and handing out Chromebooks - laptops running its own

"It only gets worse when you realize that you're actually doing more of the dirty work, or the scut work, than the actual Google employees who get paid twice as much as you" Issue 42 ∞ October 2021 13

operating system. “It's comical,” a Google employee said.

Alongside this, Google’s employment policies may actually constitute another public relations campaign - just not one aimed at local communities, workers, or consumers.

“Another really important motivation for tech companies is that by keeping more workers as contractors, and not as employees of Google, they aren't considered employees for the purposes of financial reporting,” UC Berkeley Labor Center lead researcher for the Technology and Work program, Jessie HF Hammerling, told DCD. Google is thought to employ around 130,000-150,000 TVCs across all of its offices and data centers, but no one is really sure.

Whatever the true number, it is greater than that of full-time Google employees.

“A really important financial metric that investors look at is revenue divided by headcount, which is used as an indicator of a company's productivity. So the fewer employees you have, the more productive you look on paper,” Hammerling said.

Shifting work to temporary staff may impress shareholders, but it’s just smoke and mirrors. In reality, it leads to chaos and confusion.

“It takes about one month to bring someone who has never done this work before to a basic level,” said one Google data center employee who trains TVCs. “It takes about one to two months for that person to become confident in their abilities to perform the work without having to ask multiple times. If they’re unlucky, we lose that person in three months and we cycle the process again. If they are lucky, we cycle that person through three more terms of work.

“At the end, though, if we don’t bring them on, what have we done all of this for?”

Another Googler, a manager of both TVCs and FTEs, said that constantly letting qualified temps go was a fundamentally flawed approach.

"It is absolutely stupid," they said. "The temp agencies actually do a decent job of finding people who are very passionate, or very skilled at this kind of work. And then the contract's up and all that goes out the window.”

The whole process is exhausting, the employee found. "It's just a constant cycle of having new TVCs we need to train. It doesn't make sense."

Hammerling agreed: “There's a lot of reason to think that there's a substantial cost to outsourcing. The cost of high turnover, and then the contractor fees, which can be really substantial, because these are companies that are also trying to make a profit.”

A different Google A little over two years ago, something changed at Google, multiple people DCD spoke with said.

In the past, Google has encouraged overworked temporary staff with the prospect of being taken on as full-time employees when their contracts came to a close. Now, for whatever reason, it seems that just doesn’t happen anymore. And Google and Modis haven’t communicated the change to the workers.

"It's misleading and deceptive, they're essentially using it to lure people into jobs with an expectation of advancement," Laura Padin, senior staff attorney at the non-profit National Employment Law Project (NELP), said. "Depending on how they word it, it could be illegal, because they're making a false promise."

A Googler that started after the change said: "In the time I have been with them, I have not seen one TVC promoted to FTE. I heard about it happening a number of years ago."

A colleague added: "It's messed up.

But you're dealing with a trillion-dollar corporation, and they get to do scummy stuff like that and get away with it."

Another thing Google has gotten away with is more subtle. The label TVC - temp, vendor or contractor - implies these people are temporary workers. Not so, according to those we spoke to.

Not only are these staff doing the same work as full-timers, but whole areas of work have been handed over to them. One role in particular - Data Center Technician Level 1 - is now almost exclusively filled with Modis TVCs. Level 2 roles appear to be next.

"Never have I ever been at a data center that didn't have TVCs, and it's usually a large portion of the workforce," a Google employee said. "They're not hiring L1 data center technicians anymore. That's going to be a role completely filled by temps."

It’s part of a wider shift the long-term Google staffer has seen at the company.

“Especially since [Google founders] Larry Page and Sergey Brin left, it really seriously seems to be about doing what shareholders want now. We brought in executives from other companies like Oracle and Walmart.

At the time, some people were worried that it was going to change the culture at the company, others blew it off. Well, the hens

have come home to roost," they said.

“The culture is changing. It's certainly not the same place that it was when I got hired."

The changes are getting more pronounced and disruptive, impacting the quality of operations, both Googlers and TVCs told us. Part of this is due to an internal tussle - which is built into the structure of data centers. These are buildings designed to support IT equipment, and filled with servers, but which is the most fundamental - the tech, or the building that nurtures it?

Right across the data center field, Hardware and Facilities teams have vied for power, budget, and positions - and for all its efforts to rewrite the book on technology, Google is no different.

An effort to merge the two groups proved disastrous. "It was a mess,” another Googler said. “The Facilities side got boosted up further than the Hardware side, and most of the managers got placed into positions that had them running both sides with zero idea of what happened on one side or the other,” they remembered. “Bad days.”

In February 2021, Google appointed Walmart executive Monique Picou as its head of Product, Tech Strategy, & Server Operations, with a mandate of improving the situation.

"When Monique started, her idea was to split the groups again,” the Googler said.

“Hardware was actually happy about this, and we were excited to get back to doing our job with a fair shake. Then the change got changed again. Then again.”

If tech roles get demoted to become temporary positions, this could be fallout from the turf wars between the building managers and the IT - and it could be bad news for technicians.

As one contact put it: “We were once told all Level 1 and 2 positions would never be fully TVC'd. Suddenly, here we are, watching both of those levels being moved to TVConly roles. We were told, in subtle ways, to keep our resumes up to date and constantly watch for new job opportunities that got us out of the data centers as ‘bigger changes are coming.’ Our faith is completely gone.”

Many feel Google has now prioritized shorter-term profits over grand visions, embraced cost-cutting, and pared-back once-lofty ideals as it evolves towards a more standard embodiment of corporate America.

That has been expressed in financial

“I know what it feels like to be alone, to think that you're the only one who has that issue. To feel that you're on an empty data center floor, and nobody else cares. It's horrible”

14 DCD Magazine • datacenterdynamics.com

How Google Treats Its Workers decisions, like paying TVCs less and reducing their benefits, often making their lives a struggle. But it has also led to apparent pettiness and cruelty, some told DCD. “The food team would come in and prepare a snack,” a Googler said. “And then for whatever reason, it'd be only for Googlers, and TVCs couldn’t have the cookie or other treats. There’s lots of things like that, I don’t know why.”

It was this unnecessary unkindness that proved the final straw for Shannon Wait.

Breaking point “At the very end of January 2021, I had this seal-proof water bottle that was given to me by the company,” she said. The rubber cap broke off, something Wait soon learned had also happened to a friend of hers - a friend who happened to be a Googler.

The two went to the Google employee that had given them the bottles in the first place. “I was told that my friend could have one because she is a Googler but that contractors do not get a replacement. I thought that's messed up, because she is sitting in a conference room all day and I am on the floor in 85-degree heat swapping bat boxes and pulling out heavy trays from the breadboard.

“It was like a slap in the face, but I let it go.”

Then Modis management sent around an email to TVCs telling them not to ask Googlers for anything, and emphasizing that they only ever get one water bottle.

“I was outraged by that email, it was very demeaning and condescending.”

The bottle episode was not the worst thing to happen to Wait or her friends at the Google data center. On several occasions, managers told her not to discuss salaries with other staff - something that is illegal in the US. On other occasions, the company misled TVCs about pandemic bonuses.

But the water bottle was the culmination of years of growing dissatisfaction and disappointment.

“I don't know, I hit my limit. I went home, I wrote this little Facebook post,” she said. In it, she expressed her anger and her sadness about the incident. The response was mixed, with some Googlers and TVCs agreeing, while others said she should just be happy that she had a job.

The next day, she didn’t.

“Security approached me on the data center floor and took me into a conference room where all four of the Modis program managers were on a video chat screen,” she said.

“And they said that something on my Facebook was a security risk to Google, and that I needed to be escorted off and suspended with pay until they investigated and found out

whether I broke my NDA or not.”

It was an unfortunate situation for Wait, but it turned out to be equally unfortunate for Google and Modis. It just so happened that she already had a meeting arranged with someone from the fledgling Alphabet Workers Union - a new body set up to push for better conditions for all workers at Google.

Wait called to cancel the meeting, in which she’d planned to discuss conditions at Moncks.

“I told them ‘I can't do the meeting, I think I just got fired,’” she recollected. “And as soon as I told them that, they quickly had me in touch with the attorneys at Communications Workers of America, the parent of AWU.” The union told Wait that what she had said on Facebook was protected activity.

“They didn't think the issue was as small as I thought it was. And the level of solidarity that I felt from them is so hard to explain that I want everyone who works at a Google data

center, no matter what company they work for, to feel that same sense of solidarity,” she said.

“I know what it feels like to be alone, to think that you're the only one who has that issue. To feel that you're on an empty data center floor, and nobody else cares. It's horrible.”

The union filed an Unfair Labor Practice complaint with the National Labor Relations Board (NLRB), and soon won the case.

Google and Modis had to reinstate Wait, and post notices around the data center saying that it would not punish her, and that staffers could discuss salaries and unionization.

The victory was historic. But it was also fleeting. Wait’s contract was near its two-year limit, so she left - and she doesn’t want to go back.

A few notices may have been posted in Moncks Corner, but managers had broken the law by telling Wait and others not to discuss pay. None of them were punished.

Issue 42 ∞ October 2021 15

“Somebody got promoted,” she said. “This is how the company rewards bad behavior; it is how America rewards corruption. I know that these four program managers are still talking like this to other workers.”

Two other Modis TVCs told DCD that they, or someone working alongside them, have been told not to discuss salaries since the NLRB settlement. One Googler said that he had heard it happen “often.” It is also standard practice for TVCs to be told not to talk to FTEs about any matter beyond the job at hand, including being told not to contact Google HR or whistleblower lines (both of which were described as "unhelpful, anyway" by the Googler).

An FTE technician told DCD: “Unionizing is talked about in hushed tones and out of the way conversation that is never near a manager. We have a few at my site who are trying to fight for the TVCs and even FTEs there, but we have to be quiet about it lest, well... you know how it goes, right?”

Another said: “I think Google did a really good job of localizing the impact of that [case], especially with the documents that they had to post publicly saying we're not allowed to tell you that you can't discuss your pay.

“It was disappointing that that only got posted in Moncks Corner, I would have liked to have seen those messages posted at all of their sites.”

The employee called out a specific senior Modis manager accused of breaking labor laws (DCD is not disclosing their identity, as we were not able to independently verify the incident). “The fact that [they are] still working there is mind-boggling,” the Googler said. “[They] broke the law and didn't even get a slap on the wrist. It's frustrating, the company can break whatever law, and as long as it doesn't stir up too much trouble, it just gets forgotten about.”

Weak labor laws mean that even repeated infractions are unlikely to amount to much, NELP’s Padin said. “The most you can get is essentially reinstatement and an employer notice, so it’s not the type of penalty that prevents employers from violating the law multiple times.

“The penalties are so minimal that many employers violate labor law almost like a cost of doing business.”

Wait, however, remains hopeful: “It's only

a matter of time till I find another worker who reminds me of myself in the way that they want to speak up. It's only a matter of time before someone is not scared anymore, and Google and Modis will get in trouble again.”

An inclusive company Every day he went to work, Phares Lee would walk past a large sign saying 'Don't be Evil.' "I'd have to go past that and into a bathroom I didn't identify with. It was absolutely heartbreaking."

A transgender man, Lee had been open about the fact that he was transitioning when he applied for a security job with contractor G4S, and was told that not only was it perfectly fine, but that the client was very publicly “a supporter of the LGBTQ+ community.”

That support did not translate to the G4S training facility, where Lee was forced to use the female toilet, nor to the Google data center - where Lee had just one simple request: to change his name badge from his “deadname” - or birth name. G4S said no.

"And so working there, it just got progressively worse and worse. There would be trainees that would come in and they'd ask ‘what's your real name?’ ‘Well, you're not a real guy,’ ‘blah, blah.’ I would try to shut them down as quickly as possible. And if it got worse, then I’d take it to a supervisor.

But nothing was ever getting done.

“I was reporting this over and over. It wasn't just me, there were at least two other transgender individuals on security who are going through very similar situations. There was even one cisgender young lady whose haircut is really short, and was getting a lot of transphobic comments just because of that.”

Together, all four wrote an email to management about the spiraling incidents.

“And, again, nothing was ever done. One of those transgender individuals ended up leaving and going to find a different job, because it just wasn't a safe environment for them."

Lee did not plan to stay at the data center long - he had been training as a US Marine, but left with the Trump Administration's transgender ban. The plan was to re-enlist when the ban was repealed. "That time just kept getting pushed out. So now we're three years into this process," he said. He still

“This is how the company rewards bad behavior; it is how America rewards corruption. I know that these four program managers are still talking like this to other workers” 16 DCD Magazine • datacenterdynamics.com

works at G4S, helping keep a Google data center secure.

Google's policies for trans employees are by no means perfect, but it still has specific systems and resources for those transitioning, changing names, or experiencing discrimination.

Unsurprisingly, most are not available for TVCs like Lee. Like Wait, Lee found that Google only reacted when he went public, again with the help of the Alphabet Workers Union.

"A lot of the changes that have been made are very specific to me. They fixed my badge as much as they can at this point, and they've given me access to the Employee Resource Groups. But again, it's all very me-centric.

“That's not why I did this. As much as I love my job, it isn't what I plan on doing for the rest of my life, and I want to make a difference in this company for the transgender individuals that are going to come after me."

Publicly calling out an employer is not a scalable solution. Transitioning can be hard enough, let alone doing it in the public eye, while risking one’s job.

"It's very, very difficult for me to speak publicly, I get very socially anxious," Lee said. "But because of the society we live in, and because it's so difficult to live as a transgender person in this day and age, I have to say something for those who can't.

“I am privileged to be in a stable enough place where I have a roof over my head, and I don't have to worry about a transphobic landlord, and I have enough money to pay my bills. But not every transgender person has those privileges, and this puts me in a safe enough position to speak publicly on behalf of others.”

Even then, it required Lee and his husband to spend a considerable amount of time discussing whether it was worth the risk of going public. “We talked about what our options were - our finances were already tight, especially in this economy.”

G4S contracts are longer and less restrictive than Modis ones, offering some level of job security. But there are other ways they can freeze out those deemed troublemakers.

"You can be moved to a different site that might be 50 miles away from your house,” Lee said. “Or you are given a new schedule that you cannot physically make work because you have kids or whatever home situation, despite the fact that you’ve told them several times that it can’t work.

“There have been officers that have had to deal with that. And, oftentimes, they are forced to quit because they can’t make it work. It’s not an overt retaliation, but it does happen."

How Google Treats Its Workers

“If every TVC went on strike, they would have a hard time for a month or more at Google. Imagine if this is something that happened in November or December..."

Where the power lies Worker mistreatment, two-tiered systems, and contractor abuse are not Googlespecific problems. While it is key to hold a company that has pitched itself as a different, more benevolent form of capitalism to account, it is important to understand how much these practices are now a common part of the US labor landscape.

"It's what we call fissured work, this increasing corporate practice of contracting out work," NELP's Padin said. "What we see as the commonality here is that these systems are really about degrading wages and working conditions, while also allowing the company to distance itself from its workers, and avoid accountability for the conditions that it creates.

"It is very widespread - contracting out in the tech industry seems pretty enormous.”

Most in the tech sector use TVCs for all sorts of jobs; not just for non-core roles like catering and security, but for roles fundamentally identical to FTE roles.

Microsoft was successfully sued by long-term temporary workers in 2000, who won $100m after claiming they had essentially become "common law'' permanent employees. It is thought that the arbitrary two-year-on, six-monthoff limit for Modis contractors is entirely aimed at avoiding a similar case being brought against Google.

"I think it all comes down to the fact that they really enjoy being able to evade responsibility as an employer," NELP's Padin said. "Companies are often contracting out the most undesirable work, the harder work, and so they can essentially disclaim responsibility for those job conditions."

Google is far from alone in deciding to pursue this strategy. But the situation at the company might be changing.

Labor movements have traditionally been undermined by successful efforts to split workers up: be it between black and white employees, domestic and immigrant labor forces, and full time and contractor workers.

This has worked at Google for much of its existence, but there is growing evidence that Google employees are

willing to fight for the rights of TVCs. Following protests and petitions on the matter, the Alphabet Workers Union launched this year with a concerted push for pay parity for TVCs, and a call to hire them on as FTEs. Hundreds of Google employees across the company have joined the appeal.

“I think that is a sea change,” NELP’s Laura Padin said. “I think a union saying we are inclusive of all people working for this company, regardless of if you're classified as a direct employee, or contractor or temp is an important shift. I have not seen that before, honestly.”

Even alone, TVCs have more power than they might think. “TVCs are scared and feel replaceable all of the time,” Wait said. “Together, they're so strong.”

Due to background and employment checks, it takes at least two weeks to onboard a data center TVC. It takes weeks more to get them up to speed. “So, if every TVC went on strike, just as a hypothetical, they would have a hard time for a month or more at Google,” she mused.

“Imagine if this is something that happened in November or December when Black Friday and Christmas shopping is happening and everybody is going online.

I mean, that would be a really great time for TVCs to act, should they ever decide to.”

There are less extreme options, including ‘slowdown’ - an industrial action in which employees are still at work, but seek to reduce productivity or efficiency. Googlers can help too. “Google management and Google employees can do their part to advocate for us as well, they could say they’ve had enough of this and want people hired as data center technician L1s. It would make the data center [productivity] better, and the quality of life for those people would get so much better.”

Ultimately, such change could prove most beneficial for Google. Several longtime FTE data center employees expressed exhaustion and disillusionment at the direction of the company, with many considering quitting, or reducing their personal investment in a job they no longer feel comfortable with.

"Honestly, I am feeling less and less

A problem with lifts There’s something wrong with Google’s server lift carts.

The company originally used standard systems nicknamed 'sporklifts,' which did the job, but were prone to breaking, or being left unplugged.

So, the company designed a lift in-house. "Oh Lord, these things," one Googler told us. "Dumbest idea put into practice since the invention of the Shake Weight," they added, listing a number of specific grievances with the hardware. “This is bad design.”

Another explained: "The idea was to have basically a cart with a work surface that had an integrated lift, where you could just raise the lift and pull the machine onto the lift and basically reduce the amount of time you were actually spending lifting and moving the machine around.

"Well, you add all that machinery and batteries for it, and it gets heavy.

So it was motorized and the controls were on the handles, and yeah, it crushed somebody."

Shannon Wait remembered the incident: "These new lifts have handlebars, and it's very awkward.

You have to steer it from the back. So someone was driving this and pulling it towards them while they were standing against a rack. And the lift just ran into them and pinned them to the rack."

Managers told FTEs and TVCs to still use the lift, but advised to be careful.

"They also said that they would add a flashing light for people to see it coming, because you might not hear it in a loud data center," Wait said. "So it's got this bright flashing light, and it was causing people to have headaches... [some of us] stopped using it."

proud to be at Google," one said.

Another recalled why they joined Google. "I remember being 14 years old, and there was this documentary on Google, and I thought about how cool it would be to work in their data center," they said. "It was really my dream job. And I got my dream job, and it kind of sucks."

The employee said that they loved working at the data center, but said that "all the things that are changing are getting pretty shitty, especially the way they treat people."

They added: "I'm having a harder and harder time justifying working here."

Issue 42 ∞ October 2021 17

Keeping a cool head Fossil fueled electricity, and a potentially volatile politics might trouble some executives - but not Sunevision’s Raymond Tong

H

ong Kong is one of the fastest growing data center markets in APAC. But there’s a couple of clouds over its future: the territory relies on electricity from a grid which has no immediate prospect of shifting to renewables; and its political tensions regularly make international headlines.

Where does that leave operators in the region? If Raymond Tong, CEO of Hong Kong’s largest data center company Sunevision is an indication, we’d say they are remarkably positive.

Hong Kong’s special status The Hong Kong data center market

Peter Judge Global Editor

"When subsea cables started arriving, Hong Kong was strategically located in Asia, [firms arrived] looking for space for racks” accounted for 54 percent of the APAC data center investment in the first half of 2020, according to Cushman and Wakefield. By the end of the first quarter of 2021, it already had a total of more than eight million square feet of data center space, Cushman and Wakefield predict another four million square feet of data center space will be supplied by 2025.

18 DCD Magazine • datacenterdynamics.com

Part of Hong Kong’s draw is connection, but there’s a lot more to it, Tong tells us: “There are 13 intra-Asia subsea cables, and 11 of these land in Hong Kong” Raymond Tong said in a frank and friendly phone interview.

“Hong Kong is geographically the center of Asia.”

Besides the geography and the fiber, Hong Kong has been a gateway to China, so many

Sunevision's Vision corporations place their headquarters there.

No enterprise can ignore the Chinese market, but they cannot enter it freely. As a former colony that has become a “special administrative region” of China, the region has been a bridge between the two systems: one where the local democracy makes Western businesses feel at home - although increasing Chinese administrative control has threatened this balance, and sparked recent protests.

There are more than 20 service providers and more than 40 data centers, with AWS, Facebook, Google, and Alibaba Cloud all present in some form.

Veteran player Tong speaks so knowledgeably about the local data center scene, it’s a surprise to find he’s a newcomer to it. He ran various local food and beverage companies before 2018, when he was appointed CEO of Sunevision.

By contrast, Sunevision is a veteran player, dating back to 2000, when Hong Kong’s largest property developer, Sun Hun Kai Property Limited, saw the opportunity in the first Internet boom in 2000.

“When subsea cables started arriving, Hong Kong was strategically located in Asia,” Tong says. Firms arrived in the region, “looking for space for racks.”

Sunevision listed on the Hong Kong Stock Exchange in 2000, and quickly opened a most unusual data center. In a world where most data centers are one or two stories high, Mega-i is a towering 30 story building on Hong Kong island, designed to house over 4,000 racks with a total gross floor space of more than 350,000 square feet.

It’s not only one of the oldest purposebuilt data centers still going, it’s also one of the largest.

The company was one of the rare dotcom babies to be unfazed by the dotcom crash.

“The data center business continued to grow,” says Tong. New floors opened in Mega-i as each one was filled, and then other buildings were added: “Today, we have five buildings.”

Originally founded for more general tech property development, Sunevision has focused on data centers, while other parts of the business have “faded away.” The company is 75 percent owned by the major shareholder, with the rest floated in that IPO.

“We are carrier neutral. Other data centers in Hong Kong are mostly owned by telcos, and other telcos hesitate to land their cables in those data centers.

Nine of the 11 cables terminate in Mega-i” Cables land in neutral space Starting as a property company helped Sunevision in two major ways. Obviously, finding viable space in the crowded Hong Kong region takes deep knowledge, but another factor is Sunevision’s status in telecoms.

“We are carrier neutral. Other data centers in Hong Kong are mostly owned by telcos, and other telcos hesitate to land their cables in those data centers. Of the 11 subsea cables landing in Hong Kong, nine terminate in Mega-i.”

Alongside Mega-i, the company has Mega Two, in the Fo Tan neighborhood, with 490,000 sq ft of space, One, a 20,000 sq ft facility in Kwun Tong, and Jumbo, 120,000 sq ft space in Tsuen Wan. Some three years ago, it launched Mega Plus, a high-tier greenfield site in Tseung Kwan O, aimed at cloud players, which has a 474,000 sq ft development.

That gives it a total of around 70MW. “We had accumulated enough experience to acquire a piece of land to build a data center for hyperscaler customers, the US and Chinese cloud guys,” explains Tong. “We can offer 5MW, 10MW, or 20MW. Hyperscalers really move the needle.”

The Mega Plus development is now close to being fully occupied, and Sunevision is looking at further expansion. The 20MW Mega Gateway is due to open in 2022, as is the 180MW Mega IDC. It’s also announced plans for an eighth site, 10MW in a converted warehouse.

As well as being carrier-neutral, the facilities are cloud-neutral, allowing onramps to all the major providers.

These factors helped the company to its leading position. It has some 30 percent of the space, closely followed by the communications company PCCW - a company which Tong points out, “cannot be carrier-neutral.” However, PCCW's data center

“We have accumulated enough experience to acquire a piece of land to build a data center for hyperscaler customers - the US and Chinese cloud guys"

business has now been sold to DigitalBridge, and wrapped under the Vantage brand.

He’s equally confident in a contest with the world colocation leader Equinix: “In Hong Kong, Equinix is a very respectable partner.”

Equinix is also a close neighbor: its HK1, HK2, and HK3 facilities are close to Sunevision’s Jumbo, HK4 is near Mega 2, and HK5 is in the Tseung Kwan O business park next to Mega Plus. Public information suggests Sunevision has more cross-connects than Equinix.

Recycle buildings Unlike operators in less developed areas, Sunevision builds most of its facilities in converted industrial or commercial buildings.

That’s an environmentally sound decision, and one that is also a response to rules from the authorities driven by the very limited availability of land in Hong Kong.

Hong Kong has a large number of old industrial estates, left abandoned as traditional manufacturing moves elsewhere.

The administration is reportedly very much in favor of converting these into data centers, as long as they are upgraded to meet the needs for electric power and floor loading.

Reusing the building reduces some costs and cuts waste, but there’s a bigger reason: time. “If you tear it down, it will take another four to five years. If you know what you want, you can change it.”

Stay local Two things normally happen to strong local providers: they either attempt to expand beyond their borders, or else they sell to a bigger player like Equinix or DRT, in order to achieve that expansion. Sunevision doesn’t plan on either, says Tong.

“In Hong Kong, we enjoy our core competency,” he says, listing out skills in land acquisition, design know-how, and a knowledge of how to work with local government. “We’ve been serving customers in Hong Kong for 21 years. We have an endto-end core competence.”

The region gives Sunevision all the expansion it needs, though some customers have offered to be an anchor tenant if the company expanded elsewhere in Asia or greater China: “We don’t expand for the sake of expanding.”

As to selling up: “We are not for sale. That’s exactly why hyperscalers want to work with

Issue 42 ∞ October 2021 19

us,” Privately owned data center operators are rare, but Sunevision is effectively controlled

by Raymond Kwok, the chair of Sun Hun Kai Property, a member of one of Asia’s wealthiest families, and someone who sees a future in data centers.

A big part of that future is as a partner to hyperscalers, says Tong, as building in Hong Kong is a commitment: “From acquisition to final approval, any project takes four to five years.”

That’s a big contrast to underdeveloped areas in the US or China where buildings can go up in a matter of months - but with a lot of competition on price: “We look at our friends in China operating data centers, and I don’t think any are in net-positive profit territory.”

By contrast, Sunevision has a healthy balance sheet, partly because Hong Kong data center space is at a premium.

Living on Hong Kong’s fossil grid As well as land, power must be an issue - and renewable energy is intractable in Hong Kong.

Tong declined to say much directly, but the Hong Kong grid is heavily regulated, with two monopoly players. Hong Kong Electric serves the island where Mega-i is located, while CLP serves the rest of the “New Territory,” where Sunevision’s other facilities are.

In a controlled economy any major project such as a new data center, which might require a new substation, needs government approval.

But there’s a potential issue here. Hong Kong has very little renewable energy. There are some renewable energy certificates (RECs) but they cost 50 percent more than the regular tariff.

And funding renewable projects through power purchase agreements (PPAs) as happens elsewhere in the world, is simply forbidden in Hong Kong.

Tong says he supports moves to get more renewable energy on the grid, which will bring the price down and break the chickenand-egg situation.

But China currently has a weak plan on decarbonization, and the Hong Kong administration echoes this. Chief executive Carrie Lam has promised Hong Kong will be carbon neutral by 2050, a goal which environmentalists judge to be too slow to meet the needs of the planet - and not realistically achievable with the tools available.

“Hong Kong has flourished since the British handover to China. For my team and myself, we still believe in Hong Kong in a big way” The Chinese government has said China’s emissions will continue to grow till 2030, then decline. Although CLP has 50 percent natural gas generation, Hong Kong’s electricity is still heavily dependent on coal the most polluting fossil fuel, which the UN’s Intergovernmental panel on Climate Change (IPCC) says must change - and fast.

“There must be no new coal plants built after 2021,” UN Secretary-General António Guterres said in response to the latest IPCC report. “This report must sound a death knell for coal and fossil fuels, before they destroy our planet.”

China doesn’t currently plan to adopt that suggestion. It’s still building coal plants within its borders, though it has said it will eventually stop building coal plants for other countries.

It’s not yet clear whether China will even attend the COP26 climate change summit, but if it agrees to any more binding commitments there, it could have an impact on anything which increases the load on Hong Kong’s fossil grid.

One logical idea could be some kind of pause in data center building - at least until more renewable energy can be delivered.

There’s absolutely no sign of such a move at present, but other cities such as Amsterdam and Singapore have applied a moratorium, faced with a less drastic crisis in space and energy.

Politics Tong isn’t apparently worried by such a prospect. He’s also not concerned with political instability, which has drawn the attention of media around the world.

“We’ve been blessed,” he says of Hong Kong’s “one country, two systems” approach.

“Hong Kong has flourished since the British handover to China. For my team and myself, we still believe in Hong Kong in a big way.”

Last year, Hong Kong implemented aggressive security laws written by Beijing policymakers.

The new National Security Law grants significant powers to Chinese authorities to

help them combat vague national security threats, including criminalizing seeking to “split” Hong Kong from China, or “colluding” with “external forces” to spy on China. Such crimes could lead to life imprisonment possibly in labor camps. National security suspects can be detained for six months before they are charged, and trials can happen behind closed doors.

Beijing has the power to interpret the law, rather than Hong Kong officials.

In particular, the law states (translated) that "when the Hong Kong Special Administrative Region Police Service maintains the national security department to handle crimes against the national security, it may take various measures... [including] interception of communications and covert surveillance of persons who have reasonable grounds to suspect involvement in crimes against national security."

One stabilizing factor is the way Hong Kong is embedded in the world economy, through the international framework of legal accounting and privacy rules - and those 11 subsea cables.

Of course, the last year has seen some difficulties on the cable runs, as US companies like Facebook have pulled out of cables terminating in Hong Kong, because of US fears of Chinese surveillance.

Tong says the data still gets through, but has to shift to cables in Taiwan or the Philippines: “It was initially a direct flight, but it has become a transit flight that stops in the Philippines en route to Hong Kong. That’s not ideal for sure, there are latency issues.”

He believes the basic need for data to get through will ensure the situation works out: “Politics can change tomorrow. There are now more intra-Asia cables coming into Hong Kong. Data needs to hop somewhere, and Hong Kong is in demand. If direct flights cannot be done, regional flights will be more frequent.”

It could even be an opportunity for new landing stations for regional cables, he told us. “A year ago we acquired land and will build a cable landing station.“

"Politics can change tomorrow. There are now more intra-Asia cables coming into Hong Kong. Data needs to hop somewhere, and Hong Kong is in demand.

If direct [cable links] cannot be done, regional [links] will be more frequent” 20 DCD Magazine • datacenterdynamics.com

Sponsored by

Colo & Cloud Supplement

INSIDE

How colocation is changing in the age of the hyperscale cloud The rebirth of colo

Get serious about security

How Dropbox went hybrid

> Early cloud adopters are coming back to colocation services - but with different demands

> Colos need to adapt to the new threats the rise of cloud computing brings

> We explore its Magic Pocket hybrid cloud push, and whether others could do the same

Data center experts deploy Data center experts deploy

with less risk. with less risk.

EcoStruxure™ for Data Center delivers efficiency, performance, and predictability.

EcoStruxure™ for Data Center delivers efficiency, • Rules-based designs accelerate the deployment of your performance, predictability. micro, row, pod, and or modular data centers.

• Lifecycle services drive continuous performance.

• Rules-based designs accelerate the deployment of your • Cloud-based management and services help maintain micro, row, pod, or modular data centers. uptime and manage alarms.

• Lifecycle services drive continuous performance.

• Deliver sustainability, lower TCO and maximize floor space • Cloud-based management and services help maintain uptime and manage alarms.

•#WhatsYourBoldIdea Deliver sustainability, lower TCO and maximize floor space se.com/datacenter #WhatsYourBoldIdea se.com/datacenter © 2021 Schneider Electric. All Rights Reserved. Life Is On Schneider Electric is a trademark and the property of Schneider Electric SE, its subsidiaries and affiliated companies.

998_21431513

Galaxy™ VL Galaxy™ VL

Colo & Cloud Supplement

Sponsored by

Contents 24. The rebirth of colocation Cloud adopters are returning to colo - with a new set of demands 28. Advertorial: How to tackle environmental impacts and meet sustainability goals 30. Why colos need to get serious about security Colo providers need to adapt to new threats in a changing landscape 34. H ow Dropbox pulled off its hybrid cloud transition We explore Magic Pocket, and whether others could do the same

24 34 30

34

Born-again colocation

T

he public cloud is a very good idea, but it's been hugely oversold.

That's why we're now seeing a mass movement, as early adopters return to colocation.

But they don't want traditional colocation services. They've been changed by their cloud journey, and want more.

Onramps and metal There are many reasons why public cloud has lost its luster for early adopters. Among others, it turns out that renting virtual instances by the hour can work out expensive.

Ten years ago, we expected it would be possible to do everything in AWS, Azure, or Google Cloud.

Now it's obvious that there are many things that don't make sense in the public cloud.

But there are still some things that are perfect for AWS or the others. Those things that lured us into public cloud are still good.

So this isn't an exodus from public cloud. It's a movement to combine colocation with those services we want to keep in public cloud.

That means two things: onramps that can connect the new in-house instances with those in the cloud, and bare metal servers, which offer some of the flexibility of cloud resources, with the privacy and ownership which some applications need.

All of which means that colocation companies must step up and invest to meet the new demands.

That changes the dynamics, and could prompt yet more consolidation amongst colocation

providers, because not everyone can add all the necessary features to a small local portfolio of data centers space.

The future of colocation is not fixed, and we expect a lot more changes (p24).

Security blanket With new technology comes new security risks. It has always been that way, and the new colocation world is no exception.

As new kinds of ransomware strike, colocation providers will be well outside their comfort zone.

The security arms race will always continue, and we look at the next steps in keeping up (p30).

Magic Pocket One widely-publicized example of a company where the cloud lost its initial charms is Dropbox.

Fast growth in its online file sharing business left the company addicted to Amazon's S3 storage service, with costs potentially out of control.

The answer was Magic Pocket, one of the world's biggest data migrations, that put Dropbox back in control of its storage, and allowed it to introduce better technologies at a pace it wanted.

But the story really is a prime example of the New Colo, because Dropbox has not made a total, oneway migration. If times change, Dropbox can move data in the opposite direction.

If Amazon ever offers a better deal, lead developer Preslav Le told us, Dropbox's data can go right back there.

As everyone eventually discovers, there is no single right answer.

Colo to Cloud Supplement 23

The rebirth of colocation Early cloud adopters are coming back to colocation services. But the born-again colo customers are very different, and providers face completely new challenges

F

or some years, there’s been a school of thought that colocation is out of date, and will eventually wither away in favor of the cloud. But that idea runs counter to the facts.

The colo market is stubbornly growing.

But it’s not the same market. Early cloud adopters are partially returning to colocation - and these born-again colo users are very different to the old school.

It’s been fashionable to see the cloud as an all-consuming future. The cloud can handle massive workloads, services are easy to buy, and are scalable. So why would anyone go to the trouble of buying racks and servers and installing them in retail colocation space? Surely you should let the cloud handle the grunt work, and get on with your real job!

Market figures tell a different story.

Averaging out forecasts from a bunch of providers, it seems the colocation market as a whole is growing massively, at around 16 percent per year. Over the next ten years, that adds up to a market that will quadruple in size, going from roughly $46 million in 2020, to $200 billion in 2030.

Market researchers say the retail colocation sector is bigger than wholesale colocation, where whole data centers are rented by large operators - and retail colo will keep its lead at least till 2030. What’s going on?

Cloud is massive - and efficient First off, it’s more complicated than that.

Cloud data centers really are massive because, alongside the ones leased in wholesale colo deals, hyperscalers own a massive number of sites, which they’ve built themselves. These are huge beasts, with power demands up to 1,000MW. “They’re dominating the market today,” says Yuval Bachar, a hyperscale veteran with stints at Microsoft Azure, Facebook, Cisco, and LinkedIn. “These mega data centers actually account for about 70 percent of the data center business in the world - from the power consumption as

Peter Judge Global Editor

“The public cloud as we know it has been around for 12 years, right? Everyone sees the growth, everybody sees people going pure cloud, and just running to the cloud, drinking the Kool-Aid" well as from a floor space perspective.”

But hyperscale includes some behemoths which are actually giant in-house IT services, like Facebook, Bachar points out: “Facebook is probably one of the biggest data center operators in the world nowadays. But they're serving their own enterprise needs.

They're not a public cloud service - they're running their own internal cloud.”

Bachar says hyperscale cloud data centers do indeed have a big advantage over other sectors, in their ability to deliver cheap IT power: “These sites are usually located in remote areas where the land is inexpensive, and power is available from multiple green sources.”

If those sites don’t have connectivity, the hyperscalers have the muscle to provide it: “The large companies who are building those mega data centers need to bring connectivity into those sites and be creative to create the network backbone.

And each and every one of them is creating their own backbone.”

On these sites, hyperscalers “start with one or two buildings, and then expand in a replication mode, on the same site,” Bachar says. “They create a very high level of efficiency operating the data center with a PUE of 1.06 to 1.1.”

In his view, the hyperscalers are “creating a very, very significant level of green data centers.”

Colocation has challenges Smaller colocation sites are very different, he says. They were set up to host physical servers owned by enterprises which “decided not to actually build their own data center but actually to put part of their

24 DCD Supplement • datacenterdynamics.com

IT load into a colocation site.

“These are small sites between 50 and 75MW, and in some cases can be even smaller than 15MW. They are closer to urban areas - because historically those sites actually have been put closer to the headquarters of their customers.”

These colo providers have big challenges, says Bachar: “These buildings are not scalable. Because they're sitting in urban areas, the size they have been built to this the size they're actually going to operate under for the remainder of their life. They don't have expansion space.“ A second challenge is, “they are heavily regulated - because the closer you get to the middle of the city, the heavier you are regulated for emissions, power availability and every aspect that impacts the environment around you.”

So the odds are stacked against smaller colocation companies. But their market share resolutely refuses to decrease and there’s a surprising reason for this.

According to Greg Moss, a partner at cloud advisory firm Upstack, large numbers of early cloud adopters are moving capacity out of the cloud.

Cloud defectors come back to colo “The public cloud as we know it has been around for 12 years, right? I mean, the big three - GCP, Azure, and AWS. Everyone sees the growth, everybody sees people going pure cloud, and just running to the cloud kind of drinking the Kool-Aid. What they don’t realize is there's two sides to that coin.”

According to Moss, the early adopters, the “sexy, innovative” companies who

Colocation Reborn went all-in on the cloud twelve years ago, “are now at a point where they're pulling out at least a portion of their environment, it could be 20 percent, it could be 80 percent, and hybridizing, because what they've realized over the last 12 years, that cloud isn't perfect. To really get the efficiencies from an economic and technical perspective, you really need to be in some sort of hybrid environment.”

Companies started with a “knee jerk reaction” to put everything in AWS, he says: “Why? Because some board member mandated it, or because our competitors are doing it, or because it's the rage right now.”

Later on it goes sour, because in a lot of cases, renting capacity on demand costs a lot more than owning the hardware: “Someone's losing their job, because they realize they're spending 30 percent more than they were - and the whole exercise was around cost reduction and innovation!”

would have liked to pull out a portion of their environment six years ago, but they can't because they have no headcount.

There's a big deficit in the industry for talent.”

And there’s company politics: “There’s a person who's been there 15 years, who just doesn't want to do more than what he's doing. He picks up his kid every day at school at three, and he knows that if the IT sits in AWS, he can continue to do his job and leave at three and pick up his kid. He could be the gatekeeper.

“I've seen large companies dismiss $50 million a year savings because the gatekeeper, a $150,000 employee, just

doesn't let the management know that there's an opportunity.”

Sooner or later, those early adopters can get past the gatekeepers, and start shifting the balance of their IT provision towards a hybrid model with some loads returning to colocation. But these customers are a new generation, and they will want more than just the resilient racks with power and networking, that were good enough in days gone by.

Born-again colo needs: bare metal and cloud onramp “You can't just have great resiliency, you have to have a total solution. That

The trouble with cloud It turns out that going to the cloud isn’t a simple answer to all questions: “It doesn't solve anything. It just hands your data center environment to a different company. If the data center just went away, and is miraculously living in the ozone, then fine. But it's not. You're just shifting infrastructure around in a different billable model. It makes sense: some people want to consume hardware in a day to day or hour by hour function.”

The hyperscale cloud operators can afford to lose some custom, says Moss, because they still have massive growth due to the late adopters: “AWS, GCP, and Azure are still seeing so much growth right now, because of healthcare, because of not-for-profit, because of legal, because of all the non-sexy companies that are just now getting comfortable enough to move to the cloud.”

But the early adopters really aren’t happy - and they have problems: “They're stuck for five to 10 years, because no one's going to pull out of a massive migration or massive decision after just doing it regardless of the outcome. So that's why the early adopters are now exiting. Finally!

After 10 or 12 years.”

But it’s still not easy: “They probably

“I've seen large companies dismiss $50 million a year savings because the gatekeeper, a $150,000 employee, just doesn't let the management know that there's an opportunity" Colo to Cloud Supplement 25

means big buckets - a data center that's resilient. And some sort of bare metal or custom managed component, like Equinix Metal for instance. And then there's the connectivity to the large public clouds through a partner like Megaport or a direct onramp. Those are the three components that make up hybridization.”

The capacity speaks for itself, while bare metal is a way to own dedicated capacity in someone else’s infrastructure. Customers can need this to meet privacy rules which require customer data to have a specific location away from shared hardware.

And the need for on-ramps to the public cloud is obvious. If customers are building hybrid clouds that include public cloud services as well as their own colocated servers, there should be easy to use links between the two.

Unlike the early cloud enthusiasts, the born-again colocation customers are thinking ahead, says Moss. Privacy rules might force some loads onto bare metal in future. Or they might open up a new commerce branch which would have seasonal peaks - and that could require a quick link to the cloud.

They’re thinking ahead because of the trouble they’re experiencing coming off their cloud addiction, but also because, if they pick the wrong colo, they could have to move all their IT. And, as Moss says, “nobody wants to move a data center. It's the biggest pain in the ass.”

There are companies that will physically move racks of servers from one facility to another, but Moss says: “They charge $5,000 in insurance for every million dollars in hardware, even if you're moving three blocks away. If you move $10 million worth of hardware, your insurance cost is going to be upwards of $50,000. And will they even turn back on?”

Power and networking According to Bachar, the new colo customers have another demand: they are much more power-hungry: “If we look at the technologies in the mega data centers and the colos, 80 percent of the IT load is compute and storage servers now. We're

“Nobody wants to move a data center.

It's the biggest pain in the ass”

"If we look at the technologies in the mega data centers and the colos, 80 percent of the IT load is compute and storage servers now" starting to see the emergence of AI and GPU servers, which are growing at a much faster pace than the compute and storage servers, and specialty storage servers going hand in hand with the GPUs and AI. “And the reason for that is that we're starting to deal with very large data sets.

And to process those very large data sets, we need a server, which is beyond the standard compute server.”

But GPU servers, and GPUs integrated standard compute servers demand more power: “Those high power servers are challenging our infrastructure. If you look at a typical high-end GPU server, like the ones from Nvidia, these servers are running between 6000W and 8000W watts for every six rack units (RU). That is very difficult to fit into a standard colocation where the average power per rack is 6kW to 8kW.” On those figures, a standard rack is 42 RU, so a full rack of GPU servers could demand a sevenfold increase in power.

One thing which would help is more flexibility: “Am I taking a high power rack or a low power rack? Can I actually mix technology within the rack. We need a very flexible capability in the data centers.”

New apps also need more network bandwidth, says Bachar: “Networking today is 100 and 400 Gigabit Ethernet as a baseline. We will continue to grow this to 800G and the 1.2Tbits in the future.”

Can small colos cope?

All these changes are placing huge demands on small colocation firms, while there’s a surge in demand for what they provide, and that is a big factor driving the current surge in colocation mergers and acquisitions, says Moss.

Smaller colos realize that they can’t actually fund all the changes they need to be truly successful: “So you see a lot of these smaller data centers selling off to the larger guys.”

Meanwhile, he says: “The larger guys are buying them because it speeds their go-to-market - because the infrastructure is already in place. It takes a long time to build a data center. You could probably get away with a brownfield build in the US within 18 months. If it's Greenfield, it's more likely in three years.

A lot of requests are on a shorter

26 DCD Supplement • datacenterdynamics.com

timescale than that: “Imagine you are Equinix, you have three data centers in a market and they're all bursting at the seams. You have very little inventory left.

But one of your largest customers, or an RFP from a new customer, says ‘In 12 months, we're going to need a megawatt and a half.’ But you can't build in that time.”

In that situation, the large player can buy a smaller regional player, whose data center is only 30 percent full, and put that customer in there.

“You invest some money in upgrades, you bring it up to standards, and you get certain certificates that aren't there, and you now have an anchor tenant, and maybe the facility is 60 percent full,” says Moss. “The bank loves it, because the bank takes on the existing customer leases to finance, and they also take the new signature tenant lease, that's probably 10 years long.”

The other customers are happy too, as the data center gets a perhaps-overdue facelift, along with the addition of those new must-have features, bare metal services and on-ramps.

The odds are on big colo players Small colo players often rail against giants like Equinix or Digital Realty (DRT), claiming they overcharge for basics like power and cooling, as well as services like cross-connects - links between two servers in the network. It’s very cheap for a large colo to activate a network link between two of its customers, who may even be in the same building - and yet customers are charged a high price for those crossconnects.

Multinationals don’t see that as a problem, says Moss: “A company like Equinix or DRT has everything that you would need to be successful. You are going to pay a premium, but that premium, if utilized properly, isn't really a premium.

If I'm using Equinix in three countries, I may be paying 30 percent more in space and power, but I'm saving a hell of a lot of money in my replication costs across those three data centers because I'm riding on their fabric.

“A local 200 person business in Pennsylvania, whose network engineer wants to touch every part of the hardware, is going to TierPoint, because it's two miles down the road,” he says. “He doesn't have

Colocation Reborn this three country deployment, he has just, 10 racks in a cage and wants to make sure he's there if something fails. There's still plenty of that going on in the country, but most of the money's being spent with companies like Equinix and DRT.” Bigger issues on the horizon But there are more issues to come, which will have even the largest players struggling. Bachar sums these up as Edge and Climate.

Colocation providers are going to have to keep providing their services, offering increasing power capacity, from a grid which is having to shift to renewable energy to avert climate catastrophe.

“Our power system is in transition,” says Bachar. “We're trying to move the grids into a green grid. And that transformation is creating instability. Grids are unstable in a lot of places in the world right now, because of that transition into a green environment.”

At the same time, capacity is needed in the urban locations where grids are facing the biggest crisis.

At present, all Internet data has to go through exchange points. “In the United States, there are 28 exchange points covering the whole country. If you're sending a WhatsApp message from, from your phone to another phone, and you’re both in Austin, Texas, the traffic has to go through Chicago.”

The next stage of colo will need localized networks, says Bachar: “In the next three to five years, we're going to have to either find solutions to process at the Edge, or create stronger and better backbone networks. We're having a problem with Edge cloud. It's not growing fast enough.”

The colocation data centers of the future will have to be in urban areas: “They will have to complement and live in those areas without conflict,” says Bachar. That means they must be designed with climate change in mind - meeting capacity needs without raising emissions.

“We cannot continue to build data centers like we used to build them 15 years ago, it doesn't work. It doesn't help us to move society forward and create an environment for our children or grandchildren.”

"If I'm using Equinix in three countries, I may be paying 30 percent more in space and power, but I'm saving a hell of a lot of money in my replication costs across those three data centers" Colo to Cloud Supplement 27

Colocation Providers – How to Tackle Environmental Impacts and Meet Sustainability Goals Data center need to take sustainability seriously - setting clear goals, and working hard to beat them, Schneider Electric’s Greg Jones says

C

loud and colocation

On the one hand, there is tremendous

uptime. But there are ways that colocation

providers have an important

pressure coming from regulatory bodies, from

providers can do both – build reliable and

role to play in the global

new standards, and from shareholders to

cost-efficient data centers while also making

efforts toward sustainability.

swap out their current infrastructure for more

them sustainable.

As critical leaders in the data

efficient solutions and more environmentally

center industry, they can

friendly business models.

influence others by shifting infrastructure

At the same time, customers don’t typically

Sustainability roadmap for colocation providers

towards more energy efficient and renewable

put sustainability high on their priority list

A 451 Research survey that polled 800+ data

energy sources. Yet, there are pressures to

when they are shopping for a colocation

center professionals said sustainability is a

achieving these sustainability goals.

provider – it’s more about SLAs, reliability, and

competitive differentiator, but only 43 percent

28 DCD Supplement • datacenterdynamics.com

Schneider Electric | Advertorial

“You can start that journey by setting sciencebased targets around carbon neutrality, net-zero emissions, and climate neutrality” and climate neutrality.

This concept of ‘green’ colocation or data center facilities is widely popular. I also spoke with Datacenter Dynamics CEO George Rockett in a recent podcast. In our chat, we recognize the global impact and variance in maturity models across colocation companies and their sustainable practices. Hint: Some colocation providers are further along than others. One key takeaway is that most colocation providers want to move forward but need to develop actionable sustainability strategies. There are ways colocation data centers can get started by setting specific goals, developing metrics, and putting procedures in place to measure and monitor progress.

Getting started: A framework for achieving sustainability The climate and business case are clear colocation providers are challenged to ramp up sustainability efforts.

A critical first step is to figure out how to develop or define their organization’s sustainability strategy. Wherever you are · I mportance of sustainability: it’s not just

on your journey towards carbon neutrality,

improvement plans for their infrastructure.

a feel-good concept; why sustainability

net zero, or sustainable and energy efficient

Many questions still exist on how to balance

is a competitive differentiator driven

design, I encourage you to check out these

meeting the day-to-day demands of running

by shareholder and evolving consumer

two sessions to ascertain additional actionable

a colocation facility along with ramping up

demands.

insights: Innovation Talk: How Colocation

have developed sustainability initiative

sustainability efforts.

So, in a recent Innovation Talk Webinar,

·M omentum and urgency throughout the

Facilities Can Tackle Environmental Impacts

data center industry is building: there’s

with Energy Targets, and DCD>Podcast:

I spoke with colleagues from Schneider’s

a mass movement among corporations

Green Colocation with Greg Jones.

Energy Sustainability Services organization

toward climate action. Meanwhile, 74

about how to tackle environmental impacts

percent of colocation providers say their

feel free to get in touch. Let’s keep the

and meet energy targets. Here are some key

customers expect contractually binding

discussion going.

takeaways from our discussion:

efficiency and sustainability commitment,

·T hree megatrends directing the

If you have additional input or questions,

Greg Jones is vice president of Strategy &

but only 43 percent of colocation providers

Offer Management - Cloud & Service Provider

have a comprehensive sustainability

Segment, Schneider Electric

new energy landscape: digitization,

program.

decarbonization, and decentralization are

So, there’s work to be done.

reshaping market and energy demands. ·E nergy transition challenges require new

Setting sustainability targets: a greater

ways of thinking: evolving technology

focus needs to be placed on setting climate

and climate change is shifting the C-suite

goals and energy targets. You can start that

perspective and driving new investment

journey by setting science-based targets

strategies around sustainability.

around carbon neutrality, net-zero emissions,

Issue 42 ∞ October 2021 29

Why colos need to get serious about security Dan Swinhoe News Editor

Simple colo services are increasingly making way for a hybrid mix of cloud models and platforms. Colo providers need to adapt to new security threats amid this changing landscape

T

raditional colocation providers are well-versed in physical security. The norms of constructing a resilient building and restricting access both to the building and individual customer cages are well established.

But as facilities get smarter and operators evolve to become hybrid or private cloud providers, the security landscape changes.

As those cybersecurity risks change, the relationship and responsibilities around security between the operator and the customer also need to change.

More IT, more risk for colo providers As security becomes more of a concern for organizations of all shapes and sizes, colocation providers sit in the unenviable position of needing to not only manage physical security of a data center portfolio alongside the core IT of their own organization, but also secure a growing selection of software and services being offered to customers.

Many companies might have a large remit in terms of what needs securing, but few CISOs and security leaders will be bound to as many customers in terms of uptime requirements and SLAs as colo CISOs. In September 2020, Equinix suffered a ransomware attack that didn't affect customers (see p42). But other colo and hosting providers haven’t been so lucky in recent years. A ransomware attack on CyrusOne in 2019 led to a number of customers – mostly serviced by the company’s New York Data Center – being affected.

The same year, QuickBooks cloud hosting firm iNSYNQ was also hit with a MegaCortex ransomware attack in July. The company said it was a “carefully planned ransomware attack” on one of its primary data centers, affecting more than 50 percent of its customer base. The malware entered its network via a phishing email and spread rapidly through its

network, including some backups.

2019 also saw hosting firm A2 Hosting go down for more than two weeks after a ransomware attack encrypted some of their Windows hosting servers and virtual private servers. A compromised RDP connection infected A2’s data center in Singapore before spreading to its US facilities as well as some customer backups. Full service wasn’t resumed for more than a month.

A bad year for ransomware attacks on hosting providers, 2019 also saw ASP.NET hosting provider SmarterASP.NET as well as cloud hosting provider Dataresolution. net hit. In late 2020, Managed.com suffered an attack that bought customer sites offline.

Montreal-based service provider Web Hosting Canada suffered a lengthy outage in August 2021 it blamed on unauthorized activity by an undisclosed third-party service provider.

“No organization from a CSO perspective is there to eliminate all risks,” explains Michael Montoya, CSO, Equinix. “But our role is to help balance risk for the company; understand our risk and mitigate that risk as much as possible.”

“From a data center perspective and product perspective, we drive security across protecting the physical elements of our HVACs, our PDUs, our UPS devices, all of our power distribution, access control into our IBX facilities,” he adds.

“Then we have to protect our core critical IT assets that run our financial systems and core business infrastructure, and we have to identify our key suppliers and make sure that our data is protected within those suppliers.”

IoT and OT: increasingly integrated, increasingly targeted The broad collection of industrial control systems – often grouped together as what’s known as Operational Technology (OT) – are relatively simple in operation, but key to ensuring systems such as HVACs function normally. Their simplicity, though, can often be an advantage for attackers.

OT is often viewed separately to traditional IT systems, meaning it can lack the same controls, maintenance, and security as more standard hardware and applications despite sitting on the same network. This means they can be both easy targets to compromise if connected to the Internet and vulnerable to attack if connected to a compromised IT network.

“Hackers attacking back-office systems, such as building automation and building management systems are common,” says William Schultz, VP of technology at Netrality Data Centers. “Hackers will use monitoring systems as a backdoor to access the broader network in order to circumvent front-end layer security. These back-office systems are generally not as well protected.”

Recent years have seen large-scale OTbased attacks increase. A 2020 survey of OT executives by Fortinet found just eight percent of respondents had seen no intrusions in the previous 12 months, yet 65 percent had seen three or more incidents. Another survey from Honeywell noted that three-quarters of facility managers were worried about the security of OT systems and improving security posture

“Hackers will use monitoring systems as a backdoor to access the broader network in order to circumvent front-end layer security. These backoffice systems are generally not as well protected”

30 DCD Supplement • datacenterdynamics.com

Secure Thinking

was a priority over the next 12-18 months.

Montoya notes that there are around 13 threat actor groups that are actively building tools and technology responsible for OT-related attacks. “Unfortunately in the industry there's been this perception with OT environments that it's air-gapped,” he says.

“There's been this, in my opinion, very false sense of security that’s put OT environments years behind IT security.

“But if you look at the latest breaches that just happened with Colonial Pipeline in the United States, with the large meat provider GBS, or with the Florida Water System recently, a lot of organizations are finally waking up with some of these more visible breaches that are happening.

“That's been a big focus for us for years; we’ve spent tremendous efforts on doing the right level of segmentation on a physical side as well as to control access to those systems and facilities, and then ensuring that that is very well tied into our data lake so if we do see some anomaly, we can triangulate that against some of our IT assets that they may touch and how do we sort of understand more if there's a threat environment happening inside of our facility space.”

At the same time, as more Internet of Things (IoT) devices make their way into data centers, a new playground for potential

attackers to compromise opens up.

New sensors might make data centers much smarter when it comes to monitoring operations, but it creates added complexity and potential vulnerability as each device can potentially be a new point of failure or route in for an attacker.

“When it comes to those IoT types of things, you want to try and isolate those things as much as possible,” notes Flexential’s VP of cyber security Will Bass. “You don't want those devices being on the same network as customer data traffic for instance.”

Managing those Industrial IoT (IIoT) systems starts to look increasingly similar to managing a traditional IT stack, requiring constant security monitoring, regular patching cycles, restricted access controls, and the ability to respond to any unusual activity quickly.

“IoT devices, such as CCTV cameras and HVAC systems, are often the targeted entry point due to vulnerable security within deployed systems,” explains Michael Carr, head of strategic development at UK IT firm Six Degrees. “This often leads to access into the corporate networking environment.”

Separation of IoT and building systems from both core IT and customer environments – no matter what kinds of services an operator might be providing – is key, as is robust monitoring and access management. Regular penetration testing and patch management processes should also be adopted.

“At our data centers, all supporting infrastructure is both physically and logically

separated from customer environments,” Carr says. “Physical security controls – including door access, CCTV, and HVAC systems – operate on separate networks within facilities, and all segmented control networks and systems are monitored through event collection into a SIEM platform analyzed 24x7 by our SOC facility.”

New services mean new security challenges for colo providers Colo companies are increasingly offering software and service solutions that blur the lines between traditional colocation and cloud.

“You still have customers that come in and just want to buy data center space,” says Bass.

“But we're also having more customers come in and want some colo space, some private cloud, some help with disaster recovery.

“You definitely see that merging and changing for data center companies,” Bass continues. “Protecting the HVAC is definitely much different than having a VMware stack that has customer data on it, and we have to have the right processes and alerting and monitoring in place.”

As colos evolve their offerings, the cybersecurity focus has to change too.

Software development requires constant consideration around security, but even more so when applications and services being developed are being consumed externally.

Colo providers need to ensure they are adopting the latest advice and methodologies around securely developing applications, such

Colo to Cloud Supplement 31

as OWASP top ten or NIST’s Secure Software Development Framework, to ensure they offer resilient products.

“As we move more to the software element, we have put a lot of focus into ensuring that we've got the right security controls around our software fabric or metal service, starting with how we do development overall of our fabric solutions,” says Montoya.

“We're running a very strict automated CI/CD pipeline; we work very closely with our product organization to control that instrumentation and ensure that we have visibility across that pipeline so that before it hits production we are able to sign off and ensure that all of the right security gates are made.

“Starting from the threat modeling, all the way to the build, into the actual scanning of code as well as anything in production that we need to manage once it gets into our production facilities.”

Colos becoming clouds means new security responsibilities Major vulnerabilities in IaaS providers’ cloud stacks are rare, while companies leaving themselves accidentally exposed due to configuration errors are nearly daily occurrences.

Exposed AWS S3 Buckets leaking information have been a common configuration faux pas for a number of years, but AWS will always reaffirm its platform is secure. Such cloud compromises are usually rooted in human errors; something that cloud providers often offer a service to help with, but would never take the blame for.

Cloud providers have spent years informing customers about the cloud security shared responsibility model and the notion that they will secure the hardware and underlying software, but everything to do with configuration, access, and monitoring of data and applications remain firmly in the customers’ hands.

Where the traditional roles and responsibilities of colo operator and customer have long been well understood, those old lines have become blurred as more colo providers offer cloud services. And as yet there isn’t an equivalent shared responsibility model for the new cloudy colo firms for who owns what risks.

“As organizations seek to take advantage of colocation services, we find that there isn’t always a clear delineation for which entity is responsible for network security,” says Mike O’Malley, SVP of technical advisory firm SenecaGlobal. “Companies often incorrectly assume that the colocation provider is handling all aspects of cybersecurity, protecting their servers, applications, and digital assets in a sort of electronic vault.

“Colocation providers that clearly communicate to clients how they protect the

“There's a lot of work we need to do as colo providers to really help people understand where those demarcations are, and how we play in the overall shared inherited risk model" physical colocation premise and network infrastructure – and what security protections for applications and data need to be handled by the client – are in a better position to protect the entire ecosystem.”

Equinix’s Montoya acknowledges that no such shared responsibility model exists for the new world of cloud and service-based colocation, and that the industry as a whole probably has to get better at educating both customers and operators on who owns what risks.

“There's a lot of work we need to do as colo providers to really help people understand where those demarcations are, and how we play in the overall shared inherited risk model,” he says. “I think as a community there's a lot more dialog that needs to happen and collaboration around thinking about inherited risk and shared security overall.

“This is an incredible opportunity for us as a community to create more standardization, so that we all are speaking the same language, and we're all able to build support around a very sort of common approach to how we're dealing with shared security.”

Quite what that shared responsibility model between colo and customers could look like in the future hybrid world is still up for debate, but for now the onus is still very much on the customer to do their homework.

“That responsibility aspect is definitely different from someone that's in our colo than it is someone that's in our private cloud,” adds Flexential’s Bass, “But it's figuring out where do those responsibilities stop [that is difficult]; every company needs to do that risk assessment.”

Changing customers, changing risks As much as colo providers need to ensure attackers don’t use compromised company IT or building systems to attack customers, at the same time they have to be vigilant that their customers aren’t posing a potential risk to the company or its other customers.

“Are our customers a risk? Absolutely,” acknowledges Montoya, “We have to understand our customer base to understand what risks they may bring us.”

Montoya notes that there are many threat actors that are interested in using colo companies to perform what he calls upstreaming - also known as island hopping or supply chain attacks - where a provider is

32 DCD Supplement • datacenterdynamics.com

compromised in order to disrupt or pivot to customers.

“They're not necessarily interested in us, but maybe they're interested in just disrupting our customers,” he says. “Our concern is how they would use our facilities or our services to try to disrupt the services of our customers.

“You think about some of the big system integrator and telecom breaches that have happened over recent years; it was less about going after those companies and more about going after their customers.”

As a result, Montoya says Equinix does a lot of analysis on who would be interested in its customers, whether that’s through disrupting a facility or compromising a network and attempting to pivot into a customer’s environment (which he unsurprisingly says would be ‘incredibly hard to do’).

He does note, however, that companies hosting problematic content on Equinix infrastructure are higher on that risk register than an actor hopping from their cage into its interconnection environment.

“Our concerns with customers are less around can they pivot to one of our physical services and probably more who are our customers and are they bringing other concerns to us, such as the events of January in the US.” Bass agrees that customers can attract added interest from unwanted eyes, saying Flexential has a number of clients it doesn’t talk about to avoid becoming more of a potential target for sophisticated actors.

For now, however, the industry standard of colos protecting the building and leaving customer hardware well alone remains in place, leaving operators forced to remain vigilant but hands-off.

“We do see, in some cases, activities that come to us because customers may have poor hygiene in their environment,” says Montoya.

“We will alert them and help them understand the potential risk in their environment. But we don't have control over how our customers perform their own hygiene.”

Flexential offers incident response services, but can only help if requested by the customer. Like other colo firms, it needs to make sure customer incidents aren’t in danger of bleeding out while remaining hands largely hands-off. He notes it is often the smaller ‘mom and pop’ businesses that end up having security challenges.

Secure Thinking “We want to make sure that we understand exactly what's happening on the edges of all customer environments so that we can see if they're having some sort of security incident or issue. We want to ensure that that is not it's not getting out and going to anyone else.”

“But on the flip side, it is their environment.

We're not going to go in and make changes to it without them and working with them on those issues.”

Supply chain security gets a new focus A number of companies DCD spoke to noted that the recent SolarWinds breach – where attackers compromised the company’s Orion IT monitoring and management software to gain highly privileged access to its customer’s networks – has driven growing interest and focus on supply chain security.

Enterprise customers are now wanting to make sure the supply chains of their own supply chain are secure. Audits from those customers about controls, compliance, security are growing in number and detail, leading to their suppliers asking the same from colo providers. “[Our customers] are making sure that they are secure so that they can prove to their customers that they are secure; that customer

data is secure in their environment, which could also be part of our environment,” says Bass.

As a result, merely being compliant with any given compliance requirement or standard – whether NIST, ISO, Cyber Essentials, SOC, HIPAA, PCI, or any number of others – is no longer good enough.

Montoya notes that not only are the number of audit requests increasing significantly, but the intensity of those audits have also increased. Where in previous years customers would be happy with a copy of the desired compliance certificate, they are becoming more knowledgeable and creating their own audits with customized controls.

“A lot of customers now like to create their own control view and bring increased inspection on controls,” he says. “Where they might have previously had 20 additional controls, suddenly we see in some of these audits they’re doing 100 additional customized controls for review.”

At the same time, colo providers must take closer looks at their own supply chains.

Every vendor employed – whether to help the company operate its own business or provide a service to customers – creates a potential risk for both the colo and its customers.

Target’s 2014 data breach via a

“We will alert them and help them understand the potential risk in their environment. But we don't have control over how our customers perform their own hygiene”

compromised HVAC provider remains one of the most notorious examples of supply chain breach, and one that’s very relevant for a data center industry reliant on air conditioning. But risks can come from almost any supplier.

“You really have to understand the supply chain that you're relying on to deliver your services, whether those services are to protect your core data, protect your core business or products, or protect your customers,” explains Montoya.

“We've implemented a third-party audit process as well as what we call continuous assurance which helps us take our key suppliers and evaluate them for their cyber risk in a much more real-time basis.”

The future of colo The reality is the colo landscape is both changing rapidly while still remaining the same; some companies will always want some standard hosting services, and providers will still need to protect their core IT and their buildings, just with the added complexity of multi-cloud.

“I think traditional colo is always going to be around,” says Bass. “Even if it's not the small company coming by and buying colo from you, data has to live somewhere, all these SaaS applications have to live somewhere.

“The customer profile might change and I certainly think we're going to see a more mixed hybrid type of approach coming,” concludes Bass.

Some customers will always only ever want you to be a landlord to host their cages, while others will want much more. It’s up to colo providers to be ready to offer what customers need, but do it securely.

Colo to Cloud Supplement 33

How Dropbox pulled off its hybrid cloud transition

Sebastian Moss Editor

We explore Magic Pocket, and whether others could do the same

W

hen file hosting service Dropbox first announced its hybrid cloud effort Magic Pocket in 2016, many saw it as a sign that the company was done with Amazon Web Services and was betting on an on-premise future.

But the reality is more nuanced, lead developer Preslav Le told DCD.

The company has always had its own data center presence, but Dropbox needed more capacity and soon grew to become a major customer of Amazon S3 (Amazon Simple Storage Service) after joining in 2013. It didn’t take long for the company to wonder whether it made more sense to do it themselves.

"We used AWS S3 because storage at scale was an extremely hard problem to solve," Le said. "It was only a few years later,

34 DCD Supplement • datacenterdynamics.com

when we really believed we could tackle this problem better for our needs, that we even tried."

The result was Magic Pocket, one of the largest data migrations off the cloud in web history. This, Le said, has allowed for significant cost savings and more control - but is not something that most other companies could easily replicate.

Over a two-and-a-half-year period, the company built its own massive on-

Sync to Colo

"The cloud have to solve really broad problems just imagine all the different usage patterns for S3.

Our usage patterns are much simpler"

premises platform, officially launching it in 2015. This involved a huge amount of software work - including switching from programming language Go to Rust midway through to reduce memory use - and getting deeply involved with the hardware to ensure that every ounce of possible storage was squeezed out of a rack.

"It's not only the language we changed," Le said. "We also significantly improved the architecture. We moved from using a file system to just managing the drive directly we literally open the drive as a block device and then we have our own formats. This allowed us to gain a lot of efficiencies from avoiding the file system, but also move quite a bit faster."

For example, the company could adopt shingled magnetic recording (SMR) hard disk drives without waiting for drivers to support them. SMD disks can be much denser by writing new tracks that overlap part of the previously written magnetic track, somewhat

like overlapping roof shingles.

"This is one of the examples where we were able to work closely with hard drive companies and were able to move much faster than some other companies," Le said.

"They need to build a new file system, etc.

Some of the big players still don't use SMR." The company helps design its own custom servers, cramming more and more storage into its data centers. "We replace our hardware every four years, but have at least a couple of new generations in those four years," Le said. "Back when we started, we worked with four terabyte drives. Now we have 20 terabytes... but we also increased the number of drives per chassis so we really increased the density quite a bit."

By 2016, the company said that it had moved around 90 percent of its files over to on-prem, comprising four data centers.

"What we've seen in the last couple of years is that we tend to move more things onprem than towards the cloud for our core storage production," Le said, but declined to share the exact percentage.

The initial move was a big risk. "Looking back, it really turned out to be a great investment for both our velocity and the business," Le said. "Amazon and the cloud have to solve really broad problems - just imagine all the different usage patterns for S3. Our usage patterns are much simpler, and we understand them, so we can [build for them]."

So does this mean Dropbox has dropped the cloud, and is essentially an on-premises business now?

Not so, Le argues. "Magic Pocket is this very famous system, and often people say 'what's the Magic Pocket team?' We don't have one, we have the Storage Team. The reason we call it Storage is because their job is not to do Magic Pocket.

"Their job is to provide the best, most reliable and cost-efficient storage for Dropbox. So if ever Amazon can innovate and they're better than us, and they're cheaper, or we can secure better deals wherever makes sense, their job is to advocate us moving the data back."

Indeed, in places where Dropbox doesn't have the scale, or prices differ, it still relies on S3 - including the UK, mainland Europe, Japan, and much of the non-American world. It does, however, operate its own Point of Presence network.

It’s all about keeping one’s options open,

Le said. "For the initial migration out of S3 to Magic Pocket, we built the ability to move data back and forth between the two locations. Over the years, we decided that it's worth retaining that capability.

"So if we ever decide because of supply chain issues, Covid, or whatever, that you want to spin over some capacity to S3, we can just do it with a click of a button - we don't need to write code, you don't need to deploy, you can literally click a button and then some data can go back."

He added: "If adopting other cloud providers made sense, we'd do that too.”

There are other areas where the cloud comes first, too. "Some workloads from our analytics and dev box and other auxiliary things, we've moved to the cloud, where we can allow people to move faster and the cost is acceptable."

The cloud still makes sense for most businesses, Le said. "I think if you're starting a company, just go use the cloud.

Operating your own infrastructure comes with a cost.

“And the only way to justify it is if A) You have a very good understanding of the problem. B) You have the right scale usually, that means a huge scale: with Magic Pocket we store exabytes of data. And then there’s C) Do you have the right talent?"

Dropbox is also fortunate that it is primarily a storage-focused company, so it's hard to get locked into the cloud.

Users of more specialized cloud services or databases are increasingly finding themselves trapped on platforms that are hard to extricate their workloads from.

"Sometimes vendor lock-in is ok when building a prototype. It's a small scale, it's not expensive, just go use AWS. But if you're building something where your business margins are seriously affected, then you should seriously think of vendor lock-in."

That's why, if you have the scale and the team, "you should try to really embrace hybrid cloud," he said.

The cost of R&D on Magic Pocket "has not been hard to sustain" since the initial flurry of investment in the shift. "There are all these other costs like hardware and data center operations but whenever we compare costs, we take all those things into account.

"Magic Pocket was a really sound investment that really paid off multiple times over."

Colo to Cloud Supplement 35

IT professionals manage with

at the edge using EcoStruxure™ IT.

Gain the flexibility you need to optimize uptime at the edge.

• Gain visibility and actionable insights into the health of your IT sites to assure continuity of your operations.

• Instead of reacting to IT issues, take advantage of analytics and data-driven recommendations for proactive management.

• Keep costs under control by managing your IT sites more efficiently.

Choose to outsource to Schneider Electric’s experts, address issues yourself, or leverage a mix of both approaches.

ecostruxureit.com

©2021 Schneider Electric. All Rights Reserved. Schneider Electric | Life Is On and EcoStruxure are trademarks and the property of Schneider Electric SE, its subsidiaries, and affiliated companies.

998-21556505_GMA-US

EcoStruxure IT Expert

A Dry Article

Data center water usage remains hidden We don’t know how much water data centers use. We just know it’s a lot

T

his September, as a brutal drought dragged on in the Southwestern United States, the National Oceanic and Atmospheric Administration issued a stark warning: This is only going to get worse.

The region is suffering the worst water shortage on record. Reservoirs are at all-time lows. Drinking supplies, irrigation systems, hydropower generation, fishing stocks, and more are at risk of collapse.

NOAA linked this drought, and others across the US, to climate change - a manmade problem that we appear unlikely to adequately combat any time soon.

Every sector will have to face this reality - that they will need to reduce water usage as supplies dwindle, and they will need to become better citizens in struggling communities.

The booming data center industry is no different. As facilities have cropped up across the country, they have added extra pressure

"Roughly three-fourths of US data centers’ operational water footprint is from indirect water dependencies"

Sebastian Moss Editor

to regions already challenged with meeting public, agricultural, and industrial needs.

It begs the question - how much water do US data centers use?

"We don't really know," Lawrence Berkeley National Laboratory research scientist Dr. Arman Shehabi explained.

Best known in the industry for his landmark work on quantifying how much power US data centers consume (around 205TWh in 2018), Shehabi is now trying to do the same for data center water.

"I never thought it could be worse transparency than on the energy side, but we actually know less," he said.

In a study recently published in Environmental Research Letters, researchers Shehabi, Landon Marston, and Md Abu Bakar Siddik tried to estimate the data center industry’s water use in the US, with what data

Issue 42 ∞ October 2021 37

is currently available.

When calculating water use, it's important to not only look at the water used directly to cool data centers, but also at the water used by power plants to generate that 205TWh. The researchers also tracked the water used by wastewater treatment plants due to data centers, as well as the water used by power plants to power that portion of the wastewater treatment site's workload.

Server manufacturing and wider data center lifecycle aspects were not included as they are comparatively smaller.

"The total annual operational water footprint of US data centers in 2018 is estimated at 5.13 × 108 m3," the paper states, with the industry relying on water from 90 percent of US watersheds. 5.13 billion cubic meters equates to 5.13 trillion liters (1.128 trillion imperial gallons, 1.36 trillion US gallons) of water.

"Roughly three-fourths of US data centers’ operational water footprint is from indirect water dependencies. The indirect water footprint of data centers in 2018 due to their electricity demands is 3.83 × 108 m3, while the indirect water footprint attributed to water and wastewater utilities serving data centers is several orders of magnitude smaller (4.50 × 105 m3)."

"Data centers' heavy reliance on water-scarce basins to supply their direct and indirect water requirements highlight the industry’s role in local water scarcity" Overall, the researchers estimated that 1MWh of data center energy consumption required 7.1 m3 of water, but that there is a huge variation at the local level (range 1.8– 105.9 m3) based on the power plants used.

For example, while there are not that many data centers in the Southwest subbasin, the disproportionate amount of electricity from water-intensive hydroelectricity facilities and the high evaporative potential in this arid region means that the facilities are responsible for much more water usage.

Direct water consumption of US data centers in 2020 is estimated at 1.30 × 108 m3.

"Collectively, data centers are among the topten water-consuming industrial or commercial industries in the US," the paper states.

A lot of that water is potable - that is drinking - water coming straight from the utility. "That's pretty unusual for an industry,” Shehabi said.

38 DCD Magazine • datacenterdynamics.com

“If you put data centers in the same category as other sorts of manufacturing sectors, well you don’t have a steel plant or a textile plant using water from the local utility.

It means that there's more embodied energy associated with that water, because it has gone through the whole treatment process.”

Unfortunately, that large water use disproportionately impacts drier areas. A water scarcity footprint (WSF) is a calculation of the pressure exerted by consumptive water use on available freshwater within a river basin and determines the potential to deprive other societal and environmental water users from meeting their water demands.

"The WSF of data centers in 2018 is 1.29 × 109 m3 of US equivalent water consumption,” the researchers found.

Data centers are more likely to "utilize water resources from watersheds experiencing greater water scarcity than average," particularly in the Western US. Changing the location of a data center can help reduce water use and impact, but may require tradeoffs with efforts to reduce carbon footprint. Equally, some areas could lead to less water usage - but draw it from subbasins

A Dry Article facing higher levels of water scarcity.

"In general, locating a data center within the Northeast, Northwest, and Southwest will reduce the facility's carbon footprint, while locating a data center in the Midwest and portions of the Southeast, Northeast, and Northwest will reduce its WSF." Strategically placing hyperscale and cloud data centers in the right regions could reduce future WSF by 90 percent and CO2 emissions by 55 percent, the researchers said.

"Data centers' heavy reliance on waterscarce basins to supply their direct and indirect water requirements not only highlight the industry’s role in local water scarcity, but also exposes potential risk since water stress is expected to increase in many watersheds due to increases in water demands and more intense, prolonged droughts due to climate change."

Unfortunately, there is no accurate metric for balancing water usage, versus the water stress of a region, versus the carbon intensity of the power. It gets even more complicated when you consider the differences in water used (treated or untreated). “It’s something we need to work on, we need to come up with that metric, and start taking into account how water is being used,” Shehabi said.

The estimates were based on publicly available data on water usage, cooling requirements, and extrapolations based on data center power usage. But for the problem to be tackled, the industry needs to be more forthcoming with data, Shehabi implored.

“I consider it now a major sector like textiles and chemical industry, and for these other industries a lot of data is collected by the federal government,” he said. “And there’s a lot that's even put out by trade groups globally. We know how much electricity steel manufacturing uses in the world.

“In the US, we have a good sense of textile industry power usage, and we have that because it's reported. We don't need a bunch of PhDs coming up with sophisticated models to estimate it.”

Data center companies are often cautious about sharing any information they don’t have to. Many are not even collecting data at all.

This summer, a survey by industry body the Uptime Institute found that only around half of data center managers track water usage at any level. “Most say it’s because there is no business justification, which suggests

a low priority for management — be it cost, risk, or environmental considerations," Uptime found. "Yet even some of those who do not track say they want to reduce their water consumption.”

"We can't even get a baseline of what 'good' water consumption is," Shehabi said.

"Google or Facebook may claim they are very efficient in terms of water usage, but there's no average.

"That's one of those things that I'm hoping we can shed more light on. Because without us being able to know how they're doing, there's really no push for them to improve in that way."

Google often makes local cities sign non-disclosure agreements prohibiting them from disclosing how much water they use.

Sometimes glimpses slip out: Google's Berkeley County data center campus in South Carolina is permitted to use up to 549 million gallons of groundwater each year for cooling, a fact that was only revealed after a two-year battle with local conservation groups to stop the company taking water from a shrinking aquifer.

Facebook is a little more open about how much it consumes.

Planning documents disclosed the scale of its demands at a planned Mesa, Arizona, data center: Initially, the data center will use 550 acre-feet of water per year increasing to 1,100 acre-feet per year for Phase 2 and 1,400 acre-feet per year for Phase 3. An acre-foot is 325,851 gallons of water, so the Phase 1 facility would consume 180 million gallons of water a year, while Phase 3 would require 500 million gallons of water.

This year, Arizona's Department of Water Resources said that water use "is over-allocated and the groundwater is overcommitted. The amount of groundwater rights issued and the amount we are pumping far exceeds our capacity, and Arizona will not reach its 2025 safe yield goal of preserving groundwater."

One of the worst hit by the ongoing droughts, Arizona faces a stark future. In August, the US declared the first-ever water shortage from the Colorado River, triggering mandatory cuts to its supply. The agricultural industry, already suffering, is preparing for a difficult few years as more cuts are planned.

"As we form Mesa's climate action plan and embark upon the first phase of the seven-state drought contingency plan, making cutbacks to agriculture, I cannot in

"We can't even get a baseline of what 'good' water consumption is. Google or Facebook may claim they are very efficient in terms of water usage, but there's no average"

good conscience approve this mega data center using 1.7 million gallons per day at total build-out, up to 3 million square feet on 396 acres," Mesa Vice Mayor Jenn Duff said, in a vote over the Facebook project. She was the only one to vote against the facility.

That's partially because, for all their flaws and lack of transparency, hyperscalers like Facebook, Google, and Microsoft are at least promising to tackle their water usage - even as they target desert regions for their largest facilities.

Facebook claims that it will restore more water than the new data center will consume.

It has invested in three water restoration projects that will together restore over 200 million gallons of water per year in the Colorado River and Salt River basins.

However, such efforts are not without caveats. Some of these projects technically do not replenish lost water - instead, they minimize water loss elsewhere, by planting different crops or fixing leaky pipes. While beneficial, they are ideally things that should be done in and of themselves, and not to offset new water usage.

There is also a finite number of such fixes, which reduces the space for other companies (including smaller data center operators) to do the same.

Facebook hopes to be water positive by 2030, while Microsoft said it would replenish more water than it uses by that date, and Google said it would add 120 percent of its water usage by the end of the decade. In each case, the goal appears to target direct water usage.

Again, there is little transparency into these efforts. "Not all water is the same," Shehabi said. "The water that's being used for irrigation - is that the same water that's actually being used at the data center? Is it coming from the same source? Maybe, maybe not."

None of the companies have to disclose how they are replenishing water, and there is no independent analysis of whether any projects are successful in returning water to the affected regions.

Shehabi hopes some of the companies will be more open about their usage and mitigation efforts. Those worried about disclosing proprietary data, he said, could share it with him in an anonymized fashion.

If not, he warned, spreading droughts and growing public awareness of data centers could lead to regulation that forces disclosure.

"Water might be the resource that pushes there to be more transparency, because it's more critical than energy," he said.

"Things feel more severe when water starts going down, and there's a drought, and people need to start rationing and there's concerns about how that's affecting the entire economy and people's own livelihoods.”

Issue 42 ∞ October 2021 39

Sumitomo Electric Lightwave’s Freeform Ribbon® allows for dense fiber packing and a small cable diameter with a non-preferential bend axis, increasing density in space-constrained applications. For high density and small duct space, Freeform Ribbon® Technology is designed to fit more fiber inside a space. With this technology, there can be a double the fiber within the cable and can help speed up a fusion splicing project within minutes, instead of hours. Freeform Ribbon® is compatible with SEL’s fusion splicers, Splice-On Connectors, and Hardware.

®

QUANTUM Q102-M12 RIBBON FIBER FUSION SPLICER Sumitomo Electric Lightwave announces

Connectors, and Sumitomo Electric’s patented

its newest addition to the industry-leading

dual independent ovens make this fusion

Quantum® Fusion Splicer portfolio. Sumitomo

splicer a preferred choice for extensive and

Electric’s new Type-Q102-M12 is accompanied

diverse optical fiber networks. The new ribbon

with new features and benefits capable of

fiber splicer is capable of handling 250um and

deploying next generation hyperscale networks.

200um ecosystems and is accompanied by SEL’s 24/7 fusion splicer technical support and

Consistent and quality low-loss splicing, the

a three-year warranty.

compatibility with Lynx Custom-Fit® 2 Splice-On

FEATURES & BENEFITS Dual Independent Ovens for Productivity

Active ACAS • Optimizes splice loss 0.05 dB

Reduce Your Work Time • Splicing 11s • Heating 35s

Touch Optimized User Interface and High Resolution Display

FEATURED ACCESSORIES FHM-12 Holders, 12 ct. Ribbon Fiber, Mass Splicer

FC-8R Cleaver, Handheld w/ Auto Blade Adjust

JR-6+ Jacket Remover, Plus, Heated, DC-in, Lithium ion Battery

For more information on the Q102-M12 and our other fusion splicers, please visit SumitomoElectricLightwave.com

Dan Swinhoe News Editor

NetWalker: A year on from Equinix’s ransomware incident Equinix CSO Michael Montoya talks to Dan Swinhoe about its ransomware incident

R

ansomware is big business.

Cybercriminals know that companies hate to see operations grind to a halt, and compromising one machine can quickly and easily bring down entire networks.

The likes of CyrusOne, iNSYNQ, A2 Hosting, SmarterASP.NET, and DataResolution.net were all hit with ransomware during 2019, with many of

those seeing customers impacted.

Last year Equinix became the largest data center victim of ransomware in recent memory. But unlike many of its peers, it managed to contain the attack quickly and keep its customers’ operations unaffected by the incident.

“The investments that we made as a company over the past several years paid dividend in our own ransomware attack because that lateral movement into our IBX

42 DCD Magazine • datacenterdynamics.com

facilities was not possible thanks to a lot of the things that the company has done over the past several years,” Equinix’s CSO Michael Montoya tells DCD. The attack begins On September 9, 2020, Equinix posted a statement saying it was ‘investigating a security incident that involves ransomware on some of our internal systems.’

The company moved quickly to contain

Colo Security

"The biggest impact that we had is they were able to get access to some file servers that were in the process of being moved to the cloud" and remediate the incident. After a month of investigation, it found that although some information accessed included internal references to Equinix customers, the data contained no sensitive information on customer operations or other material customer information.

Via previous stints at FireEye and Microsoft, Montoya said he had dealt with thousands of security incidents over the years through helping clients. But this was his first as CSO. “It definitely feels very different when you're on this side of the table. There's no doubt about that,” he said.

Where CyrusOne’s 2019 ransomware incident reportedly affected six managed service customers served primarily from its New York data center, at no point were customer operations within Equinix facilities affected.

Equinix hasn’t released much information publicly, but according to a report by BleepingComputer a few days after the attack, the company was hit with the NetWalker strain of ransomware, and attackers asked for $4.5 million in ransom.

The publication reported that systems affected held financial information, payroll, accounting, audits, and data center reports.

What is NetWalker ransomware?

According to Crowdstrike, the NetWalker ransomware variant was reportedly created by a Russian-speaking cybercrime group known as ‘Circus Spider’ in 2019. The malware is sold to criminals via an ‘asa-Service’ model where buyers rent the capabilities from the creators for a fee or percentage of profits.

NetWalker encrypts files on the local system, maps network shares, and enumerates the network for additional shares, attempting to access them using the security tokens from all logged-in users on the victim’s system. Attackers often follow up the initial encryption of data with the threat to release information publicly.

The University of California San Francisco (UCSF) was another victim of NetWalker and revealed that it paid roughly $1.14 million in order to recover its data.

The Australian logistics giant Toll Group, Pakistani power utility K-Electric, and a number of healthcare organizations have also been hit by the ransomware.

ChainAnalysis said it had tracked more than $46 million worth of funds in NetWalker ransoms since it first came on the scene in August 2019 across hundreds of victim organizations.

In January the Department of Justice, alongside law enforcement in Bulgaria, said it had arrested Canadian national Sebastien Vachon-Desjardins of Gatineau and had taken down the portals used by NetWalker ransomware affiliates to provide payment instructions and communicate with victims. According to Canadian press, Vachon-Desjardins is being denied bail while he awaits extradition to the US. He was described as a ‘key player’ to NetWalker’s operations and a likely flight risk if released.

Equinix falls victim Montoya says the attacker was able to get into Equinix’s network through a ‘configuration management deviation’ in one of its cloud environments, which allowed a threat actor to get in through via a Remote Desktop Protocol (RDP) session.

“That RDP session was misconfigured, and in a cloud environment that maybe didn't have the right level of management oversight. And, as a result, they were able to get into our environment and make some pivots. The biggest impact that we had is they were able to get access to some file servers that were in the process of being moved to the cloud.”

Montoya says the information the threat actors accessed were in the process of being moved to a file storage system and were not business-or mission-critical, but also acknowledges they were able to gain access to “some of our IT management systems, like our software distribution systems and a few other systems.

“But once they moved from this cloud environment into our core environment, we were able to detect very quickly within a few hours. So our defense mechanisms

worked with precision.”

Montoya said the company was able to get all the encrypted data back using its own backups, and at no time did users not have access to their email, the corporate environment, sales information, customer information, etc.

“The business operated with no interruption. The biggest impact to our user base is we did force a password change to our users, and that's just more of a hygienerelated activity. We were able to operate businesses as normal because the blast radius of this impact was pretty small and it was pretty focused on some file systems,” he explained.

Equinix’s response According to IBM, the average time to identify a breach in 2020 was 228 days, while the average time to contain a breach was 80 days. It’s not uncommon for firms to become aware of incidents via thirdparties.

Though the attacker was present for almost three weeks, Montoya says Equinix were able to self-detect the incident within six hours once the threat actor weaponized their attack.

“We were able to basically detect this within 19 days, which is still horrible because there's somebody who potentially had access for 19 days,” says Montoya.

“But the material time where that attack was being weaponized was less than six hours before we detected it. So I think our response time from an industry perspective was pretty amazing.”

Equinix utilizes the OODA Loop (Observe, Orient, Decide, Act); a fourstep process taken from the military and commonly used in cybersecurity. So rather than instantly act, the data center firm’s security team waited before acting.

“Within 15 minutes of that detection we had our full CERT (computer emergency response team) on the incident, observing what movements are they making, where they are trying to do some stuff. We did a lot of observation because we tried to figure out what type of attack is this, what are they doing and wanted to learn something about the threat actor here. And in that learning, we observed for some time and oriented ourselves in a way that we could make the right decisions on how we respond.

“The business operated with no interruption. The biggest impact to our user base is we did force a password change to our users, and that's just more of a hygiene-related activity" Issue 42 ∞ October 2021 43

"We want to even have greater levels of resiliency in our backups and we took additional measures there as well” “Within an hour, we started our response to that. Once we started our response after we did the right level of observation, we were able to one get a live body on the other side, [and] force them into moving faster on what they were trying to accomplish.”

Initial containment was achieved that day, within around eight hours of initial detection. And full containment was declared within a few days.

“We then pulled in our crisis plan, which included bringing in a third party Incident Response provider to help us do a formal investigation and response, pulled in obviously outside counsel, activated our full executive team and our board, so that we could properly respond to this from not necessarily the technical perspective but all the things that come with a breach right in terms of communication, etc.”

The company also created a private GitHub repository where it just shared a lot of intelligence with its customers in the security community.

Montoya says he is a “big believer” in intelligence sharing and that intelligence was a way to help partners while ‘intelligence engines caught up’.

Never let a good crisis go to waste The relative lack of impact was a testament to previous investments and proof the current security strategy was working, but as with any incident, there were learnings to be had.

“The biggest issue for us honestly was just more urgency. We had the right strategy, we just needed to get more urgency around some key areas,” says Montoya. “Even though our segmentation was a big proactive thing for us, we're taking even additional controls and efforts around segmentation to even give us further degrees of protection and capabilities there.

“We've improved our backup strategy; although our backups were a big advantage because we were able to restore from backups, we recognized that we want to even have greater levels of resiliency in our backups and we took additional measures there as well.”

Ransomware best practices: The basic things that all companies should do Segment: The more separated systems and networks are, the harder it is for attackers to pivot from their entry point to their eventual target. Keep networks as separated as possible, and only put onto the Internet what needs to be.

Patch: Harder than it sounds for large networks, but having a well-defined, risk-based patching process for systems and ensuring it is stuck to can prevent high-risk systems from being compromised by attacks that could have been easily prevented.

Authenticate: Using multi-factor authentication and technologies such as zero-trust authentication and Single Sign On (SSO) can help protect users from having accounts hijacked and flag any potential suspect log-ins.

Know your enemy: Some actors will happily target anyone and everyone, but many have a narrow set of targets and a well-defined set of tools and techniques.

Understanding who might want to attack you, why, and how they would do it can help you take a risk-based approach and better prepare for more likely scenarios.

Conduct Pen Tests: Conduct regular penetration tests to spot potential entry points into your network. Consider running bug bounty programs for a more continuous threat program.

Monitor: You can’t protect what you can’t see. Having a granular view of systems, data, and users is key to spotting potentially malicious behavior. Try and understand what ‘normal’ looks like on your network to spot anomalies.

Have backups: In the event something goes wrong, having a recent and readilyavailable backup is important. Test your backups regularly; having slow or outdated backups are the reason many companies decide to risk paying ransoms.

Have an Incident Response plan: Knowing how to respond to an incident can speed response and quicken containment, meaning less damage is done. Having a well-defined incident response plan, playbooks for different scenarios, and having a specialist IR company on retainer can help get a jump on attackers.

Practice, practice, practice; No plan survives contact with the enemy, but regularly testing your plans via war games and tabletop exercises can help better define roles in emergencies and help identify potential weak spots. Regular testing of both frontline staff as well as executives and other key stakeholders is important; everyone has a role to play in such events.

Communicate: Customers don’t want to be left in the dark. Be open and clear with customers and partners about what is happening. Even if you don’t have all the answers, it's better than radio silence.

Retiring those files servers more quickly was a key lesson, and though Equinix was already on a roadmap to adopting zero trust, more speed was needed there as well: “We've accelerated all of our technology refresh plans around zero trust from a multi-year strategy to a fraction of that time as a result of this event because we realized that we had the right strategy, but we needed to move faster.”

In addition, the company decided to bring its engineering function and security operations under the same structure so the company can take more of a continuous engineering approach to security operations and perform continuous

44 DCD Magazine • datacenterdynamics.com

instrumentation on its technology stack.

“That's been of huge value for us to sort of in our approach to automation; creating this more continuous engineering loop between our security engineers, and our security operators.”

Overall, the attack on Equinix could have been far worse, but this was largely due to the efforts and decisions made before the incident, not during or afterwards.

“The defenses we built over the years, not just months or days, were critical in helping us respond I think very effectively,” reflects Montoya. “We know our assets, and we've done a lot of segmentation and hardening to restrict lateral movement.”

Under the Sea

Submarine cables find new impetus under hyperscalers The tides of subsea cable trends are changing rapidly amid new investments and development

T

he first submarine cables – built to send telegram signals – were laid in the mid-1800s and helped create the foundations for today’s Internet infrastructure.

The first submarine telephone cable, TAT-1, was laid between 1955 and 1956.

A joint project, the cable was paid for by a consortium of the UK Post Office (which then housed BT), the American Telephone and Telegraph company (now AT&T), and the Canadian Overseas Telecommunications Corporation. The first fiber optic cable came a little over 30 years later, and again was laid in partnership between AT&T, France Télécom, and British Telecom.

Today there are almost 500 subsea cables in operation or development. And while telecoms companies are still very much part of the picture, trends are increasingly shifting away from telco consortiums and more towards cable projects led by Over-The-Top (OTT) hyperscalers.

Just as how they making some of the largest investments in data centers, the likes of Facebook, Google, Microsoft, and Amazon are

investing hundreds of millions of dollars in a multitude of cable projects across the world and changing the industry in the process.

“Having accounted for less than 10 percent of total usage prior to 2012, Google, Facebook, Microsoft, and Amazon are currently invested in over 40 submarine cables around the world, supporting somewhere in the region of 250Tbps each,” says Brian Lavallee, Ciena submarine networking solutions expert.

“Apple, Dropbox, Netflix, Alibaba, Akamai, and Limelight represent much of the remainder.”

Hyperscalers take to the seas Where once telecoms and communications companies ruled the seas and shared ownership of subsea fiber, the hyperscalers

Dan Swinhoe News Editor

are increasingly investing in submarine cables both as part of traditional consortiums and as private projects. Where they were once major customers of subsea capacity, they are now peers or even leaders on new projects; the four companies have invested in more than 30 cable systems over the last ten years.

“20 years ago they were all consortium cables; all the traditional telcos all chipped in and created a consortium group, and they funded these projects,” explains Gil Santaliz, founder and CEO of New Jersey cable landing station NJFX. “Today, the folks that are supporting these projects being built are smaller groups, and they're the companies that are social media and cloud companies, because they realized they needed the owner

“Having accounted for less than 10 percent of total usage prior to 2012, Google, Facebook, Microsoft, and Amazon are currently invested in over 40 submarine cables around the world" Issue 42 ∞ October 2021 45

economics, they needed to control the architecture.”

One of the biggest hyperscale investors in subsea cables is search engine giant Google.

According to SubmarineNetworks.com, Google has invested in at least 19 submarine cables since 2010, including six private cables.

Google’s first submarine cable investment was the Unity cable around 2010. The search giant joined a consortium of Bharti Airtel, Global Transit, KDDI Corp., Pacnet (now Telstra), and SingTel to build a cable connecting Chikura, Japan to Los Angeles, US. Today the company is making largescale investments in both private and consortium cables that are amongst the biggest in the world and has a number of other cables in development; The 16 fiber pair, 350Tbps Grace Hopper cable will connect the US, UK, and Spain, and is due to come online in 2022. Its Equiano Portugal-to-South Africa cable is due for completion later in the year. Announced in June, Firmina will run from the East Coast of the US to Las Toninas, Argentina, with landings in Praia Grande, Brazil, and Punta del Este, Uruguay.

“Systems built even 20 years ago lack adequate capacity and important new technical capabilities, and can no longer manage the job,” says Mark Sokol, senior director of infrastructure at Google Cloud. “Updated technology is required to address this continued growth in demand for capacity, driven by cloud computing, future Internet advances, backend replication, and the many breakthrough innovations that lie ahead.

“We are building out our network to provide the best possible experience to our end users and customers. By designing and funding subsea cables we are able to plan effectively for the future capacity needs of our customers and users around the world, and add a layer of security beyond what’s available over the public Internet.”

The search giant’s Virginia-to-France Dunant cable went live earlier this year. The company claims the 6,400km 12 fiber-pair cable is capable of 250Tbps, making it the world's highest capacity submarine cable. That will be surpassed by the company’s 340Tbps UK-US Grace Hopper cable, which recently landed in Cornwall and is due to come online soon.

Even if new cable projects are funded through consortiums, the OTT hyperscalers are happy to partner together; Facebook has partnered with Google, Microsoft, and Amazon for different cable projects.

Google and Amazon have both invested in the US-Denmark HAVFRUE cable, while Microsoft and Amazon both own fiber pairs on the Spain-US MAREA cable. Facebook is part of both projects, as well as a number of others including the Echo and Bifrost systems to connect the West Coast US to Singapore and Indonesia.

"By designing and funding subsea cables we are able to plan effectively for the future capacity needs of our customers and users around the world, and add a layer of security beyond the public Internet” “These companies have dedicated teams and in-house talent that know how to manage a project,” says Santaliz. “They know how to partner with others for like-minded opportunities, and they're working on multiple projects at the same time. It's not the way it was 20 years ago when you have 22 companies involved in a project that met four times a year on one project.”

While the hyperscalers are happy to share capacity on cables, it’s unlikely they’ll start reselling capacity. Alan Mauldin, research director at TeleGeography, previously told Wired that they "would never do that” because they would then become a carrier and subject to being licensed as a carrier. In the same piece, Urs Hӧlzle, a senior vice president of technical infrastructure at Google's Cloud division confirmed that Google “doesn’t want to be a service provider as an ISP” and is going it alone because consortiums can slow the process of building new cables.

Hyperscalers want control of cables and capacity The drive by the hyperscalers and other OTT providers is driving up cable capacity at a rapid pace; Telegeography.com reports that the amount of international capacity deployed by content networks rose over nine-fold, to 962Tbps, between 2015 and 2019. The research firm says that having accounted for less than 10 percent of total usage prior to 2012, content providers’ share of total cable capacity was over 65 percent in 2020.

“ISPs are building their own submarine cable infrastructure to increase capacity, offer richer services, and deliver international access to their coveted content,” says Ciena’s Lavelle. “This is certainly having a positive effect on the wider submarine cable industry, as it results in more competition and route diversity for increased availability, therefore accelerating access to increasingly reliable global connectivity.”

As well as new cables, existing cables are being continually upgraded with new technology. Following a trial earlier this year, the trans-Atlantic MAREA cable – originally designed with a capacity of 160Tbps – saw its potential capacity increase from an already upgraded 200Tbps to 224Tbps.

Santaliz says much of the rapid progress being delivered in cable technology is being driven by the hyperscalers demanding more of

46 DCD Magazine • datacenterdynamics.com

their suppliers.

“Google and Facebook have really been the pioneers and pushing these companies like SubCom and Alcatel to up their game,” he explains. “They've been asking them for better solutions, they've done a great job of hiring talent inside their companies to push the envelope and get the manufacturers to invent more. They've reinvented the industry in the last five years.

“They're no longer just sitting there waiting for a proposal, they're driving innovation they're driving applications are sitting side by side with the telecom operators, and in some cases they're telling them here's a better way of doing it.”

Santaliz says one of the reasons the large hyperscalers are investing in cables is to ensure continuous uptime and connectivity; lost connectivity hurts the brand.

“Whether your brand is social media or banking you need to make sure your network is always available. This round of investments for subsea networks [is the likes of] Facebook, Google, Microsoft; [but] I'm even starting to see US banks invest in subsea cables.”

Will other OTT players want to invest in cables?

With the large cloud players only growing their investments in data center infrastructure globally – Microsoft alone has said it’s on course to build 50 to 100 new data centers each year for the foreseeable future – their demand for new cables and increased capacity is unlikely to be satisfied any time soon. But will other large companies follow?

Santaliz says only the largest enterprise customers that will take the time to learn the subsea industry and learn the architecture to do it themselves on subsea cables, but notes that the banking industry is one that could be next.

“The banking industry has always been just a customer to telecommunications, and they've been watching the OTTs go from being customers to partners. Transactions are important; the idea of not being able to process a payment or finalize a trade or close on a deal because your app’s not available, means they realize that 'never down' and their brands are related.”

We may already be seeing it; Morgan Stanley has bought 49 percent of Altice's regional fiber network. And Santaliz notes that while they might have bought the infrastructure as a

Under the Sea financial investment, it is certainly also going to improve the firm’s digital resiliency.

“Morgan Stanley's not going to have an outage on anything related to New York Metro on that network because they have an owner’s perspective now they're not just the customer anymore.”

While the biggest OTT players have been investing in data centers and cables, smaller content companies have been more focused on Content Delivery Networks (CDNs) and ensuring their content is cached close to their customers. But could that change in the future?

Derek Long, head of telecoms and mobile at Cambridge Consultants, has previously discussed with DCD the possibility of large content providers like Disney jumping into Internet infrastructure through High altitude pseudo-satellites (HAPS), perhaps the same could happen with subsea cables.

“If we will see more OTTs outside of the ‘Big 5’ take this route is unclear, but it is highly likely within the next five years,” says Mattias Fridstrom, chief evangelist at Telia Carrier.

Where do telcos fit in an OTT world?

As well as driving innovation around the technology, the rise of OTT players investing in their own cables means the rest of the cable industry must also adapt how they do business.

In July, Global Cloud Xchange canceled its Italyto-Singapore Eagle cable because of a lack of an anchor tenant.

“The old adage of ‘build it and they will come’ no longer works,” CEO Carl Grivner recently told Capacity.

“There was nothing wrong with Eagle – just no anchor tenant. It’s an OTT [over-the-top] world and if you’re getting a consortium of wholesale carriers together you need an anchor tenant. I looked at the business case and it wasn’t working.”

For the ISPs, Ciena’s Lavalle says he expects to see a 16 percent decline in wholesale pricing year-on-year, and a 28 percent decline by 2027.

“ISPs are looking for more capacity in addition to lower costs per bit to remain financially viable,” he says. “Pricing is, therefore, the main threat to business.”

As incumbent cables are retired and hyperscalers make more direct investments in new cables, the traditional economics for telcos may not stack up as they once did. But despite this, they still have a role to play in the industry.

“On the larger traffic paths (cross-Atlantic and cross-Pacific), I think we have seen the

last consortiums with telcos involved,” explains Fridstrom. “Most new cables will be built by one or a few owners with a fiber pair as the least denominator.

“On more un-traditional routes to ‘newer’ areas like Africa, I can still see Telcos being involved to secure a lower cost path towards other regions for their domestic traffic. In Europe and North America, Telcos needs to focus on meeting up subsea cables for further transport inland on their land networks.”

Santaliz says the carriers aren't going to be the ‘go-to guys’ when it comes to innovation, and will instead invest in “interesting architectures.” An example he gives is Telecom Italia and Omantel’s Italy-to-India Blue-Raman cable, which bypasses the overcrowded Suez Canal by going overland into Israel for a portion of the cable. Google is also involved in the project.

“They're going to be rewarded because they were innovative and they were able to solve for that issue. That's where the carriers will find their place and in providing their traditional values of point-to-point solutions and partnering with the OTTs and partnering with the enterprise.”

Gavin Rea, CTO at Gulf Bridge International (GBI), notes that some cable owners have decided not to involve OTTs directly in their cables, and much depends on the kind of relationship the carriers want with the OTT players in the future.

“Each cable owner needs to make their own decisions on whether to work directly with the OTTs as partners in a new cable or maintain a customer relationship selling them capacity.”

In the longer term, Michael Ourabah, CEO of infrastructure and connectivity provider BSO, warns the ecosystem around cable operations needs to grow in order to keep up with demand, lest the hyperscalers take over.

“The supporting ecosystem of repair assets like cable repair ships and specialist equipment vendors will not have enough capacity to support the growing volume of subsea cables; that could lead to the development of a twotier system where high-volume routes run by hyperscalers stockpile repair assets and are well served by them but lower volume routes are not well provisioned and deteriorate,” he says.

“The only way to remain competitive for consortiums will be to cooperate with OTTs and hyperscalers and look to offer complementary services that extend the reach of networks to areas where cables cannot yet readily reach.”

“Google and Facebook have really been the pioneers and pushing these companies like SubCom and Alcatel to up their game.. [and] to push the envelope"

Subsea cables as an alternative to intercountry ground fiber While hyperscalers are investing in a number of large-scale international projects, there’s still plenty of room for carriers to invest in ‘alternative’ routes, and Santaliz suggests domestic subsea cables could be one such area of opportunity.

In the US, the Confluence-1 cable will run across the US eastern seaboard from Wall Township, New Jersey to Miami, Florida via Virginia Beach, Virginia; Myrtle Beach, South Carolina; and Jacksonville Beach, Florida. The cable will run to 2,571 kilometers, providing 24 fiber pairs and offering more than 500Tbps in capacity once complete.

“In the US, we never dreamed that we needed to go in the water to connect between the US, we assumed that we'd use railroads, use highways, and we would just interconnect the US with those arteries that we've always had, because it was good enough,” says Santaliz.

While there have been some intercountry subsea cables in the likes of Brazil before, they are becoming more common and Santaliz says this trend is due to continue. Australian company Fibre Expressway is planning a 10,000-kilometer, 16-fiber pair carrierneutral subsea cable that would have seven cable landing stations between Perth and Darwin in the northwest of the country before running north to Indonesia, Singapore, and Malaysia.

In India, Reliance Jio’s IAX will link Mumbai in the west with Chennai in the east via Sri Lanka, with the possible addition of Vizag in the future, before landing in Thailand and Singapore.

Santaliz says that amid evergrowing capacity demands, subsea cables now offer the economics and resiliency that make domestics subsea cables increasingly viable, especially in and around the US. “What we're realizing now is that it's actually more economical and it's safer to put a cable in the water to connect the US north and south. If you had brought that up 10 years ago, you'd get laughed out of the room, now the reality is the economics are better to be in the water, you're going to get a more resilient network.”

Issue 42 ∞ October 2021 47

Batteries play a key role in the transition to a green world FIAMM complete battery solutions for data center projects across Europe

T

he world is rapidly

are something that are taken for granted and

FIAMM Reserve Power Solutions’ customer-

transitioning away from

the important role they play in our world is

oriented approach

carbon-based power to a green

often overlooked. Their purchase is often seen

FIAMM Energy Technology is a global name

future. Recent environmental

as a last-minute decision in the design stages

in the supply and development of batteries

events across the globe have

of most projects. But early planning leads to

for industrial use. The company has been

escalated the imperative to

better solutions.

manufacturing batteries for over eighty years

change even faster. The data center industry

Lead-acid chemistry battery is the

and has an extensive suite of product types

is stepping up to this challenge and is already

workhorse of the global battery industry. Since

used in a wide range of applications including

setting an example for many other industries

its early development over 100 years ago, it has

UPS, telecommunication, rail, and general

to follow.

undergone many changes and improvements

industry.

Batteries are a critical link between

such as longer life, no need for maintenance,

generated power, and when the source is

and higher levels of energy and power. In the

the design of a solution for data centers,

intermittent and reliant on nature. A small

UPS industry, lead-acid batteries are the first

each have a unique need. Each customer

power spike or a momentary interruption of

choice for UPS systems, and data centers in

application needs to be carefully defined and

power from a public utility or other sources

particular.

a specific solution offered, based on a range of

can lead to a break in the data transmission.

Batteries are used to fill this instantaneous need.

Further reasons for its popularity are its

Industrial battery applications, including

criteria that includes life expectancy, operating

availability, the ability to be fully recycled,

temperature range, maintenance routines,

offering residual value, and a strong safety

space availability and budget.

Lead-acid batteries still dominate - safe, 100

from the risk of fire. Furthermore, as a result of

percent recyclable and proven by time

industry cooperation and conformity, there are

network is set up to provide this deep level

A battery is a combination of chemistry and

standardized footprints that makes switching

of service both before- and post-sale. The

mechanical design. For most people, batteries

and replacement easy for customers.

company has offices in major regional centers

48 DCD Supplement • datacenterdynamics.com

The FIAMM sales and project engineering

FIAMM | Advertorial and it is also supported by a strong and fully trained distributor network. All sales and product solution services must have a strong engineering-based knowledge to be able to engage with customers and interpret their specific need.

FIAMM lead-acid batteries have many key features for customers. The company not

The role of batteries for the future is of the utmost importance as the energy world continues to transition to green solutions and we move away from fossil fuel power sources.

only supplies batteries but all the components necessary for a complete installed solution.

This includes racks, monitoring systems for remote web-based monitoring, and many other accessories. The company has also moved forward meeting with the growing needs of the data center industry where operating room temperatures are high.

FIAMM has recently introduced a new hightemperature product designed to improve life in higher operating conditions.

A FIAMM solution: 500kW with 15’ back up time – part of the total solution.

on a combination of photovoltaic panels

‘Blackbox’ which can be moved around an

and a hydro plant sourced from the local

installation or even other sites as required.

and abundant river water supply making it a

1MW N+1 Stackable power modules inside the Aruba Center Data centers are a major investment and a UPS system is a significant part of that cost.

showcase of the industry.

The site concept was developed in 2016 and

The FIAMM-Aruba story is just one example of how the company develops a solution.

Customers are encouraged to assess battery

began operational rollout in 2017. The full case

backup needs early in the development of a

history of the site and its development can be

data center or other applications. This early

seen in a YouTube video.

approach can assist in design considerations

The solution must offer operational integrity

FIAMM Energy Technology worked with

as well as value. The FIAMM products have a

the Aruba team at an early stage to provide the

proven record over many decades.

best battery solution. Part of the requirement

and best practices to get the best life performance from the system.

The role of batteries for the future is of

was to provide an expandable solution as the

the utmost importance as the energy world

FIAMM and Aruba… a case history

independent data centers within the campus

continues to transition to green solutions and

Aruba is a European provider of cloud and

were developed and come online. All design

we move away from fossil fuel power sources.

data center services. The company already

and development were done entirely by the

FIAMM Energy Technology is committed

operates with eight major data centers based in

Aruba team.

to developing solutions that support this

six European countries and offers customers

To accommodate the growth of the

transition. The company’s products meet all

a cloud-based network with enterprise levels

different inner data centers, an ingenious

the key drivers of this transition as they are

services and localized support.

solution of stackable modular containers was

proven, safe, fully recyclable.

FIAMM Energy Technology was chosen as the battery supplier for a new Italian data

developed. Each one comprising Batteries, UPS, and PC system.

center campus developed by the company.

The site, when fully operational, will have a maximum power requirement of 90MW

A Blackbox battery solution

making it the biggest data center in the Aruba

FIAMM Energy Technology believes the

family as well as the largest campus of its type

concept of a modular stackable UPS system

in Italy. The site is located in Northern Italy

has great potential for the future. It means that

and it is powered by 100 percent green energy

the company can effectively supply a drop-in

Issue 42 ∞ October 2021 49

Processing the universe Exploring the computing and networking requirements of the Square Kilometer Array

D

ecades in the making, construction has finally begun on one of the largest scientific endeavors in human history.

The SKA Observatory, better known as the Square Kilometer Array, will be the largest and most sensitive radio telescope in the world, collecting more data on our universe than ever before.

And all that data will need to be processed.

Across a huge tract of land in Australia, hundreds of thousands of Christmas treeshaped antennas are set to be installed to process signals from the dawn of time, in the low portion of the radio spectrum.

Over in South Africa, hundreds of large satellite dishes will record mid-spectrum,

Sebastian Moss Editor

"It's a journey of tens of thousands of kilometers for a piece of data, to go from the analog radio wave, to being turned into something digital, to being turned into something scientifically useful" also unlocking the mysteries of the universe.

SKAO Low and Mid will operate in tandem on different continents, each spread over thousands of kilometers, requiring a careful and complex collaboration.

"In the Low case, once we do the digitalization of the signals, we're producing raw data at the rate of about

50 DCD Magazine • datacenterdynamics.com

two petabytes per second," SKAO's computing infrastructure lead Miles Deegan said.

"We then go through several stages of reducing that data - we've got some FPGAs, and what have you - we do a little processing, and then from the deserts we go to Perth and Cape Town respectively.”

In those cities, large supercomputers will crunch what comes in - with the two

The Stars Like Data

‘Science Data Processor’ HPC systems expected to total at least 250 petaflops, although that number could still rise.

“They will do further reduction on this data, and hopefully turn it in something which is scientifically useful,” Deegan said. That useful data will accumulate at the rate of about 300 petabytes per telescope per year, once the system is in full operations from 2027-2030.

"And then that's not the end of the story,” he said. The data has to be sent to the scientific community around the globe. “So we will need dedicated links to distribute this 300 petabytes per annum to a network of what we call SK regional centers.

"It's a journey of tens of thousands of kilometers for a piece of data, to go from the analog radio wave, to being turned into something digital, to being turned into something scientifically useful, and then to be sent off to the scientists."

That journey first has to start at the dish or antenna, and be heavily processed and compressed before it travels to the data center.

That's handled by the Central Signal Processor (CSP), which is actually a number of different semiconductors, algorithms, and processes, rather than a single chip.

Some of this is done on site, in a desert - radio astronomers have to build in remote regions because even the smallest of manmade radio emissions can drown everything out. That means dealing with

extreme heat, all while trying to not emit anything.

"We've got the RFI challenges; that gives us a bit of a problem," SKAO network architect Peter Lewis said of the equally difficult data transfer efforts.

"If you look at the SKA Mid, within the dishes there's a kind of a Faraday cage, a shielded cabinet which has got no air-con in there. We're gonna put a load of kit in there in the middle of nowhere where temperatures can get to 40°C (104°F).

"We're looking at putting industrial

Ethernet switches in where we can. Where we can't, we just have to use standard technology… it will just be running fairly hot.”

With SKA Mid, the group has taken the decision to shift a bunch of the CSP out of the desert and near to the Cape Town data center.

"We are in the process of actually trying to move some stuff around," Deegan said.

"In the case of the Low telescope, we're just about getting around finishing off the analysis as to whether we should make the same move. My guess is it will happen, but the jury's still out.”

It is perhaps fitting that a project aiming to study the beginnings of the universe has required a level of long-term thinking and planning humans generally aren't that well suited to.

First conceived in the early 1990s, designed in the late-2010s, the SKAO has gone through numerous iterations - whether it be because of new ideas, political infighting, or technological advancements.

This has made designing the compute and networking infrastructure for a shifting project set years in the future quite challenging.

"We did some designs back in 2017 and, clearly, we're not going to deploy the things that we designed then, because technology's just moved on so quickly,”

"When we first started, the data rates were quite enormous. With a project that's taking this long, obviously, technologies have kind of caught up" Issue 42 ∞ October 2021 51

A different kind of network

Lewis said. "Even the designs that we're looking at today, will be different by the time it goes live."

It’s a constant challenge, he said, of dealing with “people asking us what does the rack layout look like, what's the power draw, what cooling do you need? And all we can do is give what we can do today, with some kind of view as to how the power curve will drop over time.”

But, in other ways, time has been the friend of the networking and computing team at SKAO, which in 2013 thought it would have to have a fiber network with greater capacity than global Internet traffic.

"To be honest, when we first started, the data rates were quite enormous," Lewis said. "With a project that's taking this long, obviously, technologies have kind of caught up."

In those early days, it seemed like there would be simply no way to transfer the data far from the dishes.

SKAO helped fund projects like the IBM Dome in 2014, which was a hot watercooled microserver that they hoped could process 14 exabytes of data on site and filter out one petabyte of useful information.

Dome was ultimately a failure, although parts of the project found their way into IBM's (now waning) HPC business, including the SuperMUC supercomputer in Zurich. But the amount of data has changed, as has the ability to send it over greater distances, meaning that SKAO doesn't rely on something like Dome to be successful.

That could change once again. "The ambition is to ultimately build a second phase of these SKAO telescopes, and maybe even further telescopes,” Deegan said. “The observatory is supposed to have a lifetime of at least 50 years and things aren't going to be static “If we can successfully deliver these two telescopes, we will then look to expand them further. We’re looking at a factor of 10 increase in the number of antennas and that really increases the data and the amount of computing for algorithmic scaling reasons, especially if you start spreading these things out over large distances,” he said.

“It’s not like we will deliver whatever is needed on day one, and then just feed and water those systems and plod along for forevermore. It's just gonna be an ongoing problem for 50 years."

52 DCD Magazine • datacenterdynamics.com

"This is not like any data center - if you were to go to an AWS facility or an on-prem site, it would look nothing like it," SKAO's networking engineer Peter Lewis said.

"Data's going one way, there's nothing coming back, essentially," he explained. "You don't need different policies to require different users, it's a completely different scenario than you would see in a normal data center."

After working at Cisco for 20 years, Lewis described the network as "quite unique," adding "everything is one way, the data is going left to right, there's virtually nothing coming back."

The dishes and antenna produce terabytes of data, but need barely anything back in return. "You have to think about it differently - all of your experience in traditional networks and data centers is kind of useful, but it's very different."

The systems do require some data and interconnection, however. "We have to provide a very precise timing across the array," Lewis said, to ensure that all the systems are working in tandem. This is handled on a separate, smaller, network.

Computing infrastructure lead Miles Deegan added: "You need exquisitely accurate timing - at the picosecond."

Take a Bath

Giving Victorian baths a second life An abandoned Victorian bathhouse in Greater Manchester, the Ashton Old Baths has now reopened as a council-owned data center

M

ost data centers aren’t very aesthetically pleasing; most are non-descript boxes on the edge of town.

But occasionally a new data center catches the eye for its visual impact.

Ashton Old Baths in Greater Manchester UK is certainly one of those. A 19th-century public swimming baths house built in Greater Manchester, UK, the building is Grade II* listed by Historic England for its architectural merit, putting it on a par with the iconic Battersea Power Station in London.

Despite its beauty, the baths in West Ashton-under-Lyne lay empty for 40 years.

But they’ve now been converted into a tech

hub, offering more than 10,000 sq ft of office and coworking space. In its most recent phase of development, a new data center was added to the building.

As well as housing local council and NHS infrastructure, space in the new facility will be offered commercially through a cooperative. Tameside Council said this data center was less about money and more about providing the digital infrastructure needed to enable and keep digital businesses in the local area.

Old Ashton Baths: A tech hub with history Built in 1870, the Ashton Swimming Baths were designed in the Italianate style of architecture by architects Paull and Robinson.

Dan Swinhoe News Editor

The building boasts Flemish bond brick with slate roof and stone dressings and Historic England described it as an “important early example of a municipal swimming bath” having “exceptional forceful architectural massing and use of stylistic features.”

The swimming pool was said at the time to be the second largest covered bath in Europe. Operating as a bathhouse during the entire time it was open, it also hosted concerts as an orchestra venue, a tennis club, and a restaurant.

The Baths closed in 1975, after which it changed hands a number of times. Grade II* listing restricts the changes that can be made to a building, and it was years before it found a new permanent owner or new use.

Issue 42 ∞ October 2021 53

For some years Historic England listed it on its Heritage at Risk register as an important building at risk of being lost.

“It was a beautiful Victorian baths, which up until about 30 years old was still in use, but since then was going downhill very rapidly,” according to Tim Rainey, assistant director of, Digital Tameside, a division of Tameside Metropolitan Borough Council (TMBC).

Manchester-based developer PlaceFirst bought the site from Ask Developments in 2014, and, began the task of transforming it into its new role. Listed buildings can be eligible for grants, and the project got funding from the Heritage Lottery Fund, the European Regional Development Fund, and the Borough Council.

Sara Hilton, head of the Heritage Lottery Fund North West previously said: “Ashton Old Baths is a fantastic example of 19th-century architecture which has served and inspired generations of local residents. Its fascinating history is tightly interwoven with the social history and culture of Ashton and Tameside.”

Scrubbing up The council bought the site from PlaceFirst

and the renovated building reopened in 2016.

Paul Stevens, director at project architects MCAU called the development an “amazing opportunity to form a catalyst to help regenerate the whole area in Tameside.” The conservation architect on the project was Stephen Levrant Heritage Architecture.

“Because of its listing we had to be very careful,” says Rainey. “There were an awful lot of rules around what we could do with the building. So in essence what we've done inside is built an ark. The Baths building has been brought back to its full glory, and then in the middle of it is a huge oak ark, which houses the offices for the digital startup businesses.”

This approach removed some of the

complications of trying to make significant changes to listed buildings, and helped the building achieve a BREEAM 'Very Good' rating - normally a difficult thing for an older structure.

This year, the second phase was completed, which added a new council data center to the building, to replace a facility that was demolished several years ago.

“We razed our HQ to the ground about six or seven years ago and as part of that we lost our data center,” says Rainey. “We temporarily moved all of the council data center equipment and computer system into Rochdale Council's data center.”

“The intention always was to move them back into Tameside into a joint facility with the NHS. It took longer than we probably first thought it would.”

The 150 sqm (1,600 sq ft) data center is a ‘room within a room’ design built to Tier III standards with a steel frame and modular wall and ceiling panel construction. It was built in the original spectators’ gallery, next to the original Victorian pool, which is now filled in.

The data center has two independent UPS systems in a separate plant room next to the data hall. Data center developer Sudlows, with a partner, put together a bespoke BMS (building management system).

Sudlows looked at other design options including splitting the facility over two floors, but eventually opted for a single story, 36 rack data center consisting of four pods, according to Anthony Duffy, senior account manager.

“The first pod is for colo, another pod was going to be given to the NHS and then the others are for the council, but it's not been totally defined,” he says.

The data center has standalone security controls – including CCTV and biometric door access – not connected to the main building. Sudlows can provide a full audit trail of who was in and out of the data center and what racks were accessed.

The facility delivers around 2.5kW per cabinet, and can be expanded to meet demand, says Rainey.

“To start with we didn't want to go with the maximum amount of power, because we were conscious that from day one we wouldn't need all of that power,” he said. “We started off with the intention of growing it as

“There were an awful lot of rules around what we could do with the building. So in essence what we've done inside is built an ark. The Baths building has been brought back to its full glory"

54 DCD Supplement • datacenterdynamics.com

Take a Bath the facility itself becomes fuller.”

According to tender documents, Sudlows won a £700,000 tender ($970,000) from TMBC May, for data center facilities management at the Bath. Rainey says the agreement for management and maintenance will run for ten years.

Aesthetics and heritage Most data centers are anonymous whitewalled rooms, but Tameside Council wanted more aesthetics.

“We were very careful; one of the briefs with our architects and with Sudlows was we didn't just want a warehouse with a lot of racks in it,” explains Rainey. “We wanted something that would visually fit with the Ashton Baths offices.

“We went to quite a lot of trouble to make sure that the data center itself has got windows and glass walls with colored lights.

As you wander around the annex, in the middle is this data center which you can see into, with color-changing lights in the walls so that it looks an attractive feature. It just fits in with the history of the building, but also fits in with the technology side of the building as well.”

Most retrofits are in repurposed warehouses or factories, so it’s rare for a data center firm to be involved in a building with so much history and local cultural cachet.

“As a building, it was really exciting,” says Sudlow’s associate director of data center design, Zac Potts. “Being a Manchester-based company, some of the engineers working on the projects live in and around Ashton, and have been driving past the building for years.”

“From an engineering point of view, as a data engineer, I think a lot of data centers look quite good. But this is a totally different type of facility. The building itself has got so much history in it. It is a unique building where we're dealing with constraints, English Heritage, and trying to respect the architecture of the building.”

The company had to consider carefully where to drill holes and interact with the listed parts of the building. Duffy says that the majority of correspondence with English Heritage was going through the architecture partners, but when they did deal with the organization they were “very sympathetic” around what had to be done to bring the building up to required standards.

“We had to be very considerate of the space and work within the constraints,” adds Potts. “One of the reasons for the roomwithin-a-room approach was so that the space itself was almost independent of the existing structure, so that we didn't have to damage the structure by bringing it up to what you need within a data center.

“The aesthetics within the data center, trying to get that balance as to how we can

meet both the requirements of this being a focal point of the building, while still being a live, operational, and secure data center was an interesting kind of part of the project.”

Digital cooperatives and community This phase of development cost around £5 million ($6.8m), bringing the total project to around £15m ($20m). As well as hosting data and applications for the council, the NHS and other public sector partners, the new resilient facility will offer space to commercial tenants, through a cooperative.

“It took us quite a long time to decide that the co-op was the right operating vehicle, for the commercialization of the public sector investment that had happened in Tameside,” says Rainey. “The NHS, and the council,

and the colleges, and the various housing associations have all chipped in to make the fiber network that the data center is at the heart of.”

The Cooperative Network Infrastructure brings together public and private sector organizations to create and share digital infrastructure; public sector partners invest in new infrastructure assets – whether fiber, ducts, or rack space – where they have a business case; those assets are then joined together to form a coherent, integrated infrastructure that can be shared using the cooperative, and offered to telcos and other companies.

“It's open to everybody and everybody gets the same price,” says Rainey. “If you're Virgin or if you're a small independent IT company

Issue 42 ∞ October 2021 55

based in Tameside, you get charged exactly the same for accessing rack space, or fiber, or duct, it’s a completely level playing field.”

In the network, there is no single owner of the passive layer; rather a ‘thin layer’ cooperative ‘presents’ the disparate assets as a single infrastructure to multiple operatorservice providers [whether in-house IT departments, ISPs, carriers, and specialist VPN suppliers, or systems integrators].

Rainey explains that all public sector bodies that have made those infrastructure investments get rental income from the coop as it rents fiber, duct space, and rack space to the commercial telcos, which the telcos themselves then can go out and market to businesses and residents around Tameside.

“They just rent the infrastructure that's already there without having to make big investments or putting their own infrastructure in place.”

As well as Tameside Council, members of the cooperative include CityFibre, Virgin Media, Zayo, the Brighton Digital Exchange, a number of NHS Trusts and local colleges, as well as councils such as Lancaster City, Blackpool, West Sussex, and Manchester City.

Ensuring the data center was built to Tier III standards was important to Tameside’s commercial plan, guaranteeing it was on an equal footing with other facilities.

“We always had a view that we needed to commercialize this. It's the first and only commercial data center in Tameside, and we wanted to make sure that it was Tier III and the kind of facility that all businesses, as well as government, will be comfortable to use,” Rainey says.

“We took the time with Sudlows making sure that we had the right infrastructure and security in place; biometric locks, and CCTV, and the correct doors. We've ended up with a unique data center site inside an absolutely fantastic, renovated building in Tameside.”

But ultimately the goal wasn’t money, but creating an ecosystem which can allow digital business to grow and hopefully enable them to stay local for longer.

“The data center is important for creating a digital hub, around which we want digital

“We always had a view that we needed to commercialize this. It's the first and only commercial data center in Tameside, and we wanted to make sure that it was Tier III" businesses to coalesce and we want to grow that digital creative sector,” says Rainey.

“We've invested a lot of money in Digital Tameside in fiber in the ground and ducting and data centers, and we've done that and saved the public sector a lot of money. But the prize was never really saving money; the prize was always the economy,” says Rainey.

“We knew that if we didn't make investments like this in Tameside, businesses that need connectivity, rack space, and data center space would end up leaving.”

Rainey says that as telcos begin to deploy fiber more widely, Tameside would not be high on the list of areas to invest in, which could have harmed the local economy.

“Places like Airport City, in the northern quarter of Manchester, would steal a march and we'd end up losing business or losing jobs. Businesses would go to areas that have got that kind of connectivity and those kinds of facilities.”

Does the North need hyperscalers?

While all of the big cloud providers have a presence in the UK, their facilities are mostly located in and around London and the South of England.

Sudlows’ Potts says that Manchester and

56 DCD Magazine • datacenterdynamics.com

the Northwest have got the capacity and the ability to house a large hyperscale facility for sure, but there is also “big demand” for more Edge-type facilities.

Rainey isn’t shy of saying he feels there should be more data centers in and around Manchester and greater connectivity across the North in general.

“I think that one of the problems that the UK has is the London-centric focus of the digital infrastructure,” says Rainey. “IX London is huge. Which is fine, except that there's no resilience beyond that.”

LINX, the London Internet Exchange, the UK’s central Internet exchange point (IXP), does operate regional IXP in Manchester, Scotland and Wales, but Rainey says the Manchester IXP is nowhere near the same scale as London despite being the second-largest exchange in the UK. By comparison, Germany operates a number of larger regional and sub-regional Internet Exchanges.

“Manchester should have a thriving Internet exchange, it should have the data centers to support it because, as we move more into cloud computing, the latency becomes more important, so will be the proximity of businesses to their systems.

“I think the North will be disadvantaged if it continues as it is, with the great big data centers down in Slough or over in Dublin or across in Denmark, because that latency will be critical.”

Orbital Barrages tensions, and a number of satellites have been destroyed in military demonstrations.

Since the 1960s we have come to rely on satellites for connectivity, positioning data, and scientific research. Can governments and militaries continue rattling their space sabers without impacting global communications, and their own domestic industries? Or is the world going to have to learn the hard way again?

Dan Swinhoe News Editor

Wargames in space threaten satellite security and sustainability We need to relearn lessons from the Cold War - about weapons in space

D

uring the late 1950s and early 1960s, just as the space race was heating up and Cold War tensions were growing frosty, both the US and the Soviet Union were busy developing weapons to bring down satellites almost as quickly as they were being sent up.

Within days of the Soviet Union launching

its first satellite, Sputnik, in October 1957, the US began work on a response from the ground, in mid-air and from space-borne platforms.

In the 1960s, high-altitude nuclear tests showed the unintentional fallout of using weapons in space - and the fragility of satellites.

But despite this, space weaponry is still being developed. After a relatively peaceful period, recent years have seen increased political

Starfish Prime and lessons in space sustainability After Sputnik launched, the CIA quickly issued a report warning of the potential for Soviet reconnaissance by satellite.

The US response was to develop antisatellite (ASAT) weapons. The idea of spaceborne weapons was quickly conceived, but the earliest response was a series of missile systems launched from the ground or air.

The High Virgo, or Weapons System 199C (WS-199C), missile failed, but it was followed by Bold Orion, or Weapons System 199 (WS-199B) the first successful demonstration of an ASAT. In October 1959 a missile was launched from a B-47 at an altitude of 35,000 feet (11,000 m); the missile came within four miles (6.4 km) of Explorer 6, a US science satellite studying radiation, at an altitude of 156 miles (251 km).

The test was deemed a success; if the missile had a nuclear warhead, the satellite would have been destroyed.

Around the same time, another way to counter the satellite threat was considered: the US conducted a series of nuclear explosions in the atmosphere.

In 1958, the Hardtack program exploded three nuclear weapons over the Pacific. The 1.7 kiloton Yucca was the first high-altitude nuclear explosion; the warhead was attached to a large helium-filled balloon, and detonated at a relatively modest altitude of 26.2 km. The 3.4 megaton Teak was detonated at an altitude of 76.2 kilometers (47.3 mi), in the stratosphere.

Orange – the largest high altitude test conducted at 3.8 megatons – detonated at 43 kilometers (27 mi).

While Teak and Orange caused local outages and communication disruption in the surrounding area, neither were high enough to cause damage to satellites that orbit above 250km.

Also in 1958, the US sought to test whether a small number of nuclear bombs detonated high over the South Pacific could create an electromagnetic shield that would destroy the on-board electronics of incoming Soviet warheads.

The Argus program launched three 1.7 kiloton missiles at altitudes of 200km, 240km, and 400km, which is the highest such test ever conducted.

The electromagnetic effects were tested by Explorer 4 and 5 satellites, showing that in-orbit operations were seemingly unaffected, any

Issue 42 ∞ October 2021 57

such electron shield created by the explosions wouldn’t last long enough to be useful.

Operation Fishbowl, designed to explore electromagnetic pulses, was the last such series of tests and was also the most damaging to what was then a relatively small satellite community. The first two attempts, Bluegill and Starfish, were abandoned after launch before they could detonate. Starfish Prime, detonated successfully and, in the process, demonstrated the dangers of weapons in space.

A W49 thermonuclear warhead attached to a Thor rocket was launched from the Johnston Atoll in the Pacific Ocean in July 1962. The 1.4 megaton explosion at an altitude of some 400 km (250 mi) created an electromagnetic pulse detected in Hawaii more than 1,400 kilometers (900 mi) away, reportedly knocking out around 300 streetlights, setting off numerous burglar alarms, and caused the failure of a telecoms firm’s microwave repeating station.

In orbit, a number of satellites were temporarily disabled in the explosion’s wake.

Many more were damaged or lost in the following months, likely as a result of the test’s resultant radiation belt – including the USowned Traac, Transit 4B, Injun I, Explorer 14 and 15, Relay 1, and Telstar I, as well as the UK satellite Ariel, and the Soviet-owned Cosmos V. With only around 20-25 active satellites in orbit at the time, this means that the Starfish Prime explosion had impacted nearly half the world’s satellites. Today there are closer to 4,000 operational satellites, so any similar test would likely have a much larger impact.

Starfish Prime probably should have been the end of such tests, but the US military continued Operation Fishbowl after a few months’ pause; Bluegill Prime and DoublePrime both failed to launch properly, while the 10-20 kiloton Checkmate was successfully detonated in October 1962. Bluegill Triple Prime, Kingfish, and Tightrope also launched later that year, with no noted damage to satellites. Tightrope was the final atmospheric test conducted by the United States, after a program of five nuclear detonations as part of Fishbowl and 11 in total.

The Soviet Union carried out four of its own high-altitude tests in 1961 and three in 1962 in what is known as the K project, one of which started a fire that burned down the Karaganda power plant in Kazakhstan.

Satellites make the world go round The satellite industry is now a massive industry generating huge amounts of value for multiple industries, well beyond military and national defense uses. Disrupting such machines would have a significant knock-on effect on our daily lives.

Global positioning system (GPS) satellites alone have generated an estimated $1.4 trillion in economic benefits since the system was made available for non-military use in the 1980s. As well as providing location services,

such global navigation systems provide timekeeping for many digital systems.

A report by The Aerospace Corporation’s Center for Space Policy and Strategy (CSPS) notes satellites are important in areas such as agriculture, fishing and maritime, environmental monitoring and weather forecasting, financial trade, and even emergency services.

“Satellites are so important to our daily lives,” says Makena Young, research associate with the Aerospace Security Project at the Center for Strategic and International Studies (CSIS).

“If a main satellite system went down, it would just wreak havoc on our daily society; things like banking would really slow down or stop completely for a few days.”

DCD has previously written on the growing complexity of space operations amid the huge increase of commercial satellites from the likes of SpaceX and OneWeb. SpaceX’s Starlink satellites are involved in almost half of all close encounters between spacecraft every week (though the majority are between its own satellites), but the company says it is working hard at ensuring greater coordination with both commercial and government satellite operators.

Advocates for greater oversight of space traffic management are also growing in number, arguing that change is needed to ensure space sustainability. However, in the military domain, the rhetoric around weaponization is increasing.

In the US, military personnel such as General John Hyten have previously spoken about the need to make more public demonstrations of the US’ space weapon capabilities in order to deter adversaries.

Essentially, treating these weapons as military secrets means that potential enemies don’t know they exist, so they have no deterrent effect. Breaking Defense reports Hyten saying the US “over-classify everything in space,” and that “deterrence does not happen in the classified world.” However, such demonstrations in a global operating environment such as orbit creates risks for everyone.

While geostationary satellites are large, multi-ton machines built to withstand the harsh environs of space, there’s little that can protect them from a targeted ASAT missile strike.

And as well as taking out the large GEO satellites, ASAT use would create a large cloud

of debris which would prove instantly fatal to the growing number of low earth orbit (LEO) satellites that are much smaller, lighter, and faster.

A destroyed satellite can cause a cascade; each trackable piece of debris can potentially destroy another satellite, which in turn can create more debris that can destroy other satellites. Kessler Syndrome warns of a scenario where an excess of space debris begins to collide uncontrollably with satellites, causing an unstoppable cascade of collisions and debris that could render orbits unusable by satellites or manned missions for generations.

The indiscriminate nature of space means that as well as hurting local industry and global connectivity, military operations can easily be disrupted by debris and conjunction events.

Earlier this year, the Yunhai 1-02 Chinese military satellite collided with a piece of debris from the Zenit-2 rocket that launched Russia's Tselina-2 spy satellite in September 1996 and has since broken up. Any prolonged combat in space would likely impact a high number of satellites for years to come.

“ASATs are really dangerous for everyone,” says CSIS’ Young. “Space is very different than on the ground... If you blow up a satellite, that satellite breaks up into thousands of different pieces and then they just continue to orbit, and they can hit any number of satellites.”

“If Russia were to blow up a US satellite,

“If a main satellite system went down, it would just wreak havoc on our daily society; things like banking would really slow down or stop completely for a few days"

58 DCD Magazine • datacenterdynamics.com

Orbital Barrages that space debris could easily take out a dozen Russian satellites in the same orbit.”

Wargames in space threaten everyone’s satellites The 1963 Partial Nuclear Test Ban Treaty prevented further high-altitude nuclear tests and the 1963 Outer Space Treaty prohibited the placing of nuclear weapons in space, but the US and Soviet Union continued testing and developing a number of ASAT weapons systems that didn’t rely on nuclear weapons in order to avoid breaking the treaty or damaging their own satellites with EMP. Many ASAT systems are kinetic weapons relying on missiles. Building on the High Virgo and Bold Orion tests and following Program 505 – where counter-missile systems were explored as anti-satellite weapons – the US Military’s Program 437 anti-satellite weapons program continued from the early 1960s until 1975. The US also had the Satellite Interceptor Program (SIP). But despite all the years of testing, only two US satellites have been actually destroyed since Starfish Prime.

In 1985, despite an incoming block on ASAT testing in the US from the Senate, an ASM-135 missile launched from an F-15 was used to destroy the Solwind P78-1 satellite in a test; the debris from the explosion meant NASA had to enhance shielding for its planned space station.

The last piece of tracked debris from this test fell out of orbit more than 15 years later in 2002.

In February 2008, the US Navy destroyed the malfunctioning US spy satellite USA-193 from a decaying orbit using a modified RIM-161 Standard Missile 3 fired from a warship in the Pacific. The satellite was officially destroyed due to the presence of hydrazine and beryllium on board and the potentially hazardous effect the chemicals could have if they leaked over a populated region.

The Soviet Union also developed its own ASAT capabilities via ballistic missiles during the 1960s and 1970s. The program was named Istrebitel Sputnikov, meaning "destroyer of satellites" and did exactly that to several orbiting systems.

The Soviets explored using anti-satellite lasers at the Terra-3 laser testing center, but seemingly were unsuccessful at the time. The US also tested ground-based anti-satellite lasers to varying levels of success up to the late 1990s.

Both countries are assumed to have nonkinetic, energy-based ASAT systems that aren’t publicly disclosed.

Since the fall of the Soviet Union, Russia has continued to develop and operate ASAT weaponry, and has conducted a number of tests without destroying any satellites in recent years.

New millenium, same threats The 1990s saw little in the way of space conflict or demonstrations, but since the turn of the new millennium tensions have gradually risen

“The Chinese test was the absolute worst because they destroyed a large one-ton satellite right in the most popular orbit, at an altitude where the debris takes a long time to come down" and weaponization has escalated.

In January 2001, a space commission headed by Donald Rumsfeld recommended that “the US government should vigorously pursue the capabilities called for in the National Space Policy to ensure that the president will have the option to deploy weapons in space to deter threats to, and, if necessary, defend against attacks on US interests.” The US then withdrew from the Anti-Ballistic Missile Treaty in 2002.

That same year, the Pentagon announced the US would continue the “development and testing of space-based defenses, specifically space-based kinetic energy [hit-to-kill] interceptors and advanced target tracking satellites.”

After more than 20 years without incident, there have been three intentional ballistic tests on satellites in the last 15 years. Today, four countries have demonstrated ASAT capabilities; the US, Russia, India, and China. Though none have been recorded as used in conflict, such demonstrations often lead to similar reactions by rival nations and increased tensions. Israel is also reportedly developing an ASAT system.

“Every new space power seems to have some desire to show that they can do it,” says Dr. Holger Krag, head of the Space Safety Programme Office for the European Space Agency. “My hope is that once they have shown it, they don't need to repeat it. There are disruptions to spaceflight every time it happens.”

“We have already many many unintentional events. And these cannot be prevented. For those that you can control, if they cannot be avoided, they must be done in a very careful way.”

China has reportedly been working on ASAT capabilities since the 1960s; originally through the Program 640 missile development initiative and then through the wide-reaching Program 863 for high technology development.

China has also reportedly been developing laser/energy-based weapons that could disable enemy satellites without destroying them and causing debris; in 2006, China reportedly illuminated a US satellite with a ground-based laser, perhaps more than once.

The Chinese military launched an antisatellite weapon on January 11 2007 destroying the Fengyun-1C (FY-1C) weather satellite. The 750 kilogram (1,650 lb) satellite was struck by a missile at an altitude of 865 kilometers (537 mi). It was the first such destruction of a satellite since 1985.

In the aftermath, Chinese Foreign Ministry spokesman Liu Jianchao stated that "there's no need to feel threatened about this" and argued that "China will not participate in any kind of arms race in outer space."

China has since tested a number of ASAT missiles without destroying more satellites. US officials have said its destruction of its USA-193 in 2008 was not a demonstration of its own capabilities in response to the Chinese test.

The aftermath of the 2007 Chinese test is still being felt today. Where USA-193 created 174 pieces of orbital debris, the last of which re-entered the atmosphere some 20 months later in October 2009, the FY-1C demonstration created 40,000 pieces of debris larger than 1cm.

Several thousand remain in orbit today, more than 10 years later.

“The Chinese test was the absolute worst because they destroyed a large one-ton satellite right in the most popular orbit, at an altitude where the debris takes a long time to come down,” says astrophysicist Jonathan McDowell of the Harvard-Smithsonian Center for Astrophysics.

India has developed its own ASAT capabilities, and successfully conducted an ASAT test under the project name Mission Shakti. In March 2019, a missile fired from Kalam Island struck a test satellite – thought to be Microsat-R – at a relatively low 283-kilometer (176 mi) altitude. More than 200 pieces of debris were created, though only four pieces remained in orbit at the start of 2021.

“The Indian test, they thought they would try to be good, and relatively speaking, they were good,” says McDowell. “But when you have these collisions some of the debris goes up to higher apogees. Most of their debris went down right away but they did have a couple of hundred pieces of debris that were tracked in orbit for months.”

Planet Satellites at the time said “destroying satellites on-orbit severely threatens the long term stability of the space environment for all space operators. Planet urges all space-capable nations to respect our orbital commons.”

At the time, then-NASA Administrator Jim Bridenstine called the Indian test a “terrible, terrible thing” that is “not compatible with the future of human spaceflight.”

“It’s unacceptable and NASA needs to be very clear about what its impact to us is,” he said, adding that all commercial and civil space activities “are placed at risk” by such tests and

Issue 42 ∞ October 2021 59

that “when one country does it, other countries feel like they have to do it as well.”

How about arming the satellites?

But there’s another, possibly more alarming development, with its roots going back to the very first response to Sputnik. Instead of aiming earth-bound weapons at satellites from Earth, nations have been seriously considering arming the satellites themselves.

The first proposed anti-satellite weapon wasn’t ground or aircraft-base d missile, but an in-orbit anti-satellite satellite. The idea for Project SAINT (SAtellite INTerceptor) was conceived days after Sputnik was launched, and contracts were signed in 1960.

The project continued until 1962, though it never launched.

At first, the US military considered arming its own reconnaissance satellites, such as MIDAS and SAMOs, but the idea evolved to SAINT - a dedicated satellite-killing satellite.

Each SAINT module was expected to weigh around 4,400 pounds (2,000 kg), and be 14 feet (4.3 m) long, with the ability to fly to within 40 ft (12 m) of a target to inspect and potentially destroy it. Various potential ‘kill’ designs were considered; one would have seen the satellite perform a kamikaze collision, another would have used a laser to blind the target, and of course nuclear weapons were also considered.

Interestingly, a more passive attack concept would have seen the machine coat its target with black paint, blocking its ability to see or transmit, rendering it inoperable.

SAINT was canceled without ever launching due to high project costs and the fact that the Soviets weren’t placing warheads in orbit but had reportedly armed their satellites with

In the 1970s, a Russian military space station was armed with the Shchit-1 system; a customized auto-cannon designed for aircraft that could theoretically fire up to 2,600 rounds per minute proximity-based self-destruct mechanisms to prevent close observation or interference.

But this was far from the last time militaries thought about placing weapons in space. The 1963 treaty didn’t and still doesn’t provide an outright ban on the weaponization of space and weapons have occasionally been put into orbit.

In the 1970s the Almaz Salyut 3/OPS-2 military space station was armed with the Shchit-1 system; a customized auto-cannon designed for aircraft that could theoretically fire up to 2,600 168-gram rounds per minute.

The weapon was apparently fired only once, as a remote test, after the crew had disembarked. The results are unknown, but some reports say the cannon was fired at target satellites at ranges from 500 to 3,000 meters (1,600 to 9,800 ft).

The OPS-4 space station was reportedly armed with rockets in lieu of auto-cannons, but was canceled before launch.

Such examples of in-orbit weaponry date from the height of the Cold War - and are rare.

While ground or platform-based anti-satellite weaponry is a superpower status weapon, inorbit weaponry became taboo as nations tried to keep the peace in space.

But as global tensions rise, the possibility

60 DCD Magazine • datacenterdynamics.com

and feasibility of anti-satellite satellites and inorbit weaponry are gaining traction again.

“The main three space powers are the US, Russia, and China, and they're not allies in a lot of these situations so there's definitely a lot of competition in these military space systems,” says CSIS’ Young. “If you see an adversary have a weapon, then you automatically also want to have that capability.”

“We are seeing satellites that have the capability to blind on-orbit,” she adds. “I think we are seeing a lot more hardening of the physical satellite to combat against threats like lasers. It is possible to put a gun on a satellite but it's not very likely.”

In 2001, a Chinese newspaper stated that the PRC was testing a parasitic micro-satellite that could latch onto another satellite and destroy it on command, though no such device has yet been tested in space to public knowledge.

In 2008 Chinese astronauts aboard the Shenzhou-7 space station released the BX-1 micro-satellite, which flew within 27 miles of the International Space Station at a relative speed of 17,000 mph.

Also in 2008, Russia proposed a new treaty banning space weapons, but has had a change of heart. Dr. Brian Weeden, director of program planning, Secure World Foundation, now says that Russia is “a bigger counterspace threat than China,” after a number of demonstrations of potentially destructive weapons.

Since 2011 Russia has been thought to be working on a space-based ASAT. “Burevestnik” - also known as Project 14K168, and not to be confused with the name-sake Burevestnik nuclear-powered cruise missile - is thought to potentially involve a launch vehicle for small interceptor satellites that can approach and disable enemy satellites.

Russia is also developing a terrestrial ASAT ballistic-missile system called Nudol.

The US Space Force, founded in 2019, has raised concerns about Russian ASATs, calling a 2020 ASAT test “yet another example that the threats to US and allied space systems are real, serious and growing.”

The Space Force has also noted that two Russian satellites may be armed - they ‘exhibited characteristics of a space weapon’ and were observed conducting manoeuvrers near a US Government satellite that would be “interpreted as irresponsible and potentially threatening in any other domain.”

In July 2020, Kosmos 2543 reportedly

Orbital Barrages released an unknown object at ‘a fairly high relative velocity’ of around 700km/h (434 mph) into orbit close to another Russian satellite.

The Russian Defense Ministry said 2543 did not create a threat for other spacecraft and merely “carried out a check of a Russian spacecraft at close range with the use of specialized small spacecraft apparatus,” but the object was described by the UK MoD to be a ‘projectile with [the] characteristics of a weapon.’

2543 had itself been released from another satellite, Cosmos 2542 in what was described as a kind of "nesting doll" mechanism. The parent 2542 satellite had already caused concerns amongst US military, after it was seen to be tailing a US satellite. It orbited in the same plane as USA 245, a KH-11 spy satellite operated by the National Reconnaissance Office, and got as close as 150km.

This was the third case of Russia seemingly launching ‘nesting’ satellites that break off from one another after launch.

The Russian Luch/Olymp-K has also been known to creep up on other satellites, including several Intelsat machines, before moving on to different orbits. Why it’s been doing this and whether it has any other missions beside close observation – such as data interception – is unknown.

US Space Command head General James Dickinson, has also written about concerns with China’s Shijian-17/SJ-17 satellite, which is reportedly equipped with a robotic arm, warning the technology “could be used in a future system for grappling other satellites.”

China has maintained the satellite is an ‘experimental communications satellite’ featuring novel imaging, solar, and propulsion technology. Like the Russian satellites, it too has changed orbits repeatedly to sidle up to other satellites, though mostly Chinese-owned and operated machines. It did, however, spend time near Indonesian communications satellite Telkom 3S, in late 2017 and early 2018.

The US, through DARPA and contracted defense firms, is also exploring the use of robot arms on satellites, though most would say the primary focus is repair and maintenance.

The very formation of the US Space Force itself – though mocked in the press at the time as one of then-President Donald Trump’s more fantastical ideas – shows not only how seriously the military is taking the idea of weaponizing space but also how ready it is to take action. Air Force Secretary Frank Kendall recently warned that “China has moved aggressively to weaponize space.”

Breaking Defense reports the US is close to revealing a new ‘space weapon program’ and preparing a demonstration of its capabilities.

Speculation on the currently-classified weapon ranges from a terrestrially-based mobile laser used for blinding satellites, to on-board proximity triggered radio-frequency jammers, to a microwave system that can fry electronics carried on maneuverable bodyguard satellites.

Increasing tension between the major space actors is leading other countries to consider their own space defense measures.

France recently conducted Europe’s first space wargame to stress test its satellite resilience.

Run from France’s Space Command operations room in Toulouse, the four-day ‘AsterX’ operation was joined by Germany, Italy, and the US. AsterX simulated an attack on one of France’s allies, from an unnamed space-capable power - an attack which centered on taking out a communications satellite.

France is also reportedly considering arming its next generation of Syracuse satellites with weapons. Reports suggest these weapons could take the form of machine guns capable of destroying the aggressor's solar panels or lasers which can blind or destroy an enemy satellite.

Such weaponized machines could be in orbit by 2030.

"We do not want to embark on a space arms race," French Minister of Defense Florence Parly said in 2019. "We will conduct a reasoned arsenalization."

Japan has also reportedly been considering developing and launching its own satellite interceptor as a means to counter-attack satellites from China and Russia.

Reasons to be peaceful In June 2021, NATO expanded Article 5 – which requires member states to come to the aid of any member state subject to an armed attack – to include attacks against a country’s space-borne assets. This means that any attack on a NATOmember satellite could cause retaliation in space and on Earth from up to 30 countries.

“We consider that attacks to, from, or within space present a clear challenge to the security of the Alliance, the impact of which could threaten national and Euro-Atlantic prosperity, security, and stability, and could be as harmful to modern societies as a conventional attack,” NATO said in a communique. “Such attacks could lead to the invocation of Article 5.”

But not all global efforts are aimed at ramping up tensions in space. In December

“There's a strong need for us to go 'no, just don't do that.’ We have enough problems with accidental debris. Don't create deliberate debris, it's just not the thing to be doing'”

2020 the UN adopted resolution 75/36 aimed at “reducing space threats through norms, rules and principles of responsible behaviors” that will hopefully lead to a UN-wide set of principles and rules on “responsible behaviors” for space. Another resolution, 75/60, was passed reaffirming non-proliferation of ballistic missiles in space.

There are other proposed international agreements, such as the treaty on the Prevention Of The Placement Of Weapons In Outer Space, The Threat or Use of Force Against Outer Space Objects (PPWT), the EU International Space Code of Conduct, as well as the Prevention Of An Arms Race In Space (PAROS) Treaty that would prevent signatories from placing objects carrying any type of weapon into orbit, installing weapons on celestial bodies, and threatening to use force against objects in outer space.

None so far have been successfully adopted globally. The US often votes down any legal regimes or other mechanisms that would restrict its access to or use of space, including any arms control proposals that would impinge on military space acquisitions or operations.

Resolution 75/36, which was introduced by the UK, seems the most likely to succeed in the long term, but it could be years before any sort of binding agreement emerges. But at least it shows that countries around the world recognize the importance of keeping the peace in space. Hopefully, we won’t have another Starfish Prime before then.

But if an agreement can’t be made between world leaders, it is hoped that the space industry itself can make a case for keeping the peace.

“In just the past few years commercial space has started to totally dominate versus military and civilian space,” says McDowell. “If your country is active in military space but it also has a big commercial space sector, there's an internal political discouragement from screwing up things for the commercial people.

“That might, I hope, discourage some of these countries from actually doing more of these tests because the commercial people are not going to be happy with the extra debris.”

“There's a strong need for us to go 'no, just don't do that.’ We have enough problems with accidental debris. Don't create deliberate debris, it's just not the thing to be doing.'”

CSIS’ Young notes that the dynamic is changing as the military and Government become more dependent on commercial operators. This could potentially give large companies more influence to help reduce potential weaponization in space.

“Commercial actors are becoming so much more vocal in space. A lot of commercial companies are vital to giving information to the government and that relationship is becoming a lot more important and we rely on some of these commercial companies more than we have in the past.”

Issue 42 ∞ October 2021 61

Difficulties Staying Up

Don’t be stupid

T

his section of the magazine is usually reserved for the editorial team’s thoughts or concerns about the industry, be it over the influx of clueless investors, the lack of concern over climate change, or the dangers of centralized services.

This time, however, it’s being used for a simple public service announcement: Don’t be stupid.

Outages happen, software glitches happen, human errors occur. We know this.

Why, then, are some large and technologically advanced companies still pretending otherwise? Just as we prepared to publish this magazine, Facebook went down due to a BGP misconfiguration error, dragging WhatsApp, Instagram, and Oculus offline with it.

While there are sure to be lessons in the causes of the lengthy outage, more can be learned by the failure of basic company design. Not only was the controversial social media site offline, but so was the platform staff used to communicate, as were internal diagnostic tools, and back-end systems.

Oh, and they couldn’t get into their offices. Panicked employees, desperate to get to work to fix the site were unable to use entry badges that relied on Facebook.com being online. Those inside were equally unable to enter

62 DCD Magazine • datacenterdynamics.com

conference rooms or access other sites.

Sure, Facebook has been busy lately fighting against growing evidence that its platform is destructive to mental health and the fabric of society, but surely there must have been some discussions in the past about redundancy and resiliency.

It is basic best practice to ensure that the tools and systems used to help recover from outages don’t rely on there not being an outage.

This is an extreme example of a surprisingly common issue. When AWS US East-1 suffered issues late last year, the company was unable to update its status page.

Equally, when Microsoft had a huge outage in 2018, its status page went down completely.

In both cases, the platform designed to let people know what was happening during outages relied on the service operating normally.

This likely isn’t by intentional design, but by small short-term decisions to consolidate services on infrastructure platforms in the name of efficiency, cost reduction, or ensuring that they don’t use a competitor.

For whatever reason, each of these companies forgot a basic tenant of redundancy. Make sure you don’t do the same.

POWER SOLUTIONS FOR A BRIGHTER FUTUREFOR POWER SOLUTIONS A BRIGHTER FUTURE

At Cat Electric Power, we are striving for a world where society’s basic needs are met; needs for shelter, clean water, sanitation, food and reliable power. We offer customers scale-able, sustainable products and solutions designed to meet your power needs. Our low- and no-carbon power system options, enable economic growth through sustainable infrastructure and energy development.

At Cat Electric Power, we are striving for a world where society’s basic needs are met; needs for shelter, clean water, sanitation, foodmore and reliable We offer customers scale-able, sustainable products and solutions designed to To find out visitpower. cat.com/sustainablepower meet your power needs. Our low- and no-carbon power system options, enable economic growth through sustainable infrastructure and energy development.

To find out more visit cat.com/sustainablepower

24

7

H₂ HVO SOLAR & ENERGY STORAGE

LOW - CARBON INTENSITY RENEWABLE FUELS

AFTERMARKET SUPPORT 24

7

No matter the environment, Starline’s at the center.

Hyperscale, Colocation, Enterprise.

Time-tested power distribution for every environment.

Starline Track Busway has been the leading overhead power distribution provider—and a critical infrastructure component—for all types of data centers over the past 30 years. The system requires little to no maintenance, and has earned a reputation of reliability due to its innovative busbar design.

StarlineDataCenter.com/DCD