The DCD Automation Supplement by DCD Magazine

Sponsored by

Contents Self-driving data centers

4. The network conundrum Automated networks can be a doubleedged sword. We need intelligent automation

10. Augmented reality for building Engineering grade augmented reality can help get a project right first time 13. Drones in a data center Security guards may need a helping hand to patrol today’s mega-campuses

igital infrastructure is expanding at a colossal rate, and only technology can keep pace with that. In other words, data centers must automate to survive.

The job of managing our digital infrastructure, from the physical buildings all the way up to the fibers connections, network links and code, is too much for unaided humans. So data center builders and operators are turning to technology for help. This supplement looks at the progress we are making towards a data center that controls, heals and patrols itself. And maybe one day, can even build itself.

Networks that think

It used to be that the network was a fixed thing, and the software had to fit in with it. Now the network has to adapt to the demands of applications, which run on multiple clouds to precise performance levels. The only way to deal with that situation is with an intelligent network which can adapt its performance ot handle what is thrown its way. The first attempts to automate networks created complex systems of rules, which broke when they encountered the unexpected. Now, network equipment is approaching a level which could actually be called intelligent. It understands the goals of designers and the intent of users, and works to deliver the best possible performance - and knows when to call for help.

Headsets that think Augmented reality (AR) tends to look science fictional and a bit unrealistic, while construction is the ultimate hands-on, muddy boot occupation. And yet, data center builders are among the first in the construction industry to see the present and future of their project combined through AR headsets. It's all down to having a headset that can locate real and virtual data points to within millimeters, and spot errors before the happen. We don't have a self-building data center, but this is a serious step to help automate the construction process

Robots on patrol Physical security has always meant staff pounding a physical beat, patrolling perimeters with their eyes and ears open. Maybe not for much longer. Autonomous drones can make independent airborne patrols of the campus perimeters, reporting back on anomalies. Truly integrated drone security is in its early stages, but data centers are getting a head start on using smart flying devices.

What's coming next? More and more of the data center sector is becoming self-managing and self-monitoring. This may be cause for some concern. When outages happen, it's often because complex systems are running automatically with no one seeing every consequence of a change. Next time we return to automation, we may have to check out security and resilience.

Automation Supplement | 3

NETWORK AUTOMATION:

the complexity conundrum

Peter Judge Global Editor

Automation can be a double-edged sword in the network arena. What we need is intelligent automation

ata center networks have changed, and there’s no going back. They have become so complex, that they cannot be understood in real time by human beings.

They have to be automated. But that brings a danger: how can you be sure they have been automated correctly?

The complexity comes from the demands of the applications which are running - and the need to run them on multiple clouds and infrastructure. “There are a lot of new demands and pressures being placed on data center networks,” says analyst Brad Casemore of IDC, speaking at a DCD online event. “Application architectures have really redefined data center networking requirements. As a result of the evolution of application architectures, there's a need for modernization within data centers.”

DCD Supplement • datacenterdynamics.com

Network Automation  Applications rule the network

Fast is not good enough

Networks used to dictate terms to the applications that ran on them, says Sanjeevan Srikrishnan, senior global solutions architect at Equinix: “In the good old days, we'd go out and build a killer network. We'd be like hey, I want 100G backbone links and all of this crazy infrastructure. Then the business would come to us and say, ‘Can I run my app on your network?’ and we’d say “No, sorry, it doesn't meet our needs.’”

The services that run on these hybrid distributed networks have to respond instantly - but also very consistently - says White: “When I work on hyper scaling networks, it's not even really the delay that matters. It's the jitter.”

Network architects could actually ask the business to go away and rebuild applications to suit the network: “Take it away, break it into these three tiers. Bring it back to me like this.”

With all these different parts of the network to manage, it’s impossible for network administrators to respond quickly enough to keep up with changing demands.

It’s not like that anymore: “Nobody does it that way anymore. Application is the king Kahuna. It's the bottom of the triangle. It’s the base of the technology equivalent of Maslow's hierarchy of needs. User experience is king.” Meeting user experience demands would be complex enough, but networks are now constructed from diversified parts, and have to respond coherently: “Everything is really responsive to the applications,” says Sagi Brody, CTO of managed service provider Opti9. “And the production environment for an enterprise organization today is typically hybrid. It spans across colocation, private clouds, public clouds, and SaaS.” These complex networks have been put together from parts which were historically siloed - and under the covers some of that hasn’t changed: “You're seeing organizations go from fixed and siloed configurations, into this new digital world.” says Srikrishnan, “and it's never a clean migration. You always have “tech debt” that sits there and it may stick around for 10 to 15 years.” Alongside that, responsibilities shift: “Many of the things you thought the provider was going to own you still own. You are jumbling together four different types of services, and you have to own the compliance and security of all of them those individually, as well as how they work together,” says Brody. These networks also distribute more network decisions, says Russ White, infrastructure architect at Juniper: “From a network architecture perspective, how do I build these networks that can handle this Edge traffic and distribute stuff intelligently and still have some sort of a core?”

Delay is when network packets take a long while to arrive. Jitter is when they arrive, but the delay is variable, garbling real-time traffic such as voice calls, he explains: “Consistency is a huge key right now, how can I make the network perform consistently all the time?”

“The hybrid use cases are forcing us into scenarios where we need to deploy things like VPN, and VxLAN,” says Sagi Brody, CTO of managed service provider Opti9. “These are just literally not configurable by hand anymore.” The obvious thing to do is to use automated tools to control the network’s response to changing conditions, and to take the burden off the admin: “What I believe in automating the crap out of everything,” says Srikrishnan. But what exactly is being automated? Srikrishnan says the network is a “nebulous” term. “Are we talking about the virtual networks that the developers see? Or are we talking about the underlying infrastructure that powers all of that? Those are two very different things.” Another issue is that automation is not simple. The first approach was to make a set of rules which provide a canned version of the response an administrator would make to specific events. That works fine most of the time, but if an event is slightly outside the possibilities considered by the network programmer, the response may actually be wrong - and sometimes disastrously so.

Fast automation is dangerous Brody says: “Automation is important. But it could also be dangerous, it has to be done right. It has to be use case-specific.” Brody says: “The intelligence has to come in and add some layers of logic,” to check if any action will cause problems. “An example is IPAM [IP address management]. If the IPAM says that a subnet is free and not in use, before we go and assign an IP address, let's check if it's routable.”

Automation Automation Supplement Supplement || 55

People used to think they could build things as complex as they wanted, he says, “as long as we automate it. And I think we need to get away from that line of thinking and start thinking about how do I make my network more intelligent, so I can actually automate less, but have the automation be more intelligent.” Brody says: “I think we're moving away from a world where you can halfautomate, and half do things manually. We have to focus on simplicity, everything has to be as simple as possible.” Shrikrishnan thinks the answer may be automating early, from the ground up: “If we talk about automation early on, and you use best practice, you're not using hands to keyboard to deploy anything, unless you're using a product like Terraform or Ansible to push your code up into production into your infrastructure. As you do this, you should be validating it.”

Observability for security

intelligent: “You add some intelligence, you add some logical checking, and so on.”

Casemore wants to see verification: “So when you automate a change at scale, it's not going to cause all sorts of problems or potentially break down part of your network.” “We're in a transition phase,” says White. “We had all these really smart people who could type on the keyboard and get the console working. And we thought we would just automate them out of a job. But we haven't turned that corner.” Brody has been through that cycle: “Years ago I built a lot of network automation myself. There's always this natural progression. You build the automation, and then at some point, it fails. At some point, it takes down your network, it does the exact opposite of what you want it to do.”

Intelligent automation Brody says the answer is to make a network which is not just automated, but

White agrees: “I want the network to be down as little as possible. And I think we're almost over-relying on automation, and under-relying on intelligent automation. We should put the emphasis on intelligence and not on automation.” IDC’s Casemore says: “The automation not only becomes more comprehensive, but it becomes smarter and a little more anticipatory. we move to a more proactive form of automation.” But this has to be done without adding layers of complexity. Brody wants to bring it back to a more simple view: “We have to turn it around. We need to focus on declarative models, imposing our ideal configuration on the network.” Instead of the automation configuring the network, he wants to see a “single point of truth”, a configuration imposed on the physical network. “This is a new paradigm,” he says. “We need to move towards machineto-machine interfaces. And we need to rethink.”

DCD Supplement • datacenterdynamics.com

Network behavior also has to be “observable,” a key word emerging in network discussions. “I think it's a whole new genre of software,” says Brody. “I was at [the AWS tech event] Re:Invent this year, and the big buzzword was observability. Because we've made things so complex, we now have this new challenge of how do we observe what's happening where? And how do we troubleshoot it? That wasn't a problem years ago.” Shrikirshnan agrees that “observability is huge,” and say a network has to be able to “receive logs and respond to events in real time.” For instance, what if a user is normally in Toronto, but suddenly shows up in Manila? “What's going on there? Is this a legitimate use case? Or is this a bad actor?” says Shrikrishnan. “That user in Manila may have left their iPad at home, and the iPad is now checking in for emails, but the user is physically in Manila. “Do you now take the traditional SecOps approach and kill their user account because you notice malicious activity? Or do you say, hey, wait a minute, this could be legitimate? Let me prompt them for credentials. And if it is a legitimate use case, do I need to now spin up

Network Automation 

digital infrastructure in Singapore to support them? because they need reliable secured connectivity back to my core infrastructure.” In a zero-trust network, users authentication is automatic, and continuous, says White: “When I was at Cisco, and I talked about network security, I had a slide which said we could do a crunchy edge with a really nice DMZ [De-Militarized Zone]. And the inside of the network could be really chewy, like a chocolate chip cookie, Nowadays, I’m sorry, but the entire network has to be crunchy all the way through security has got to be built in from the ground up and all the way through.”

Lifecycle Network automation also has to be able to handle the lifecycle of a network, during which time it will be maintained by multiple people with different levels of skill. “if you're a senior network engineer, and you've been in the trenches, you know what to build,” says Brody. “But someone newer and more junior may be tasked with simply deploying hardware and plugging it into some automation software. My fear is, how do I ensure that it's not going to do more damage than good?” It‘s tempting to design a network for Day Zero and deliver it on Day One, expecting it to carry on working, says Casemore: “It’s not just Day Zero and Day One. When you plan and design something, you have to deal with things like troubleshooting and remediation, and that closed loop.” Automation has to work on Day N, he says: “So you're able to optimize change management, and ensure that the network is continually refined so that it produces the results that it needs to deliver for those applications that it supports.” For Brody, the important thing is to have a reference architecture that determines how different clouds and services can be combined as needed. White says it’s a matter of trying to build networks things in a simple, modular way that can be automated: “Because there's a limit to how much you can hold your head. And if you've made it too complex,

you can't be flexible, because nobody can figure out how to make it work.”

when you add things to it you're pushing things into the graph using a graphical interface.”

How it works in practice

The system can continually probe whether the graph matches the intent, ie whether there is a fault or a failure.

So far, so theoretical. But what happens when you want to actually deliver an automated network? Let’s take as an example, the Apstra network automation system that Juniper acquired and uses. Apstra coined the term “intent-based” network for the jump from automation to intelligent automation, explains Juniper network engineer Mikko Kiukkanen: “You're describing the intent. What you want to do, not how you get there.” Some tools automate tasks like IP address generation, but don’t verify them. An intent-based network will be based on a reference design or “graph” which describes what the network is meant to achieve. This is mapped onto a network which can be made from multiple vendors’ hardware. “We generate the syntax, after validating that the configuration and the design is correct, and push the configuration to the switches,” he says. “This happens on day one, where you implement it, and hand it over to operations. After that, we do the dayto-day operations, which means the monitoring and troubleshooting side of things. The network behavior is generated from the network design, which is stored in a graph database on Day Zero, say Kiukkanen: “It's a complex data store, which is connected to a router. It gives us now a much, much more granular view into the data center.” When the network is running, the graph runs in sync with the real network he says. “Rather than querying devices and looking at log files in real time, we can query the graph, because it's a single source of truth.”

The intents can include service levels, so if a network link needs to operate at no more than 90 percent capacity, the system will flag up when a change breaks that intent, he says. “If there is an anomaly like a duplicate address, we flag it. And then you just hit one click, and it will say don't let it happen,” he says. “We can make an alarm for an anomaly,” he says, and issue a trouble ticket automatically for the fix if human intervention is required. “It’s like autonomous cars,” says Kiukkanen. “We want a self-driving or selfoperating network. Are we there yet? Not quite. But we have the pieces.”

No going back In the pandemic, network automation was put to the test, as thousands of users started to work from home: “An inflexible core data center architecture would not have allowed that.” Intelligent networks have to operate autonomously, adjusting to deal with faults and surges in demands. “This is the coolest time when you're talking about intelligence and automation,” says Shrikrishnan. But it’s always going to be a limited kind of autonomy, he says: “I don't want to put the intention forward or the message forward that we're trying to build Skynet here with Intelligent Automation. It's a little different.” 

The automation runs on the “control plane”, the management interface of the switches, not the the “data plane”, the general bits they transfer: “This gives you the flexibility to add equipment, because

Automation Automation Supplement Supplement || 77

Demystifying Data Center Complexity in 2022 Approaches for creating smarter data centers in the year ahead

he complexity of data center environments was growing by leaps and bounds before 2020. When the COVID-19 pandemic hit, that complexity accelerated to a new level.

Operators who thought they had a handle on managing their assets prior to the pandemic were left scrambling to keep up with the accelerating trend toward distributed IT environments, demand for greater capacity to meet new business requirements and problems finding qualified staff, among other challenges. As data center operators seek to navigate the continuing complexity of the landscape in 2022, many will do so amid mounting pressures to be more cost-effective, higher performing and more sustainable as well. Thankfully, there are tools and approaches available to help untangle this complexity in a way that can lay a foundation for future flexibility and growth.

A multitude of challenges The increasing complexity of IT and technology advances and business demands has upended the idea of the traditional, centralized data center environment and even redefined the concept of the data center itself. Because of this, each environment is becoming more unique in such a way that simple, one-size-fitsall solutions are insufficient to meet the growing list of challenges operators face. One of those challenges is the explosion of compute and storage at the edge. The push to bring business-critical data closer to the end user while helping alleviate capacity and bandwidth issues in the core data center has pushed more critical infrastructure to distributed environments, creating an unwieldy architecture that becomes increasingly difficult to manage. According to the Uptime Institute, 58% of companies still expect to see a significant increase in edge computing going forward. As more

DCD Supplement • datacenterdynamics.com

distributed environments come online, it becomes more challenging to monitor and address issues such as power outages when they inevitably occur. In the face of greater demands and a finite amount of data center space, operators should consider whether there’s an opportunity to optimize assets in their core data center and better manage capacity. The cloud may be attractive to some, but the same Uptime Institute survey found 73% of operators were unwilling to shift critical workloads to the public cloud. Rather than buying capacity in colocation facilities or building a new data center, both of which are costly and result in less control over assets, many wonder how to better use the capacity and assets that currently exist in the core data center. Operators weigh these concerns while managing the same significant hiring struggles that have impacted many industries amid the pandemic. The Uptime Institute study found that roughly half of operators have difficulty finding qualified staff to fill

EATON | Advertorial 

Mike Jackson, Director – Offer Management, Data Centers, Eaton

As the Director of Data Center Offer Management in Eaton's Digital Office, Mike Jackson and team are responsible for executing product management of data center software and digital services as well as the global go-to-market strategies for Eaton in the data center segment. You can find more information at Eaton.com/BrightlayerDataCenters.

open positions in the data center, leading to a concerning lack of on-site support or staff to manage critical functions. This can make addressing issues with capacity and workloads even more daunting, as operators are forced to be more efficient with fewer hands.

•

These challenges alone, and the complexity caused by them, are enough to keep even the most seasoned operator up at night. Thankfully, there are ways to employ both technology and processes to help mitigate these issues while enabling flexibility and the opportunity to optimize for future growth.

Unraveling the complexity There are many tools and approaches available to help data center operators address the growing complexity of their environments. Some of these include: •

Best-of-breed monitoring: Remote monitoring has emerged as a highly effective solution for operators seeking to manage the complexity of their environments better, especially as they continue their move to the edge. Remote monitoring tools are essential for understanding the overall health of data center infrastructure (in both traditional centralized and edge environments) in real-time and over time. They can provide alerts and alarms to help operators understand when problems occur, get a better handle on assets to save energy and costs, and often feature reporting, trending and dashboards that provide insights on growth over time to help further address challenges.

•

Capacity management: Capacity management tools can help operators better utilize their assets today while providing a data-driven approach to expanding in the face of increasing capacity demands. Deploying these tools can help automate the tracking of data center assets and how they are used, overcoming human errors and helping the operator make smarter decisions to expand intelligently. For example, capacity management tools allow an operator to proactively analyze the impact of a new deployment project by evaluating if enough space, power, cooling, network ports, etc., are available before deploying a project, not during the physical deployment. Data center automation and managed services: Given the staffing challenges many operators experience today, leveraging data center automation can help them better manage time and resources and focus on delivering business-critical services to customers. Additionally, operators can partner with managed service providers to help take on time-consuming tasks and workflows, alleviating some of the burdens that come from a lack of adequate support staff until they can make the right hires for open positions.

optimizing both core data center assets as well as edge deployments. This all starts with documenting and ensuring data is accurate.

Unique data centers, unique solutions Many of the tools outlined in this article are available in Data Center Infrastructure Management (DCIM) software solutions. However, as mentioned earlier, every data center is unique, and a one-size-fits-all approach will likely be insufficient to pinpoint and prioritize every issue that must be addressed. Operators must identify a vendor who can work with them to understand their unique challenges and be flexible enough to deploy solutions that achieve the specific outcomes they seek. While technology can be a key solution, having the right human knowledge and processes are essential to overcome obstacles. Operators should understand what tools they currently have in place – and their role – to ensure that, as they embrace new solutions, their organizations are prepared to grow into any process changes that may come with them. When deployed effectively, these tools can help operators demystify data center complexity while helping position themselves for future growth and success. 

Data is the key theme in helping effectively use these tools to reduce complexity. Technology can help automate processes and workflows, but its real value comes from having accurate data and insights into data connections across the devices managed. Data drives views into state of health, capacity planning, load/environment balancing, and

Automation Supplement | 9

Peter Judge Global Editor

onstruction might seem like the ultimate muddy-boot occupation, but there’s a company aiming to disrupt it with augmented reality (AR). And unlike a lot of AR headsets, this is not just a high-tech toy: XYZ Reality reckons that a high-tech hard hat on an engineer’s head can pay for itself in six months. It’s hard to grasp the proposition at first. The XYZ Reality Atom headset looks more like a futuristic bike helmet than something you’d see on a construction site. Shimmering mirrors glint inside the goggles. Videos of the HoloSite AR system show virtual buildings springing into existence before the engineers’ augmented eyes. But CEO and founder David Mitchell is very definite that the headsets are not enabling science-fiction castles in the sky. “I do like sci-fi,” he tells us. “But I am very much focused on the human aspect, and this can allow humans to build better.”

HEADSETS

that think

Engineering grade augmented reality headsets could make the construction process more reliable

The headset includes laser-driven AR that is accurate to within 5mm, and can load up schematics for every aspect of the building including civil, mechanical, architectural and structural, so the wearer sees the electrics, pipework, and the eventual physical structure, all overlaid over the current state of the project.

Right first time “The value proposition is, build it right the first time,” Mitchell tells us. “The industry is plagued by rework.” All building projects suffer when parts are not quite correct, and need to be worked over, he says: ”Seven to 11 percent of each project is rework. That is wasted effort.” AR can change that, he says, citing a very concrete example from one project. “After they broke ground, two months in, we came onsite with the headset, and they spotted problems left, right and center.” One concrete pad had been made 500mm too high. “That would have incurred substantial delays later,” he explains, “It normally takes two weeks to identify that kind of error. And if you verify the work after it is completed, the cost has already been incurred.” With a two-week delay, the project managers would have been faced with the choice between

10 DCD Supplement • datacenterdynamics.com

Headsets that Think

knocking down the work so far, or engaging in expensive adaptations. As it was, the fix was relatively quick. But the benefits could have been far greater: “That’s using our tech as a reactive process. Drive it deeper into the quality process, and you can go in advance.” If the AR system had been in use at the start of the project, the error would never have occurred. It would have been spotted and fixed when the rebar was installed, before the concrete was poured.

Architect’s view Mitchell has a family background in construction. He learned to walk on his father’s building sites on the West Coast of Ireland and got involved in commercial construction, helping to build small tower blocks and hotels. “Then I became a bit of an architect,” he says, studying the subject and working in Paris for a time. “But I missed the construction side, and moved back to London. I got involved in some incredible projects - like the Shard and Battersea Power Station.” He also helped build hyperscale data centers in Europe, though he’s not saying who the clients were.

“I started questioning why builders use 2D drawings on site,” he tells us. Architects and engineers use 3D CAD systems and the designs exist as BIM (building information modeling) data. But on site, the construction staff have to work from a hard copy, which holds less information and is fiendishly hard to marry up with the real world. “As an architect, sometimes I struggled to understand my own drawings,” he says. Working for J. Coffey Construction, he wanted to give the building workers something better. “Back in 2015, we deployed paperless constructions on a building in Ireland,” he says. The project gave the builders more direct access to BIM data. “It was a huge success - everyone was sold on the idea.” At that stage, he could see the potential, and wanted to take it further. He told his firm: ”This is the future. I’m going for it.” And Coffey agreed, giving him pre-seed funding. Five years later, XYZ Reality has 70 staff. Backed by $26.5 million in venture funding, its headsets have showed up at projects worth a total of nearly $2 billion. Data centers are a major part of its market - albeit a secretive one. All Mitchell will say about a major current project is it’s a hyperscale data center in Europe. Elsewhere, however, XYZ’s partner PM Group mentions Denmark, so we can be pretty sure we’re talking about the Meta/Facebook site currently expanding in Odense.

Engineering grade AR XYZ designed a health and safety compliant helmet, with waveguides to project a hologram in front of the wearer’s eyes. The company opted for a laser-driven positioning system to improve the accuracy over consumer-grade AR systems which sometimes use cameras for positioning: “You cannot rely on feature points and the camera position.” The positioning system works with a site coordinate system to give millimeter accuracy, and the images are validated by laser-scanned reality capture. “There are a number of prosumer devices that are marketed in a way that makes this kind of thing look feasible, but we quickly realized that, for AR to be usable, it needs to be accurate within construction tolerances.” Prosumer devices like Microsoft’s Hololens can often drift by 100mm, and be affected by factors such as sunlight he says: “We guarantee we can position AR with engineering-grade accuracy.” Those holograms can selectively show different disciplines including civil, mechanical, architectural, and structural engineering. “We are 3D creatures. It makes more sense to see things positioned accurately,” he says. “When someone puts on that headset, they get a big smile on their face.” He sums up: “Ultimately our vision is builders building from holograms.”

Automation Supplement | 11

“We are not potentially replacing people. We can delay the invasion of the robots. But we could feature ‘cobots’ that interface with people” Data centers - the perfect use case Historically construction is seen as conservative, but data centers are more aware of the potential of technology - and they also have a skills shortage that has become acute during a surge in demand for new facilities. Those facilities are also often very similar, with one providing wanting to build the same building, over and over again. And that’s a big opportunity for a new datadriven technology to show its worth. “Data centers are cookie-cutter projects, so we can measure benefits directly against previous buildings,” says Mitchell. Data center builders that adopt AR can directly see that, where one project required 10 percent rework, the next one needed less than one percent. Given the large costs of that rework, he says the return on investment can work out well, with one system paying for itself nine times over in six months: “The drop off in rework happens instantly.” It’s also been used to verify the equipment going into the data center. “We have done factory witness tests, streaming into the factory environment.” In these tests, engineers measure skids of equipment designed for a data center before they are shipped. “We’re able to check that the site can host it accurately, and all key interfaces are built correctly Mitchell won’t say how much the headsets cost, explaining that they are

Audits and support

Plenty of virtual reality and augmented reality headsets have already been seen in working data centers, and in virtual simulations of them. Some have been demonstrations or prototypes, while others have helped with troubleshooting. Back in 2017, Future Facilities, a company that optimized airflow using computational fluid dynamics (CFD), investigated using VR to explore CFD simulations of data centers. Future Facilities used VR to let users view the internals of several different generations of data center, including temperature levels and airflow.

provided as-a-service: “The cost depends on the duration of the project. We have a subscription model, charging per unit per month.” This means that customers can increase the number of users with headsets as required, during construction peaks, and can roll on headsets to new projects. “One project started with one headset, now it has three, and it’s moving to eight-plus. They keep coming back for more.”

Can you automate it? But can the system evolve into a level of automation? Mitchell says it already allows for an increase in productivity from existing workers: “This industry is suffering from a massive skills shortage. Giving this to people in the industry, assisted reality will assist them in building what they are building.” The system also allows remote staff to share the view from the headset: “We are happy to stream into the headset,” he says. “We prioritized streaming and remote working as soon as pandemic came in,” he says. “We wanted to help the market.” Adapting what the company had XYZ patched users into Teams calls so some customers would work remotely on-site using Teams and HoloSite. “We reduced the amount of labor and travel,” he says. “40 people streaming into a headset were able to do a virtual handover In 2018, veteran data center engineer Greg Sherry, launched a company called Virtual Augmented Reality for Critical Environment Technical Infrastructure (VARceti), proposing a training program based on VR and AR technologies aimed at data center engineers and technicians, called Avros DC. Describing it at the time as a “flight simulator for data centers,” Sherry told DCD. “We can start a fire in the data center, or flood it, or make a tank come crashing through it.” During the Covid-19 pandemic, Microsoft's cloud operations team used the company’s HoloLens 2 AR glasses to conduct data center audits remotely. This allowed the team to comply with Covid-19 precautions and made the inspections quicker and cheaper. The team used Microsoft Dynamics

12 DCD Supplement • datacenterdynamics.com

and walk down of the project.” He doesn’t see it going further any time soon, where are a robot or drone provides the remote eyes. “We are not potentially replacing people. We can delay the invasion of the robots. But we could feature ‘cobots’ that interface with people.” So the headset isn’t a step towards a self-building data center, but there are fascinating additions in the pipeline. “The next phases involve ‘assisted reality’” he explains. In this development, the headset can monitor aspects beyond the current user’s job definition, and passively detect what’s been done on site. Once again, it’s about spotting errors, but in a wider context. The system can be worn by a construction worker, and passively detect a mechanical system is out of tolerance. It would then pass an alert directly to the mechanical engineering team - assigning the work to the right people automatically. “We might have made mistakes historically,” says Mitchell. “Now we know it will be built right first time.” 

365 Remote Assist running on its Azure cloud, with HoloLens 2, having first assured themselves that the Azure platform complied with standards such as the PCI data security specifications. Meanwhile, some hardware manufacturers have added AR capabilities to their remote management products. Again, inspired by travel restrictions early in the pandemic, ABB Electrification added immersive augmented reality to its support and maintenance offerings. The company launched Closer, a set of interactive step-by-step guides to help lesstechnical staff get through troubleshooting tasks. It also offered a product called Raise for remotely guided repairs and maintenance of ABB products. 

Drones 

Drones at (and even in) a data center Do today’s mega-campuses need aerial security as much as human patrols?

ata centers are continually driving to become more efficient, and to get more useful work out of staff. In the case of giant hyperscale data centers, increasing automation may be calling time on the practice of people patrolling thousands of yards of data center perimeters. Novva data centers, led by former C7 CEO Wes Swenson and backed by CIM Group, has a 100-acre flagship in West Jordan, Utah - a campus which could reach to over 1.5 million sq ft (139,350 sq m) of data center space. The first phase, involving a 300,000 sq ft (28,000 sq m) data center was completed in late 2021 and includes a 120MW substation as well as an 80,000 sq ft (7,500 sq m) office building for

Novva’s headquarters. To cover that much space, Novva is turning to autonomous drones and robots. The company is deploying Boston Dynamics’ Spot robot to patrol data halls, as well as semi-automated security drones (also known as Unmanned Ariel Vehicles, or UAVs). “When you run a 100-acre campus, you really should have an aerial view of the operation,” says Swenson.

Drones take flight In 2021, Novva deployed two Blackbird drones from Nightingale Security – and plans to have four on the Utah campus. “For the most part, it just does its own thing and then just autonomously goes back to its landing site,” says Swenson.

Dan Swinhoe News Editor

Equipped with 4K cameras, LiDAR and infrared, the quadcopter drones perform regular autonomous perimeter checks, as well as responding to ad hoc alerts. The drones run around 10 pre-defined missions, providing perimeter checks, facility inspections, and can react to certain alerts automatically. “We have detectors on our fencing that detect any kind of vibration, and an algorithm that detects whether that vibration is due to the wind speed or some other interference,” says Swenson. “If we think it’s somebody trying to climb over the fence or cut it, the drone will automatically launch towards that sensor and get eyes on it.” Robots in data centers are still niche: while

Automation Automation Supplement Supplement || 13 13

they might be used during the construction phase, use of drones at data centers for security and ongoing inspection is rare. As well as Novva, Nightingale has been reportedly working with ‘a very large Silicon Valley company that runs their own data centers’ for around two years, and is in the process of conducting a pilot with an e-commerce company in Washington. The company was deployed at the Ohio data center site of a US investment bank in during the construction phase. Data center operator BSC is also considering drones at its facilities. Hector Castenada, senior director, service delivery and technology for BCS, says the company is hoping to deploy a pilot project at a 300,000 sq ft data center close to its Texas HQ by Q3 of 2022. As well as pervasive security drones, Castenada is considering drones for facilities inspection – so visiting engineers can inspect chillers or other equipment on the roof without climbing up themselves – as well as wider land and data center surveying. While the project is still in the research phase, he would expect to see around two permanent security drones at each facility – ensuring one can patrol while another recharges – alongside smaller held-held machines engineers can bring on-site for inspections. “I definitely see this becoming the standard in a few years,” he says.

Integrating data center and machines Nightingale offers its drones, base station, and software as a package to buy outright alongside a yearly subscription, or on a lease basis. Swenson says Nightingale was the UAV company which offered Novva the closest to a complete package, as opposed to point solutions. “Most drone companies are basically remote control aircraft companies,” says Jack Wu, founder & CEO, Nightingale Security. “They’re just building something based on RC [remote control] technology that’s been around for decades. We’re trying to marry the autonomous technology, with robotic software and robotic infrastructure.” Drones have improved in leaps and bounds in recent years. They can now

remain in the air for long periods and are able to fly autonomously in difficult conditions. The software and wider support ecosystem isn’t quite as advanced. If data centers are to become more automated, the robots deployed there will need to be easily integrated with existing management systems, something which is still in the nascent stages. “Automation is a huge trend,” says Wu. “You’re going to have ground-based robots, drones, other automated systems. We’re not at the age where we’re all integrated yet. But that is coming.” Wu suggests we are still in the ‘discovery phase’ where customers learn and understand the potential use cases of different robots, while the robotics companies are working on getting each platform working properly, independently. After that, the industry can focus more on integration and putting more robot platforms into a single dashboard. “First phase is the introduction of these select platforms. And then after that, it will be the phase of integration, where all the different platforms will start working together.” While Swenson says Nightingale was one of the most ‘enterprise-ready’ offerings on the market, deployment still takes work. “This is not something that you just go buy off the shelf and drop in. It does take some work. It takes some on-site consultations and setup,” he says. “The drones themselves are pretty mature, but to integrate it and run missions, that is still pretty advanced.” Castenada says BSC is researching the integration of drones for extended duties. “I’m really keen to use the drone not only for perimeter security, but to interact with, a chiller or an external generator; external devices and external critical equipment.”

“What we’re looking for is integration capabilities with our BMS [building management systems]. Right now we’re researching which drone to integrate with video management systems like Genetec.” Currently, Azur Drones of France appears to be the only UAV company with a publicly announced integration with Genetec, says Castaneda. He also says there are a few data centers that integrate with drones. The company is also reportedly exploring Wingtra, which offers a vertical take-off fixed-wing drone that can be used for surveying.

Humans need drone training While different drone companies will have different processes, prior to deployment Nightingale will come to a location, survey the environment, map the buildings and potential obstacles, and help train the staff. Wu says the process normally takes about a week. “Each local environment is different,” he adds. “They have different wind patterns, different temperature, and also their operational tempo is different; some use it 12-13 times a day, others use it twice a day.” The software system includes predefined limits around altitude and no-fly areas. At Novva drones won’t fly over the on-site substation or peek into neighboring buildings. Nightingale will help establish those initial restrictions but the customer can change those as required. It’s surprisingly straightforward to pass FAA requirements to fly a drone. The Nightingale flight system is designed to meet US Army standards. Wu says this means they are usable by people with a minimum IQ of around 80. “Part 107 [the FAA’s drone certification program] is only a 60-question, multiplechoice questionnaire,” he explains. “Anybody

“If your data center is in a residential, super metropolitan area, it’d be a little bit risky to be sending a drone up in tight quarter. People are gonna get sensitive about it and you’re gonna get calls; they’re just too dystopian"

14 DCD Supplement • datacenterdynamics.com

Drones 

who can use an Uber app and Google Maps can use our software.” Nightingale put DCD in charge of a drone in a live demonstration, allowing this writer to fly a drone round the company’s facility in California, from the comfort of a flat in London. Moving the machine was a simple case of point and click, and despite my efforts I wasn’t able to crash or break through the no fly zones. During the flight, the drone was able to identify and flag vehicles in the company’s parking lot outside usual working hours. Swenson says everyone at Novva’s control center, including himself, is FAAcertified to fly a drone, and the company has different internal pilot statuses. Less certified members of the team can pause and rewind pre-set missions to check for anything the drones may have missed – and control viewing angles – but those with top-flight status can take full control of the drones. “The drone itself requires very little human intervention once it’s preprogrammed,” says Swenson. “And even then it has all sorts of safeguards; emergency landing zones, no-fly zones, altitude zoning.” “It’s also something that we felt like we could copy and paste to other facilities and build a training structure for every data center. It’s something we can replicate more easily.” Wu is clear, however, that his company’s machines are complementary to humans, rather than a replacement. “We’re not here to replace all human guards because there are tons of things that humans can do that we can’t. So robots are good at repetitive tasks, dangerous environments, and the ability to be very cost-effective, but we can’t answer telephones, and we can’t provide directions.” “And we definitely can’t discern the context of a bunch of kids smoking weed, or a bunch of guys with guns, in the parking lot. The context is very important, and humans have that context.”

Building a base for your robots The Blackbirds live within a designated enclosure with a retractable roof that acts as a landing pad and charging pad. Swenson describes Utah as a ‘cold desert’, just as likely to see snow as intense heat, so it is important the base station is weatherproof and can deal with snow without needing a person to clear the landing pads.

The razor-wire-covered drone compound where the base stations reside sits at ground level on the Utah campus. It measures around 16 feet by 20 feet – the base stations are around 7 feet by 7 feet – and the pad has dedicated power and fiber running to the stations. “We needed something that could sit in a weatherized vestibule. You could put this on the top of the building but it does take some planning; it’s not something that you can just go in and drop in in a day with a WiFi signal. But compared to building a data center, putting in one of these drones was relatively easy.” Wu describes the base station as an ‘Edgebased computing device’; the station stores the video feeds and other data collected by the drones and runs the footage collected through machine learning algorithms, for example, to detect cars or people. “It has a large PC inside, tons of memory, tons of storage, and a lot of horsepower to crunch numbers,” says Wu, adding that processing and storing the video streams at the base saves cloud costs and provides added security. When asked if the drones have helped prevent or deal with any incidents, Swenson says the company has not caught anybody trying to get into the facility since it opened, but the drone has made some avian friends. “Sometimes we have some birds of prey out here that find the drone quite interesting when it’s flying around; we have a few eagles and some hawks that find it interesting but for the most part, they keep their distance.”

When drones makes sense Wu suggests campuses less than 30 acres probably won’t need a drone for patrols, as people and cameras can do the job well enough. For larger campuses where it can take minutes to get across site even in a car, drones potentially offer a more efficient patrol and quicker response option. “If you have large perimeters, it just doesn’t make sense for you to put cameras on them because it’s very expensive,” he says. “It’s not just the cameras - you also have to run power. And then you also have to run data, so that becomes a huge cost factor.” While large remote industrial sites are suitable for drones, deploying a drone at

an inner-city carrier hotel might not make sense. Wu notes the more urban a proposed site is, the more difficult it is to gain approval from regulators to fly. “If your data center is in a residential, super metropolitan area, it’d be a little bit risky to be sending a drone up in tight quarters,” adds Swenson. “People are gonna get sensitive about it and you’re gonna get calls; they’re just too dystopian, and people might feel like they’re being spied on.” While the company helps train staff, Wu notes it’s ultimately up to the customer company to understand restrictions and potential liabilities with relation to flying drones, likening the relationship to that between a carmaker and the driver.

Drones inside data halls In future, Wu suggests the systems will be more autonomous and less reliant on a dashboard and point-and-click interfaces. “Eventually you’ll just install the system, and the person will just talk to it: ‘Hey, Blackbird. Can you do a perimeter patrol of Section A or Parking Lot 1’, and the drone will take off go and do it.” “And then when it lands, it will give an alert to the person the mission is complete, or, if it sees something suspicious will automatically say, ‘you should take a look at this’ and provide the live video.’” Castenada says if his test pilot goes well, he’d like to see BSC start to use headsets such as Microsoft’s HoloLens to control the drones. He also wants to share the drone’s footage with customers. “We would like to introduce transparency. A customer wants to see their facility, and we’d like to offer a secure portal for them to be able to view and watch their property.” Today Novva only has medium-sized drones that operate outdoors, but in the future, the company wants to have microdrones within the data halls - perhaps flying eight to 10 microdrones per building. “Right now we’re experimenting with smaller drones that are three inches in diameter,” says Swenson. “They would operate within the data center and be able to monitor things [and watch for] anomalies within an aerial view.” 

Automation Supplement | 15

Brightlayer Data Centers suite The data center’s best kept secret to monitoring and managing end-toend IT and power infrastructure

Ensure business continuity with predictive capacity planning

Visualize your entire operation in real time to optimize staffing efficiency

Reduce operational expenses up to 30%

16 DCD Supplement • datacenterdynamics.com