20 minute read
Who’s a good bot?
As Novva deploys Boston Dynamics’ Spot dog robot at its Utah data center, CEO Wes Anderson says larger data centers need automated security
Dan Swinhoe News Editor
Imagine the scenario: you arrive at a data center in Utah around the holiday season. As you walk the aisles of the data hall, a four-legged robot dog in a Christmas jumper walks up to you and greets you by name before returning to its rounds.
Such a scene isn’t some poorly-written and far-fetched fan fiction, but a potential reality. Novva data centers, a new firm led by former C7 CEO Wes Anderson and backed by CIM Group, has partnered with Utah’s Brigham Young University to deploy customized Spot dog robots developed by Boston Dynamics in its data center.
First announced in September, Novva said the Wire (Wes’ Industrious Robot Employee) machines will fulfill “multiple missioncritical roles” at its Utah campus, including temperature and equipment monitoring, greeting guests, confirming building occupant security clearance via facial scanning and recognition, and other general ‘tasks and missions.’
As we’ve previously written, robots in data centers are often talked about, but rarely progress past the prototype or pilot project stage. But Anderson tells DCD bigger and bigger data centers will need automation and robotics to ensure optimal operations.
“We're trying everything we can to build a better data center, so why not give it a shot?” says Anderson. “So far we're actually really happy with what it does - but it is cutting edge. It is not for the light-hearted. You really have to be committed to it.”
The promise of robots in the data center
Data centers have a long and patchy history with deploying robots. The long-heralded lights-out data center maintained by robots remains out of reach, and deployments of robots cohabiting with humans have had mixed results.
Back in 2013, IBM hacked an iRobot (similar to the Roomba autonomous vacuum cleaner) to travel around a data center tracking temperature and other data. The pilot project was quietly shelved.
The Korea Advanced Institute of Science and Technology tried a similar idea; its Scout bot patrolled the KAIST iCubeCloud Data Center using vision-based monitoring to look for issues. A follow-up study promised to attach a robotic arm that could work on servers. It was never published.
Naver’s Cloud Ring data center in Sejong City, South Korea, also promised to use a number of robots for maintenance, monitoring, and security. The facility was due to come into operation in 2022. It has made a number of announcements about deploying robots at its offices as well, but the machines don’t appear to have been rolled out yet.
However, some companies have seemingly managed to successfully deploy robots at certain facilities. German Internet exchange company DE-CIX has rolled out a family of automated “patch robots," including Patchy McPatchbot, Sir Patchalot, and Margaret Patcher. These are based on X-Y gantries, and can locate a socket in an optical distribution frame, similar to a traditional patch panel, and then plug a fiber optic cable into it.
“During the 2020 11.11 Global Shopping Festival, Alibaba Cloud data centers had its Tianxun inspection robot upgraded to the second generation,” Wendy Zhao, Senior Director & Principal Engineer, Alibaba Cloud Intelligence, previously told DCD.
“The second-generation Tianxun robot is AI-powered and can work without human intervention to automatically replace any faulty hard disks. The whole replacement process, including automatic inspection, faulty disk locating, disk replacing, and charging, can be completed quickly and smoothly. The disk can be replaced in four minutes.”
Switch is developing its own robot, the Switch Sentry, essentially a 360-degree camera and heat sensors on wheels that can act as a security guard. It travels autonomously, but humans take over remotely when an incident occurs. The company said it hopes to turn the
robot into its own business line, but no such timeline has been announced.
Spot the Boston Dynamics dog has been used by a number of companies, including Tesla, for facilities surveillance. However, even though Boston Dynamics’ former parent Google has more than enough data centers, Novva is apparently the first and possibly only data center Spot has patrolled.
Why bother with robots? Novva’s flagship data center is a $1bn, 100-acre campus in West Jordan, Utah. Construction is taking place over four phases and will total over 1.5 million sq ft (139,350 sqm) of data center space when finished. The first phase, involving a 300,000 sq ft (28,000 sq m) data center was completed in later 2021 and includes a 120MW substation as well as an 80,000 sq ft (7,500 sq m) office building for Novva's headquarters.
“With such large facilities, it really requires kind of a different approach. We had not seen this before in another data center, but it was really out of necessity that we did something because of the scale of our data centers.”
Anderson says it is taking a “holistic” approach to security and observation, with the Spot robots, drones outside – and eventually inside – and maybe the use of Boston Dynamics’ humanoid Atlas robots.
“Our whole automation approach is to get to where the humans can monitor for anomalies, rather than sitting there tweaking things manually,” he says.
Research firm Gartner recently predicted that half of cloud data centers will be leveraging advanced robots by 2025.
“The gap between growing server and storage volumes at data centers, and the number of capable workers to manage them all is expanding,” said Sid Nag, research vice president at Gartner. “The risk of doing nothing to address these shortcomings is significant for enterprises.
“Data center operations will only increase in complexity as organizations move more diverse workloads to the cloud. Data centers are an ideal sector to pair robots and AI to deliver a more secure, accurate and efficient environment that requires much less human intervention,” he said.
There have been a number of announcements around Spot and other robots dogs used for construction and facilities maintenance this year; IBM has partnered with Boston Dynamics to explore how to increase the utility of mobile robots used in industrial environments, while Tesla and Hyundai have reportedly the robots patrolling their manufacturing facilities.
Construction firm Robins and Morton has deployed Spot to help with the development of a HostDime data center in Eatonville, Florida. Verizon has partnered with Ghost Robotics to explore how 5G-enabled robots can support inspection, surveillance, mapping, and security.
Spot the dog at a data center
Anderson says the idea was to use Spot, or Wire to do some of the mundane tasks humans may tire of in a large facility. The machine runs pre-determined missions throughout the data center to collect data, monitor equipment, and report anything unusual.
“There are some things that humans get tired of, like the repetition,” he says. “We put QR codes on every piece of machinery in the data center. When human patrols or inspectors go around, they have to scan those QR codes as a routine facility check.”
Currently, Spot takes temperature readings and performs security tasks ensuring there are no unexpected visitors in the data halls.
“As it's doing its rounds, it's feeding back all of that information into us into the control center, where we do statistical quality control and watch for anomalies,” says Anderson. As well as taking a reading of the ambient temperature of the room or data aisle, it can also read the displays on equipment and feed that back to the control center.
It also performs a security role: The dog is integrated with Novva’s security database, and will scan the faces of people in the data center against the company’s list of authorized or expected personnel. Recognized visitors will be greeted by name via a generated audio file, strangers are reported to the control center.
“When it recognizes a face, It will check the database and it will say good morning to your name,” says Anderson. “If it doesn't recognize you then it reports it back to the control center as an alert; it marks the GPS location, takes an automatic photograph, and then the control center will follow up on it. The robot will then either stay put or continue its rounds.”
Developing Spot for data center use
Novva partnered with BYU to build upon Spot’s Software Development Kit (SDK) and create a more user-friendly software package that made it easier to edit and improve going forward.
“We employed BYU's engineering department to help us advance that programming, and get it to a stage in which the software is easier for my technicians to adjust missions,” says Anderson. “BYU did a great job with it. They had it for almost a year; that kind of gives you an idea that it takes quite a bit to do this.”
Joseph LeCheminant, a Mechanical Engineering MS Student at Brigham Young University Capstone, was involved in the project as part of his undergraduate degree.
He was part of a team of seven students; a mix of mechanical, electrical, and computer engineers that worked on the project for two semesters (around eight months in total).
“It’s definitely something really exciting to see in person. You just look at it and [think] how is this real?” he says. “It was fun to branch out a little bit from traditional mechanical engineering.”
Part of LeCheminant’s role in the project was to help write the custom API to connect Spot’s facial recognition capabilities developed by BYU to Novva’s own facial recognition database and generate the audio files to say the person’s name.
The robot comes with advanced capabilities for walking and object avoidance; the team added software to use Spot’s cameras for facial and meter readings, as well develop its autonomous route navigation capabilities. On the hardware side, the team added the thermometer to take a reading of the atmospheric temperature in the computer rooms.
LeCheminant says the team had something of a learning curve with Boston Dynamics’ SDK for Spot, especially as some of the students had no experience of coding, let alone coding in Python as required.
“We the mechanical engineers had to learn a bunch of things that the electrical and computer engineers already knew, and then we needed to understand the SDK that came with Spot to develop it to customize it,” explains. “None of the mechanical engineers had ever used Python ever, and by the end we could write a Python script from scratch in Python and make Spot do things.”
Novva has since upgraded to a later model of Spot, but the one the BYU team were developing on didn’t have battery charging capabilities, which meant the students had to regularly change batteries by hand.
At the time, Spot also was limited on connectivity and wouldn’t connect to WPA Enterprise Wi-Fi networks – though support for that has since been added – so the team had to buy its own routers and set up its own network to run the robot dog.
During development, the students had regular meetings with Novva to discuss requirements and development progress. The students’ involvement in the project finished before the robot was actually deployed in the data center, but both BYU and Novva regard the project as a success in terms of the work the
students did.
LeCheminant describes the project as a “huge privilege to us as students.”
“I think it's a really cool statement that they've made,” he says. “It was great to work with them. They were easy to work with, extremely flexible, and just happy to provide what we needed to succeed and give us the learning experiences students.”
The software packages developed by BYU should be easily editable for future Novva development and allow for further improvements in the future.
Anderson says that while some of the project may have been developed using open source tools, Novva currently owns the custom software packages developed during the project, and would be unlikely to license or resell them to a competitor currently. It would, however, consider contributing part of what was developed to an open-source foundation or project dedicated to robotics.
Deploying Spot in Utah
Novva currently has two Spot robots, but aims to increase the number as it expands its campus and also use Spot in other facilities.
“There will probably be three to four dogs per facility,” says Anderson, “So in Utah, you would see anywhere from 16 to 20 dogs. We like to be N+1, so we want an extra dog on hand. ”
Novva acquired a second data center in Colorado in September, and Anderson says that the campus will have around six dogs once all three buildings are fully developed. Other planned facilities the company has in the works will also see dog deployments.
Spot comes with a control panel, and has to be manually walked around the facility to learn the building. As well as teaching Spot the current layout of the data center, the company also has mapped equipment that isn’t yet installed so that Spot’s patrol routes don’t need regular updating.
“A lot of the cabinets aren’t in place yet but the robot is programmed to operate as if they are so it doesn't take us a lot to deploy it,” says Anderson.
In its YouTube videos, Boston Dynamics often has QR-code like signage around its facilities and objects its machines are interacting with. Novva has adopted a similar system.
“We also have a QR type code system that sits about three feet above the floor throughout the building. Its eyes can recognize those to verify where it's at.”
In the near future, Anderson hopes to deploy the Spot/Wire dogs outside to respond to alerts and provide ground support to the drones.
“We are waiting for the next major upgrade of the robot from Boston Dynamics where it will actually have an arm, and one robot can open the door for another robot.”
“Eventually, I think even next year, we can get the robots to run payloads from the warehouse out to the data centers; they'll be able to pull small trailers and walk behind you.”
The uncanny valley, but a good dog
Anyone who has watched Boston Dynamics’ carefully choreographed videos is bound to have felt a sense of eeriness about the machines; while clearly not human or dog, the movements of Atlas and Spot can look strangely natural.
Anderson and LeCheminant both acknowledge the “uncanny valley” feeling the machines can inspire, but note the robots have a certain mesmerizing allure.
“The feedback from friends and family was this is a little bit uncanny and they definitely wouldn't want to come across this and have it come chasing them,” says LeCheminant. “But, as the developers, you understand that it only does what it's told. It does have some pre-programmed autonomy, but it's not gonna roam off on its own somewhere.”
LeCheminant adds one of the reasons the ability to greet visitors by name was added was to try and “break down” any barriers people might have about the machine.
“People get a little bit surprised by it, but it is captivating,” adds Anderson. “It stops you in your tracks when you see it.”
“Some people do get a little nervous when it gets too close. These things are quite large; they weigh a few hundred pounds, and they can be slightly dystopian in appearance”
Occasionally through our call, Anderson genders the robot before correcting himself, but accepts he and the team have a certain fondness for the canine-like machine.
“We kind of embrace it as a living thing, you almost can't avoid it. Even though it's just this robotic object, you do take some endearment to it. We're even having a dog sweater made for it for the holidays!”
When the machines are deployed outside, Anderson is keen to note that the machines are still only for surveillance and won’t be performing any kind of K9 heroics.
“It's not going to interfere. It's not going to attack, it's not going to defend, it's not going to do any of that stuff. And we don't really want it to. If it's something we need to call the police for, we would do that.”
Not the perfect monitoring machine
“For the most part, it runs its missions and then it comes back to its dog house where it sits down on a platform and recharges itself so it doesn't really require any human intervention to do that.”
“You have to clean the 360-degree camera once in a while but other than that, it's pretty hands-off.”
While everybody in the control center is trained on how to use Spot, the company currently has one developer internally to manage the robot’s ongoing development.
“If the operators are seeing something with the dog or the drone, then they just register a ticket with IT and we'll fix it. Or, if it's an improvement – like changing the angle of the gaze, things like that – then we put in a ticket and see if we can improve it.”
“At this stage, it doesn't take a team of people, but that's really due to the fact that BYU did so much advanced engineering on it and made the software interface, the user interface more approachable for us.”
While impressive, the machine isn’t yet totally enterprise-ready in terms of day-today operations. While data centers require constant uptime, Spot still requires regular downtime and isn’t quite ready for continuous development in terms of software updates.
As part of Novva’s contract with Boston Dynamics, the machines are entitled to regular software and hardware updates, but rather than OTT software updates or on-site maintenance, Novva has to send the machines away.
“Right now, we actually physically send the dog back and they may send us another dog or replacement dog with upgraded software or some kind of mechanical system. We may not get the exact same dog back.”
Anderson says Novva has to unequip the external equipment – LiDAR, 360 camera etc – before the company can send a machine back. And once a replacement arrives, Novva has to go through a QA process to check that whatever software upgrades implemented don’t clash with Novva’s software packages, and then make any amendments if required.
“Things like that are a little bit complex and burdensome. It's not the most plug and play technology at this stage. It does take some dedication of resource to make it work.”
While the company is hoping to make a number of software updates and enable new capabilities, it plans to only do large-scale changes every four to six months.
“We're trying to keep it in static mode right now. We're trying to not spend more time programming it than we are actually using it, so we're trying to limit its revisions.”
Another issue is connectivity. While the machine now has support for enterprise Wi-Fi networks, it has to be connected at all times
and currently has no option for the likes of LTE or any IoT wireless protocols.
“The robot has to work off of Wi-Fi connectivity; it's not really built for 5G and it doesn't operate outside,” says Anderson. “If it loses some kind of connectivity or things like that, it will just stop in its tracks, lie down, and alert the control center.”
Anderson says he hopes that Boston Dynamics will add some 4G/5G capabilities to the machine soon, but if not, Novva will look to customize the machine further to add that. And if that proves difficult, the company will expand its Wi-Fi to cover the wider campus beyond the offices and data halls.
He also notes there’s just no competing with human understanding and intuition on some things.
“It doesn't think on its own. It doesn't make decisions, when it sees something it doesn't react to it. It would take a lot of programming for a robot to hear friction and equipment and be able to notify the control center that it hears that; things like that are difficult and probably not worth it yet to program.”
More robots, but not less humans
The Explorer model of Spot costs $74,500, while the Enterprise model is more expensive. Extra equipment – such as the 360-degree inspection camera ($29,750), LiDAR sensor ($18,450), or on-board GPU processor ($24,500) – cost extra.
Anderson says that including maintenance and equipment, Novva is spending around $200,000 per dog; a price point the company finds effective in order to add an extra layer of monitoring and security it didn't have before.
Novva is also seemingly interested in Boston Dynamic’s human Atlas robot and keen to test its capabilities in a data center environment once the robotics company makes the machine available.
“We would very much like to try Atlas as well when they feel like that it’s ready for primetime,” said Anderson.
“Not that we would abandon the dogs altogether, but if we could get to the more humanoid [robot] where it's on two legs and it stands up and it can open its own doors and carry payloads, and lift the servers into the cabinets for the clients - that's where we're looking to in the future.”
But he reiterates that even deploying Atlas robots won’t lead to an en-masse replacement of human workers.
“It's really a complement to the human side of the business,” says Anderson. “There is really no replacement of humans for this. It is really complementary. In fact, I would say we've had to add personnel to program these missions.”
“It's not perfect, it's not plug and play and it still takes some programming adjustment. But we see this is a critical element of what we do in the coming years.”
Drones in a data center
As well as Boston Dynamics’ Spot robot, Novva is currently rolling out semiautomated drones for security purposes.
Equipped with 4K cameras, LiDAR and infrared, the quadcopter drones perform regular autonomous perimeter checks, as well as responding to ad hoc alerts.
“When you run a 100-acre campus, you really should have an aerial view of the operation,” says Anderson. “For the most part, it just does its own thing and then just autonomously goes back to its landing site.”
The company currently operates two drones – models known as Blackbirds and supplied by Nightingale Security – but plans to have four at its Utah campus. The machines live within a designated enclosure known as a base station with a retractable roof that acts as a landing pad and charging pad.
Novva has around 10 pre-defined missions the drones can run. As well as routine perimeter checks, the drones can react to certain alerts.
“We have detectors on our fencing that detect any kind of vibration, and it has an algorithm that detects whether that vibration is due to the wind speed or if it's due to some kind of other interference,” says Anderson.
“If we think it's somebody trying to climb over the fence or cut it, the drone will automatically launch towards that sensor and get eyes on it.”
The system includes pre-defined limits around altitude, no-fly areas (such as over the company’s on-site substation), and won’t look at certain neighboring buildings. Anderson says everyone at Novva’s control center, including himself, has to be certified with the FAA to fly a drone, and then internally the company has different pilot statuses.
Less certified members of the team can pause and rewind pre-set missions to check for anything the drones may have missed, but those with top flight status can take full control of the drones.
“The drone itself requires very little human intervention once it's preprogrammed. And even then it has all sorts of safeguards; emergency landing zones, no-fly zones, altitude zoning.”
Today Novva only has medium-sized drones that operate outdoors, but in the future, the company is aiming to have microdrones operate within the data halls.
“Right now we're experimenting with smaller drones that are three inches in diameter,” says Anderson. “They would operate within the data center and be able to monitor things [and watch for] anomalies within an aerial view.”
Anderson says that once deployed, the company aims to have eight to 10 microdrones per building.
