The Critical Power Supplement

Critical thinking

Adata

center's critical power enters from outside the building.

It comes from a grid which is shared by every other business and consumer in the area.

So it's clear that the critical power of a facility requires the operator to think outside the box.

Taking a long view, all developments in data center critical power will have an impact around the site, either by using shared power resources, or by creating local power (and emissions).

This supplement looks at some of the ways experts in the data center field are reshaping data center critical power to move that integration forward.

Backup to the fore Electricity grids are under stress. The war in Ukraine has pushed up energy prices in Europe, grids everywhere are shifting to renewable energy, and they are taking on new loads including electric vehicles. At the same time, increases in extreme weather are creating new burdens.

Data centers will be looking to increase their resilience in this world. European operators have been stockpiling diesel fuel for the short term risks of the winter (p13).

But there's a need for longer term thinking too. Running on diesel for longer will create emissions, and make data centers less reliable.

And there's an opportunity here to look wider. Data centers have treated backup as a private resource, but in an energy crisis, they can boost their reputation by sharing it.

Goodbye to diesel

Looking further out, operators are aware that their diesel backup generators are a very visible contradiction to their plans to lead the way to carbon neutrality.

Those gensets sit outside their data center where everyone can see them. They may only be used a few hours a year, but their emissions are more tangible than those from the day-to-day energy consumption of the facility.

Given that, it's no surprise to see data center operators looking for ways to eliminate the diesel they burn in backup systems (p4).

And there are plenty of options to achieve this.

Diesels can be replaced by batteries, fuel cells or other power supplies on a microgrid.

Ambitiously, they could be replaced by the first implementations of a potential new energy source - hydrogen.

Alternatively, they can simply be refueled with renewable diesel alternatives - and genset makers are working on ways to reduce test burns.

Using power twice

There's another very important aspect to data center power usage, and that is to consider getting the best use of all the critical power consumed.

A radical idea would be to actually use every Watt twice, providing useful heat, at the same time as digital processing.

That's the pitch of digital boiler makers like Qarnot, and it's one that has taken years to being to come to fruition (p10).

There's no reason why critical power should not be shared.

How many ways can you cut your UPS emissions?

UPS systems produce a very small part of the emissions from a data center, but operators are being very inventive in ways to reduce them

Data centers are supposed to be available all the time, but the power grid sometimes fails (see page 13). Facilities traditionally have an uninterruptible power system (UPS), which ensures that they keep going when the grid does not.

Typically, the UPS consists of diesel generators that fire up when needed, and deliver enough power to keep the data center running. The UPS also includes switches, and rooms full of batteries that store enough energy to keep the facility up till the diesel is running, and can support it.

Data center operators are all committing to become carbon neutral. They use renewable energy for their regular power needs, and are now turning their attention to replacing the fossil power in their backup systems.

Why do it?

It’s worth putting this in context. In most developed countries, power outages are rare and shortlived. So diesel generators are only fired up rarely. If there are no outages, they will be used for a total of a few hours per year, to test them.

If the operator does a full load test for half an hour, every couple of months, that amounts to a solid amount of energy used, with associated emissions. But it’s

a tiny fraction (less than a thousandth) of the energy used by the data center as a whole over that period.

However, if the operator switches its regular power to a renewable source, then the emissions from backup tests will remain stubbornly static, and become a (slightly) larger proportion of the overall emissions caused by that site.

Another factor is that, unlike the grid energy used to power the facility, the generator at the site is visible, along with its tank of fuel, and creates noise and emissions that the public can trace where seen, at the site.

Diesels are also unpopular and highly visible.

Ten years ago in 2012, Microsoft

Peter Judge Global Editor

suffered a humiliating campaign against its Quincy data center, led by former Mayor, environmental campaigner Patty Martin. The campaign was taken up by the New York Times , in what was one of the strongest rebukes issued to data center development till that date.

The campaign started with emissions, and brought in a whole raft of complaints against the then little-known facilities. Some of the Times’ criticisms were offbeam, but that public pillorying appears to have stung Microsoft - its name will feature large in what follows.

No diesel at all?

What are the alternatives? For Microsoft, the first approach was to simply try to cut out the diesel. In what may have been a direct response to the Quincy campaign, Microsoft announced in 2012 that it would run some data centers with no diesel generators.

The company tried the idea in Boyton Virginia and Chicago, as well as Quincy. Microsoft’s David Gauthier, who was director of data center architecture and design management, said: “There are tens of megawatts in this data center and other data centers Microsoft runs. They do not have diesel generators behind them.”

At roughly the same time, in 2013, eBay set up a 6MW subsidiary data center in Salt Lake City using containers, and

Images by Microsoft / John Brecher

powered it with fuel cells as a primary source . The fuel cells ran from biogas, and the electricity grid took the place of the backup generators.

At the time, eBay’s data center architect, Dean Nelson said the change was worth it because it allowed for other changes, such as eliminating the hardware cost of the generators: “You can’t justify fuel cells without redesigning the core of the site,” he told this writer in 2013.

During that decade, there was optimism that multiple cloud data centers could use software-level redundancy to provide reliability. Loads would run in multiple sites, so a power outage in one place would not stop the software running.

In 2017, Andy Lawrence of Uptime Institute predicted that "over the next decade, for a sizeable but as yet undetermined number of operators, resiliency and redundancy at the individual data center level will, in whole or part, be complemented and/or replaced by resiliency at the IT level. This is not necessarily a resiliency tactic or strategy, but an inevitable move, resulting from the fact that applications themselves are becoming more distributed."

In five years we’ve not seen operators doing away with their UPSs, however.

eBay is understood to have switched to more conventional UPS approaches after Nelson left in 2016. Two years later, Microsoft also backtracked on its early enthusiasm for facilities with no generators. Effectively admitting failure, it issued a single request for 72 diesel generators at Quincy.

The idea won’t die But in 2021, the concept re-emerged. After its 2020 commitment to be carbon negative by 2030 , Microsoft needed a roadmap to eliminate its diesel consumption, along with other sources of emissions.

“Finding solutions to reduce reliance on traditional diesel fuels represents a substantial contribution to the technology pathways necessary for deep decarbonization,” the company said as it revisited the idea of changing its UPS approach.

This time round, it’s attacking the problem with a range of alternatives to petroleum diesel fuel.

“As a company, we're exploring all of the possible options,” says Microsoft

senior program manager Casey Miller.

“In the medium and long term, there’s different competing technologies - we're looking at technologies that may have an even lower carbon footprint.”

Let’s take a look at what those are.

Bio-methane microgrid

In San Jose, Microsoft is using renewable natural gas instead of diesel to power its backup. Enchanted Rock is installing a 100MW microgrid at an Azure data center being built on a site Microsoft bought in 201, with 224 natural gas generators, each with a 0.45 MW capacity

The generators are fed by a natural gas supply. Enchanted Rock will source net-zero biogas captured from rotting food waste and agricultural operations, and inject it into the pipes upstream of the facility to offset any fossil gas burnt by Microsoft in the event of an outage.

As well as reducing greenhouse gases, the project helps the facility meet California’s tough clean air rules. Californian Democrat Senator Ben Hueso said: “I am hopeful that one day all data centers will replace their backup power systems with carbon-negative, clean renewable natural gas. Today’s announcement will set a precedent I believe all of California’s roughly 240 data centers should follow.”

Fuel cells and hydrogen

Fuel cells are a strong contender to replace diesel generators in the back up of data centers. They can be fed by various gases, including methane from natural gas or biogas, or hydrogen, which can be produced without emissions.

The fuel cell effectively ”burns” the gas fuel, combining it chemically with oxygen. If methane is used, then carbon dioxide is released into the atmosphere, which counts as green if the methane comes from a recent biological source.

If hydrogen is used, the only byproduct is water.

Microsoft’s efforts with fuel cells go back to 2013, when the company tested solid oxide fuel cells (SOFCs). These run at high efficiency, and can deliver a high load continuously - but as they don’t respond well to variable loads, they turned out not to be the best solution for backup systems which have to start quickly.

By 2018, the company switched to proton exchange membrane (PEM) fuel cells which can start up quickly as

required by backup applications.

In 2020, Microsoft demonstrated a 250kW PEM system which provided continuous power for 48 hours to a bank of 10 racks of servers at a Microsoft energy lab in Seattle.

For two years, led by principal engineer Mark Monroe, the company ramped that up with the goal of creating a complete fuel cell system that could be plug-compatible with a diesel generator.

In 2022, it showed a 3MW power generation system powered by hydrogen, built by its partner Plug Power, and operating at Plug’s Latham, New York headquarters.

Plug created some larger (125kW) fuel cells, which enabled the project to fit into a unit the size of two 40ft shipping container, but - as a prototype - it is a lot more expensive than ordering a diesel generator. Still, the system passed the acceptance tests for a data center backup system, powering a simulated data center load (a set of heaters or “load banks”) during a full break in the grid power.

Microsoft’s director of data center research Sean James described this as “a moon landing moment. We have a generator that produces no emissions. It’s mind-blowing.”

The next step is to productize the prototype. Plug plans to make a smaller, commercial-grade version. Microsoft will test this at a real data center - possibly in an urban area where diesel generators are forbidden.

This test will be as much about the practicality of storing and using hydrogen, which is as big an issue as the fuel cells themselves.

Hydrogen fuel cells are only as green as the fuel they consume, and hydrogen can be a variety of colors.

“Green” hydrogen is produced by electrolyzing water using renewable electricity. That’s not widely available yet, so Microsoft is feeding its prototype with hydrogen produced as a byproduct in the industrial production of chlorine and sodium hydroxide.

Microsoft describes this as "blue" hydrogen - hydrogen which produces some emissions, but can be cleaned up.

So Microsoft wants to bring into being a hydrogen supply chain, and fuel cell backup units it can buy in quantity.

“We’ve got a commitment to be completely diesel-free, and that supply chain has got to be robust,” says James.

“We’ve got to talk about scale across the entire hydrogen industry.”

The vegetarian option

A more basic and simple way to reduce diesel emissions, is to replace the fuel, not the generators. Diesels can burn vegetable oils, which release greenhouse gases that have only just been captured by plants, making them a "renewable" resource.

Petroleum by comparison, release greenhouse gases stored millions of years ago, which should left in the ground to avert the climate crisis.

First generation plant-based fuels didn’t do well, however. Biodiesel (EN14214) is produced from plants such as rapeseed grown for the purpose

“Biodiesels were just a nightmare,” comments Lex Coors, CTO at Interxion’s Digital Realty. “They were clogging up the systems, we need fuel polishers, and we needed, to replace the fuel every two or three years, regardless of the fact that we had all the polishers in place.”

Engine manufacturers hated biodiesels. “We even had to prohibit the usage of this kind of fuel with our engines, because it was really a disaster,” says Jacky Pluchon, vice president of EMEA power solutions at engine maker Kohler. “I mean, those engines were not built for that kind of fuel.”

By contrast, HVO (hydrotreated vegetable oil - EN15940) is a second generation fuel, produced from raw materials such as residues and waste.

Surprisingly, HVO scores a double whammy. It’s not just made from a waste product, making it part of a circular economy, it also can be used directly in existing diesel generators.

Most engine manufacturers are in the process of certifying their engines to burn HVO and say there’s no performance hit. In fact, performance improves.

“We’ve tested a lot of our generators, and we continue to do it, when they are sold to our customers, and the result is

really good,” says Pluchon. “In fact, this kind of synthetic fuel is even better for the engine, because it doesn't create any impurities inside the engine. Technically speaking, it's really a great solution.”

He adds: “It was a very good surprise to us when we tested those generators, because we've tested other types of biofuel in the past, and the performance wasn’t the same.”

It’s also more stable, and easier to store than petroleum diesel fuel, and can be mixed with regular fuel: “You can keep HVO fuel five or six years,” says Pluchon. “That’s not the case with regular diesel. You need to have a fuel polishing system inside your tanks, or you need to change, change it on a regular basis.”

Data center operators like Coors’ Digital Realty want to use HVO100 (100 percent HVO) as a sustainable fuel: “In an emergency, if there is no HVO100 available, you can mix it to any percentage with normal diesel fuel.”

Coors backs up Pluchon’s performance claims: “The capacities are the same, we have no loss of capacity. The only disadvantage is the cost.”

HVO costs around double the price of conventional diesel, but Coors shrugs that off: “The time that we will need to earn the diesel is maybe an hour a year. That will not hurt our opex at all.”

Large batteries

Another option is to restructure the whole UPS system, with much larger batteries,

which can support the data center long term.

There’s an obvious drawback to this approach, as batteries only store energy, they don’t convert it from a portable form like diesel generators do.

If an outage goes on for longer than a few hours, as can happen with incidents like floods and wildfires, then data centers can potentially carry on for long periods on no grid power, if they can replenish their tanks of diesel.

You can only replace a generator with a large battery if you are pretty sure no outage will last longer than the lifetime of the energy stored in that battery.

Despite this, Google is going ahead with replacing some diesel capacity with batteries , at its St. Ghislain, Belgium data center.

The search giant is replacing a diesel generator with a battery that can deliver 2.75MW of power, and hold 5.5MWh of energy. The positive benefit is that this battery can be connected bidirectionally to the grid, and support the grid when necessary.

The installation uses Gridstack batteries from Fluence, and is being connected up by Centrica, using Centrica's FlexPond software to store and discharge energy, so it can support the data center during power cuts and blackouts, and also provide up to half of the stored energy to the grid on a short term basis, to improve grid stability.

The batteries use LFP (lithium iron

Images by Microsoft / John Brecher

Microsoft fuel cell demo Christian Belady, Sean James, Microsoft Advanced Energy

phosphate) batteries made by Chinese battery maker Contemporary Amperex Technology Co. This has a lower power density than more usual Li-ion chemistries, but has a better duty cycle and safety record, while using less toxic raw materials.

The installation only backs up a small proportion of the power needs at a single one of Google’s many facilities, but could be an important proof of concept.

Meanwhile, Microsoft is using Saft batteries in some UPS, for a test being carried out with a partner which is evidently desperate for environmental credibility - the oil company Total. Needless to say, the emissions saved in this test will be utterly meaningless compared with those produced by Total as a whole.

Less testing

It’s worth mentioning there’s another way to reduce emissions from diesels, and that is to run them less.

Considering that data centers mostly switch their generators on only during tests, it makes a big difference if they are tested less.

Kohler is moving towards diesels that can be trusted without fully firing them up in tests, and it’s doing it because of pressure from the data center industry.

“It will require a different maintenance from the way it is done traditionally, but it's how we can improve - and I think everybody will follow that direction,” says

Pluchon. “Data centers was the industry pushing for that. It didn't come from any other industry at this point.

Coors likes the prospect: “Kohler is one of the first companies who wholeheartedly supported my suggestion to go to zero run hours on diesel. We now see other companies also starting to change, because that's the only way forward. If you don't need a diesel, don't start it. Just start it when you need it.”

Positive spin-off

All these options are more expensive than diesel generators, or less available. In the case of hydrogen, they need a whole new production and distribution infrastructure.

But advocates argue that the data sector has high margins, and can afford to use more expensive options. In fact, it has a responsibility to do so, because it will stimulate a growing market for them.

Speaking of HVO, Miller says: “A potential influence we can have, as an industry, is to push legislation in that direction and bring more attention to adoption. Having an actual renewable diesel standard would go a big way. That's probably step one, making sure that there's a target that you have to hit with whatever you're designing your generators too.”

“We create the demand, and that will be followed by larger industries,” says Coors.

Of course, that raises a question. If demand ramps up, is there really enough

fatty waste feedstock to support an expanding industry based on HVO?

“The source of the waste can be a question,” admits Pluchon. “There is still research done as to whether there are sources such as algae in the future. For now though, we produce a lot of waste on this planet, unfortunately. With the kind of resources there are, you will have a lot of it for the coming years.”

Even the long-term bets, like hydrogen, could pay off as well.

If data centers stimulate a market for distributing hydrogen, then a subsidiary market has to evolve for very practical reasons that Monroe pointed out in a DCD podcast. The small size of hydrogen molecules means there’s inevitable gradual discharge, so it can’t be stored for ever.

But that could be a positive spin-off. Hydrogen fuel cells would be a good power source for automotives, but there’s never been an available distributed source of green hydrogen.

The hydrogen slowly leaking from data center energy storage plants could be offered as fuel for cars powered by hydrogen fuel cells, and a full hydrogen economy could evolve.

"Once green hydrogen is available and economically viable, this type of stationary backup power could be implemented across industries, from data centers to commercial buildings and hospitals," says Monroe. 

In a somewhat confusing development, both Microsoft and AWS have in the past claimed to be able to do away with the UPS, using batteries installed in the racks.

In 2015, Microsoft announced it was installing in-rack batteries for individual power supplies in its data centers. Instead of a large room full of lead acid or lithium-ion batteries, have a series of smaller batteries supporting individual servers or racks, creating what it described as a “distributed UPS.”

AWS showed a similar approach in a keynote speech in 2020, replacing the UPS, in order to do away with the reliability issue of powering a data center from a single large battery room.

“Rather than using a big third-party UPS, we now use small battery packs and custom power supplies that we integrate into every rack. You can think about this as a micro-UPS, but it's far less complicated,” said AWS engineer and evangelist Peter DeSantis “Now the batteries can also be removed and replaced in seconds rather than hours. And you can do this without turning off the system. So this allows us to drastically reduce the risk of maintenance we need to do.”

It’s not clear how far these ideas have been implemented in production systems. Microsoft has said little since its 2015 announcement, though its data centers clearly still have large battery rooms, based on recent announcements, such as its plans to share battery energy with the grid in Ireland.

AWS claimed in its 2022 sustainability report that its data centers no long have a central UPS: “In AWS, the team spent considerable resource innovating on power efficiency, removing the central Uninterruptible Power Supply from our data center design, integrating power supplies into our racks.”

DCD has not been able to confirm with AWS whether this quote is in fact “aspirational” or if the company has actually stopped buying UPSs.

In any case, the energy stored in in-rack battery systems would only be enough to keep systems online for a very short time, so these systems would only replace the batteries and switchgear of the UPS.

For outages of more than a minute or two, a diesel or similar long term power supply would be needed.

the UPS but keep the diesel?

Fuel Cell Update: Moving Toward Commercialized Solutions

Data center operators are continuing to drive efficiency and utilization improvements to reduce both energy costs and greenhouse gas emissions. But some are also looking ahead to the next step on the journey to net zero operations and exploring opportunities to power their facilities with clean energy.

One of the most promising of those opportunities is fuel cells. Fuel cells allow operators to transform intermittent renewable energy into continuous data center power. The key to that transformation is hydrogen. Some fuel cells use hydrogen as their fuel source, generating electricity through a process whose only byproduct is water. And hydrogen can be produced from renewable energy and water, resulting in a clean fuel for a clean power generation process.

Like batteries, there are multiple types of fuel cells. Two have the characteristics required by data centers.

Arturo Di Filippi Smart Power Global Offering Manager, Vertiv

Applying Fuel Cells to the Data Center

Like batteries, there are multiple types of fuel cells, and two have been identified as having the performance characteristics required by data centers: proton-exchange membrane (PEM) and solid oxide fuel cells (SOFCs). While both are suitable for the data center, they have different operating characteristics that will influence how they are used.

PEM fuel cells use hydrogen as their fuel source and operate at relatively low temperatures that enable startup and shutdown times comparable to the diesel generators used for data center backup power. SOFCs typically use natural gas as a fuel source but are increasingly being adapted to run on hydrogen. They have higher electrical efficiency than PEM but are more expensive and operate at

much higher temperatures, requiring longer startup and shutdown times. The high temperatures can also reduce the number of on\off cycles in the life of a SOFC, making them better suited for continuous duty.

Based on these characteristics, PEM fuel cells are being piloted for use in data center backup power applications, and SOFCs are being piloted in primary power applications.

The Emergence of the Hydrogen Economy

The big question around the viability of using fuel cells to provide clean power to data centers focuses on how quickly hydrogen production and distribution can scale to meet the industry’s requirements. Significant private and public investments are being made today to add the necessary capacity and drive down costs. One of the pilot fuel cell projects Vertiv is participating in is supported by the European Union’s Horizon 2020 research and innovation programme, and the U.S. Department of Energy’s Earthshot initiative has set a goal of driving down the cost of clean hydrogen to $1 per kilogram by the end of this decade.

Managing Transients From Fuel Cells

One of the challenges presented by fuel cells is that they generate electrical transients in response to load changes. This is being addressed by configuring the fuel cells and the UPS system’s lithium-ion batteries in parallel. This allows the UPS battery system to absorb and store the transient energy that results from load changes.

When fuel cells are used for backup power, this configuration means the load never has to transfer to the fuel cells during an outage. Instead, the batteries are continuously charged by the fuel cells and support the load until stored hydrogen is depleted. Integrating the fuel cells with the UPS system will also enable the fuel cells to support peak shaving and other grid services.

Solid oxide fuel cells using natural gas as their fuel are being piloted for primary data center power, with the grid serving as the backup power source in conjunction with lithium-ion batteries. As with backup power applications, the UPS will eventually control the interface between the batteries and fuel cells, managing fuel cell set points and excess power from transients.

The lack of a hydrogen distribution network is less of an issue for backup than primary power applications. Enough hydrogen can be stored on site to provide 48 hours of backup power for even large data centers. Consequently, commercial data center backup power solutions using fuel cells should advance more quickly than primary power solutions. Based on the current state of development, it isn’t unreasonable to expect these solutions to reach the market in the next several years.

Using SOFCs that operate on natural gas enables operators to leverage the natural gas distribution network for primary power applications. This isn’t a zero-emissions solution, but it does offer emissions reductions in most cases, and some fuel cell manufacturers will likely provide ways to convert natural gas SOFCs to hydrogen in the future.

Vertiv is working closely with multiple fuel cell manufacturers to ensure fuel cells can meet data center requirements for reliability and performance. For a more in-depth review of the latest developments related to fuel cells in the data center, including an overview of a fuel cell solution, read the Vertiv white paper, Evaluating the Potential of Fuel Cells for Backup and Primary Data Center Power 

Making the most of your waste heat

After years of hot air, the digital boiler market is heating up

As data center operators strive to make their facilities more efficient, some are embracing the concept of reusing waste heat.

Greenhouses, fish farms, and apartments have benefited from the idea - with servers using the heat that they don't want to warm those that need it.

But it requires data centers to be near district heating systems or farms, and is viewed as a secondary priority after the main job of being a data center. What if we flip the idea on its head? What if we find the places that are already using electricity to create heat, and stick compute in the middle?

A few companies have tried to build digital boilers and heaters over the past decade, but the idea has struggled to find traction, with several startups going bust or pivoting out of the sector. Now, French company Qarnot thinks it has managed to solve the challenge.

"10 years ago, we were considered to be crazy," Paul Benoit, the company's CEO, told DCD.

"And what is very interesting is that for a year now, what we do is considered the way to go to deploy more IT infrastructure with less energy," he proclaimed.

It's a bold statement that may slightly oversell the company's success - many are still uncertain about the concept - but there are signs things are changing. In 2020, the company raised around $6.5 million, adding to the $2.5m it had received from Data4 Group a few years earlier.

Qarnot also has clients it can point to, including Société Générale, KTH Royal

Sebastian Moss Editor-in-Chief

Images by: Qarnot

Institute of Technology, and animation house Illumination (best known for the Minions franchise).

Before we unpick how it got to those deals, first let's understand how we got here. "In the beginning, we made a space heater for houses," Benoit explained.

The concept was that, instead of using your electricity to produce heat through a traditional radiator, use the same amount of electricity on a Qarnot box that looks like a radiator. At the same time as producing heat, a side product would be compute, which could be sold to other companies as distributed computing.

But the economics were tricky as the equipment cost significantly more than a comparable heater. There was also a fundamental flaw: Data centers require 24x7 compute, while people don't require 24x7 heating. There were also security concerns about putting servers in people’s homes, and limitations to residential fiber connectivity.

In the Netherlands, these issues led to rival e-radiator business Nerdalize declaring bankruptcy in 2019.

"So now we go to social housing [and large apartment blocks], where we can run 24x7 for hot showers and stuff," Benoit explained. "We're stopping doing heaters and going more for boilers,” he said

One exception, a much-publicized

crypto-heater launched in 2018, was mostly a marketing stunt, Benoit admits.

“It was B2C, which we’re not doing anymore, and it was quite expensive,” he said, but added that those that bought it would have nearly doubled their money on Ethereum mining at the currency’s peak. Benoit would not disclose how many units were sold, but said it was small. “We’re focused on B2B now.”

There's still a limit to housing blocks, though: "The scaling is not great, because you cannot put 500kW of hot water 24x7 in that housing [complex], even if it's a large one," he said. But Europe is building more local district heating, covering multiple such properties from one location - "this is our strategic focus."

The hope is to convince such sites to use their energy twice, both for heating and computing, while paying only for the heating. The digital boilers themselves are more expensive than conventional ones, because you're replacing simple heating equipment with semiconductors and IT infrastructure. The IT infrastructure also becomes obsolete much faster than normal boilers do.

"We provide the chassis, and replace the servers every five years," Benoit said. The company works with circular economy firm IT Renew to use recertified and new Open Compute Project servers.

To offset the higher costs, Qarnot partners with a heating company "and then we sell the heat we produce at a much lower price to them than they sell the heat themselves."

Traditional air-cooled data centers produce waste heat at around 30°C (86°F); Qarnot’s boilers use direct water cooling to deliver water at at 65°C (149°F) through 2cm copper pipes.

"Megawatts of heat at 30 degrees? It's crap, you cannot do anything with it," Benoit said.

At the same time as it sells the heat, Qarnot then sells its compute to customers, at a low price because the electricity costs have been offset.

That compute infrastructure, which is either paid for by the customer upfront or on demand, is accessed through interconnection points at investor Data4's data centers, and managed through a custom software stack.

While the deployments are close to potential end users, it’s important to note that it is currently not an ‘Edge’ system focused on low latency. Instead, Qarnot is targeting

batch-processing high-performance computing workloads.

“Most of our clients are not very sensitive to latency, because they are doing simulations, and workloads like that,” Benoit said.There is still an opportunity for placing the boilers in the same building as clients, or at least nearby, but Benoit does not see it as a priority. “It would be great,” he said. “But the problem is more on the business side, because if you have to discuss with the heat guy of the building and the IT guy of the company, it may take 10 years to make the deal. We don't do this, but it can be a way to go [in the future].”

French bank Société Générale is trialing

using the system for risk computation. "It's a first proof of concept, but the project [is expected to grow to] 500kW-1MW sites, the sweet spot for district heating that is 24x7," he said.

As Qarnot deployments grow larger, they bump into a different challenge: Connectivity. Nobody wants their residential block's Internet being sapped by their boiler, and nobody wants to pay for a data center that is limited to residential fiber rates.

"Historically when we deployed heaters in the building, we would deploy 100-300 of them and use fiber to the building," Benoit said. "Now we're building larger sites, so we deploy dark fiber because the investment is reasonable compared to the infrastructure and it can be financed by the end user."

The company currently rolls out dark fiber for 500kW and above projects, but plans to do it for sites in the lower hundreds moving forwards.

It may also need to build in more redundancy as its sites grow bigger. "It's totally possible if clients want it, and some do, but most do not ask for it," Benoit said. "We have to be able to understand the needs of the client and then discuss whether they want to pay for it.”

The next difficulty is finding those sites, with district heating systems only available in some countries. “In France, district heating penetration is very low,” Benoit admitted. “We work in Finland and want to work there more, as well as in all the Nordic countries, where we have partners. We also have discussions in Canada, and in Japan - but Japan is special in terms of infrastructure.”

His hope is that, as countries invest in net zero infrastructure, more district heating systems will begin to be rolled out across the world - at least in the colder regions.

“The opportunity is gigantic,” he said. 

Images by: Qarnot

Backup comes to the fore

Peter Judge Global Editor

With electric power grids stressed, and long term energy supplies in question, data centers may find themselves making more use of diesel backup power in the coming months

On September 2, 2022, an electrical substation in Flevoland, the Netherlands, caught fire. A second substation swiftly ignited, and the local grid shut down.

Local data centers switched to their backup generators, as one would expect. But there were problems with the generators at A2B Internet. A simultaneous outage in the mobile network impacted their controls.

The issue brought the reliability of backup power to the fore. And this was not by any means an isolated incident.

There are multiple factors that mean data centers round the world could be switching on their diesel generators more often in the near future. And that might be a problem in itself.

Power grids under stress

In January, a power line failed in Kazakhstan disconnecting the whole country, as well as much of neighboring former Soviet states Uzbekistan and Kyrgyzstan.

In March, a malfunction at the Hsinta Power Plant in Taiwan caused blackouts in the south of the country, hitting Internet connectivity to the whole nation.

These outages may be coming our way more frequently, for a variety of reasons. European nations will very likely be experiencing a shortage of natural gas this winter due to the war in Ukraine, which could lead to electricity blackouts.

And even before the war provoked an energy crisis, multiple nations and metro areas have been predicting a difficulty in serving demand for electricity. Ireland’s EirGrid, for instance, warned of “rolling blackouts” in 2021, due to increasing demand for electricity.

In London and Northern Virginia, this is because data centers have been approved faster than new electricity capacity can be commissioned, along with the infrastructure to distribute that energy.

In other cases, planners are warning that attempts to reach net-zero targets will create extra strain on the grid, because of a double effect - increasing demands on the grid, while destabilizing supplies.

New demands will be placed on the network as countries attempt to shift from petrol-driven vehicles to electric ones, and switch home and office heating from gas to electricity.

And at the same time, nations are trying to shift the power supply to their grid, over to renewable sources such as wind and solar, The proboem is that these sources are variable and can’t be managed as easily as a baseload source such as coal or gas, which can provide power on demand.

As the effects of global warming begin to be felt, it’s likely that natural disasters such as floods and wildfires will be more frequent. In August 2022, huge floods in Pakistan took down the Internet there for an extended period, and plenty of earlier disasters meant data centers were taken off the grid.

In March this year, Australia’s New South Wales experienced outages due to flooding, while at the same time, wildfires in Tennessee cut off many residents and businesses.

In February, a giant storm in Texas created outages that were exacerbated by that State’s poorly-maintained electrical grid, leaving a lot of data centers struggling.

To add one more factor to the cocktail of risks facing the grid, it has long been expected that malicious attacks could bring down sections of the electricity network. Critical equipment is often provided with interfaces for remote management across the Internet - and these interfaces can be vulnerable.

In 2016, years before the current war, it’s believed that hackers brought down a section of Ukraine’s power grid. On 23 December, half the residents of Ivano-Frankivsk, were

In the US and Europe, data center operators are realizing that diesel generators may be called upon more often, as they have been in developing nations

left without electricity for several hours.

Since the current hostilities began, Russian forces have hit power generators, and effectively treated the giant Zaporizhzhia nuclear plant as a hostage.

New role for backup

Against this background, there are signs that data center operators are treating their backup systems differently - at least in the developed world, where grid reliability has been a given till now.

In the US and Europe, diesel generators and battery rooms have always been there to cater for very rare emergencies when the electrical power fails or falters. They’re mostly seen as an insurance policy.

Now, data center operators are realizing that those backup systems may be called upon more often, perhaps approaching the experience of pioneering facilities in developing nations.

In Africa, it’s long been a fact of life that the grid is less reliable, and diesel generators are frequently fired up for backup. But that is changing, according to Funke Opeke, CEO of MainOne, in Nigeria.

“We build dedicated lines to take power from a substation, and that results in 96 percent or higher power availability from the grid,” Opeke told DCD this year, adding that the reliability of grids in other African countries may be less. “We get 95 percent in Ghana, and in Cote d’Ivoire we do better than that.”

While Africa may be emerging from an era of unreliable grids, Europe may be entering one. As winter blackouts approach, data center operators are buying up extra supplies of diesel fuel, in case they have to shift to their own generators for extended periods.

“We’ve been doing contingency planning since the war in Ukraine broke out,” Gary Aitkenhead, Equinix’s SVP of Europe, Middle East, and Africa operation told the Financial Times earlier this year.

“We don’t ever expect to have to run for more than a few hours, or at worst case a day, on diesel - but we’re prepared to run for up to a week.”

Digital Realty is also storing more diesel at its sites across Europe.

Just as nations are filling their gas reserves, data centers are pumping more diesel into tanks that have previously been kept at 60 percent capacity, filling them up to more than 90 percent.

Is it a good approach?

There are problems with this. More diesel burnt at data centers will mean more emissions, putting the operators’ own netzero ambitions out of reach.

It’s also inherently more unreliable. Once the diesel is running, and the power grid is down, then the data center has one less source of power available, and the diesel may be a single point of failure.

“Burning that much diesel isn't good for the environment or the stability of the data center,” says Opeke.

Other African providers are less optimistic about the grid’s stability, and have ambitions to bypass it.

“We don't believe that the grid can support the data centers of today, so we have to do something about it,” says Wale Ajisebutu of another Nigerian provider, 21st Century. “So our philosophy is microgrids, and captive power.”

In the long term, operators are aiming to shift to alternative sources which can keep the data center operating without creating emissions (see p4). If the backup generators are being used more regularly, this will increase the need to make that shift.

And in the short term, they will be doing all they can to secure their diesel supplies. If only because they remember how things went in New York in 2012.

When Hurricane Sandy hit in October of that year, Manhattan’s data centers discovered how crucial, and physically demanding, that supply effort could be.

When the storm hit, Internet provider Peer 1 had a problem. City regulations meant its generators were on the roof, with fuel pumped up 18 stories from main tanks in the basement.

The storm flooded Peer 1’s basement, and the pumps shorted. Data center manager Michael Mazzei had six hours’ fuel in a tank on the roof - and decided the only way to get more fuel reserves up there was by hand, in buckets.

Mazzei, his staff, and colleagues from other operators, all pitched in and Peer 1 remained online with a bucket chain providing its power for two-and-a-half days, till the pumps could be reinstated.

Ten years later, DCD caught up with Mazzei, and he was proud of the legendary achievement: given a similar situation, "I would have done it again," he told us.

But today’s Internet is more complicated than that, with multiple services connected across diverse facilities. Mazzei wouldn’t do it again.

At a very basic level, Equinix and Digital Realty may be glad they secured their diesel supplies.

"We’ve been doing contingency planning since the war in Ukraine broke out. We don’t ever expect to have to run for more than a few hours, or at worst case a day, on diesel - but we’re prepared to run for up to a week."