Connecting the power train to the value chain from grid to chip
Neil Potter Business Development Segment Team Lead, Data Centres - EMEA Eaton
Executive summary
A data centre today looks very different to one just five years ago. And it’s all thanks to the effects of increasing demand on their evolving and expanding capacity, alongside regulatory drivers designed to deliver greater sustainability and efficiency. Accelerating globalisation and the presence of hyperscalers and large colos beyond the traditional hubs are now asking questions of us all. How can we evolve with data centres and support their growth? How can we alleviate their challenges and identify new ones now and into the future? And, critically, how can we do better than before? With this as the driving force for improvements, we must ask if the current way we design, implement and operate our data centres can meet these needs. Alternatively, we need to challenge ourselves to think in a transformational and evolutionary way. This paper explores turning traditional thinking on its head by taking a systems approach to data centres to support the demands of today and tomorrow.
Introduction
It cannot be underestimated how rapid growth within data centre markets has been over recent years. This has been fuelled by the ever-increasing demands we place on the digital world and the need for business-critical functions to be resilient and reliable.
While the focus is still on the main ‘FLAPD’ countries, we’re seeing many operators, including hyperscalers, colos and even enterprises, spreading their wings to emerging markets such as Turkey, Italy, Spain, Poland, Greece, Middle East and Africa.
Emerging/current market analysis - $ value and CAGR
What are the challenges to be solved?
Although such rapid growth is a pleasing trend in uncertain times, data centre operators are facing continued challenges. These include:
• Data centre build out and complexity: site acquisition, data centre design and construction, skill sets, time, competitiveness etc
• Data centre design complexity: construction, heating, power, cooling, IT (critical systems and subsystems)
• Energy strategy: beyond buildout – around new and existing data centre operational costs, ESG goals and regulation
But with challenges come opportunities:
• Availability of specialised skills and critical services: skillsets, people, high mission-critical standards 24/7/365
• Maintaining data centre performance: resilience + redundancy + liability + efficiency + security + dynamic growth
• Differentiated value: sustainability, costs, pedigree/reputation, processes/procedures, data centre performance, client acquisition
These challenges, changes and demands are at the forefront of most operators’ and designers’ minds when focusing on their core needs for reliable, efficient, resilient and more sustainable data centres – while continuing to drive out risk. To support this, vendors are in a prime position to help meet these requirements and be part of the solution by using a ‘systems engineering’ approach.
What do we mean by systems engineering and the system level performance approach it enables?
When we refer to systems engineering we refer to the 3S’s of systems - These being:
• Systems engineering - the approach taken
• Engineered system – the combination of the systems that are to be system engineered
• System – the sub systems that make up an engineered system
To put this into practice, a systems engineering approach applies a set of 9 principles to the engineered system and associated sub-system to provide system level performance. These 9 principles being:
• Integrated stakeholder goals
• Modelling of lifecycle
• Considered levels of complexity and change
• Generating alternative solutions and concepts
• Modelled requirements for each phase
• System verification and validation
• Considers both problems and solutions
• Identified relationships between parts
• Balancing all parts for desired outcome
The goal is by applying these principles to a complex task and project enables a more simplified and effective outcome.
Systems engineering involves more than simply working with a vendor who can supply more than one product – a commonly held view. As defined by NASA, it’s a ‘methodical, multi-disciplinary approach for the design, realization, technical management, operations, and retirement of a system’. An engineered system is the ‘combination of elements that function together to produce the capability required to meet a need. The elements include all hardware, software, equipment, facilities, personnel, processes, and procedures needed for this purpose; that is, all things required to produce system-level results.’ A system is a set of sub-systems that make up the engineered system that in turn needs to be system engineered and have the 9 principles applied.
Critically, a holistic, systems engineering approach adds value beyond individual product capability, taking into account ‘big picture’ interconnectivity and the relationship of the system’s components to each other. In the context of electrical infrastructure, it extends from grid to chip at the design phase –rather than taking a ‘mix and match’ product route. It considers how design affects system implementation, how it will perform when up and running, and how to plan in for retirement when needed – all held together through the power of software and digitalisation. Systems engineering is a logical way of thinking. Not only is it about getting the design right but also getting the right design to enable fulfilment of key business goals and expectations.
Conversely, choosing not to take a systems engineering approach raises significant challenges including:
• The need for a more complex design process to coordinate mix and match products
• Additional firmware and EMS/BMS customisation
• Failure to identify and address all sources of load loss
• Inability to closely coordinate protective devices leading to component oversizing
• No defined quality metric across different manufacturers’ components
• The need for onsite alteration of equipment
• Additional plant space requirements
• Incompatibility of component properties
• Additional testing and commissioning
• Multiple maintenance contracts
• Complex end of life (EOL) disposal programmes
What’s the risk picture in mission critical infrastructure such as data centres?
There are several recurring threats or themes when it comes to failure in mission critical infrastructure including humaninfluenced outages, inadequate resilience or redundancy, negative component independencies and finally component incompatibilities.
The Uptime Institute notes that 80% of operators surveyed reported outages in the last three years. 66% of these were negligible to minimal with 30% ranking as significant to severe. Human factors tend to feature high on the cause of such outages and post-event analysis points to the majority as being avoidable, either through practice or design.
The data centre’s design may not incorporate the correct level of resilience or redundancy in some or part of the systems deployed, whether as a result of the selection of protective devices and back feed protection or the risk and danger of arc flash protection. The next thing to consider is also design-related but specifically associated with the inter-dependencies between systems covering the power train, IT system and load, cooling, operational controls and management.
Individual component and product compatibility must also be factored in. While such impact is generally mitigated for through redundancy and the designed electrical topologies, whether this is N+1, N2+1, or the tiering of the data centre design, it is still a factor in outages. Intrinsic product quality and an understanding of potential failure mechanisms at component and product level should be part of the data centre designers’ and engineers’ tool kit. Working with a vendor at the early stage of design, who can consider a fully connected electrical domain and integrated power train as part of a systems approach, can mitigate system failure.
What have previous data centre risk-control initiatives looked like?
Historically, the method of data centre electrical design has been dictated by the relationship between consultant and client. Each of these invariably has the same end goals but seldom manages to align priorities during the procurement process.
The consultant’s mix and match approach is generally dictated by historical knowledge of products and an obligation to use generic (non-product specific) tendering. While the client’s approach seldom revolves around purely technical issues, it will more frequently lean heavily on financial concerns around price, warranty and replacement strategies.
Operations teams are generally driven by maintenance issues and the need for upskilling on new products. These approaches take little account of the high complexity of data centre infrastructure and its need to be finely tuned and interconnected in order to achieve the client’s desired reliability targets. When this is coupled with the need to operate ever tighter construction programmes while costs remain within budget, it becomes obvious that a new approach is needed.
Central to the disfunction of most designers’ mix and match approach is its inability to offer closely matched component selection. A lack of alignment of breaker properties creates the need for a wider coordination gap between upstream components. This has a ripple effect not just on the breakers but also the cables, busbars and ATSs – all the way to the utility grid. This additional sizing is not in place to accommodate increased capacity but is instead just there to overcome the incompatibility of different manufacturers’ components.
Challenges such as these which arise when adopting a traditional approach have negative cost impacts throughout the whole lifecycle of the data centre, from the initial design phase through operations to end of life (EOL) disposal and replacement phases.
What can a systems approach bring to the data centre environment?
When we consider the digital transformation we’re currently witnessing, the data centre environment is right at its heart, making it all happen.
A data centre has two principal domains, data and electrical, and while they each serve different functions, how they interoperate is key for overall functionality. Although there are commonalities between them in that they both are networks, they differ in how they transmit, receive, process, switch and store data and energy.
Performance is optimised when each is designed to operate within a system. If these two domains are intelligently connected, then events in one can trigger appropriate action in the other. Coupled with the evolution of AI, connected data and electrical domains move us forward to the self-optimised, self-aware intelligent data centre of the future – and this can only be achieved when designed and connected as an engineered system.
To further highlight this point, if we consider the electrical infrastructure, the data centre is the perfect example of where this must be designed as a network and function as a system. A data centre is not a collection of functional blocks simply bolted together. Instead, all elements must be designed to function as an end-to-end system. This is about more than interoperability and sizing, it means ensuring that specific system parameters of input/output, management and operation are conceived in advance and ‘baked in’.
While risk can come down to the smallest single element in a network, system performance is defined by how all the elements not only function individually but do so together. It’s very much the case that the whole is greater than the sum of its parts.
We’ll later further explore the importance of the design and electrical infrastructure through the ‘power train’ – how a systems approach is more than just design optimisation.
At Eaton, when thinking of the DIOR data centre stages (Design, Implement, Operate and Retire), we consider these as the value chain stages – each one fundamental for how the next performs. A true systems engineering approach influences all stages of the end-to-end value chain, each one considering and conditioning the next. They connect with each other to create a self-aware, self-optimised, intelligent infrastructure that facilitates a connected electrical and data domain allowing the data centres of the future to meet today’s demands and challenges.
To think of these stages as a system, we must look beyond the traditional ‘building blocks’ of design where individual components and equipment are mix and matched to provide a functional infrastructure.
Systems engineering and the application of its principles can optimise the data centre’s data and electrical networks through a fully connected system that can contribute to overall business goals including:
• The ability to maximise output with minimum input through data insights and intelligent design
• Support for hyper efficiency by considering wider activities across the data centre’s DIOR stages to enable reduced complexity, time and cost
• Keeping digitalisation at its very core to driving an effective, efficient ‘power train’
• Delivering a more sustainable and commercially successful data centre – one focused on TCO
The design stage of a systems approach
By adopting a systems engineering approach to the design of the electrical system, many of the problems faced by mix and match design can be easily and quickly addressed. Such an approach involves considering the design not just as several functional blocks but as a group of intelligent components coordinated with each other and tuned to act as part of a complete system. Systems-level designs benefit significantly when they originate from a single manufacturer allowing for better alignment of component properties.
Key benefits include:
• In-depth design studies by the manufacture (see more on this below)
• Physical compatibility between all components and panel frames
• Modular compatibility between PDU, UPS and transformer (size compatibility)
• All components designed with central monitoring and control in mind
• Tighter coordination between protective components including
• Seamless communication between components
When a single manufacturer is employed to devise an electrical design, particular attention can be focused on those areas of critical importance which are commonly overlooked. These include:
• Arc flash design and protection
• Impact of UPS in a power system
• UPS backfeed protection
• Bypass fuse sizing
• Earth fault current detection in TT systems (3 wire installation and floating grids)
Design studies with greater depth
When using mix and match system design, electrical engineers rely on generic power analysis software which depends on up-todate manufacturers’ data for accuracy. Correspondingly, the engineer must also update the software regularly. At best this leaves a significant time delay between new product launch and its availability for analysis using generic engineering software.
Direct engagement with the manufacturer, on the other hand, guarantees that up-to-date component data is used consistently, resulting in accurate product selection. Detailed design studies using the manufacturer’s software eliminate the risk of component oversizing and inform better product selection. They can cover:
• Selective co-ordination
• Overload protection
• Short circuit capabilities
• Power load analysis
Let’s focus on some of the key design priorities and stages when taking a systems engineering approach:
The power train
A key element of a systems engineering approach is the ‘power train’, principally made up of components that make up the system itself. In its simplest form it takes and manages the power from the grid connection all the way through to the chip.
But then we need to look at the elements that are needed along the power train to deliver the grid to the chip and the complexities this brings:
While it may seem simple to describe each component as a functional block, something we must consider is the number of components and the effect – including inter-dependencies – that each has on each other.
It cannot be underestimated how software and networks are critical for the power train – they’re the glue that connects the system together. Software affects how we control, monitor and predict, allowing optimisation and utilisation to be designed in through the use of modelling and digital twinning. This simulates the optimised state, not only at the design stages but just as importantly in the operation of the data centre, also enabling integration to further equipment that branches or feeds off the power train. Considering the right type of software and its interfaces is therefore key.
To maximise systems engineering, designs should ideally be based on products and components that are fully compatible with each other, with integration, interfacing and interoperability the driving factors of choice and their interdependencies fully understood. This ensures that the most accurate real-time data and digital insights are utilised along with guaranteed seamless integration between the value chain stages. Working with a vendor like Eaton, whose hardware, software and services feature inbuilt DNA, enables such optimisation.
It’s also critical to consider the external challenges we see today such as pressures on overall costs, energy management and consumption, risk management, regulatory directives around sustainability and carbon reduction.
Here are two key examples:
• France has legislated a reduction in data centre power consumption of 40% by 2030, 50% by 2040 and 60% by 2050
• The EU’s 2030 climate & energy framework sets key targets including a 40% cut in CO2 emissions, a 32% share of the energy market for renewables and a 32.5 % improvement in energy efficiency – all by 2030.
As these influences are likely to impact the entire data centre market eventually, we must plan and introduce measures now. This is particularly the case as hyperscalers and colos broaden their reach beyond traditional locations.
A connected, integrated, interoperable and effective power train that’s finely tuned at the design stage – and fully implemented – enables self-aware, self-optimised, intelligent infrastructure to become a reality and help meet these challenges head on.
Applying a systems engineering approach to power distribution
A systems engineering approach to electrical distribution design must take the system’s three constituent parts into account. These include its:
• Individual electrical components (breakers, UPS, generator, PDU, transformer)
• Power sources (mains, emergency, grid, generator, batteries)
• Modes of operation (UPS inverter, bypass and maintenance)
The components must be chosen and tuned to react correctly in response to whichever source of power is active and the chosen modes of system operation. The system designer must consider the latter when sizing and setting protective devices to ensure selective coordination is maintained throughout. As the source of power changes from grid to generator and then to battery, so too does the level of power that the system can draw during short circuit conditions.
A fault downstream of the UPS creates its own unique risk conditions. In battery mode, the UPS can provide only limited power to clear a load-side fault. In such an event, the UPS is programmed to automatically attempt to clear the fault by transferring firstly to the static bypass and then to the maintenance bypass – each, in turn, offering incrementally greater fault clearing capacity. This technique depends on the availability of the mains to deliver the additional power to clear the fault. If the mains is unavailable, then the importance of a system-wide approach becomes apparent.
If the downstream protective device is correctly chosen and tuned with the UPS inverter, then it will clear the fault before the inverter goes into overload. By doing so it will prevent the complete UPS load from being lost. This scenario is typical of the complexity faced by the design engineer and much attention must be given to the operation of the system to ensure that all load loss scenarios are modelled and incorporated into the protection scheme.
A single manufacturer delivers closer coordination and savings
Mixing and matching manufacturers’ components within a power chain can lead to component oversizing caused by an inability to closely coordinate component properties. In such cases, this can create a ripple effect throughout the system as corresponding breakers, cables and busbar ratings all increase in size and cost. By using a single manufacturer, component selection can be finely tuned and more closely matched. This can result in downsizing of electrical components.
With every drop in component size, significant cost savings can be made across the complete electrical system.
Software: the glue that holds a system together
When we refer to software as being the glue that holds an engineered system together, it really is. Without it, a system is just a set of functional blocks. While we may consider that procuring the same vendor’s products is a system that will deliver a lot of key benefits, it’s the software/DCIM that’s the core element that delivers full optimisation by:
• Enabling consistent management and visualisation
• Linking OT (operational technology) and IT (information technology)
• Providing seamless, unified grid-to-chip performance management
• Giving key insights to support your sustainable goals – such as real-time data around PUE, water usage, carbon usage, energy sources
Common sense dictates that the electrical equipment manufacturer is better positioned to design its own management software, thereby embedding the same DNA into both components and code.
This approach allows manufacturers to build their products pre-programmed for compatibility with their own brand DCIM –avoiding the need for re-coding and the inevitable system glitches this creates. While generic DCIM systems market themselves as manufacturer-agnostic, few live up to this claim without becoming code heavy and error prone.
When we take a systems engineering approach, particularly breaking it down across DIOR data centre stages, the DCIM is the tool that provides the connection between the value chain and the power train.
It’s also through digital twinning that we’re able to predict faults and maintenance, and optimise space and performance, through fully understanding system capabilities and ‘what if’ scenarios.
Implementation
Beyond the design phase, taking a systems engineering brings added value and benefit throughout the whole data centre lifecyle. Once we’ve designed the system itself, the next priority is how this design and infrastructure should be implemented. From sourcing of products and solutions through to the installation and commissioning process, using a traditional method brings complexities of its own including:
• Multiple points of contacts
• Increased cost of product as individually procured
• More co-ordination of vendors
• Higher levels of risk in compatibility and integration of products
However, taking a systems approach and working with a vendor such as Eaton, who looks beyond the design stage to offer support end-to-end, brings added value as follows:
• Single vendor responsibility
• Seamless integration and less coding of products
• Managed site responsibility
• Greater efficiency in manufacturing of products through single vendor
• Speed of deployment
• Scheduling and installation execution efficiencies
Simplified commissioning
Data centre commissioning starts at the design stage, as it’s from here that most influence can be brought to bear to ensure a robust and reliable electrical system. The manufacturer is ideally situated to steer the electrical system through the commissioning process on a more fault-free route by incorporating rigorous testing at every stage of design and prefabrication. With a single system manufacturer, the Factor Acceptance Test can evolve beyond simple functional testing of individual modules to become a more detailed, system-level test with all modules operating together. The entire power chain can be tested as a system on the factory floor – thereby reducing valuable commissioning time onsite.
Dedicated project management
The building block approach – sourcing multiple products from multiple vendors – brings complexities and challenges in terms of co-ordination and efficiency in the construction phase. By taking the systems engineering route, the addition of project management adds value at the implementation stage. Choosing a vendor that offers a finely tuned power train while also considering the interdependencies of connected solutions means project management that enables smooth implementation by:
• Planning: developing the project plan by outlining scope, milestones, goals, timeline, budget and resource allocation
• Scheduling: creating and maintaining a detailed project schedule
• Managing risk: identifying and resolving potential risks and issues that could impact the project’s success
• Managing resources: organising and ensure all resources required for the project, including personnel, equipment and materials
• Communicating: continually and formally to the wider stakeholders all aspects of project progress
• Delivering: ensuring the project stays on track for time, budget and quality
Operational stage
Once the data centre is fully comissioned and handed over to the data centre operator, the project is at the operational stage when the design adopted in the early stages can be seen in action. Should this have been based on the traditional mix and match approach, the data centre will be functional. However, had a system engineering approach been taken, considering how each value chain stage affects and influences the next, the most appropriate design will now be in operation that goes beyond basic functionality.
Key to this is the software system or DCIM selected earlier. Choosing a Visual Capacity Optimisation Manager (VCOM) that has been tried and tested with the same vendor at component level enables the most accurate and insightful data. This information can be used to ensure maximised capacity usage, identifying areas of inefficiency, optimised power, water and mechanical usage. Through digital twinning, the effects of decisions can be modelled before they are taken to ensure they represent the right route.
Image taken from Eaton’s Brightlayer VCOM software system:
At the same time, the ability to pro-actively predict when maintenance is needed or systems are close to failure delivers improved reliability and greater resilience. Having components at every level able to provide up-to-date data about their health and status – and also predict lifecycle – gives peace of mind. This is what a systems-engineered approach reliably and accurately enables when the vendor’s service components interface seamlessly as a result of pre-designed integration.
Retirement stage
Some might query how a systems-engineered approach impacts how you retire or repurpose a data centre. As previously highlighted, it shapes how each stage in turn affects and influences the next. The retirement stage is no different. The retirment stage should be seen not as end, but the beginning of the next phase, a new chapter. By simplifying the processess and utilising data insights, we can use the best pratice, learn from challenges presented and overcome.
A systems engineering approach allows us to model best practices to be applied in future projects, drives innovation for future developments in technologies.
Through the provision of accurate total cost of ownership data at earlier, along with digital insights and twinning provision, the system will perform more effectively for longer and be pro-active in its use.
To maximise the benefit from a sustainability perspective, the circular economy should be considered. Optimised design and usage, coupled with a modular approach, enables the power train to be either re-used or decommissioned at minimal cost and environmental impact.
Systems engineering and software tools can take all these considerations into account at an early stage to ensure a fully optimised data centre not only for today but into the future.
xIntegra: Eaton’s solution to systems engineering that delivers power by design
xIntegra is the result of Eaton’s commitment to and investment in systems engineering – a solutions package that focuses on providing real value throughout the data centre lifecycle. By focusing on early engagement and utilising software as the glue that holds a system engineering approach together, it looks beyond the traditional functional blocks of components to enable a broader perspective that applies systems engineering principles to the power train in all interdependent aspects of implementation, operation and retirement.
xIntegra drives system level performance along the power train by expanding the product portfolio normally associated with Eaton in the data centre space. This includes medium-voltage switchgear, medium-voltage transformers, low-voltage distribution, low-voltage busways, PDUs, rack PDUs, and data centre software all under the Brightlayer banner.
The xIntegra solution offers access to dedicated support teams at each stage, from solutions architects at the design phase and service engineers and project managers at implementation, through to software engineers at the operational stage and a specialist team for decommissioning and repurposing. Each resource adds value through a systems- engineered approach.
Eaton is developing new digital xIntegra tools that enable different ways of measuring data centre performance. These bring together all key metrics under one platform to generate an xIntegra data centre maturity score that combines sustainability metrics, including PUE and utilisation etc., to offer real insights into overall perfomance. Look out for this in the comimg months.
xIntegra continues to evolve and, true to Eaton’s philosphy of investing for the future, will bring even more new and exciting features.
Scalability through modularity: maximising the benefits of a system approach
In recent years we’ve seen the emergence and establishment of off-site, modular build data centres as a solution to meet the increased need for speed of deployment and scalability, particularly when there are construction constraints involved. Taking the modular approach makes for the easy development of the three data centre functional blocks, namely power infrastructure, cooling and IT space.
Modularity can also incorporate the benefits of the systems engineering by working with a single vendor to capture a system in a box. This can meet all technical and commercial needs while also providing these additional value benefits:
• Full design, construct, and commission service
• Deep dive design leading to better component selection
• More coordinated system operation in normal and fault conditions
• Factory floor prefabrication to ISO 9000 quality standards
• Full in-factory component to system-level testing
• Integration of EPMS and alerting
• Integration of bespoke modules and components
• Local regulation and compliance review
• Integrated electrical, mechanical distribution systems
• Speed of deployment
• Co-ordinated and controlled implementation environment
• Reduced Total Cost of Ownership
• Improved carbon footprint through:
• Managed centralised goods and services
• Reduced transportation needs to implement the installation
• Reduced manufacturing drops
• Reduced business/employee commuting
• Reduced waste disposal
• Contribution to scope 3 requirements
• Guarantee of operation on site – reduced fixes on site.
Introducing Eaton xModular Eaton’s xModular range is our solution for modular build requirements. This takes a systems approach covering all off-site fabrication needs resulting in a technically compelling solution that can be bespoke-designed or use reference designs for typical arrangements, as shown here:
Conclusion
As the evolution of data centres continues alongside the drive for hyper-efficiency, there are inherent challenges that need to be addressed. While it’s true that the traditional mix and match method has got us to a certain point – an approach some may argue will continue to provide a functional, performing data centre – in reality it cannot fully meet the expectations of today’s and tomorrow’s customers.
The demand for more sustainable, efficient and secure data centres, alongside accelerated expansion programs and leaner build costs, makes a strong case for looking at a holistic, end-toend, systems engineering approach. This should consider all data centre lifecycle stages while also incorporating a finely tuned, systems level power train that brings higher performance and costs savings, coupled with efficiencies and greener credentials from conceptual design through to retirement.
This is the approach that will define and shape data centres not only today but into the future.
Working with recognised, industry-leading modular builders across the EMEA region, we bring together their knowledge and experience of modular construction. Combining this with Eaton’s industry leading technologies and our systems approach, the digital platforms that wrap the modules together, and the input of our experienced service and maintenance engineers, enables us to deliver a true ‘system in a box’.
