AI Drives Thermal Power Density Data Centers: The Cooling Concepts Beyond Air Cooling

Traditional air cooling is increasingly reaching its limits with AI and HPC systems. Direct-to-chip, liquid, and two-phase cooling, as well as intelligent control methods, are set to dominate the next performance class.

The thermal architecture of data centers is changing. Not only do artificial intelligence applications generate increasingly higher heat loads, but communication is also becoming a thermal issue. At 112-Gbit/s PAM4, optical I/O modules typically consume 15 to 25 W according to interconnect specialist Molex, while modules for 224-Gbit/s PAM4 consume up to 40 W each. In a 32-port switch, optical modules alone generate a power dissipation of more than 1 kW.

Air Cooling Reaches Its Limits

Forced air cooling remains practical for many servers and especially for less densely packed areas. Molex considers a rack power of around 10 kW as a manageable scale for directed air cooling. Modern data centers are therefore increasingly combining air and liquid cooling.

As power density increases, so does the effort required for managing airflow, heat sinks, and fans. Optical transceivers are particularly critical: their design is largely dictated by existing slots and backward compatibility. More power loss therefore means more heat on an almost unchanged surface area.

The Potential of Drop-Down Heat Sinks

An interim solution is drop-down heat sinks. In this design, the heat sink does not contact the transceiver during insertion. Only in the last 10% of the insertion process is it lowered onto a defined contact surface. This allows the use of TIMs (Thermal Interface Materials) without them being damaged during the insertion process.

Molex reports an improvement in heat dissipation of up to 9 °C  (16.2 °F) at 35 W compared to an optimized zipper-fin heat sink. This can be used either for lower fan speeds or for more powerful modules.

Direct-to-Chip Cooling Becomes the Standard for AI and HPC

At even higher power densities, heat dissipation moves directly to the heat source. In direct-to-chip cooling, a cold plate is placed directly on the CPU or GPU. Liquid transports the heat through a circuit to the coolant distribution unit and then to the heat sink.

Molex cites a rack power range of 25 to 50 kW as the threshold at which direct-to-chip cooling becomes relevant. However, this is not a fixed limit. The determining factors are heat flux density, chip power, coolant temperature, and system architecture. Other sources see direct-to-chip cooling and other liquid cooling concepts as applicable for AI racks in the range of about 50 to 100 kW and above. In practice, significantly higher rack power levels are already relevant today.

The key advantage of liquid cooling is its significantly higher heat transport capability compared to air. At the same time, the higher coolant temperatures can be used for heating, for example, buildings.

European HPC projects are also at the forefront of this development: serverChill is investigating the direct liquid cooling of CPU, GPU, and memory; the planned EuroHPC supercomputer HammerHAI is based on a liquid-cooled NVIDIA-GB200-NVL4 platform.

Cooling of I/O Modules: Potential and Limitations

However, the development does not stop at CPUs and GPUs. In high-speed systems with 224 Gbit/s, cooling of optical I/O modules is also becoming increasingly relevant. A single cold plate per transceiver, however, would hardly be practical: with 32 ports, up to 32 supply and return lines would be required.

A solution, therefore, is a shared cold plate for multiple modules, which, however, presents mechanical challenges. The modules do not have exactly the same height; cages, connectors, and cooling surfaces also have tolerances. As a result, a rigid cold plate would thermally contact some modules well and others insufficiently.

Molex relies on a cold plate with floating pedestals. Spring-mounted cooling contacts compensate for mechanical tolerances, provide the necessary contact pressure, and thermally connect multiple QSFP-DD modules simultaneously. With a 1×6 QSFP-DD configuration, six modules can thus be cooled via a shared cold plate.

Two-Phase Liquid Cooling for Extreme Heat Flux Densities

As power density continues to increase, two-phase liquid cooling comes into play. The cooling medium evaporates at the heat source and transports the heat via its enthalpy of vaporization. This allows high heat flux densities to be achieved with relatively small temperature differences.

The European EuroHPC project TEXTAROSSA is investigating two-phase liquid cooling for future exascale systems. The EU project DYMAN combines two-phase cooling with dynamic control of cooling performance and processor workload.

Thermal Design Must Consider the Entire Heat Path

With increasing power density, especially for optical components, it is no longer sufficient to only consider a module's case temperature. Specifying a case temperature of, for example, 70 °C does not necessarily reflect the most critical internal component. Lasers, TIAs, and DSPs may have different thermal margins.

A more precise approach is a thermal analysis of the actually critical components. For this, temperature measurements, simulations, and system tests must be correlated.

This starts at the module level, where hotspots are identified, and extends to the simulation of the entire chassis. Components such as GPUs, SSDs, add-in cards, and I/O modules mutually influence the airflow and thus the temperature distribution.

European research projects such as DYMAN and ECO-Qube are therefore pursuing integrated thermal management, where IT load, cooling circuit, and energy consumption are optimized together.

Immersion Cooling Remains an Option for Highest Power Densities

With immersion cooling, the entire electronics are submerged in an electrically non-conductive liquid. The liquid absorbs the heat directly from the components and transfers it via a heat exchanger.

This solution is suitable for very high heat loads in the range of more than 50 kW per rack. The choice between direct-to-chip, rear-door heat exchanger, and immersion depends on rack power, hardware, infrastructure, and cost-effectiveness. Direct-to-chip cooling currently appears in the literature as the most established liquid cooling solution for AI and HPC applications.

Immersion cooling also has a significant drawback in high-speed communication: the dielectric properties of the cooling liquid change the electromagnetic boundary conditions. For connectors and high-speed components designed for air, this results in altered impedance and signal integrity. Accordingly, components qualified for data rates of 112- and 224-Gbit/s are required.

Hybrid Solutions: Combinations Instead of the Ultimate Cooling System

The development does not aim at completely replacing air cooling. Instead, a hybrid thermal architecture is emerging: air cooling handles less heat-critical components, direct-to-chip cools CPUs and GPUs, cold plates increasingly reach memory, power supply, and I/O, while immersion cooling remains an option for specialized high-performance applications.

The key will be thermal integration. Cooling, mechanics, power supply, and signal transmission can no longer be optimized independently at the upcoming performance levels. Power and cooling must therefore be designed in an integrated manner. The next generation of data centers will not simply be cooled more intensively but thermally re-engineered. 

Underwater Data Center

This is How China Cools its Servers on the Seabed