Investigation of Immersion Cooling For Data Center Hardware: Progress, Opportunities And Challenges

Author: Chang and Jati, Meta Collection: 2023 Global Summit - Cooling Environments

Summary

This presentation reports Meta's multi-year investigation into single-phase immersion cooling as a path to supporting next-generation AI hardware in their data centers. Using the OCP YV3.5 1U server (4 servers per sled, 3 sleds per chassis) as the primary test vehicle, Meta's team characterized thermal performance gains, material compatibility issues, and operational scaling challenges across five levels: individual server TTV, silicon-based sled, multi-chassis, partial tank, and full custom tank. The motivation is straightforward: air cooling is like a bicycle—adequate for small trips—but immersion is the shuttle needed to move an entire building's worth of compute to the next location when AI workloads demand power densities that air cannot support.

Thermal performance results show single-phase immersion at 30°C inlet delivers 2.5× the module cooling capability and 3.1× the DIMM cooling capability vs. air cooling, while enabling 2.3× rack power density in the same floor area. Beyond raw thermal gains, immersion eliminates heatsinks for auxiliary components (voltage regulators, retimer switches, CXL ASICs, boot drives), which collectively represent significant BOM cost and board real estate. Material compatibility testing revealed EPDM as incompatible with hydrocarbon-based fluids—discoloration appeared within one week of a partial-tank test—leading the team to build a compatibility matrix covering copper, stainless steel, bakelite, polyimide, synthetic rubber, nitrile (NBR/PVC), and EPDM against representative hydrocarbon and fluorochemical fluids.

Meta's near-term vision focuses on enabling single-phase immersion with PFAS-free fluids, while remaining open to two-phase and warm-fluid exploration longer term. The presentation identifies five key challenge areas blocking scale deployment: flow distribution/bypass, need for board-native immersion designs (vs. retrofitting air-cooled boards), component warranty availability, thermal interface materials (TIM) effectiveness in fluid, and material compatibility with standardized fluid recycling protocols.