What is CFD?

Computational fluid dynamics (CFD) computes how air, heat and pressure move through a space by solving the equations of fluid flow on a computer. For a data center, that turns questions like Will this row run hot if a cooling unit fails? or How much of our cooling is reaching the racks? into questions you can ask of a model instead of the room.

What it answers

Data center cooling is a spatial problem. A room can have more total cooling capacity than its heat load and still overheat a rack, because the cold air never reaches that rack’s intake. Averages don’t show that; a spatially resolved model does. CFD lets a team:

find hot spots and trace them to their cause, such as recirculated exhaust, a starved tile or bypass air;
check whether the room could run with less cooling airflow or a warmer supply, which is where PUE improvements usually come from;
run failure cases, such as each cooling unit off in turn, on the model instead of the room; and
compare options such as containment, plenum depth or tile layout before committing capital.

How a simulation works

Every CFD study follows the same four steps.

Define the domain. Build the geometry and set the boundary conditions. For a data hall that means the room, the raised floor or ceiling plenum, racks with their heat loads and airflows, cooling units with their supply temperatures and airflow, perforated tiles, and any containment.
Mesh it. Divide the space into many small cells. The solver keeps one temperature, one air velocity and one pressure for each cell, so the mesh decides what the model can see. CoolSim builds the mesh from the model when the job is submitted.
Solve. The solver iterates until the mass, momentum and energy balances hold in every cell and the quantities being watched stop changing.
Post-process. The solution becomes something you can act on: temperature and velocity maps, pathlines showing where the supply air goes, rack intake temperatures, and metrics built from them.

CoolSim solves for the steady state: the room as it settles with its heat loads and cooling held fixed. That answers where the air and heat end up. How quickly a room heats up after a failure is a different question. It needs a transient (time-dependent) calculation, which we run as a separate cooling failure study.

The mesh

Of the four steps, the mesh gets the least attention and decides a great deal. Each cell holds a single value of each quantity, so anything smaller than a cell is averaged away. And the way neighboring cells meet decides how accurately the solver can pass heat and air between them.

Small enough to see the airflow

A 2 ft (0.6 m) floor tile is a good yardstick. We want at least three cells across every tile, and four or five is better. With two, the jet leaving the tile is described by two values across its width, so it can’t have a fast core and slower edges.

Smaller cells cost time. Halving the cell size multiplies the number of cells by eight, and the solve usually takes more than eight times as long. How much model is enough? solves one room on three meshes and shows which answers moved as the cells got smaller.

Why box-shaped cells

Cells can be any shape: boxes, tetrahedra with four triangular faces, or many-sided polyhedra. A data hall is made of boxes. Racks, cooling units, floor tiles, containment panels and plenums are rectangular and line up with the walls, so a mesh of box-shaped cells puts every one of their surfaces exactly on cell faces. Nothing has to be approximated to make the geometry fit.

The bigger reason is accuracy, and it comes down to the angle at which neighboring cells meet. Heat, momentum and turbulent mixing spread from cell to cell through the faces the cells share. To work out how much crosses a face, the solver takes the difference between the two cells’ values and divides it by the distance between their centers. That is exact only when the line between the two centers crosses the face square-on.

Two small grids of cells. A: square cells; the line from the center of cell P to the center of its neighbor N crosses their shared face at a right angle. B: slanted cells; the line from P to N crosses the shared face at an angle theta away from the face's normal. — A: on box cells, the line from one cell’s center (P) to its neighbor’s (N) crosses the face they share at a right angle, so the whole difference between P and N is change across the face. B: on slanted cells the line is θ off square, and part of the difference runs along the face instead.

On box cells the line always crosses square-on (A). On slanted cells it doesn’t (B). It meets the face at an angle, called the non-orthogonality, and part of the difference between the two cells is change along the face rather than across it. The solver has to estimate that part from the cells around them and take it out, a step called non-orthogonal correction. The estimate carries its own error, it adds work to every iteration, and as the angle grows the solution gets less stable; mesh checks commonly flag faces past about 70°. On box cells the correction is zero, because there is nothing to correct.

This is not a fine point. Turbulence is modeled as extra mixing (see Turbulence below), and the solver computes that mixing face by face in exactly this way.

What we measured

We checked the meshes of 21 customer jobs, from 20 customers, ranging from 8,400 to 17 million cells.

In 14, every face between two cells was square to the line between their centers, to within a hundredth of a degree. That includes all 51 million faces in the largest.
In five more, 98 to 99.8 percent of the faces were.
The other two mixed in tetrahedral and polyhedral cells, and 84 percent of their faces were square.

In the seven meshes with any slanted cells, the worst face in each was between 54° and 86° off square.

What box cells give up

A round pipe, a column or an angled wall doesn’t line up with box cells. The mesh either steps around it in a staircase or slants cells to fit it, and slanted cells bring the correction back. And because box cells line up in rows, a fine spacing needed in one place, such as a narrow gap beside a rack, carries on until it meets a wall or a solid object, adding cells where they aren’t needed. For a room of rectangular equipment, we think that is the right trade.

Turbulence

Air in a data hall is turbulent. The smallest eddies are far smaller than any practical cell, so turbulence is modeled rather than resolved. Its main effect is faster mixing of momentum and heat, and models represent that as an added, turbulent viscosity.

General-purpose CFD codes often compute that viscosity from two extra transport equations, the k-ε and k-ω families. CoolSim uses a simpler zero-equation, or mixing-length, model. It computes the turbulent viscosity directly from the local flow and the distance to the nearest wall, with no extra equations to solve. Zero-equation models are a common choice for enclosed rooms such as data centers, where the questions are about where air goes at the scale of racks and aisles.

Convergence

A converged solution means the equations are balanced on this mesh. It does not mean the inputs are right. A model is only as good as its rack heat loads and airflows, its cooling-unit settings, and the leakage paths it includes. For that reason, the difference between two runs of the same model (with and without containment, with one cooling unit off) is usually more trustworthy than any one absolute temperature.

The equations behind it

Fluid flow is governed by three conservation laws: mass cannot appear or disappear, force equals the rate of change of momentum, and energy is conserved. Together they are usually called the Navier–Stokes equations. Below, $\rho$ is the air density, $\mathbf{u}$ the air velocity, $p$ the pressure and $T$ the temperature.

Conservation of mass says that air flowing into any small volume either flows out again or changes the density inside it:

\frac{\partial \rho}{\partial t} + \nabla \cdot (\rho\,\mathbf{u}) = 0

Conservation of momentum relates the air’s acceleration to pressure differences, the stress $\boldsymbol{\tau}$ from viscosity and turbulent mixing, and the weight of the air, $\rho\,\mathbf{g}$ :

\frac{\partial (\rho\,\mathbf{u})}{\partial t} + \nabla \cdot (\rho\,\mathbf{u}\,\mathbf{u}) = -\nabla p + \nabla \cdot \boldsymbol{\tau} + \rho\,\mathbf{g}

The weight term is where buoyancy comes from. Air in a data hall moves too slowly for pressure to change its density noticeably, but temperature does, and that is why rack exhaust rises toward the ceiling. CoolSim computes density from temperature with the ideal gas law, at the air pressure for the site’s altitude, so warm air rises in the model as it does in the room.

Cooling is about moving heat, so an energy equation tracks temperature as the flow carries it and the air conducts it, with $k$ the conductivity (turbulent mixing included) and $S$ the heat added, for example by the equipment in a rack:

\frac{\partial (\rho\,c_p T)}{\partial t} + \nabla \cdot (\rho\,c_p\,\mathbf{u}\,T) = \nabla \cdot (k\,\nabla T) + S

For the steady state, CoolSim drops the time derivatives, the first term of each equation. The $\nabla \cdot \boldsymbol{\tau}$ and $\nabla \cdot (k\,\nabla T)$ terms are the ones the solver computes face by face from the difference between neighboring cells, the step that box-shaped cells keep free of correction. There is no closed-form solution to these equations for a real room full of racks, ducts and obstructions, which is why they are solved numerically on a mesh.

Where CoolSim fits

Traditionally, CFD meant general-purpose software and a dedicated analyst. CoolSim packages the workflow for data centers. Models are built in a Windows application from libraries of racks, cooling units, tiles and other data center objects. The meshing and the Ansys Fluent solve run in the cloud, and the results come back as a report with rack intake temperatures, maps, pathlines and the RCI and RTI cooling metrics.

For a longer introduction, with two example rooms, see WP107: Using CFD for Data Center Design and Analysis.