AI data centers: a deep dive

For most of the cloud era, I thought of data centers as boring in the best way. They were warehouses for computers. They were cooled by big air handlers, sited where land and fiber were cheap, and judged on how close they could push one efficiency ratio toward 1.0. You could go a whole career in software without knowing where your code physically ran, and that was the point.

That stopped being true sometime around 2023. The data center has become a story about power plants, transmission lines, water rights, nuclear licensing and sovereign industrial policy. Utility executives now talk about it on earnings calls the way software companies used to talk about user growth. I've spent a good part of this year reading the reports underneath the headlines, and this essay is my attempt to put the pieces in one place: what changed physically, why electricity became the binding constraint, and what that means for the very large bet now being placed on it.

I'll try to stick to numbers I could trace to a primary source, and to say plainly where the forecasts disagree. They often do.

#What AI changed inside the building

There are two very different workloads behind the phrase "AI compute," and it helps to keep them apart.

Training is a single huge job. Thousands to hundreds of thousands of accelerators work in lockstep for weeks or months, constantly exchanging gradients. It is extremely sensitive to the network between chips. One slow link stalls the whole job, so you want the GPUs as physically close together as you can manage. Training clusters reward density above almost everything else.

Inference means serving a trained model to users. It is many small, latency-sensitive jobs, and it looks more like classic cloud work. It can be spread out, placed nearer to users, and scaled with demand. For a while, the common view was that training would dominate AI's energy footprint. That view is ageing fast. Reasoning models think for longer before they answer, agents run multi-step loops, and the user base keeps growing. All of that pushes inference toward being the larger and more durable load.

Both workloads run on the same physical shift, which is the accelerator. A modern AI GPU draws on the order of a kilowatt by itself. Pack enough of them together and the rack stops looking like a rack.

#From 8 kW to 120 kW (and beyond)

The industry's survey data shows how far the average facility is from the frontier. Uptime Institute's 2024 global survey found the average rack density creeping upward but still in the single-digit kilowatts. Most operators reported that they didn't have a single rack above 30 kW¹. That's the world most existing data centers were designed for: air-cooled, with raised floors or hot and cold aisles, and power distribution sized for a few kilowatts per cabinet.

Compare that with NVIDIA's GB200 NVL72, which puts 72 Blackwell GPUs and 36 Grace CPUs into a single liquid-cooled, rack-scale system that behaves like one giant accelerator². When it was announced in March 2024, The Register put its power draw at about 120 kW per rack. Direct-to-chip liquid loops carry most of the heat away³. That is roughly fifteen times the density of a typical enterprise rack, and the roadmap goes further. NVIDIA has published a plan to move data center power distribution to 800-volt DC, explicitly to support megawatt-scale racks. It ties full-scale production to its Kyber rack systems in 2027⁴.

It's worth spelling out what a hundredfold jump in rack density does to a building. Electrical rooms get bigger than server halls. The floor has to carry liquid manifolds, coolant distribution units and much heavier racks. The electrical design moves toward medium voltage running deeper into the building. And when existing operators ask whether they can retrofit an older facility for AI, the honest answer is often "partly, at great expense." For the frontier, the answer is usually to build new.

The unit of planning for AI infrastructure is no longer the server or the rack. It's the gigawatt.

#The power problem, in numbers

The best global baseline I know of is the International Energy Agency's Energy and AI report from April 2025. It estimates that data centers used about 415 TWh of electricity in 2024, roughly 1.5% of global consumption, and that this demand has grown about 12% per year since 2017. In its base case, that figure more than doubles to around 945 TWh by 2030, about what Japan uses today. Electricity demand from AI-optimised data centers specifically is projected to more than quadruple⁵.

The concentration is the striking part. The IEA expects data centers to account for almost half of all US electricity demand growth through 2030. It projects that the US will use more electricity for processing data in 2030 than for making all of its energy-intensive goods (aluminium, steel, cement and chemicals) combined⁵.

Lawrence Berkeley National Laboratory's December 2024 report to Congress gives the US-specific picture. US data centers used about 176 TWh in 2023, or 4.4% of national electricity. By 2028, the report projects between 325 and 580 TWh, or roughly 6.7% to 12% of total US consumption⁶. The width of that range is the honest part. It reflects real uncertainty about chip shipments, utilisation, cooling choices and how much announced capacity actually gets built.

2024 United States Data Center Energy Usage Report

Berkeley Lab's congressionally requested estimate of US data center electricity use since 2014, with scenarios through 2028.

Lawrence Berkeley National Laboratory

#Why a national average undersells it

National percentages hide how local this load is. Data centers cluster, around fiber hubs, tax incentives and existing campuses, and the grid is a local machine. Ireland is the clearest example. The Central Statistics Office reports that data centers used 22% of all metered electricity in 2024, up from 5% in 2015. Their consumption grew 10% in that one year, against 3% growth for everyone else combined⁷. Northern Virginia, central Ohio and parts of Texas are heading toward similar pressure in their own utility territories.

Forecasts can also be wrong in either direction. Efficiency gains are real. Each chip generation does more work per joule, and better software (quantisation, distillation, smarter batching) cuts the energy per answer. There's also a long history of electricity forecasts that assumed growth and then didn't see it. But efficiency gains in computing have historically been spent on doing more computing, not on using less power. I would not bet on Jevons losing this time.

#The real bottleneck is the grid

If you talk to people building these facilities, the scarce resource isn't GPUs anymore. GPU supply eased through 2024 and 2025. What's scarce is an energised connection to the grid on a timeline that matches the hardware's useful life.

Berkeley Lab's annual Queued Up report tracks the backlog of power plants and storage projects waiting to connect to the US grid. At the end of 2023 it found nearly 2,600 GW of generation and storage capacity in interconnection queues. That's more than twice the installed capacity of the entire existing US power fleet⁸. Most of those projects will never be built, and the queue shrank somewhat in the following year as operators reformed their processes. But the sheer size of the backlog tells you the system was built for a slower world.

The cost shows up in prices too. In PJM, the grid operator for much of the US mid-Atlantic and Midwest, including Northern Virginia, capacity auction prices went from $28.92 per megawatt-day for 2024/25 to $329.17 for 2026/27. IEEFA's analysis attributes much of that tightening to projected data center demand. It also warns that those load forecasts may not fully materialise⁹. Either way, households in the region end up paying part of the bill. That is why data center siting has become a matter for state politics rather than a zoning detail.

The industry's response is splitting into a few strategies, none of them clean:

• Go where the power already is. Sites next to existing generation, especially nuclear plants and stranded gas, have suddenly become valuable.
• Bring your own generation. On-site gas turbines, fuel cells and batteries let a campus energise before the grid catches up. The cost is emissions and permitting fights.
• Become flexible. Grid operators are increasingly willing to connect large loads faster if they agree to curtail during peak hours. Training jobs can pause; latency-sensitive inference mostly can't.
• Sign very long contracts for firm, clean power. This is where nuclear comes in.

#Why Big Tech went nuclear

Solar and wind are the cheapest new energy in much of the world, and hyperscalers have been among the largest buyers of renewable power for a decade. But a training cluster runs around the clock. Matching a 24/7 load with intermittent supply takes storage, overbuilding or fossil backup. Nuclear offers something the others don't: large amounts of firm, carbon-free power from a single site, with a decades-long operating life.

#Restarts: Three Mile Island becomes Crane

The deal that made this real came in September 2024. Constellation announced a 20-year power purchase agreement with Microsoft that would restart Three Mile Island Unit 1, renamed the Crane Clean Energy Center. Unit 1 is not the reactor that partially melted down in 1979. It ran until it was retired for economic reasons in 2019. The restart would bring back about 835 MW of carbon-free capacity¹⁰. As of spring 2026, Constellation was still targeting a restart in the second half of 2027, ahead of the original 2028 date. The open questions were mostly about grid deliverability and interconnection rights rather than the reactor itself¹¹. That says a lot about where the bottleneck actually is.

#Uprates, co-location and the existing fleet

Amazon took a different route through Talen Energy's Susquehanna plant, also in Pennsylvania. Its original plan was to co-locate a data center campus directly behind the plant's meter, which drew regulatory scrutiny over grid cost allocation. In June 2025 the deal was restructured as a front-of-the-meter power purchase agreement for up to 1,920 MW through 2042, ramping to full volume by 2032. The companies also said they would explore small modular reactors and uprates at the site¹². The lesson: even when the power plant is right next door, the grid operator and the regulator still decide who pays for what.

#New builds: small modular reactors

Google went for new technology. In October 2024 it signed what it described as the first corporate agreement to buy power from multiple small modular reactors: up to 500 MW from Kairos Power's molten-salt-cooled design. The first unit is targeted for 2030 and further deployments through 2035¹³. This is a deliberately long-dated bet. It aims to give a new reactor design an order book so it can move down the cost curve, much as early corporate PPAs did for wind and solar.

My read is that the restart and uprate deals are about power this decade, and the SMR deals are about power next decade. Both are expensive per megawatt-hour compared with a new solar farm. The hyperscalers are paying for firmness, for the carbon accounting, and above all for certainty of supply. When a GPU cluster depreciates in five or six years, a year of delay waiting for power is very expensive.

#Cooling, water and the efficiency plateau

Every watt that goes into a chip comes out as heat, and getting that heat out of the building is the second engineering story of AI data centers. For decades, the industry measured this with power usage effectiveness (PUE). Uptime's surveys show that the industry average has barely moved: about 1.56 in 2024, flat for the fifth year in a row¹. The 2025 survey again described average PUE as essentially unchanged and hovering just above 1.5¹⁴. Newer, larger hyperscale facilities do much better. The average is held back by a long tail of older enterprise and colocation sites.

At 120 kW per rack, air cooling simply runs out of headroom. That's why the AI generation is built around direct-to-chip liquid cooling: cold plates on the GPUs and CPUs, fed by a facility water loop. Adoption across the broader industry is still gradual. In Uptime's 2025 cooling survey, about a fifth of respondents reported using direct liquid cooling in some form, and most operators still rely mainly on air¹⁵. But for new AI-first builds, liquid isn't optional anymore.

#The water nobody counts

Water is where measurement is weakest. A 2023 paper by Pengfei Li, Jianyi Yang, Mohammad Islam and Shaolei Ren, "Making AI Less 'Thirsty'," estimated that training GPT-3 in Microsoft's US data centers could have directly evaporated about 700,000 liters of clean freshwater. It also estimated that global AI demand may account for 4.2 to 6.6 billion cubic meters of water withdrawal in 2027¹⁶. Those numbers come from models with wide error bars, and the authors say so. The more lasting contribution of the paper is the framework. It counts the on-site water evaporated for cooling and the off-site water used to generate the electricity, and it shows that the timing and location of a workload change its water footprint a lot.

The engineering trade-offs are real and point in opposite directions:

1. Evaporative cooling is very energy-efficient, but it consumes water, which is worst in hot, dry places where land and sun are cheap.
2. Closed-loop liquid cooling with dry coolers uses little water but more electricity, especially in summer.
3. Warm-water liquid cooling runs coolant hot enough that you rarely need chillers, and the waste heat can sometimes be reused for district heating. That works in Nordic cities and rarely in Arizona.

My view is that water will become a local, permitting-level fight long before it becomes a global one. As with power, the national average is the wrong lens.

#The capex wave

The money is hard to grasp even when you've been watching it. By Sherwood News's tally in February 2026, Amazon, Alphabet, Meta and Microsoft spent nearly $400 billion on capital expenditure in 2025. Their guidance and analyst estimates pointed to roughly $600 billion in 2026, a 50% increase in a single year. Amazon guided to about $200 billion, Alphabet to $175–185 billion and Meta to $115–135 billion¹⁷. Several of those ranges were raised again during the year. Most of that money goes to data centers and the chips inside them.

Then there's Stargate, the joint venture of OpenAI, SoftBank, Oracle and MGX announced in January 2025 with a headline commitment of $500 billion and 10 GW. By September 2025, OpenAI said that the flagship campus in Abilene, Texas, five newly announced US sites and its CoreWeave projects together came to nearly 7 GW of planned capacity and more than $400 billion of investment over three years¹⁸. Note the unit: gigawatts, not servers. The industry now plans compute in the same units utilities use for power plants.

[1hr Talk] Intro to Large Language Models — Andrej Karpathy

#Is it a bubble?

The cleanest statement of the skeptical case is still David Cahn's essay at Sequoia, "AI's $600B Question," from June 2024. His arithmetic is simple. Take NVIDIA's data center revenue run-rate, double it to account for the rest of the data center cost (buildings, energy, cooling), then double again for the margins of whoever resells that compute. The result is the annual end-user AI revenue needed to justify the build-out, and at the time the gap between that and actual revenue was enormous¹⁹.

GPU computing is increasingly turning into a commodity, metered per hour.

I find it useful to separate the layers, because they don't all carry the same risk:

• Powered land and grid connections are the most durable asset. A substation and an interconnection agreement will be valuable for decades whatever happens to model architectures. If anything here is undervalued, it's this.
• Buildings and cooling plant last 15–20 years but are being designed for densities that keep changing. A hall built for 40 kW racks is already awkward for 120 kW racks.
• Accelerators are the fragile layer. They depreciate fast, a new generation arrives every year or two, and their value depends entirely on rental prices staying high. This is where an overbuild would hurt first.

The railway and fiber-optic comparisons are popular for a reason. Both were real technologies that changed the economy, both were overbuilt by investors who went bankrupt, and the infrastructure they left behind turned out to be enormously useful to whoever bought it cheap. If AI follows that pattern, the losers are the people who financed GPUs at peak prices with debt. The winners are whoever ends up owning the powered sites.

What would change my mind in the bullish direction is inference demand that keeps compounding as models get cheaper. Agents are the most likely source. These are systems that operate software and the web the way a person does, consuming tokens continuously instead of in short chat turns. Karpathy made a related point when OpenAI launched Operator. He argued that agents working through the same general interfaces humans use (screens, keyboards, browsers) are the digital counterpart of humanoid robots in the physical world:

If that comes true, today's capex will look early rather than excessive. If it stalls, we'll have built a lot of very expensive, very warm buildings in a hurry.

#Geography and sovereignty

A decade ago, data center siting was a real estate question. Now it's a matter of national strategy, for three reasons.

First, energy geography. Compute is moving toward cheap, abundant, firm power: Texas and the US interior, the Nordics, the Gulf. Places with a strained grid, such as Dublin, Amsterdam and Frankfurt, have used moratoria or tighter connection rules to slow new builds. The map of AI is turning into a map of electricity.

Second, sovereignty. Governments have concluded that relying entirely on foreign clouds for frontier compute is a strategic risk. The European Commission's InvestAI initiative, launched in February 2025, aims to mobilise €200 billion for AI, including a €20 billion fund for "AI gigafactories" each built around roughly 100,000 advanced chips²⁰. Similar programs exist in the Gulf states, India, Japan and elsewhere. Whether you call it industrial policy or insurance, it means demand won't come only from commercial ROI.

Third, supply chain chokepoints. Leading-edge chips, high-bandwidth memory, large power transformers and gas turbines are each produced by a small number of suppliers, some with multi-year lead times. Export controls add a geopolitical layer on top. A country can have all the power and land it wants and still be unable to build fast.

#What I'm watching

If you want to know where this goes without reading every earnings transcript, these are the signals I think matter most:

1. Utilisation and rental prices for GPUs. Falling hourly prices for current-generation accelerators would be the earliest sign of overbuild.
2. Interconnection reform and large-load tariffs. Watch how fast grid operators connect flexible loads, and whether states make data centers pay upfront for the grid upgrades they trigger.
3. The Crane restart timeline. It's the bellwether for whether nuclear can actually deliver on a data center schedule.
4. Inference's share of compute. If agentic workloads make inference the dominant load, demand becomes stickier and more distributed. It looks less like a race to build ever-larger training clusters.
5. Measured, not modelled, water and energy disclosures. The data is still thin. Mandatory reporting, as in the EU, would make every number in this essay more reliable.

The thing I keep coming back to is how physical all this is. We spent twenty years building abstractions that let us forget the machines: the cloud, serverless, APIs. AI has dragged the machines back into view, along with the turbines, the transformers, the cooling towers and the transmission lines behind them. Whatever happens to the valuations, the next decade of computing will be decided as much by electrical engineers and utility regulators as by researchers. I think that's a healthy thing to remember.