AI Training Is Going to Multiple Data Centers

In their quest to build ever-larger AI models, technology giants will soon train new AI models across multiple data center campuses.

The AI boom is driving a global surge in electricity consumption, straining power grids in places like the United States. This demand is expected to increase even further, according to SemiAnalysis, a research firm specializing in semiconductor and AI industries.

No signs of plateauing

I previously wrote about how the next generation of AI models is being trained on GPU clusters with as many as 100,000 GPUs. This trend is attributed to the scaling paradigm, where OpenAI researchers have shown that increasing processing capabilities leads to better AI models.

Mira Murati, the CTO of OpenAI, discussed this in May: “From a technology perspective, we believed in scale, and we bet on the scaling paradigm. [It’s] this idea that you throw a ton of compute and data at these large language models, and that it will lead to emerging capabilities, that models will become more powerful and be able to do more things.”

In the same interview, she mentioned there is no evidence of the scaling paradigm slowing down, and that OpenAI will continue to push this approach to make models more powerful with more compute and high-quality data.

Multiple data centers

Technology giants are planning two steps ahead, working hard to build infrastructure to support 300,000 GPU clusters. According to SemiAnalysis, this is not a distant plan but something that could happen as soon as next year.

The extensive report went on to describe how Google is currently leading with the capability to conduct gigawatt-scale training runs across multiple campuses. This is due to the millions of Tensor Processing Units (TPUs) it already operates in its data centers. TPUs are custom application-specific integrated circuits developed by Google for machine learning.

Not to be outdone, Microsoft and OpenAI have launched an ambitious infrastructure project to construct a multi-gigawatt computer system spanning multiple data center campuses. These data centers will be connected by numerous of fiber optics cables with high-speed networking equipment.

This raises the question: When does this end? And how do we manage the environmental and logistical challenges posed by such massive energy demands?

Image credit: iStock/cybrain