Scale Is All You Need? Part 3-2

Note: If you haven’t seen Part 3-1, you can read it here.

Energy

After analyzing the exponentially increasing demand for compute, which is largely driven by the increasing complexity and size of modern AI models, we are now turning our attention to one of the fundamental requirements for these developments: power. Without sufficient energy supply, even the most powerful data centers and GPUs would not be able to do their work.

But as the models continue to grow in size, energy requirements are also growing at a breathtaking pace. Each additional GPU used to train models like GPT, Llama or DALL-E consumes not only more computing power but also more electricity This leads us to a crucial bottleneck on the road to AGI: the availability and sustainability of energy. While companies like NVIDIA are working on developing more efficient chips (e.g. NVIDIA's B200 series is said to be significantly more energy efficient than the H100 series), the energy question remains unresolved: power consumption is increasing exponentially, and the efficiency gains of modern hardware can only cushion this development to a limited extent.

A glance at the infrastructure already shows that the demand for energy for AI applications has reached new dimensions. Microsoft's purchase of 5 gigawatts from the Three Mile Island nuclear power plant is a clear indicator of the amount of electricity needed to operate AI systems today. Sam Altman of OpenAI also secures access to enormous amounts of energy to ensure the future development of his AI models. 

But what happens when energy demand continues to rise? Can renewable energies or nuclear power plants keep pace with this demand? And what geopolitical implications will arise if access to electricity resources becomes the new question of power? Answering these questions is essential, because one thing is clear: without sufficient energy supply, progress in AI development and the achievement of AGI will be significantly slowed down or even stopped.

In the following, we therefore examine the current and future electricity demand of AI developments. We analyze how much energy is needed to train and operate modern models, what the forecasts are for future consumption, and whether the current infrastructure is capable of meeting this demand. One thing is clear: the path to AGI will be decisively influenced not only by compute, but also by the energy question. We then turn to the question of access and power in the context of social inequality over compute and electricity.

Data centers currently consume around 3% of the world's electricity. This is expected to rise to 6% by 2030. A study by Goldman Sachs clearly demonstrates the increasing demand.

But it could be even worse, because our electricity demand is not developing linearly, but exponentially. If, for example, the number of parameters in a neural network is doubled, the energy requirement increases not just by double, but by a multiple, since the computing time for processing these additional parameters is considerably longer. This kind of scaling effect is particularly evident in the largest AI models.

Big Tech is spending tens of billions quarterly on AI accelerators, which has led to an exponential increase in power consumption. Over the past few months, multiple forecasts and data points reveal soaring data center electricity demand, and surging power consumption. The rise of generative AI and surging GPU shipments is causing data centers to scale from tens of thousands to 100,000-plus accelerators, shifting the emphasis to power as a mission-critical problem to solve.

Demand is particularly challenging in the US because of the high level of investment in data center construction there.

And the new generation of GPUs in particular is set to increase demand even further.

Although this is not the place to start a discussion about CO2 emissions, it should be clear that the question of electricity production is an important one. Globally, efforts are now being made to focus primarily on renewable energies, but global CO2 emissions are still rising – and of course the training of new models has an impact on this.

A paper from the University of Massachusetts Amherst stated that “training a single AI model can emit as much carbon as five cars in their lifetimes.” Yet, this analysis pertained to only one training run. When the model is improved by training repeatedly, the energy use will be vastly greater. 

Consequently, it is not surprising that former Google CEO Eric Schmidt says that the climate targets are unattainable, especially in the context of AGI. 

In addition, requests to an LLM consume significantly more power than a regular Google search. Routers that select the right model (an SLM or an LLM) depending on the question could help to reduce demand a little, at least temporarily. Nevertheless, the more AI searches replace regular Google searches (e.g. Perplexity.ai), the more electricity demand increases here as well.

But even if you can generate the electricity, it also has to be supplied to the data centers. Often, the power lines are not designed for such a load. And that, in turn, poses major challenges for the power grid.

We see that, in addition to computing, electricity demand poses the greatest challenge for us Exponentially increasing demand, which collides with the demand for climate protection and an infrastructure (power lines) that are not designed for this high load. Although attempts are being made on all sides to counter this by connecting old nuclear power plants to the grid and expanding renewable energies, new GPUs and larger models will have a higher long-term demand than electricity can be supplied. A solution must therefore be found in the medium term. Because the data situation is clear.

Part 3-3 is coming out tomorrow. Subscribe to the Forward Future Newsletter to have it delivered straight to your inbox.