Why Are We Using GPUs to Run Artificial Intelligence?
If AI is mostly matrix mathematics, why do we need general-purpose GPUs — and why is so much computing power sometimes left unused?
Understanding AI ComputingIn the first part of this series, we explored how NVIDIA transformed graphics processors into a powerful computing platform with CUDA.
But that raises an interesting question.
If artificial intelligence relies heavily on matrix calculations, why are we using GPUs at all?
Wouldn’t a processor designed exclusively for AI be faster, cheaper, and more energy-efficient?
The answer is more complicated than it first appears.
And it reveals a surprising problem with modern AI hardware.
The fastest processor in the world is of little use if it spends its time waiting for data.
A GPU Is a Parallel Computing Machine
A traditional CPU is designed to execute a wide variety of operations, including complex control logic and sequential tasks.
A GPU takes a different approach.
It contains many processing units that can execute large numbers of suitable calculations simultaneously.
That makes GPUs exceptionally good at operations where the same instructions must be applied to large amounts of data.
Graphics rendering is one example.
Artificial intelligence is another.
Neural networks repeatedly multiply numerical values by learned weights, add results together, and apply mathematical transformations.
Many of these operations can be expressed as matrix multiplications.
And matrices are well suited to parallel processing.
But a GPU is still a relatively flexible processor.
It can execute many different mathematical operations, not just matrix multiplication.
That flexibility is useful, but it also means that some of its hardware is more general-purpose than a specialized AI workload requires.
What Makes an AI Accelerator Different?
An AI accelerator concentrates more of its hardware on operations commonly used by neural networks.
Matrix multiplication is particularly important.
Instead of performing every multiplication and addition through general-purpose instructions, specialized hardware can process blocks of matrix operations directly.
NVIDIA’s Tensor Cores are one example.
They are integrated into modern NVIDIA GPUs and accelerate matrix multiplication and accumulation.
Google’s Tensor Processing Units, or TPUs, take specialization further.
Their matrix units use an architecture called a systolic array, in which numerical values move through a grid of processing elements.
The same data can be reused repeatedly without constantly fetching it from external memory.
This makes suitable matrix operations extremely efficient.
Another technique is reduced numerical precision.
Neural networks often work effectively with formats such as FP16, BF16, or FP8 rather than requiring 64-bit floating-point calculations.
Smaller numbers require less storage and can allow more operations to be performed with the same hardware resources.
But AI accelerators still need to perform other tasks.
Neural networks require normalization, activation functions, data movement, and control operations.
That is why Google’s TPUs also include vector and scalar processing units.
A specialized AI processor is not necessarily a matrix calculator and nothing else. It is a processor whose architecture is optimized for AI workloads.
So Why Not Use Only Specialized AI Chips?
In fact, large computing facilities already do.
Google operates extensive TPU infrastructure for training and running AI models.
NVIDIA’s data-center GPUs also contain specialized AI hardware, alongside more flexible computing resources.
The distinction between GPUs and AI chips is therefore becoming less clear.
The more important question is how much specialization makes sense.
A highly specialized processor can be remarkably efficient when its workload closely matches its architecture.
A more flexible processor can accommodate a wider variety of algorithms and computational patterns.
And AI is still evolving.
A processor optimized for one generation of neural networks may not be equally efficient for another.
There is no universal answer.
But there is another reason why simply adding more specialized computing units doesn’t always improve performance.
The Memory Problem
Imagine a factory with a thousand workers.
Every worker can perform a calculation almost instantly.
Unfortunately, the workers must wait for someone to bring them the numbers they need.
Hiring another thousand workers will not solve the problem.
The factory needs a faster delivery system.
Something similar happens inside AI processors.
Modern accelerators can perform enormous numbers of mathematical operations per second.
But those operations require data.
Model weights, intermediate results, and other information must be retrieved from memory and delivered to the computing units.
If the data cannot arrive quickly enough, some processing capacity remains unused.
This is called a memory-bandwidth bottleneck.
It means that processing speed is limited not by how quickly calculations can be performed, but by how quickly the necessary data can be supplied.
And this is particularly important for large language models.
Training and Inference Are Different Problems
Training a neural network involves processing large quantities of data, often in batches.
Large matrix multiplications provide enough work to keep specialized computing units busy.
The processor can reuse data efficiently and perform an enormous number of calculations.
Generating text with a language model is different.
When you submit a prompt, the model first processes the input tokens.
This stage is called prefill.
Much of the work can be performed in parallel.
Then the model begins generating its response.
This stage is called decode.
In a conventional autoregressive language model, each new token depends on the preceding context.
The model generates one token, updates its internal state, and then generates the next.
For a single request, this creates relatively small computational workloads compared with large training batches.
Yet the processor still needs access to the model’s weights and the relevant cached information from previous tokens.
The result?
The computing units may be capable of performing calculations much faster than memory can supply the required data.
During this phase, a processor with enormous theoretical computing power can deliver only a fraction of its peak arithmetic throughput.
Not because the hardware is defective.
But because the workload is limited by something other than computation.
The Trick: Process More Requests Together
AI data centers have a way to improve this situation.
Instead of processing only one user’s request, they can process many requests together.
This is called batching.
If several requests use the same language model, the processor can reuse model weights across multiple calculations.
The amount of useful work performed for each memory access increases.
That allows more of the available computing capacity to be used.
But there is a trade-off.
Larger batches consume more memory and can increase the time individual users wait for their responses.
Data-center operators must balance throughput against latency.
They cannot simply maximize batch size without considering the consequences.
This also explains why a benchmark showing spectacular peak computing performance tells us relatively little about how quickly a system will answer an individual user.
The Fastest Chip Isn’t Always the Best Chip
We can now return to our original question.
Why do we still use GPUs for artificial intelligence?
Because modern AI GPUs combine specialized acceleration with the flexibility to handle other operations and changing workloads.
But specialized alternatives such as Google’s TPUs are already proving that different architectures can be effective.
And neither architecture can escape the fundamental relationship between computation and data movement.
A processor may have enough computing power to perform trillions of operations every second.
If it cannot obtain the required data quickly enough, much of that potential remains unused.
This changes how we should evaluate AI hardware.
Instead of asking only how many operations a processor can perform per second, we should also ask:
How much data can it move?
How efficiently can it reuse that data?
How much useful work can it perform before needing another memory access?
And how does it behave under real workloads rather than ideal benchmark conditions?
These questions are becoming increasingly important as AI models and data centers continue to grow.
The Real Lesson
The development of AI hardware is not simply a race to build processors with more computing units.
It is an effort to balance several competing requirements.
General-purpose flexibility.
Specialized mathematical acceleration.
Memory capacity and bandwidth.
And the ability to keep expensive processing hardware productively occupied.
Modern GPUs already incorporate dedicated AI acceleration, while specialized AI processors include enough general-purpose functionality to execute complete neural-network workloads.
The boundaries are becoming increasingly blurred.
But the fundamental problem remains.
Computing power is only valuable when you can keep it busy.
And that brings us to another question.
If expensive AI processors cannot always use their full computing capacity, what does that mean for the economics of a data center?
That is the subject of Part 3.
References & Further Reading
Google Cloud — TPU System Architecture
https://docs.cloud.google.com/tpu/docs/system-architecture-tpu-vm
NVIDIA — Mastering LLM Inference Optimization
https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/
NVIDIA — Co-Designing AI Model Attention (2026)
https://developer.nvidia.com/blog/co-designing-ai-model-attention-for-fast-interactive-long-context-inference/
This is Part 2 of Understanding AI Computing, a series exploring the hardware, software, and economics behind modern artificial intelligence.
Next: Part 3 — The Hidden Economics of AI Data Centers.
Top comments (0)