Parallel Processing Is the CPU on its Way Out? How GPUs Perform Calculations So Fast

GPUs accelerate AI, simulations, and numerical calculations. Their advantage lies in their ability to process many tasks simultaneously. Why CPUs are designed differently, and why adding more processing units alone does not result in a faster computer.

One million measured values are to be multiplied by a factor and then shifted by a constant value. The same calculation rule applies to each measured value. The result of the first value is not needed to process the second. Therefore, whether the calculations are performed sequentially or simultaneously does not affect the result.

Such tasks offer a high degree of data parallelism: The same process can be applied to many independent data sets. This is exactly what GPUs are designed for. This pattern can be found in image processing as well as in scientific computations and neural networks. However, this does not mean that a GPU is inherently faster than a CPU. The key factor is which tasks can be distributed—and which must wait for previous results.

Even a CPU Performs Parallel Computations

The idea that a CPU always performs one operation at a time is incorrect. Multiple processor cores can handle different tasks simultaneously. Furthermore, within a single high-performance core, multiple independent machine instructions can be executed in parallel.

In addition, there are vector units. They process multiple data values with a single instruction according to the SIMD principle: Single Instruction, Multiple Data. For example, a 256-bit-wide vector can hold eight 32-bit floating-point numbers. A suitable vector addition instruction then processes eight pairs of numbers. However, this does not automatically mean eight fully completed additions per clock cycle; the actual throughput depends on the microarchitecture.

Consequently, the measurement problem described at the beginning can also be parallelized on a CPU. The main difference from a GPU lies in how well the hardware is tailored to this operating mode.

The CPU Speeds Up Program Execution

Many programs consist of a series of tasks: searching through data structures, checking conditions, calling functions, and responding to intermediate results. Sometimes, the memory address needed next isn't determined until after the previous access has taken place.

High-performance CPU cores therefore devote significant resources to quickly executing a single instruction stream. A branch prediction estimates the program’s subsequent path. Out-of-order execution prioritizes independent instructions while others are still waiting. Caches keep frequently used data close to the processing units. These mechanisms can reduce wait times or partially mask them. They do not eliminate true data dependencies.

All of this requires chip area and power. These resources are not used exclusively for additional adders and multipliers, but also to provide flexible and rapid support to existing computing units. This makes sense when a small number of time-critical processes or irregular tasks determine the runtime.

The GPU Manages the Workload

A GPU allocates resources differently. Its processing units are organized into groups that handle many threads. A thread describes a single execution sequence, such as the processing of a measured value. At NVIDIA, threads are grouped into warps, each containing 32 threads. This execution principle is called SIMT: Single Instruction, Multiple Threads. Within such a group, common instructions are applied to the respective data of the active threads. Other GPU architectures use related concepts. Thus, the GPU does not simply consist of thousands of complete CPU cores. Many execution paths share parts of the instruction control unit. Therefore, the advertised number of GPU processing cores cannot be directly compared to the number of cores in a CPU.

Even if a million threads are started, that doesn't mean a million computations are running simultaneously on physically existing units. The hardware processes the tasks using its available capacity. In doing so, it keeps many threads on standby: if one group is waiting for data, other groups that are ready to run can take their turn.

Branches Reduce Utilization

Concurrent execution works particularly well when threads in the same group follow similar paths through the program. If some values must be processed using method A and others using method B, different execution paths can arise within a warp.

The GPU can handle such branching. However, it may need to execute the different paths separately, with only the corresponding threads active at any given time. As a result, compute paths remain unused at times. This divergence explains why irregular tasks can be inefficient despite high nominal computing power. Not every branch is problematic: if all threads in a group follow the same branch, no such splitting occurs.

While irregular paths slow down the GPU, other domains benefit enormously from its parallel structure—starting with neural networks. Many computational steps here can be formulated as matrix operations. In matrix multiplication, numerous result values are produced through repeated multiplications and additions. Different result ranges can be processed in parallel. In addition, cleverly selected data blocks can be reused multiple times. A value loaded once then contributes to multiple calculations. This reuse is crucial: it increases the number of computational operations relative to the amount of data moved. Specialized matrix units—such as Nvidia’s Tensor Cores—go even further. They perform matrix multiplication and accumulation operations on small blocks.

Simulations also involve suitable tasks. Local computations on many elements or grid points can often be performed in parallel. In FEM, systems of equations are also generated, and their numerical solution can take up a significant portion of the computation time. However, the benefits depend on the solution method and the data organization. Many FEM matrices are sparsely populated. Processing them requires not only numerical values but also additional structural information; memory accesses and the exchange of intermediate results can limit throughput. In some cases, it is therefore worthwhile to reformulate the algorithm. The FEM library MFEM, for example, describes “partial assembly”: the mathematical operator is applied without constructing the complete global matrix. This can reduce memory requirements and data movement, particularly when using suitable higher-order elements. For this to work, the preconditioning must also be suited to the method.

Why Don't We Build CPUs Like GPUs?

CPU developers have long been using data parallelism through vector units. However, a complete focus on GPU-like throughput would weaken capabilities that are important for other tasks: quickly processing a small number of dependent operations, responding to changing program paths, and efficiently executing irregular workloads. This can be illustrated using a counterexample to the measurement problem: If each new value depends on the result calculated immediately before it, the steps cannot easily be distributed across many computational paths. A different algorithm may sometimes help. However, additional hardware alone does not eliminate this dependency.

Overall acceleration also remains limited. Here’s a simplified calculation: If 90 percent of the previous runtime can be accelerated 100-fold, while 10 percent remains unchanged, the runtime drops to 10.9 percent. The entire process thus becomes only about 9.2 times faster. This calculation does not yet take data transfer overhead into account. That is why CPUs and GPUs complement each other. The CPU therefore continues to handle program control and serial tasks, while the GPU takes on the heavy parallel computing workloads. This does not mean the CPU is obsolete—it is their interaction that makes modern computers so powerful.