Google TPU

A TPU is a chip that does one thing: multiply matrices. Not "mostly" — the design gives up nearly everything a general processor offers in exchange for doing that single operation extremely well.

Understanding why that trade pays off means looking at what the expensive part of arithmetic actually is.

The operation

Almost all the work in a neural network reduces to multiply-and-accumulate: take a list of inputs, take a list of weights, multiply them pairwise, add the products.

3×4 + 7×1 + 2×9 = 12 + 7 + 18 = 37

That is the whole computation, repeated an enormous number of times. Multiplying two 128 × 128 matrices is 2,097,152 of those multiply-accumulates.

A general-purpose processor handles this by fetching operands, multiplying, storing the result, and moving on. The multiplication is trivially cheap. The fetching is not — reading a value from off-chip memory costs orders of magnitude more energy than the multiply it feeds, and takes far longer.

So the real question for any matrix-multiply chip is not how many multipliers it has. It is how many multiplications it gets out of each value it reads.

Passing instead of fetching

A TPU's answer is a systolic array: a grid of small multiply-accumulate cells — 128 wide by 128 tall, so 16,384 of them — wired only to their immediate neighbours.

Weights are loaded into the grid once and stay put. Input values enter from one edge and move across, one cell per clock tick. Partial sums move down. Each cell does one thing per tick: multiply the value arriving from the left by its resident weight, add the partial sum arriving from above, pass the value right and the sum down.

No cell reads memory. A value read once at the edge is used by 128 cells as it crosses.

Count the traffic for that 128 × 128 multiplication:

	values moved
every multiply fetches both operands	4,194,304
systolic array — weights once, inputs once	32,768

That is 128× less memory traffic for identical arithmetic, or 64 multiply-accumulates per value read. Since memory access dominates both the time and the energy, this ratio is essentially the whole story of why the chip is fast and why it runs cool.

The throughput follows almost for free. 16,384 cells each doing one multiply-accumulate per cycle, at around 940 MHz, is 15.4 trillion multiply-accumulates per second.

What the design costs

A pipeline this rigid has a warm-up. Values entering the left edge take 128 ticks to cross, and partial sums take 128 ticks to fall through, so roughly 255 cycles pass before the array is doing full work — and the same at the end as it empties.

For a large multiplication that overhead disappears into the total. For a small one it dominates:

columns streamed through	array utilisation
128	34%
512	67%
1,024	80%
8,192	97%

This is why a TPU wants large batches and large matrices, and why it can look unimpressive on small ones. It is not a latency device. Feed it one small request and most of its 16,384 cells are idle for most of the run.

The array's fixed 128 × 128 shape has the same character. A matrix whose dimensions are not multiples of 128 gets padded, and the padding is real work the chip performs on zeros.

Two further deliberate sacrifices:

Reduced precision. Weights and activations are held in narrow formats rather than 32-bit floats. A smaller multiplier is smaller and cheaper in silicon, and training tolerates the imprecision — one of the few places where the mathematics happens to be forgiving.

No general-purpose machinery. No branch prediction, no out-of-order execution, no large cache hierarchy. The data flow is known in advance, so none of the apparatus for coping with unpredictable access patterns is needed. Remove it and the die area goes to multipliers instead.

Which is also why a TPU cannot run ordinary code. There is nothing there to run it with.

Where it fits

TPUs sit in data centres serving and training large models, where the workload is exactly the shape the chip was built for: enormous matrix multiplications, in batches, with a known structure. Smaller variants appear in devices for on-device inference, with the same trade at a smaller scale.

The limitation is the mirror image of the strength. Any workload with unpredictable control flow, small irregular tensors, or heavy data movement between operations loses most of the advantage — the array spends its cycles filling and draining rather than multiplying.

What to take away

The interesting number in a TPU is not 16,384 multipliers. It is 64 multiply-accumulates per value read.

Everything else about the chip — the fixed grid, the appetite for big batches, the narrow number formats, the inability to run anything else — is the price paid for that ratio. On the workload it was designed for, it is a bargain.