Encoder vs Decoder
/AI/4 min read
One mask setting separates the two, and it decides two things that look unrelated: how much training signal a corpus yields, and whether generation can be made incremental. Both favour the decoder by a wide margin.
A transformer processes a sequence of tokens with attention: every position looks at other positions and gathers what it needs from them.
The only structural difference between an encoder and a decoder is which positions each one is allowed to look at. An encoder sees the whole sequence. A decoder sees only what came before.
That single restriction has consequences far beyond what it sounds like.
Why seeing both sides helps
Take a sentence where a word is ambiguous until later:
She left the plant by the window.
She left the plant at six.
The first plant is a pot of leaves; the second is a factory. When the model reaches that word it has no way to know which — the disambiguating information arrives afterwards.
An encoder reads bidirectionally, so the representation it builds for plant incorporates window or at six. It is not predicting anything; it is producing a representation of text that already exists, and there is no reason to withhold part of it.
This is why encoders suit classification, retrieval, extraction, tagging — anything where the input is complete and the job is to characterise it.
Why a decoder cannot do that
A decoder's job is to produce the next token. If it could see the next token while predicting it, the task would be trivial and it would learn nothing.
So a causal mask blocks every position from attending to anything after itself. Position 7 sees positions 1 to 7 and nothing else.
That is the whole distinction. Same layers, same attention, one mask setting — and everything else follows from it.
The first consequence: training signal
This is the part that explains which architecture won, and it is arithmetic.
An encoder is trained by hiding some tokens and asking it to recover them. Hide too many and there is not enough context left; the conventional figure is around 15%. So only 15% of positions produce a learning signal; the other 85% are context.
A decoder predicts every position from its prefix. Every token is simultaneously an input and a target, so 100% of positions produce a signal.
| prediction targets per token of corpus | over a 1-trillion-token corpus | |
|---|---|---|
| masked encoder | 0.15 | 150 billion |
| causal decoder | 1.00 | 1,000 billion |
6.67× more learning from the same text. When the binding constraint on model quality is how much data you can get through, an architecture that extracts nearly seven times the signal per token is not slightly better — it is on a different scaling curve.
The second consequence: generation can be incremental
A causal mask guarantees that a token's representation depends only on tokens before it. So once computed, it never changes.
That is precisely what makes the attention cache possible. Generate a token, keep its keys and values, and the next step only has to process the one new position.
Without that guarantee, generating 500 tokens means reprocessing the whole prefix at every step:
| token-positions processed to generate 500 tokens | |
|---|---|
| recompute the prefix each step | 125,250 |
| reuse past keys and values | 500 |
A 250× difference, and it is not an optimization you could add to a bidirectional model. In an encoder, every new token changes every earlier representation, so there is nothing stable to cache. Bidirectionality and incremental generation are mutually exclusive.
The three shapes
Encoder-only. Bidirectional, no generation. Good at understanding a fixed input — classification, retrieval, extraction. Still the right choice when the output is a label or a vector rather than text.
Decoder-only. Causal, generates. What nearly all current chat and coding models are.
Encoder-decoder. An encoder reads the input bidirectionally, a decoder writes the output while attending to that reading. Natural for transforming one text into another — translation, summarisation — where the input is complete and the output is new.
The interesting question is why decoder-only models took over tasks the other two were designed for. Partly the training-signal arithmetic above. Partly that a model good enough at predicting text turns out to be able to describe a classification in words, so the separate architecture stopped being necessary — one model, prompted differently, covers what previously needed three.
What to take away
The difference is one line of masking, and it cascades.
Blocking the future costs an encoder's bidirectional view of ambiguous words. It buys a learning signal on every token instead of one in seven, and a representation stable enough to cache — which are the two things that decided the field.