Transformers Explained: How AI Reads Everything at Once

The 2017 paper behind the T in ChatGPT — and the attention trick that made modern AI possible.

In June 2017, eight researchers published a fifteen-page paper with a deliberately provocative title: Attention Is All You Need. Their claim was that you could throw away the two workhorses of sequence modeling — recurrence and convolution — and build a translation system on attention alone. It worked embarrassingly well: their model beat the best existing systems on English-to-German and English-to-French translation, training in three and a half days on eight GPUs, a fraction of what rivals needed. The architecture they introduced was the transformer. The T in ChatGPT stands for it, and so does the T in BERT. Nearly every capable AI system you have used since runs on it.

What a transformer actually is

A transformer is a family of neural network architectures built around one mechanism: multi-head self-attention. Text goes in as tokens, each token becomes a vector of numbers through an embedding layer, and then each layer lets every token gather context from the other tokens around it — amplifying the signal of the important ones, diminishing the rest. That gathering happens in parallel, across many attention “heads” at once.

The family comes in three main flavors. Encoder-only models (like BERT) read text to understand it — good for classification and search. Decoder-only models (like GPT) generate text one token after another — the chatbots and writing tools you know. Encoder-decoder models (the original 2017 design) read something in and write something out — the classic translation setup. And transformers long ago escaped text: vision transformers read images as sequences of patches, and the same architecture now shows up in audio, robotics, and multimodal systems. The T travels well.

If the surrounding vocabulary is new, two companion pieces help: what an LLM actually is and where deep learning sits inside AI.

Why everything before it was too slow

Before 2017, the standard way to handle sequences was the recurrent neural network. An RNN reads word one, updates its internal state, reads word two, updates again — strictly in order. That creates two problems. First, nothing parallelizes: word fifty cannot be processed until words one through forty-nine are done, so training on billions of words takes forever. Second, information decays: by the time the network reaches the end of a long paragraph, the beginning has faded into a blur, like a message passed down a long line of whisperers.

The transformer’s answer was to stop pretending sequence had to mean sequence processing. Every token gets a direct line to every other token, so the distance between “cat” and a pronoun fifty words later is one step, not fifty. And because no token waits for another, the whole computation parallelizes across modern hardware. That is the entire bet of the 2017 paper, stated in its abstract: superior quality, more parallelizable, significantly less time to train (the paper on arXiv; the published NeurIPS version).

Attention, worked by hand

Attention sounds mystical until you see the arithmetic. Here is a miniature version, worked out step by step. Imagine the model is processing one word and asks a question of the sentence: which other words matter most to me? Three candidate words answer, each as a small vector. (Real vectors have hundreds of dimensions; we use two so the arithmetic stays visible. Every number below is our own calculation.)

Step 1 — score by similarity. The processing word carries a query vector q = [1, 0]. Each candidate carries a key vector: k1 = [1, 0], k2 = [0, 1], k3 = [1, 1]. The score for each candidate is the dot product — a similarity measure. High score means “you look relevant to what I’m asking about.”

s1 = 1×1 + 0×0 = 1    s2 = 1×0 + 0×1 = 0    s3 = 1×1 + 0×1 = 1

Step 2 — scale. Divide each score by √d, where d is the vector dimension (here d = 2, so √2 ≈ 1.414). This is the step people skip when explaining, and it matters: in high dimensions, raw dot products grow large, and large inputs push the next step — the softmax — into a region where one candidate grabs nearly 100% and the rest get zero, killing the model’s ability to learn fine distinctions. Scaling keeps the voting sensitive.

0.707    0    0.707

Step 3 — softmax into weights. Exponentiate and normalize so the weights sum to 1. Now it is a proper vote:

w1 ≈ 0.40    w2 ≈ 0.20    w3 ≈ 0.40

Step 4 — mix the values. Each candidate also carries a value vector — the actual content it contributes: v1 = [4, 0], v2 = [0, 4], v3 = [2, 2]. The output is the weighted mix:

output = 0.40×[4,0] + 0.20×[0,4] + 0.40×[2,2] ≈ [2.41, 1.59]

That is the whole trick. The query says what the word is looking for, the keys say what each candidate offers, the dot product scores the match, and the values get blended by the resulting weights. Run this for every word simultaneously, with eight or more independent “heads” asking different questions — one head might track grammar, another might track who-did-what-to-whom — and you have multi-head self-attention. The original paper used eight heads, and the multi-head design is one of its most-copied ideas (Wikipedia’s transformer overview; a visual walkthrough of the architecture, secondary).

Three design decisions the paper got right

Positions have to be injected. Attention, left alone, is a set operation: it does not know or care about word order, so “dog bites man” and “man bites dog” would look identical. The paper adds positional encodings to each token’s vector so order information survives. It is a small detail that prevents total nonsense.

One head is not enough. A single attention pattern can only ask one kind of question. With multiple heads running in parallel, different heads specialize: syntax in one, coreference in another, long-range topic links in a third. The model learns the division of labor itself during training.

Attention is only half the layer. After attention mixes information across positions, each position passes through a small feed-forward network that processes it independently. Mix, then think; mix, then think — stacked dozens of times. That alternating rhythm is the transformer block, and everything since 2017 has been variations on it.

The real price: attention is n-squared

Now the honest part. Every word attending to every other word means the attention scores form an n-by-n matrix, where n is the sequence length. Double the context, quadruple the scores. At the 128k-token contexts that are routine in 2026, that matrix is enormous — and this, more than anything, is why long context costs what it costs. Try it:

Where transformers won — and where the story gets complicated

The wins are historic and deserve plain stating. Transformers made training on internet-scale text practical, which made foundation models possible, which made generative AI — the GPT in everything — possible. Hugging Face’s transformers library, now the standard toolkit with hundreds of thousands of pretrained models on its hub, exists because one architecture proved general enough to be worth standardizing (the library docs).

But three complications belong in any honest account. First, the quadratic cost is real. The n-squared attention matrix is why context windows were tiny for years and why “long context” remains a premium feature. An entire subfield — sparse attention, FlashAttention, sliding windows — exists to dodge a bill the architecture itself creates.

Second, transformers are data-hungry. Attention learns relationships from scratch rather than assuming them, which is powerful but means the architecture needs enormous training data to shine. That is part of why the biggest models belong to the best-resourced labs, and part of why efficiency research like mixture-of-experts designs matters: it attacks the cost side the transformer opened up.

Third, the transformer is probably not the final answer. State-space models and other challengers nibble at the quadratic cost directly, and nobody serious claims attention is the last word in sequence modeling. What the 2017 paper proved was narrower and more durable: that attention alone, parallelized, could beat recurrence. Everything built since is commentary on that result.

Our stance, stated as editorial judgment rather than anyone’s documentation: if you build with AI, understanding attention repays the effort, because every cost, latency, and context-window limit you will ever hit traces back to that n-by-n matrix. If you only use AI, the one-paragraph version at the top is genuinely enough — and anyone telling you otherwise is selling a course.

Frequently asked questions

What is a transformer in AI, simply?

It is the neural network architecture behind modern AI: instead of reading text one word at a time, it processes the whole sequence at once, with every word attending to every other word to gather context. The T in ChatGPT and BERT stands for transformer.

How does the attention mechanism work?

Each word issues a query (what it is looking for); every other word offers a key (what it provides) and a value (its content). Dot products score how well each key matches the query, the scores are scaled and turned into weights summing to one, and the values are blended by those weights. The worked example above shows the full arithmetic.

Why are transformers better than RNNs?

Two reasons: parallelism and direct long-range connections. RNNs process words strictly in order, so training cannot be parallelized and early information fades. Transformers process all positions at once and give every word a direct one-step connection to every other word.

What is the difference between a transformer and an LLM?

A transformer is an architecture — a blueprint. An LLM is a large language model built, in nearly all modern cases, using that blueprint and trained on massive text. Saying “LLM” when you mean “transformer” is like saying “skyscraper” when you mean “steel frame.”

Are transformers only used for text?

No. The same architecture handles images (vision transformers read an image as a sequence of patches), audio, robotics control, and multimodal combinations. Anything that can be turned into a sequence is fair game.

Why is it called “attention”?

Because the mechanism decides, for each word, which other words deserve focus — amplifying the relevant ones and diminishing the rest, loosely inspired by how human attention filters a noisy room. The name is an analogy, not a claim that the model experiences anything.