iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
In a Transformer, attention lets each position build a context-aware representation by comparing a query with keys, turning those match scores into weights, and using the weights to combine values. Queries and keys determine which information matters for this calculation; values provide the information that is mixed into the output.
How query, key, and value fit together
Think of attention as a learned lookup over representations—not a literal database search and not human-like focus. A query represents what one position is looking for in the current representation space. Keys provide the features that the query is compared against. Values carry the content that can contribute to the result.
For each query, the mechanism scores its match with available keys. A stronger score leads to a larger normalized weight, so the corresponding value has more influence in the output. The score itself is not the retrieved content: it determines how much of each value is included.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Fix the driver behind crashes, sound loss and screen glitches3Repair Windows errors before they cause bigger problemsAs Ashish Vaswani and coauthors put it in “Attention Is All You Need” (2017), “An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors.”
What the scaled dot-product formula does
The original Transformer uses scaled dot-product attention:
Attention(Q, K, V) = softmax(QKᵀ / √dₖ)V
Here, Q, K, and V are matrices of queries, keys, and values; Kᵀ is the transpose of the key matrix; and dₖ is the dimension of the keys. For one query, the calculation proceeds as follows:
Rank #2
- Compare query with keys. Dot products in
QKᵀproduce a score for each key. A larger dot product indicates a stronger match in the learned representation space. - Scale the scores. Divide by
√dₖ. The Transformer paper explains that dot products can grow large as the key dimension increases, pushing softmax toward very peaked distributions and producing small gradients. Scaling moderates the scores before normalization. - Turn scores into weights. Softmax converts the scaled scores into normalized weights. For a given query, the weights sum to one.
- Combine values. Multiply the weights by the value vectors and sum them. The result is the attention output for that query: a weighted combination of the available values.
In compact terms, query-key comparisons decide the mixture; the values supply what is mixed. The original paper discusses additive attention, which computes scores with a learned feed-forward scoring function, as another approach. Its Transformer uses scaled dot-product attention instead. The paper’s method and explanation provide the original comparison.
Self-attention and cross-attention use different inputs
| Mechanism | Where queries come from | Where keys and values come from | What it lets the model do |
|---|---|---|---|
| Self-attention | The sequence representation being processed | The same sequence representation | Let positions incorporate information from other positions in that sequence |
| Cross-attention | One representation | A separate representation | Let positions in one representation draw information from another |
In self-attention, the query, key, and value vectors are learned projections of the same sequence representation. In the original Transformer, encoder layers use self-attention, while decoder encoder-attention (often called cross-attention) uses decoder representations to query encoder outputs. The 2017 paper describes both arrangements.
Rank #3
Why use multiple attention heads?
A single head performs attention in one learned projection space. Multi-head attention applies several learned projections and attention operations in parallel, then combines their outputs. Vaswani and coauthors’ stated motivation is to let the model attend to information from different representation subspaces at different positions. Heads are therefore parallel learned views of the representations, not a set of manually assigned roles. For the paper’s formulation, see the original Transformer description.
What attention does not tell you on its own
Attention connections alone do not encode sequence order. The original Transformer adds positional encodings to token embeddings so the model can use position information alongside attention. Without that distinction, it is easy to mistake the pattern of connections for an order-aware representation.
Attention weights show how a particular computation distributes weight across values, but they are not, by themselves, a complete or faithful account of everything a model has learned or why it produced an answer. They describe one part of the computation, not a full explanation of the model’s reasoning.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallWhat the original result means—and does not mean
The 2017 paper reported 28.4 BLEU for the Transformer on the WMT 2014 English-to-German translation benchmark, more than 2 BLEU above the existing best results it cited, including ensembles. This is a result reported for that specific historical benchmark in the paper, not a current leaderboard figure. The result is documented by Google Research.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
