In natural language processing and other sequence-modeling tasks, the Transformer has become a research focus thanks to its strong performance. Multi-head self-attention is one of the Transformer’s core components. What follows looks closely at how it works and how it is implemented.
What Is Self-Attention
The self-attention mechanism lets the model, while encoding a sequence, gather information from the other positions for the representation at each position and update that representation from this information. Unlike recurrent networks such as RNNs and LSTMs, which capture dependence along a sequence step by step, self-attention models dependence between positions directly through attention scores. Modeling long-range dependence directly in this way greatly increases the model’s expressive power.
How Multi-Head Attention Is Computed
Multi-head self-attention takes the attention mechanism a step further by using several independent “attention heads” to capture sequence information from different representation subspaces. The computation is as follows.
Linear projection
The input sequence \(X\) has shape \((N, T, d)\), where \(N\) is the batch size, \(T\) is the sequence length, and \(d\) is the feature dimension. We first project \(X\) into the query, key, and value spaces with a linear transformation \(W_{QKV}\), producing a representation of shape \((N, T, 3d)\). We then split it along the last dimension into \(Q\), \(K\), and \(V\), each of shape \((N, T, d)\).
Splitting into heads (multi-head)
Split \(Q\), \(K\), and \(V\) along the last dimension into \(h\) heads, each of dimension \(d/h\). The shape of \(Q\), \(K\), and \(V\) is then \((N, T, h, d/h)\). Next, swap axes 1 and 2 of \(Q\), \(K\), and \(V\), so the shape becomes \((N, h, T, d/h)\), which makes the later computation more convenient.
Scaled dot-product attention
For each head \(i\), take the dot product of \(Q_i\) and \(K_i\), divide by \(\sqrt{d/h}\) to scale it, add the mask matrix \(M\) to handle padded positions, and then apply the softmax function over the last two dimensions to obtain the attention-score matrix:
The score matrix has shape \((N, h, T, T)\).
Weighted sum
Multiply the attention scores by \(V_i\), with the matrix multiplication carried out on the last two dimensions, to obtain the weighted representation of each head:
This is a tensor of shape \((N, h, T, d/h)\).
Concatenating the heads
Concatenate the weighted representations of the \(h\) heads along the last dimension. The shape becomes \((N, h, T, d)\).
Linear transformation
An additional linear layer \(W_o\) maps the concatenated representation back to a \(d\)-dimensional space, producing the final multi-head self-attention output of shape \((N, T, d)\).
A few of the key matrix operations above can be checked for consistency by looking at their shapes:
Multiplying \(Q_i\) by \(K_i^T\): \((N, h, T, d/h) \times (N, h, d/h, T) \rightarrow (N, h, T, T)\)
Multiplying \(\text{Attention}_i\) by \(V_i\): \((N, h, T, T) \times (N, h, T, d/h) \rightarrow (N, h, T, d/h)\)
Merging the heads: \((N, h, T, d/h) \rightarrow (N, T, h, d/h) \rightarrow (N, T, d)\)
By splitting the work into several parallel subspaces, multi-head attention can take sequence information from different representation subspaces and fuse it, which increases the model’s expressive power. At the same time, an efficient implementation built on matrix operations lets multi-head self-attention encode the sequence in parallel, a kind of parallelism that recurrent neural networks cannot match.
Implementation Details
The previous section described how multi-head self-attention is computed from a theoretical point of view. Now let us look at a NumPy implementation and check it against PyTorch’s official implementation.
| |
This code implements the multi-head self-attention computation described above. A few points are worth noting:
After the linear projection produces Q, K, and V, a list comprehension reshapes them and swaps axes. This is concise and efficient.
When the attention scores are computed, the dot product of Q and K is taken first, then scaled by dividing by \(\sqrt{d/h}\), and finally the mask matrix is added as a soft constraint.
For the weighted sum, the attention scores are multiplied by V first and then by \(W_o\), which keeps the shapes matched.
Compute the maximum of \(Z\) along the last dimension (
axis=-1) and keep that dimension (keepdims=True). This produces a new array of the same shape in which every value along the last dimension equals the maximum of the corresponding slice of the original array. Subtract this array of maxima from the corresponding elements of the original array \(Z\), which ensures that every element is no greater than 1.
To verify that this implementation is correct, we can compare it with PyTorch’s official implementation:
| |
This code builds a simple multi-head self-attention example and compares the result with PyTorch’s built-in nn.MultiheadAttention module. The difference between the two outputs is close to 0, which shows that our NumPy implementation is correct.
With the implementation and the numerical check above, you should now have a more concrete and thorough understanding of multi-head self-attention. This self-attention mechanism is the core of the Transformer and of other sequence models. Mastering it helps us explore and improve these models further.
