This is one of those questions where you think you know it and then halfway through your explanation you realize you're not sure you're saying anything precise.
Start by explaining the intuition behind self-attention as a mechanism for computing context-aware representations, then walk through the mathematical derivation of Q, K, V and the scaled dot-product attention formula. Finally, describe how multi-head attention extends this and contrast with RNNs to highlight advantages for long-range dependencies.
Pro tip: Emphasize the parallelizability and constant path length between any two positions in self-attention, which directly addresses the vanishing gradient problem in RNNs. Mention that at Netflix, this enables efficient processing of long sequences like user viewing histories.
Explain that self-attention allows each position in a sequence to attend to all other positions, capturing dependencies regardless of distance. Contrast with RNNs which process sequentially and struggle with long-range dependencies.
Describe how for each input token embedding, we compute Query, Key, and Value vectors by multiplying with learned weight matrices W_Q, W_K, W_V. These projections allow the model to learn different roles for each token.
Present the scaled dot-product attention: Attention(Q, K, V) = softmax(QK^T / sqrt(d_k)) V. Explain each component: QK^T computes similarity scores, scaling by sqrt(d_k) prevents softmax saturation, softmax normalizes to attention weights, and multiplication with V produces weighted sum.
Explain that multi-head attention runs multiple attention operations in parallel with different learned projections, allowing the model to attend to different representation subspaces. The outputs are concatenated and linearly transformed.
Highlight that self-attention has O(1) path length between any two positions, enabling direct gradient flow and better long-range dependency modeling. Also, it is highly parallelizable, unlike sequential RNNs.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.