Start by outlining the overall architecture of multi-head self-attention, then implement it step-by-step in code, explaining each operation. Emphasize mask handling and complexity analysis to demonstrate depth.
Pro tip: Mention that you can use a single linear layer for Q, K, V projections by splitting the output, which is more efficient and common in practice. Also, highlight that scaling by sqrt(d_k) is crucial for stable gradients.
Use linear layers to project the input tensor into query, key, and value spaces. Typically, these are combined into one linear layer for efficiency.
Reshape the projected tensors to split the feature dimension into multiple heads, resulting in shape (batch, heads, seq_len, head_dim).
Compute attention scores as Q @ K^T / sqrt(head_dim), apply masks if needed, then softmax and multiply by V.
Concatenate the heads back together and apply a final linear projection to produce the output.
Explain how to implement causal and padding masks, and analyze the O(n^2 * d) time complexity and O(n^2) memory complexity.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.