I got the O(N² · d) time part right away but fumbled a bit explaining why memory is O(N²) specifically.
Start by stating the standard complexities: time O(N^2 * d) and memory O(N^2 + N * d). Then explain the derivation by breaking down the self-attention computation into its matrix operations, highlighting the quadratic dependence on sequence length due to the attention matrix. Finally, discuss practical implications and mention optimizations like sparse attention or linear attention that reduce complexity.
Pro tip: Emphasize that the quadratic complexity is a major bottleneck for long sequences, and mention that many efficient transformer variants aim to reduce this to O(N log N) or O(N). This shows awareness of current research and practical trade-offs.
Clearly state that self-attention has time complexity O(N^2 * d) and memory complexity O(N^2 + N * d) for a sequence of length N and model dimension d.
Explain that computing the attention scores involves multiplying Q (N x d) and K^T (d x N), resulting in an N x N matrix, which takes O(N^2 * d) time. Then applying softmax and multiplying by V (N x d) also takes O(N^2 * d) time.
Note that storing the attention matrix requires O(N^2) memory, and storing the input/output projections and intermediate activations requires O(N * d) memory, leading to overall O(N^2 + N * d) memory.
Mention that the quadratic dependence on N makes self-attention expensive for long sequences, and briefly describe approaches like sparse attention, low-rank approximations, or linear attention that reduce complexity.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.