← Meta Interview InsightsSoftware Engineer

Meta·Software Engineer·Technical Phone Screen·Senior

SeniorPrefer not to say
Jun 2026Remote

Summary

Meta research engineer screen, focused entirely on implementing self-attention from scratch. One question, pretty deep, and they wanted both the math and working code before moving on.

Questions Asked (1)

Q1

Implement scaled dot-product self-attention from scratch using NumPy or PyTorch. Given query, key, and value tensors of shape (batch, seq_len, d_k), compute the attention output using the formula softmax(QK^T / sqrt(d_k)) * V. Walk through the math first, then write the code. Optionally handle a causal mask.

System DesignTechnical Trade-offsAlgorithms & Data Structures
Author's notes

They asked me to explain the math before touching any code, which I actually appreciated because it slowed me down enough to not make the classic mistake of forgetting the sqrt(d_k) scaling.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by explaining the intuition behind scaled dot-product attention: queries and keys compute similarity scores, which are scaled to prevent softmax saturation, then used to weight values. Then implement it step-by-step in code, handling the causal mask by setting future positions to -inf before softmax. Finally, discuss trade-offs and potential optimizations.

Pro tip: Mention that scaling by sqrt(d_k) is crucial for stable gradients, and demonstrate awareness of numerical stability by subtracting the max before softmax. Also, note that in practice, efficient implementations use fused kernels like FlashAttention.

1. Explain the math

Walk through the formula: compute QK^T, scale by 1/sqrt(d_k), apply softmax over the last dimension, then multiply by V. Emphasize the shapes and the purpose of each step.

2. Implement core attention

Write code to compute scores = Q @ K.transpose(-2, -1) / sqrt(d_k), apply softmax, and return the weighted sum with V. Use NumPy or PyTorch as preferred.

3. Add causal mask (optional)

Create a lower-triangular mask, set masked positions to -inf before softmax, and ensure the mask broadcasts correctly across batch and head dimensions.

4. Test and validate

Test with small random tensors, check output shapes, and verify that masked positions have zero attention weight. Compare against a reference implementation if possible.

5. Discuss trade-offs and optimizations

Talk about computational complexity (O(n^2 d)), memory usage, and alternatives like FlashAttention or sparse attention for long sequences.

Key Points to Mention

  • Scaling factor 1/sqrt(d_k) prevents softmax from saturating and stabilizes gradients.
  • Softmax is applied along the last dimension (keys) to get attention weights.
  • Causal mask ensures autoregressive property by setting future positions to -inf before softmax.
  • Numerical stability: subtract max before exponentiation in softmax.
  • Computational complexity: O(n^2 d) time and O(n^2) memory for attention matrix.
  • Efficient implementations like FlashAttention use tiling and recomputation to reduce memory.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.