← Adobe Interview InsightsMachine Learning Engineer

Adobe·Machine Learning Engineer·Technical Phone Screen·Senior

Senior
Jun 2026

Summary

Adobe ML engineer interview that went deep into transformer internals, specifically Multi-Head Attention. The whole session felt like a graduate seminar on attention mechanics, which I was not fully prepared for.

Questions Asked (6)

Q1

Walk through the full operation order inside Multi-Head Attention, from input to output.

System DesignTechnical Trade-offs
Author's notes

I got the broad strokes right but fumbled the exact sequencing of the reshape steps relative to the projection.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by framing the purpose of multi-head attention—capturing diverse relationships in parallel—then walk through the tensor operations step by step, from input projection to output projection. Use a concrete example (e.g., batch size, sequence length, model dimension) to ground the explanation and highlight trade-offs like computational cost and expressiveness.

Pro tip: Emphasize that multi-head attention is not just parallel attention but a learned projection of queries, keys, and values into subspaces, and mention how this enables the model to attend to different representation subspaces. Also, note that the output projection after concatenation is crucial for mixing information across heads.

1. Input and Linear Projections

Describe the input tensor shape (batch_size, seq_len, d_model) and how it is linearly projected into queries, keys, and values using learned weight matrices. Mention that for multi-head, these projections are split into h heads, each with dimension d_k = d_v = d_model / h.

2. Scaled Dot-Product Attention per Head

Explain that for each head, compute attention scores as QK^T / sqrt(d_k), apply softmax to get attention weights, and then multiply by V to get the head output. Highlight that this is done in parallel for all heads.

3. Concatenation and Output Projection

Describe how the outputs from all heads are concatenated along the feature dimension to form a tensor of shape (batch_size, seq_len, d_model). Then apply a final linear projection with weight matrix W^O to produce the final output.

4. Complexity and Trade-offs

Briefly discuss computational complexity (O(n^2 d) for sequence length n and model dimension d) and how multi-head attention increases representational power at the cost of more parameters and computation. Mention that the number of heads is a hyperparameter.

Key Points to Mention

  • Input tensor shape and linear projections for Q, K, V
  • Splitting into multiple heads and parallel computation
  • Scaled dot-product attention formula: softmax(QK^T / sqrt(d_k))V
  • Concatenation of head outputs and final linear projection
  • Computational complexity and trade-offs (e.g., number of heads, d_k)
  • Role of multi-head attention in capturing diverse relationships

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q2

What are the Q, K, and V projections doing, and how does tensor reshaping fit into that process?

System DesignTechnical Trade-offs
Author's notes

Knew what Q/K/V were conceptually but got a bit tangled explaining the reshape.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by explaining the purpose of Q, K, and V projections in attention: they transform input embeddings into query, key, and value spaces to compute attention weights and weighted sums. Then describe how tensor reshaping (e.g., splitting into heads) enables parallel multi-head attention, and how it fits into the overall process. Emphasize the mathematical operations and practical implications for model performance.

Pro tip: Relate the projections and reshaping to real-world trade-offs like computational efficiency and model expressiveness, showing you understand both theory and implementation. Mention that reshaping is not just a technical detail but a design choice that impacts parallelism and memory access patterns.

1. Define Q, K, V projections

Explain that Q, K, and V are linear projections of the input, each serving a distinct role: queries represent the current token's information need, keys represent the context tokens' identities, and values carry the actual content to be aggregated.

2. Describe attention computation

Outline how Q and K are used to compute attention scores (e.g., dot product), which are scaled and softmaxed to produce weights, and then used to take a weighted sum of V.

3. Explain multi-head attention and reshaping

Detail how the projected tensors are reshaped to split the model dimension into multiple heads, enabling parallel attention computations. Mention that reshaping involves operations like view/reshape and transpose to align dimensions for batched matrix multiplication.

4. Connect reshaping to efficiency and design

Discuss how reshaping facilitates efficient parallel computation on GPUs and allows the model to attend to information from different representation subspaces, enhancing expressiveness.

5. Summarize the overall flow

Conclude by tying together the projections, attention, and reshaping as a cohesive pipeline that transforms input into context-aware representations.

Key Points to Mention

  • Linear projections: Q = XW_Q, K = XW_K, V = XW_V, where X is input and W are learned weight matrices.
  • Attention scores: softmax(QK^T / sqrt(d_k))V, with scaling for stable gradients.
  • Multi-head attention: splitting into h heads allows attending to different subspaces.
  • Reshaping operations: view/reshape to (batch, seq_len, num_heads, head_dim) and transpose to (batch, num_heads, seq_len, head_dim).
  • Computational benefits: parallelization across heads, efficient batched matrix multiplication.
  • Trade-offs: more heads increase expressiveness but also computational cost; reshaping affects memory layout and speed.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q3

How is the attention score computed and what role does softmax play?

Algorithms & Data StructuresTechnical Trade-offs
Author's notes

Straightforward if you've implemented this before.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining the attention mechanism and the formula for attention scores, then explain how softmax converts raw scores into a probability distribution. Emphasize the role of softmax in normalizing scores and enabling differentiable, weighted aggregation of values.

Pro tip: Mention that softmax's temperature parameter (often sqrt(d_k)) controls the sharpness of the distribution, and that without scaling, gradients can vanish. This shows practical awareness of implementation details.

1. Define Attention Scores

Explain that attention scores are computed as the dot product between the query and key vectors, optionally scaled by 1/sqrt(d_k) to prevent large magnitudes.

2. Apply Softmax

Describe how softmax is applied to the scores to convert them into a probability distribution over the keys, ensuring all weights are positive and sum to 1.

3. Weighted Sum of Values

Show that the final attention output is a weighted sum of the value vectors, where the weights are the softmax probabilities.

4. Role of Softmax

Highlight that softmax introduces non-linearity, ensures differentiability, and allows the model to focus on relevant parts by amplifying high scores and suppressing low ones.

5. Practical Considerations

Mention scaling, numerical stability (e.g., subtracting max before exponentiation), and the effect of temperature on the distribution's entropy.

Key Points to Mention

  • Attention score formula: QK^T / sqrt(d_k)
  • Softmax normalizes scores into probabilities
  • Scaling by sqrt(d_k) prevents softmax saturation
  • Softmax enables differentiable attention weights
  • Weighted sum of values produces context vector
  • Temperature parameter controls sharpness of distribution

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q4

Why does Multi-Head Attention use 4 linear projections rather than fewer?

System DesignTechnical Trade-offs
Author's notes

This tripped me up more than it should have.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying that the 4 projections (Q, K, V, and output) are fundamental to the attention mechanism, not arbitrary. Explain the distinct role of each projection and how they enable flexible, expressive attention. Then discuss the trade-offs of reducing projections, focusing on model capacity and performance.

Pro tip: Emphasize that the projections are learned and allow the model to attend to different representation subspaces, which is crucial for capturing complex patterns. Mention that reducing projections would limit this diversity and likely hurt performance, especially on tasks requiring nuanced understanding.

1. Define the four projections

Briefly describe each projection: Query (Q), Key (K), Value (V), and Output (O). Explain that Q and K compute attention scores, V provides the content to aggregate, and O projects the concatenated multi-head output.

2. Explain the necessity of separate Q, K, V

Discuss why Q, K, and V cannot be the same: they serve different purposes. Q and K determine where to attend, while V determines what to extract. Using the same projection would force a single representation to serve both roles, reducing flexibility.

3. Discuss the output projection

Explain that the output projection combines information from multiple heads and projects it back to the model dimension. Without it, the concatenated heads might not integrate well with the rest of the network.

4. Analyze trade-offs of fewer projections

Consider what happens if we reduce projections: e.g., sharing Q and K might simplify but limit the model's ability to learn distinct query and key representations. Fewer projections reduce parameters and computation but may sacrifice expressiveness and performance.

5. Conclude with practical implications

Summarize that the 4 projections are a deliberate design choice balancing expressiveness and efficiency. In practice, they are essential for state-of-the-art performance, and alternatives like sharing projections are rarely beneficial.

Key Points to Mention

  • Query, Key, Value roles: Q and K for compatibility, V for content.
  • Multi-head attention: each head learns different projections, enhancing representation.
  • Output projection integrates heads and matches model dimension.
  • Trade-offs: fewer projections reduce parameters but may harm performance.
  • Empirical evidence: standard Transformer uses 4 projections; alternatives underperform.
  • Flexibility: separate projections allow model to learn distinct subspaces.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q5

Can you estimate the FLOPs involved in self-attention?

Algorithms & Data StructuresTechnical Trade-offs
Author's notes

Did not prep for this at all and it showed.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining the self-attention operation and its key dimensions (sequence length n, model dimension d). Then derive the FLOPs for each matrix multiplication step, assuming standard dense operations, and sum them to get the total. Finally, discuss the quadratic dependence on sequence length and its implications.

Pro tip: Mention that in practice, FLOPs are only part of the story—memory bandwidth and parallelism often dominate, so you should also consider the arithmetic intensity. This shows you understand real-world performance beyond theoretical counts.

1. Define variables and assumptions

Let n be the sequence length and d be the model dimension. Assume single-head attention and ignore batch size for simplicity. State that you'll count multiply-adds as 2 FLOPs.

2. Break down the computation

Self-attention involves: (1) computing Q, K, V via linear projections, (2) computing attention scores QK^T, (3) applying softmax, (4) computing weighted sum with V.

3. Calculate FLOPs per step

For each step: Q,K,V projections: 3 * (2 * n * d * d) = 6 n d^2. QK^T: 2 n^2 d. Softmax: ~5 n^2 (negligible). Weighted sum: 2 n^2 d. Total: 6 n d^2 + 4 n^2 d.

4. Sum and simplify

Total FLOPs ≈ 6 n d^2 + 4 n^2 d. For large n, the n^2 term dominates, so it's O(n^2 d). Mention that multi-head attention multiplies this by the number of heads but with reduced dimension per head, so total remains similar.

5. Discuss implications

Highlight that the quadratic dependence on n makes self-attention expensive for long sequences, motivating efficient variants like sparse or linear attention.

Key Points to Mention

  • Quadratic complexity in sequence length (n^2)
  • Linear projection FLOPs: 6 n d^2
  • Attention score and weighted sum FLOPs: 4 n^2 d
  • Multi-head attention: total FLOPs similar to single-head with same total dimension
  • Softmax FLOPs are negligible compared to matrix multiplications
  • Practical considerations: memory bandwidth and parallelism often matter more than raw FLOPs

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q6

Why is the computational complexity of self-attention O(S^2) with respect to sequence length?

Algorithms & Data StructuresSystem Design
Author's notes

Recovered here after the FLOPs stumble.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining self-attention and its core operation: computing pairwise interactions between all positions in a sequence. Then, derive the O(S^2) complexity by analyzing the matrix multiplications involved, specifically the S×S attention score matrix. Conclude by contrasting with other sequence models and mentioning practical implications for long sequences.

Pro tip: Mention that while the time complexity is O(S^2), the memory complexity is also O(S^2) due to storing the attention matrix, which is a key bottleneck in practice. This shows awareness of real-world constraints beyond theoretical analysis.

1. Define self-attention

Explain that self-attention computes a weighted sum of all positions in a sequence, where weights are based on pairwise similarity between queries and keys.

2. Identify the core operation

Highlight that the attention scores are computed as the dot product of queries and keys, resulting in an S×S matrix where S is the sequence length.

3. Analyze computational steps

Break down the computation: (1) QK^T multiplication takes O(S^2 * d) time, (2) softmax over S^2 elements takes O(S^2), (3) weighted sum with values takes O(S^2 * d). Since d is typically fixed, the dominant term is O(S^2).

4. Discuss implications

Mention that this quadratic scaling makes self-attention expensive for long sequences, motivating efficient variants like sparse or linear attention.

Key Points to Mention

  • Self-attention computes pairwise interactions between all positions, leading to S^2 pairs.
  • The attention score matrix has dimensions S×S, requiring O(S^2) memory and computation.
  • Matrix multiplication QK^T involves S^2 dot products of dimension d, resulting in O(S^2 d) time.
  • Softmax and weighted sum operations also scale with S^2.
  • Typically, d (head dimension) is much smaller than S, so complexity is simplified to O(S^2).
  • This quadratic complexity is a bottleneck for long sequences, unlike RNNs which are O(S) but sequential.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.