← Adobe Interview InsightsMachine Learning Engineer

Adobe·Machine Learning Engineer·Technical Phone Screen·Senior

Senior
Apr 2026

Summary

Adobe ML engineer interview that went deep into attention mechanism internals. The whole session felt like a graduate seminar on transformer efficiency, which I was not fully prepared for.

Questions Asked (7)

Q1

Walk me through how FlashAttention works under the hood, including how it uses tiling and SRAM to avoid materializing the full attention matrix in HBM.

System DesignTechnical Trade-offs
Author's notes

This is where I started sweating.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by contrasting standard attention's memory bottleneck with FlashAttention's IO-aware design, then explain the tiling and online softmax mechanics that keep computation in SRAM. Emphasize how this reduces HBM traffic and enables exact attention without materializing the full matrix.

Pro tip: Mention that FlashAttention is not an approximation—it computes exact attention—and highlight the backward pass recomputation trade-off to show depth beyond the forward pass.

1. Set the context: standard attention's memory bottleneck

Explain that standard attention computes S = QK^T and P = softmax(S), materializing N×N matrices in HBM, which is slow and memory-heavy for long sequences.

2. Introduce tiling and SRAM

Describe how FlashAttention splits Q, K, V into blocks that fit in on-chip SRAM, loading them tile by tile to avoid writing large intermediate matrices to HBM.

3. Explain online softmax and block-wise computation

Detail how the algorithm computes attention incrementally using running statistics (max and sum) to rescale partial results, enabling exact softmax without full-row access.

4. Highlight the backward pass and recomputation

Note that FlashAttention recomputes attention during backpropagation instead of storing the full matrix, trading extra compute for reduced memory and HBM traffic.

5. Summarize benefits and trade-offs

Conclude with speed and memory improvements, exactness, and mention limitations like head dimension constraints or implementation complexity.

Key Points to Mention

  • HBM vs SRAM bandwidth and capacity differences
  • Tiling: blocking Q, K, V into chunks that fit in SRAM
  • Online softmax: incremental computation with running max and sum
  • Avoiding materialization of the N×N attention matrix in HBM
  • Backward pass recomputation to save memory
  • Exactness: FlashAttention is not an approximation

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q2

How does online softmax work and why is it necessary for FlashAttention's tiling approach?

Algorithms & Data StructuresTechnical Trade-offs
Author's notes

Blanked on the name 'online softmax' for a second, even though I knew the concept.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by explaining the standard softmax and its memory bottleneck in attention, then introduce online softmax as a streaming algorithm that computes softmax in one pass with constant memory. Finally, connect it to FlashAttention's tiling by showing how each tile updates running statistics (max and sum) to avoid storing the full attention matrix.

Pro tip: Emphasize that online softmax is not just an optimization but a fundamental enabler for tiling, and mention that it maintains numerical stability by rescaling previous accumulations when a new maximum is encountered.

1. Define standard softmax and its limitations

Explain that softmax requires computing exponentials of all scores and normalizing by their sum, which needs the full row in memory. In attention, this means storing the entire N×N attention matrix, causing quadratic memory usage.

2. Introduce online softmax as a streaming algorithm

Describe how online softmax processes elements one at a time, maintaining a running maximum and a running sum of exponentials. When a new maximum is found, it rescales the previous sum and any accumulated weighted values to maintain numerical stability.

3. Connect to FlashAttention's tiling approach

Explain that FlashAttention tiles the query and key matrices into blocks that fit in SRAM. For each query block, it iterates over key blocks, using online softmax to update the output incrementally without materializing the full attention matrix.

4. Highlight the benefits and trade-offs

Discuss how this reduces memory from O(N^2) to O(N) and enables efficient computation on GPUs by keeping data in fast memory. Mention that it may introduce extra floating-point operations but is overall faster due to memory savings.

Key Points to Mention

  • Standard softmax requires the full row of scores to compute the max and sum, leading to O(N^2) memory for attention.
  • Online softmax computes softmax in a single pass by maintaining running max and sum, rescaling previous values when the max updates.
  • FlashAttention uses tiling to break the attention computation into blocks that fit in SRAM, avoiding global memory bottlenecks.
  • Online softmax allows each tile to be processed independently and combined correctly, enabling the tiling strategy.
  • Numerical stability is preserved by subtracting the running max before exponentiation and rescaling accumulators.
  • The result is exact softmax (not an approximation) with reduced memory footprint and improved cache efficiency.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q3

What are the memory traffic tradeoffs in FlashAttention compared to standard attention, and how does reducing HBM reads and writes translate to actual speedups?

System DesignTechnical Trade-offs
Author's notes

Went okay.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by contrasting the memory access patterns of standard attention (materializing the full N×N attention matrix in HBM) with FlashAttention's tiling and recomputation strategy that keeps intermediate results in SRAM. Then quantify how the reduction in HBM traffic (from O(N²) to O(N)) translates to speedups, especially for long sequences, and discuss the tradeoff of increased compute (recomputation) for reduced memory bandwidth pressure.

Pro tip: Emphasize that FlashAttention is not just about memory savings but about achieving higher arithmetic intensity and better GPU utilization; mention that the speedup is most pronounced when the attention matrix is large enough to exceed SRAM capacity, and that it enables training with longer sequences without OOM.

1. Describe standard attention memory traffic

Explain that standard attention computes S = QK^T, P = softmax(S), and O = PV, materializing S and P in HBM, leading to O(N²) memory reads/writes.

2. Explain FlashAttention's tiling and recomputation

Detail how FlashAttention tiles Q, K, V into blocks, computes attention block-by-block in SRAM, and recomputes intermediate values during the backward pass instead of storing them.

3. Quantify memory traffic reduction

State that FlashAttention reduces HBM accesses from O(N²) to O(N), which is significant for long sequences (e.g., N=4K or 16K).

4. Connect memory reduction to speedups

Explain that HBM bandwidth is often the bottleneck; reducing reads/writes increases arithmetic intensity and allows the GPU to spend more time computing, leading to 2-4x speedups in practice.

5. Discuss tradeoffs and limitations

Mention that FlashAttention increases compute (recomputation) but this is often hidden by memory savings; also note that it requires custom CUDA kernels and may not be beneficial for very short sequences.

Key Points to Mention

  • Standard attention materializes N×N attention matrix in HBM, causing O(N²) memory traffic.
  • FlashAttention uses tiling to keep blocks in SRAM and recomputes attention during backward pass.
  • HBM bandwidth is the primary bottleneck in standard attention; reducing traffic improves speed.
  • Speedups are more significant for longer sequences (e.g., 2-4x for N=2K-4K).
  • Tradeoff: increased compute (recomputation) but overall faster due to memory-bound nature.
  • Enables training with longer sequences without out-of-memory errors.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q4

Compare multi-query attention and grouped-query attention. What problem are they solving and what do you give up?

Technical Trade-offsSystem Design
Author's notes

Solid ground for me.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by framing the core problem: standard multi-head attention (MHA) has a KV cache that scales with the number of heads, which becomes a memory and bandwidth bottleneck during autoregressive inference. Then compare MQA (all heads share a single KV head) and GQA (groups of query heads share a KV head) as trade-offs between memory efficiency and model quality, and explicitly state what is sacrificed: expressiveness and potentially some accuracy, though GQA often recovers most of it.

Pro tip: Mention that GQA is used in production models like Llama 2/3 and Mistral, and that it can be trained from scratch or by uptraining an existing MHA model with a small fraction of compute—this shows you understand practical deployment constraints, not just theory.

1. Define the baseline and the problem

Explain that standard MHA caches keys and values per head, so KV cache size grows linearly with sequence length and number of heads, causing memory and bandwidth issues during inference. This is the problem both MQA and GQA aim to solve.

2. Describe MQA

MQA uses a single shared key-value head for all query heads, drastically reducing KV cache size and memory bandwidth. This speeds up inference but can hurt model quality because all heads must attend to the same key/value representations.

3. Describe GQA

GQA groups query heads and assigns one key-value head per group, interpolating between MHA and MQA. It reduces KV cache size by a factor equal to the group size while preserving more diversity in attention than MQA.

4. Compare trade-offs

Contrast memory savings, inference speed, and model quality: MQA gives maximum savings but may degrade quality; GQA offers a tunable balance. Mention that GQA often achieves quality close to MHA with significant memory reduction.

5. State what you give up

Acknowledge that both methods reduce the number of independent key/value projections, which limits the model's ability to attend to different representation subspaces. This can lead to a slight drop in accuracy or require more training to match MHA performance.

Key Points to Mention

  • KV cache memory and bandwidth bottleneck during autoregressive inference
  • MHA: one KV head per query head; MQA: one KV head total; GQA: one KV head per group of query heads
  • Memory reduction factor: MQA reduces KV cache by number of heads; GQA by group size
  • Quality vs. efficiency trade-off: MQA fastest but least expressive; GQA balances both
  • GQA used in models like Llama 2 70B, Llama 3, and Mistral; can be obtained by uptraining MHA
  • What you give up: reduced representational diversity in attention, potential accuracy drop, need for careful group size tuning

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q5

What is paged attention and how does it help with KV cache memory management during inference?

System DesignTechnical Trade-offs
Author's notes

Knew this one from reading about vLLM.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining paged attention as a memory management technique inspired by OS virtual memory, then explain how it partitions the KV cache into fixed-size blocks to eliminate fragmentation and enable dynamic allocation. Finally, discuss the concrete benefits for inference, such as higher throughput, lower memory waste, and better scalability for long sequences.

Pro tip: Mention that paged attention is the core innovation behind vLLM, and that it can improve memory utilization by 2-4x, enabling larger batch sizes and up to 24x higher throughput compared to naive implementations. This shows you understand real-world impact.

1. Define paged attention

Explain that it's a technique to manage the KV cache in fixed-size blocks (pages), similar to virtual memory paging in operating systems, allowing non-contiguous storage.

2. Describe the problem with traditional KV cache

Highlight that traditional KV cache pre-allocates contiguous memory for the maximum sequence length, leading to internal and external fragmentation, and wasted memory.

3. Explain how paged attention works

Detail that the KV cache is divided into blocks, each holding a fixed number of tokens' keys and values. A block table maps logical positions to physical blocks, enabling dynamic allocation and sharing.

4. List benefits for memory management

Discuss how it reduces memory waste (no over-allocation), allows flexible sequence lengths, enables memory sharing across sequences (e.g., parallel sampling), and improves cache utilization.

5. Connect to inference performance

Conclude that these memory savings allow larger batch sizes and longer sequences, boosting throughput and reducing latency, which is critical for serving LLMs at scale.

Key Points to Mention

  • KV cache stores key and value tensors for autoregressive generation, growing with sequence length.
  • Traditional contiguous allocation causes fragmentation and limits batch size.
  • Paged attention uses fixed-size blocks and a block table for non-contiguous memory mapping.
  • Enables dynamic allocation and deallocation of memory as sequences grow or finish.
  • Supports memory sharing across sequences (e.g., beam search, parallel sampling).
  • Implemented in vLLM, achieving significant throughput improvements (e.g., 2-4x memory savings, up to 24x throughput).

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q6

Can you explain sparse attention and linear attention as alternatives to full quadratic attention? What are the practical limitations?

Technical Trade-offsAlgorithms & Data Structures
Author's notes

Sparse attention I handled fine, patterns like sliding window or strided blocks.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining full quadratic attention and its O(n^2) complexity, then explain sparse and linear attention as methods to reduce this complexity. Compare their mechanisms and trade-offs, and discuss practical limitations like accuracy, hardware efficiency, and implementation complexity. Conclude with when to use each approach.

Pro tip: Mention that linear attention often requires a kernel feature map and can suffer from low-rank bottlenecks, while sparse attention patterns must be chosen carefully to match data structure; also note that real-world speedups depend on hardware and kernel implementations.

1. Define full attention and its complexity

Explain that full attention computes pairwise interactions between all tokens, leading to O(n^2) time and memory complexity, which is prohibitive for long sequences.

2. Explain sparse attention

Describe sparse attention as restricting each token to attend to a subset of tokens (e.g., local windows, strided patterns, or learned sparsity), reducing complexity to O(n sqrt(n)) or O(n log n) depending on pattern.

3. Explain linear attention

Describe linear attention as approximating the softmax attention with a kernel feature map, allowing computation to be reordered to achieve O(n) complexity in time and memory.

4. Discuss practical limitations

Cover limitations: sparse attention may miss long-range dependencies and requires pattern design; linear attention can suffer from reduced expressiveness and low-rank bottlenecks; both may have implementation overhead and hardware inefficiencies.

5. Conclude with trade-offs and use cases

Summarize when to use each: sparse for structured data with local patterns, linear for very long sequences where efficiency is critical, and note that hybrid approaches exist.

Key Points to Mention

  • Quadratic complexity of full attention and its memory bottleneck
  • Sparse attention patterns: local, strided, block-sparse, and learned
  • Linear attention via kernel feature maps and associative property
  • Trade-offs: accuracy vs. efficiency, expressiveness, and long-range dependency capture
  • Hardware considerations: actual speedups depend on GPU/TPU kernels and memory access patterns
  • Examples: Longformer, BigBird, Performer, Linformer, and their limitations

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q7

How does quantizing the KV cache affect inference quality and memory usage, and what are the risks?

Technical Trade-offsSystem Design
Author's notes

Talked through int8 and fp8 KV cache quantization.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining KV cache quantization and its purpose: reducing memory footprint and enabling longer context or larger batch sizes. Then explain the trade-offs: lower precision (e.g., 8-bit or 4-bit) reduces memory but can introduce quantization error, affecting output quality, especially for tasks requiring high precision. Finally, discuss risks such as accuracy degradation, increased latency from dequantization, and potential need for calibration or mixed-precision strategies.

Pro tip: Emphasize that the impact on quality depends on the model, task, and quantization method; always validate with task-specific metrics and consider fallback to higher precision for critical layers or tokens.

1. Define KV cache and quantization

Briefly explain what the KV cache is and why quantization is applied (memory savings, throughput).

2. Memory usage impact

Quantify memory reduction: e.g., 8-bit halves memory, 4-bit quarters it, enabling longer sequences or larger batches.

3. Inference quality impact

Discuss how quantization error can degrade output quality, especially for tasks needing high precision; mention that effects vary by model and task.

4. Risks and mitigation

Outline risks: accuracy loss, increased latency from dequantization, calibration overhead; suggest mitigations like mixed-precision, per-channel quantization, or outlier handling.

5. Evaluation and trade-off decision

Conclude with the need to empirically evaluate quality vs. memory/latency trade-offs for the specific use case.

Key Points to Mention

  • Memory reduction: 8-bit halves, 4-bit quarters the KV cache size
  • Quantization error can cause degradation in perplexity, BLEU, or task-specific metrics
  • Impact varies by model size, task, and quantization granularity (per-tensor vs per-channel)
  • Risks: accuracy loss, increased latency from dequantization, calibration complexity
  • Mitigation: mixed-precision (keep critical layers in FP16), outlier-aware quantization, or dynamic quantization
  • Trade-off: longer context or larger batch vs. potential quality drop; always validate on downstream tasks

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.