← Adobe Interview InsightsMachine Learning Engineer

Adobe·Machine Learning Engineer·Technical Phone Screen·Senior

Senior
Jun 2026

Summary

Adobe MLE interview that went deep into transformer internals and GPU performance. Felt like they wanted someone who actually understands why things are slow, not just someone who can use a library.

Questions Asked (5)

Q1

Why are transformer models considered memory-bound rather than compute-bound, and what does that mean in practice?

System DesignTechnical Trade-offs
Author's notes

This is where I felt most confident.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining memory-bound vs compute-bound, then explain why transformers are memory-bound due to their large parameter counts and attention mechanisms. Discuss practical implications such as memory bandwidth limitations, the need for techniques like quantization and pruning, and how this affects deployment and scaling.

Pro tip: Mention that while transformers are memory-bound during inference, training can be compute-bound due to backpropagation and large batch sizes. This nuance shows depth and helps tailor solutions to the specific phase.

1. Define the concepts

Clearly explain what memory-bound and compute-bound mean: memory-bound means performance is limited by data movement (memory bandwidth), while compute-bound means limited by arithmetic operations (FLOPS).

2. Explain why transformers are memory-bound

Highlight that transformers have many parameters (e.g., billions) and the attention mechanism requires storing and accessing large key/value caches, leading to high memory traffic relative to compute.

3. Discuss practical implications

Describe how this affects real-world deployment: inference speed is often limited by memory bandwidth, not FLOPS; techniques like quantization, pruning, and efficient attention (e.g., FlashAttention) help mitigate memory bottlenecks.

4. Address training vs inference

Note that training is often compute-bound due to gradient computations and large batch sizes, while inference is typically memory-bound, especially for autoregressive generation.

5. Conclude with trade-offs and solutions

Summarize that understanding this distinction guides optimization strategies, such as using hardware with high memory bandwidth or algorithmic improvements to reduce memory footprint.

Key Points to Mention

  • Memory-bound vs compute-bound definitions
  • Large parameter counts and memory footprint of transformers
  • Attention mechanism's memory access patterns (key/value caching)
  • Inference vs training differences
  • Optimization techniques: quantization, pruning, FlashAttention, model parallelism
  • Hardware considerations: memory bandwidth vs FLOPS

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q2

What is the cost of materializing the full attention matrix, and how does that affect system design decisions?

System DesignTechnical Trade-offs
Author's notes

Blanked for a second on how to quantify it precisely.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by quantifying the memory and compute costs of materializing the full attention matrix, which scales quadratically with sequence length. Then discuss how these costs drive system design choices such as memory-efficient attention algorithms, hardware-aware optimizations, and trade-offs between accuracy and efficiency. Conclude with practical implications for deploying models at scale, especially in resource-constrained environments.

Pro tip: Mention that while full attention is simple and parallelizable, its quadratic cost often makes it impractical for long sequences; instead, highlight that many production systems use approximations like FlashAttention or sparse attention to balance performance and quality. This shows you understand real-world constraints beyond theory.

1. Quantify the cost

Explain that the attention matrix has size O(n^2) for sequence length n, leading to O(n^2) memory and O(n^2 d) compute for head dimension d. Emphasize that both memory and compute grow quadratically.

2. Identify bottlenecks

Discuss how materializing the full matrix causes high memory bandwidth usage and limits batch size or sequence length, often making it the bottleneck in training and inference.

3. Explore alternatives

Introduce memory-efficient attention methods like FlashAttention, which avoid materializing the full matrix by tiling and recomputation, reducing memory to O(n) and improving speed.

4. Connect to system design

Explain how these costs influence decisions: choosing attention variants, optimizing hardware utilization, and designing for scalability (e.g., distributed training, model parallelism).

5. Discuss trade-offs

Weigh the simplicity and accuracy of full attention against the efficiency of approximations, and mention that the choice depends on application requirements like latency, throughput, and sequence length.

Key Points to Mention

  • Quadratic memory and compute complexity O(n^2) for sequence length n
  • Memory bandwidth and GPU utilization challenges due to large intermediate matrices
  • FlashAttention and other IO-aware algorithms that reduce memory footprint
  • Impact on batch size, sequence length, and model parallelism strategies
  • Trade-offs between exact attention and approximate methods (e.g., sparse, low-rank)
  • Real-world implications for deployment on edge devices or in latency-sensitive applications

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q3

Can you explain HFU and MFU and how you'd use them to evaluate GPU efficiency?

Technical Trade-offsSystem Design
Author's notes

MFU vs HFU is one of those distinctions that sounds pedantic until you're actually debugging why your training run is underperforming.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining HFU and MFU clearly, then explain how they measure GPU efficiency by comparing achieved FLOPS to theoretical peak. Discuss how you would use them in practice to identify bottlenecks and guide optimizations, and mention their limitations and complementary metrics.

Pro tip: Emphasize that MFU is often the more practical metric for real-world workloads, but HFU can be useful for diagnosing whether low utilization is due to algorithmic inefficiency or hardware limitations. Also, mention that these metrics should be tracked over time to detect regressions.

1. Define HFU and MFU

Explain that HFU (Hardware FLOPS Utilization) is the ratio of achieved FLOPS to the GPU's theoretical peak FLOPS (e.g., from spec sheets), while MFU (Model FLOPS Utilization) is the ratio of achieved FLOPS to the peak FLOPS achievable for a given model architecture (accounting for sparsity, precision, etc.).

2. Explain how to calculate them

Describe the formula: HFU = achieved FLOPS / peak hardware FLOPS; MFU = achieved FLOPS / peak model FLOPS. Mention that achieved FLOPS can be measured via profiling tools like NVIDIA's Nsight or PyTorch Profiler.

3. Discuss their role in evaluating GPU efficiency

Explain that high HFU indicates the hardware is being used near its maximum capability, while high MFU indicates the model is running efficiently relative to its theoretical best. Low values suggest bottlenecks such as memory bandwidth, kernel inefficiencies, or poor parallelization.

4. Provide a practical example

Give an example: If training a transformer yields 30% MFU, you might investigate optimizations like mixed precision, better kernel fusion, or increasing batch size. If HFU is also low, the issue might be in data loading or communication overhead.

5. Mention limitations and complementary metrics

Note that MFU can be misleading if the model's theoretical peak is not well-defined, and HFU can be unattainable due to hardware constraints. Suggest also looking at memory bandwidth utilization, SM occupancy, and power consumption for a holistic view.

Key Points to Mention

  • Definition of HFU and MFU with formulas
  • Difference between theoretical peak and achievable peak FLOPS
  • How to measure achieved FLOPS using profiling tools
  • Common bottlenecks that reduce HFU/MFU (e.g., memory bandwidth, kernel launch overhead)
  • Optimization strategies to improve HFU/MFU (e.g., mixed precision, kernel fusion, larger batch sizes)
  • Limitations of HFU/MFU and the need for complementary metrics

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q4

How would you identify and address GPU underutilization in a training or inference workload?

System DesignRoot Cause Analysis
Author's notes

Talked about profiling tools and looking at kernel timelines.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by explaining how you would measure GPU utilization using tools like nvidia-smi, Nsight Systems, or PyTorch Profiler to establish a baseline. Then, systematically diagnose common bottlenecks such as data loading, small batch sizes, inefficient kernels, or CPU-GPU synchronization. Finally, propose targeted optimizations like increasing batch size, using mixed precision, optimizing data pipelines, or leveraging CUDA graphs, and validate improvements with metrics.

Pro tip: Emphasize the importance of profiling before optimizing—many candidates jump to solutions without confirming the actual bottleneck. Also, mention that underutilization can be intentional (e.g., for latency-sensitive inference) and that the goal is to balance utilization with other constraints like latency or cost.

1. Measure and Baseline

Use profiling tools (nvidia-smi, Nsight Systems, PyTorch Profiler) to quantify GPU utilization, memory usage, and kernel execution times. Establish a baseline to identify periods of underutilization.

2. Identify Bottlenecks

Analyze the profile to pinpoint bottlenecks: data loading (CPU-bound), small batch sizes, inefficient kernels, excessive synchronization, or memory transfers. Check if the GPU is waiting on data or if kernels are underutilized.

3. Prioritize Optimizations

Based on the bottleneck, prioritize fixes: increase batch size, use mixed precision, optimize data pipeline with prefetching and parallel loading, fuse operations, or use CUDA graphs to reduce launch overhead.

4. Implement and Validate

Apply the optimizations incrementally and re-profile to measure impact. Ensure that changes improve utilization without degrading model accuracy or increasing latency beyond acceptable limits.

5. Monitor and Iterate

Set up continuous monitoring in production to detect underutilization regressions. Iterate on optimizations as workload or hardware changes.

Key Points to Mention

  • Profiling tools: nvidia-smi, Nsight Systems, PyTorch Profiler, TensorBoard
  • Common bottlenecks: data loading, small batch size, CPU-GPU synchronization, kernel launch overhead
  • Optimization techniques: mixed precision training, larger batch sizes, data prefetching, CUDA graphs, model parallelism
  • Trade-offs: utilization vs. latency, cost, and accuracy
  • Importance of iterative profiling and validation
  • Consideration of workload type: training vs. inference (e.g., inference may have low utilization due to latency constraints)

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q5

What is kernel fusion and when would you use it? How do CUDA graphs fit into that picture?

Technical Trade-offsSystem Design
Author's notes

Kernel fusion reduces the number of round-trips to global memory by combining ops into a single kernel.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining kernel fusion as combining multiple GPU operations into a single kernel to reduce memory traffic and launch overhead. Explain when to use it: when operations are memory-bound, have compatible data layouts, and can be fused without excessive register pressure. Then distinguish CUDA graphs as a complementary technique that captures a sequence of kernel launches (fused or not) into a single graph to reduce CPU launch overhead, especially for static workloads.

Pro tip: Mention that fusion and CUDA graphs solve different bottlenecks: fusion reduces GPU-side memory traffic and kernel count, while CUDA graphs reduce CPU-side launch overhead. Combining both can yield multiplicative gains, but beware of reduced flexibility and increased compile-time complexity.

1. Define kernel fusion

Explain that kernel fusion merges multiple GPU kernels into one, eliminating intermediate global memory writes/reads and reducing kernel launch overhead.

2. When to use fusion

Discuss scenarios: memory-bound elementwise chains (e.g., activation + dropout), small operations with low arithmetic intensity, and when data dependencies allow. Avoid when register pressure or shared memory limits cause spills or occupancy drops.

3. Introduce CUDA graphs

Describe CUDA graphs as a way to capture a sequence of kernel launches (including fused kernels) into a DAG, then replay it with minimal CPU overhead.

4. How they fit together

Explain that fusion reduces the number of nodes in the graph, while CUDA graphs reduce the per-node launch cost. They are complementary: fuse first to minimize work, then graph to minimize launch overhead.

5. Trade-offs and practical considerations

Mention that fusion can increase compile time and reduce flexibility (e.g., dynamic shapes), while CUDA graphs require static shapes and memory addresses. Both are best for inference or repeated training steps with fixed shapes.

Key Points to Mention

  • Kernel fusion reduces memory bandwidth pressure by keeping intermediate results in registers or shared memory.
  • Fusion is most beneficial for memory-bound operations with low arithmetic intensity.
  • CUDA graphs capture kernel launches and memory operations into a single graph to reduce CPU launch overhead.
  • Fusion and CUDA graphs are complementary: fusion reduces GPU work, graphs reduce CPU overhead.
  • Trade-offs: fusion may increase register pressure and reduce occupancy; CUDA graphs require static shapes and addresses.
  • Real-world examples: fused kernels in PyTorch (e.g., fused Adam) and CUDA graphs in TensorRT or PyTorch's graph mode.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.