← Pinterest Interview InsightsMachine Learning Engineer

Pinterest·Machine Learning Engineer·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say
Jun 2026

Summary

Pinterest ML infra interview, system design round focused entirely on GPU memory budgeting for model serving. Pretty deep dive, more math-heavy than I expected for a design question.

Questions Asked (1)

Q1

You have a GPU with 60 GB of memory. Walk through how you'd figure out the optimal batch size for serving a model, covering parameter memory, activation memory, KV cache for transformer models, framework overhead, and the latency vs throughput trade-off. How do you validate your choice empirically?

System DesignTechnical Trade-offs
Author's notes

This question is deceptively wide.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by breaking down the memory budget into components: model parameters, activations, KV cache, and framework overhead. Then, derive a theoretical maximum batch size and validate it empirically by measuring throughput and latency across a range of batch sizes, considering the latency-throughput trade-off and SLOs.

Pro tip: Always measure actual memory usage and latency with profiling tools, as theoretical estimates can be off due to fragmentation and framework-specific overhead. Also, consider using techniques like gradient checkpointing or quantization if memory is tight.

1. Estimate memory components

Calculate memory for model parameters (weights), activations (intermediate tensors), KV cache (for transformers), and framework overhead (CUDA context, cuDNN workspaces).

2. Derive maximum batch size

Sum the memory components and subtract from total GPU memory to find available memory for batch data. Divide by per-sample memory to get theoretical max batch size.

3. Benchmark empirically

Run experiments with varying batch sizes, measuring throughput (samples/sec) and latency (p50, p99). Use profiling tools to monitor actual memory usage.

4. Analyze trade-offs

Plot throughput vs batch size and latency vs batch size. Identify the knee point where throughput gains diminish or latency exceeds SLO.

5. Select optimal batch size

Choose the batch size that maximizes throughput while meeting latency requirements, considering hardware utilization and cost.

Key Points to Mention

  • Parameter memory: model size in bytes (e.g., FP16: 2 bytes per parameter).
  • Activation memory: depends on batch size, sequence length, and model architecture; can be reduced with checkpointing.
  • KV cache: for transformers, memory grows with batch size, sequence length, and number of layers; formula: 2 * batch_size * seq_len * num_layers * hidden_size * bytes_per_element.
  • Framework overhead: CUDA context, cuDNN, and other libraries consume memory; typically 1-2 GB.
  • Latency vs throughput trade-off: larger batches increase throughput but also increase latency; need to balance based on SLO.
  • Empirical validation: use tools like nvidia-smi, PyTorch profiler, or TensorBoard to measure memory and latency; test with realistic input shapes.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.