← Pinterest Interview InsightsMachine Learning Engineer
Start by breaking down the memory budget into components: model parameters, activations, KV cache, and framework overhead. Then, derive a theoretical maximum batch size and validate it empirically by measuring throughput and latency across a range of batch sizes, considering the latency-throughput trade-off and SLOs.
Pro tip: Always measure actual memory usage and latency with profiling tools, as theoretical estimates can be off due to fragmentation and framework-specific overhead. Also, consider using techniques like gradient checkpointing or quantization if memory is tight.
Calculate memory for model parameters (weights), activations (intermediate tensors), KV cache (for transformers), and framework overhead (CUDA context, cuDNN workspaces).
Sum the memory components and subtract from total GPU memory to find available memory for batch data. Divide by per-sample memory to get theoretical max batch size.
Run experiments with varying batch sizes, measuring throughput (samples/sec) and latency (p50, p99). Use profiling tools to monitor actual memory usage.
Plot throughput vs batch size and latency vs batch size. Identify the knee point where throughput gains diminish or latency exceeds SLO.
Choose the batch size that maximizes throughput while meeting latency requirements, considering hardware utilization and cost.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.