Hippocratic AI·Machine Learning Engineer·Onsite - System Design / Architecture
- Design a high-throughput multi-GPU LLM inference serving system from scratch. Walk through the full stack including memory management, GPU communication, and request handling.
- How do you handle All-Reduce communication overhead in tensor parallelism, and what techniques reduce the bottleneck between computation and communication across GPUs?
- An extremely long prompt arrives and blows past your pre-allocated KV-cache pool. How do you prevent the node from OOM-ing or crashing, and what are the trade-offs between your recovery options?
“I started with the KV-cache layer because that felt most concrete to me.”