Start by clarifying requirements: expected request rate, latency SLOs, model size, and workload type (online vs. batch). Then design a single-GPU inference service that maximizes throughput via batching, caching, and model optimization, while ensuring reliability through queuing and autoscaling of stateless components. Finally, discuss trade-offs and potential scaling strategies if demand exceeds GPU capacity.
Pro tip: Emphasize that with only one GPU, the goal is to maximize utilization without violating latency SLOs; dynamic batching and model quantization are often the highest-leverage optimizations. Also, mention that you would monitor GPU utilization and queue depth to trigger alerts and potential fallback to CPU or a second GPU if needed.
Ask about expected QPS, latency targets, model size, input/output sizes, and whether the workload is online or batch. Confirm that only one GPU is available and whether it can be upgraded or supplemented later.
Propose a stateless API layer (e.g., FastAPI) that queues requests to a single GPU worker. Use a message queue (e.g., Redis, RabbitMQ) to decouple request ingestion from GPU processing and enable backpressure.
Implement dynamic batching to group requests, use model quantization (FP16/INT8) and pruning to reduce compute, and consider caching frequent requests. Choose an efficient runtime like TensorRT or ONNX Runtime.
Add health checks, retries, and circuit breakers. Autoscale the API layer based on queue depth, and plan for graceful degradation (e.g., fallback to CPU or a smaller model) if GPU is saturated.
Acknowledge that a single GPU is a single point of failure and limits throughput. Discuss trade-offs between latency and throughput, and outline a path to multi-GPU or distributed inference if demand grows.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.