← Nuro Interview Insights

Nuro·Machine Learning Engineer·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say
May 2026

Summary

System design round at Nuro for an MLE role. One GPU, cloud inference setup, and I was not prepared. Pretty rough.

Questions Asked (1)

Q1

Design a cloud-based inference system with only a single GPU available.

System DesignTechnical Trade-offs
Author's notes

I fumbled this pretty badly.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements: expected request rate, latency SLOs, model size, and workload type (online vs. batch). Then design a single-GPU inference service that maximizes throughput via batching, caching, and model optimization, while ensuring reliability through queuing and autoscaling of stateless components. Finally, discuss trade-offs and potential scaling strategies if demand exceeds GPU capacity.

Pro tip: Emphasize that with only one GPU, the goal is to maximize utilization without violating latency SLOs; dynamic batching and model quantization are often the highest-leverage optimizations. Also, mention that you would monitor GPU utilization and queue depth to trigger alerts and potential fallback to CPU or a second GPU if needed.

1. Clarify Requirements and Constraints

Ask about expected QPS, latency targets, model size, input/output sizes, and whether the workload is online or batch. Confirm that only one GPU is available and whether it can be upgraded or supplemented later.

2. Design the Serving Architecture

Propose a stateless API layer (e.g., FastAPI) that queues requests to a single GPU worker. Use a message queue (e.g., Redis, RabbitMQ) to decouple request ingestion from GPU processing and enable backpressure.

3. Optimize GPU Utilization

Implement dynamic batching to group requests, use model quantization (FP16/INT8) and pruning to reduce compute, and consider caching frequent requests. Choose an efficient runtime like TensorRT or ONNX Runtime.

4. Ensure Reliability and Scalability

Add health checks, retries, and circuit breakers. Autoscale the API layer based on queue depth, and plan for graceful degradation (e.g., fallback to CPU or a smaller model) if GPU is saturated.

5. Discuss Trade-offs and Future Scaling

Acknowledge that a single GPU is a single point of failure and limits throughput. Discuss trade-offs between latency and throughput, and outline a path to multi-GPU or distributed inference if demand grows.

Key Points to Mention

  • Dynamic batching to increase throughput while meeting latency SLOs
  • Model optimization techniques: quantization, pruning, and using TensorRT/ONNX
  • Queuing and backpressure to handle bursts and avoid GPU overload
  • Caching of frequent requests to reduce GPU load
  • Monitoring GPU utilization, queue depth, and latency for autoscaling and alerting
  • Trade-offs: single GPU as a bottleneck and single point of failure; strategies for graceful degradation

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.