← Waymo Interview InsightsMachine Learning Engineer

Waymo·Machine Learning Engineer·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say
Apr 2026

Summary

Waymo ML engineer system design round, focused on building an inference system at massive scale. Pretty intense scope for a single session, lots of back-and-forth on the math.

Questions Asked (1)

Q1

Design an inference system that can serve 100 million daily active users. Walk through your memory and bandwidth estimates, and explain how you'd handle latency and out-of-memory failures.

System DesignTechnical Trade-offs
Author's notes

This one sprawled fast.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the workload characteristics (e.g., model size, request rate, latency SLA) and then walk through a structured capacity plan: estimate QPS, memory footprint, and bandwidth needs. Propose a scalable architecture with caching, sharding, and autoscaling, and discuss specific strategies for latency reduction and graceful degradation under memory pressure.

Pro tip: Emphasize that for autonomous driving, safety-critical inference must be deterministic and have fallback mechanisms; mention that you'd design for worst-case latency, not average, and include redundancy to handle out-of-memory without impacting vehicle safety.

1. Clarify Requirements and Assumptions

Ask about model size, input/output sizes, latency SLA, request patterns, and hardware constraints. State assumptions clearly (e.g., average model size 500MB, 1000 QPS peak).

2. Estimate QPS, Memory, and Bandwidth

Calculate average and peak QPS from 100M DAU. Estimate total memory needed for model replicas and bandwidth for data transfer, considering factors like batching and compression.

3. Design Scalable Architecture

Propose a multi-tier architecture with load balancers, model servers (e.g., Triton), caching layers, and sharding. Discuss horizontal scaling, autoscaling, and use of edge vs. cloud inference.

4. Address Latency Optimization

Detail techniques: model quantization, pruning, batching, caching frequent requests, using faster hardware (GPUs/TPUs), and optimizing network paths.

5. Handle Out-of-Memory Failures

Describe strategies: memory pooling, graceful degradation (fallback to smaller model), circuit breakers, monitoring, and auto-recovery. Emphasize safety-critical fallbacks.

Key Points to Mention

  • QPS calculation: 100M DAU * average requests per user per day / 86400 seconds, with peak factor (e.g., 2-3x).
  • Memory estimation: model size * number of replicas + overhead; consider model parallelism for large models.
  • Bandwidth: input/output data size * QPS; use compression and batching to reduce.
  • Latency SLA: for Waymo, likely <100ms; use edge inference, model optimization, and caching.
  • Out-of-memory handling: pre-allocate memory, use OOM killer with priorities, fallback to simpler models, and implement health checks.
  • Scalability: sharding by user/region, autoscaling based on load, and use of spot instances for cost efficiency.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.