← Waymo Interview InsightsSoftware Engineer

Waymo·Software Engineer·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say
Jun 2026

Summary

Waymo system design round, one big open-ended question about building an evaluation platform from scratch. Dense problem with a lot of moving parts and I felt like I was constantly choosing which thread to pull on.

Questions Asked (1)

Q1

Design an evaluation platform that supports both human reviewers and LLM-based judges. Your design should cover how tasks get ingested and routed, how you manage a mixed pool of human and automated evaluators, quality control mechanisms like gold tasks and inter-annotator agreement, how you aggregate results into a consensus, throughput and SLA guarantees, cost trade-offs between human and LLM evaluation, and what APIs you expose to customers.

System DesignTechnical Trade-offsAPI & Integrations
Author's notes

This is a beast of a question.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements and scale, then present a high-level architecture covering ingestion, routing, evaluation pools, quality control, aggregation, and APIs. Emphasize trade-offs between human and LLM evaluators, and how you ensure reliability and cost-efficiency. Conclude with how you'd measure success and iterate.

Pro tip: Anchor your design in concrete numbers (e.g., expected QPS, latency SLAs, cost per evaluation) and explicitly discuss failure modes and mitigation strategies, as this demonstrates production maturity.

1. Clarify Requirements and Scope

Ask questions to understand scale (tasks/day), latency SLAs, budget constraints, and customer expectations. Define what 'evaluation' means for the use case (e.g., correctness, safety, preference).

2. Design Ingestion and Routing

Outline how tasks are ingested via API, validated, and routed to appropriate evaluators based on task type, complexity, and cost. Include a priority queue and dynamic routing logic.

3. Manage Evaluator Pools and Quality Control

Describe how to onboard and manage human reviewers and LLM judges, including gold tasks, inter-annotator agreement, and continuous calibration. Explain how to detect and handle low-quality evaluators.

4. Aggregate Results and Ensure Consensus

Detail the aggregation strategy (e.g., majority vote, weighted average, Dawid-Skene) to combine human and LLM outputs into a final score. Discuss conflict resolution and confidence scoring.

5. Define APIs, SLAs, and Cost Trade-offs

Specify customer-facing APIs for task submission and result retrieval, along with throughput and latency guarantees. Compare cost and quality of human vs. LLM evaluation and propose a hybrid approach.

Key Points to Mention

  • Hybrid routing: use LLMs for high-volume, low-complexity tasks and humans for nuanced or high-stakes evaluations.
  • Gold tasks and inter-annotator agreement (e.g., Cohen's kappa) to measure and maintain evaluator quality.
  • Aggregation methods like majority voting or probabilistic models to combine heterogeneous judgments.
  • SLA guarantees: define throughput (tasks/sec), latency (p95), and availability, with autoscaling and fallback mechanisms.
  • Cost optimization: dynamic pricing, spot instances for LLMs, and tiered human expertise to balance budget and quality.
  • API design: RESTful endpoints for task submission, status polling, and result retrieval, with webhooks for async updates.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.