← DoorDash Interview InsightsMachine Learning Engineer

DoorDash·Machine Learning Engineer·Onsite - System Design / Architecture·Senior

Senior
Jun 2026

Summary

DoorDash ML system design round focused entirely on infrastructure, not modeling. The scope was wide enough that I kept second-guessing whether I was going too deep on one area and neglecting others.

Questions Asked (7)

Q1

Design the end-to-end ML infrastructure for a large-scale product, covering the full training and inference pipeline at a high level.

System DesignTechnical Trade-offs
Author's notes

I started with data ingestion and worked forward, which felt logical but probably wasted time on stuff they didn't care about.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the product requirements and scale (e.g., number of users, requests per second, data volume) to ground your design. Then walk through the ML lifecycle: data ingestion, feature engineering, training, deployment, and monitoring, highlighting key components and trade-offs at each stage. Emphasize scalability, reliability, and how you would iterate quickly while maintaining low latency for inference.

Pro tip: Anchor your design around a concrete DoorDash use case (e.g., delivery time prediction or search ranking) to make it tangible, and proactively discuss trade-offs like batch vs. online inference and model freshness vs. cost.

1. Clarify Requirements and Scale

Ask questions to understand the product, data volume, latency requirements, and update frequency. This ensures your design is tailored to the specific use case.

2. Data and Feature Pipeline

Describe how data is ingested, processed, and stored for training and serving. Include batch and streaming pipelines, feature stores, and data versioning.

3. Training Infrastructure

Outline the training workflow: experiment tracking, distributed training, hyperparameter tuning, and model registry. Discuss how to handle large datasets and retraining.

4. Model Deployment and Inference

Explain how models are deployed for online and batch inference, including serving infrastructure, scaling, and latency optimization. Mention A/B testing and canary deployments.

5. Monitoring and Iteration

Cover monitoring for data drift, model performance, and system health. Describe feedback loops for continuous improvement and retraining.

Key Points to Mention

  • Feature store for consistent features between training and serving
  • Distributed training frameworks (e.g., TensorFlow, PyTorch) and hardware accelerators (GPUs/TPUs)
  • Model registry and versioning for reproducibility and rollback
  • Online serving with low-latency requirements (e.g., using TensorFlow Serving, TorchServe, or custom solutions)
  • Monitoring for data drift, model decay, and system metrics
  • Trade-offs between batch and real-time inference, model complexity vs. latency, and cost vs. performance

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q2

How would you design a feature store that supports both online low-latency lookups and offline batch training?

System DesignTechnical Trade-offsData Modeling
Author's notes

This is where I felt most comfortable.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the requirements: online serving needs sub-10ms latency and high availability, while offline training needs point-in-time correctness and scalability. Then propose a dual-store architecture with a unified feature definition layer, and discuss trade-offs around consistency, cost, and complexity.

Pro tip: Emphasize point-in-time correctness for offline training to avoid data leakage, and mention that DoorDash's delivery logistics require features like real-time courier location and restaurant prep time, so the online store must handle high-throughput updates.

1. Clarify Requirements and Constraints

Ask about latency SLAs, data volume, feature freshness, consistency needs, and existing infrastructure. This shows you understand that design depends on specific use cases.

2. Define Unified Feature Schema and Transformations

Propose a central feature registry with versioned definitions and transformation logic that can be applied both online (streaming) and offline (batch). This ensures consistency between training and serving.

3. Design Dual Storage Architecture

Use a low-latency online store (e.g., Redis, DynamoDB) for serving and a scalable offline store (e.g., data lake, warehouse) for training. Discuss how data is ingested and kept in sync.

4. Address Point-in-Time Correctness and Backfilling

Explain how to generate training datasets with time-travel queries to avoid leakage, and how to backfill features for new models. Mention the need for event timestamps and versioning.

5. Discuss Trade-offs and Operational Concerns

Cover consistency models (e.g., eventual vs. strong), cost, monitoring, and failure modes. Highlight how you would handle feature drift and ensure low-latency updates.

Key Points to Mention

  • Point-in-time correctness to prevent data leakage in training
  • Online-offline consistency via a unified transformation layer
  • Choice of storage: key-value store for online, columnar store for offline
  • Streaming ingestion (e.g., Kafka) for real-time features
  • Feature versioning and metadata management
  • Monitoring and alerting for feature freshness and drift

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q3

Walk through how you'd handle experiment management, including A/B tests, shadow deployments, canary rollouts, and rollbacks.

A/B Testing & ExperimentationSystem Design
Author's notes

Shadow deployments tripped me up a bit.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by framing the experimentation lifecycle: define metrics and hypotheses, then choose the right deployment strategy (A/B test, shadow, canary) based on risk and goals. Walk through each method with concrete examples from your experience, emphasizing how you monitor, analyze, and roll back. Conclude with how you'd apply this at DoorDash to balance speed and reliability.

Pro tip: Always tie experiments to business metrics (e.g., delivery time, order conversion) and mention guardrail metrics to catch regressions. Show you understand that not every change needs a full A/B test—shadow and canary are for risk mitigation, not causal inference.

1. Define goals and metrics

Clarify the hypothesis, primary success metrics, and guardrail metrics before choosing an experiment type. Ensure metrics align with business objectives like conversion or delivery efficiency.

2. Select deployment strategy

Choose A/B testing for causal impact, shadow deployment for safe validation without user impact, or canary rollout for gradual exposure. Consider risk, traffic volume, and need for statistical power.

3. Implement and monitor

Set up logging, dashboards, and alerts for real-time monitoring. For A/B tests, ensure proper randomization and sample size; for canary, define success criteria and progression gates.

4. Analyze and decide

Use statistical tests to evaluate A/B results, check guardrails, and decide to ship, iterate, or rollback. For shadow/canary, compare against baseline and assess operational metrics.

5. Rollback and learn

Have a clear rollback plan with automated triggers (e.g., error rate spike). Post-mortem to capture learnings and improve future experiments.

Key Points to Mention

  • A/B testing: randomization, sample size, statistical significance, and avoiding peeking
  • Shadow deployment: mirror production traffic to a new model without affecting users, compare outputs offline
  • Canary rollout: gradual traffic shift, automated rollback on metric degradation, and progressive delivery
  • Rollback strategies: feature flags, versioned models, and automated alerts for quick reversal
  • Guardrail metrics: latency, error rates, and business KPIs to prevent negative impact
  • DoorDash context: experimentation on delivery logistics, personalization, and fraud detection with high traffic and real-time constraints

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q4

How would you ensure reproducibility across training runs?

System DesignTechnical Trade-offs
Author's notes

Short answer: seed everything, version your data snapshots, pin dependencies, log all hyperparameters.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining reproducibility in the ML context—same data, code, and environment yielding the same model—then outline a layered strategy covering data, code, environment, and experiment tracking. Emphasize practical trade-offs and how you'd implement this at scale, referencing tools like Docker, Git, and MLflow.

Pro tip: Mention the importance of capturing non-determinism sources (e.g., GPU operations, random seeds) and how you balance strict reproducibility with iteration speed—showing you understand real-world constraints.

1. Define Reproducibility Scope

Clarify what level of reproducibility is needed (exact, statistical, or directional) and the associated costs. This sets the stage for trade-off discussions.

2. Version Control Everything

Use Git for code, DVC or similar for data, and semantic versioning for models. Ensure all artifacts are traceable to a specific commit.

3. Containerize the Environment

Use Docker or Conda to freeze dependencies, including OS libraries and CUDA versions. Store images in a registry for reuse.

4. Manage Randomness and Determinism

Set random seeds for Python, NumPy, and frameworks; configure deterministic algorithms where possible. Document any non-deterministic operations.

5. Track Experiments and Artifacts

Log parameters, metrics, and artifacts with tools like MLflow or Weights & Biases. Link runs to code, data, and environment versions.

Key Points to Mention

  • Data versioning (e.g., DVC, Delta Lake) to ensure identical training data.
  • Environment isolation (Docker, Conda) with pinned dependencies.
  • Random seed management and deterministic GPU operations.
  • Experiment tracking (MLflow, W&B) for logging parameters and artifacts.
  • Trade-offs between reproducibility and iteration speed.
  • Automated pipelines (CI/CD) to enforce reproducibility checks.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q5

How would you design the serving stack to meet sub-100ms online inference latency at scale, and what tradeoffs exist between CPU and GPU serving?

System DesignTechnical Trade-offs
Author's notes

CPU vs GPU is one of those questions where the answer is genuinely 'it depends' but you can't just say that.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the requirements: what model, what scale (QPS), and what latency SLO (p99 vs p50). Then propose a layered serving architecture that addresses model optimization, infrastructure choices, and dynamic batching, and discuss CPU vs GPU tradeoffs in terms of latency, throughput, cost, and operational complexity.

Pro tip: Emphasize that sub-100ms latency is often achievable with optimized CPU serving for small models, and GPUs are only necessary for large models or high throughput; always quantify tradeoffs with concrete numbers (e.g., cost per million inferences) to show business awareness.

1. Clarify Requirements and Constraints

Ask about model size, QPS, latency SLO (p50/p99), and cost constraints. This ensures your design is tailored to the actual problem.

2. Optimize the Model for Inference

Discuss techniques like quantization, pruning, distillation, and compilation (e.g., TensorRT, ONNX Runtime) to reduce model size and latency.

3. Design the Serving Infrastructure

Propose a layered architecture: load balancer, model server (e.g., Triton, TorchServe), dynamic batching, caching, and autoscaling. Mention edge vs centralized serving.

4. Analyze CPU vs GPU Tradeoffs

Compare CPU and GPU in terms of latency, throughput, cost, scalability, and ease of deployment. Highlight when each is appropriate.

5. Address Monitoring and Iteration

Explain how to monitor latency, throughput, and errors, and how to continuously optimize based on production data.

Key Points to Mention

  • Model optimization techniques: quantization, pruning, knowledge distillation, and hardware-specific compilers (TensorRT, OpenVINO).
  • Dynamic batching and request queuing to maximize hardware utilization while meeting latency SLOs.
  • CPU serving advantages: lower cost, easier scaling, no cold start, and sufficient for small models with optimized runtimes.
  • GPU serving advantages: higher throughput for large models, but higher cost, potential cold starts, and complexity in autoscaling.
  • Caching strategies: embedding caches, feature stores, and result caching to reduce computation.
  • Monitoring and autoscaling: track p99 latency, QPS, GPU/CPU utilization, and implement horizontal pod autoscaling.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q6

What's your approach to monitoring for data drift, model drift, and training/serving skew in production?

System DesignRoot Cause Analysis
Author's notes

I talked through statistical tests for distribution shift and threshold-based alerting.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining the three types of drift and skew clearly, then outline a layered monitoring strategy that covers data quality, model performance, and feature consistency. Emphasize proactive detection with automated alerts and a feedback loop for retraining, tailored to DoorDash's real-time logistics context.

Pro tip: Tie your monitoring approach to business impact—e.g., how detecting drift in delivery time predictions can prevent late deliveries and customer dissatisfaction. This shows you think beyond metrics to operational outcomes.

1. Define and Baseline

Clearly define data drift, model drift, and training/serving skew, and establish baseline statistics for features, predictions, and performance metrics from training and initial deployment.

2. Implement Monitoring Layers

Set up monitoring for input data distributions (e.g., PSI, KL divergence), model outputs (e.g., prediction distribution), and performance metrics (e.g., accuracy, latency) with automated alerts for deviations.

3. Detect Training/Serving Skew

Compare feature distributions and transformations between training and serving pipelines, using tools like TensorFlow Data Validation or custom checks, to catch inconsistencies early.

4. Root Cause Analysis and Response

When drift is detected, investigate causes (e.g., upstream data changes, concept drift) and trigger retraining or pipeline fixes, with a clear escalation path.

5. Iterate and Automate

Continuously refine thresholds and automate retraining pipelines, integrating monitoring with CI/CD and model registry for seamless updates.

Key Points to Mention

  • Statistical methods for drift detection (e.g., PSI, KL divergence, KS test)
  • Feature stores and their role in ensuring consistency between training and serving
  • Automated retraining triggers and model versioning
  • Monitoring tools (e.g., Prometheus, Grafana, Evidently AI, WhyLabs)
  • Business metrics alignment (e.g., delivery time accuracy, order completion rate)
  • Handling concept drift vs. data drift in a dynamic environment like DoorDash

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q7

How would you handle multi-tenant isolation and cost optimization in a shared ML infrastructure?

System DesignTechnical Trade-offs
Author's notes

Came at the end and I was running low on steam.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the requirements and constraints of the multi-tenant ML infrastructure, such as tenant scale, isolation levels, and cost drivers. Then propose a layered architecture that enforces isolation at compute, data, and network levels while leveraging resource sharing and elasticity for cost efficiency. Finally, discuss trade-offs between isolation and cost, and how to monitor and optimize continuously.

Pro tip: Emphasize that isolation and cost optimization are not mutually exclusive: use logical isolation with strong access controls and resource quotas to achieve both. Also, mention that cost optimization should be tenant-aware, e.g., chargeback/showback to encourage efficient usage.

1. Clarify Requirements and Constraints

Ask about tenant scale, isolation requirements (e.g., regulatory, security), workload types (training vs. inference), and cost constraints. This ensures your solution is tailored to DoorDash's context.

2. Design Isolation Layers

Propose isolation at multiple levels: compute (namespaces, virtual clusters, or dedicated nodes), data (separate storage buckets, encryption keys), and network (VPCs, service meshes). Choose the level based on tenant needs.

3. Implement Cost Optimization Strategies

Use resource sharing (e.g., Kubernetes with quotas), auto-scaling, spot instances, and model optimization (quantization, pruning). Implement chargeback to hold tenants accountable.

4. Balance Trade-offs and Monitor

Discuss trade-offs: stronger isolation may increase cost, while aggressive sharing may risk noisy neighbors. Set up monitoring for performance, cost, and isolation breaches, and iterate.

5. Summarize and Conclude

Recap how your approach meets both isolation and cost goals, and highlight any assumptions or next steps.

Key Points to Mention

  • Logical vs. physical isolation: namespaces, virtual clusters, dedicated instances
  • Resource quotas and limits to prevent noisy neighbors
  • Auto-scaling and spot instances for cost savings
  • Data isolation: encryption, access controls, separate storage
  • Chargeback/showback for cost accountability
  • Monitoring and observability for isolation and cost

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.