← DoorDash Interview InsightsMachine Learning Engineer
I started with data ingestion and worked forward, which felt logical but probably wasted time on stuff they didn't care about.
Start by clarifying the product requirements and scale (e.g., number of users, requests per second, data volume) to ground your design. Then walk through the ML lifecycle: data ingestion, feature engineering, training, deployment, and monitoring, highlighting key components and trade-offs at each stage. Emphasize scalability, reliability, and how you would iterate quickly while maintaining low latency for inference.
Pro tip: Anchor your design around a concrete DoorDash use case (e.g., delivery time prediction or search ranking) to make it tangible, and proactively discuss trade-offs like batch vs. online inference and model freshness vs. cost.
Ask questions to understand the product, data volume, latency requirements, and update frequency. This ensures your design is tailored to the specific use case.
Describe how data is ingested, processed, and stored for training and serving. Include batch and streaming pipelines, feature stores, and data versioning.
Outline the training workflow: experiment tracking, distributed training, hyperparameter tuning, and model registry. Discuss how to handle large datasets and retraining.
Explain how models are deployed for online and batch inference, including serving infrastructure, scaling, and latency optimization. Mention A/B testing and canary deployments.
Cover monitoring for data drift, model performance, and system health. Describe feedback loops for continuous improvement and retraining.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by clarifying the requirements: online serving needs sub-10ms latency and high availability, while offline training needs point-in-time correctness and scalability. Then propose a dual-store architecture with a unified feature definition layer, and discuss trade-offs around consistency, cost, and complexity.
Pro tip: Emphasize point-in-time correctness for offline training to avoid data leakage, and mention that DoorDash's delivery logistics require features like real-time courier location and restaurant prep time, so the online store must handle high-throughput updates.
Ask about latency SLAs, data volume, feature freshness, consistency needs, and existing infrastructure. This shows you understand that design depends on specific use cases.
Propose a central feature registry with versioned definitions and transformation logic that can be applied both online (streaming) and offline (batch). This ensures consistency between training and serving.
Use a low-latency online store (e.g., Redis, DynamoDB) for serving and a scalable offline store (e.g., data lake, warehouse) for training. Discuss how data is ingested and kept in sync.
Explain how to generate training datasets with time-travel queries to avoid leakage, and how to backfill features for new models. Mention the need for event timestamps and versioning.
Cover consistency models (e.g., eventual vs. strong), cost, monitoring, and failure modes. Highlight how you would handle feature drift and ensure low-latency updates.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by framing the experimentation lifecycle: define metrics and hypotheses, then choose the right deployment strategy (A/B test, shadow, canary) based on risk and goals. Walk through each method with concrete examples from your experience, emphasizing how you monitor, analyze, and roll back. Conclude with how you'd apply this at DoorDash to balance speed and reliability.
Pro tip: Always tie experiments to business metrics (e.g., delivery time, order conversion) and mention guardrail metrics to catch regressions. Show you understand that not every change needs a full A/B test—shadow and canary are for risk mitigation, not causal inference.
Clarify the hypothesis, primary success metrics, and guardrail metrics before choosing an experiment type. Ensure metrics align with business objectives like conversion or delivery efficiency.
Choose A/B testing for causal impact, shadow deployment for safe validation without user impact, or canary rollout for gradual exposure. Consider risk, traffic volume, and need for statistical power.
Set up logging, dashboards, and alerts for real-time monitoring. For A/B tests, ensure proper randomization and sample size; for canary, define success criteria and progression gates.
Use statistical tests to evaluate A/B results, check guardrails, and decide to ship, iterate, or rollback. For shadow/canary, compare against baseline and assess operational metrics.
Have a clear rollback plan with automated triggers (e.g., error rate spike). Post-mortem to capture learnings and improve future experiments.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Short answer: seed everything, version your data snapshots, pin dependencies, log all hyperparameters.
Start by defining reproducibility in the ML context—same data, code, and environment yielding the same model—then outline a layered strategy covering data, code, environment, and experiment tracking. Emphasize practical trade-offs and how you'd implement this at scale, referencing tools like Docker, Git, and MLflow.
Pro tip: Mention the importance of capturing non-determinism sources (e.g., GPU operations, random seeds) and how you balance strict reproducibility with iteration speed—showing you understand real-world constraints.
Clarify what level of reproducibility is needed (exact, statistical, or directional) and the associated costs. This sets the stage for trade-off discussions.
Use Git for code, DVC or similar for data, and semantic versioning for models. Ensure all artifacts are traceable to a specific commit.
Use Docker or Conda to freeze dependencies, including OS libraries and CUDA versions. Store images in a registry for reuse.
Set random seeds for Python, NumPy, and frameworks; configure deterministic algorithms where possible. Document any non-deterministic operations.
Log parameters, metrics, and artifacts with tools like MLflow or Weights & Biases. Link runs to code, data, and environment versions.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
CPU vs GPU is one of those questions where the answer is genuinely 'it depends' but you can't just say that.
Start by clarifying the requirements: what model, what scale (QPS), and what latency SLO (p99 vs p50). Then propose a layered serving architecture that addresses model optimization, infrastructure choices, and dynamic batching, and discuss CPU vs GPU tradeoffs in terms of latency, throughput, cost, and operational complexity.
Pro tip: Emphasize that sub-100ms latency is often achievable with optimized CPU serving for small models, and GPUs are only necessary for large models or high throughput; always quantify tradeoffs with concrete numbers (e.g., cost per million inferences) to show business awareness.
Ask about model size, QPS, latency SLO (p50/p99), and cost constraints. This ensures your design is tailored to the actual problem.
Discuss techniques like quantization, pruning, distillation, and compilation (e.g., TensorRT, ONNX Runtime) to reduce model size and latency.
Propose a layered architecture: load balancer, model server (e.g., Triton, TorchServe), dynamic batching, caching, and autoscaling. Mention edge vs centralized serving.
Compare CPU and GPU in terms of latency, throughput, cost, scalability, and ease of deployment. Highlight when each is appropriate.
Explain how to monitor latency, throughput, and errors, and how to continuously optimize based on production data.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
I talked through statistical tests for distribution shift and threshold-based alerting.
Start by defining the three types of drift and skew clearly, then outline a layered monitoring strategy that covers data quality, model performance, and feature consistency. Emphasize proactive detection with automated alerts and a feedback loop for retraining, tailored to DoorDash's real-time logistics context.
Pro tip: Tie your monitoring approach to business impact—e.g., how detecting drift in delivery time predictions can prevent late deliveries and customer dissatisfaction. This shows you think beyond metrics to operational outcomes.
Clearly define data drift, model drift, and training/serving skew, and establish baseline statistics for features, predictions, and performance metrics from training and initial deployment.
Set up monitoring for input data distributions (e.g., PSI, KL divergence), model outputs (e.g., prediction distribution), and performance metrics (e.g., accuracy, latency) with automated alerts for deviations.
Compare feature distributions and transformations between training and serving pipelines, using tools like TensorFlow Data Validation or custom checks, to catch inconsistencies early.
When drift is detected, investigate causes (e.g., upstream data changes, concept drift) and trigger retraining or pipeline fixes, with a clear escalation path.
Continuously refine thresholds and automate retraining pipelines, integrating monitoring with CI/CD and model registry for seamless updates.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Came at the end and I was running low on steam.
Start by clarifying the requirements and constraints of the multi-tenant ML infrastructure, such as tenant scale, isolation levels, and cost drivers. Then propose a layered architecture that enforces isolation at compute, data, and network levels while leveraging resource sharing and elasticity for cost efficiency. Finally, discuss trade-offs between isolation and cost, and how to monitor and optimize continuously.
Pro tip: Emphasize that isolation and cost optimization are not mutually exclusive: use logical isolation with strong access controls and resource quotas to achieve both. Also, mention that cost optimization should be tenant-aware, e.g., chargeback/showback to encourage efficient usage.
Ask about tenant scale, isolation requirements (e.g., regulatory, security), workload types (training vs. inference), and cost constraints. This ensures your solution is tailored to DoorDash's context.
Propose isolation at multiple levels: compute (namespaces, virtual clusters, or dedicated nodes), data (separate storage buckets, encryption keys), and network (VPCs, service meshes). Choose the level based on tenant needs.
Use resource sharing (e.g., Kubernetes with quotas), auto-scaling, spot instances, and model optimization (quantization, pruning). Implement chargeback to hold tenants accountable.
Discuss trade-offs: stronger isolation may increase cost, while aggressive sharing may risk noisy neighbors. Set up monitoring for performance, cost, and isolation breaches, and iterate.
Recap how your approach meets both isolation and cost goals, and highlight any assumptions or next steps.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.