← Datadog Interview Insights

Datadog·Machine Learning Engineer·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say
Jun 2026

Summary

Datadog ML engineer round focused entirely on one meaty system design problem about monitoring web server latency at scale. No fluff, no behavioral warmup, just straight into the weeds on alerting pipelines and on-call policy design. Walked out unsure if I covered enough of the human-in-the-loop angle.

Questions Asked (1)

Q1

Design a system to monitor latency across dozens of web servers (each reporting average response time every second), detect degraded performance, and decide when to page an on-call engineer. Include human-in-the-loop controls like ack, snooze, and re-page intervals, and discuss how you'd tune the policy to balance alert fatigue against missed regressions.

System DesignTechnical Trade-offsProduct Analytics & Metrics
Author's notes

This is a big question and I did not scope it well at first.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements and scale, then propose a streaming architecture that ingests per-second metrics, computes robust baselines, and applies multi-window anomaly detection. Describe the alerting state machine with human-in-the-loop controls, and finish by discussing policy tuning using precision/recall and feedback loops to balance alert fatigue and missed regressions.

Pro tip: Emphasize that alerting is a product with its own SLOs—measure alert precision and time-to-acknowledge, and treat every page as a data point for continuous tuning. Also, mention that you'd start with simple statistical methods (e.g., EWMA, median absolute deviation) before jumping to complex ML, as interpretability and fast iteration matter in on-call systems.

1. Clarify Requirements and Scale

Ask about the number of servers, expected latency distribution, acceptable false positive/negative rates, and existing monitoring stack. Confirm that each server reports average response time every second, and that we need to detect degradation and page on-call.

2. Design Data Ingestion and Baseline Computation

Propose a scalable ingestion pipeline (e.g., Kafka) to collect metrics, and compute per-server baselines using rolling statistics (e.g., EWMA, median, MAD) to handle diurnal patterns and noise. Consider aggregating across servers for fleet-wide anomalies.

3. Implement Anomaly Detection and Alerting Logic

Define detection rules: e.g., latency exceeds baseline by X MADs for Y consecutive seconds, or a sudden spike. Use multi-window (short and long) to reduce false positives. Then design the alerting state machine with states: OK, PENDING, ALERTING, ACKED, SNOOZED, RESOLVED.

4. Incorporate Human-in-the-Loop Controls

Detail how ack, snooze, and re-page intervals work: ack stops re-paging until resolved; snooze temporarily suppresses alerts for a duration; re-page interval escalates if unacknowledged. Ensure these actions are logged and fed back into tuning.

5. Tune Policy and Balance Trade-offs

Discuss tuning thresholds, windows, and re-page intervals using historical data and feedback. Use metrics like precision, recall, MTTD, MTTA, and alert volume. Propose A/B testing or simulation to find the sweet spot between alert fatigue and missed regressions.

Key Points to Mention

  • Use robust statistical baselines (e.g., median absolute deviation) to handle outliers and non-normal latency distributions.
  • Implement multi-window detection (e.g., short window for spikes, long window for gradual degradation) to reduce false positives.
  • Design an alert state machine with clear transitions and idempotent actions to avoid duplicate pages.
  • Incorporate human feedback (ack, snooze) to dynamically adjust thresholds and suppression rules.
  • Define and monitor alerting SLOs (e.g., precision > 90%, MTTD < 1 min) to quantify alert fatigue.
  • Consider fleet-wide aggregation to detect correlated anomalies that individual server checks might miss.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.