← Google Interview Insights

Google·Data Scientist·Technical Phone Screen·Senior

Senior
May 2026

Summary

A/B testing design question for a DS role at Google. The whole interview was basically this one problem stretched across the full session, and they wanted you to cover every angle without being prompted.

Questions Asked (1)

Q1

Design an A/B test to compare two search ranking algorithms. Walk through your hypothesis, primary and guardrail metrics, randomization unit, sample size and power, test duration, segmentation, novelty effects, and how you'd decide whether to ship.

A/B Testing & ExperimentationProduct Analytics & MetricsTechnical Trade-offs
Author's notes

This is the kind of question that looks manageable until you realize they want you to go deep on every single component, not just name-drop the concepts.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by framing the experiment around a clear hypothesis about how the new ranking algorithm improves user experience, then systematically cover each required element (metrics, randomization, power, duration, segmentation, novelty, ship decision) in a logical order. Emphasize trade-offs and justify choices with statistical reasoning and product context, showing you understand both the technical and business sides.

Pro tip: Mention that you'd pre-register the experiment design and decision criteria to avoid p-hacking, and discuss how to handle interference between units (e.g., via cluster randomization or interleaving) since search ranking often has network effects.

1. Define hypothesis and success criteria

State a clear, testable hypothesis about how the new ranking algorithm will improve user outcomes (e.g., higher relevance, more clicks) and specify the primary metric and guardrail metrics upfront.

2. Choose metrics and randomization unit

Select primary metrics (e.g., CTR, dwell time, success rate) and guardrails (e.g., latency, revenue, long-term satisfaction). Decide on the randomization unit (user, session, or query) considering interference and analysis feasibility.

3. Plan sample size, power, and duration

Calculate required sample size using power analysis for the primary metric, accounting for expected effect size and variance. Determine test duration to cover full weekly cycles and avoid novelty effects, while ensuring sufficient power.

4. Address segmentation and novelty effects

Predefine segments (e.g., new vs. returning users, query types) for heterogeneous treatment effect analysis. Mitigate novelty effects by running the test long enough and monitoring early vs. late behavior.

5. Analyze results and make ship decision

Use appropriate statistical tests (e.g., t-test, sequential testing) to evaluate primary and guardrail metrics. Consider practical significance, segment-level results, and long-term impact before deciding to ship, iterate, or abandon.

Key Points to Mention

  • Hypothesis should be specific and tied to user value (e.g., 'New algorithm increases successful search rate by 2%').
  • Primary metric should be sensitive and aligned with long-term goals; guardrails prevent regressions (e.g., latency, revenue).
  • Randomization unit choice: user-level avoids contamination but may need large samples; query-level increases power but risks interference.
  • Power analysis: specify minimum detectable effect, alpha, power (typically 80%), and account for multiple testing corrections.
  • Test duration: at least 1-2 weeks to capture weekly seasonality and allow novelty to fade; use holdout or long-term monitoring.
  • Ship decision: combine statistical significance with practical significance, check guardrails, and consider segment-level trade-offs; possibly run a follow-up experiment.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.