← Meta Interview InsightsMachine Learning Engineer

Meta·Machine Learning Engineer·Technical Phone Screen·Senior

Senior
Jun 2026

Summary

Meta MLE interview focused on a pretty meaty ML system design question around feature engineering for the News Feed ranking model. The follow-up about offline vs online metric gaps was where things got interesting and honestly a bit uncomfortable.

Questions Asked (4)

Q1

There's an existing model predicting the probability a user likes a News Feed post. How would you add a new feature capturing whether the user liked posts from a specific author (or a specific post) in the last 6 months? Walk through feature engineering, how you'd serve it at inference time, and how you'd avoid label leakage during training.

System DesignTechnical Trade-offsData Modeling
Author's notes

This is the kind of question where I started confident and slowly realized I was glossing over the hard parts.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Frame your answer around three distinct phases: feature engineering (defining the signal and its variants), online serving (how to compute and cache the feature at low latency), and training integrity (preventing label leakage with strict temporal splits). Ground every decision in real-world constraints like latency budgets, data freshness, and storage costs that Meta-scale systems face.

Pro tip: Explicitly distinguish between author-level and post-level engagement signals and discuss how you'd handle cold-start (new authors or posts with no history) — interviewers at Meta love seeing candidates think about edge cases and graceful degradation rather than just the happy path.

1. Define the Feature Variants

Identify the two distinct signals: (a) user–author affinity — did the user like posts from this author in the last 6 months, and (b) user–post affinity — did the user previously interact with this specific post. Decide on raw counts, binary flags, and normalized/decayed versions (e.g., exponential time-decay to weight recent likes more heavily).

2. Feature Engineering & Storage

Precompute user–author like counts over a rolling 6-month window and store them in a low-latency key-value store (e.g., ZippyDB or a Redis-like cache) keyed by (user_id, author_id). For the post-level signal, maintain a user–post interaction log and look up at serving time; consider bucketing counts (0, 1–5, 6–20, 20+) to reduce cardinality and improve generalization.

3. Online Serving & Latency

At inference time, perform a point lookup in the precomputed store using the (user_id, author_id) or (user_id, post_id) key, targeting sub-10ms p99 latency. Discuss a fallback strategy for cache misses — either a default value (0 or global mean) or a lightweight real-time aggregation from a stream processor like Flink.

4. Avoiding Label Leakage During Training

Use strict temporal splits: the feature value must be computed using only interactions that occurred before the label timestamp (i.e., the moment the post was shown). Implement point-in-time correct feature generation by replaying the historical interaction log up to each training example's request_time, ensuring the 6-month window is anchored to that moment and not to the dataset creation date.

5. Validation & Monitoring

Measure offline AUC lift from adding the feature and run an ablation to confirm it adds signal beyond existing engagement features. Post-launch, monitor feature coverage (% of requests with a non-null value), distribution drift over time, and whether the feature's SHAP importance remains stable — a sudden spike could indicate a data pipeline bug or leakage.

Key Points to Mention

  • Point-in-time correctness: feature values during training must only use data available before the label event to prevent leakage
  • Time-decay weighting (e.g., exponential decay) to make the 6-month window more nuanced than a flat count
  • Precomputed vs. real-time feature trade-offs: batch precomputation for low latency vs. streaming for freshness (e.g., Apache Flink or Spark Structured Streaming)
  • Cold-start handling for new authors or users with no interaction history — default values, global priors, or fallback to content-based signals
  • Storage and scalability considerations at Meta scale: billions of (user, author) pairs, TTL-based eviction, and memory footprint
  • Feature normalization and bucketing to handle power-law distributions in like counts and improve model robustness

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q2

Once you have that feature, how do you integrate it into the existing model? Should it go in as a raw numeric feature, get embedded, or be constructed as an interaction term with something else? What's the risk to existing features?

Technical Trade-offsProduct Analytics & Metrics
Author's notes

Spent probably too long on the embedding angle when the interviewer seemed more interested in the interaction feature idea.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Frame your answer around a principled decision tree: start by understanding the feature's distribution and relationship to the target, then choose the integration strategy (raw, embedded, or interaction) based on that analysis. Explicitly address the risks to existing features like multicollinearity, feature importance dilution, and training instability, showing you think holistically about the model ecosystem rather than just the new feature in isolation.

Pro tip: At Meta scale, mention shadow scoring or a staged rollout where the new feature is logged and evaluated offline before being introduced to the live model — this signals you understand production ML risk management and not just modeling theory.

1. Characterize the Feature

Analyze the feature's distribution (skewness, sparsity, cardinality), its correlation with the target, and its relationship to existing features. This diagnostic step determines which integration strategy is appropriate and surfaces potential risks upfront.

2. Choose Integration Strategy

Decide between raw numeric (for dense, well-scaled continuous features), embedding (for high-cardinality categoricals or sparse IDs), or interaction terms (when the feature's signal is conditional on another feature). Justify the choice with the feature's statistical properties and the model architecture in use.

3. Assess Risk to Existing Features

Evaluate multicollinearity with existing features, potential for feature importance redistribution, and whether the new feature could dominate gradients and destabilize training. Run a correlation matrix and variance inflation factor (VIF) analysis as a baseline check.

4. Validate Offline Before Deployment

Run controlled offline experiments comparing model performance with and without the new feature using held-out data, checking both aggregate metrics (AUC, NDCG) and slice-level performance to ensure no regression on key user segments.

5. Stage the Online Rollout

Deploy via shadow scoring or a small A/B test to monitor live metric impact, feature value distributions in production, and any training-serving skew before full rollout. Define clear rollback criteria in advance.

Key Points to Mention

  • Feature cardinality and sparsity as primary drivers of the raw vs. embedding decision — high-cardinality IDs (user, item) almost always warrant embeddings in deep learning pipelines
  • Interaction terms are valuable when a feature's effect is non-additive and conditional, e.g., 'time since last action' interacting with 'session length', but add model complexity and should be motivated by domain knowledge
  • Multicollinearity risk: a new feature highly correlated with an existing one can destabilize coefficients in linear models or cause redundant splits in tree models, diluting both features' measured importance
  • Training-serving skew: ensure the feature is available at inference time with the same latency and freshness guarantees as during training, especially for real-time ranking systems at Meta
  • Normalization and scaling: raw numeric features should be normalized (log-transform for heavy-tailed distributions, z-score for Gaussian-like) to prevent gradient dominance in neural models
  • Offline-to-online gap: validate that offline metric gains translate to online gains by monitoring proxy metrics in a live experiment, since feature interactions in production can differ from historical data

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q3

How would you validate whether this feature actually helps? What offline metrics would you look at, and how would you slice the analysis?

Product Analytics & MetricsA/B Testing & Experimentation
Author's notes

Pretty standard ground here.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by anchoring the answer to the product goal the feature is meant to achieve, then map that goal to measurable offline proxy metrics before discussing how to slice the analysis for robustness. Demonstrate that you understand the gap between offline metrics and real-world impact, and show awareness of potential pitfalls like metric gaming or Simpson's paradox in slicing.

Pro tip: At Meta scale, interviewers love when candidates proactively mention that a single aggregate metric can mask harm to specific user cohorts — explicitly calling out fairness slices (e.g., new vs. returning users, mobile vs. desktop, geographic regions) signals senior-level thinking and awareness of responsible ML practices.

1. Clarify the Feature's Goal

Restate what the feature is trying to optimize — engagement, relevance, retention, or something else — so your metric choices are clearly motivated. This shows structured thinking and avoids proposing metrics in a vacuum.

2. Define Primary Offline Metrics

Identify the core offline metrics that serve as proxies for the business goal, such as AUC-ROC, NDCG, precision@k, or calibration error depending on the task type. Explain why each metric is a valid proxy and acknowledge its limitations.

3. Add Guardrail & Secondary Metrics

Propose guardrail metrics to ensure the feature doesn't degrade other important dimensions — for example, latency, coverage, or diversity — even if the primary metric improves. This demonstrates awareness that optimizing one metric can harm others.

4. Design the Slicing Strategy

Describe how you would segment the analysis across meaningful dimensions: user cohorts (new vs. power users), content types, device/platform, geographic regions, and time windows. Explain that slicing catches cases where aggregate gains hide losses for specific subgroups.

5. Validate Offline-to-Online Correlation

Discuss how you would sanity-check that your offline metrics historically correlate with online A/B test outcomes, and outline the plan to move from offline validation to a live experiment. Mention shadow scoring or replay evaluation as intermediate steps if applicable.

Key Points to Mention

  • Connecting offline metrics to the specific product or business objective (e.g., relevance → NDCG, click prediction → AUC/logloss)
  • Calibration and ranking metrics as complementary — a model can rank well but be poorly calibrated, which matters for downstream systems
  • Guardrail metrics to detect regressions in latency, coverage, or content diversity
  • Cohort slicing: new vs. returning users, high vs. low activity users, and geographic or demographic segments to surface fairness concerns
  • Temporal slicing to detect concept drift or performance degradation over time windows
  • Offline-to-online metric correlation as a prerequisite for trusting offline evaluation, and the plan to bridge to A/B testing

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q4

Say your offline metrics improve but the online A/B test shows no meaningful gain. What are the possible explanations and how do you debug it?

A/B Testing & ExperimentationRoot Cause AnalysisProduct Analytics & Metrics
Author's notes

This is where I felt the pressure.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Structure your answer by first acknowledging that offline-online metric gaps are common and expected, then systematically walk through the possible root causes across data, model, and experimentation dimensions. Demonstrate a methodical debugging process that shows both technical depth and product intuition, concluding with how you'd prioritize investigations and make a ship/no-ship decision.

Pro tip: Mentioning 'surrogate metric misalignment' — where your offline metric (e.g., AUC, NDCG) doesn't faithfully proxy the true business metric (e.g., long-term engagement, revenue) — signals senior-level thinking and is a very common real-world culprit at companies like Meta.

1. Validate the Experiment Setup

First rule out instrumentation and experimental design issues: check for sample ratio mismatch (SRM), ensure proper randomization, verify that logging is consistent between control and treatment, and confirm the experiment ran long enough to avoid novelty effects or insufficient power.

2. Examine the Metric Gap (Offline vs. Online)

Investigate whether the offline metric is a faithful proxy for the online business metric — misalignment here is a primary culprit. Check if the offline evaluation dataset is representative of live traffic distribution, and whether label quality or feedback loops differ between offline and online settings.

3. Diagnose Model & Feature Issues in Production

Confirm the model is actually being served correctly by checking prediction distributions, feature value distributions, and model version in production logs. Look for training-serving skew, where features computed offline differ from how they're computed at serving time.

4. Analyze Segment-Level and Metric Decomposition

Break down the online results by user segments, surfaces, or content types to see if gains exist in subpopulations but are diluted in aggregate. Also check if the primary metric is moving but secondary or guardrail metrics are offsetting it, masking real impact.

5. Synthesize Findings and Decide Next Steps

Summarize the most likely root cause with supporting evidence and propose a concrete action: fix the offline metric, iterate on the model, extend the experiment, or redesign the evaluation protocol. Communicate the tradeoffs clearly to stakeholders before making a ship/no-ship recommendation.

Key Points to Mention

  • Sample Ratio Mismatch (SRM) and experiment validity checks as a first debugging step
  • Training-serving skew — differences in feature computation between offline training and online inference
  • Surrogate metric misalignment — offline metrics (AUC, NDCG, RMSE) not faithfully representing the true online business objective
  • Dataset distribution shift — offline evaluation data not reflecting live traffic patterns, user behavior, or seasonality
  • Novelty effect and insufficient statistical power — experiment may need more time or traffic to detect real but small effects
  • Segment-level analysis — gains may exist in specific cohorts but be diluted or counteracted in aggregate metrics

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.