← Pinterest Interview InsightsMachine Learning Engineer

Pinterest·Machine Learning Engineer·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say
Jun 2026

Summary

Pinterest ML Engineer interview, system design round focused entirely on fraud detection. Pretty deep dive, they wanted the full stack from problem framing through to feedback loops, not just a model architecture sketch.

Questions Asked (6)

Q1

Design a real-time fraud detection system for banking transactions, end to end.

System DesignTechnical Trade-offs
Author's notes

This is a big open-ended one and I spent probably too long on the problem framing before getting to anything concrete.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements and constraints (latency, scale, fraud types, data availability), then sketch a high-level architecture covering data ingestion, feature computation, model serving, and feedback loops. Dive into key components like streaming feature store, model selection (e.g., gradient boosting for tabular data), and online/offline consistency, while discussing trade-offs between precision/recall, latency, and cost.

Pro tip: Emphasize the importance of a feedback loop and monitoring for concept drift, as fraud patterns evolve rapidly; also mention how you'd handle class imbalance and label delay in training.

1. Clarify Requirements and Constraints

Ask about transaction volume, latency requirements (e.g., <100ms), fraud types (card-not-present, account takeover), available data (transaction metadata, user behavior), and regulatory constraints. Define success metrics like precision, recall, and false positive rate.

2. High-Level Architecture

Outline the end-to-end flow: data sources (transaction streams, user profiles), ingestion (Kafka), feature computation (streaming and batch), model training pipeline, real-time scoring service, and action layer (block/flag). Include storage for features (feature store) and model registry.

3. Feature Engineering and Data Pipeline

Describe how to compute features in real-time (e.g., sliding windows, aggregations) and ensure consistency between training and serving. Discuss handling of missing values, categorical encoding, and feature versioning.

4. Model Selection and Training

Choose models suitable for low-latency inference and tabular data (e.g., XGBoost, LightGBM, or a simple neural network). Address class imbalance (resampling, class weights), label delay (using semi-supervised or active learning), and retraining cadence.

5. Deployment, Monitoring, and Iteration

Explain deployment strategy (canary, A/B test), monitoring for drift and performance degradation, and feedback loops to capture confirmed fraud for retraining. Discuss trade-offs between model complexity and latency, and how to handle false positives.

Key Points to Mention

  • Streaming feature computation with exactly-once semantics and low latency (e.g., using Flink or Kafka Streams)
  • Feature store for online/offline consistency (e.g., Feast, Tecton)
  • Handling class imbalance and evaluation metrics beyond accuracy (precision-recall AUC, F1)
  • Model interpretability for regulatory compliance and debugging (SHAP, LIME)
  • Concept drift detection and automated retraining pipelines
  • Trade-offs between rule-based systems and ML models, and hybrid approaches

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q2

How would you handle class imbalance in a fraud detection model, and what trade-offs does that create between precision and recall?

Technical Trade-offsProduct Analytics & Metrics
Author's notes

Went fine until they asked what the actual business cost of a false positive is.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by acknowledging that fraud detection is a highly imbalanced classification problem where false negatives (missed fraud) are often costlier than false positives. Then discuss a range of techniques to handle imbalance, such as resampling, class weighting, and anomaly detection, and explain how each affects the precision-recall trade-off. Finally, emphasize that the optimal trade-off depends on business costs and should be tuned via threshold optimization and evaluated with metrics like PR-AUC.

Pro tip: Frame the trade-off in terms of business impact: quantify the cost of false positives (e.g., customer friction, manual review) versus false negatives (e.g., financial loss, chargebacks) to justify your choice of operating point. Mention that you would monitor the model in production and adjust thresholds as fraud patterns evolve.

1. Define the problem and costs

Clarify the business context: what is the cost of a false positive vs. false negative? This determines the target precision-recall balance.

2. Choose imbalance handling techniques

Discuss methods like resampling (SMOTE, undersampling), class weighting, or using anomaly detection algorithms. Explain pros and cons of each.

3. Select evaluation metrics

Avoid accuracy; use precision, recall, F1, PR-AUC, and cost-based metrics. Explain why PR-AUC is more informative than ROC-AUC for imbalanced data.

4. Tune decision threshold

After training, optimize the probability threshold to achieve the desired precision-recall trade-off based on business costs.

5. Monitor and iterate

Deploy with monitoring for drift and feedback loops; periodically re-evaluate the trade-off as fraud patterns and business costs change.

Key Points to Mention

  • Class imbalance techniques: resampling (SMOTE, undersampling), class weighting, anomaly detection, ensemble methods.
  • Precision-recall trade-off: increasing recall often decreases precision and vice versa; the optimal point depends on costs.
  • Evaluation metrics: PR-AUC, F1, precision@k, recall@k, and cost-sensitive metrics.
  • Threshold optimization: adjusting the decision threshold post-training to align with business objectives.
  • Business impact: quantifying costs of false positives (e.g., customer friction) and false negatives (e.g., financial loss).
  • Production considerations: monitoring, retraining, and adapting to evolving fraud patterns.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q3

What features would you engineer for a fraud detection model, specifically around velocity, graph relationships, and device signals?

System DesignData Modeling
Author's notes

Graph features over account-merchant relationships was the part I fumbled.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by framing fraud detection as a real-time classification problem where feature engineering must balance predictive power with low-latency computation. Then organize your answer around the three requested signal families—velocity, graph, and device—explaining how each captures different fraud patterns and how they complement each other. Finally, discuss how you would validate and productionize these features, including handling data leakage and ensuring scalability.

Pro tip: Emphasize that velocity features must be computed with strict time-windowing and point-in-time correctness to avoid leakage, and mention that graph features often require approximate or sampled computations to meet latency budgets. This shows you understand the trade-offs between accuracy and real-time serving.

1. Clarify the problem and constraints

Ask about the fraud types (e.g., account takeover, fake engagement, spam), data volume, and latency requirements. This ensures your feature choices align with business goals and system constraints.

2. Engineer velocity features

Propose time-windowed aggregates (e.g., counts, sums, averages) over user actions like logins, posts, follows, and messages. Include rate-of-change and acceleration features to capture sudden bursts.

3. Engineer graph-based features

Describe features derived from the user-item or user-user interaction graph, such as degree, clustering coefficient, PageRank, and community detection metrics. Highlight how these reveal coordinated fraud rings.

4. Engineer device and session signals

List device fingerprinting features (OS, browser, IP, device ID), session-based features (session length, clickstream patterns), and anomalies like multiple accounts per device or rapid device switching.

5. Address validation and productionization

Explain how to avoid leakage with point-in-time joins, use time-based splits, and monitor feature drift. Discuss computational efficiency (e.g., streaming aggregations, approximate graph algorithms) for real-time serving.

Key Points to Mention

  • Time-windowed aggregations (e.g., 1h, 24h, 7d) with exponential decay or sliding windows for velocity features.
  • Graph features like degree centrality, connected components, and label propagation to detect fraud rings.
  • Device fingerprinting and anomaly detection (e.g., impossible travel, device sharing) as strong fraud signals.
  • Point-in-time correctness and avoiding target leakage when computing historical features.
  • Scalability considerations: using approximate algorithms (e.g., HyperLogLog for cardinality) and streaming pipelines (e.g., Flink, Spark Streaming).
  • Feature importance and interpretability to explain model decisions to trust and safety teams.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q4

How do you ensure consistency between your offline training features and the features served at inference time?

System DesignTechnical Trade-offs
Author's notes

Feature store question.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining training-serving skew and its impact on model performance. Then describe a comprehensive strategy that includes a shared feature engineering library, a feature store, and rigorous validation. Emphasize trade-offs between consistency, latency, and development velocity.

Pro tip: Highlight the importance of logging serving features and joining them with training data to detect skew, and mention that at Pinterest scale, a feature store like Feast or Tecton is essential for consistency.

1. Define the Problem

Explain what training-serving skew is and why it matters, including examples of how it can degrade model performance in production.

2. Shared Feature Engineering

Describe using a common codebase or library for feature computation in both training and serving, ensuring identical logic and transformations.

3. Feature Store

Introduce a feature store that centralizes feature definitions, provides point-in-time correctness, and serves features consistently for training and inference.

4. Validation and Monitoring

Discuss techniques like logging serving features, comparing distributions, and setting up alerts for skew detection in production.

5. Trade-offs and Best Practices

Address trade-offs between consistency, latency, and complexity, and share best practices like versioning and testing.

Key Points to Mention

  • Training-serving skew and its consequences
  • Shared feature engineering code (e.g., using Python libraries or SQL)
  • Feature store (e.g., Feast, Tecton, or custom) for consistency and point-in-time correctness
  • Logging and comparing serving features with training features
  • Monitoring and alerting for feature drift and skew
  • Trade-offs: latency vs. consistency, complexity vs. maintainability

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q5

How would you set up the feedback loop from analyst fraud reviews back into model retraining?

System DesignA/B Testing & Experimentation
Author's notes

Honestly the part I found most interesting.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by describing the end-to-end pipeline: how analyst fraud review labels are captured, stored, and fed into the training data. Then explain how you would validate and prioritize these labels for retraining, and finally how you would measure the impact of retraining on model performance and business metrics.

Pro tip: Emphasize the importance of closing the loop quickly and safely: use a shadow deployment or canary to validate new models before full rollout, and track analyst agreement rates to ensure label quality.

1. Capture and store analyst feedback

Design a system where analysts can easily label fraud cases (e.g., confirmed fraud, false positive) and store these labels in a structured database with metadata (timestamp, analyst ID, confidence).

2. Curate and prioritize retraining data

Aggregate analyst labels, deduplicate, and weight recent or high-confidence labels more heavily. Ensure a balanced dataset by including both fraud and non-fraud examples.

3. Trigger and execute retraining

Set up automated retraining pipelines triggered by data volume thresholds or performance degradation. Use versioning for datasets and models to ensure reproducibility.

4. Validate and deploy updated model

Evaluate the retrained model offline on a holdout set and online via A/B test or shadow mode. Monitor key metrics like precision, recall, and analyst workload.

5. Monitor and iterate

Continuously track model performance and analyst feedback to detect drift. Close the loop by feeding new analyst reviews back into the next retraining cycle.

Key Points to Mention

  • Label quality and analyst agreement metrics
  • Data versioning and reproducibility
  • Automated retraining triggers (e.g., data drift, performance drop)
  • Offline evaluation metrics (precision, recall, AUC) and online A/B testing
  • Feedback loop latency and its impact on fraud detection
  • Scalability and infrastructure considerations (e.g., feature store, pipeline orchestration)

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q6

How would you monitor for model drift in a fraud detection system, given that fraudsters actively adapt their behavior?

Technical Trade-offsRoot Cause Analysis
Author's notes

Short answer: adversarial drift is different from natural distribution shift and most standard drift detection methods don't catch it fast enough.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by acknowledging that fraud is adversarial, so drift monitoring must be continuous and multi-faceted. Propose a layered monitoring system that tracks data drift, concept drift, and model performance, with automated alerts and a feedback loop for retraining. Emphasize the need to balance detection sensitivity with false positive costs in a production environment.

Pro tip: Frame drift monitoring as a product decision, not just a technical one: tie alert thresholds to business impact (e.g., fraud loss vs. user friction) and propose a shadow deployment to test retrained models before full rollout.

1. Define drift types and metrics

Distinguish between data drift (changes in input feature distributions) and concept drift (changes in the relationship between features and fraud label). Choose metrics like PSI, KL divergence, and performance metrics (precision, recall, AUC) that are robust to class imbalance.

2. Set up real-time monitoring pipelines

Implement streaming or batch pipelines to compute drift metrics on a rolling window (e.g., daily or hourly). Use tools like Evidently, WhyLabs, or custom dashboards to visualize trends and trigger alerts when thresholds are breached.

3. Incorporate delayed labels and proxy signals

Since fraud labels are delayed, use proxy signals like chargebacks, user reports, or rule-based flags to estimate performance. Monitor these proxies for sudden changes that indicate drift.

4. Automate retraining and validation

When drift is detected, trigger retraining with recent data. Validate the new model on a holdout set and via A/B test or shadow mode to ensure it improves fraud detection without increasing false positives.

5. Close the feedback loop with investigation

Route alerts to fraud analysts for root cause analysis. Use their feedback to refine features, thresholds, and the monitoring system itself, creating an adaptive loop.

Key Points to Mention

  • Adversarial nature of fraud: fraudsters adapt, so drift is expected and monitoring must be proactive.
  • Data drift vs. concept drift: different detection methods and implications for model retraining.
  • Metrics: PSI, KL divergence, and performance metrics like precision-recall AUC, with attention to class imbalance.
  • Delayed label problem: use proxies and weak supervision to get timely performance estimates.
  • Automated retraining pipeline with human-in-the-loop validation to avoid overfitting to noise.
  • Business impact: align drift thresholds with cost of false positives vs. false negatives.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.