← Credit Karma Interview Insights

Credit Karma·Machine Learning Engineer·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say
Jun 2026

Summary

ML system design round at Credit Karma focused almost entirely on recommendation and ranking systems for their financial product marketplace. Heavy emphasis on multi-objective modeling, label sparsity, and architecture tradeoffs. Pretty deep interview, not a lot of fluff.

Questions Asked (10)

Q1

How would you design an ML system for Credit Karma's marketplace to maximize revenue?

System DesignPricing & MonetizationProduct Strategy
Author's notes

This was the anchor question that everything else branched off of.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the business objective and constraints, then outline a two-sided marketplace system that balances user experience with partner revenue. Propose a modular ML architecture covering ranking, pricing, and allocation, and discuss how to measure and optimize for long-term revenue without harming user trust.

Pro tip: Emphasize that revenue maximization must be constrained by user experience and fairness; propose guardrail metrics and A/B testing to avoid short-term gains that erode long-term trust.

1. Clarify Business Goals and Constraints

Ask questions to understand Credit Karma's marketplace dynamics: who are the users, partners, and what are the revenue streams? Identify constraints like user experience, regulatory compliance, and fairness.

2. Define ML Objectives and Metrics

Translate business goals into ML objectives: maximize revenue per user session while maintaining user satisfaction. Define primary metrics (e.g., revenue, conversion rate) and guardrail metrics (e.g., user retention, NPS).

3. Design System Architecture

Outline a modular system: candidate generation, ranking (e.g., learning to rank), pricing (e.g., bid optimization), and allocation (e.g., matching users to offers). Consider real-time and batch components.

4. Address Key Challenges

Discuss cold start, exploration vs exploitation, feedback loops, and fairness. Propose techniques like contextual bandits, counterfactual evaluation, and multi-task learning.

5. Evaluate and Iterate

Describe offline evaluation (replay, counterfactual) and online A/B testing with guardrails. Plan for monitoring, retraining, and incorporating business feedback.

Key Points to Mention

  • Two-sided marketplace dynamics: balancing user experience with partner revenue
  • Ranking and pricing models: learning to rank, bid shading, reserve pricing
  • Exploration vs exploitation: contextual bandits for new offers
  • Guardrail metrics: user satisfaction, retention, fairness, regulatory compliance
  • Feedback loops and counterfactual evaluation to avoid bias
  • Real-time serving and scalability considerations

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q2

Should you use a single model across the entire funnel or separate models for each stage like click, application, and approval?

Technical Trade-offsSystem Design
Author's notes

I argued for separate models and they pushed back a bit.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying that the answer depends on the specific funnel stages, data availability, and business objectives. Then discuss the trade-offs between a single model (simplicity, shared learning, less overfitting) and separate models (specialization, handling stage-specific patterns, better performance). Conclude with a recommendation for a hybrid approach, such as a multi-task model or shared embeddings with stage-specific heads, and emphasize the importance of experimentation and monitoring.

Pro tip: Frame your answer around business impact: Credit Karma cares about optimizing conversion and user experience, so highlight how your modeling choice affects metrics like approval rates and revenue. Also, mention that you would start simple and iterate based on A/B test results.

1. Clarify the funnel and objectives

Define the stages (click, application, approval) and the key metrics for each (CTR, application rate, approval rate). Understand the business goal, such as maximizing approvals or revenue.

2. Assess data and feature availability

Evaluate whether you have sufficient data per stage and if features are shared or stage-specific. Consider data sparsity and potential leakage across stages.

3. Compare single vs. separate models

Discuss pros and cons: single model offers simplicity, shared learning, and less overfitting; separate models allow specialization and better handling of stage-specific patterns but require more data and maintenance.

4. Propose a hybrid or multi-task approach

Suggest architectures like multi-task learning with shared layers and task-specific heads, or a two-stage model where the first stage predicts click and the second predicts approval given click.

5. Recommend experimentation and iteration

Emphasize the need for offline evaluation and online A/B testing to measure impact. Start with a simple baseline and iterate based on performance and business metrics.

Key Points to Mention

  • Trade-offs between bias and variance: single model may underfit stage-specific nuances, separate models may overfit due to limited data.
  • Data leakage and target leakage: ensure features used in early stages are available at prediction time for later stages.
  • Multi-task learning and shared representations: can leverage common patterns while allowing specialization.
  • Business metrics alignment: each stage may have different optimization objectives (e.g., click vs. approval).
  • Scalability and maintenance: separate models increase operational complexity and require more monitoring.
  • Cold start and data sparsity: separate models may struggle with rare events like approvals.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q3

How do you handle conversion labels that don't arrive until days or weeks after the original impression?

System DesignTechnical Trade-offs
Author's notes

Blanked for a second here.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Acknowledge the delayed feedback problem as a core challenge in online advertising and ML systems, then propose a solution that combines short-term proxy labels with long-term true labels, using techniques like importance weighting or delayed feedback models. Emphasize the trade-offs between model freshness and accuracy, and how you would validate the approach.

Pro tip: Mention that you would monitor the distribution shift between proxy and true labels over time and set up a feedback loop to retrain models as delayed labels arrive, ensuring the system adapts to changing user behavior.

1. Define the problem and impact

Explain how delayed conversion labels cause a mismatch between training and serving data, leading to biased models and suboptimal decisions. Quantify the delay distribution and its impact on business metrics.

2. Choose a modeling strategy

Select an approach such as using proxy labels (e.g., clicks, add-to-cart) for immediate training, or employing delayed feedback models like survival analysis or recurrent neural networks to incorporate delays.

3. Handle label delay in training

Apply techniques like importance weighting, label correction, or waiting windows to adjust for delayed labels. Consider online learning to update models as new labels arrive.

4. Evaluate and validate

Design offline and online evaluations that account for delayed labels, such as time-based cross-validation or using a holdout set with fully observed labels. Monitor for concept drift.

5. Iterate and monitor

Set up production monitoring to track the gap between proxy and true labels, and retrain models periodically. Use A/B testing to measure the impact of the chosen strategy.

Key Points to Mention

  • Delayed feedback problem in online advertising and recommendation systems
  • Proxy labels vs. true labels and their trade-offs
  • Importance weighting and inverse propensity scoring
  • Delayed feedback models (e.g., survival analysis, RNNs)
  • Online learning and incremental model updates
  • Evaluation challenges with delayed labels and concept drift

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q4

Conversion rates in this setting can be extremely low, around 0.1 to 0.5 percent. How do you deal with that kind of label sparsity?

Technical Trade-offsSystem Design
Author's notes

Oversampling, negative downsampling, focal loss, auxiliary tasks as proxies.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Acknowledge the challenge of extreme label sparsity and frame it as a problem of learning from highly imbalanced data. Discuss a combination of data-level, algorithm-level, and evaluation strategies, emphasizing that the right approach depends on the business objective and the cost of false positives/negatives. Highlight the importance of proper validation and metric selection.

Pro tip: Don't just list techniques; tie them to business impact. For example, explain how you'd optimize for precision at a fixed recall to minimize wasted outreach, and mention that you'd monitor for concept drift since conversion behavior can change.

1. Clarify the business objective and constraints

Understand what the model is used for: is it to rank users for targeting, or to predict conversion probability? This determines whether you need calibrated probabilities or just good ranking, and what false positive/negative costs are.

2. Choose appropriate evaluation metrics

Avoid accuracy; use metrics like PR-AUC, recall at high precision, or lift at top deciles. For ranking, use NDCG or precision@k. Ensure you use stratified sampling and proper cross-validation.

3. Apply data-level techniques

Consider oversampling the minority class (e.g., SMOTE), undersampling the majority, or using class weights. Be cautious: oversampling can lead to overfitting, so validate on original distribution.

4. Leverage algorithm-level approaches

Use models that handle imbalance well (e.g., tree-based ensembles with class weights, focal loss for neural networks). Consider anomaly detection or one-class classification if positives are extremely rare.

5. Validate and iterate with business metrics

Evaluate on a holdout set with the original distribution. Simulate business impact (e.g., expected profit) and iterate. Monitor for drift and retrain periodically.

Key Points to Mention

  • Class imbalance techniques: resampling, class weights, SMOTE, and their trade-offs.
  • Evaluation metrics: PR-AUC, recall at high precision, lift, and why accuracy is misleading.
  • Algorithm choices: tree-based models (XGBoost, LightGBM) with scale_pos_weight, focal loss, or anomaly detection.
  • Calibration: if probabilities are needed, use Platt scaling or isotonic regression after resampling.
  • Business impact: aligning model thresholds with business costs and benefits.
  • Monitoring and drift: conversion rates can change, so continuous evaluation is key.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q5

If you use negative sampling during training, how do you correct for that bias at inference time?

Technical Trade-offsAlgorithms & Data Structures
Author's notes

Classic calibration correction, you adjust the output probability by the inverse of the sampling rate.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Clarify that negative sampling introduces a bias by training on a modified distribution, and that correcting it requires importance weighting or calibration. Explain that at inference, you can either adjust scores using the known sampling probabilities or avoid the issue by using a full softmax if feasible. Emphasize that the correction depends on the specific model and loss function.

Pro tip: Mention that in practice, many systems ignore the bias because ranking is preserved, but for calibrated probabilities (e.g., in credit risk), you must apply the correction. This shows you understand both theory and business impact.

1. Define the bias

Explain that negative sampling changes the distribution of negatives, so the model learns P(positive | sampled context) instead of the true P(positive | full context).

2. Derive the correction

Show that the true probability can be recovered by multiplying the sampled probability by the ratio of sampling probabilities (importance weighting).

3. Apply at inference

Describe how to adjust scores: for each negative sample, weight by 1/(sampling probability), or use a calibration layer trained on a small unbiased set.

4. Consider practical alternatives

Mention that if only ranking matters, correction may be unnecessary; if calibrated probabilities are needed, use the correction or a post-hoc calibration method like Platt scaling.

5. Evaluate impact

Discuss how to validate the correction: compare corrected scores to a model trained without negative sampling on a held-out set, or measure calibration metrics.

Key Points to Mention

  • Negative sampling biases the model towards the sampled negatives, altering the learned probability distribution.
  • Importance weighting: multiply the predicted probability by the inverse of the sampling probability to correct the bias.
  • For ranking tasks, the bias often doesn't affect the order, so correction may be skipped.
  • For calibrated probabilities (e.g., in credit risk), correction is crucial; use a validation set to calibrate.
  • Alternative: use full softmax if computationally feasible, or use noise contrastive estimation (NCE) which inherently corrects.
  • Mention that the correction factor depends on the negative sampling distribution (e.g., uniform vs. popularity-based).

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q6

You're training on multiple objectives like CTR, conversion, and approval rate. How do you combine these into a single ranking score at the end?

System DesignPricing & MonetizationTechnical Trade-offs
Author's notes

This is where it got interesting.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by explaining that multiple objectives are combined into a single ranking score using a weighted sum or a learned model, but the weights should reflect business priorities and be tuned via online experiments. Emphasize that the combination must be calibrated and may involve constraints or guardrails to prevent one objective from dominating.

Pro tip: Mention that the weights are not static; they should be dynamically adjusted based on context (e.g., user segment, time of day) and business goals, and that you would use multi-objective optimization techniques like Pareto efficiency to balance trade-offs.

1. Clarify business objectives and constraints

Understand the relative importance of each objective (CTR, conversion, approval rate) and any hard constraints (e.g., approval rate must not drop below X%).

2. Choose a combination method

Select a method such as weighted linear sum, multiplicative utility, or a learned ranking model (e.g., LambdaMART with multiple objectives) based on complexity and interpretability needs.

3. Calibrate and normalize scores

Ensure each objective's score is on a comparable scale (e.g., via min-max scaling or probability calibration) so weights are meaningful and not biased by scale differences.

4. Tune weights via offline and online experiments

Use historical data to simulate and narrow down weight ranges, then run A/B tests to optimize for business metrics like revenue or user satisfaction.

5. Monitor and adapt

Continuously monitor performance and adjust weights or model as business goals evolve, using techniques like multi-armed bandits for dynamic weighting.

Key Points to Mention

  • Weighted sum is simple but assumes linear trade-offs; consider non-linear methods if interactions exist.
  • Normalization is crucial to prevent one objective from dominating due to scale.
  • Business constraints (e.g., approval rate floor) can be handled via penalty terms or constrained optimization.
  • Online A/B testing is essential to validate offline tuning and capture real-world effects.
  • Consider using a multi-task learning model to predict all objectives and then combine at serving time.
  • Dynamic weighting based on context can improve personalization and adapt to changing business needs.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q7

What is the actual objective function you'd use for the final ranking step?

Algorithms & Data StructuresSystem Design
Author's notes

I went with a weighted product of probabilities, something like pCTR times pCVR times expected value.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Clarify the business objective first—Credit Karma likely wants to maximize user engagement or financial action while balancing relevance and fairness. Then propose a concrete objective function, such as a weighted sum of predicted click-through rate and expected value of downstream actions, with constraints or regularization to prevent bias. Explain how you'd train and evaluate it, and mention trade-offs like exploration vs exploitation.

Pro tip: Show you understand that the 'final ranking step' often involves a re-ranking or business-rule layer on top of the ML model, so the objective function should incorporate business KPIs and fairness constraints, not just model loss.

1. Clarify the goal and constraints

Ask what the primary business metric is (e.g., clicks, conversions, revenue) and any fairness or regulatory constraints. This ensures the objective aligns with stakeholder needs.

2. Define the objective mathematically

Propose a formula, e.g., maximize sum over items of (w1 * pCTR + w2 * pCVR * value) subject to diversity or fairness constraints. Explain how weights are chosen.

3. Discuss training and optimization

Mention how you'd train the model (e.g., learning to rank with LambdaMART) and optimize the final objective, possibly using constrained optimization or post-processing.

4. Address evaluation and iteration

Describe offline metrics (NDCG, business KPIs) and online A/B tests. Emphasize monitoring for feedback loops and bias.

Key Points to Mention

  • Business objective alignment (e.g., maximizing user financial progress)
  • Multi-task learning: combining pCTR, pCVR, and other engagement signals
  • Weighted sum or constrained optimization to balance multiple objectives
  • Fairness and diversity constraints to avoid filter bubbles or discrimination
  • Offline evaluation metrics (NDCG, AUC) and online A/B testing
  • Exploration vs exploitation trade-off in ranking

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q8

When does it make sense to use a two-stage retrieval plus reranking setup versus just scoring all candidates directly?

System DesignTechnical Trade-offs
Author's notes

Scale is the obvious answer, you can't run a heavy model over millions of items in real time.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining the trade-off between latency and accuracy, then explain when a two-stage approach (fast retrieval + expensive reranking) is preferable to direct scoring. Use concrete examples like large candidate sets, complex scoring functions, or strict latency budgets to illustrate your reasoning.

Pro tip: Emphasize that the decision hinges on the cost ratio between retrieval and scoring, and that you should always measure the recall of the first stage to ensure the reranker has a chance to succeed.

1. Clarify the problem constraints

Identify the size of the candidate set, latency requirements, and the computational cost of the scoring function. This sets the stage for the trade-off analysis.

2. Compare direct scoring vs. two-stage

Explain that direct scoring is feasible when the candidate set is small or the scoring function is cheap; two-stage is beneficial when the candidate set is large and scoring is expensive.

3. Discuss the role of recall in retrieval

Highlight that the first stage must have high recall to avoid missing relevant items, and that reranking improves precision among the retrieved candidates.

4. Consider latency and throughput

Analyze how two-stage can reduce overall latency by limiting expensive scoring to a subset, but note the added complexity and potential for pipeline delays.

5. Provide a recommendation with metrics

Suggest evaluating both approaches with metrics like NDCG, latency, and cost, and choosing based on the specific application requirements.

Key Points to Mention

  • Candidate set size and its impact on computational cost
  • Latency constraints and real-time vs. batch processing
  • Cost and complexity of the scoring function (e.g., deep learning model vs. simple heuristic)
  • Recall of the first-stage retrieval and its effect on final quality
  • Trade-off between accuracy and efficiency
  • Use of approximate nearest neighbor (ANN) for retrieval and its parameters

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q9

Walk through the pros and cons of two-stage versus single-stage ranking architectures.

Technical Trade-offsSystem Design
Author's notes

Latency, scalability, recall tradeoffs versus model complexity and training simplicity.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining both architectures and their typical use cases, then systematically compare them across key dimensions like latency, scalability, and complexity. Use a concrete example (e.g., a recommendation system) to illustrate trade-offs, and conclude with when to choose each based on business and technical constraints.

Pro tip: Emphasize that the choice depends on the specific application's requirements—there's no one-size-fits-all. Mention that hybrid approaches (e.g., using a lightweight single-stage for some traffic and two-stage for others) can balance trade-offs, showing you think beyond binary choices.

1. Define the architectures

Briefly explain single-stage ranking (one model scores all items) and two-stage ranking (a candidate generation stage followed by a ranking stage). Clarify that two-stage is common in large-scale systems like search and recommendations.

2. Compare on key dimensions

Discuss pros and cons across dimensions such as latency, scalability, accuracy, complexity, and maintainability. For each dimension, highlight how each architecture performs.

3. Illustrate with an example

Walk through a concrete scenario (e.g., Credit Karma's recommendation engine) to show how each architecture would handle it, including trade-offs in real-time constraints and personalization.

4. Discuss when to choose each

Provide guidelines: single-stage for small item catalogs or low-latency needs; two-stage for large catalogs, complex ranking, and scalability. Mention hybrid approaches if relevant.

5. Conclude with a balanced summary

Summarize that the choice depends on specific requirements, and emphasize the importance of evaluating trade-offs in the context of the business goals and technical constraints.

Key Points to Mention

  • Latency and throughput: two-stage can reduce latency by pruning candidates early, but adds overhead of two models.
  • Scalability: two-stage scales better with large item catalogs (e.g., millions of items) by reducing the ranking set.
  • Accuracy and personalization: two-stage allows more complex ranking models on a smaller set, potentially improving relevance.
  • Complexity and maintenance: two-stage requires maintaining two models and a candidate generation strategy, increasing operational overhead.
  • Cold start and coverage: single-stage may handle new items better if the model generalizes, while two-stage may miss items not retrieved by the candidate generator.
  • Real-world examples: mention systems like YouTube or Amazon that use two-stage, and simpler systems that use single-stage.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q10

Under what conditions would you choose a two-stage architecture over a single-stage one?

Technical Trade-offsSystem DesignAdaptability & Ambiguity
Author's notes

Corpus size, latency constraints, and whether your retrieval recall is good enough.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying that the choice depends on the problem's complexity, data characteristics, and business constraints. Then, walk through the key trade-offs—such as modularity, data requirements, and latency—and give a concrete example where a two-stage approach is preferable, like when you need to separate retrieval and ranking in a recommendation system.

Pro tip: Emphasize that two-stage architectures are often chosen when you need to balance efficiency and accuracy: use a fast, lightweight first stage to filter a large candidate set, then a more complex second stage to refine the top candidates. This shows you understand real-world production systems.

1. Clarify the problem and constraints

Ask about the scale of data, latency requirements, and accuracy goals to ground your answer in the specific context.

2. Discuss single-stage limitations

Explain scenarios where a single-stage model struggles, such as handling large candidate sets or needing to optimize multiple objectives.

3. Highlight two-stage advantages

Describe how a two-stage approach can improve efficiency, allow specialized models, and handle complex tasks like retrieval and ranking.

4. Provide a concrete example

Illustrate with a real-world use case, such as a recommendation system where a candidate generator feeds a ranker.

5. Acknowledge trade-offs and alternatives

Mention that two-stage adds complexity and may not be needed for small-scale problems, showing balanced judgment.

Key Points to Mention

  • Scalability: two-stage handles large candidate sets by filtering first.
  • Latency: first stage is fast, second stage can be more complex but runs on fewer items.
  • Modularity: separate models can be developed and improved independently.
  • Data requirements: two-stage may need labeled data for both stages, but can leverage weak supervision.
  • Business impact: improved relevance and user engagement in systems like recommendations.
  • Complexity trade-off: two-stage increases system complexity and maintenance overhead.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.