← Credit Karma Interview Insights

Credit Karma·Machine Learning Engineer·Technical Phone Screen·Senior

Senior
May 2026

Summary

Went through a technical phone screen for an ML Engineer role at Credit Karma. The questions were pretty deep on training dynamics and optimization, less about standard ML theory and more about what you'd actually hit in production.

Questions Asked (3)

Q1

What are the likely causes of a multi-task model's loss going NaN or becoming unstable after an initial period of decrease?

Root Cause AnalysisTechnical Trade-offs
Author's notes

This one is deceptively specific.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by acknowledging that NaN loss after initial decrease often stems from numerical instability or optimization issues, then systematically cover data, model, and training dynamics. Structure your answer by grouping causes into categories (e.g., gradient explosion, learning rate, data issues) and suggest diagnostic steps for each.

Pro tip: Mention that you would first check for NaNs in inputs and gradients, and consider using gradient clipping or a lower learning rate as quick fixes, but emphasize root cause analysis to prevent recurrence.

1. Check for numerical instability

Investigate gradient explosion/vanishing, especially in multi-task settings where task losses may have different scales. Use gradient clipping and normalization techniques.

2. Review learning rate and optimizer settings

A learning rate that is too high can cause divergence after initial convergence. Consider learning rate schedules, warm-up, or adaptive optimizers like Adam with appropriate epsilon.

3. Examine data and labels

Look for NaNs, infinities, or outliers in input features or labels. In multi-task, one task's data might have corrupted samples causing instability.

4. Analyze loss weighting and task balancing

If task losses are combined with static weights, one task may dominate and cause instability. Use dynamic weighting or uncertainty-based weighting.

5. Inspect model architecture and initialization

Poor initialization or architecture choices (e.g., unbounded activations) can lead to NaN. Consider layer normalization or gradient clipping.

Key Points to Mention

  • Gradient explosion or vanishing, especially with multi-task learning where gradients from different tasks may conflict.
  • Learning rate too high or inappropriate optimizer settings (e.g., Adam's epsilon).
  • Data issues: NaNs, infinities, or outliers in features or labels, possibly task-specific.
  • Loss weighting: static weights causing one task to dominate, leading to instability.
  • Numerical instability in operations like log or division (e.g., in loss functions).
  • Lack of gradient clipping or normalization techniques.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q2

How would you go about debugging training instability that only shows up after several hundred steps?

Root Cause AnalysisSystem Design
Author's notes

The delayed onset part is what trips people up.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by acknowledging that late-stage instability often stems from subtle issues like numerical precision, data ordering, or optimizer state accumulation. Then outline a systematic debugging process: reproduce, isolate, instrument, and fix, emphasizing the need for controlled experiments and logging.

Pro tip: Use gradient and parameter norm tracking from the start; sudden spikes often precede loss divergence, giving you an early warning. Also, consider that the bug might be in your evaluation or data pipeline, not the model itself.

1. Reproduce and characterize

Run the training multiple times with the same seed to confirm reproducibility. Log loss, gradients, and metrics at every step to identify the exact point of divergence.

2. Isolate the cause

Simplify the setup: reduce model size, use a subset of data, and disable components (e.g., dropout, batch norm) to see if instability persists. Check for data ordering issues or rare samples that appear later.

3. Instrument and monitor

Add detailed logging for gradient norms, parameter norms, learning rate, and activation statistics. Use tools like TensorBoard or Weights & Biases to visualize trends over time.

4. Test hypotheses

Form hypotheses (e.g., numerical instability, learning rate too high, bad data) and test them with controlled experiments. For example, try gradient clipping, lower learning rate, or different initialization.

5. Fix and validate

Apply the fix (e.g., gradient clipping, learning rate schedule, data cleaning) and validate by running multiple seeds to ensure stability. Document the root cause and solution.

Key Points to Mention

  • Gradient clipping to prevent exploding gradients
  • Learning rate schedules and warmup
  • Numerical precision (e.g., float16 vs float32, loss scaling)
  • Data pipeline issues: shuffling, rare samples, or data leakage
  • Batch normalization and its interaction with small batches
  • Regularization techniques and their impact on stability

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q3

What loss functions and optimization strategies would you use for a sparse conversion prediction problem?

Technical Trade-offsProduct Analytics & Metrics
Author's notes

Class imbalance is the whole game here.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the business context and data characteristics (e.g., conversion rate, class imbalance, volume). Then discuss loss functions suited for sparse binary outcomes, such as weighted binary cross-entropy or focal loss, and optimization strategies like Adam with learning rate scheduling, regularization, and resampling techniques. Emphasize evaluation metrics beyond accuracy (e.g., AUC-ROC, PR-AUC) and the trade-offs between model complexity and interpretability.

Pro tip: Credit Karma values data-driven decisions, so quantify the impact of your choices—e.g., how focal loss improved recall by X% in a similar sparse conversion scenario—and mention how you'd validate with A/B testing to ensure business alignment.

1. Clarify the problem and data

Ask about conversion rate, data volume, feature availability, and business goal (e.g., maximizing conversions vs. minimizing cost). This ensures your approach is tailored to the specific sparsity and imbalance.

2. Select loss functions

Propose loss functions robust to class imbalance, such as weighted binary cross-entropy, focal loss, or asymmetric loss. Explain why they help with sparse positive examples.

3. Choose optimization strategies

Discuss optimizers (e.g., Adam, SGD with momentum), learning rate schedules, regularization (L1/L2, dropout), and techniques like batch normalization. Mention handling of sparse gradients if using embeddings.

4. Address data-level techniques

Consider resampling (oversampling positives, undersampling negatives), synthetic data generation (SMOTE), or class weighting. Note trade-offs like potential overfitting or information loss.

5. Define evaluation and iteration plan

Specify metrics (PR-AUC, recall@k, lift) and validation strategy (stratified k-fold, time-based split). Outline how you'd iterate and potentially A/B test in production.

Key Points to Mention

  • Class imbalance and sparsity: conversion events are rare, so standard loss functions may fail; use weighted or focal loss.
  • Focal loss: down-weights easy negatives, focuses on hard examples, useful for extreme imbalance.
  • Optimization: Adam optimizer with learning rate decay, early stopping, and regularization to prevent overfitting.
  • Resampling techniques: oversampling positives or undersampling negatives, but beware of overfitting and distribution shift.
  • Evaluation metrics: PR-AUC is more informative than ROC-AUC for imbalanced data; also consider recall at fixed precision.
  • Business impact: align model objective with business KPI (e.g., conversion rate lift) and plan for online testing.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.