← Upstart Interview Insights

Upstart·Machine Learning Engineer·Technical Phone Screen·Senior

SeniorPrefer not to say
Jun 2026Remote

Summary

Upstart ML Engineer interview with a hands-on case study using the California housing dataset. The twist was a deliberate train/test distribution shift by income bracket, so it wasn't just about building a model but about diagnosing and adapting to covariate shift. More depth expected on the analysis side than I initially anticipated.

Questions Asked (1)

Q1

You're given the California housing dataset where train and test come from different income brackets. Build a regression model from scratch to predict house prices, then analyze and optimize it with the distribution shift in mind.

Technical Trade-offsRoot Cause AnalysisData Modeling
Author's notes

I went straight into feature engineering and spent way too long there.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by framing the problem as a distribution shift between train and test sets, then build a baseline regression model (e.g., linear regression) from scratch, evaluate it on both in-distribution and shifted data, and iteratively apply techniques like importance weighting, domain adaptation, or robust feature engineering to optimize for the shifted test distribution. Emphasize validation strategies that simulate the shift (e.g., splitting train by income brackets) and discuss trade-offs between model complexity and generalization.

Pro tip: Demonstrate awareness that the shift is in the input distribution (income brackets), not just the label distribution, so methods like importance weighting or adversarial validation are more appropriate than simple resampling. Also, mention that you would monitor performance on a held-out set that mimics the test shift to avoid overfitting to the training distribution.

1. Understand the data and shift

Explore the California housing dataset, identify the income brackets in train and test, and quantify the distribution shift (e.g., compare histograms, compute statistics). Recognize that the shift is in the input features (income) and may affect the relationship with the target.

2. Build a baseline model from scratch

Implement a simple regression model (e.g., linear regression with gradient descent) from scratch, using only the training data. Evaluate its performance on a validation set that mimics the test distribution (e.g., hold out a portion of train with different income brackets).

3. Diagnose the impact of distribution shift

Compare model performance on in-distribution vs. shifted validation sets. Analyze errors, feature importance, and residuals to understand how the shift affects predictions. Consider using adversarial validation to confirm which features cause the shift.

4. Apply mitigation techniques

Implement methods to handle the shift: importance weighting (reweight training samples to match test distribution), domain adaptation (e.g., CORAL, adversarial training), or robust feature engineering (e.g., removing or transforming shifted features). Discuss trade-offs of each approach.

5. Optimize and validate

Iterate on model and preprocessing, using a validation strategy that simulates the shift (e.g., cross-validation with income-based splits). Evaluate final model on a held-out set that closely resembles the test distribution, and report metrics like RMSE and MAE.

Key Points to Mention

  • Distribution shift: covariate shift vs. concept shift, and how income bracket shift affects the feature-target relationship.
  • Importance weighting: estimating density ratio between test and train distributions to reweight samples.
  • Adversarial validation: training a classifier to distinguish train from test to identify shifted features.
  • Domain adaptation techniques: CORAL, adversarial domain adaptation, or simple feature normalization.
  • Validation strategy: creating a validation set that mimics the test shift (e.g., splitting by income brackets) to avoid overfitting.
  • Trade-offs: complexity vs. robustness, computational cost of reweighting, and potential negative transfer.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.