I went straight into feature engineering and spent way too long there.
Start by framing the problem as a distribution shift between train and test sets, then build a baseline regression model (e.g., linear regression) from scratch, evaluate it on both in-distribution and shifted data, and iteratively apply techniques like importance weighting, domain adaptation, or robust feature engineering to optimize for the shifted test distribution. Emphasize validation strategies that simulate the shift (e.g., splitting train by income brackets) and discuss trade-offs between model complexity and generalization.
Pro tip: Demonstrate awareness that the shift is in the input distribution (income brackets), not just the label distribution, so methods like importance weighting or adversarial validation are more appropriate than simple resampling. Also, mention that you would monitor performance on a held-out set that mimics the test shift to avoid overfitting to the training distribution.
Explore the California housing dataset, identify the income brackets in train and test, and quantify the distribution shift (e.g., compare histograms, compute statistics). Recognize that the shift is in the input features (income) and may affect the relationship with the target.
Implement a simple regression model (e.g., linear regression with gradient descent) from scratch, using only the training data. Evaluate its performance on a validation set that mimics the test distribution (e.g., hold out a portion of train with different income brackets).
Compare model performance on in-distribution vs. shifted validation sets. Analyze errors, feature importance, and residuals to understand how the shift affects predictions. Consider using adversarial validation to confirm which features cause the shift.
Implement methods to handle the shift: importance weighting (reweight training samples to match test distribution), domain adaptation (e.g., CORAL, adversarial training), or robust feature engineering (e.g., removing or transforming shifted features). Discuss trade-offs of each approach.
Iterate on model and preprocessing, using a validation strategy that simulates the shift (e.g., cross-validation with income-based splits). Evaluate final model on a held-out set that closely resembles the test distribution, and report metrics like RMSE and MAE.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.