← Uber Interview InsightsMachine Learning Engineer
The closed-form part was fine, augmenting X with a ones column and inverting XᵀX is pretty mechanical.
Start by clarifying assumptions (e.g., whether to include an intercept, data format) and outline the math behind both solutions. Then implement the normal equation with a bias term and gradient descent with configurable hyperparameters, ensuring numerical stability. Finally, implement predict and discuss trade-offs between the two methods.
Pro tip: Mention numerical stability: for the normal equation, use np.linalg.solve or pseudo-inverse instead of explicit matrix inversion, and for gradient descent, consider feature scaling to speed convergence.
Confirm input format (X, y), whether to include an intercept, and expected output. Initialize parameters and add a bias column to X if needed.
Compute theta using the normal equation: theta = (X^T X)^(-1) X^T y, using a numerically stable method like np.linalg.solve or np.linalg.pinv.
Initialize theta, define the cost function (MSE), and iteratively update theta using the gradient with configurable learning rate and iterations. Optionally track loss for convergence monitoring.
Given new X, add bias term and return X @ theta. Ensure consistency with training preprocessing.
Compare computational complexity, scalability, and numerical stability of both methods. Mention when to use each (e.g., normal equation for small features, gradient descent for large datasets).
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Said something about large D making the matrix inversion expensive (cubic in D) and large N making it memory-heavy to store XᵀX.
Start by contrasting the computational and statistical trade-offs between closed-form normal equations and gradient descent, emphasizing when each is preferable. Then dive into the practical concerns of an ill-conditioned XᵀX matrix, covering numerical stability, regularization, and algorithmic workarounds. Conclude with a concrete recommendation for production systems.
Pro tip: Mention that while normal equations are elegant, in practice you often use iterative methods like conjugate gradient or LSQR for large sparse problems, and always monitor condition number as a diagnostic. This shows you understand real-world deployment beyond textbook theory.
Explain that normal equations require O(n³) for inversion (or O(n²) with Cholesky) and O(mn²) for forming XᵀX, while gradient descent scales with iterations and is preferable for large n or when n is huge. Highlight that normal equations are exact and avoid hyperparameter tuning like learning rate.
State scenarios: small to medium n (e.g., n < 10,000), well-conditioned XᵀX, need for exact solution, or when you want to avoid iterative convergence issues. Also mention that it's useful for prototyping and when the design matrix fits in memory.
Describe that ill-conditioning (high condition number) makes XᵀX nearly singular, leading to numerical instability, amplified errors in coefficients, and overfitting. Mention that it can cause the normal equations to fail or give unreliable estimates.
List solutions: regularization (ridge adds λI to XᵀX), using QR or SVD instead of normal equations for better numerical stability, feature scaling, or switching to gradient descent with regularization. Also mention monitoring condition number and using iterative solvers like LSQR for large sparse problems.
Summarize that for small, well-conditioned problems, normal equations are fine; for large or ill-conditioned problems, prefer gradient descent or regularized iterative methods. Emphasize that understanding the condition number guides the choice.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by explaining the concept of L2 regularization and its effect on the loss function. Then, show how to modify the closed-form solution by adding a regularization term to the normal equations, and how to adjust the gradient descent update rule by adding the regularization gradient. Finally, discuss practical considerations such as feature scaling and the intercept term.
Pro tip: Emphasize that L2 regularization introduces bias but reduces variance, and mention that the closed-form solution with L2 is always invertible due to the added λI term, which improves numerical stability.
Explain that L2 regularization adds a penalty term λ||w||^2 to the loss function, discouraging large weights. Clarify that the intercept is typically not regularized.
Derive the modified normal equations: w = (X^T X + λI)^(-1) X^T y, where I is the identity matrix (possibly excluding the intercept). Mention that this makes the matrix invertible even if X^T X is singular.
Show the updated gradient: ∇J = (1/m) X^T (Xw - y) + (λ/m) w (or λw depending on convention). Explain that the update rule becomes w := w - α[(1/m) X^T (Xw - y) + λw].
Discuss the importance of feature scaling for gradient descent, the need to tune λ via cross-validation, and the option to exclude the intercept from regularization.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by defining MSE and R-squared, explaining what each measures and their limitations. Then describe how to use them together to assess model fit, and finally outline a systematic approach to distinguish underfitting from overfitting using training vs. validation metrics and learning curves.
Pro tip: Emphasize that R-squared can be misleading for non-linear models and that MSE is scale-dependent, so always compare against a baseline and consider the business context (e.g., Uber's pricing or ETA prediction) when interpreting these metrics.
Explain MSE as the average squared difference between predicted and actual values, and R-squared as the proportion of variance explained. Mention that MSE penalizes large errors more and is in squared units, while R-squared is unitless and ranges from -∞ to 1.
Describe how to compute these metrics on training and validation sets. A low MSE and high R-squared on both sets indicate good fit, but always compare to a baseline (e.g., mean predictor) to ensure the model adds value.
Underfitting occurs when the model is too simple: both training and validation MSE are high, and R-squared is low on both. The model fails to capture the underlying pattern.
Overfitting occurs when the model is too complex: training MSE is very low and training R-squared is high, but validation MSE is much higher and validation R-squared is much lower. This indicates poor generalization.
Plot learning curves (training vs. validation error as a function of training set size or model complexity) to visualize under/overfitting. Mention techniques like cross-validation, regularization, and early stopping to mitigate these issues.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.