← ByteDance Interview InsightsMachine Learning Engineer
Started fine with MSE vs MAE, talked about outlier sensitivity and when you'd prefer one over the other.
Structure your answer by grouping loss functions by task type (regression vs. classification), then for each loss explain its mathematical intuition, sensitivity to outliers, and typical use cases. Emphasize trade-offs and practical considerations like dataset noise and business impact.
Pro tip: Mention that loss function choice should align with the evaluation metric and business objective—e.g., using MAE when outliers are noise but MSE when large errors are costly—and note that Huber combines the best of both.
Briefly separate regression losses (MSE, MAE, Huber) from classification losses (cross-entropy, hinge, focal) to set a clear structure.
For MSE, MAE, and Huber, describe their formulas, sensitivity to outliers, and when to use each (e.g., MSE for Gaussian noise, MAE for robustness, Huber as a compromise).
For cross-entropy, hinge, and focal, explain their probabilistic or margin-based nature, typical use cases (e.g., cross-entropy for probabilistic outputs, hinge for SVMs, focal for class imbalance).
Discuss how loss choice affects optimization, convergence, and final model behavior, and mention practical considerations like class imbalance, noisy labels, and evaluation metrics.
Summarize the key decision factors (outliers, imbalance, probabilistic needs) and emphasize aligning loss with business goals.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start with the formal definition of KL divergence, then explain its asymmetry through the lens of directionality and support mismatch. Connect it to cross-entropy by decomposing cross-entropy into entropy plus KL divergence, and finally discuss when KL is preferable to simpler losses like MSE, emphasizing probabilistic modeling and distribution matching.
Pro tip: Mention that KL divergence is not a true distance metric (no symmetry, no triangle inequality) and that its asymmetry has practical implications: forward KL (MLE) is mass-covering, while reverse KL is mode-seeking. This shows depth beyond textbook definitions.
Define KL divergence between distributions P and Q as KL(P||Q) = E_P[log(P(x)/Q(x))] = ∫ p(x) log(p(x)/q(x)) dx (or sum for discrete). Emphasize it's the expected log-likelihood ratio under P.
Explain that KL(P||Q) ≠ KL(Q||P) because the expectation is taken under different distributions. Highlight the consequences: forward KL penalizes Q for assigning low probability where P has mass (zero-avoiding), while reverse KL penalizes Q for assigning mass where P has none (zero-forcing).
Show that cross-entropy H(P,Q) = H(P) + KL(P||Q). Since H(P) is constant w.r.t. Q, minimizing KL(P||Q) is equivalent to minimizing cross-entropy. This explains why cross-entropy is used as a loss in classification.
Discuss scenarios: when modeling probability distributions (e.g., variational inference, generative models), when the output is a distribution (e.g., softmax), or when you care about relative probabilities rather than absolute errors. Contrast with MSE which assumes Gaussian noise and penalizes squared differences.
Mention numerical stability (KL can be infinite if Q has zero where P has mass), the need for smoothing, and the choice between forward and reverse KL depending on the application (e.g., forward for density estimation, reverse for variational inference).
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.