← Dandy Interview Insights

Dandy·Machine Learning Engineer·Technical Phone Screen·Senior

Senior
Mar 2026

Summary

Interviewed for an ML Engineer role at Dandy and got hit with a pretty meaty stats/modeling question about cross-validation tradeoffs. Not the typical coding grind I expected.

Questions Asked (1)

Q1

Compare leave-one-out cross-validation and k-fold cross-validation (say, 5-fold) as estimators of test error. Walk through the bias-variance tradeoff between them and discuss computational cost. Which would you actually use in practice?

Technical Trade-offsA/B Testing & Experimentation
Author's notes

I knew the high-level answer but fumbled the variance explanation a bit.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining both methods and their purpose as estimators of test error, then systematically compare them along bias, variance, and computational cost. Conclude with a practical recommendation, emphasizing that k-fold (especially 5-fold) is typically preferred due to its balance between bias and variance and manageable computational cost, while LOO can be useful for very small datasets but is often computationally prohibitive and can have high variance.

Pro tip: Mention that LOO can sometimes be approximated efficiently for certain models (e.g., linear regression) using closed-form solutions, but in practice, k-fold is more robust and widely used. Also, note that for large datasets, even 5-fold can be expensive, so consider using a single validation set or 3-fold if data is abundant.

1. Define the methods

Briefly explain leave-one-out cross-validation (LOO) and k-fold cross-validation (e.g., 5-fold), including how they partition data and estimate test error.

2. Bias comparison

Discuss that LOO uses nearly all data for training, so it has lower bias than k-fold, which uses (k-1)/k of data. However, the difference diminishes as k increases.

3. Variance comparison

Explain that LOO can have higher variance because the training sets are highly correlated (differ by only one sample), leading to similar models and correlated errors. k-fold, with less overlap, often yields lower variance in the error estimate.

4. Computational cost

Compare computational costs: LOO requires n model fits (n = number of samples), which is expensive for large n. k-fold requires k fits, typically much smaller (e.g., 5), making it more efficient.

5. Practical recommendation

Conclude that in practice, k-fold (especially 5 or 10) is preferred due to its good bias-variance tradeoff and computational efficiency. LOO may be used for very small datasets or when computational cost is not an issue, but it's often not worth the extra cost.

Key Points to Mention

  • LOO has lower bias but higher variance compared to k-fold; k-fold offers a balance.
  • Computational cost: LOO requires n fits, k-fold requires k fits, so k-fold is much cheaper for large n.
  • Correlation between training sets in LOO leads to high variance in the error estimate.
  • k-fold with k=5 or 10 is a common choice; 5-fold is often sufficient and computationally efficient.
  • For very small datasets, LOO might be considered, but variance can be a problem; repeated k-fold can help.
  • In practice, use k-fold (e.g., 5-fold) unless there's a specific reason to use LOO, such as extremely small data or closed-form LOO for certain models.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.