I knew the high-level answer but fumbled the variance explanation a bit.
Start by defining both methods and their purpose as estimators of test error, then systematically compare them along bias, variance, and computational cost. Conclude with a practical recommendation, emphasizing that k-fold (especially 5-fold) is typically preferred due to its balance between bias and variance and manageable computational cost, while LOO can be useful for very small datasets but is often computationally prohibitive and can have high variance.
Pro tip: Mention that LOO can sometimes be approximated efficiently for certain models (e.g., linear regression) using closed-form solutions, but in practice, k-fold is more robust and widely used. Also, note that for large datasets, even 5-fold can be expensive, so consider using a single validation set or 3-fold if data is abundant.
Briefly explain leave-one-out cross-validation (LOO) and k-fold cross-validation (e.g., 5-fold), including how they partition data and estimate test error.
Discuss that LOO uses nearly all data for training, so it has lower bias than k-fold, which uses (k-1)/k of data. However, the difference diminishes as k increases.
Explain that LOO can have higher variance because the training sets are highly correlated (differ by only one sample), leading to similar models and correlated errors. k-fold, with less overlap, often yields lower variance in the error estimate.
Compare computational costs: LOO requires n model fits (n = number of samples), which is expensive for large n. k-fold requires k fits, typically much smaller (e.g., 5), making it more efficient.
Conclude that in practice, k-fold (especially 5 or 10) is preferred due to its good bias-variance tradeoff and computational efficiency. LOO may be used for very small datasets or when computational cost is not an issue, but it's often not worth the extra cost.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.