← Disney Interview Insights

Disney·Machine Learning Engineer·Technical Phone Screen·Intermediate

Intermediate
Apr 2026

Summary

Disney ML Engineer interview that was basically a full notebook exercise on spam classification. They walked you through the whole pipeline live, which I wasn't expecting to be as involved as it was.

Questions Asked (3)

Q1

Build a text classifier in a Jupyter notebook to label messages as spam or not-spam using a Naive Bayes model. Walk through the full pipeline from raw text to evaluation.

Technical Trade-offsAlgorithms & Data Structures
Author's notes

This was the meat of the whole thing.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Walk through the end-to-end pipeline in a logical order: data loading and exploration, text preprocessing, feature extraction, model training with Naive Bayes, and evaluation. Emphasize the reasoning behind each choice, especially why Naive Bayes is suitable for text classification and how you handle class imbalance and evaluation metrics. Keep the explanation clear and structured, as if you were narrating your notebook.

Pro tip: Mention that you would start with a simple baseline (e.g., Multinomial Naive Bayes with TF-IDF) and then iterate, and highlight the importance of using precision/recall and AUC-ROC instead of just accuracy for imbalanced spam data.

1. Data Loading and Exploration

Load the dataset (e.g., SMS Spam Collection), inspect class distribution, and examine sample messages to understand text characteristics and potential preprocessing needs.

2. Text Preprocessing

Clean and normalize text: lowercasing, removing punctuation/numbers, tokenization, stopword removal, and optionally stemming/lemmatization. Explain how each step affects the model.

3. Feature Extraction

Convert text to numerical features using techniques like Bag-of-Words or TF-IDF. Discuss trade-offs (e.g., TF-IDF downweights common words) and why it suits Naive Bayes.

4. Model Training and Tuning

Split data into train/test, train a Multinomial Naive Bayes classifier, and optionally tune hyperparameters (e.g., alpha smoothing) using cross-validation.

5. Evaluation and Interpretation

Evaluate using appropriate metrics (precision, recall, F1, AUC-ROC), inspect confusion matrix, and interpret model coefficients to understand influential words.

Key Points to Mention

  • Why Naive Bayes is a good baseline for text classification (probabilistic, handles high-dimensional sparse data, fast).
  • Handling class imbalance: use stratified splits, class weights, and appropriate metrics.
  • Text preprocessing choices and their impact (e.g., stopword removal may hurt if spam words are stopwords).
  • Feature extraction: TF-IDF vs CountVectorizer, n-grams, and the effect of max_features.
  • Hyperparameter tuning: alpha smoothing in Naive Bayes to avoid zero probabilities.
  • Evaluation metrics: precision, recall, F1, AUC-ROC, and why accuracy is misleading for imbalanced data.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q2

How would you handle class imbalance in this spam classification task, and what effect does it have on your evaluation metrics?

Product Analytics & MetricsTechnical Trade-offs
Author's notes

F1 score came up here and I was glad I'd thought about it beforehand.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by acknowledging that spam classification is typically imbalanced, then discuss both data-level and algorithm-level techniques to handle it. Emphasize that evaluation metrics must be chosen carefully to reflect business costs, such as false positives (blocking legitimate emails) versus false negatives (allowing spam).

Pro tip: Tie your metric choice to Disney's business impact: for example, false positives could mean missing important guest communications, so precision or a cost-sensitive metric may matter more than recall.

1. Acknowledge the imbalance

State that spam datasets often have a minority class (spam) and majority class (ham), and that accuracy is misleading. Mention the typical ratio and why it matters.

2. Choose handling techniques

Discuss data-level methods like resampling (oversampling, undersampling, SMOTE) and algorithm-level methods like class weights, cost-sensitive learning, or ensemble methods. Explain trade-offs.

3. Select appropriate metrics

Recommend metrics beyond accuracy: precision, recall, F1, AUC-ROC, AUC-PR, and confusion matrix. Explain how each is affected by imbalance and which to prioritize based on business goals.

4. Align with business costs

Connect metric choice to the cost of false positives vs false negatives. For Disney, false positives (blocking legitimate emails) may be more costly, so precision or a cost-weighted metric could be key.

5. Validate and iterate

Emphasize using stratified cross-validation, monitoring performance on both classes, and iterating on the approach as data drifts.

Key Points to Mention

  • Accuracy paradox: high accuracy can be achieved by predicting majority class only.
  • Resampling techniques: SMOTE, random oversampling/undersampling, and their risks (overfitting, information loss).
  • Class weights and cost-sensitive learning in algorithms like logistic regression, SVM, or tree-based models.
  • Evaluation metrics: precision, recall, F1, AUC-ROC, AUC-PR, and confusion matrix; AUC-PR is more informative for imbalanced data.
  • Business context: cost of false positives vs false negatives, and how that dictates metric prioritization.
  • Stratified k-fold cross-validation to ensure representative class distribution in each fold.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q3

After fitting your model, how would you sanity-check it on a few example inputs before trusting it in production?

Root Cause AnalysisTechnical Trade-offs
Author's notes

Short answer: just feed in some obvious examples and see if they make sense.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by emphasizing that sanity-checking is a critical step before deployment, focusing on a few diverse, representative examples. Walk through a structured process: select edge cases, manually inspect predictions, compare against baselines, and validate with domain knowledge. Highlight the importance of catching obvious errors and building trust in the model's behavior.

Pro tip: Frame your sanity-check as a risk mitigation exercise: mention that you'd automate these checks into a test suite to catch regressions, showing you think about long-term production reliability.

1. Select diverse and edge-case examples

Choose a small set of inputs that cover typical cases, edge cases (e.g., missing values, outliers), and known difficult scenarios. This ensures you test the model's robustness across the input space.

2. Manually inspect predictions and intermediate outputs

For each example, examine the model's prediction, confidence scores, and any intermediate representations (e.g., feature importances). Look for nonsensical outputs or unexpected patterns.

3. Compare against baselines and expectations

Contrast the model's predictions with a simple baseline (e.g., heuristic, previous model) and with domain knowledge. Check if the model's behavior aligns with intuitive expectations.

4. Validate with domain experts and stakeholders

If possible, review the examples with domain experts to confirm that predictions make sense in the business context. This helps catch subtle errors that metrics might miss.

5. Automate checks for continuous monitoring

Codify these sanity checks into automated tests that run on new data or model updates. This ensures ongoing reliability and quick detection of issues in production.

Key Points to Mention

  • Importance of selecting representative and edge-case examples
  • Manual inspection of predictions and confidence scores
  • Comparison with baselines and domain knowledge
  • Involvement of domain experts for validation
  • Automation of sanity checks for production monitoring
  • Potential pitfalls: overfitting to test set, data leakage, distribution shift

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.