← Databricks Interview Insights

Databricks·Machine Learning Engineer·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say
Apr 2026

Summary

System design round at Databricks for an MLE role. The whole question was about predicting OOM failures in cloud notebooks, which sounds niche but actually opens up into a pretty deep ML systems problem once you start pulling on the threads.

Questions Asked (1)

Q1

Design an ML system that predicts Out-of-Memory (OOM) failures for cloud-hosted notebook sessions, covering problem framing, feature engineering, labeling, serving architecture, user-facing actions, and the evaluation and feedback loop.

System DesignTechnical Trade-offsProduct Analytics & Metrics
Author's notes

This was a single sprawling question that kept branching.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the problem scope: define OOM as a binary classification task predicting whether a notebook session will fail due to memory exhaustion within a time horizon. Then walk through the ML lifecycle: data collection and labeling, feature engineering, model training and serving, user-facing interventions, and evaluation with feedback loops. Emphasize trade-offs between precision and recall, latency requirements, and integration with the Databricks platform.

Pro tip: Focus on actionable predictions: the goal is not just to predict OOM but to trigger preventive actions (e.g., auto-scaling, warnings) that improve user experience. Also, consider that OOM is rare, so use techniques like anomaly detection or cost-sensitive learning to handle class imbalance.

1. Problem Framing and Labeling

Define the prediction target: binary classification of whether a session will OOM within the next N minutes. Label data by identifying OOM events from logs (e.g., exit codes, error messages) and creating positive/negative examples with appropriate time windows.

2. Feature Engineering

Engineer features from session metadata (e.g., cluster size, libraries), runtime metrics (e.g., memory usage trends, GC activity), user behavior (e.g., code execution patterns), and historical data (e.g., past OOMs). Include temporal features like memory growth rate.

3. Model Training and Serving

Train a model (e.g., gradient boosted trees, LSTM) with class imbalance handling. Serve predictions in real-time via a streaming pipeline that computes features from live metrics and returns a risk score. Ensure low latency and scalability.

4. User-Facing Actions

Design interventions based on risk score: low risk (no action), medium risk (warn user, suggest optimization), high risk (auto-scale cluster, kill session). Integrate with Databricks UI and APIs for seamless user experience.

5. Evaluation and Feedback Loop

Evaluate offline with precision/recall, AUC, and business metrics (e.g., reduction in OOM incidents). Monitor online performance, collect user feedback (e.g., whether warnings were helpful), and retrain periodically to adapt to changing patterns.

Key Points to Mention

  • Class imbalance: OOM events are rare, so use techniques like oversampling, SMOTE, or anomaly detection.
  • Feature engineering: include memory usage trends, cluster specs, user history, and code complexity metrics.
  • Serving architecture: real-time streaming pipeline with low latency, possibly using Spark Structured Streaming and MLflow for model deployment.
  • User-facing actions: tiered interventions (warnings, auto-scaling, session termination) with clear communication.
  • Evaluation metrics: beyond accuracy, use precision/recall, F1, and business impact like reduced OOM incidents and user satisfaction.
  • Feedback loop: collect implicit (user actions) and explicit (surveys) feedback to continuously improve model.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.