← Databricks Interview Insights
This was a single sprawling question that kept branching.
Start by clarifying the problem scope: define OOM as a binary classification task predicting whether a notebook session will fail due to memory exhaustion within a time horizon. Then walk through the ML lifecycle: data collection and labeling, feature engineering, model training and serving, user-facing interventions, and evaluation with feedback loops. Emphasize trade-offs between precision and recall, latency requirements, and integration with the Databricks platform.
Pro tip: Focus on actionable predictions: the goal is not just to predict OOM but to trigger preventive actions (e.g., auto-scaling, warnings) that improve user experience. Also, consider that OOM is rare, so use techniques like anomaly detection or cost-sensitive learning to handle class imbalance.
Define the prediction target: binary classification of whether a session will OOM within the next N minutes. Label data by identifying OOM events from logs (e.g., exit codes, error messages) and creating positive/negative examples with appropriate time windows.
Engineer features from session metadata (e.g., cluster size, libraries), runtime metrics (e.g., memory usage trends, GC activity), user behavior (e.g., code execution patterns), and historical data (e.g., past OOMs). Include temporal features like memory growth rate.
Train a model (e.g., gradient boosted trees, LSTM) with class imbalance handling. Serve predictions in real-time via a streaming pipeline that computes features from live metrics and returns a risk score. Ensure low latency and scalability.
Design interventions based on risk score: low risk (no action), medium risk (warn user, suggest optimization), high risk (auto-scale cluster, kill session). Integrate with Databricks UI and APIs for seamless user experience.
Evaluate offline with precision/recall, AUC, and business metrics (e.g., reduction in OOM incidents). Monitor online performance, collect user feedback (e.g., whether warnings were helpful), and retrain periodically to adapt to changing patterns.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.