This is one question but it's really like four questions stapled together.
Structure your answer as a clear, step-by-step derivation: start with the model formulation, then introduce the loss function with justification, derive the gradient, and finally explain gradient descent with the update rule. Write the math explicitly on the whiteboard, narrating each step to show your reasoning.
Pro tip: Emphasize the probabilistic interpretation of the sigmoid and the maximum likelihood derivation of the loss, as this demonstrates deep understanding beyond rote memorization. Also, mention practical considerations like feature scaling and learning rate selection to show real-world awareness.
Define the linear combination z = w^T x + b, then apply the sigmoid function σ(z) = 1/(1+e^{-z}) to get the predicted probability p(y=1|x). Explain that the sigmoid maps real values to (0,1), making it suitable for binary classification.
Introduce the binary cross-entropy loss (log loss): L(w) = -[y log(p) + (1-y) log(1-p)]. Justify it via maximum likelihood estimation, showing that minimizing this loss is equivalent to maximizing the likelihood of the observed data.
Derive the gradient of the loss with respect to weights: ∇_w L = (p - y) x. Show the steps using the chain rule, highlighting that the sigmoid derivative simplifies the expression elegantly.
Explain gradient descent: iteratively update weights in the opposite direction of the gradient: w := w - α ∇_w L, where α is the learning rate. Discuss the role of the learning rate and potential issues like slow convergence or divergence.
Mention extensions like regularization (L1/L2), stochastic gradient descent, and convergence checks. Briefly note how to handle the bias term and the importance of feature scaling for gradient descent.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.