This is the kind of question where nodding along in study sessions does not prepare you.
Start by clearly defining the network architecture and notation, then systematically derive the forward pass, loss, and backward pass using the chain rule. Write out the equations layer by layer, emphasizing the recursive nature of backpropagation and the role of activation derivatives.
Pro tip: When deriving, explicitly state the dimensions of each matrix and vector to demonstrate rigor and catch shape mismatches—this shows practical implementation awareness that interviewers value.
Specify the network with L layers, layer sizes, weight matrices W^l, bias vectors b^l, and activation functions σ^l. Use clear index notation for samples and layers.
For each layer l, write z^l = W^l a^{l-1} + b^l and a^l = σ^l(z^l), with a^0 = x. Define the final output and loss function (e.g., MSE or cross-entropy).
Define the error term δ^l = ∂L/∂z^l. Derive the recursive relation δ^l = (W^{l+1}^T δ^{l+1}) ⊙ σ'^l(z^l) for hidden layers, and the output layer δ^L = ∇_a L ⊙ σ'^L(z^L).
Using δ^l, write ∂L/∂W^l = δ^l (a^{l-1})^T and ∂L/∂b^l = δ^l. Show that these follow from the chain rule applied to z^l = W^l a^{l-1} + b^l.
Present the full backpropagation algorithm: forward pass to cache activations, backward pass to compute δ and gradients, then update parameters. Mention vectorization and computational cost.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.