← Instacart Interview Insights
Said 'yes' pretty confidently at first because MSE is convex and I figured both optimizers are just finding the minimum.
Clarify that with MSE loss, batch gradient descent and SGD converge to the same global minimum for convex models (e.g., linear regression), but may converge to different local minima or points for non-convex models (e.g., neural networks). Emphasize that the loss surface and optimization dynamics determine convergence, not just the loss function.
Pro tip: Mention that in practice, SGD's noise can help escape shallow local minima and sometimes find better solutions, but it never guarantees the same point as batch GD. Also note that with a decaying learning rate, SGD can converge to a local minimum, but not necessarily the same as batch GD.
State that MSE loss defines a loss surface, and both batch GD and SGD are iterative methods to minimize it. The convergence point depends on the convexity of the loss function with respect to the model parameters.
For convex models like linear regression, MSE is convex, so both batch GD and SGD (with appropriate learning rate schedule) converge to the same global minimum.
For non-convex models like neural networks, MSE is non-convex. Batch GD follows the full gradient and may converge to a local minimum; SGD's stochasticity can lead to different local minima or even saddle points.
Mention that SGD requires a decreasing learning rate to converge to a minimum, while batch GD can converge with a fixed learning rate if it's small enough. Even then, the points may differ due to different optimization paths.
Conclude that they do not always converge to the same point; only in convex settings with proper schedules. In deep learning, differences are expected and can be beneficial.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.