← Startups.com Interview Insights
This question has two parts and I fumbled the transition between them.
Start by clarifying the requirements: a plain tensor function using torch.where or masking, and a custom autograd.Function with explicit forward and backward. Then implement both, ensuring the backward pass correctly computes the gradient as a mask (1 where input > 0, 0 elsewhere) and handles broadcasting. Finally, test with various shapes and verify gradients using torch.autograd.gradcheck.
Pro tip: Mention that using torch.where is more memory-efficient than multiplying by a boolean mask because it avoids creating an intermediate tensor, and that in the custom Function, you should save the input or the mask in forward to avoid recomputation in backward.
Confirm that the function should work for any shape, support autograd, and that the gradient is 1 for x>0 and 0 otherwise (including at x=0, where the derivative is typically 0).
Use torch.where(x > 0, x, torch.zeros_like(x)) or x * (x > 0).float() to compute ReLU while preserving autograd.
Subclass torch.autograd.Function, define static forward that saves the input or mask, and backward that returns grad_output * (input > 0).float().
Verify outputs for positive, negative, and zero inputs, and use torch.autograd.gradcheck to ensure backward matches numerical gradients.
Compare the two approaches: plain function is simpler but may use more memory; custom Function allows memory savings by saving the mask instead of the input, and can be fused with other ops.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.