← Startups.com Interview Insights

Startups.com·Machine Learning Engineer·Technical Phone Screen·Intermediate

Intermediate
Mar 2026

Summary

Interviewed for an ML Engineer role at Startups.com and got a pretty deep PyTorch fundamentals question about implementing ReLU from scratch, both as a plain tensor function and as a custom autograd.Function. Not the vibe I expected from a startup interview but here we are.

Questions Asked (1)

Q1

Implement the ReLU activation function from scratch in PyTorch without using built-in ReLU utilities. Your solution should handle tensors of any shape, support autograd with correct gradients (1 where x > 0, 0 elsewhere), and be written both as a plain tensor function and as a custom torch.autograd.Function subclass with explicit forward and backward methods.

Technical Trade-offsAlgorithms & Data StructuresSystem Design
Author's notes

This question has two parts and I fumbled the transition between them.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the requirements: a plain tensor function using torch.where or masking, and a custom autograd.Function with explicit forward and backward. Then implement both, ensuring the backward pass correctly computes the gradient as a mask (1 where input > 0, 0 elsewhere) and handles broadcasting. Finally, test with various shapes and verify gradients using torch.autograd.gradcheck.

Pro tip: Mention that using torch.where is more memory-efficient than multiplying by a boolean mask because it avoids creating an intermediate tensor, and that in the custom Function, you should save the input or the mask in forward to avoid recomputation in backward.

1. Clarify requirements and edge cases

Confirm that the function should work for any shape, support autograd, and that the gradient is 1 for x>0 and 0 otherwise (including at x=0, where the derivative is typically 0).

2. Implement plain tensor function

Use torch.where(x > 0, x, torch.zeros_like(x)) or x * (x > 0).float() to compute ReLU while preserving autograd.

3. Implement custom autograd.Function

Subclass torch.autograd.Function, define static forward that saves the input or mask, and backward that returns grad_output * (input > 0).float().

4. Test correctness and gradients

Verify outputs for positive, negative, and zero inputs, and use torch.autograd.gradcheck to ensure backward matches numerical gradients.

5. Discuss trade-offs and optimizations

Compare the two approaches: plain function is simpler but may use more memory; custom Function allows memory savings by saving the mask instead of the input, and can be fused with other ops.

Key Points to Mention

  • ReLU definition: f(x) = max(0, x), derivative: 1 if x > 0 else 0.
  • Using torch.where or boolean masking to avoid in-place operations that break autograd.
  • In custom Function, forward must return output and save any tensors needed for backward (e.g., input or mask).
  • Backward must return gradients for all inputs; if only one input, return a single tensor, not a tuple.
  • Handling of x=0: derivative is 0 (subgradient), which is standard in PyTorch.
  • Testing with gradcheck and comparing to torch.relu for correctness.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.