← Startups.com Interview Insights

Startups.com·Machine Learning Engineer·Technical Phone Screen·Intermediate

Intermediate
Apr 2026

Summary

Interviewed for an ML Engineer role at Startups.com and got hit with a from-scratch softmax implementation. Not a vibe-check round, they wanted real PyTorch internals knowledge and numerical stability awareness.

Questions Asked (1)

Q1

Implement the Softmax function from scratch in PyTorch without using any built-in softmax utilities. Your implementation should handle an arbitrary dim argument, be numerically stable via max subtraction before exponentiation, and support autograd either through native PyTorch ops or a custom autograd Function with an explicit backward pass using the softmax Jacobian.

Technical Trade-offsAlgorithms & Data StructuresSystem Design
Author's notes

This one had layers I did not fully anticipate.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the requirements: arbitrary dim, numerical stability, and autograd support. Then implement the forward pass using max subtraction and exponentiation, and either rely on PyTorch's autograd or define a custom Function with an explicit backward pass using the softmax Jacobian. Finally, discuss trade-offs between the two autograd approaches.

Pro tip: Mention that while a custom backward pass is educational, using native PyTorch ops is generally preferred for performance and reliability in production. Also, highlight that max subtraction is crucial for numerical stability, especially with large logits.

1. Clarify requirements and constraints

Confirm that the implementation must handle arbitrary dim, be numerically stable, and support autograd. Discuss whether to use native ops or a custom Function.

2. Implement numerically stable forward pass

Compute the maximum along the specified dim, subtract it from the input, exponentiate, and normalize by the sum of exponentials.

3. Choose autograd approach

Decide between relying on PyTorch's automatic differentiation or writing a custom autograd.Function with an explicit backward pass.

4. Derive and implement backward pass (if custom)

For the custom Function, compute the gradient using the softmax Jacobian: grad_input = softmax * (grad_output - sum(grad_output * softmax, dim, keepdim=True)).

5. Test and validate

Verify correctness against PyTorch's built-in softmax, check numerical stability with large inputs, and test gradient computation using torch.autograd.gradcheck.

Key Points to Mention

  • Numerical stability via max subtraction to prevent overflow/underflow.
  • Handling arbitrary dim by using keepdim=True in max and sum operations.
  • The softmax Jacobian: for a vector, the Jacobian is diag(s) - s s^T; for batched inputs, it's applied per sample.
  • Trade-offs between using native PyTorch ops (simpler, optimized) and a custom autograd Function (educational, more control).
  • Testing with torch.autograd.gradcheck to ensure correct gradients.
  • Performance considerations: in-place operations, memory usage, and vectorization.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.