← OpenAI Interview InsightsMachine Learning Engineer

OpenAI·Machine Learning Engineer·Technical Phone Screen·Senior

Senior
Jun 2026

Summary

Got a fairly niche ML question from what seemed like an OpenAI technical screen. Just the one question but it was the kind that made me realize I'd been thinking about entropy in a way that only works when you have all the data upfront.

Questions Asked (1)

Q1

If logits are arriving in a streaming fashion, block by block, how would you compute entropy in an online manner?

Algorithms & Data StructuresTechnical Trade-offs
Author's notes

I fumbled the start because my brain went straight to the standard entropy formula and then I had to back up and actually think about what 'online' means here.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the problem: entropy of what distribution? Assume we need the entropy of the softmax distribution over logits. Then explain that entropy requires a sum over all elements, so we can maintain running sums of exp(logit) and logit*exp(logit) across blocks, and compute entropy at the end or incrementally. Discuss numerical stability and memory trade-offs.

Pro tip: Mention that you can compute entropy in a single pass using the log-sum-exp trick to avoid overflow, and that if the number of classes is huge, you might need to approximate or use a streaming algorithm for the partition function.

1. Clarify the problem

Confirm that entropy is for the softmax distribution over logits, and that logits arrive in blocks. Ask if we need exact entropy or an approximation, and if the number of classes is known and fixed.

2. Derive the formula

Entropy H = -sum p_i log p_i, where p_i = exp(z_i)/Z, Z = sum exp(z_j). So H = log Z - (1/Z) sum z_i exp(z_i). This requires accumulating Z and S = sum z_i exp(z_i).

3. Design online accumulation

For each block, compute local sums of exp(z_i) and z_i exp(z_i) using a stable method (e.g., subtract max within block). Then combine with global running sums, possibly using log-sum-exp to merge.

4. Handle numerical stability

Use the log-sum-exp trick to avoid overflow/underflow. Maintain global max or use a streaming log-sum-exp algorithm to combine block-wise statistics.

5. Discuss trade-offs and extensions

Mention memory: O(1) beyond block size. If classes are huge, consider approximate methods like sampling or sketching. Also note that if logits are unnormalized, entropy is well-defined only after softmax.

Key Points to Mention

  • Entropy formula for softmax: H = log Z - (1/Z) sum z_i exp(z_i)
  • Need to maintain running sums of exp(z_i) and z_i exp(z_i)
  • Use log-sum-exp trick for numerical stability
  • Combine block-wise statistics using associative operations
  • Memory complexity O(1) beyond block size
  • Approximation methods for large number of classes (e.g., sampling, sketching)

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.