The naive version blows up fast and I knew that going in, so I started with the log-sum-exp trick right away.
Start by explaining the naive formula for entropy of softmax and its numerical pitfalls (overflow/underflow). Then present the log-sum-exp trick to stabilize both the softmax probabilities and the log probabilities, and derive a stable expression for entropy. Finally, discuss implementation details and trade-offs (e.g., using log_softmax directly).
Pro tip: Mention that many deep learning frameworks provide a numerically stable log_softmax function, and you can compute entropy as -sum(exp(log_probs) * log_probs) or equivalently using logsumexp and expected logits. This shows practical awareness beyond the math.
State the formula for entropy H(p) = -sum(p_i log p_i) and softmax p_i = exp(z_i)/sum(exp(z_j)). Highlight that direct computation can overflow/underflow.
Explain subtracting the max logit m = max(z) before exponentiation: p_i = exp(z_i - m)/sum(exp(z_j - m)). This avoids overflow in exp and underflow in the denominator.
Derive log p_i = (z_i - m) - log(sum(exp(z_j - m))). Use this to avoid taking log of small probabilities directly.
Write entropy as -sum(p_i * log p_i) using the stable p_i and log p_i. Alternatively, use H = log(sum(exp(z_i - m))) + m - sum(p_i * z_i) to avoid computing log p_i explicitly.
Mention using built-in log_softmax, handling large vectors, and potential underflow when probabilities are extremely small (but log p_i remains finite).
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
This is where it got interesting and also where I fumbled a bit.
Start by defining the online softmax update using a running maximum and running sum of exponentials, then derive the entropy update from these statistics. Emphasize numerical stability techniques like the log-sum-exp trick and discuss pitfalls such as overflow, underflow, and loss of precision.
Pro tip: Mention that maintaining the running maximum is crucial not only for stability but also for efficient updates when a new maximum arrives, and that using log-domain computations can further mitigate underflow.
Clarify that we process logits z_i sequentially and need to maintain entropy H = -∑ p_i log p_i where p_i = exp(z_i)/∑ exp(z_j). Introduce running statistics: current maximum m and sum of exponentials s = ∑ exp(z_j - m).
When a new logit z arrives, update m' = max(m, z) and s' = s * exp(m - m') + exp(z - m'). This avoids overflow by shifting exponentials by the running maximum.
Show that entropy can be computed as H = log s + m - (1/s) ∑ z_i exp(z_i - m). Maintain the weighted sum T = ∑ z_i exp(z_i - m) alongside s and m, updating T' = T * exp(m - m') + z * exp(z - m').
Highlight risks: overflow when exponentiating large logits (mitigated by max-shifting), underflow when s becomes very small (use log-domain or scaling), and precision loss when adding many small terms (use compensated summation or double precision).
Conclude that each new logit requires O(1) time and memory, and that the approach is numerically stable with proper handling of the running maximum and sum.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.