They only showed this after you finished greedy and top-k, which I thought was a nice way to structure it but also meant I was already a bit mentally drained by the time I got here.
Start by clarifying the problem and edge cases, then outline the algorithm step-by-step, and finally discuss implementation details and trade-offs. Emphasize the importance of numerical stability and efficiency in a production setting.
Pro tip: Mention that you would use a stable sort and consider the computational cost of sorting at each step, and that you might optimize by using a partial sort or thresholding. Also, highlight that nucleus sampling is a trade-off between diversity and coherence, and that p is a hyperparameter that can be tuned.
Restate the problem to ensure understanding, and ask clarifying questions about the input (e.g., logits or probabilities), output format, and handling of edge cases like p=0, p=1, or when no tokens exceed p.
Describe the steps: compute softmax probabilities, sort tokens in descending order, compute cumulative sum, find the smallest subset where cumulative probability >= p, renormalize, and sample from the subset.
Explain how to implement efficiently: use a stable sort, handle ties, ensure numerical stability (e.g., subtract max logit before softmax), and consider using a heap or partial sort for large vocabularies.
Describe the loop: at each step, feed the current sequence to the model, get logits for the next token, apply nucleus sampling, append the sampled token, and repeat until EOS or max length.
Mention trade-offs: nucleus sampling vs. greedy/beam/top-k, the effect of p on diversity and coherence, and potential optimizations like caching, batching, and using a threshold to avoid full sort.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.