This one actually went okay for me up to a point.
Start by explaining that randomness comes from the probabilistic nature of sampling from the model's output distribution. Then, clearly define each decoding strategy (greedy, temperature, top-k, top-p) and how they modify that distribution, using a simple example to illustrate. Finally, connect these concepts to practical trade-offs in real-world applications, especially in a data science context at Walmart Labs.
Pro tip: Emphasize that temperature, top-k, and top-p are not mutually exclusive; they are often combined in practice (e.g., temperature scaling followed by top-p). Also, mention that greedy decoding is deterministic but can lead to repetitive or suboptimal outputs, which is why sampling methods are preferred for creative tasks.
Describe how LLMs output a probability distribution over the vocabulary for the next token, and randomness arises when we sample from this distribution instead of always picking the highest-probability token.
Explain that greedy decoding always selects the token with the highest probability, making the output deterministic (no randomness) but potentially repetitive or dull.
Explain that temperature modifies the distribution's sharpness: low temperature (e.g., 0.1) makes it peakier (more deterministic), high temperature (e.g., 1.5) flattens it (more random).
Top-k restricts sampling to the k most likely tokens, while top-p (nucleus) sampling restricts to the smallest set of tokens whose cumulative probability exceeds p. Both truncate the distribution to reduce randomness and avoid unlikely tokens.
Summarize how these methods balance diversity, coherence, and determinism, and give examples of when to use each (e.g., greedy for factual QA, top-p for creative writing).
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.