Went straight to sampling and temperature, which is the obvious answer, but I made sure to actually explain the mechanism rather than just name-dropping the terms.
Start by clarifying that the question is about non-determinism in inference, not training. Then explain the primary sources: stochastic sampling methods and non-deterministic GPU operations, and optionally mention other factors like dropout if the model is in training mode. Conclude with how to achieve reproducibility and the trade-offs involved.
Pro tip: Emphasize that in production, especially in finance, reproducibility is critical for debugging and compliance, so you often need to set seeds and use deterministic algorithms, but this may cost performance. Show awareness of the trade-off between creativity and determinism.
Confirm that the model is in inference mode and that the input is truly identical. Distinguish between training and inference, as dropout and other regularization are typically disabled during inference.
Explain that decoder-based models often use stochastic decoding strategies like sampling with temperature, top-k, or nucleus sampling, which introduce randomness. Also, non-deterministic GPU operations (e.g., atomic adds) can cause slight numerical differences that amplify over layers.
Mention other potential sources: dropout if the model is inadvertently in training mode, random seed initialization for any random operations, and framework-specific behaviors (e.g., cuDNN autotuner selecting different algorithms).
Describe steps to make outputs reproducible: set random seeds, use deterministic algorithms (e.g., torch.use_deterministic_algorithms(True)), disable sampling (use greedy or beam search), and ensure consistent hardware/software environments.
Discuss the trade-offs: deterministic settings may reduce performance or diversity. In finance, reproducibility is often required for auditing, so determinism is preferred, but for creative tasks, stochasticity is desired.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.