The greedy part was straightforward, argmax in a loop until you hit EOS, not much to overthink.
Start by clarifying the interface and constraints: the model's forward method returns logits for the next token, and we need to generate tokens until EOS. For greedy decoding, simply take argmax at each step; for top-k sampling, select the top-k logits, renormalize them into probabilities, and sample from that distribution. Ensure both functions return only the generated tokens (excluding the initial prompt) and stop when EOS is produced.
Pro tip: Mention that top-k sampling should be done without replacement and that you must handle the case where k is larger than the vocabulary size by capping k at vocab size. Also, emphasize that the model is deterministic, so greedy decoding will always produce the same sequence, which is useful for testing.
Review the provided GPT-like class: it has a fixed vocab size, an EOS token, and a forward method that returns logits given input tokens. Note that the forward method is deterministic.
Iteratively call forward with the current sequence, take argmax of the logits to get the next token, append it to the generated tokens, and stop if it equals EOS.
At each step, get logits, select the top-k values, renormalize them into probabilities (e.g., using softmax), sample one token from this distribution, append it, and stop if it equals EOS.
Ensure k is capped at vocab size, handle empty generation, and return only the generated tokens (excluding the initial input).
Test both functions with the given input and verify that the generated sequences match the expected exact sequences.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.