Mentioned the causal mask and that seemed to be the answer they were looking for.
Start by defining the core architectural difference: encoders use bidirectional self-attention to build rich contextual representations, while decoders use masked self-attention and cross-attention to generate sequences autoregressively. Then connect this to practical implications for system design and trade-offs, such as latency, parallelism, and use cases.
Pro tip: Emphasize that the decoder's causal masking is what enables autoregressive generation, and mention that encoder-only models are often better for representation learning while decoder-only models dominate generative tasks—this shows you understand the architectural spectrum beyond just the textbook definition.
Explain that an encoder processes the entire input sequence in parallel using bidirectional self-attention, allowing each token to attend to all others. This produces contextualized representations suitable for tasks like classification or encoding.
Describe that a decoder generates output sequences one token at a time, using masked self-attention to prevent attending to future tokens, and cross-attention to incorporate encoder outputs when present.
Contrast the attention mechanisms: encoders use unmasked bidirectional attention, while decoders use masked (causal) attention. This difference dictates whether the model can see the full context or must generate autoregressively.
Mention that transformers can be encoder-only (e.g., BERT), decoder-only (e.g., GPT), or encoder-decoder (e.g., T5). This shows awareness of how the components are combined in practice.
Explain implications: encoders enable parallel processing and are efficient for representation tasks, while decoders are inherently sequential during inference, affecting latency and throughput. This ties back to real-world ML engineering decisions.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.