← Adobe Interview InsightsMachine Learning Engineer

Adobe·Machine Learning Engineer·Technical Phone Screen·Senior

Senior
May 2026

Summary

Adobe ML engineer interview that went pretty deep into transformer internals. The whole session felt like they wanted to see if you actually understood the architecture or just knew the buzzwords.

Questions Asked (4)

Q1

What is the difference between encoder-only and decoder-only transformer architectures, and what are they each used for?

Technical Trade-offsSystem Design
Author's notes

Felt okay on this one.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining the core architectural difference: encoder-only models use bidirectional attention to build rich contextual representations, while decoder-only models use causal (unidirectional) attention to generate text autoregressively. Then map each architecture to its typical use cases—encoder-only for understanding tasks like classification and embedding, decoder-only for generation tasks like chatbots and code synthesis. Finally, connect this to Adobe's needs by discussing trade-offs in latency, memory, and suitability for multimodal or creative applications.

Pro tip: Mention that decoder-only models have become dominant for generative AI due to their scalability and in-context learning, but encoder-only models remain superior for tasks requiring deep bidirectional understanding, such as retrieval or semantic search—a key component in Adobe's creative tools.

1. Define the architectures

Explain that encoder-only transformers (e.g., BERT) use bidirectional self-attention to encode input sequences, while decoder-only transformers (e.g., GPT) use masked self-attention to predict the next token autoregressively.

2. Explain attention mechanisms

Contrast bidirectional attention (sees full context) with causal attention (only sees previous tokens), and note how this affects training objectives: masked language modeling vs. next-token prediction.

3. Map to use cases

List typical applications: encoder-only for classification, named entity recognition, sentiment analysis, and embeddings; decoder-only for text generation, chatbots, code generation, and few-shot learning.

4. Discuss trade-offs and system design

Compare inference speed, memory usage, and suitability for real-time vs. batch processing. Mention that decoder-only models can be fine-tuned for understanding tasks but may underperform bidirectional models on some benchmarks.

5. Relate to Adobe's context

Connect to Adobe's products: encoder-only for content tagging, search, and recommendation; decoder-only for generative features like text-to-image or code assistants. Highlight hybrid approaches if relevant.

Key Points to Mention

  • Bidirectional vs. causal attention masking
  • Training objectives: masked language modeling vs. autoregressive generation
  • Examples: BERT (encoder-only) vs. GPT (decoder-only)
  • Use cases: understanding tasks (classification, NER) vs. generation tasks (chat, code)
  • Trade-offs: encoder-only often better for embeddings and low-latency retrieval; decoder-only excels at open-ended generation and in-context learning
  • Scalability and emergence of decoder-only models for large-scale generative AI

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q2

How does bidirectional attention differ from causal masked attention, and why does each approach make sense for its respective use case?

Technical Trade-offsAlgorithms & Data Structures
Author's notes

This is where I think I actually did well.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining both attention mechanisms clearly, then contrast their information flow and training objectives. Explain why bidirectional attention suits understanding tasks like classification or masked language modeling, while causal attention suits generation tasks like autoregressive language modeling. Use examples to illustrate the trade-offs.

Pro tip: Mention that bidirectional attention can be seen as a special case of causal attention with a full attention mask, and that some models like T5 use a unified approach with relative position biases to handle both. This shows depth beyond the basics.

1. Define Bidirectional Attention

Explain that in bidirectional attention, each token can attend to all other tokens in the sequence, both left and right. This is used in models like BERT for tasks that require full context understanding.

2. Define Causal Masked Attention

Explain that in causal attention, each token can only attend to previous tokens and itself, preventing information flow from future tokens. This is used in autoregressive models like GPT for generation tasks.

3. Compare Information Flow and Training Objective

Highlight that bidirectional attention allows full context, suitable for masked language modeling or classification, while causal attention enforces a left-to-right order, suitable for next-token prediction.

4. Relate to Use Cases

Discuss why each makes sense: bidirectional for understanding tasks where future context is available, causal for generation tasks where future tokens are unknown at inference.

5. Mention Trade-offs and Hybrid Approaches

Note that some models use hybrid attention (e.g., prefix LM) or relative position biases to combine benefits. Also mention computational implications: causal attention allows efficient incremental decoding.

Key Points to Mention

  • Bidirectional attention allows each token to attend to all positions, enabling full context.
  • Causal attention uses a triangular mask to prevent attending to future tokens.
  • Bidirectional is used in autoencoding models (e.g., BERT) for tasks like classification, QA, and masked language modeling.
  • Causal is used in autoregressive models (e.g., GPT) for text generation and language modeling.
  • Causal attention enables efficient left-to-right generation with KV caching.
  • Some models (e.g., T5, BART) use encoder-decoder architectures where the encoder uses bidirectional attention and the decoder uses causal attention.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q3

How does cross-attention work in an encoder-decoder model, specifically in terms of where the keys, values, and queries come from?

System DesignTechnical Trade-offs
Author's notes

Blanked for a second on the exact mechanics.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining cross-attention as the mechanism that lets the decoder attend to the encoder's output, then clearly state that queries come from the decoder while keys and values come from the encoder. Explain the step-by-step computation and highlight how this enables the decoder to focus on relevant parts of the input sequence.

Pro tip: Emphasize that cross-attention is crucial for tasks like machine translation and summarization, and mention that it allows the decoder to dynamically align with the encoder's representations, which is key for handling long-range dependencies.

1. Define cross-attention

Explain that cross-attention is an attention mechanism where the queries come from one sequence (the decoder) and the keys and values come from another sequence (the encoder).

2. Identify sources of Q, K, V

State that in cross-attention, the queries are derived from the decoder's previous layer output, while the keys and values are derived from the encoder's final output.

3. Describe the computation

Walk through how the attention scores are computed by taking the dot product of queries with keys, scaling, applying softmax, and then using the weights to combine the values.

4. Explain the purpose

Discuss how this allows the decoder to focus on relevant parts of the input sequence at each decoding step, enabling effective sequence-to-sequence learning.

5. Contrast with self-attention

Briefly contrast cross-attention with self-attention, where Q, K, and V all come from the same sequence, to highlight the difference.

Key Points to Mention

  • Queries come from the decoder, keys and values come from the encoder.
  • The attention scores are computed as softmax(QK^T / sqrt(d_k))V.
  • Cross-attention enables the decoder to attend to the entire input sequence, not just the previous outputs.
  • It is used in transformer-based encoder-decoder models like those for translation and summarization.
  • The encoder's output provides a rich representation that the decoder can query.
  • Cross-attention helps mitigate the bottleneck of fixed-length context vectors in traditional seq2seq models.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q4

Why don't encoder models need a KV cache during inference?

System DesignTechnical Trade-offs
Author's notes

Short answer and I gave a short answer.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by contrasting encoder and decoder architectures, emphasizing that encoders process the entire input in parallel with full bidirectional attention, so each token's representation is computed once without needing to remember past states. Then explain that KV caching is an optimization for autoregressive decoding, where each new token depends on all previous tokens, requiring cached keys and values to avoid recomputation. Conclude by noting that encoders are typically used for tasks like classification or embedding, not sequential generation, so caching is unnecessary.

Pro tip: Mention that while encoders don't need a KV cache, they can still benefit from other optimizations like attention masking or pruning, showing you understand the broader inference optimization landscape.

1. Define KV Cache

Explain that a KV cache stores key and value tensors from previous tokens to avoid recomputing them during autoregressive generation.

2. Describe Encoder Processing

Highlight that encoders process the entire input sequence in parallel, with each token attending to all others, so no sequential dependency exists.

3. Contrast with Decoder Inference

Contrast this with decoder-only models, where generation is autoregressive and each new token requires attending to all previous tokens, making caching essential.

4. Conclude on Necessity

Conclude that because encoders compute all token representations in one forward pass, there is no need to cache intermediate states for future steps.

Key Points to Mention

  • Encoder models use bidirectional attention, allowing each token to attend to all others in a single pass.
  • KV caching is an optimization for autoregressive decoding, where tokens are generated one at a time.
  • In encoders, the entire input is known upfront, so no sequential dependency exists.
  • Encoders are typically used for tasks like classification, embedding, or feature extraction, not generation.
  • Without autoregression, there is no need to store past keys and values for future use.
  • Caching would add unnecessary memory overhead without performance benefits.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.