← Oracle Interview Insights

Oracle·Machine Learning Engineer·Technical Phone Screen·Senior

SeniorPrefer not to say
May 2026

Summary

Oracle ML Engineer interview that was basically one big coding deep-dive into attention mechanisms. No fluff, just 'implement this from scratch and explain every decision.' Felt like a grad school exam more than a job interview.

Questions Asked (1)

Q1

Implement multi-head self-attention from scratch in PyTorch or NumPy, without using any built-in attention modules. Walk through projecting inputs into Q, K, V, reshaping for multiple heads, computing scaled dot-product attention, and recombining the heads with a final linear projection. Also discuss how you'd handle causal and padding masks, and what the computational complexity looks like.

System DesignTechnical Trade-offsAlgorithms & Data Structures
Author's notes

This was the whole interview, basically.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by outlining the overall architecture of multi-head self-attention, then implement it step-by-step in code, explaining each operation. Emphasize mask handling and complexity analysis to demonstrate depth.

Pro tip: Mention that you can use a single linear layer for Q, K, V projections by splitting the output, which is more efficient and common in practice. Also, highlight that scaling by sqrt(d_k) is crucial for stable gradients.

1. Project inputs to Q, K, V

Use linear layers to project the input tensor into query, key, and value spaces. Typically, these are combined into one linear layer for efficiency.

2. Reshape for multiple heads

Reshape the projected tensors to split the feature dimension into multiple heads, resulting in shape (batch, heads, seq_len, head_dim).

3. Compute scaled dot-product attention

Compute attention scores as Q @ K^T / sqrt(head_dim), apply masks if needed, then softmax and multiply by V.

4. Recombine heads and final projection

Concatenate the heads back together and apply a final linear projection to produce the output.

5. Discuss masks and complexity

Explain how to implement causal and padding masks, and analyze the O(n^2 * d) time complexity and O(n^2) memory complexity.

Key Points to Mention

  • Scaling factor 1/sqrt(d_k) to prevent softmax saturation
  • Causal mask: upper triangular matrix with -inf to prevent attending to future tokens
  • Padding mask: mask out padding tokens to avoid attending to them
  • Complexity: O(n^2 * d) time and O(n^2) memory for sequence length n
  • Efficiency: combining Q, K, V projections into one linear layer
  • Numerical stability: using -inf for masked positions before softmax

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.