← UiPath Interview Insights

UiPath·Machine Learning Engineer·Technical Phone Screen·Senior

SeniorPrefer not to say
May 2026

Summary

Interviewed for an MLE role at UiPath and got a pretty focused theory question on transformer internals. Not a lot of fluff, they went straight into the technical stuff.

Questions Asked (1)

Q1

For a sequence of length N with model dimension d, what is the time and memory complexity of self-attention?

Technical Trade-offsSystem DesignAlgorithms & Data Structures
Author's notes

I got the O(N² · d) time part right away but fumbled a bit explaining why memory is O(N²) specifically.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by stating the standard complexities: time O(N^2 * d) and memory O(N^2 + N * d). Then explain the derivation by breaking down the self-attention computation into its matrix operations, highlighting the quadratic dependence on sequence length due to the attention matrix. Finally, discuss practical implications and mention optimizations like sparse attention or linear attention that reduce complexity.

Pro tip: Emphasize that the quadratic complexity is a major bottleneck for long sequences, and mention that many efficient transformer variants aim to reduce this to O(N log N) or O(N). This shows awareness of current research and practical trade-offs.

1. State the standard complexities

Clearly state that self-attention has time complexity O(N^2 * d) and memory complexity O(N^2 + N * d) for a sequence of length N and model dimension d.

2. Derive time complexity

Explain that computing the attention scores involves multiplying Q (N x d) and K^T (d x N), resulting in an N x N matrix, which takes O(N^2 * d) time. Then applying softmax and multiplying by V (N x d) also takes O(N^2 * d) time.

3. Derive memory complexity

Note that storing the attention matrix requires O(N^2) memory, and storing the input/output projections and intermediate activations requires O(N * d) memory, leading to overall O(N^2 + N * d) memory.

4. Discuss implications and optimizations

Mention that the quadratic dependence on N makes self-attention expensive for long sequences, and briefly describe approaches like sparse attention, low-rank approximations, or linear attention that reduce complexity.

Key Points to Mention

  • Time complexity O(N^2 * d) due to matrix multiplications QK^T and attention*V.
  • Memory complexity O(N^2 + N * d) due to storing the attention matrix and activations.
  • The quadratic term N^2 dominates for long sequences, making it a bottleneck.
  • Optimizations: sparse attention, linear attention, low-rank approximations, etc.
  • Practical trade-offs: accuracy vs. efficiency, and the need for efficient implementations.
  • Comparison with other layers: e.g., feed-forward networks have O(N * d^2) complexity.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.