← Uber Interview InsightsMachine Learning Engineer

Uber·Machine Learning Engineer·Technical Phone Screen·Senior

Senior
Jun 2026

Summary

Uber ML engineer round that was pretty much one focused coding question on implementing CLIP-style contrastive loss. Felt like a niche topic if you haven't touched multimodal stuff before, but the math itself isn't too bad once you see the pattern.

Questions Asked (1)

Q1

Implement a contrastive loss function for a vision-language model given a batch of image and text embeddings. The loss should be symmetric: compute cross-entropy over the similarity matrix for images, do the same transposed for texts, then average them.

Algorithms & Data StructuresTechnical Trade-offs
Author's notes

I knew the general idea behind contrastive learning but had never written this specific formulation from scratch under pressure.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the setup: a batch of N image and N text embeddings, and the goal is a symmetric contrastive loss (like CLIP). Then walk through the implementation step-by-step: normalize embeddings, compute the N×N similarity matrix with a temperature scale, apply cross-entropy in both directions (rows for images, columns for texts), and average the two losses. Finally, discuss practical considerations like numerical stability, temperature as a learnable parameter, and handling of in-batch negatives.

Pro tip: Mention that the temperature parameter is typically learned (as in CLIP) and that using logits = similarity / temperature helps with numerical stability; also note that the loss implicitly uses all other samples in the batch as negatives, so batch size matters.

1. Clarify inputs and goal

Confirm that you have a batch of N image embeddings and N text embeddings, and the loss should treat the paired (image_i, text_i) as positives and all other combinations as negatives. State that the loss is symmetric: average of image-to-text and text-to-image cross-entropy.

2. Normalize and compute similarity matrix

L2-normalize the embeddings along the feature dimension to ensure cosine similarity. Compute the N×N similarity matrix as the dot product of normalized image and text embeddings, optionally scaled by a temperature parameter (logit_scale).

3. Compute cross-entropy in both directions

For images: apply cross-entropy loss with logits as rows of the similarity matrix and labels as the diagonal indices (0 to N-1). For texts: apply cross-entropy on the transpose of the similarity matrix with the same labels. Average the two losses.

4. Address numerical stability and implementation details

Use stable cross-entropy implementations (e.g., log_softmax) and consider clamping the temperature or using a learnable logit_scale. Mention that the loss can be computed efficiently with matrix operations and that no explicit negative sampling is needed.

5. Discuss trade-offs and extensions

Talk about how batch size affects the number of negatives, the role of temperature, and potential improvements like hard negative mining or using a memory bank. Also mention that the symmetric loss helps balance the two modalities.

Key Points to Mention

  • L2 normalization of embeddings to compute cosine similarity
  • Temperature scaling (logit_scale) and its learnable nature
  • Cross-entropy with diagonal labels for both directions
  • Symmetric loss averaging to balance modalities
  • In-batch negatives and the effect of batch size
  • Numerical stability (log_softmax, clamping)

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.