← Amazon Interview InsightsMachine Learning Engineer

Amazon·Machine Learning Engineer·Technical Phone Screen·Intermediate

Intermediate
May 2026

Summary

Interviewed for an ML Engineer role at Amazon, got asked a pretty foundational question about the difference between inference and training. Short screen, nothing too crazy.

Questions Asked (1)

Q1

What is the difference between inference and training in machine learning?

Technical Trade-offsSystem Design
Author's notes

Knew this one but fumbled the delivery a bit.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining training and inference at a high level, then contrast them across key dimensions like computational patterns, resource requirements, and system design implications. Emphasize how these differences influence real-world ML system architecture, especially in a production context like Amazon.

Pro tip: Highlight that inference often demands low-latency, high-throughput serving with optimized models, while training focuses on throughput and convergence; mentioning techniques like model quantization or batch vs. real-time inference shows practical maturity.

1. Define training and inference

Clearly state that training is the process of learning model parameters from data, while inference is using the trained model to make predictions on new data.

2. Contrast computational characteristics

Explain that training involves forward and backward passes, gradient updates, and large datasets, whereas inference typically involves only forward passes and smaller batches or single instances.

3. Discuss resource and infrastructure differences

Mention that training requires significant compute (GPUs/TPUs), memory, and storage for checkpoints, while inference prioritizes low latency, high throughput, and cost-efficient deployment (e.g., CPU vs. GPU, edge vs. cloud).

4. Highlight system design implications

Explain how these differences affect architecture: training pipelines focus on data processing and distributed training, while inference systems emphasize model serving, scaling, and monitoring.

5. Connect to trade-offs and optimizations

Discuss common trade-offs such as model size vs. latency, and optimizations like quantization, pruning, or batching that are more relevant for inference than training.

Key Points to Mention

  • Training requires labeled data and iterative optimization; inference uses the trained model on unseen data.
  • Computational patterns: training uses forward and backward propagation; inference uses only forward propagation.
  • Resource intensity: training is compute-heavy and often distributed; inference is latency-sensitive and optimized for cost.
  • Infrastructure: training on powerful GPUs/TPUs with large memory; inference may use CPUs, GPUs, or specialized accelerators like AWS Inferentia.
  • Model optimization techniques: quantization, pruning, and knowledge distillation are primarily applied for inference.
  • Serving considerations: batch vs. real-time inference, auto-scaling, and monitoring for drift are critical in production.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.