← Expedia Interview Insights

Expedia·Machine Learning Engineer·Onsite - System Design / Architecture·Senior

Senior
May 2026

Summary

System design round at Expedia for an MLE role, focused almost entirely on building a large-scale event ingestion and transformation pipeline. It went deep fast and covered a lot of ground, more like a data engineering interview than anything ML-specific.

Questions Asked (6)

Q1

Design a data pipeline that ingests events from many upstream producers, transforms them, and delivers them to analytics-ready storage. Walk through the full system end to end.

System DesignTechnical Trade-offsData Modeling
Author's notes

This was the main question and it basically ate the whole session.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements and constraints, then walk through the pipeline stages: ingestion, processing, storage, and serving. Emphasize trade-offs at each stage, especially around scalability, fault tolerance, and data quality, and tie decisions back to ML use cases like feature freshness and training reproducibility.

Pro tip: Anchor your design around concrete SLAs (e.g., end-to-end latency, data freshness) and explicitly discuss how you'd monitor and evolve the pipeline over time—this shows operational maturity beyond just architecture diagrams.

1. Clarify Requirements and Constraints

Ask about data volume, velocity, variety, latency requirements, and downstream ML use cases. Confirm non-functional needs like fault tolerance, cost, and compliance.

2. Design Ingestion Layer

Choose between batch, streaming, or hybrid ingestion. Discuss producers, message queues (e.g., Kafka, Kinesis), schema registry, and handling of late/duplicate events.

3. Design Transformation and Processing

Outline ETL/ELT steps: validation, enrichment, aggregation, and feature computation. Decide between stream processing (e.g., Flink, Spark Streaming) and batch (e.g., Spark, dbt).

4. Design Storage and Serving Layers

Select storage for raw, processed, and analytics-ready data (e.g., data lake, warehouse, feature store). Ensure partitioning, indexing, and access patterns support ML training and serving.

5. Address Reliability, Monitoring, and Evolution

Discuss fault tolerance, exactly-once semantics, backpressure, data quality checks, lineage, and how to evolve schema and pipeline without downtime.

Key Points to Mention

  • Trade-offs between batch and streaming processing (latency vs. throughput vs. complexity)
  • Exactly-once semantics and idempotency in ingestion and processing
  • Schema evolution and data contract management across producers
  • Partitioning and storage format choices (e.g., Parquet, Delta Lake) for efficient analytics
  • Feature store integration for ML training/serving consistency
  • Monitoring, alerting, and data quality validation (e.g., Great Expectations)

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q2

How would you handle exactly-once delivery semantics across the pipeline, and where would you relax that to at-least-once and why?

System DesignTechnical Trade-offs
Author's notes

Blanked for a second.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining exactly-once semantics as idempotent processing with transactional guarantees, then map where they are essential (e.g., billing, final model updates) versus where at-least-once with deduplication suffices (e.g., logging, feature aggregation). Emphasize trade-offs between complexity, latency, and cost, and propose a hybrid architecture with idempotent sinks and checkpointing.

Pro tip: Demonstrate maturity by acknowledging that exactly-once is often a distributed systems illusion; focus on end-to-end idempotency and business impact rather than chasing theoretical guarantees.

1. Clarify semantics and requirements

Define what exactly-once means in the pipeline context and identify business-critical stages where duplicates or data loss are unacceptable.

2. Design for exactly-once where needed

Use idempotent writes, transactional sinks, and checkpointing (e.g., Kafka transactions, Flink two-phase commit) to achieve exactly-once for critical stages.

3. Relax to at-least-once with deduplication

For non-critical stages, allow at-least-once delivery and handle duplicates via idempotent consumers or deduplication keys to reduce complexity and cost.

4. Monitor and validate end-to-end

Implement monitoring for duplicates, lag, and data quality, and use reconciliation jobs to detect and correct inconsistencies.

5. Iterate based on business impact

Continuously evaluate where exactly-once is truly needed and adjust based on cost, latency, and evolving requirements.

Key Points to Mention

  • Idempotency as the foundation for exactly-once semantics
  • Transactional guarantees in stream processing frameworks (e.g., Kafka, Flink)
  • Trade-offs: exactly-once increases latency, cost, and complexity
  • At-least-once with deduplication is often sufficient for analytics and logging
  • Checkpointing and state management for fault tolerance
  • Business impact: financial transactions vs. model training data

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q3

How do you approach schema evolution in a high-throughput event pipeline without breaking downstream consumers?

System DesignTechnical Trade-offs
Author's notes

Schema registry with backward-compatible changes was my answer, Confluent-style.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by acknowledging the tension between schema flexibility and consumer stability, then outline a strategy that combines schema registry, compatibility rules, and versioning. Emphasize proactive governance and consumer-driven contracts to prevent breakage, and mention how you'd handle ML-specific concerns like feature drift and model retraining.

Pro tip: Always design for backward and forward compatibility, but also implement a canary deployment for schema changes and monitor consumer lag and error rates to catch issues early. In ML pipelines, treat feature schemas as part of the model contract and version them alongside models.

1. Assess current state and requirements

Understand the existing pipeline, consumer expectations, and throughput requirements. Identify critical consumers and their tolerance for schema changes.

2. Adopt a schema registry and compatibility policy

Use a schema registry (e.g., Confluent Schema Registry) to enforce compatibility rules (backward, forward, full). Define a policy that balances flexibility with safety.

3. Design schema evolution patterns

Choose patterns like adding optional fields, using union types, or versioned topics. For breaking changes, use dual-write or migration windows with consumer coordination.

4. Implement monitoring and rollback

Monitor consumer lag, deserialization errors, and data quality. Set up alerts and a rollback plan for schema changes that cause issues.

5. Align with ML lifecycle

Treat feature schemas as model contracts. Version them with models, and ensure training/serving consistency. Use feature stores to manage schema evolution centrally.

Key Points to Mention

  • Schema registry and compatibility types (backward, forward, full)
  • Consumer-driven contracts and impact analysis
  • Versioning strategies (topic per version, schema IDs in messages)
  • Monitoring and observability (consumer lag, error rates, data quality)
  • ML-specific concerns: feature drift, model retraining, training-serving skew
  • Trade-offs between flexibility and stability, and cost of breaking changes

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q4

Walk me through your storage layer choices, specifically around table formats and the tradeoffs between options like Iceberg, Delta, and Hudi on top of object storage.

System DesignTechnical Trade-offsData Modeling
Author's notes

This was the part I felt most confident in.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by framing the storage layer as a critical component for ML reproducibility and feature consistency, then compare Iceberg, Delta, and Hudi based on Expedia's needs like ACID transactions, schema evolution, and time travel. Conclude with a recommendation that balances tradeoffs and aligns with ML workflows such as feature stores and model training.

Pro tip: Emphasize that the choice often depends on ecosystem integration and operational maturity—e.g., Delta's tight Spark integration vs. Iceberg's engine-agnostic design—and mention how you'd validate with a proof-of-concept on real ML pipelines.

1. Clarify Requirements

Ask about data volume, update patterns, query engines, and ML-specific needs like point-in-time correctness and feature versioning.

2. Compare Table Formats

Highlight key differences: Iceberg's hidden partitioning and engine neutrality, Delta's Spark optimization and unified batch/streaming, Hudi's upsert efficiency and incremental processing.

3. Evaluate Tradeoffs

Discuss tradeoffs in performance, cost, complexity, and ecosystem support, tying them to ML use cases like feature engineering and model training.

4. Recommend and Justify

Propose a format (or combination) with rationale, considering Expedia's existing infrastructure and team expertise.

5. Address Migration and Operations

Briefly cover how you'd handle migration, monitoring, and maintenance to ensure long-term success.

Key Points to Mention

  • ACID transactions and isolation levels for concurrent ML workloads
  • Schema evolution and enforcement for feature consistency
  • Time travel and snapshot isolation for reproducibility
  • File layout and metadata management impact on query performance
  • Integration with processing engines (Spark, Flink, Trino) and ML frameworks
  • Operational overhead and community support for each format

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q5

How would you design the observability and data quality layer for this pipeline, including SLA monitoring and handling bad data?

System DesignProduct Analytics & Metrics
Author's notes

Talked about dead-letter queues for malformed events, row count and null rate checks at each stage, and alerting on lag metrics in the streaming layer.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the pipeline's purpose and SLAs, then propose a layered observability architecture covering data quality, pipeline health, and model performance. Emphasize proactive monitoring, automated bad data handling, and alignment with business SLAs.

Pro tip: Tie every metric and alert to a business impact—like booking conversion or customer experience—to show you think beyond technical thresholds. Also, mention that you'd implement data quality checks as code and version them alongside the pipeline for reproducibility.

1. Define SLAs and Data Contracts

Work with stakeholders to define clear SLAs for data freshness, completeness, and accuracy, and establish data contracts with upstream producers.

2. Implement Data Quality Checks

Embed automated checks (e.g., schema validation, null checks, distribution drift) at ingestion and transformation stages, using tools like Great Expectations or Deequ.

3. Set Up Monitoring and Alerting

Monitor pipeline health (latency, throughput, errors) and data quality metrics, with alerts routed to on-call and dashboards for visibility.

4. Handle Bad Data Gracefully

Define strategies for bad data: quarantine, dead-letter queues, fallback to last good data, and automated retries with backoff.

5. Close the Loop with Model Performance

Monitor model predictions for drift and degradation, linking data quality issues to model performance and triggering retraining when needed.

Key Points to Mention

  • Data quality dimensions: accuracy, completeness, timeliness, consistency, validity, uniqueness
  • Tools: Great Expectations, Deequ, Monte Carlo, Apache Airflow for orchestration, Prometheus/Grafana for monitoring
  • SLA monitoring: define SLIs (e.g., % of data arriving on time), SLOs, and error budgets
  • Bad data handling: quarantine tables, dead-letter queues, alerting, and manual review workflows
  • Model observability: drift detection, performance metrics, and feedback loops
  • Cost and scalability considerations: sampling, tiered storage, and efficient alerting

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q6

How do you think about cost optimization and scaling for a pipeline processing millions of events per second?

System DesignTechnical Trade-offs
Author's notes

Honestly the question I was least prepared for.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the pipeline's goals and constraints (latency, throughput, cost budget), then propose a layered architecture that decouples ingestion, processing, and storage to independently scale and optimize each. Discuss trade-offs between cost and performance, and highlight specific techniques like partitioning, batching, and autoscaling with concrete examples from ML pipelines.

Pro tip: Emphasize that cost optimization is not just about reducing resources but about aligning spend with business value—e.g., using spot instances for non-critical batch jobs while ensuring SLA-critical paths remain reliable. Also, mention that scaling for millions of events per second often requires a streaming-first mindset with backpressure handling and dead-letter queues to avoid data loss during spikes.

1. Clarify Requirements and Constraints

Ask about latency requirements, data volume patterns, budget constraints, and SLAs to frame the problem. This ensures your solution is tailored to Expedia's travel platform needs, such as real-time pricing or recommendation updates.

2. Design for Horizontal Scalability

Propose a distributed architecture using sharding, partitioning (e.g., by user ID or event type), and parallel processing. Leverage managed services like Kafka, Kinesis, or Pub/Sub for ingestion and Flink or Spark Streaming for processing.

3. Implement Cost Optimization Techniques

Discuss strategies like right-sizing instances, using spot instances for fault-tolerant workloads, autoscaling based on load, and tiered storage (hot vs. cold). Also, consider data compression, efficient serialization (e.g., Avro, Protobuf), and batching to reduce overhead.

4. Address Trade-offs and Monitoring

Explain how you balance cost, latency, and reliability—e.g., choosing eventual consistency for cost savings vs. strong consistency for critical data. Describe monitoring and alerting on cost metrics (e.g., cost per event) and performance to continuously optimize.

5. Tie Back to ML-Specific Considerations

Highlight ML pipeline aspects like feature engineering, model inference, and retraining. For example, use batch inference for cost efficiency where real-time isn't needed, and optimize feature stores to avoid redundant computations.

Key Points to Mention

  • Partitioning and sharding strategies to distribute load and enable parallel processing.
  • Autoscaling and spot instances to match resource allocation with demand and reduce costs.
  • Data compression, efficient serialization formats, and batching to lower I/O and network overhead.
  • Trade-offs between latency, throughput, cost, and consistency (e.g., CAP theorem implications).
  • Use of managed streaming services (Kafka, Kinesis) and serverless options for operational efficiency.
  • Monitoring and observability with cost metrics (e.g., cost per million events) to drive continuous optimization.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.