← Affirm Interview Insights

Affirm·Software Engineer·Onsite - System Design / Architecture·Senior

Senior
May 2026

Summary

System design round at Affirm for a software engineer role. The whole session was basically one big question about building an A/B testing platform from scratch, and it went deeper than I expected.

Questions Asked (6)

Q1

Design an A/B testing platform that allows product teams to run controlled experiments at scale, covering experiment configuration, user assignment, metrics collection, and statistical analysis.

A/B Testing & ExperimentationSystem DesignTechnical Trade-offs
Author's notes

This one sprawled fast.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements and scale, then walk through the end-to-end architecture: experiment configuration, deterministic user assignment, metrics collection, and statistical analysis. Emphasize trade-offs between consistency, latency, and accuracy, and how you would handle common pitfalls like sample ratio mismatch and network effects.

Pro tip: Always discuss how you would prevent metric dilution and ensure experiment isolation—e.g., using a dedicated assignment service with consistent hashing and a separate metrics pipeline to avoid interference. Also, mention the importance of guardrail metrics and automated shutdown for harmful experiments.

1. Clarify Requirements and Scale

Ask about the number of experiments, users, metrics, and latency requirements. Understand whether the platform needs real-time assignment, batch analysis, or both.

2. Design Experiment Configuration and Management

Outline how product teams define experiments: targeting rules, variants, allocation percentages, and metrics. Consider versioning, approval workflows, and a UI/API for configuration.

3. Implement User Assignment and Bucketing

Describe a deterministic assignment service using consistent hashing (e.g., on user ID + experiment salt) to ensure stable bucketing. Discuss caching, fallbacks, and handling of overlapping experiments.

4. Build Metrics Collection and Data Pipeline

Explain how to collect exposure and metric events (e.g., via logging or streaming), aggregate them, and store in a data warehouse. Address data quality, deduplication, and real-time vs. batch processing.

5. Perform Statistical Analysis and Reporting

Cover statistical methods (frequentist vs. Bayesian), power analysis, sequential testing, and multiple comparison corrections. Discuss how to surface results, detect anomalies, and automate decisions.

Key Points to Mention

  • Deterministic hashing for consistent user assignment and avoiding bias
  • Sample ratio mismatch (SRM) detection and mitigation
  • Metrics pipeline design: event ingestion, aggregation, and storage
  • Statistical rigor: p-values, confidence intervals, power, and sequential testing
  • Guardrail metrics and automated experiment shutdown for safety
  • Scalability considerations: caching, sharding, and handling high throughput

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q2

How would you handle the assignment service hot path to keep latency under 10ms at the 99th percentile?

System DesignTechnical Trade-offs
Author's notes

Talked about pushing experiment configs to edge caches so assignment never hits a central DB.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the scope and constraints of the assignment service, then systematically break down the hot path into components and propose optimizations for each. Emphasize a data-driven approach: measure, identify bottlenecks, and apply targeted solutions while considering trade-offs.

Pro tip: Quantify the impact of each optimization and discuss how you would validate the 99th percentile latency, showing you understand the importance of measurement and iteration. Also, mention that you'd consider the cost and complexity trade-offs, as over-engineering can be as harmful as under-engineering.

1. Clarify Requirements and Scope

Ask questions to understand the assignment service's responsibilities, expected load, data size, and existing architecture. Confirm that the 10ms p99 latency is for the entire service or specific operations.

2. Profile and Identify Bottlenecks

Describe how you would measure current latency and identify bottlenecks using profiling tools, tracing, and metrics. Focus on the hot path: from request ingestion to response.

3. Optimize Critical Components

Propose optimizations for each component: e.g., caching, database indexing, query optimization, asynchronous processing, connection pooling, and reducing network hops.

4. Consider Architectural Changes

Discuss higher-level changes like sharding, read replicas, denormalization, or moving to an in-memory data store. Evaluate trade-offs between consistency, availability, and latency.

5. Validate and Iterate

Explain how you would test the changes under load, monitor p99 latency, and set up alerts. Emphasize continuous improvement and regression testing.

Key Points to Mention

  • Caching strategies (e.g., Redis, local cache) with appropriate invalidation policies
  • Database optimizations: indexing, query tuning, connection pooling, and read replicas
  • Asynchronous processing and queueing to offload non-critical work
  • Code-level optimizations: reducing allocations, efficient data structures, and avoiding unnecessary serialization
  • Infrastructure considerations: CDN, load balancing, and network latency
  • Monitoring and observability: distributed tracing, metrics, and p99 latency tracking

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q3

Walk me through how you'd handle the high volume of exposure and conversion events flowing into the platform.

System DesignA/B Testing & Experimentation
Author's notes

Standard event firehose setup, Kafka in front, consumers writing to a columnar store.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the scale and requirements (e.g., events per second, latency, accuracy) and then propose a high-level architecture that decouples ingestion, processing, and storage. Focus on scalability, reliability, and real-time analytics, and discuss trade-offs between different technologies and approaches.

Pro tip: Emphasize the importance of exactly-once processing and idempotency to avoid double-counting events, which is critical for accurate A/B testing and conversion metrics. Also, mention the need for monitoring and alerting on lag and error rates to ensure data quality.

1. Clarify Requirements

Ask about the expected volume (events per second), latency requirements (real-time vs. batch), data retention, and accuracy needs (e.g., exactly-once semantics).

2. Design Ingestion Layer

Propose a scalable, fault-tolerant ingestion system using a distributed message queue like Apache Kafka or AWS Kinesis to handle high throughput and buffer events.

3. Processing and Enrichment

Describe a stream processing layer (e.g., Apache Flink, Spark Streaming, or Kafka Streams) to transform, enrich, and aggregate events in real-time, ensuring exactly-once processing with checkpointing and idempotent writes.

4. Storage and Serving

Choose appropriate storage for different use cases: a data lake (e.g., S3) for raw events, a data warehouse (e.g., Redshift, BigQuery) for analytics, and a real-time database (e.g., Druid, ClickHouse) for low-latency queries.

5. Monitoring and Scalability

Discuss monitoring (e.g., lag, throughput, error rates), auto-scaling, and disaster recovery. Mention partitioning strategies and backpressure handling.

Key Points to Mention

  • Use of distributed messaging systems (Kafka, Kinesis) for ingestion
  • Stream processing frameworks (Flink, Spark Streaming) for real-time processing
  • Exactly-once processing semantics and idempotency to avoid data duplication
  • Scalable storage solutions (data lake, warehouse, real-time DB) for different access patterns
  • Monitoring and alerting on key metrics (lag, throughput, errors)
  • Partitioning and scaling strategies to handle high volume

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q4

How would you deal with late-arriving events in your metrics pipeline, and what impact does that have on your analysis?

A/B Testing & ExperimentationProduct Analytics & MetricsTechnical Trade-offs
Author's notes

This is where I got a bit stuck.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining late-arriving events and their common causes in a metrics pipeline. Then outline a technical strategy to handle them, such as windowing and watermarks, and discuss the trade-offs between accuracy and latency. Finally, explain the impact on analysis, including potential biases and how to mitigate them.

Pro tip: Emphasize that the choice of handling late data depends on the business question: for real-time dashboards, approximate results may suffice, but for A/B testing, accuracy is critical. Mention that you would monitor the volume of late events and adjust watermarks dynamically.

1. Define the problem

Explain what late-arriving events are and why they occur (e.g., network delays, mobile offline mode, clock skew).

2. Technical handling strategies

Describe methods like event-time processing, watermarks, allowed lateness, and reprocessing. Mention tools like Apache Flink, Beam, or Kafka Streams.

3. Trade-offs

Discuss the trade-off between latency and completeness: waiting longer improves accuracy but delays results. Also consider cost and complexity.

4. Impact on analysis

Explain how late data can skew metrics, especially in A/B tests, leading to false positives/negatives. Mention the need for statistical corrections or waiting for data completeness.

5. Monitoring and iteration

Propose monitoring late event rates and adjusting strategies dynamically. Suggest backfilling or reprocessing for critical analyses.

Key Points to Mention

  • Event time vs. processing time
  • Watermarks and allowed lateness
  • Windowing strategies (tumbling, sliding, session)
  • Impact on A/B testing: novelty effects, sample ratio mismatch, peeking
  • Trade-offs: latency vs. accuracy, cost vs. completeness
  • Tools: Apache Flink, Beam, Kafka Streams, or custom solutions

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q5

How would you evolve this platform to support ML-driven multi-armed bandits instead of traditional fixed-allocation A/B tests?

A/B Testing & ExperimentationSystem DesignTechnical Trade-offs
Author's notes

Genuinely enjoyed this part.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the current A/B testing architecture and the goals for adopting multi-armed bandits, such as faster convergence and improved user experience. Then outline a phased evolution: from offline simulation to a shadow mode, and finally to a production system with a bandit service that integrates with existing experimentation infrastructure. Emphasize trade-offs around statistical rigor, engineering complexity, and business metrics.

Pro tip: Highlight the importance of guardrail metrics and a fallback mechanism to fixed A/B tests to mitigate risks, and mention that bandits require a shift from batch to online learning, which impacts data pipelines and model deployment.

1. Clarify Requirements and Constraints

Ask about the current experimentation platform, scale, latency requirements, and business objectives to tailor the solution. Identify key metrics and guardrails.

2. Design the Bandit Service Architecture

Propose a microservice that handles arm selection, reward computation, and model updates. Ensure it integrates with existing feature flags, logging, and monitoring systems.

3. Choose Bandit Algorithms and Exploration Strategy

Discuss options like Thompson Sampling or UCB, and how to balance exploration and exploitation. Consider contextual bandits if user features are available.

4. Address Data and Feedback Loops

Explain how to collect rewards in near real-time, handle delayed conversions, and avoid bias. Implement logging for offline evaluation and counterfactual analysis.

5. Plan Rollout and Risk Mitigation

Propose a phased rollout: offline simulation, shadow mode, small-scale online test, then full deployment. Include guardrails, fallback to A/B tests, and monitoring for drift.

Key Points to Mention

  • Trade-offs between statistical power and adaptivity: bandits reduce regret but complicate inference.
  • Infrastructure changes: need for low-latency decision service, real-time data pipelines, and model serving.
  • Algorithm selection: Thompson Sampling, UCB, or contextual bandits based on use case.
  • Guardrail metrics and safety mechanisms to prevent degradation of user experience.
  • Integration with existing experimentation platform and gradual migration path.
  • Monitoring and evaluation: how to measure bandit performance and detect issues like feedback loops.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q6

How would you detect and handle a sample ratio mismatch in a running experiment?

A/B Testing & ExperimentationProduct Analytics & Metrics
Author's notes

SRM checks were something I'd read about but never implemented.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by explaining what a sample ratio mismatch (SRM) is and why it's critical to detect early in an experiment. Then outline a systematic approach: monitor the ratio continuously, use statistical tests to detect deviations, and have a clear protocol for handling SRM when it occurs. Emphasize the importance of root cause analysis and preventing invalid results from influencing decisions.

Pro tip: Proactively mention that you would set up automated alerts for SRM using a chi-squared test with a low p-value threshold (e.g., p < 0.001) to avoid false positives, and that you'd document the incident and communicate transparently with stakeholders.

1. Define expected ratio and monitoring

Establish the expected traffic split (e.g., 50/50) and set up real-time monitoring of the actual ratio as data accumulates. Use dashboards and automated alerts to track deviations.

2. Statistical detection

Apply a statistical test such as the chi-squared goodness-of-fit test to compare observed vs. expected sample sizes. Use a stringent significance level (e.g., p < 0.001) to minimize false alarms.

3. Investigate root cause

If SRM is detected, immediately investigate potential causes: bugs in randomization, data pipeline issues, bot traffic, or uneven exposure due to logging errors. Check logs, code, and data quality.

4. Mitigate and decide

Depending on the cause, either fix the issue and restart the experiment, or if the mismatch is due to a known benign factor, adjust analysis methods (e.g., use weighted metrics). If invalid, discard results and communicate.

5. Prevent recurrence

Implement safeguards such as automated SRM checks in the experimentation platform, regular audits, and training to prevent future mismatches.

Key Points to Mention

  • Definition of SRM: when the observed sample sizes in experiment groups deviate from the expected ratio.
  • Statistical tests: chi-squared test, sequential testing, or Bayesian methods for detection.
  • Common causes: randomization bugs, data pipeline issues, bot traffic, uneven logging, or user behavior differences.
  • Impact: SRM can invalidate experiment results, leading to false conclusions and poor business decisions.
  • Handling: stop experiment, investigate, fix, and potentially restart; avoid analyzing biased data.
  • Prevention: automated monitoring, alerts, and robust experiment design.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.