← Affirm Interview Insights

Affirm·Software Engineer·Onsite - System Design / Architecture·Senior

Senior
Jun 2026

Summary

System design round at Affirm for a software engineer role. The whole thing was one big question about building an A/B testing platform from scratch, which sounds manageable until you realize how many moving pieces they actually want you to cover.

Questions Asked (4)

Q1

Design an A/B testing platform for a large product organization, covering experiment definition, user assignment, exposure logging, metric collection, analysis, and experiment lifecycle management.

A/B Testing & ExperimentationSystem DesignTechnical Trade-offs
Author's notes

This is a beast of a question and I underestimated the scope going in.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements and scale, then walk through the end-to-end architecture covering experiment definition, deterministic user assignment, exposure logging, metric collection, and analysis. Emphasize trade-offs between consistency, latency, and statistical validity, and discuss lifecycle management including guardrails and rollout.

Pro tip: Highlight the importance of a single source of truth for experiment assignments and exposures, and explain how you would handle cross-device consistency and metric trustworthiness through logging and validation.

1. Clarify Requirements and Scale

Ask about scale (users, experiments, metrics), latency requirements, and consistency needs. Define key terms like experiment, variant, exposure, and metric.

2. Design Experiment Definition and Assignment

Propose a service for experiment configuration and a deterministic assignment service using hashing (e.g., user ID + experiment ID) to ensure consistent bucketing. Discuss targeting and segmentation.

3. Implement Exposure Logging and Metric Collection

Design a low-latency exposure logging pipeline (e.g., via Kafka) and a metric collection system that aggregates user actions. Ensure exposure events are logged only when the experiment is active and the user is exposed.

4. Build Analysis and Reporting

Outline a batch or streaming analysis pipeline that computes metrics per variant, performs statistical tests (e.g., t-test, sequential testing), and generates reports. Include guardrail metrics and anomaly detection.

5. Manage Experiment Lifecycle

Describe the lifecycle: draft, review, ramp-up, run, analyze, and conclude. Include features like kill switches, automated stopping rules, and integration with feature flags for rollout.

Key Points to Mention

  • Deterministic hashing for consistent user assignment and avoiding sample ratio mismatch (SRM).
  • Exposure logging only when the experiment is active and the user is actually exposed to the variant.
  • Metric collection with idempotency and exactly-once semantics to avoid double counting.
  • Statistical methods: frequentist vs. Bayesian, sequential testing, and multiple comparison corrections.
  • Guardrail metrics and automated alerts for degradation.
  • Integration with feature flags and CI/CD for seamless rollout and rollback.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q2

How would you handle deterministic, low-latency user bucketing that stays consistent across sessions and remains mutually exclusive across overlapping experiments?

A/B Testing & ExperimentationSystem DesignAlgorithms & Data Structures
Author's notes

The hash plus salt approach came to me pretty fast.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements: deterministic hashing, low latency, session consistency, and mutual exclusivity. Then propose a layered architecture using a stable hash of user ID and experiment salt, with a centralized experiment registry to enforce exclusivity. Finally, discuss trade-offs and optimizations like caching and precomputed assignments.

Pro tip: Emphasize that mutual exclusivity is often achieved by assigning users to non-overlapping experiment layers or using a global experiment registry with conflict detection. Mention that consistent hashing with a salt per experiment ensures deterministic bucketing without coordination.

1. Clarify Requirements and Constraints

Confirm the need for determinism, low latency (e.g., <10ms), cross-session consistency, and mutual exclusivity. Ask about scale, number of experiments, and whether users can be in multiple experiments.

2. Design Deterministic Bucketing

Use a hash function (e.g., MurmurHash) on a composite key of user ID and experiment salt to produce a uniform bucket number. Ensure the hash is stable across sessions and services.

3. Enforce Mutual Exclusivity

Implement a centralized experiment registry that tracks active experiments and their overlapping user segments. Use layer-based assignment or conflict detection to prevent users from being in overlapping experiments.

4. Optimize for Low Latency

Cache bucketing results in memory or a fast store (e.g., Redis) with TTL. Precompute assignments for large user sets if needed. Use efficient data structures for quick lookups.

5. Address Edge Cases and Scalability

Handle experiment changes (e.g., ramping up) by rehashing with a new salt or using consistent hashing. Ensure the system scales horizontally and maintains consistency during deployments.

Key Points to Mention

  • Deterministic hashing (e.g., MurmurHash, SHA-256) with user ID and experiment salt
  • Consistent bucketing across sessions via stable user identifiers
  • Mutual exclusivity through experiment layers or a global registry
  • Low-latency caching strategies (in-memory, Redis) and precomputation
  • Handling experiment changes (ramping, stopping) without breaking consistency
  • Trade-offs between simplicity and flexibility (e.g., salt per experiment vs. global salt)

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q3

What guardrails would you put in place to catch sample ratio mismatch or peeking problems in your analysis engine?

A/B Testing & ExperimentationProduct Analytics & Metrics
Author's notes

I knew what SRM was but explaining it under pressure made me sound less confident than I actually am.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by acknowledging that sample ratio mismatch (SRM) and peeking are critical validity threats in A/B testing, then describe a layered defense: automated statistical checks, real-time monitoring, and cultural/process guardrails. Emphasize that these guardrails should be integrated into the analysis engine to prevent false positives and ensure trustworthy results.

Pro tip: Mention that SRM detection should be based on a chi-squared test with a pre-registered threshold (e.g., p < 0.001) and that peeking can be mitigated by using sequential testing or always-valid p-values, which are more robust than fixed-horizon tests.

1. Automated SRM Detection

Implement a daily (or real-time) chi-squared test comparing observed vs. expected sample sizes per variant. If p-value falls below a strict threshold (e.g., 0.001), flag the experiment and halt analysis.

2. Peeking Prevention via Sequential Testing

Use sequential testing methods (e.g., group sequential boundaries, alpha spending) or always-valid inference to allow continuous monitoring without inflating Type I error.

3. Real-time Monitoring & Alerts

Build dashboards that track sample ratios and p-values over time, with automated alerts to experiment owners when anomalies are detected.

4. Pre-registration & Analysis Lock

Require pre-registration of primary metrics, sample size, and analysis plan. Lock the analysis until the planned sample size is reached, unless sequential testing is used.

5. Education & Process Enforcement

Train teams on the risks of peeking and SRM, and enforce guardrails through code reviews, experiment review boards, and automated checks in the deployment pipeline.

Key Points to Mention

  • Sample Ratio Mismatch (SRM) definition and common causes (e.g., bot traffic, logging errors, uneven randomization).
  • Chi-squared test for SRM detection and appropriate significance thresholds.
  • Peeking problem: inflated false positive rate when repeatedly checking results.
  • Sequential testing, alpha spending, or always-valid p-values as solutions.
  • Automated monitoring and alerting systems for early detection.
  • Pre-registration of analysis plans and sample size to prevent p-hacking.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q4

How would you design the exposure logging system to handle billions of events per day with minimal overhead on the request path?

System DesignA/B Testing & ExperimentationTechnical Trade-offs
Author's notes

Async logging was the obvious answer and I said it immediately.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements: scale (billions/day), latency budget, and consistency needs. Then propose an asynchronous, client-side logging pipeline that decouples event capture from processing, using a write-optimized buffer like Kafka and a scalable storage layer. Emphasize trade-offs between overhead, cost, and data completeness.

Pro tip: Mention that you would sample or aggregate events at the edge when full fidelity isn't needed, and use a separate logging endpoint to avoid impacting critical request paths. This shows you prioritize system reliability and cost-efficiency.

1. Clarify Requirements and Constraints

Ask about event volume, latency tolerance, data retention, and consistency requirements. Confirm that minimal overhead means low latency impact on user requests.

2. Design the Client-Side Capture

Propose a lightweight client that batches events and sends them asynchronously (e.g., via HTTP or UDP) to a dedicated logging endpoint, avoiding blocking the main request.

3. Build a Scalable Ingestion Pipeline

Use a distributed message queue like Kafka to buffer events, enabling decoupling and backpressure handling. Ensure the pipeline can scale horizontally.

4. Choose Storage and Processing

Store raw events in a cost-effective, scalable store (e.g., S3, HDFS) and process them with a stream processor (e.g., Flink, Spark Streaming) for real-time analytics or batch jobs.

5. Address Reliability and Trade-offs

Discuss handling failures (retries, dead-letter queues), monitoring, and trade-offs between overhead, cost, and data loss. Mention sampling or aggregation if full fidelity isn't required.

Key Points to Mention

  • Asynchronous, non-blocking logging to avoid request path impact
  • Use of a distributed message queue (e.g., Kafka) for buffering and decoupling
  • Horizontal scalability and partitioning strategies
  • Cost-effective storage and tiered retention policies
  • Sampling or aggregation to reduce volume when appropriate
  • Monitoring, alerting, and failure handling (e.g., dead-letter queues)

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.