← American Express Interview Insights

American Express·AI Engineer·Onsite - System Design / Architecture·Senior

Senior
Jun 2026

Summary

System design round at Amex for an AI Engineer role, pretty deep on the infrastructure side of multi-agent systems. The whole session was basically one big question broken into layers, and they kept pushing further than I expected.

Questions Asked (6)

Q1

How would you design a multi-user AI agent system that keeps each user's context fully isolated while sharing underlying infrastructure like model servers, KV-cache pools, and vector stores?

System DesignTechnical Trade-offs
Author's notes

This is where I spent most of my time and still felt like I left things on the table.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by framing the core tension: sharing infrastructure for efficiency while enforcing strict per-user isolation. Then walk through a layered architecture—request routing, context management, and shared resource pooling—emphasizing how isolation is enforced at each layer (e.g., tenant IDs, namespacing, access controls). Finally, discuss trade-offs like cost, latency, and compliance, and how you'd validate isolation.

Pro tip: Mention that isolation must be enforced at multiple levels (API, data, and compute) and that you'd use automated tests to verify no cross-tenant leakage—this shows you think about security and reliability, not just design.

1. Clarify requirements and constraints

Ask about scale (number of users, requests per second), latency SLAs, data sensitivity (e.g., PII, financial data), and compliance requirements (e.g., GDPR, PCI-DSS). This shapes isolation strictness and sharing strategy.

2. Design the request routing and authentication layer

Propose an API gateway that authenticates users and attaches a tenant ID to each request. This ID is propagated through all downstream services to enforce isolation.

3. Isolate context at the application layer

Store per-user context (conversation history, preferences) in a dedicated namespace within a shared database or vector store. Use tenant ID as a partition key and enforce access controls so queries only return that user's data.

4. Share model servers and KV-cache with logical isolation

Use a shared model serving infrastructure (e.g., Triton, vLLM) with per-request KV-cache partitioning. Ensure that cache entries are tagged with tenant ID and evicted or isolated to prevent cross-user leakage.

5. Address trade-offs and validation

Discuss trade-offs: sharing reduces cost but may increase risk; strict isolation may add latency. Propose monitoring, auditing, and automated tests to verify isolation (e.g., penetration tests, canary requests).

Key Points to Mention

  • Tenant ID propagation and enforcement at every layer (API, data, compute).
  • Namespacing and access control in vector stores (e.g., separate collections or metadata filtering).
  • KV-cache isolation techniques: per-request cache keys, cache partitioning, or dedicated cache instances per tenant if needed.
  • Model server sharing: use of multi-model endpoints or dynamic batching with tenant-aware scheduling.
  • Trade-offs between isolation and efficiency: cost, latency, complexity, and compliance.
  • Validation: automated isolation tests, audit logs, and monitoring for cross-tenant access attempts.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q2

Walk me through how you'd implement fair queueing and rate limiting across concurrent users hitting shared model inference endpoints.

System DesignTechnical Trade-offs
Author's notes

Went with a token bucket per user plus a global weighted fair queue.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the requirements and constraints of the shared inference endpoints, then propose a layered architecture that combines rate limiting and fair queueing. Explain your design choices, trade-offs, and how you would monitor and adapt the system in production.

Pro tip: Emphasize the importance of fairness metrics and tenant isolation, and discuss how you would handle priority users or SLAs without starving others. Mention that you'd start with simple algorithms and iterate based on real traffic patterns.

1. Clarify Requirements and Constraints

Ask about expected traffic volume, user tiers, latency SLAs, and whether the system is multi-tenant. Understand if fairness means equal share per user, per API key, or weighted by priority.

2. Design Rate Limiting

Choose appropriate rate limiting algorithms (e.g., token bucket, leaky bucket, sliding window) and where to enforce them (API gateway, sidecar, or application layer). Consider distributed rate limiting using Redis or a centralized service.

3. Implement Fair Queueing

Select a fair queueing algorithm such as Weighted Fair Queueing (WFQ) or Deficit Round Robin (DRR) to allocate inference capacity. Ensure queues are per-user or per-tenant and that scheduling respects weights and priorities.

4. Address Concurrency and Backpressure

Handle concurrent requests with bounded queues and backpressure mechanisms (e.g., 429 responses, retry-after headers). Use async processing and worker pools to manage model inference concurrency.

5. Monitor, Test, and Iterate

Define metrics like per-user throughput, latency percentiles, and queue wait times. Simulate load tests and gradually roll out, adjusting parameters based on observed fairness and performance.

Key Points to Mention

  • Token bucket vs. leaky bucket for rate limiting and their trade-offs
  • Weighted Fair Queueing (WFQ) or Deficit Round Robin (DRR) for fair scheduling
  • Distributed rate limiting using Redis or centralized counters
  • Backpressure strategies: 429 status codes, Retry-After headers, and client-side retries with jitter
  • Multi-tenancy and isolation: per-user queues, resource quotas, and priority tiers
  • Observability: metrics for fairness (e.g., Jain's fairness index), latency, and throughput

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q3

How do you balance request batching for inference throughput against per-user latency SLAs when load is uneven?

System DesignTechnical Trade-offs
Author's notes

Blanked for a second here.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by framing the trade-off: batching improves throughput but can increase per-request latency, so you need a dynamic strategy. Then describe a concrete approach like adaptive batching with latency budgets, and explain how you'd monitor and adjust based on load patterns and SLAs.

Pro tip: Mention that you'd set a maximum batch size and a maximum wait time, and use a priority queue to ensure high-priority or latency-sensitive requests are batched separately or expedited. This shows you understand real-world constraints and customer impact.

1. Clarify requirements and constraints

Ask about the specific SLAs (e.g., p99 latency targets), the nature of the load (bursty vs. steady), and the cost implications of over-provisioning. This ensures your solution aligns with business needs.

2. Define batching strategy with latency budget

Propose a dynamic batching system where each request has a deadline (latency budget). The batcher waits up to a maximum time or until a maximum batch size is reached, whichever comes first.

3. Implement adaptive batching based on load

During high load, increase batch size to maximize throughput; during low load, reduce batch size or timeout to minimize latency. Use metrics like queue depth and request rate to adjust in real-time.

4. Prioritize and isolate latency-sensitive requests

Use separate queues or priority levels for different user tiers or request types. For example, premium users or real-time requests get smaller batches or dedicated resources.

5. Monitor, measure, and iterate

Continuously track latency percentiles (p50, p95, p99) and throughput. Set up alerts for SLA violations and use A/B testing or canary deployments to tune parameters.

Key Points to Mention

  • Dynamic batching with a maximum wait time (e.g., 10ms) and maximum batch size (e.g., 32) to bound latency.
  • Latency SLAs are often defined as percentiles (e.g., p99 < 100ms), so focus on tail latency, not just average.
  • Use of a priority queue or separate queues for different request classes to meet differentiated SLAs.
  • Adaptive batching: adjust batch size based on current load (e.g., using a PID controller or simple thresholds).
  • Trade-off between throughput and latency: larger batches improve GPU utilization but increase queueing delay.
  • Monitoring and observability: track queue wait time, batch size, and latency percentiles to inform tuning.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q4

If multiple concurrent sessions can update shared agent state, what consistency guarantees would you provide and how would you implement them?

System DesignAPI & Integrations
Author's notes

Probably my weakest answer.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the requirements and constraints, then propose a consistency model that balances correctness with performance. Describe a concrete implementation using versioning, optimistic concurrency control, and conflict resolution, and explain how you would handle edge cases and failures.

Pro tip: Emphasize that consistency guarantees should be driven by business needs—for example, financial data may require strong consistency, while less critical state can tolerate eventual consistency. Show you understand the trade-offs and can communicate them to stakeholders.

1. Clarify Requirements and Constraints

Ask about the nature of the shared state, expected concurrency levels, latency requirements, and tolerance for conflicts. Determine if the system is read-heavy or write-heavy and if there are regulatory or business requirements for consistency.

2. Choose a Consistency Model

Select an appropriate consistency model (e.g., strong, eventual, causal) based on requirements. Justify your choice by explaining the trade-offs between consistency, availability, and latency (CAP theorem).

3. Design the Implementation

Propose a concrete mechanism such as optimistic concurrency control with version numbers, pessimistic locking, or a distributed consensus protocol (e.g., Raft). Describe how updates are applied, how conflicts are detected and resolved, and how clients are notified.

4. Address Failure and Edge Cases

Discuss how the system handles network partitions, node failures, and concurrent conflicting updates. Explain retry logic, idempotency, and compensation actions if needed.

5. Evaluate Trade-offs and Alternatives

Summarize the pros and cons of your approach and mention alternative solutions. Highlight how your design meets the specific needs of the scenario.

Key Points to Mention

  • CAP theorem and the trade-offs between consistency, availability, and partition tolerance
  • Optimistic vs. pessimistic concurrency control
  • Versioning and conflict resolution strategies (e.g., last-write-wins, merge)
  • Distributed consensus algorithms (e.g., Raft, Paxos) for strong consistency
  • Idempotency and retry mechanisms to handle failures
  • Monitoring and alerting for consistency violations

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q5

How would you instrument a multi-user agent system for observability, specifically per-user distributed tracing across tool calls and model invocations?

System DesignProduct Analytics & Metrics
Author's notes

Easier than the others.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining the observability goals for a multi-user agent system, emphasizing per-user traceability across tool calls and model invocations. Then describe a distributed tracing architecture using unique trace IDs propagated through all components, with context enrichment for user identity and session data. Finally, discuss how to leverage this instrumentation for debugging, performance monitoring, and product analytics.

Pro tip: Highlight the importance of sampling strategies to balance observability costs with data completeness, and mention how you'd handle PII and compliance in a regulated industry like finance.

1. Define Observability Requirements

Identify key metrics (latency, error rates, token usage) and the need for per-user tracing to debug and optimize agent behavior. Consider compliance and data retention policies.

2. Design Trace Propagation

Use a unique trace ID per user session, propagated via headers or context objects across all services, including tool calls and model invocations. Ensure trace context is maintained in asynchronous and streaming scenarios.

3. Instrument Components

Add instrumentation to capture spans for each tool call and model invocation, including metadata like user ID, session ID, tool name, model name, and token counts. Use OpenTelemetry or similar standards for consistency.

4. Collect and Store Traces

Send traces to a distributed tracing backend (e.g., Jaeger, Zipkin) with appropriate sampling. Enrich traces with user context and ensure secure storage with access controls.

5. Analyze and Act

Use trace data to identify bottlenecks, errors, and user-specific issues. Build dashboards and alerts for key metrics, and feed insights back into product and engineering improvements.

Key Points to Mention

  • Unique trace ID per user session and propagation across services
  • Instrumentation of tool calls and model invocations with relevant metadata
  • Use of OpenTelemetry or similar standards for interoperability
  • Sampling strategies to manage cost and volume
  • Privacy and compliance considerations (e.g., PII redaction, data retention)
  • Integration with existing monitoring and analytics platforms

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q6

How would you test concurrency correctness in an AI agent system, particularly for race conditions in shared state and session isolation?

System DesignTechnical Trade-offs
Author's notes

Said chaos-style load tests with synthetic concurrent users plus deterministic replay tests where you inject specific interleavings.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the system architecture and the specific shared state and session boundaries, then outline a multi-layered testing strategy that combines static analysis, deterministic simulation, and stress testing. Emphasize how you would reproduce race conditions and verify session isolation under concurrent load, using tools and techniques tailored to AI agent systems.

Pro tip: Mention that you would first instrument the system with detailed logging and tracing to capture interleavings, then use deterministic replay to debug any failures—this shows you understand that concurrency bugs are often non-deterministic and hard to reproduce.

1. Identify Shared State and Session Boundaries

Map out all shared mutable state (e.g., conversation history, tool caches, model contexts) and define what constitutes a session. Determine the isolation requirements and potential contention points.

2. Design Deterministic Concurrency Tests

Create unit and integration tests that simulate concurrent access using controlled scheduling (e.g., barriers, latches) to force specific interleavings. Use mocking to isolate components and verify atomicity of operations.

3. Stress Test and Fuzz for Race Conditions

Run high-concurrency stress tests with many parallel sessions and random delays to surface race conditions. Employ fuzzing to generate diverse sequences of operations and monitor for data corruption or session leakage.

4. Verify Session Isolation

Test that sessions do not interfere by injecting unique markers into each session and asserting that no cross-session data appears. Use multi-tenant scenarios to ensure isolation under load.

5. Monitor and Debug in Production

Implement runtime monitoring for concurrency anomalies (e.g., deadlocks, inconsistent state) and use distributed tracing to diagnose issues. Plan for canary releases and rollback strategies.

Key Points to Mention

  • Use of thread-safe data structures and synchronization primitives (e.g., locks, atomic operations) in AI agent state management.
  • Deterministic simulation and record-replay techniques to reproduce race conditions.
  • Session isolation testing via unique session identifiers and data leakage checks.
  • Stress testing with tools like JMeter, Locust, or custom harnesses to simulate high concurrency.
  • Static analysis and formal verification methods (e.g., model checking) for critical sections.
  • Observability: logging, metrics, and tracing to detect and debug concurrency issues in production.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.